Thank you for the PM xyzzy as I got lost in all of the insanity. Ensuring people aren't penalized for WUs because of the upgrade outages is something I had already promised.

For all WU's with a report_deadline between 2011-10-09 00:00:00 - 2011-10-13 00:00:00 GMT I have added 48 hours to the report_deadline. Actually this would have decreased server load earlier.

What happens when the deadline passes is a 2nd WU is generated and sent out. Whichever WU is reported first is granted credit and the server sends an abort at the next network communication to the other client. This is the default BOINC system and in some cases may unfairly punish the 2nd cruncher.

feedback requested: If you have fallen victim to the server outages and think the adjustment needs to be different from the above then please post here or PM me. We absolutely do not want to have earned credit lost because of issues on our side.

The validator can have the following states for a result:
Initial (waiting for validation)
Valid
Invalid
Skipped
Inconclusive (doesn't apply as we don't use things like adaptive replication)
Too Late

If the result arrived after another user had turned one in (that was valid) then you would see Too Late.

Invalid means the validator found a problem with the result: either the result is bad or the validator itself has problems. Though, I see that you generally have valid results. I generated the same workunit and got the same result:
md5sums:
b7cc894a0b222ecb5d749f0d73a97a01 7488199 md5_mixalpha-numeric-all-space#1-8_24_422000x500000 - 147301000000_0_0
b7cc894a0b222ecb5d749f0d73a97a01 result

That result was also marked invalid a good 20 hours before the 2nd result was received.

Actually your result is more mysterious than just that:
distrrtgen_validator1.log.0.gz:2011-10-11 03:54:05.8529 [CRITICAL] [RESULT#14953884 7488199 md5_mixalpha-numeric-all-space#1-8_24_422000x500000 - 147301000000_0]charset mixalpha-numeric-all-space (1 - 8) not supported
distrrtgen_validator1.log.0.gz:2011-10-11 03:54:05.8543 [RESULT#14953884 7488199 md5_mixalpha-numeric-all-space#1-8_24_422000x500000 - 147301000000_0] Invalid [HOST#25256]

I've got 1 other report of an invalid WU that I looked at that had this error. It doesn't make any sense though because if the character set was not supported - no results would validate.

The system did crash within about 20 minutes of that validation and may have already been on some sort of death spiral. I have several more invalid results to check out in this thread and via pms. If I find this case repeating itself then there is some serious investigating to do.

Hrm, first see my message id 137 in response to Michael.

distrrtgen_validator1.log.0.gz:2011-10-11 03:59:49.1006 [CRITICAL] [RESULT#15147774 7625669 md5_mixalpha-numeric-all-space#1-8_32_422000x500000 - 55056000000_1]charset mixalpha-numeric-all-space (1 - 8) not supported
distrrtgen_validator1.log.0.gz:2011-10-11 03:59:49.1030 [RESULT#15147774 7625669 md5_mixalpha-numeric-all-space#1-8_32_422000x500000 - 55056000000_1] Invalid [HOST#24211]

distrrtgen_validator1.log.0.gz:2011-10-11 03:49:25.7456 [CRITICAL] [RESULT#15137890 7654087 md5_mixalpha-numeric-all-space#1-8_32_422000x500000 - 66765000000_0]charset mixalpha-numeric-all-space (1 - 8) not supported
distrrtgen_validator1.log.0.gz:2011-10-11 03:49:25.7474 [RESULT#15137890 7654087 md5_mixalpha-numeric-all-space#1-8_32_422000x500000 - 66765000000_0] Invalid [HOST#6870]

distrrtgen_validator2.log:2011-10-11 03:35:03.1444 [CRITICAL] [RESULT#15135790 7652082 md5_mixalpha-numeric-all-space#1-8_32_422000x500000 - 65762500000_0]charset mixalpha-numeric-all-space (1 - 8) not supported
2011-10-11 03:35:03.1460 [RESULT#15135790 7652082 md5_mixalpha-numeric-all-space#1-8_32_422000x500000 - 65762500000_0] Invalid [HOST#6870]

7649798 is weirder yet - logs for the valid one but not for when yours was marked invalid

distrrtgen_validator1.log.0.gz:2011-10-11 03:38:21.0794 [CRITICAL] [RESULT#15132931 7649347 md5_mixalpha-numeric-all-space#1-8_32_422000x500000 - 64395000000_0]charset mixalpha-numeric-all-space (1 - 8) not supported
distrrtgen_validator1.log.0.gz:2011-10-11 03:38:21.0811 [RESULT#15132931 7649347 md5_mixalpha-numeric-all-space#1-8_32_422000x500000 - 64395000000_0] Invalid [HOST#24211]

I find 9 invalid WUs for you. With 1 exception they were all reported between 11 Oct 2011 3:39:46 UTC and 11 Oct 2011 4:17:39 UTC which is just prior to the server crash. It also looks like the same mystery of my last two responses in this thread.

Oh, should have mentioned before I replied to anyone in this thread there was already a bug ticket regarding this specific validation failure. It doesn't seem to be unique to any host, user, WU, etc. at this point.

I looked up the first result and it is the same server side validation error where it fails to find the character set as valid. I just went through the code and there are several messages that do not get logged that would let us find if the failure is earlier in the process and narrow it down.

We are having server stability issues that is leading to the system rebooting about every 24 or 48 hours. I suspect this is a kernel panic (the system is set to restart on a panic instead of staying hung) and am also working to capture the core file for the panic since nothing in the logs is of any use.

Debug data gave us the exact information we needed on where and what the problem is. It is still a mysterious problem but one that a full work around to prevent it from recurring is easy to do. If Martin doesn't get the chance to get the fix deployed before heads to bed then I'll take care of it after work this evening in ~4hours.

What's UT?

Ok patch deployed to the validator. I will keep an eye on the logs. However, if anyone starts getting new batches of invalid WUs then alert me and I'll check that we haven't found some other issue. The gist: sometimes the validator would be unable to read charset.txt (rather odd) but it was also reading and parsing the file for *every* WU it validated *every* time. In any case it has been integrated into the code to avoid whatever caused it to fail reading and decent chunk of wasted I/O and performance.

3 Boxes with Validation errors here.
____________

Both tasks suffered from the issue prior to my fix. Anything failing validation *after* 2011-10-18 00:44 UTC indicates either a new issue, problem with the fix, or real validation failure.

I show 3 WUs that failed validation in the last 2 hours project wide. Your last validation failure was 17 Oct 2011 19:50:40 UTC. I don't show any aborted, compute error, etc. WUs for you since then either. Am I missing something?

cuda support for hybrid2 is in 3.45 and so a mix of the standard long cuda WUs and some of the short 60k chain length hybrid2 WUs are being distributed. That chain length is appropriate for CPU only use of completed tables. (Even a 200k chain length completed table is essentially unusable without a GPU). I'm still clinging to attempting to balance CPU and GPU users both from a generation perspective and from a usability perspective of the final product. I own 1 and only 1 GPU and I purchased it for dev work for this project but have many more cpus and cpu cores than 1.

From my GTX470 (x86_64 linux) the 422k md5 WUs run in about 1,081.05 seconds (about 18 minutes) and the 60k ntlm hybrid2 WUs run in about 153.74 seconds (about 2.5 minutes). In this case it seems to scale correctly on my card as 1081.05/153.74 is ~7.03x and 422/60 (chain length - or 4220/600 credit if you prefer) is about 7.03x. The reason I said a mix of them is being sent is because at that chain length my ability to generate with just that 1 GPU exceeds my upload bandwidth. (There are people connected to this project with enough GPU power that even at all 422k length chains they are uploading essentially 24 hours a day).

In practice while it should scale like that it does not seem to especially on the windows side. Though, the win64 cuda builds end up having a high rate of failure so the win32 builds may actually be the biggest factor. Of course I probably should benchmark again with 3.45 and compare both platforms.

Time for some sobering and perhaps painful news. The validation failures before the server side fix will most likely stay as they are. We offer our deepest apologies but that of course does not replace cobblestones.

The intent of my original post was to handle a relatively easy condition: increasing the report_deadline *before* results for a WU expired and a new result was sent out. I wanted feedback for anyone needing me to bump it by another 24 hours or something of that nature.

Once a result times out or is marked invalid then a new result is generated and sent out. While things become more difficult at that point, results can still be re-validated (assuming you know it was some temporary server side issue and catch it in time), but once quorum is reached (1 for us) and that result and WU are marked valid then things are near impossible to fix. The valid result and WU have the uploaded result deleted fairly soon after validation.

In order to re-run a result through the validator you have to mark both the result *and* WU as needing validation. However, that forces it to try to validate the already valid result that no longer has the data file and it in turn gets marked as invalid. I tried a variety of approaches of changes the quorum, deadlines, max success results, etc. etc. The only solution would be to essentially rewrite the validator into a very strange one off to try to rerun just the invalid result, grant credit, ignore various times and count limits, and other fun things. At this point I have already spent near 10 hours across 2 evenings just to realize the only approach to fixing it is what I was trying to avoid in the first place.

In 10 days 2.8% of the results have been marked as failing validation. Though, the majority of this is actually concentrated on a few time frames. This has impacted up to 760 users of 1537 active users. Of these users here is the breakdown on number of results invalid:
< 5 invalid results: 366 users
< 10 invalid results: 512 users
< 20 invalid results: 622 users
> 20 and < 40: 60 users
> 40 and < 60: 26 users
> 60: 41 users

I can count myself as one of the 60 in the 20 to 40 invalid result range (to the tune of about 7 hours of lost GPU credits).

I certainly do not enjoy writing notes such as this one. However, the validation problem on the server side has been resolved. I have made the data public so that I can get feedback on if 5, 10, or even 60 lost result credits is something we can all live with. I also do not believe in trying to hide the scope of problems as there is nothing to be gained by such a practice, and did not know the scope was this large until getting the data to write this post.

quel

Hrm all 4 of those are legitimately invalid. I regenerated them and don't match your results. The validation failures all look like this (well failures at different parts of the results but similar issues):
2011-10-19 17:19:10.1741 [CRITICAL] [RESULT#15603915 8069215 ntlm_hybrid2(alpha#1-1,loweralpha#5-5,loweralpha-numeric#2-2,numeric#1-3)#0-0_3_60000x500000 - 63329000000_0] Endpoint index verification failed at step 57343 with number 50637919046268187 > keyspace 444393878722560
2011-10-19 17:19:10.1762 [RESULT#15603915 8069215 ntlm_hybrid2(alpha#1-1,loweralpha#5-5,loweralpha-numeric#2-2,numeric#1-3)#0-0_3_60000x500000 - 63329000000_0] Invalid [HOST#18218]

Checking the end points (8 bytes each at 500000 per WU) are within the keyspace is one of the validation checks. Those were all hybrid2 WUs so I additionally went and found a non-hybrid2 WU and you're hitting the same problem. It seems that some of your WUs are coming back as valid though.

Looks like that host has 3 GPUs. We had code that would hit GPU > 0 harder (assumption was headless but we do have users with 4 GPUs and 4 monitors) but apparently we have no logging data about which GPU was selected.

I'm going to need more data (and likely at least have more debug data in general if not a custom debug build for you to gather more data) and anything you can detail is helpful. Feel free to PM instead of posting here anything that might help such as:
Any changes you've had that might play a role (or if this isn't an issue in other projects that's good to know too)
Historically how the system has handled this project since I take it you have some successful history in the past.

I'm getting mostly invalids today the 24th as well as a bunch back on the 16th-17th. I'm now past 60 invalids. That's 15+ hours of wasted crunching. Please fix this and the #46 & #75 bug ticket bugs once and for all. Or at least manually compensate in the boinc server credit settings for the rest of the 422000s until you do. It takes almost 3x longer than the 200000s so credit should be in the 5500-5800 range for the actual run time at the moment compared to the 200000s run time. Once run times are fixed than you can revert back to the linear credit code. After that you need to fix the server stability and other GPU app issues before anything else. People elsewhere complaining about this too.

status update