Stats has lost one of my processors

[bapriebe]

Out of the blue I discover that the stats page for "Brian_Priebe" lists only 4 active processors when 5 have been working for weeks. One was "lost" in the last week as it's the 7 days stats that are in error. As near as I can tell from FAHLOG's,they are all working just fine except for a surprise "UNSTABLE_MACHINE" report on WU 4747 with the GPU2 client on August 30. (Everything is totally stock clock-wise here and always has been.) GPU2 has been grinding away just fine ever since and returning results without incident.

Any ideas?

[bruce]

Here is a list of the last WU received from each of your machines

0B212ADB69XXXXXXXX Windows-Vista 2008-08-29 02:22:13 2451 132 28 17
333954D315XXXXXXXX Windows-XP 2008-09-04 20:12:17 4594 18 182 29
2A92623976XXXXXXXX Windows-XP 2008-09-05 04:39:21 5113 18 55 0
0B212ADB69XXXXXXXX Windows-Vista 2008-09-05 06:23:01 2452 11 34 16
333954D315XXXXXXXX PS3 2008-09-05 14:34:54 4743 3 32 2

If you post FAHlog from the top machine on the list showing what it has been doing since 2008-08-29 we may be able to help you.

[bapriebe]

0B212ADB69XXXXXXXX Windows-Vista 2008-08-29 02:22:13 2451 132 28 17
333954D315XXXXXXXX Windows-XP 2008-09-04 20:12:17 4594 18 182 29
2A92623976XXXXXXXX Windows-XP 2008-09-05 04:39:21 5113 18 55 0
0B212ADB69XXXXXXXX Windows-Vista 2008-09-05 06:23:01 2452 11 34 16
333954D315XXXXXXXX PS3 2008-09-05 14:34:54 4743 3 32 2

This synopsis made me scratch my head. I gather the prefix of the user ID is the first field on each line but I have exactly zero PS3's and Vista machines here. The stats collector does not understand ATI GPU2 nor Windows Server 2003? I can't make out what the timestamps are. The returned WU project 2452 (Run 11, Clone 34, Gen 16) from machine ID #2 on the same processor that hosts the laggard machine ID #1 was posted at:

Code: Select all

[12:44:37] Sending work to server
[12:44:37] Project: 2452 (Run 11, Clone 34, Gen 16)
[12:44:37] - Read packet limit of 540015616... Set to 524286976.
[12:44:37] + Attempting to send results [September 5 12:44:37 UTC]
[12:44:37] - Reading file work/wuresults_04.dat from core
[12:44:37]   (Read 2441179 bytes from disk)
[12:44:37] Connecting to http://171.65.103.162:8080/
[12:44:57] Posted data.
[12:44:57] Initial: 0000; - Uploaded at ~113 kB/s
[12:44:58] - Averaged speed for that direction ~93 kB/s
[12:44:58] + Results successfully sent
[12:44:58] Thank you for your contribution to Folding@Home.

(Where did this info come from incidentally? When I click on Stats Official I can't drill down past the total count of work units returned for a particular project let alone attribute them to a particular user ID and machine ID.)

The top processor is a 1GHz Pentium III running the 6.20 CPU client under Windows Server 2003. It had trouble on September 1 returning a work unit (4 retries required) and had to retry 9 times to get new work. It did manage to upload the stuck WU to 171.64.122.76:8080 the same day (September 1) it completed it at 22:53:26 UTC. The missing stats are for Project: 2621 (Run 88, Clone 55, Gen 33). It's been working on a new WU ever since so hasn't transmitted anything for the last 4 days.

Code: Select all

<snip until August 29>

[07:43:10] Folding@home Core Shutdown: FINISHED_UNIT
[07:43:13] CoreStatus = 64 (100)
[07:43:13] Unit 3 finished with 92 percent of time to deadline remaining.
[07:43:13] Updated performance fraction: 0.913094
[07:43:13] Sending work to server
[07:43:13] Project: 2451 (Run 132, Clone 28, Gen 17)
[07:43:13] - Read packet limit of 540015616... Set to 524286976.


[07:43:13] + Attempting to send results [August 29 07:43:13 UTC]
[07:43:13] - Reading file work/wuresults_03.dat from core
[07:43:13]   (Read 2677553 bytes from disk)
[07:43:13] Connecting to http://171.65.103.162:8080/
[07:43:35] Posted data.
[07:43:35] Initial: 0000; - Uploaded at ~113 kB/s
[07:43:36] - Averaged speed for that direction ~112 kB/s
[07:43:36] + Results successfully sent
[07:43:36] Thank you for your contribution to Folding@Home.

[07:43:36] + Number of Units Completed: 3

[07:43:41] Trying to send all finished work units
[07:43:41] + No unsent completed units remaining.
[07:43:41] - Preparing to get new work unit...
[07:43:41] + Attempting to get work packet
[07:43:41] - Will indicate memory of 1279 MB
[07:43:41] - Connecting to assignment server
[07:43:41] Connecting to http://assign.stanford.edu:8080/
[07:43:41] Posted data.
[07:43:41] Initial: 40AB; - Successful: assigned to (171.64.65.65).
[07:43:41] + News From Folding@Home: Welcome to Folding@Home
[07:43:41] Loaded queue successfully.
[07:43:41] Connecting to http://171.64.65.65:8080/
[07:43:47] Posted data.
[07:43:47] Initial: 0000; - Receiving payload (expected size: 3641343)
[07:44:04] - Downloaded at ~209 kB/s
[07:44:04] - Averaged speed for that direction ~305 kB/s
[07:44:04] + Received work.
[07:44:04] Trying to send all finished work units
[07:44:04] + No unsent completed units remaining.
[07:44:04] + Closed connections
[07:44:04]
[07:44:04] + Processing work unit
[07:44:04] Core required: FahCore_78.exe
[07:44:04] Core found.
[07:44:04] Working on queue slot 04 [August 29 07:44:04 UTC]
[07:44:04] + Working ...
[07:44:04] - Calling '.\FahCore_78.exe -dir work/ -suffix 04 -checkpoint 15 -verbose -lifeline 1988 -version 620'

[07:44:04]
[07:44:04] *------------------------------*
[07:44:04] Folding@Home Gromacs Core
[07:44:04] Version 1.90 (March 8, 2006)
[07:44:04]
[07:44:04] Preparing to commence simulation
[07:44:04] - Looking at optimizations...
[07:44:04] - Created dyn
[07:44:04] - Files status OK
[07:44:25] - Expanded 3640831 -> 18751629 (decompressed 515.0 percent)
[07:44:26] - Starting from initial work packet
[07:44:26]
[07:44:26] Project: 2621 (Run 88, Clone 55, Gen 33)
[07:44:26]
[07:44:26] Assembly optimizations on if available.
[07:44:26] Entering M.D.
[07:44:37] Protein: p2621_p1475_tet1_03_1 t= 20000.00000
[07:44:37]
[07:44:37] Writing local files
[07:44:55] Extra SSE boost OK.
[07:44:57] Writing local files
[07:44:57] Completed 0 out of 125000 steps  (0%)

<snip...fast forward to September 1>

[18:43:56] Completed 125000 out of 125000 steps  (100%)
[18:43:57] Writing final coordinates.
[18:44:02] Past main M.D. loop
[18:45:02]
[18:45:02] Finished Work Unit:
[18:45:02] - Reading up to 3024264 from "work/wudata_04.arc": Read 3024264
[18:45:02] - Reading up to 5236712 from "work/wudata_04.xtc": Read 5236712
[18:45:02] goefile size: 0
[18:45:02] logfile size: 36330
[18:45:02] Leaving Run
[18:45:05] - Writing 8346834 bytes of core data to disk...
[18:45:23] Done: 8346322 -> 8122903 (compressed to 97.3 percent)
[18:45:24]   ... Done.
[18:45:25] - Shutting down core
[18:45:25]
[18:45:25] Folding@home Core Shutdown: FINISHED_UNIT
[18:45:28] CoreStatus = 64 (100)
[18:45:28] Unit 4 finished with 91 percent of time to deadline remaining.
[18:45:28] Updated performance fraction: 0.913200
[18:45:28] Sending work to server
[18:45:28] Project: 2621 (Run 88, Clone 55, Gen 33)
[18:45:28] - Read packet limit of 540015616... Set to 524286976.


[18:45:28] + Attempting to send results [September 1 18:45:28 UTC]
[18:45:28] - Reading file work/wuresults_04.dat from core
[18:45:28]   (Read 8123415 bytes from disk)
[18:45:28] Connecting to http://171.64.65.65:8080/
[18:45:28] - Couldn't send HTTP request to server
[18:45:28]   (Got status 503)
[18:45:28] + Could not connect to Work Server (results)
[18:45:28]     (171.64.65.65:8080)
[18:45:28] + Retrying using alternative port
[18:45:28] Connecting to http://171.64.65.65:80/
[18:45:28] - Couldn't send HTTP request to server
[18:45:28]   (Got status 503)
[18:45:28] + Could not connect to Work Server (results)
[18:45:28]     (171.64.65.65:80)
[18:45:28] - Error: Could not transmit unit 04 (completed September 1) to work server.
[18:45:28] - 1 failed uploads of this unit.
[18:45:28]   Keeping unit 04 in queue.
[18:45:28] Trying to send all finished work units
[18:45:28] Project: 2621 (Run 88, Clone 55, Gen 33)
[18:45:28] - Read packet limit of 540015616... Set to 524286976.


[18:45:28] + Attempting to send results [September 1 18:45:28 UTC]
[18:45:28] - Reading file work/wuresults_04.dat from core
[18:45:28]   (Read 8123415 bytes from disk)
[18:45:28] Connecting to http://171.64.65.65:8080/
[18:45:28] - Couldn't send HTTP request to server
[18:45:28]   (Got status 503)
[18:45:28] + Could not connect to Work Server (results)
[18:45:28]     (171.64.65.65:8080)
[18:45:28] + Retrying using alternative port
[18:45:28] Connecting to http://171.64.65.65:80/
[18:45:29] - Couldn't send HTTP request to server
[18:45:29]   (Got status 503)
[18:45:29] + Could not connect to Work Server (results)
[18:45:29]     (171.64.65.65:80)
[18:45:29] - Error: Could not transmit unit 04 (completed September 1) to work server.
[18:45:29] - 2 failed uploads of this unit.
[18:45:29] - Read packet limit of 540015616... Set to 524286976.


[18:45:29] + Attempting to send results [September 1 18:45:29 UTC]
[18:45:29] - Reading file work/wuresults_04.dat from core
[18:45:29]   (Read 8123415 bytes from disk)
[18:45:29] Connecting to http://171.64.122.76:8080/
[18:45:29] - Couldn't send HTTP request to server
[18:45:29]   (Got status 503)
[18:45:29] + Could not connect to Work Server (results)
[18:45:29]     (171.64.122.76:8080)
[18:45:29] + Retrying using alternative port
[18:45:29] Connecting to http://171.64.122.76:80/
[18:45:29] - Couldn't send HTTP request to server
[18:45:29]   (Got status 503)
[18:45:29] + Could not connect to Work Server (results)
[18:45:29]     (171.64.122.76:80)
[18:45:29]   Could not transmit unit 04 to Collection server; keeping in queue.
[18:45:29] + Sent 0 of 1 completed units to the server
[18:45:29] - Preparing to get new work unit...
[18:45:29] + Attempting to get work packet
[18:45:29] - Will indicate memory of 1279 MB
[18:45:29] - Connecting to assignment server
[18:45:29] Connecting to http://assign.stanford.edu:8080/
[18:45:30] Posted data.
[18:45:30] Initial: 0000; + No appropriate work server was available; will try again in a bit.
[18:45:30] + Couldn't get work instructions.
[18:45:30] - Attempt #1  to get work failed, and no other work to do.
Waiting before retry.

<snip...still September 1>

[19:07:45] - Attempt #9  to get work failed, and no other work to do.
Waiting before retry.
[19:29:13] + Attempting to get work packet
[19:29:13] - Will indicate memory of 1279 MB
[19:29:13] - Connecting to assignment server
[19:29:13] Connecting to http://assign.stanford.edu:8080/
[19:29:14] Posted data.
[19:29:14] Initial: 41AB; - Successful: assigned to (171.65.103.162).
[19:29:14] + News From Folding@Home: Welcome to Folding@Home
[19:29:14] Loaded queue successfully.
[19:29:14] Connecting to http://171.65.103.162:8080/
[19:29:17] Posted data.
[19:29:17] Initial: 0000; - Receiving payload (expected size: 1657697)
[19:29:24] - Downloaded at ~231 kB/s
[19:29:24] - Averaged speed for that direction ~290 kB/s
[19:29:24] + Received work.
[19:29:24] Trying to send all finished work units
[19:29:24] Project: 2621 (Run 88, Clone 55, Gen 33)
[19:29:24] - Read packet limit of 540015616... Set to 524286976.


[19:29:24] + Attempting to send results [September 1 19:29:24 UTC]
[19:29:24] - Reading file work/wuresults_04.dat from core
[19:29:24]   (Read 8123415 bytes from disk)
[19:29:24] Connecting to http://171.64.65.65:8080/
[19:29:24] - Couldn't send HTTP request to server
[19:29:24]   (Got status 503)
[19:29:24] + Could not connect to Work Server (results)
[19:29:24]     (171.64.65.65:8080)
[19:29:24] + Retrying using alternative port
[19:29:24] Connecting to http://171.64.65.65:80/
[19:29:25] - Couldn't send HTTP request to server
[19:29:25]   (Got status 503)
[19:29:25] + Could not connect to Work Server (results)
[19:29:25]     (171.64.65.65:80)
[19:29:25] - Error: Could not transmit unit 04 (completed September 1) to work server.
[19:29:25] - 3 failed uploads of this unit.
[19:29:25] - Read packet limit of 540015616... Set to 524286976.


[19:29:25] + Attempting to send results [September 1 19:29:25 UTC]
[19:29:25] - Reading file work/wuresults_04.dat from core
[19:29:25]   (Read 8123415 bytes from disk)
[19:29:25] Connecting to http://171.64.122.76:8080/
[19:29:25] - Couldn't send HTTP request to server
[19:29:25]   (Got status 503)
[19:29:25] + Could not connect to Work Server (results)
[19:29:25]     (171.64.122.76:8080)
[19:29:25] + Retrying using alternative port
[19:29:25] Connecting to http://171.64.122.76:80/
[19:29:25] - Couldn't send HTTP request to server
[19:29:25]   (Got status 503)
[19:29:25] + Could not connect to Work Server (results)
[19:29:25]     (171.64.122.76:80)
[19:29:25]   Could not transmit unit 04 to Collection server; keeping in queue.
[19:29:25] + Sent 0 of 1 completed units to the server
[19:29:25] + Closed connections
[19:29:25]
[19:29:25] + Processing work unit
[19:29:25] Core required: FahCore_78.exe
[19:29:25] Core found.
[19:29:25] Working on queue slot 05 [September 1 19:29:25 UTC]
[19:29:25] + Working ...
[19:29:25] - Calling '.\FahCore_78.exe -dir work/ -suffix 05 -checkpoint 15 -verbose -lifeline 1988 -version 620'

[19:29:25]
[19:29:25] *------------------------------*
[19:29:25] Folding@Home Gromacs Core
[19:29:25] Version 1.90 (March 8, 2006)
[19:29:25]
[19:29:25] Preparing to commence simulation
[19:29:25] - Looking at optimizations...
[19:29:25] - Created dyn
[19:29:25] - Files status OK
[19:29:32] - Expanded 1657185 -> 8430209 (decompressed 508.7 percent)
[19:29:32] - Starting from initial work packet
[19:29:32]
[19:29:32] Project: 2452 (Run 19, Clone 42, Gen 19)
[19:29:32]
[19:29:33] Assembly optimizations on if available.
[19:29:33] Entering M.D.
[19:29:41] Protein: p2452_Fragment-Receptor HDS
[19:29:41]
[19:29:41] Writing local files
[19:29:45] Extra SSE boost OK.
[19:29:46] Writing local files
[19:29:46] Completed 0 out of 400000 steps  (0%)
<snip>
[22:52:19] - Autosending finished units... [September 1 22:52:19 UTC]
[22:52:19] Trying to send all finished work units
[22:52:19] Project: 2621 (Run 88, Clone 55, Gen 33)
[22:52:19] - Read packet limit of 540015616... Set to 524286976.


[22:52:19] + Attempting to send results [September 1 22:52:19 UTC]
[22:52:19] - Reading file work/wuresults_04.dat from core
[22:52:19]   (Read 8123415 bytes from disk)
[22:52:19] Connecting to http://171.64.65.65:8080/
[22:52:19] - Couldn't send HTTP request to server
[22:52:19]   (Got status 503)
[22:52:19] + Could not connect to Work Server (results)
[22:52:19]     (171.64.65.65:8080)
[22:52:19] + Retrying using alternative port
[22:52:19] Connecting to http://171.64.65.65:80/
[22:52:19] - Couldn't send HTTP request to server
[22:52:19]   (Got status 503)
[22:52:19] + Could not connect to Work Server (results)
[22:52:19]     (171.64.65.65:80)
[22:52:19] - Error: Could not transmit unit 04 (completed September 1) to work server.
[22:52:19] - 4 failed uploads of this unit.
[22:52:19] - Read packet limit of 540015616... Set to 524286976.


[22:52:19] + Attempting to send results [September 1 22:52:19 UTC]
[22:52:19] - Reading file work/wuresults_04.dat from core
[22:52:19]   (Read 8123415 bytes from disk)
[22:52:19] Connecting to http://171.64.122.76:8080/
[22:53:26] Posted data.
[22:53:26] Initial: 0000; - Uploaded at ~118 kB/s
[22:53:26] - Averaged speed for that direction ~113 kB/s
[22:53:26] + Results successfully sent
[22:53:26] Thank you for your contribution to Folding@Home.
[22:53:26] + Number of Units Completed: 4

[22:53:26]   Successfully sent unit 04 to Collection server.
[22:53:27] + Sent 1 of 1 completed units to the server
[22:53:27] - Autosend completed

[22:53:27] + Working...

<snip>

[bruce]

This information is from the Moderator-only database that we have so we can take some of the routine questions and avoid distracting the Pande Group whenever possible (They do have things to do that we can't help with.)

The PS3 is clearly an error though it's the first one I've seen in that database. I suspect it really is a GPU. (Updating the transaction to support all the new hardware and new OSs isn't high on Stanford's priority list, but we make do with what we have.) The fact that UserID = 333954D315XXXXXXXX appears twice means that you are using two different values for MachineID on that hardware so it's got to be the same OS that was reported to be WindowsXP.

No, we don't have any way to identify Windows Server 2003.

The timestamps record when the points were credited in the Stanford database. It's in PDT (California time = UTC-7) and since the stats are only updating every 2 hours, it is generally quite some time after the actual upload.

The PC client on 333954D315XXXXXXXX has turned in a new WU: 2008-09-05 20:38:03 5113 12 65 1 and if I look for more information on that WU, I can see it was uploaded at 20:26:42 and credited at 20:38:03.

[bapriebe]

The PS3 is clearly an error though it's the first one I've seen in that database. I suspect it really is a GPU. ...The fact that UserID = 333954D315XXXXXXXX appears twice means that you are using two different values for MachineID on that hardware so it's got to be the same OS that was reported to be WindowsXP.

Correct: they both run under XP 32-bit. It runs the 6.20 ATI GPU2 client on one core, and a 6.20 CPU client on the other.

So the question remains: what happened with the crediting of the WU from the Server 2003 box machine ID #1 (a.k.a. User ID 0B212ADB69XXXXXXXX that's misreported as being Vista)? Machine ID #2 seems to be up-to-date stats-wise...

[bruce]

So the question remains: what happened with the crediting of the WU from the Server 2003 box machine ID #1 (a.k.a. User ID 0B212ADB69XXXXXXXX that's misreported as being Vista)? Machine ID #2 seems to be up-to-date stats-wise...

You apparently have two clients on that machine. One returned a WU on 2008-09-05 06:23:01 (or later) and the other client has not returned a WU since 2008-08-29 02:22:13 Look at FAHlog.txt for each of those clients. Confirm that they're both running, that they have different values of MachineID, that they both have the correct UserName and Team number, and that both have produced WUs since 08-29. By that time, if you have not discovered a problem, find the PRCG identifiers of the last WU that uploaded from each one and post the appropriate segments of FAHlog from each one.

[bapriebe]

All of that information was posted with my first response.

[ChelseaOilman]

Is the missing credit from Project: 2621 (Run 88, Clone 55, Gen 33)?

If so, can you post the section of the log showing the upload? Did the WU upload to the collection server instead of the work server?

[ChelseaOilman]

The PS3 is clearly an error though it's the first one I've seen in that database. I suspect it really is a GPU.

My nVidia cards are reported as "S390".

[bruce]

The PS3 is clearly an error though it's the first one I've seen in that database. I suspect it really is a GPU.

My nVidia cards are reported as "S390".

Yes, an the ATI GPU seems to be what's misidentified as a PS3 running Windows within that transaction. It's not a big deal once I figured it out. Now I remember that you mentioned it earlier.

[bapriebe]

Is the missing credit from Project: 2621 (Run 88, Clone 55, Gen 33)?

Per my original response: yes.

If so, can you post the section of the log showing the upload?

Per my original response, refer to timestamp 22:53:26 UTC on September 1 of the 2nd log (a.k.a machine ID #1 on the misreported "Windows Vista" box).

Did the WU upload to the collection server instead of the work server?

The original work server 171.64.65.65 was apparently down or overloaded (HTTP status 503). Per the log, the results were finally uploaded after four retries to 171.64.122.76 on port 8080. The server status page shows this to be "CS 2".

[bruce]

The problem with 171.64.122.76 has been identified and the CS team is working on it. I believe that all of work, both WUs and points, can be recovered but it may take some time before you see them.