The P8101 is killing my SR2

[-alias-]

I have a SR2 folding rig that flawlessly have delivered work units since I bought it in January this year, most 6903 and 6904 and some 6901. Yesterday I got down a 8101 which apparently folded fine, but this afternoon it went wrong, I see can in the log. Is P8101 tougher with the hardware than the other bigadv? I attach the entire log so you can see the error message and the whole story. I have little desire to start to tune on all my rigs (8 in number) down only because 8101 has arrived. And, is there others who have experiencing the same thing as me with this WU?

Code: Select all

[20:10:15] *------------------------------*
[20:10:15] Folding@Home Gromacs SMP Core
[20:10:15] Version 2.27 (Thu Feb 10 09:46:40 PST 2011)
[20:10:15] 
[20:10:15] Preparing to commence simulation
[20:10:15] - Looking at optimizations...
[20:10:15] - Not checking prior termination.
[20:10:17] - Expanded 30310301 -> 33158016 (decompressed 109.3 percent)
[20:10:17] Called DecompressByteArray: compressed_data_size=30310301 data_size=33158016, decompressed_data_size=33158016 diff=0
[20:10:17] - Digital signature verified
[20:10:17] 
[20:10:17] Project: 8101 (Run 1, Clone 1, Gen 20)
[20:10:17] 
[20:10:17] Assembly optimizations on if available.
[20:10:17] Entering M.D.
[20:10:24] Mapping NT from 24 to 24 
[20:10:27] Completed 0 out of 250000 steps  (0%)
[20:14:54] Mapping NT from 24 to 24 
[20:15:06] Resuming from checkpoint
[20:15:11] Verified work/wudata_07.log
[20:15:11] Verified work/wudata_07.trr
[20:15:12] Verified work/wudata_07.xtc
[20:15:12] Verified work/wudata_07.edr
[20:15:12] Completed 320 out of 250000 steps  (0%)
[20:35:32] Completed 2500 out of 250000 steps  (1%)
[20:58:52] Completed 5000 out of 250000 steps  (2%)
[21:22:15] Completed 7500 out of 250000 steps  (3%)
[21:45:36] Completed 10000 out of 250000 steps  (4%)
[22:08:55] Completed 12500 out of 250000 steps  (5%)
[22:32:16] Completed 15000 out of 250000 steps  (6%)
[22:55:36] Completed 17500 out of 250000 steps  (7%)
[23:18:57] Completed 20000 out of 250000 steps  (8%)
[23:42:16] Completed 22500 out of 250000 steps  (9%)
[00:05:33] Completed 25000 out of 250000 steps  (10%)
[00:28:55] Completed 27500 out of 250000 steps  (11%)
[00:52:15] Completed 30000 out of 250000 steps  (12%)
[01:15:35] Completed 32500 out of 250000 steps  (13%)
[01:38:53] Completed 35000 out of 250000 steps  (14%)
[02:02:12] Completed 37500 out of 250000 steps  (15%)
[02:09:45] - Autosending finished units... [April 23 02:09:45 UTC]
[02:09:45] Trying to send all finished work units
[02:09:45] Project: 6904 (Run 1, Clone 23, Gen 63)

[02:09:45] + Attempting to send results [April 23 02:09:45 UTC]
[02:09:45] - Reading file work/wuresults_05.dat from core
[02:09:45]   (Read 222357914 bytes from disk)
[02:09:45] Connecting to http://130.237.232.237:8080/
[02:16:01] Posted data.
[02:16:01] Initial: 0000; - Uploaded at ~577 kB/s
[02:16:01] - Averaged speed for that direction ~576 kB/s
[02:16:01] + Results successfully sent
[02:16:01] Thank you for your contribution to Folding@Home.
[02:16:01] + Number of Units Completed: 48

[02:25:37] Completed 40000 out of 250000 steps  (16%)
[02:48:55] Completed 42500 out of 250000 steps  (17%)
[03:12:15] Completed 45000 out of 250000 steps  (18%)
[03:35:32] Completed 47500 out of 250000 steps  (19%)
[03:58:51] Completed 50000 out of 250000 steps  (20%)
[04:22:09] Completed 52500 out of 250000 steps  (21%)
[04:45:29] Completed 55000 out of 250000 steps  (22%)
[05:08:49] Completed 57500 out of 250000 steps  (23%)
[05:32:09] Completed 60000 out of 250000 steps  (24%)
[05:55:30] Completed 62500 out of 250000 steps  (25%)
[06:19:37] Completed 65000 out of 250000 steps  (26%)
[06:42:56] Completed 67500 out of 250000 steps  (27%)
[07:06:16] Completed 70000 out of 250000 steps  (28%)
[07:29:36] Completed 72500 out of 250000 steps  (29%)
[07:52:57] Completed 75000 out of 250000 steps  (30%)
[08:16:12] - Autosending finished units... [April 23 08:16:12 UTC]
[08:16:12] Trying to send all finished work units
[08:16:12] + No unsent completed units remaining.
[08:16:12] - Autosend completed
[08:16:21] Completed 77500 out of 250000 steps  (31%)
[08:39:44] Completed 80000 out of 250000 steps  (32%)
[09:03:04] Completed 82500 out of 250000 steps  (33%)
[09:26:24] Completed 85000 out of 250000 steps  (34%)
[09:49:45] Completed 87500 out of 250000 steps  (35%)
[10:13:07] Completed 90000 out of 250000 steps  (36%)
[10:36:28] Completed 92500 out of 250000 steps  (37%)
[10:59:47] Completed 95000 out of 250000 steps  (38%)
[11:23:08] Completed 97500 out of 250000 steps  (39%)
[11:46:30] Completed 100000 out of 250000 steps  (40%)
[12:09:51] Completed 102500 out of 250000 steps  (41%)
[12:33:12] Completed 105000 out of 250000 steps  (42%)
[12:56:32] Completed 107500 out of 250000 steps  (43%)
[13:19:54] Completed 110000 out of 250000 steps  (44%)
[13:43:18] Completed 112500 out of 250000 steps  (45%)
[14:06:43] Completed 115000 out of 250000 steps  (46%)
[14:16:12] - Autosending finished units... [April 23 14:16:12 UTC]
[14:16:12] Trying to send all finished work units
[14:16:12] + No unsent completed units remaining.
[14:16:12] - Autosend completed
[14:30:08] Completed 117500 out of 250000 steps  (47%)
[14:53:32] Completed 120000 out of 250000 steps  (48%)
[15:16:58] Completed 122500 out of 250000 steps  (49%)
[15:40:22] Completed 125000 out of 250000 steps  (50%)
[16:03:47] Completed 127500 out of 250000 steps  (51%)
[16:04:49] CoreStatus = 8B (139)
[16:04:49] Client-core communications error: ERROR 0x8b
[16:04:49] Deleting current work unit & continuing...
[16:05:02] Trying to send all finished work units
[16:05:02] + No unsent completed units remaining.
[16:05:02] - Preparing to get new work unit...
[16:05:02] Cleaning up work directory
[16:05:03] + Attempting to get work packet
[16:05:03] Passkey found
[16:05:03] - Will indicate memory of 12033 MB
[16:05:03] - Connecting to assignment server
[16:05:03] Connecting to http://assign.stanford.edu:8080/
[16:05:05] Posted data.
[16:05:05] Initial: ED82; - Successful: assigned to (130.237.232.237).
[16:05:05] + News From Folding@Home: Welcome to Folding@Home
[16:05:05] Loaded queue successfully.
[16:05:05] Sent data
[16:05:05] Connecting to http://130.237.232.237:8080/
[16:05:11] Posted data.
[16:05:11] Initial: 0000; - Receiving payload (expected size: 24861471)
[16:05:28] - Downloaded at ~1428 kB/s
[16:05:28] - Averaged speed for that direction ~997 kB/s
[16:05:28] + Received work.
[16:05:28] + Closed connections
[16:05:33] 
[16:05:33] + Processing work unit
[16:05:33] Core required: FahCore_a5.exe
[16:05:33] Core found.
[16:05:33] Working on queue slot 08 [April 23 16:05:33 UTC]
[16:05:33] + Working ...
[16:05:33] - Calling './FahCore_a5.exe -dir work/ -nice 19 -suffix 08 -np 24 -checkpoint 3 -verbose -lifeline 2028 -version 634'

[16:05:34] 
[16:05:34] *------------------------------*
[16:05:34] Folding@Home Gromacs SMP Core
[16:05:34] Version 2.27 (Thu Feb 10 09:46:40 PST 2011)
[16:05:34] 
[16:05:34] Preparing to commence simulation
[16:05:34] - Looking at optimizations...
[16:05:34] - Created dyn
[16:05:34] - Files status OK
[16:05:36] - Expanded 24860959 -> 30796292 (decompressed 123.8 percent)
[16:05:36] Called DecompressByteArray: compressed_data_size=24860959 data_size=30796292, decompressed_data_size=30796292 diff=0
[16:05:36] - Digital signature verified
[16:05:36] 
[16:05:36] Project: 6901 (Run 20, Clone 11, Gen 260)
[16:05:36] 
[16:05:36] Assembly optimizations on if available.
[16:05:36] Entering M.D.
[16:05:42] Mapping NT from 24 to 24 
[16:05:44] Completed 0 out of 250000 steps  (0%)
[16:10:05] Entering M.D.
[16:10:11] Using Gromacs checkpoints
[16:10:13] Mapping NT from 24 to 24 
[16:10:22] Resuming from checkpoint
[16:10:26] Verified work/wudata_08.log
[16:10:26] Verified work/wudata_08.trr
[16:10:26] Verified work/wudata_08.xtc
[16:10:26] Verified work/wudata_08.edr
[16:10:27] Completed 675 out of 250000 steps  (0%)
[16:18:41] Completed 2500 out of 250000 steps  (1%)
[16:30:01] Completed 5000 out of 250000 steps  (2%)
[16:41:23] Completed 7500 out of 250000 steps  (3%)
[16:52:44] Completed 10000 out of 250000 steps  (4%)
[17:04:02] Completed 12500 out of 250000 steps  (5%)
[17:15:23] Completed 15000 out of 250000 steps  (6%)
[17:26:43] Completed 17500 out of 250000 steps  (7%)
[17:38:04] Completed 20000 out of 250000 steps  (8%)
[17:49:25] Completed 22500 out of 250000 steps  (9%)
[18:00:43] Completed 25000 out of 250000 steps  (10%)
[18:12:03] Completed 27500 out of 250000 steps  (11%)
[18:23:24] Completed 30000 out of 250000 steps  (12%)
[18:34:44] Completed 32500 out of 250000 steps  (13%)
[18:46:02] Completed 35000 out of 250000 steps  (14%)
[18:57:20] Completed 37500 out of 250000 steps  (15%)
[19:08:39] Completed 40000 out of 250000 steps  (16%)
[19:19:58] Completed 42500 out of 250000 steps  (17%)
[19:31:15] Completed 45000 out of 250000 steps  (18%)
[19:42:34] Completed 47500 out of 250000 steps  (19%)
[19:53:54] Completed 50000 out of 250000 steps  (20%)
[20:05:12] Completed 52500 out of 250000 steps  (21%)
[20:16:12] - Autosending finished units... [April 23 20:16:12 UTC]
[20:16:12] Trying to send all finished work units
[20:16:12] + No unsent completed units remaining.
[20:16:12] - Autosend completed
[20:16:30] Completed 55000 out of 250000 steps  (22%)
[20:27:49] Completed 57500 out of 250000 steps  (23%)
[20:39:08] Completed 60000 out of 250000 steps  (24%)
[20:50:27] Completed 62500 out of 250000 steps  (25%)
[21:01:47] Completed 65000 out of 250000 steps  (26%)
[21:13:03] Completed 67500 out of 250000 steps  (27%)
[21:24:23] Completed 70000 out of 250000 steps  (28%)

[bollix47]

Code: Select all

8B

CoreStatus = 8B (139)
Client-core communications error: ERROR 0x8B
Deleting current work unit & continuing...

Triggered by the OS, probably due to overclocking/overheating or a memory failure

I have little desire to start to tune on all my rigs (8 in number) down only because 8101 has arrived.

The 8101 requires 16 thread capability so only two or your rigs should get assigned to them. Your tyan might be okay so it's probably just the sr-2 that may need the overclock dropped a bit or maybe improving the cooling will work.

[KMac]

Yes, many have reported P8101 drawing more wattage from the wall, higher temps and the need to lower OC or increase voltage for stability as compared to P6903,6904.

[-alias-]

The 8101 requires 16 thread capability so only two or your rigs should get assigned to them. Your tyan might be okay so it's probably just the sr-2 that may need the overclock dropped a bit or maybe improving the cooling will work.

Thank you, my signature is not completely updated for the time and it will come 4P 64 core and 48 core 4P instead of the 2 x 980X shown there. I did currently displaced that 8101 was one of the new 16-core WUs.

It can not be due to overheating since my SR2 is water cooled and the memory is quality chips that will handle most jobs. The machine is running at 3.6GHz that has been swell stable over a long time. I do not think I like 8101 very much. It reminds me a lot about the P2684, which also caused a number of such messages on some machines.

[Patriot]

The 8101 requires 16 thread capability so only two or your rigs should get assigned to them. Your tyan might be okay so it's probably just the sr-2 that may need the overclock dropped a bit or maybe improving the cooling will work.

Thank you, my signature is not completely updated for the time and it will come 4P 64 core and 48 core 4P instead of the 2 x 980X shown there. I did currently displaced that 8101 was one of the new 16-core WUs.

It can not be due to overheating since my SR2 is water cooled and the memory is quality chips that will handle most jobs. The machine is running at 3.6GHz that has been swell stable over a long time. I do not think I like 8101 very much. It reminds me a lot about the P2684, which also caused a number of such messages on some machines.

Overclocked chips are more sensitive to heat, you should still check your temps even if watercooled (my 4p is watercooled).
Yes this is the evil twin of 2684... necromongered up lol...
I had to up my voltage on my chips to stay stable and Grandpa had to reduce his overclock.
My 4p draws about 50w more power running them and runs about 4c hotter.

[Grandpa_01]

I think they are cousins both sucked the life out of machines. There really is not much you can do about it Stanford is forcing these on the MP folders. And left the rest of the bigadv WU's to the single proccesor machines. And they are all saying just grin and bear it, you can lower your OC or switch to smp there are a few ways around these for the time being but if I post them it will just upset the people who say we must fold these.

[-alias-]

Overclocked chips are more sensitive to heat, you should still check your temps even if watercooled (my 4p is watercooled). Yes this is the evil twin of 2684... necromongered up lol...
I had to up my voltage on my chips to stay stable and Grandpa had to reduce his overclock.
My 4p draws about 50w more power running them and runs about 4c hotter.

For Ubuntu (11.04), I have not found software that can display the temperature of CPU cores, but by removing the side door and use an external digital temperature sensor, I can measure the surface of the cooler to 35 Celsius. Something that tells me or indicates that the temperature in the CPU can not be very high. When I tested the machine under Windows 7 with CPUID HWMonitor measured my maximum core temperature to about 45-48 Celsius under 100% load in folding. So if the 8101 with Ubuntu causes an increase in temperature of about 10 - 15% will be something around 53-56 Celsius, which I think is correct. If any of you have found software to measure this in Ubuntu 11.04 and I will take in the tips appreciated.

So I think that the reason the P8101 got the machine to crash is that it simply is not enough voltage to the CPU. I have therefore increased the voltage to the next level and I then will see if it holds. The rig uses about. 400 watts from the wall as it stands now and chews on a P6903.

[Grandpa_01]

Here you go you can use this. https://help.ubuntu.com/community/SensorInstallHowto

[-alias-]

Thank you, they are a little higher than I thought with the P6903, but acceptable nonetheless.


It might be because the ambient temperature at the moment is 23 Celsius, while the normal temperature is around 18 in the room. This program (lm-sensors) https://help.ubuntu.com/community/SensorInstallHowto is working well on my SR2 rigg, but not on my P4 Tyan S8812 & 6176SE where the program shows only a few temperatures, and not from any CPU.

It looks like the G34 rig handle 8101 better than SR2 rig did, but PPD is pretty much worse than what the P6901 provides. Electricity consumption rose from 635 to 665Watt from the wall.

Code: Select all

 Name: SR2Rigg1 P6901 400Watt (2 x Xeon 5645 3.6GHz)
 Path: M:\
 Number of Frames Observed: 300

 Min. Time / Frame : 00:10:53 - 164 674,2 PPD
 Avg. Time / Frame : 00:11:16 - 156 341,8 PPD


Name: SR2Rigg1 P8101 435Watt  (2 x Xeon 5645 3.6GHz)
 Path: M:\
 Number of Frames Observed: 194

 Min. Time / Frame : 00:23:17 - 154 612,4 PPD
 Avg. Time / Frame : 00:24:19 - 144 862,5 PPD


Name: G34 Opteron 4P P6901 635Watt  (4 x 6176SE 2.3GHz)
 Path: T:\
 Number of Frames Observed: 300

 Min. Time / Frame : 00:06:34 - 351 358,7 PPD
 Avg. Time / Frame : 00:06:44 - 338 394,4 PPD

Name: G34 Opteron 4P P8101 665Watt  (4 x 6176SE 2.3GHz)
 Path: T:\
 Number of Frames Observed: 23

 Min. Time / Frame : 00:14:17 - 321 786,3 PPD
 Avg. Time / Frame : 00:14:32 - 313 519,1 PPD
 Cur. Time / Frame : 00:14:18 - 319 864,1 PPD
 R3F. Time / Frame : 00:14:20 - 318 836,6 PPD
 All  Time / Frame : 00:14:32 - 312 786,4 PPD
 Eff. Time / Frame : 00:14:49 - 304 538,3 PPD

[sbinh]

Ah darn .. I'm not the only one. Either raise vcore (way to hot) or lower speed .... wow ...

[Leonardo]

(lm-sensors)...is working well on my SR2 rigg, but not on my P4 Tyan S8812 & 6176SE where the program shows only a few temperatures, and not from any CPU

lm-sensors is, as you've discovered, dysfunctional on some platforms. Get and install Turion Power Control (TPC). I'm running it on my AMD multi-processor Folding rig and it's completely reliable.

http://hardforum.com/showpost.php?p=103 ... ostcount=1

[-alias-]

Thanks, this looks awesome, hope it works on the Tyan S8812. Will test it tomorrow and report back if it works.

[Leonardo]

For your Intel rig(s), at least single socket systems (maybe multi as well), check out i7z for Linux. You may or may not prefer it over lm sensors. I don't know, as I never got lm sensors to work on my Intel boxes, i7z works perfectly.

[-alias-]

Turion Power Control (TPC) works just great in my Tyan S8812 system. The temperatures of the CPU does not exceed 45 Celsius with a P8101 running, and I'm very happy with that. The rig is water cooled on the CPUs. Thanks again, this was exciting, and now I have control of all my AMD systems.

My next goal is to install the remote control on all the rigs as they are without keyboards and monitors around in my house. But now I'm a bit off-topic so I stop there.

One more question?
At the end of this page (in the link) there is listed it up a few lines, are listed below.
http://hardforum.com/showpost.php?p=103 ... ostcount=1

BIOS settings to check
- PowerNow set to Disabled (see above for the check)
- NUMA/SRAT set to Enabled
- Node Interleaving set to Disabled
- Bank/channel interleaving set to Auto or Enabled
- Tyan boards - Ungang DCTs set to Always

I can not find NUMA/SRAT in the bios in my Tyan S8812, anybody who know what this means?

[RozSummer]

-alias-

I thought this was a particularly good article explaining NUMA and how that relates to Node Interleaving and the construction of a System Resource Allocation Table (SRAT). From this article, it makes sense to have NUMA enabled and Node Interleaving disabled. NUMA is a memory access scheme used by both Intel and AMD. Perhaps your Tyan board defaults to "NUMA Enabled", which in that case you'd simply need to make sure Node Interleaving is disabled.

http://frankdenneman.nl/2010/12/node-in ... r-disable/

Roz