40 BAD_WORK_UNIT errors in 15 minutes!!!

SteveWillis · Post by **SteveWillis** » Sat Jan 06, 2018 3:58 pm

Very strange. On a machine that was folding perfectly both before and after this incident. It produced 40 BAD_WORK_UNIT errors in 15 minutes. I guess it would have gone on for hours except that I have a script running that forces a reboot if a FS goes for a while without making any progress (the 99.99% hang). After the reboot the machine folded normally again.

Code: Select all

2018-01-04:23:29:13:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:13:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:2587 clone:2 gen:245 core:0x21 unit:0x00000126ab436c9d585e06e1e8f2672a
2018-01-04:23:29:17:WARNING:WU06:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:18:WU06:FS00:Sending unit results: id:06 state:SEND error:FAULTY project:9414 run:1336 clone:1 gen:513 core:0x21 unit:0x00000255ab436c9d585e069a0f2c17da
2018-01-04:23:29:22:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:22:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:2066 clone:0 gen:665 core:0x21 unit:0x00000301ab436c9d585e06dccef5dd12
2018-01-04:23:29:26:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:27:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:214 clone:1 gen:56 core:0x21 unit:0x00000043ab436c9d585e06ca7eb27704
2018-01-04:23:29:31:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:32:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:83 clone:3 gen:95 core:0x21 unit:0x00000071ab436c9d586fdd34934c1573
2018-01-04:23:29:36:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:37:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:384 clone:3 gen:58 core:0x21 unit:0x00000048ab436c9d586fdd3765490976
2018-01-04:23:29:41:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:42:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:1411 clone:4 gen:50 core:0x21 unit:0x0000003eab436c9d586fdd3fa4b2a364
2018-01-04:23:29:45:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:46:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:9414 run:837 clone:1 gen:459 core:0x21 unit:0x00000220ab436c9d585e069659a3736d
2018-01-04:23:29:49:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:50:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:1241 clone:1 gen:416 core:0x21 unit:0x000001e3ab436c9d585e0699ae7ef0c9
2018-01-04:23:29:55:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:55:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:327 clone:3 gen:78 core:0x21 unit:0x0000005aab436c9d586fdd36020a56ca
2018-01-04:23:34:01:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:01:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:2500 clone:0 gen:537 core:0x21 unit:0x00000275ab436c9d585e06e0df341911
2018-01-04:23:34:05:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:05:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:518 clone:1 gen:319 core:0x21 unit:0x00000179ab436c9d586fdd3863938bbc
2018-01-04:23:34:09:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:09:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:412 clone:2 gen:458 core:0x21 unit:0x0000021eab436c9d586fdd37fa3a5c4e
2018-01-04:23:34:13:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:13:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:738 clone:4 gen:170 core:0x21 unit:0x000000c7ab436c9d585e06cfd59c10e8
******************************* Date: 2018-01-04 *******************************
2018-01-04:23:34:17:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:17:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:1464 clone:0 gen:637 core:0x21 unit:0x000002f4ab436c9d586fdd3f2612ae5c
2018-01-04:23:34:21:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:21:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:1506 clone:1 gen:640 core:0x21 unit:0x000002e1ab436c9d585e069b6871db88
2018-01-04:23:34:25:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:25:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:758 clone:2 gen:46 core:0x21 unit:0x00000038ab436c9d586fdd39b9dd104d
2018-01-04:23:34:28:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:28:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:1750 clone:0 gen:678 core:0x21 unit:0x0000031fab436c9d585e069d38f35b74
2018-01-04:23:34:32:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:32:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:1041 clone:2 gen:171 core:0x21 unit:0x000000c9ab436c9d586fdd3c993265c4
2018-01-04:23:34:36:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:36:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:2601 clone:2 gen:228 core:0x21 unit:0x00000111ab436c9d585e06a475ef079a
2018-01-04:23:37:28:WARNING:WU00:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:28:WU00:FS04:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:1641 clone:1 gen:32 core:0x21 unit:0x00000026ab436c9d585e06d87053f116
2018-01-04:23:37:32:WARNING:WU01:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:32:WU01:FS04:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:1773 clone:1 gen:543 core:0x21 unit:0x0000027bab436c9d585e06d900ac7a75
2018-01-04:23:37:35:WARNING:WU00:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:35:WU00:FS04:Sending unit results: id:00 state:SEND error:FAULTY project:9414 run:1590 clone:4 gen:84 core:0x21 unit:0x0000006dab436c9d585e069c8bad9896
2018-01-04:23:37:39:WARNING:WU01:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:39:WU01:FS04:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:2585 clone:1 gen:593 core:0x21 unit:0x000002baab436c9d585e06e163a101b8
2018-01-04:23:37:43:WARNING:WU00:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:43:WU00:FS04:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:1358 clone:3 gen:153 core:0x21 unit:0x000000b5ab436c9d586fdd3e5391203d
2018-01-04:23:37:47:WARNING:WU01:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:47:WU01:FS04:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:1371 clone:0 gen:773 core:0x21 unit:0x0000039dab436c9d585e069a82514d93
2018-01-04:23:37:51:WARNING:WU00:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:51:WU00:FS04:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:1289 clone:0 gen:765 core:0x21 unit:0x0000036eab436c9d585e06d4a8123af2
2018-01-04:23:37:54:WARNING:WU01:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:54:WU01:FS04:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:2092 clone:1 gen:587 core:0x21 unit:0x000002bfab436c9d585e06dc14b5c146
2018-01-04:23:37:57:WARNING:WU00:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:37:57:WU00:FS04:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:1139 clone:1 gen:467 core:0x21 unit:0x0000021bab436c9d585e06d36f0ec8ce
2018-01-04:23:38:01:WARNING:WU01:FS04:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:38:01:WU01:FS04:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:1690 clone:4 gen:217 core:0x21 unit:0x0000010aab436c9d585e06d852404ce8
2018-01-04:23:43:23:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:24:WU00:FS02:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:1386 clone:1 gen:17 core:0x21 unit:0x00000018ab436c9d586fdd3e33f42c46
2018-01-04:23:43:28:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:28:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:1331 clone:4 gen:17 core:0x21 unit:0x00000015ab436c9d586fdd3e9335831a
2018-01-04:23:43:31:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:31:WU00:FS02:Sending unit results: id:00 state:SEND error:FAULTY project:9414 run:1052 clone:1 gen:339 core:0x21 unit:0x0000018eab436c9d585e06972c3625e4
2018-01-04:23:43:35:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:35:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:2090 clone:4 gen:31 core:0x21 unit:0x0000002aab436c9d585e06a0a8dd44ca
2018-01-04:23:43:38:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:38:WU00:FS02:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:229 clone:4 gen:152 core:0x21 unit:0x000000bdab436c9d585e06cbf199e37f
2018-01-04:23:43:42:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:43:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:204 clone:1 gen:246 core:0x21 unit:0x00000126ab436c9d586fdd3597297873
2018-01-04:23:43:46:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:46:WU00:FS02:Sending unit results: id:00 state:SEND error:FAULTY project:9414 run:98 clone:2 gen:25 core:0x21 unit:0x0000001fab436c9d585e06903db55421
2018-01-04:23:43:49:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:49:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:1325 clone:1 gen:20 core:0x21 unit:0x00000018ab436c9d585e069af5de3458
2018-01-04:23:43:52:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:53:WU00:FS02:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:1044 clone:1 gen:310 core:0x21 unit:0x0000017bab436c9d585e06d24802de54
2018-01-04:23:43:57:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:43:57:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:706 clone:2 gen:511 core:0x21 unit:0x0000024eab436c9d586fdd39b175a3f6

Here is part of the log. The error appears to occur immediately after the WU starts.

Code: Select all

2018-01-04:23:28:13:WU00:FS00:0x21:Completed 6187500 out of 6250000 steps (99%)
2018-01-04:23:28:21:WU02:FS02:0x21:Completed 4875000 out of 6250000 steps (78%)
2018-01-04:23:28:49:WU05:FS01:0x21:Completed 5875000 out of 6250000 steps (94%)
2018-01-04:23:28:51:WU03:FS03:0x21:Completed 2312500 out of 6250000 steps (37%)
2018-01-04:23:28:54:WU04:FS04:0x21:Completed 5750000 out of 6250000 steps (92%)
2018-01-04:23:29:04:WU02:FS02:0x21:Completed 4937500 out of 6250000 steps (79%)
2018-01-04:23:29:09:WU00:FS00:0x21:Completed 6250000 out of 6250000 steps (100%)
2018-01-04:23:29:10:WU01:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:10:WU01:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:10:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:10:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:11:WU00:FS00:0x21:Saving result file logfile_01.txt
2018-01-04:23:29:11:WU00:FS00:0x21:Saving result file checkpointState.xml
2018-01-04:23:29:11:WU00:FS00:0x21:Saving result file checkpt.crc
2018-01-04:23:29:11:WU00:FS00:0x21:Saving result file log.txt
2018-01-04:23:29:11:WU00:FS00:0x21:Saving result file positions.xtc
2018-01-04:23:29:11:WU00:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
2018-01-04:23:29:11:WU01:FS00:Downloading 5.18MiB
2018-01-04:23:29:11:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
2018-01-04:23:29:11:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9431 run:698 clone:3 gen:233 core:0x21 unit:0x00000124ab436c9d586fdd39fd8d7aea
2018-01-04:23:29:11:WU00:FS00:Uploading 13.61MiB to 171.67.108.157
2018-01-04:23:29:11:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:13:WU01:FS00:Download complete
2018-01-04:23:29:13:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9415 run:2587 clone:2 gen:245 core:0x21 unit:0x00000126ab436c9d585e06e1e8f2672a
2018-01-04:23:29:13:WU01:FS00:Starting
2018-01-04:23:29:13:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:13:WU01:FS00:Started FahCore on PID 23523
2018-01-04:23:29:13:WU01:FS00:Core PID:23527
2018-01-04:23:29:13:WU01:FS00:FahCore 0x21 started
2018-01-04:23:29:13:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:13:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:2587 clone:2 gen:245 core:0x21 unit:0x00000126ab436c9d585e06e1e8f2672a
2018-01-04:23:29:13:WU01:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:13:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:14:WU06:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:14:WU01:FS00:Upload complete
2018-01-04:23:29:14:WU01:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:14:WU01:FS00:Cleaning up
2018-01-04:23:29:14:WU06:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:14:WU06:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:14:WU06:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:15:WU06:FS00:Downloading 5.17MiB
2018-01-04:23:29:17:WU00:FS00:Upload 64.76%
2018-01-04:23:29:17:WU06:FS00:Download complete
2018-01-04:23:29:17:WU06:FS00:Received Unit: id:06 state:DOWNLOAD error:NO_ERROR project:9414 run:1336 clone:1 gen:513 core:0x21 unit:0x00000255ab436c9d585e069a0f2c17da
2018-01-04:23:29:17:WU06:FS00:Starting
2018-01-04:23:29:17:WU06:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 06 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:17:WU06:FS00:Started FahCore on PID 23537
2018-01-04:23:29:17:WU06:FS00:Core PID:23541
2018-01-04:23:29:17:WU06:FS00:FahCore 0x21 started
2018-01-04:23:29:17:WARNING:WU06:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:18:WU06:FS00:Sending unit results: id:06 state:SEND error:FAULTY project:9414 run:1336 clone:1 gen:513 core:0x21 unit:0x00000255ab436c9d585e069a0f2c17da
2018-01-04:23:29:18:WU06:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:18:WU06:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:18:WU01:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:18:WU06:FS00:Upload complete
2018-01-04:23:29:18:WU06:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:18:WU06:FS00:Cleaning up
2018-01-04:23:29:19:WU01:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:19:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:19:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:20:WU01:FS00:Downloading 5.15MiB
2018-01-04:23:29:20:WU00:FS00:Upload complete
2018-01-04:23:29:20:WU00:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:20:WU00:FS00:Final credit estimate, 57178.00 points
2018-01-04:23:29:20:WU00:FS00:Cleaning up
2018-01-04:23:29:21:WU01:FS00:Download complete
2018-01-04:23:29:22:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9415 run:2066 clone:0 gen:665 core:0x21 unit:0x00000301ab436c9d585e06dccef5dd12
2018-01-04:23:29:22:WU01:FS00:Starting
2018-01-04:23:29:22:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:22:WU01:FS00:Started FahCore on PID 23559
2018-01-04:23:29:22:WU01:FS00:Core PID:23563
2018-01-04:23:29:22:WU01:FS00:FahCore 0x21 started
2018-01-04:23:29:22:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:22:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:2066 clone:0 gen:665 core:0x21 unit:0x00000301ab436c9d585e06dccef5dd12
2018-01-04:23:29:22:WU01:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:22:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:23:WU01:FS00:Upload complete
2018-01-04:23:29:23:WU00:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:23:WU01:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:23:WU01:FS00:Cleaning up
2018-01-04:23:29:24:WU00:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:24:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:24:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:24:WU00:FS00:Downloading 5.19MiB
2018-01-04:23:29:26:WU00:FS00:Download complete
2018-01-04:23:29:26:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9415 run:214 clone:1 gen:56 core:0x21 unit:0x00000043ab436c9d585e06ca7eb27704
2018-01-04:23:29:26:WU00:FS00:Starting
2018-01-04:23:29:26:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:26:WU00:FS00:Started FahCore on PID 23571
2018-01-04:23:29:26:WU00:FS00:Core PID:23575
2018-01-04:23:29:26:WU00:FS00:FahCore 0x21 started
2018-01-04:23:29:26:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:27:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:214 clone:1 gen:56 core:0x21 unit:0x00000043ab436c9d585e06ca7eb27704
2018-01-04:23:29:27:WU00:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:27:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:27:WU01:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:27:WU00:FS00:Upload complete
2018-01-04:23:29:27:WU00:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:27:WU00:FS00:Cleaning up
2018-01-04:23:29:27:WU01:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:27:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:27:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:28:WU01:FS00:Downloading 8.84MiB
2018-01-04:23:29:31:WU01:FS00:Download complete
2018-01-04:23:29:31:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9431 run:83 clone:3 gen:95 core:0x21 unit:0x00000071ab436c9d586fdd34934c1573
2018-01-04:23:29:31:WU01:FS00:Starting
2018-01-04:23:29:31:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:31:WU01:FS00:Started FahCore on PID 23589
2018-01-04:23:29:31:WU01:FS00:Core PID:23593
2018-01-04:23:29:31:WU01:FS00:FahCore 0x21 started
2018-01-04:23:29:31:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:32:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:83 clone:3 gen:95 core:0x21 unit:0x00000071ab436c9d586fdd34934c1573
2018-01-04:23:29:32:WU01:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:32:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:32:WU01:FS00:Upload complete
2018-01-04:23:29:32:WU00:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:32:WU01:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:32:WU01:FS00:Cleaning up
2018-01-04:23:29:32:WU00:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:32:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:32:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:33:WU00:FS00:Downloading 8.84MiB
2018-01-04:23:29:36:WU00:FS00:Download complete
2018-01-04:23:29:36:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9431 run:384 clone:3 gen:58 core:0x21 unit:0x00000048ab436c9d586fdd3765490976
2018-01-04:23:29:36:WU00:FS00:Starting
2018-01-04:23:29:36:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:36:WU00:FS00:Started FahCore on PID 23606
2018-01-04:23:29:36:WU00:FS00:Core PID:23610
2018-01-04:23:29:36:WU00:FS00:FahCore 0x21 started
2018-01-04:23:29:36:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:37:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:384 clone:3 gen:58 core:0x21 unit:0x00000048ab436c9d586fdd3765490976
2018-01-04:23:29:37:WU00:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:37:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:37:WU00:FS00:Upload complete
2018-01-04:23:29:37:WU01:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:37:WU00:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:37:WU00:FS00:Cleaning up
2018-01-04:23:29:37:WU01:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:37:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:37:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:38:WU01:FS00:Downloading 8.83MiB
2018-01-04:23:29:41:WU01:FS00:Download complete
2018-01-04:23:29:41:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9431 run:1411 clone:4 gen:50 core:0x21 unit:0x0000003eab436c9d586fdd3fa4b2a364
2018-01-04:23:29:41:WU03:FS03:0x21:Completed 2375000 out of 6250000 steps (38%)
2018-01-04:23:29:41:WU01:FS00:Starting
2018-01-04:23:29:41:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:41:WU05:FS01:0x21:Completed 5937500 out of 6250000 steps (95%)
2018-01-04:23:29:41:WU01:FS00:Started FahCore on PID 23725
2018-01-04:23:29:41:WU01:FS00:Core PID:23729
2018-01-04:23:29:41:WU01:FS00:FahCore 0x21 started
2018-01-04:23:29:41:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:42:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:1411 clone:4 gen:50 core:0x21 unit:0x0000003eab436c9d586fdd3fa4b2a364
2018-01-04:23:29:42:WU01:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:42:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:42:WU01:FS00:Upload complete
2018-01-04:23:29:42:WU00:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:42:WU01:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:42:WU01:FS00:Cleaning up
2018-01-04:23:29:43:WU00:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:43:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:43:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:43:WU00:FS00:Downloading 5.14MiB
2018-01-04:23:29:45:WU00:FS00:Download complete
2018-01-04:23:29:45:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9414 run:837 clone:1 gen:459 core:0x21 unit:0x00000220ab436c9d585e069659a3736d
2018-01-04:23:29:45:WU00:FS00:Starting
2018-01-04:23:29:45:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:45:WU00:FS00:Started FahCore on PID 23743
2018-01-04:23:29:45:WU00:FS00:Core PID:23747
2018-01-04:23:29:45:WU00:FS00:FahCore 0x21 started
2018-01-04:23:29:45:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:46:WU02:FS02:0x21:Completed 5000000 out of 6250000 steps (80%)
2018-01-04:23:29:46:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:9414 run:837 clone:1 gen:459 core:0x21 unit:0x00000220ab436c9d585e069659a3736d
2018-01-04:23:29:46:WU00:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:46:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:46:WU00:FS00:Upload complete
2018-01-04:23:29:46:WU01:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:46:WU00:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:46:WU00:FS00:Cleaning up
2018-01-04:23:29:47:WU01:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:47:WU01:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:47:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:47:WU01:FS00:Downloading 5.15MiB
2018-01-04:23:29:49:WU01:FS00:Download complete
2018-01-04:23:29:49:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9414 run:1241 clone:1 gen:416 core:0x21 unit:0x000001e3ab436c9d585e0699ae7ef0c9
2018-01-04:23:29:49:WU01:FS00:Starting
2018-01-04:23:29:49:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:49:WU01:FS00:Started FahCore on PID 23756
2018-01-04:23:29:49:WU01:FS00:Core PID:23760
2018-01-04:23:29:49:WU01:FS00:FahCore 0x21 started
2018-01-04:23:29:49:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:50:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:1241 clone:1 gen:416 core:0x21 unit:0x000001e3ab436c9d585e0699ae7ef0c9
2018-01-04:23:29:50:WU01:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:50:WU01:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:50:WU00:FS00:Connecting to 171.67.108.45:80
2018-01-04:23:29:50:WU01:FS00:Upload complete
2018-01-04:23:29:50:WU01:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:50:WU01:FS00:Cleaning up
2018-01-04:23:29:51:WU00:FS00:Assigned to work server 171.67.108.157
2018-01-04:23:29:51:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:29:51:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:51:WU00:FS00:Downloading 8.83MiB
2018-01-04:23:29:54:WU00:FS00:Download complete
2018-01-04:23:29:54:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9431 run:327 clone:3 gen:78 core:0x21 unit:0x0000005aab436c9d586fdd36020a56ca
2018-01-04:23:29:54:WU00:FS00:Starting
2018-01-04:23:29:54:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-04:23:29:54:WU00:FS00:Started FahCore on PID 23776
2018-01-04:23:29:54:WU00:FS00:Core PID:23780
2018-01-04:23:29:54:WU00:FS00:FahCore 0x21 started
2018-01-04:23:29:55:WARNING:WU00:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:29:55:WU00:FS00:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:327 clone:3 gen:78 core:0x21 unit:0x0000005aab436c9d586fdd36020a56ca
2018-01-04:23:29:55:WU00:FS00:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:29:55:WU00:FS00:Connecting to 171.67.108.157:8080
2018-01-04:23:29:55:WU00:FS00:Upload complete
2018-01-04:23:29:55:WU00:FS00:Server responded WORK_ACK (400)
2018-01-04:23:29:55:WU00:FS00:Cleaning up
2018-01-04:23:29:59:WU04:FS04:0x21:Completed 5812500 out of 6250000 steps (93%)
2018-01-04:23:30:27:WU02:FS02:0x21:Completed 5062500 out of 6250000 steps (81%)
2018-01-04:23:30:30:WU03:FS03:0x21:Completed 2437500 out of 6250000 steps (39%)
2018-01-04:23:30:32:WU05:FS01:0x21:Completed 6000000 out of 6250000 steps (96%)
2018-01-04:23:31:02:WU04:FS04:0x21:Completed 5875000 out of 6250000 steps (94%)
2018-01-04:23:31:08:WU02:FS02:0x21:Completed 5125000 out of 6250000 steps (82%)
2018-01-04:23:31:19:WU03:FS03:0x21:Completed 2500000 out of 6250000 steps (40%)
2018-01-04:23:31:24:WU05:FS01:0x21:Completed 6062500 out of 6250000 steps (97%)
2018-01-04:23:31:49:WU02:FS02:0x21:Completed 5187500 out of 6250000 steps (83%)
2018-01-04:23:32:06:WU04:FS04:0x21:Completed 5937500 out of 6250000 steps (95%)
2018-01-04:23:32:08:WU03:FS03:0x21:Completed 2562500 out of 6250000 steps (41%)
2018-01-04:23:32:15:WU05:FS01:0x21:Completed 6125000 out of 6250000 steps (98%)
2018-01-04:23:32:30:WU02:FS02:0x21:Completed 5250000 out of 6250000 steps (84%)
2018-01-04:23:32:57:WU03:FS03:0x21:Completed 2625000 out of 6250000 steps (42%)
2018-01-04:23:33:06:WU05:FS01:0x21:Completed 6187500 out of 6250000 steps (99%)
2018-01-04:23:33:10:WU04:FS04:0x21:Completed 6000000 out of 6250000 steps (96%)
2018-01-04:23:33:11:WU02:FS02:0x21:Completed 5312500 out of 6250000 steps (85%)
2018-01-04:23:33:46:WU03:FS03:0x21:Completed 2687500 out of 6250000 steps (43%)
2018-01-04:23:33:52:WU02:FS02:0x21:Completed 5375000 out of 6250000 steps (86%)
2018-01-04:23:33:57:WU05:FS01:0x21:Completed 6250000 out of 6250000 steps (100%)
2018-01-04:23:33:58:WU00:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:33:58:WU00:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:33:58:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:33:58:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:33:59:WU05:FS01:0x21:Saving result file logfile_01.txt
2018-01-04:23:33:59:WU05:FS01:0x21:Saving result file checkpointState.xml
2018-01-04:23:33:59:WU05:FS01:0x21:Saving result file checkpt.crc
2018-01-04:23:33:59:WU05:FS01:0x21:Saving result file log.txt
2018-01-04:23:33:59:WU05:FS01:0x21:Saving result file positions.xtc
2018-01-04:23:33:59:WU00:FS01:Downloading 5.18MiB
2018-01-04:23:33:59:WU05:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
2018-01-04:23:33:59:WU05:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
2018-01-04:23:33:59:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:9431 run:1817 clone:3 gen:54 core:0x21 unit:0x00000042ab436c9d586fdd42a3a17344
2018-01-04:23:33:59:WU05:FS01:Uploading 13.64MiB to 171.67.108.157
2018-01-04:23:33:59:WU05:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:01:WU00:FS01:Download complete
2018-01-04:23:34:01:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9415 run:2500 clone:0 gen:537 core:0x21 unit:0x00000275ab436c9d585e06e0df341911
2018-01-04:23:34:01:WU00:FS01:Starting
2018-01-04:23:34:01:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:01:WU00:FS01:Started FahCore on PID 24442
2018-01-04:23:34:01:WU00:FS01:Core PID:24446
2018-01-04:23:34:01:WU00:FS01:FahCore 0x21 started
2018-01-04:23:34:01:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:01:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9415 run:2500 clone:0 gen:537 core:0x21 unit:0x00000275ab436c9d585e06e0df341911
2018-01-04:23:34:01:WU00:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:01:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:02:WU00:FS01:Upload complete
2018-01-04:23:34:02:WU01:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:02:WU00:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:02:WU00:FS01:Cleaning up
2018-01-04:23:34:02:WU01:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:02:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:02:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:02:WU01:FS01:Downloading 8.85MiB
2018-01-04:23:34:05:WU05:FS01:Upload 24.29%
2018-01-04:23:34:05:WU01:FS01:Download complete
2018-01-04:23:34:05:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9431 run:518 clone:1 gen:319 core:0x21 unit:0x00000179ab436c9d586fdd3863938bbc
2018-01-04:23:34:05:WU01:FS01:Starting
2018-01-04:23:34:05:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:05:WU01:FS01:Started FahCore on PID 24460
2018-01-04:23:34:05:WU01:FS01:Core PID:24464
2018-01-04:23:34:05:WU01:FS01:FahCore 0x21 started
2018-01-04:23:34:05:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:05:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9431 run:518 clone:1 gen:319 core:0x21 unit:0x00000179ab436c9d586fdd3863938bbc
2018-01-04:23:34:05:WU01:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:05:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:06:WU01:FS01:Upload complete
2018-01-04:23:34:06:WU00:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:06:WU01:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:06:WU01:FS01:Cleaning up
2018-01-04:23:34:06:WU00:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:06:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:06:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:07:WU00:FS01:Downloading 8.83MiB
2018-01-04:23:34:09:WU00:FS01:Download complete
2018-01-04:23:34:09:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9431 run:412 clone:2 gen:458 core:0x21 unit:0x0000021eab436c9d586fdd37fa3a5c4e
2018-01-04:23:34:09:WU00:FS01:Starting
2018-01-04:23:34:09:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:09:WU00:FS01:Started FahCore on PID 24474
2018-01-04:23:34:09:WU00:FS01:Core PID:24478
2018-01-04:23:34:09:WU00:FS01:FahCore 0x21 started
2018-01-04:23:34:09:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:09:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:412 clone:2 gen:458 core:0x21 unit:0x0000021eab436c9d586fdd37fa3a5c4e
2018-01-04:23:34:09:WU00:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:09:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:10:WU00:FS01:Upload complete
2018-01-04:23:34:10:WU01:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:10:WU00:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:10:WU00:FS01:Cleaning up
2018-01-04:23:34:10:WU01:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:10:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:10:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:11:WU05:FS01:Upload 68.73%
2018-01-04:23:34:11:WU01:FS01:Downloading 5.15MiB
2018-01-04:23:34:12:WU01:FS01:Download complete
2018-01-04:23:34:12:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9415 run:738 clone:4 gen:170 core:0x21 unit:0x000000c7ab436c9d585e06cfd59c10e8
2018-01-04:23:34:12:WU01:FS01:Starting
2018-01-04:23:34:12:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:12:WU01:FS01:Started FahCore on PID 24487
2018-01-04:23:34:12:WU01:FS01:Core PID:24491
2018-01-04:23:34:12:WU01:FS01:FahCore 0x21 started
2018-01-04:23:34:13:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:13:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9415 run:738 clone:4 gen:170 core:0x21 unit:0x000000c7ab436c9d585e06cfd59c10e8
2018-01-04:23:34:13:WU01:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:13:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:13:WU01:FS01:Upload complete
2018-01-04:23:34:13:WU00:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:13:WU01:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:13:WU01:FS01:Cleaning up
2018-01-04:23:34:13:WU00:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:13:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:13:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:14:WU00:FS01:Downloading 8.83MiB
2018-01-04:23:34:15:WU04:FS04:0x21:Completed 6062500 out of 6250000 steps (97%)
2018-01-04:23:34:15:WU05:FS01:Upload complete
2018-01-04:23:34:15:WU05:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:15:WU05:FS01:Final credit estimate, 58902.00 points
2018-01-04:23:34:15:WU05:FS01:Cleaning up
******************************* Date: 2018-01-04 *******************************
2018-01-04:23:34:16:WU00:FS01:Download complete
2018-01-04:23:34:16:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9431 run:1464 clone:0 gen:637 core:0x21 unit:0x000002f4ab436c9d586fdd3f2612ae5c
2018-01-04:23:34:16:WU00:FS01:Starting
2018-01-04:23:34:16:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:16:WU00:FS01:Started FahCore on PID 24499
2018-01-04:23:34:16:WU00:FS01:Core PID:24503
2018-01-04:23:34:16:WU00:FS01:FahCore 0x21 started
2018-01-04:23:34:17:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:17:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:1464 clone:0 gen:637 core:0x21 unit:0x000002f4ab436c9d586fdd3f2612ae5c
2018-01-04:23:34:17:WU00:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:17:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:17:WU01:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:17:WU00:FS01:Upload complete
2018-01-04:23:34:17:WU00:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:17:WU00:FS01:Cleaning up
2018-01-04:23:34:18:WU01:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:18:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:18:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:19:WU01:FS01:Downloading 5.15MiB
2018-01-04:23:34:20:WU01:FS01:Download complete
2018-01-04:23:34:20:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9414 run:1506 clone:1 gen:640 core:0x21 unit:0x000002e1ab436c9d585e069b6871db88
2018-01-04:23:34:20:WU01:FS01:Starting
2018-01-04:23:34:20:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:20:WU01:FS01:Started FahCore on PID 24512
2018-01-04:23:34:20:WU01:FS01:Core PID:24516
2018-01-04:23:34:20:WU01:FS01:FahCore 0x21 started
2018-01-04:23:34:21:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:21:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:1506 clone:1 gen:640 core:0x21 unit:0x000002e1ab436c9d585e069b6871db88
2018-01-04:23:34:21:WU01:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:21:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:21:WU01:FS01:Upload complete
2018-01-04:23:34:21:WU01:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:21:WU00:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:21:WU01:FS01:Cleaning up
2018-01-04:23:34:21:WU00:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:21:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:21:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:22:WU00:FS01:Downloading 8.88MiB
2018-01-04:23:34:24:WU00:FS01:Download complete
2018-01-04:23:34:24:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9431 run:758 clone:2 gen:46 core:0x21 unit:0x00000038ab436c9d586fdd39b9dd104d
2018-01-04:23:34:24:WU00:FS01:Starting
2018-01-04:23:34:24:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:24:WU00:FS01:Started FahCore on PID 24529
2018-01-04:23:34:24:WU00:FS01:Core PID:24534
2018-01-04:23:34:24:WU00:FS01:FahCore 0x21 started
2018-01-04:23:34:25:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:25:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:758 clone:2 gen:46 core:0x21 unit:0x00000038ab436c9d586fdd39b9dd104d
2018-01-04:23:34:25:WU00:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:25:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:25:WU01:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:25:WU00:FS01:Upload complete
2018-01-04:23:34:25:WU00:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:25:WU00:FS01:Cleaning up
2018-01-04:23:34:26:WU01:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:26:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:26:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:26:WU01:FS01:Downloading 5.16MiB
2018-01-04:23:34:27:WU01:FS01:Download complete
2018-01-04:23:34:27:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9414 run:1750 clone:0 gen:678 core:0x21 unit:0x0000031fab436c9d585e069d38f35b74
2018-01-04:23:34:27:WU01:FS01:Starting
2018-01-04:23:34:27:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:28:WU01:FS01:Started FahCore on PID 24542
2018-01-04:23:34:28:WU01:FS01:Core PID:24547
2018-01-04:23:34:28:WU01:FS01:FahCore 0x21 started
2018-01-04:23:34:28:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:28:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9414 run:1750 clone:0 gen:678 core:0x21 unit:0x0000031fab436c9d585e069d38f35b74
2018-01-04:23:34:28:WU01:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:28:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:28:WU01:FS01:Upload complete
2018-01-04:23:34:28:WU00:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:28:WU01:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:28:WU01:FS01:Cleaning up
2018-01-04:23:34:29:WU00:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:29:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:29:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:29:WU00:FS01:Downloading 8.82MiB
2018-01-04:23:34:31:WU00:FS01:Download complete
2018-01-04:23:34:31:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9431 run:1041 clone:2 gen:171 core:0x21 unit:0x000000c9ab436c9d586fdd3c993265c4
2018-01-04:23:34:31:WU00:FS01:Starting
2018-01-04:23:34:31:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-04:23:34:32:WU00:FS01:Started FahCore on PID 24559
2018-01-04:23:34:32:WU00:FS01:Core PID:24563
2018-01-04:23:34:32:WU00:FS01:FahCore 0x21 started
2018-01-04:23:34:32:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
2018-01-04:23:34:32:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9431 run:1041 clone:2 gen:171 core:0x21 unit:0x000000c9ab436c9d586fdd3c993265c4
2018-01-04:23:34:32:WU00:FS01:Uploading 5.50KiB to 171.67.108.157
2018-01-04:23:34:32:WU00:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:32:WU00:FS01:Upload complete
2018-01-04:23:34:32:WU00:FS01:Server responded WORK_ACK (400)
2018-01-04:23:34:32:WU01:FS01:Connecting to 171.67.108.45:80
2018-01-04:23:34:32:WU00:FS01:Cleaning up
2018-01-04:23:34:33:WU01:FS01:Assigned to work server 171.67.108.157
2018-01-04:23:34:33:WU01:FS01:Requesting new work unit for slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 from 171.67.108.157
2018-01-04:23:34:33:WU01:FS01:Connecting to 171.67.108.157:8080
2018-01-04:23:34:33:WU02:FS02:0x21:Completed 5437500 out of 6250000 steps (87%)

Code: Select all

steve@fah02 /var/lib/fahclient/logs $ head -600  log-20180104-235819.txt |grep -v topology
*********************** Log Started 2018-01-02T11:33:45Z ***********************
2018-01-02:11:33:45:******************* Folding@home Client ********************
2018-01-02:11:33:45:      Website: http://folding.stanford.edu/
2018-01-02:11:33:45:    Copyright: (c) 2009-2016 Stanford University
2018-01-02:11:33:45:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
2018-01-02:11:33:45:         Args: --child --lifeline 1676 /etc/fahclient/config.xml --run-as
2018-01-02:11:33:45:               fahclient --pid-file=/var/run/fahclient.pid --daemon
2018-01-02:11:33:45:       Config: /etc/fahclient/config.xml
2018-01-02:11:33:45:************************** Build ***************************
2018-01-02:11:33:45:      Version: 7.4.16
2018-01-02:11:33:45:         Date: Jan 6 2017
2018-01-02:11:33:45:         Time: 08:08:33
2018-01-02:11:33:45:   Repository: Git
2018-01-02:11:33:45:     Revision: e12187cbb0bd6937c067b9749af011374563b7b9
2018-01-02:11:33:45:       Branch: master
2018-01-02:11:33:45:     Compiler: GNU 4.9.2
2018-01-02:11:33:45:      Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
2018-01-02:11:33:45:               -fno-unsafe-math-optimizations -msse2
2018-01-02:11:33:45:     Platform: linux2 4.8.0-2-amd64
2018-01-02:11:33:45:         Bits: 64
2018-01-02:11:33:45:         Mode: Release
2018-01-02:11:33:45:************************** System **************************
2018-01-02:11:33:45:          CPU: AMD FX(tm)-6300 Six-Core Processor
2018-01-02:11:33:45:       CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
2018-01-02:11:33:45:         CPUs: 6
2018-01-02:11:33:45:       Memory: 7.70GiB
2018-01-02:11:33:45:  Free Memory: 7.05GiB
2018-01-02:11:33:45:      Threads: POSIX_THREADS
2018-01-02:11:33:45:   OS Version: 4.4
2018-01-02:11:33:45:  Has Battery: false
2018-01-02:11:33:45:   On Battery: false
2018-01-02:11:33:45:   UTC Offset: -6
2018-01-02:11:33:45:          PID: 1678
2018-01-02:11:33:45:          CWD: /var/lib/fahclient
2018-01-02:11:33:45:           OS: Linux 4.4.0-53-generic x86_64
2018-01-02:11:33:45:      OS Arch: AMD64
2018-01-02:11:33:45:         GPUs: 5
2018-01-02:11:33:45:        GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
2018-01-02:11:33:45:        GPU 1: Bus:8 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
2018-01-02:11:33:45:        GPU 2: Bus:9 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
2018-01-02:11:33:45:        GPU 3: Bus:10 Slot:0 Func:0 NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
2018-01-02:11:33:45:        GPU 4: Bus:13 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1080] 8873
2018-01-02:11:33:45:CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:9.1
2018-01-02:11:33:45:CUDA Device 1: Platform:0 Device:1 Bus:8 Slot:0 Compute:6.1 Driver:9.1
2018-01-02:11:33:45:CUDA Device 2: Platform:0 Device:2 Bus:9 Slot:0 Compute:6.1 Driver:9.1
2018-01-02:11:33:45:CUDA Device 3: Platform:0 Device:3 Bus:10 Slot:0 Compute:6.1 Driver:9.1
2018-01-02:11:33:45:CUDA Device 4: Platform:0 Device:4 Bus:13 Slot:0 Compute:6.1 Driver:9.1
2018-01-02:11:33:45:       OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
2018-01-02:11:33:45:               libOpenCL.so: cannot open shared object file: No such file or
2018-01-02:11:33:45:               directory
2018-01-02:11:33:45:************************************************************
2018-01-02:11:33:45:<config>
2018-01-02:11:33:45:  <!-- Client Control -->
2018-01-02:11:33:45:  <fold-anon v='true'/>
2018-01-02:11:33:45:
2018-01-02:11:33:45:  <!-- Folding Slot Configuration -->
2018-01-02:11:33:45:  <gpu v='false'/>
2018-01-02:11:33:45:
2018-01-02:11:33:45:  <!-- Logging -->
2018-01-02:11:33:45:  <log-date v='true'/>
2018-01-02:11:33:45:  <log-rotate-max v='128'/>
2018-01-02:11:33:45:
2018-01-02:11:33:45:  <!-- Network -->
2018-01-02:11:33:45:  <proxy v=':8080'/>
2018-01-02:11:33:45:
2018-01-02:11:33:45:  <!-- Slot Control -->
2018-01-02:11:33:45:  <power v='full'/>
2018-01-02:11:33:45:
2018-01-02:11:33:45:  <!-- User Information -->
2018-01-02:11:33:45:  <passkey v='********************************'/>
2018-01-02:11:33:45:  <team v='224497'/>
2018-01-02:11:33:45:  <user v='DarthMouse_ALL_1GD5nCZbh7gNo1SESPLT24xEd2Jsu4rTP9'/>
2018-01-02:11:33:45:
2018-01-02:11:33:45:  <!-- Work Unit Control -->
2018-01-02:11:33:45:  <next-unit-percentage v='100'/>
2018-01-02:11:33:45:
2018-01-02:11:33:45:  <!-- Folding Slots -->
2018-01-02:11:33:45:  <slot id='0' type='GPU'>
2018-01-02:11:33:45:    <gpu-index v='0'/>
2018-01-02:11:33:45:    <opencl-index v='0'/>
2018-01-02:11:33:45:  </slot>
2018-01-02:11:33:45:  <slot id='1' type='GPU'>
2018-01-02:11:33:45:    <gpu-index v='1'/>
2018-01-02:11:33:45:    <opencl-index v='1'/>
2018-01-02:11:33:45:  </slot>
2018-01-02:11:33:45:  <slot id='2' type='GPU'>
2018-01-02:11:33:45:    <gpu-index v='2'/>
2018-01-02:11:33:45:    <opencl-index v='2'/>
2018-01-02:11:33:45:  </slot>
2018-01-02:11:33:45:  <slot id='3' type='GPU'>
2018-01-02:11:33:45:    <gpu-index v='3'/>
2018-01-02:11:33:45:    <opencl-index v='3'/>
2018-01-02:11:33:45:  </slot>
2018-01-02:11:33:45:  <slot id='4' type='GPU'>
2018-01-02:11:33:45:    <gpu-index v='4'/>
2018-01-02:11:33:45:    <opencl-index v='4'/>
2018-01-02:11:33:45:  </slot>
2018-01-02:11:33:45:</config>
2018-01-02:11:33:45:Switching to user fahclient
2018-01-02:11:33:45:Trying to access database...
2018-01-02:11:33:45:Successfully acquired database lock
2018-01-02:11:33:45:Enabled folding slot 00: READY gpu:0:GP102 [GeForce GTX 1080 Ti] 11380
2018-01-02:11:33:45:Enabled folding slot 01: READY gpu:1:GP102 [GeForce GTX 1080 Ti] 11380
2018-01-02:11:33:45:Enabled folding slot 02: READY gpu:2:GP102 [GeForce GTX 1080 Ti] 11380
2018-01-02:11:33:45:Enabled folding slot 03: READY gpu:3:GP102 [GeForce GTX 1080 Ti] 11380
2018-01-02:11:33:45:Enabled folding slot 04: READY gpu:4:GP104 [GeForce GTX 1080] 8873
2018-01-02:11:33:45:WU02:FS02:Starting
2018-01-02:11:33:45:WU02:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 02 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 2 -cuda-device 2 -gpu 2
2018-01-02:11:33:45:WU02:FS02:Started FahCore on PID 1688
2018-01-02:11:33:45:WU02:FS02:Core PID:1692
2018-01-02:11:33:45:WU02:FS02:FahCore 0x21 started
2018-01-02:11:33:45:WU01:FS04:Starting
2018-01-02:11:33:45:WU01:FS04:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 4 -cuda-device 4 -gpu 4
2018-01-02:11:33:45:WU01:FS04:Started FahCore on PID 1693
2018-01-02:11:33:45:WU01:FS04:Core PID:1697
2018-01-02:11:33:45:WU01:FS04:FahCore 0x21 started
2018-01-02:11:33:45:WU00:FS01:Starting
2018-01-02:11:33:45:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 1 -cuda-device 1 -gpu 1
2018-01-02:11:33:45:WU00:FS01:Started FahCore on PID 1698
2018-01-02:11:33:45:WU00:FS01:Core PID:1702
2018-01-02:11:33:45:WU00:FS01:FahCore 0x21 started
2018-01-02:11:33:45:WU03:FS00:Starting
2018-01-02:11:33:45:WU03:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 03 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 0 -cuda-device 0 -gpu 0
2018-01-02:11:33:45:WU03:FS00:Started FahCore on PID 1703
2018-01-02:11:33:45:WU03:FS00:Core PID:1707
2018-01-02:11:33:45:WU03:FS00:FahCore 0x21 started
2018-01-02:11:33:46:WU04:FS03:Starting
2018-01-02:11:33:46:WU04:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 04 -suffix 01 -version 704 -lifeline 1678 -checkpoint 15 -gpu-vendor nvidia -opencl-device 3 -cuda-device 3 -gpu 3
2018-01-02:11:33:46:WU04:FS03:Started FahCore on PID 1713
2018-01-02:11:33:46:WU04:FS03:Core PID:1717
2018-01-02:11:33:46:WU04:FS03:FahCore 0x21 started
2018-01-02:11:33:46:WU03:FS00:0x21:*********************** Log Started 2018-01-02T11:33:46Z ***********************
2018-01-02:11:33:46:WU03:FS00:0x21:Project: 9431 (Run 844, Clone 1, Gen 458)
2018-01-02:11:33:46:WU03:FS00:0x21:Unit: 0x0000021fab436c9d586fdd3abd0151b2
2018-01-02:11:33:46:WU03:FS00:0x21:CPU: 0x00000000000000000000000000000000
2018-01-02:11:33:46:WU03:FS00:0x21:Machine: 0
2018-01-02:11:33:46:WU03:FS00:0x21:Digital signatures verified
2018-01-02:11:33:46:WU03:FS00:0x21:Folding@home GPU Core21 Folding@home Core
2018-01-02:11:33:46:WU03:FS00:0x21:Version 0.0.18
2018-01-02:11:33:46:WU00:FS01:0x21:*********************** Log Started 2018-01-02T11:33:46Z ***********************
2018-01-02:11:33:46:WU00:FS01:0x21:Project: 9431 (Run 580, Clone 2, Gen 123)
2018-01-02:11:33:46:WU00:FS01:0x21:Unit: 0x000000a3ab436c9d586fdd387ea49cf3
2018-01-02:11:33:46:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
2018-01-02:11:33:46:WU00:FS01:0x21:Machine: 1
2018-01-02:11:33:46:WU00:FS01:0x21:Digital signatures verified
2018-01-02:11:33:46:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
2018-01-02:11:33:46:WU00:FS01:0x21:Version 0.0.18
2018-01-02:11:33:47:WU01:FS04:0x21:*********************** Log Started 2018-01-02T11:33:46Z ***********************
2018-01-02:11:33:47:WU01:FS04:0x21:Project: 9414 (Run 1634, Clone 2, Gen 55)
2018-01-02:11:33:47:WU01:FS04:0x21:Unit: 0x00000045ab436c9d585e069c935cb8f3
2018-01-02:11:33:47:WU01:FS04:0x21:CPU: 0x00000000000000000000000000000000
2018-01-02:11:33:47:WU01:FS04:0x21:Machine: 4
2018-01-02:11:33:47:WU01:FS04:0x21:Digital signatures verified
2018-01-02:11:33:47:WU01:FS04:0x21:Folding@home GPU Core21 Folding@home Core
2018-01-02:11:33:47:WU01:FS04:0x21:Version 0.0.18
2018-01-02:11:33:47:WU02:FS02:0x21:*********************** Log Started 2018-01-02T11:33:46Z ***********************
2018-01-02:11:33:47:WU02:FS02:0x21:Project: 9415 (Run 1236, Clone 1, Gen 601)
2018-01-02:11:33:47:WU02:FS02:0x21:Unit: 0x000002c3ab436c9d585e06d441d75634
2018-01-02:11:33:47:WU02:FS02:0x21:CPU: 0x00000000000000000000000000000000
2018-01-02:11:33:47:WU02:FS02:0x21:Machine: 2
2018-01-02:11:33:47:WU02:FS02:0x21:Digital signatures verified
2018-01-02:11:33:47:WU02:FS02:0x21:Folding@home GPU Core21 Folding@home Core
2018-01-02:11:33:47:WU02:FS02:0x21:Version 0.0.18
2018-01-02:11:33:47:WU04:FS03:0x21:*********************** Log Started 2018-01-02T11:33:46Z ***********************
2018-01-02:11:33:47:WU04:FS03:0x21:Project: 9414 (Run 1183, Clone 2, Gen 124)
2018-01-02:11:33:47:WU04:FS03:0x21:Unit: 0x0000009aab436c9d585e0699016d77b4
2018-01-02:11:33:47:WU04:FS03:0x21:CPU: 0x00000000000000000000000000000000
2018-01-02:11:33:47:WU04:FS03:0x21:Machine: 3
2018-01-02:11:33:47:WU04:FS03:0x21:Digital signatures verified
2018-01-02:11:33:47:WU04:FS03:0x21:Folding@home GPU Core21 Folding@home Core
2018-01-02:11:33:47:WU04:FS03:0x21:Version 0.0.18
2018-01-02:11:33:47:WU01:FS04:0x21:  Found a checkpoint file
2018-01-02:11:33:47:WU02:FS02:0x21:  Found a checkpoint file
2018-01-02:11:33:47:WU04:FS03:0x21:  Found a checkpoint file
2018-01-02:11:33:47:WU03:FS00:0x21:  Found a checkpoint file
2018-01-02:11:33:47:WU00:FS01:0x21:  Found a checkpoint file
2018-01-02:11:34:04:WU04:FS03:0x21:Completed 4700000 out of 6250000 steps (75%)
2018-01-02:11:34:04:WU04:FS03:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2018-01-02:11:34:04:WU01:FS04:0x21:Completed 2700000 out of 6250000 steps (43%)
2018-01-02:11:34:04:WU01:FS04:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2018-01-02:11:34:04:WU02:FS02:0x21:Completed 650000 out of 6250000 steps (10%)
2018-01-02:11:34:04:WU02:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2018-01-02:11:34:04:WU00:FS01:0x21:Completed 3750000 out of 6250000 steps (60%)
2018-01-02:11:34:04:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2018-01-02:11:34:05:WU03:FS00:0x21:Completed 5950000 out of 6250000 steps (95%)
2018-01-02:11:34:05:WU03:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2018-01-02:11:34:30:WU02:FS02:0x21:Completed 687500 out of 6250000 steps (11%)

As I said it's folding fine now so I don't really need any help.
I added code to my auto-reboot script to check the log.txt file once a minute for BAD_WORK_UNIT errors and reboot if I get two of them.
I guess I'm just wondering if anyone had seen this before.

Post by **bruce** » Sun Jan 07, 2018 12:33 am

I only checked the first one, but the WU was reassigned and completed shortly after your error.

Hi DarthMouse_ALL....
Your WU (P9415 R2587 C2 G245) was added to the stats database on 2018-01-04 16:15:24 for 0 points of credit.

Hi xxxx (team xxx),
Your WU (P9415 R2587 C2 G245) was added to the stats database on 2018-01-04 17:15:21 for 49518 points of credit.

I have to suppose that something strange happened to your GPUs or the drivers. which was cleared by the reboot. I''m not sure if the early message about failure to find openCL.so is related or not.

SteveWillis · Post by **SteveWillis** » Sun Jan 07, 2018 1:11 am

Thanks, Bruce. All three of my machines have the openCL error in all their logs but it has never caused any problem that I know of. I just keep happily folding away. I agree that there was nothing wrong with the actual WU projects. There didn't seem to be any pattern there. I'm just glad I had the reboot script running. It could have gone on for hours otherwise. I guess I was just curious whether anyone else had ever had it happen. Anyway, I haven't seen any problem since.

Folding Forum

40 BAD_WORK_UNIT errors in 15 minutes!!!

40 BAD_WORK_UNIT errors in 15 minutes!!!

Re: 40 BAD_WORK_UNIT errors in 15 minutes!!!

Re: 40 BAD_WORK_UNIT errors in 15 minutes!!!