Folding Forum

Sorry it's taken me so long to get back, very sorry, but a lot of irons in the fire right now. ALL my GPU jobs have failed, 15 of them on 5 machines. I presume this isn't happening to most people since I see no reference to it in new posts. It appears that on all my machines this stuff started, at varying times, but all on 02/01/17. I have some doubts as to whether what I have collected from just 2 of my rigs will fit here for purposes of posting, but I'll try.


MACHINE 1 BY GPU

01

*********************** Log Started 2017-01-31T14:04:37Z ***********************
15:32:01:ERROR:WU01:FS01:Exception: Server did not assign work unit
******************************* Date: 2017-01-31 *******************************
******************************* Date: 2017-02-01 *******************************
06:54:19:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:54:28:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:55:10:WARNING:WU05:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:55:11:ERROR:WU03:FS01:Exception: Server did not assign work unit
******************************* Date: 2017-02-01 *******************************
08:16:59:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:17:15:WARNING:WU04:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:17:26:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:17:34:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:15:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:24:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:25:ERROR:WU00:FS01:Exception: Server did not assign work unit
08:18:36:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:44:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:50:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:19:02:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-03 *******************************
******************************* Date: 2017-02-03 *******************************

02

*********************** Log Started 2017-01-31T14:04:37Z ***********************
******************************* Date: 2017-01-31 *******************************
******************************* Date: 2017-02-01 *******************************
06:53:48:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:54:24:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:55:08:WARNING:WU03:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:56:07:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:56:16:WARNING:WU04:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:56:25:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:56:32:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:57:36:WARNING:WU04:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:57:44:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:57:52:WARNING:WU04:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-03 *******************************
******************************* Date: 2017-02-03 *******************************

03

*********************** Log Started 2017-01-31T14:04:37Z ***********************
******************************* Date: 2017-01-31 *******************************
******************************* Date: 2017-02-01 *******************************
03:53:38:WARNING:WU01:FS03:FahCore returned: CORE_OUTDATED (110 = 0x6e)
06:53:46:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:54:25:WARNING:WU05:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:55:40:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:55:50:WARNING:WU04:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
******************************* Date: 2017-02-01 *******************************
08:34:26:WARNING:WU00:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:35:06:WARNING:WU03:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:35:15:WARNING:WU00:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:36:11:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:36:21:WARNING:WU03:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:36:37:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:37:32:WARNING:WU02:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:37:37:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:38:17:WARNING:WU02:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:38:24:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-03 *******************************
******************************* Date: 2017-02-03 *******************************

MACHINE 1 WARNINGS AND ERRORS

*********************** Log Started 2017-01-31T14:04:37Z ***********************
14:04:40:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.45:8080': Empty work server assignment
14:04:41:WARNING:WU00:FS00:Failed to get assignment from '171.64.65.35:80': Empty work server assignment
14:04:41:ERROR:WU00:FS00:Exception: Could not get an assignment
14:04:42:WARNING:WU00:FS00:Failed to get assignment from '171.67.108.45:8080': Empty work server assignment
14:04:42:WARNING:WU00:FS00:Failed to get assignment from '171.64.65.35:80': Empty work server assignment
14:04:42:ERROR:WU00:FS00:Exception: Could not get an assignment
15:32:01:ERROR:WU01:FS01:Exception: Server did not assign work unit
******************************* Date: 2017-01-31 *******************************
******************************* Date: 2017-02-01 *******************************
03:53:38:WARNING:WU01:FS03:FahCore returned: CORE_OUTDATED (110 = 0x6e)
06:53:45:WU01:FS03:0x21:ERROR:Force RMSE error of 401.267 with threshold of 5
06:53:46:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:53:48:WU02:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
06:53:48:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:53:49:WU03:FS01:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
06:54:19:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:54:24:WU01:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
06:54:24:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:54:25:WU05:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
06:54:25:WARNING:WU05:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:54:28:WU02:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
06:54:28:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:54:38:WU03:FS02:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
06:55:08:WARNING:WU03:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:55:09:WU05:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
06:55:09:WU01:FS03:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
06:55:10:WARNING:WU05:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:55:11:ERROR:WU03:FS01:Exception: Server did not assign work unit
06:55:40:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:55:49:WU04:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
06:55:50:WARNING:WU04:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:56:07:WU02:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
06:56:07:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:56:15:WU04:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 2178 0
06:56:16:WARNING:WU04:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:56:25:WU02:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
06:56:25:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:56:31:WU02:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
06:56:32:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:57:05:WU04:FS02:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
06:57:36:WARNING:WU04:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:57:44:WU02:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
06:57:44:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
06:57:51:WU04:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
06:57:52:WARNING:WU04:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
******************************* Date: 2017-02-01 *******************************
08:16:59:WU00:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
08:16:59:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:17:14:WU04:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
08:17:15:WARNING:WU04:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:17:26:WU00:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
08:17:26:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:17:33:WU03:FS01:0x21:ERROR:Force RMSE error of 727.184 with threshold of 5
08:17:34:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:17:45:WU00:FS01:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
08:18:15:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:24:WU03:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1235 0
08:18:24:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:25:ERROR:WU00:FS01:Exception: Server did not assign work unit
08:18:35:WU00:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
08:18:36:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:43:WU03:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
08:18:44:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:18:50:WU00:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 3360 0
08:18:50:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:19:02:WU03:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
08:19:02:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:34:26:WU00:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
08:34:26:WARNING:WU00:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:35:05:WU03:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 5206 0
08:35:06:WARNING:WU03:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:35:15:WU00:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
08:35:15:WARNING:WU00:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:35:41:WU01:FS03:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
08:36:11:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:36:21:WU03:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
08:36:21:WARNING:WU03:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:36:36:WU01:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
08:36:37:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:37:02:WU02:FS03:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
08:37:32:WARNING:WU02:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:37:37:WU01:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
08:37:37:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:37:46:WU02:FS03:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
08:38:17:WARNING:WU02:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
08:38:24:WU01:FS03:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
08:38:24:WARNING:WU01:FS03:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-03 *******************************
******************************* Date: 2017-02-03 *******************************



MACHINE 2, GPU 01

10:00:18:WU02:FS01:0x21:Completed 7350000 out of 7500000 steps (98%)
10:02:22:WU02:FS01:0x21:Completed 7425000 out of 7500000 steps (99%)
10:02:23:WU01:FS01:Connecting to 171.67.108.45:80
10:02:23:WU01:FS01:Assigned to work server 171.64.65.92
10:02:23:WU01:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:GP104 [GeForce GTX 1080] from 171.64.65.92
10:02:23:WU01:FS01:Connecting to 171.64.65.92:8080
10:02:23:WU01:FS01:Downloading 2.52MiB
10:02:24:WU01:FS01:Download complete
10:02:24:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9194 run:0 clone:15 gen:327 core:0x21 unit:0x000001ddab40415c57cb2df010bcacac
10:04:26:WU02:FS01:0x21:Completed 7500000 out of 7500000 steps (100%)
10:04:28:WU02:FS01:0x21:Saving result file logfile_01.txt
10:04:28:WU02:FS01:0x21:Saving result file checkpointState.xml
10:04:28:WU02:FS01:0x21:Saving result file checkpt.crc
10:04:28:WU02:FS01:0x21:Saving result file log.txt
10:04:28:WU02:FS01:0x21:Saving result file positions.xtc
10:04:28:WU02:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
10:04:58:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
10:04:58:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11712 run:0 clone:58 gen:62 core:0x21 unit:0x0000004c8ca304e758332b4fe4596f07
10:04:58:WU02:FS01:Uploading 11.80MiB to 140.163.4.231
10:04:58:WU02:FS01:Connecting to 140.163.4.231:8080
10:05:04:WU02:FS01:Upload 38.66%
10:05:10:WU02:FS01:Upload 75.74%
10:05:17:WU02:FS01:Upload complete
10:05:17:WU02:FS01:Server responded WORK_ACK (400)
10:05:17:WU02:FS01:Final credit estimate, 118625.00 points
10:05:17:WU02:FS01:Cleaning up
10:50:06:WU01:FS01:Starting
10:50:06:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
10:50:06:WU01:FS01:Started FahCore on PID 10921
10:50:06:WU01:FS01:Core PID:10925
10:50:06:WU01:FS01:FahCore 0x21 started
10:50:07:WU01:FS01:0x21:*********************** Log Started 2017-02-01T10:50:06Z ***********************
10:50:07:WU01:FS01:0x21:Project: 9194 (Run 0, Clone 15, Gen 327)
10:50:07:WU01:FS01:0x21:Unit: 0x000001ddab40415c57cb2df010bcacac
10:50:07:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
10:50:07:WU01:FS01:0x21:Machine: 1
10:50:07:WU01:FS01:0x21:Reading tar file core.xml
10:50:07:WU01:FS01:0x21:Reading tar file system.xml
10:50:07:WU01:FS01:0x21:Reading tar file integrator.xml
10:50:07:WU01:FS01:0x21:Reading tar file state.xml
10:50:07:WU01:FS01:0x21:Digital signatures verified
10:50:07:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
10:50:07:WU01:FS01:0x21:Version 0.0.18
10:50:41:WU01:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
10:50:41:WU01:FS01:0x21:Saving result file logfile_01.txt
10:50:41:WU01:FS01:0x21:Saving result file log.txt
10:50:41:WU01:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:51:11:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:51:11:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9194 run:0 clone:15 gen:327 core:0x21 unit:0x000001ddab40415c57cb2df010bcacac
10:51:11:WU01:FS01:Uploading 2.45KiB to 171.64.65.92
10:51:11:WU01:FS01:Connecting to 171.64.65.92:8080
10:51:12:WU01:FS01:Upload complete
10:51:12:WU01:FS01:Server responded WORK_ACK (400)
10:51:12:WU01:FS01:Cleaning up
10:51:12:WU00:FS01:Connecting to 171.67.108.45:80
10:51:12:WU00:FS01:Assigned to work server 171.67.108.159
10:51:12:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 171.67.108.159
10:51:12:WU00:FS01:Connecting to 171.67.108.159:8080
10:51:12:WU00:FS01:Downloading 24.81MiB
10:51:16:WU00:FS01:Download complete
10:51:16:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9179 run:9 clone:7 gen:181 core:0x21 unit:0x00000133ab436c9f57bdce0465ed9bf5
10:51:16:WU00:FS01:Starting
10:51:16:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
10:51:16:WU00:FS01:Started FahCore on PID 11031
10:51:16:WU00:FS01:Core PID:11035
10:51:16:WU00:FS01:FahCore 0x21 started
10:51:17:WU00:FS01:0x21:*********************** Log Started 2017-02-01T10:51:16Z ***********************
10:51:17:WU00:FS01:0x21:Project: 9179 (Run 9, Clone 7, Gen 181)
10:51:17:WU00:FS01:0x21:Unit: 0x00000133ab436c9f57bdce0465ed9bf5
10:51:17:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
10:51:17:WU00:FS01:0x21:Machine: 1
10:51:17:WU00:FS01:0x21:Reading tar file core.xml
10:51:17:WU00:FS01:0x21:Reading tar file integrator.xml
10:51:17:WU00:FS01:0x21:Reading tar file state.xml
10:51:17:WU00:FS01:0x21:Reading tar file system.xml
10:51:17:WU00:FS01:0x21:Digital signatures verified
10:51:17:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
10:51:17:WU00:FS01:0x21:Version 0.0.18
10:52:17:WU00:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
10:52:17:WU00:FS01:0x21:Saving result file logfile_01.txt
10:52:17:WU00:FS01:0x21:Saving result file log.txt
10:52:17:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:52:48:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:52:48:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9179 run:9 clone:7 gen:181 core:0x21 unit:0x00000133ab436c9f57bdce0465ed9bf5
10:52:48:WU00:FS01:Uploading 7.00KiB to 171.67.108.159
10:52:48:WU00:FS01:Connecting to 171.67.108.159:8080
10:52:48:WU00:FS01:Upload complete
10:52:48:WU00:FS01:Server responded WORK_ACK (400)
10:52:48:WU00:FS01:Cleaning up
10:52:48:WU00:FS01:Connecting to 171.67.108.45:80
10:52:49:WU00:FS01:Assigned to work server 140.163.4.242
10:52:49:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 140.163.4.242
10:52:49:WU00:FS01:Connecting to 140.163.4.242:8080
10:52:49:WU00:FS01:Downloading 4.22MiB
10:52:51:WU00:FS01:Download complete
10:52:51:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:11403 run:8 clone:19 gen:137 core:0x21 unit:0x000000e58ca304f255ed4f8434eb7e41
10:52:51:WU00:FS01:Starting
10:52:51:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
10:52:51:WU00:FS01:Started FahCore on PID 11141
10:52:51:WU00:FS01:Core PID:11145
10:52:51:WU00:FS01:FahCore 0x21 started
10:52:51:WU00:FS01:0x21:*********************** Log Started 2017-02-01T10:52:51Z ***********************
10:52:51:WU00:FS01:0x21:Project: 11403 (Run 8, Clone 19, Gen 137)
10:52:51:WU00:FS01:0x21:Unit: 0x000000e58ca304f255ed4f8434eb7e41
10:52:51:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
10:52:51:WU00:FS01:0x21:Machine: 1
10:52:51:WU00:FS01:0x21:Reading tar file core.xml
10:52:51:WU00:FS01:0x21:Reading tar file system.xml
10:52:51:WU00:FS01:0x21:Reading tar file integrator.xml
10:52:51:WU00:FS01:0x21:Reading tar file state.xml
10:52:52:WU00:FS01:0x21:Digital signatures verified
10:52:52:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
10:52:52:WU00:FS01:0x21:Version 0.0.18
10:53:54:WU00:FS01:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
10:53:54:WU00:FS01:0x21:Saving result file logfile_01.txt
10:53:54:WU00:FS01:0x21:Saving result file log.txt
10:53:54:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:55:25:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:55:25:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:11403 run:8 clone:19 gen:137 core:0x21 unit:0x000000e58ca304f255ed4f8434eb7e41
10:55:25:WU00:FS01:Uploading 2.53KiB to 140.163.4.242
10:55:25:WU00:FS01:Connecting to 140.163.4.242:8080
10:55:25:WU04:FS01:Connecting to 171.67.108.45:80
10:55:25:WU04:FS01:Assigned to work server 171.67.108.159
10:55:25:WU04:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 171.67.108.159
10:55:25:WU04:FS01:Connecting to 171.67.108.159:8080
10:55:26:WU04:FS01:Downloading 24.41MiB
10:55:26:WU00:FS01:Upload complete
10:55:26:WU00:FS01:Server responded WORK_ACK (400)
10:55:26:WU00:FS01:Cleaning up
10:55:29:WU04:FS01:Download complete
10:55:29:WU04:FS01:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:9179 run:28 clone:3 gen:112 core:0x21 unit:0x000000beab436c9f57bdce04c7ab3ff2
10:55:29:WU04:FS01:Starting
10:55:29:WU04:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 04 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
10:55:29:WU04:FS01:Started FahCore on PID 11297
10:55:29:WU04:FS01:Core PID:11301
10:55:29:WU04:FS01:FahCore 0x21 started
10:55:30:WU04:FS01:0x21:*********************** Log Started 2017-02-01T10:55:29Z ***********************
10:55:30:WU04:FS01:0x21:Project: 9179 (Run 28, Clone 3, Gen 112)
10:55:30:WU04:FS01:0x21:Unit: 0x000000beab436c9f57bdce04c7ab3ff2
10:55:30:WU04:FS01:0x21:CPU: 0x00000000000000000000000000000000
10:55:30:WU04:FS01:0x21:Machine: 1
10:55:30:WU04:FS01:0x21:Reading tar file core.xml
10:55:30:WU04:FS01:0x21:Reading tar file integrator.xml
10:55:30:WU04:FS01:0x21:Reading tar file state.xml
10:55:30:WU04:FS01:0x21:Reading tar file system.xml
10:55:30:WU04:FS01:0x21:Digital signatures verified
10:55:30:WU04:FS01:0x21:Folding@home GPU Core21 Folding@home Core
10:55:30:WU04:FS01:0x21:Version 0.0.18
10:56:58:WU04:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
10:56:58:WU04:FS01:0x21:Saving result file logfile_01.txt
10:56:58:WU04:FS01:0x21:Saving result file log.txt
10:56:58:WU04:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:57:28:WARNING:WU04:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:57:28:WU04:FS01:Sending unit results: id:04 state:SEND error:FAULTY project:9179 run:28 clone:3 gen:112 core:0x21 unit:0x000000beab436c9f57bdce04c7ab3ff2
10:57:28:WU04:FS01:Uploading 7.00KiB to 171.67.108.159
10:57:28:WU04:FS01:Connecting to 171.67.108.159:8080
10:57:28:WU04:FS01:Upload complete
10:57:29:WU04:FS01:Server responded WORK_ACK (400)
10:57:29:WU04:FS01:Cleaning up
10:57:29:WU00:FS01:Connecting to 171.67.108.45:80
10:57:29:WU00:FS01:Assigned to work server 171.67.108.105
10:57:29:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 171.67.108.105
10:57:29:WU00:FS01:Connecting to 171.67.108.105:8080
10:57:29:WU00:FS01:Downloading 20.47MiB
10:57:33:WU00:FS01:Download complete
10:57:33:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9178 run:5 clone:14 gen:42 core:0x21 unit:0x00000045ab436c6957b24c29a6157cc5
10:57:33:WU00:FS01:Starting
10:57:33:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
10:57:33:WU00:FS01:Started FahCore on PID 11455
10:57:33:WU00:FS01:Core PID:11459
10:57:33:WU00:FS01:FahCore 0x21 started
10:57:33:WU00:FS01:0x21:*********************** Log Started 2017-02-01T10:57:33Z ***********************
10:57:33:WU00:FS01:0x21:Project: 9178 (Run 5, Clone 14, Gen 42)
10:57:33:WU00:FS01:0x21:Unit: 0x00000045ab436c6957b24c29a6157cc5
10:57:33:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
10:57:33:WU00:FS01:0x21:Machine: 1
10:57:33:WU00:FS01:0x21:Reading tar file core.xml
10:57:33:WU00:FS01:0x21:Reading tar file integrator.xml
10:57:33:WU00:FS01:0x21:Reading tar file state.xml
10:57:33:WU00:FS01:0x21:Reading tar file system.xml
10:57:33:WU00:FS01:0x21:Digital signatures verified
10:57:33:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
10:57:33:WU00:FS01:0x21:Version 0.0.18
10:58:34:WU00:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
10:58:34:WU00:FS01:0x21:Saving result file logfile_01.txt
10:58:34:WU00:FS01:0x21:Saving result file log.txt
10:58:34:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:59:05:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:59:05:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9178 run:5 clone:14 gen:42 core:0x21 unit:0x00000045ab436c6957b24c29a6157cc5
10:59:05:WU00:FS01:Uploading 7.00KiB to 171.67.108.105
10:59:05:WU00:FS01:Connecting to 171.67.108.105:8080
10:59:05:WU00:FS01:Upload complete
10:59:05:WU00:FS01:Server responded WORK_ACK (400)
10:59:05:WU00:FS01:Cleaning up
10:59:05:WU01:FS01:Connecting to 171.67.108.45:80
10:59:05:WU01:FS01:Assigned to work server 140.163.4.245
10:59:05:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 140.163.4.245
10:59:05:WU01:FS01:Connecting to 140.163.4.245:8080
10:59:06:WU01:FS01:Downloading 20.67MiB
10:59:12:WU01:FS01:Download 62.28%
10:59:12:WU01:FS01:Download complete
10:59:13:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:10494 run:11 clone:37 gen:169 core:0x21 unit:0x000001178ca304f555de949a6a22b7bc
10:59:13:WU01:FS01:Starting
10:59:13:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
10:59:13:WU01:FS01:Started FahCore on PID 11563
10:59:13:WU01:FS01:Core PID:11567
10:59:13:WU01:FS01:FahCore 0x21 started
10:59:13:WU01:FS01:0x21:*********************** Log Started 2017-02-01T10:59:13Z ***********************
10:59:13:WU01:FS01:0x21:Project: 10494 (Run 11, Clone 37, Gen 169)
10:59:13:WU01:FS01:0x21:Unit: 0x000001178ca304f555de949a6a22b7bc
10:59:13:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
10:59:13:WU01:FS01:0x21:Machine: 1
10:59:13:WU01:FS01:0x21:Reading tar file core.xml
10:59:13:WU01:FS01:0x21:Reading tar file system.xml
10:59:14:WU01:FS01:0x21:Reading tar file integrator.xml
10:59:14:WU01:FS01:0x21:Reading tar file state.xml
10:59:17:WU01:FS01:0x21:Digital signatures verified
10:59:17:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
10:59:17:WU01:FS01:0x21:Version 0.0.18
11:00:49:WU01:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
11:00:49:WU01:FS01:0x21:Saving result file logfile_01.txt
11:00:49:WU01:FS01:0x21:Saving result file log.txt
11:00:49:WU01:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
11:01:19:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
11:01:19:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:10494 run:11 clone:37 gen:169 core:0x21 unit:0x000001178ca304f555de949a6a22b7bc
11:01:19:WU01:FS01:Uploading 2.46KiB to 140.163.4.245
11:01:19:WU01:FS01:Connecting to 140.163.4.245:8080
11:01:20:WU01:FS01:Upload complete
11:01:20:WU01:FS01:Server responded WORK_ACK (400)
11:01:20:WU01:FS01:Cleaning up
11:01:20:WU00:FS01:Connecting to 171.67.108.45:80
11:01:20:WU00:FS01:Assigned to work server 171.67.108.157
11:01:20:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 171.67.108.157
11:01:20:WU00:FS01:Connecting to 171.67.108.157:8080
11:01:20:WU00:FS01:Downloading 5.18MiB
11:01:21:WU00:FS01:Download complete
11:01:21:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9414 run:92 clone:0 gen:4 core:0x21 unit:0x00000005ab436c9d585e0690fc10bb9b
11:01:21:WU00:FS01:Starting
11:01:21:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
11:01:21:WU00:FS01:Started FahCore on PID 11705
11:01:21:WU00:FS01:Core PID:11709
11:01:21:WU00:FS01:FahCore 0x21 started
11:01:22:WU00:FS01:0x21:*********************** Log Started 2017-02-01T11:01:21Z ***********************
11:01:22:WU00:FS01:0x21:Project: 9414 (Run 92, Clone 0, Gen 4)
11:01:22:WU00:FS01:0x21:Unit: 0x00000005ab436c9d585e0690fc10bb9b
11:01:22:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:01:22:WU00:FS01:0x21:Machine: 1
11:01:22:WU00:FS01:0x21:Reading tar file core.xml
11:01:22:WU00:FS01:0x21:Reading tar file integrator.xml
11:01:22:WU00:FS01:0x21:Reading tar file state.xml
11:01:22:WU00:FS01:0x21:Reading tar file system.xml
11:01:22:WU00:FS01:0x21:Digital signatures verified
11:01:22:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:01:22:WU00:FS01:0x21:Version 0.0.18
11:02:52:WU00:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
11:02:52:WU00:FS01:0x21:Saving result file logfile_01.txt
11:02:52:WU00:FS01:0x21:Saving result file log.txt
11:02:52:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
11:03:22:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
11:03:22:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9414 run:92 clone:0 gen:4 core:0x21 unit:0x00000005ab436c9d585e0690fc10bb9b
11:03:22:WU00:FS01:Uploading 7.00KiB to 171.67.108.157
11:03:22:WU00:FS01:Connecting to 171.67.108.157:8080
11:03:22:WU00:FS01:Upload complete
11:03:22:WU00:FS01:Server responded WORK_ACK (400)
11:03:22:WU00:FS01:Cleaning up
11:03:23:WU00:FS01:Connecting to 171.67.108.45:80
11:03:23:WU00:FS01:Assigned to work server 140.163.4.245
11:03:23:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 140.163.4.245
11:03:23:WU00:FS01:Connecting to 140.163.4.245:8080
11:03:23:WU00:FS01:Downloading 14.50MiB
11:03:29:WU00:FS01:Download 28.02%
11:03:31:WU00:FS01:Download complete
11:03:31:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10496 run:181 clone:3 gen:5 core:0x21 unit:0x000000088ca304f556bbb2dbb637e8e1
11:03:31:WU00:FS01:Starting
11:03:31:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
11:03:31:WU00:FS01:Started FahCore on PID 11762
11:03:31:WU00:FS01:Core PID:11766
11:03:31:WU00:FS01:FahCore 0x21 started
11:03:32:WU00:FS01:0x21:*********************** Log Started 2017-02-01T11:03:32Z ***********************
11:03:32:WU00:FS01:0x21:Project: 10496 (Run 181, Clone 3, Gen 5)
11:03:32:WU00:FS01:0x21:Unit: 0x000000088ca304f556bbb2dbb637e8e1
11:03:32:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:03:32:WU00:FS01:0x21:Machine: 1
11:03:32:WU00:FS01:0x21:Reading tar file core.xml
11:03:32:WU00:FS01:0x21:Reading tar file system.xml
11:03:33:WU00:FS01:0x21:Reading tar file integrator.xml
11:03:33:WU00:FS01:0x21:Reading tar file state.xml
11:03:36:WU00:FS01:0x21:Digital signatures verified
11:03:36:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:03:36:WU00:FS01:0x21:Version 0.0.18
11:04:44:WU00:FS01:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
11:04:44:WU00:FS01:0x21:Saving result file logfile_01.txt
11:04:44:WU00:FS01:0x21:Saving result file log.txt
11:04:44:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
11:06:14:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
11:06:14:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:10496 run:181 clone:3 gen:5 core:0x21 unit:0x000000088ca304f556bbb2dbb637e8e1
11:06:14:WU00:FS01:Uploading 2.51KiB to 140.163.4.245
11:06:14:WU00:FS01:Connecting to 140.163.4.245:8080
11:06:14:WU00:FS01:Upload complete
11:06:14:WU00:FS01:Server responded WORK_ACK (400)
11:06:14:WU00:FS01:Cleaning up
11:06:14:WU01:FS01:Connecting to 171.67.108.45:80
11:06:15:WU01:FS01:Assigned to work server 171.67.108.105
11:06:15:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 171.67.108.105
11:06:15:WU01:FS01:Connecting to 171.67.108.105:8080
11:06:15:WU01:FS01:Downloading 22.62MiB
11:06:18:WU01:FS01:Download complete
11:06:18:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9178 run:1 clone:5 gen:283 core:0x21 unit:0x00000183ab436c6957b24c29402f927c
11:06:18:WU01:FS01:Starting
11:06:18:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
11:06:18:WU01:FS01:Started FahCore on PID 11837
11:06:18:WU01:FS01:Core PID:11841
11:06:18:WU01:FS01:FahCore 0x21 started
11:06:18:WU01:FS01:0x21:*********************** Log Started 2017-02-01T11:06:18Z ***********************
11:06:18:WU01:FS01:0x21:Project: 9178 (Run 1, Clone 5, Gen 283)
11:06:18:WU01:FS01:0x21:Unit: 0x00000183ab436c6957b24c29402f927c
11:06:18:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:06:18:WU01:FS01:0x21:Machine: 1
11:06:18:WU01:FS01:0x21:Reading tar file core.xml
11:06:18:WU01:FS01:0x21:Reading tar file integrator.xml
11:06:18:WU01:FS01:0x21:Reading tar file state.xml
11:06:18:WU01:FS01:0x21:Reading tar file system.xml
11:06:18:WU01:FS01:0x21:Digital signatures verified
11:06:18:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:06:18:WU01:FS01:0x21:Version 0.0.18
11:06:54:WU01:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
11:06:54:WU01:FS01:0x21:Saving result file logfile_01.txt
11:06:54:WU01:FS01:0x21:Saving result file log.txt
11:06:54:WU01:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
11:07:25:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
11:07:25:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:9178 run:1 clone:5 gen:283 core:0x21 unit:0x00000183ab436c6957b24c29402f927c
11:07:25:WU01:FS01:Uploading 7.00KiB to 171.67.108.105
11:07:25:WU01:FS01:Connecting to 171.67.108.105:8080
11:07:25:WU01:FS01:Upload complete
11:07:25:WU01:FS01:Server responded WORK_ACK (400)
11:07:25:WU01:FS01:Cleaning up
11:07:25:WU00:FS01:Connecting to 171.67.108.45:80
11:07:25:WU00:FS01:Assigned to work server 171.64.65.84
11:07:25:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] from 171.64.65.84
11:07:25:WU00:FS01:Connecting to 171.64.65.84:8080
11:07:25:WU00:FS01:Downloading 2.58MiB
11:07:26:WU00:FS01:Download complete
11:07:26:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9188 run:0 clone:84 gen:369 core:0x21 unit:0x00000207ab40415457cb2b42b6867b48
11:07:26:WU00:FS01:Starting
11:07:26:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 0 -gpu-vendor nvidia
11:07:26:WU00:FS01:Started FahCore on PID 11865
11:07:26:WU00:FS01:Core PID:11869
11:07:26:WU00:FS01:FahCore 0x21 started
11:07:27:WU00:FS01:0x21:*********************** Log Started 2017-02-01T11:07:26Z ***********************
11:07:27:WU00:FS01:0x21:Project: 9188 (Run 0, Clone 84, Gen 369)
11:07:27:WU00:FS01:0x21:Unit: 0x00000207ab40415457cb2b42b6867b48
11:07:27:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
11:07:27:WU00:FS01:0x21:Machine: 1
11:07:27:WU00:FS01:0x21:Reading tar file core.xml
11:07:27:WU00:FS01:0x21:Reading tar file system.xml
11:07:27:WU00:FS01:0x21:Reading tar file integrator.xml
11:07:27:WU00:FS01:0x21:Reading tar file state.xml
11:07:27:WU00:FS01:0x21:Digital signatures verified
11:07:27:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
11:07:27:WU00:FS01:0x21:Version 0.0.18
11:08:29:WU00:FS01:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
11:08:29:WU00:FS01:0x21:Saving result file logfile_01.txt
11:08:29:WU00:FS01:0x21:Saving result file log.txt
11:08:29:WU00:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
11:08:59:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
11:08:59:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:9188 run:0 clone:84 gen:369 core:0x21 unit:0x00000207ab40415457cb2b42b6867b48
11:08:59:WU00:FS01:Uploading 2.45KiB to 171.64.65.84
11:08:59:WU00:FS01:Connecting to 171.64.65.84:8080
11:08:59:WU00:FS01:Upload complete
11:08:59:WU00:FS01:Server responded WORK_ACK (400)
11:08:59:WU00:FS01:Cleaning up
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-03 *******************************
******************************* Date: 2017-02-03 *******************************
10:29:59:FS01:Finishing

02

******************************* Date: 2017-02-01 *******************************
06:05:24:WU03:FS02:0x21:Completed 2050000 out of 5000000 steps (41%)
06:07:44:WU03:FS02:0x21:Completed 2100000 out of 5000000 steps (42%)
06:10:06:WU03:FS02:0x21:Completed 2150000 out of 5000000 steps (43%)
06:12:26:WU03:FS02:0x21:Completed 2200000 out of 5000000 steps (44%)
06:14:46:WU03:FS02:0x21:Completed 2250000 out of 5000000 steps (45%)
06:17:08:WU03:FS02:0x21:Completed 2300000 out of 5000000 steps (46%)
06:19:27:WU03:FS02:0x21:Completed 2350000 out of 5000000 steps (47%)
06:21:49:WU03:FS02:0x21:Completed 2400000 out of 5000000 steps (48%)
06:24:09:WU03:FS02:0x21:Completed 2450000 out of 5000000 steps (49%)
06:26:28:WU03:FS02:0x21:Completed 2500000 out of 5000000 steps (50%)
06:28:50:WU03:FS02:0x21:Completed 2550000 out of 5000000 steps (51%)
06:31:10:WU03:FS02:0x21:Completed 2600000 out of 5000000 steps (52%)
06:33:32:WU03:FS02:0x21:Completed 2650000 out of 5000000 steps (53%)
06:37:21:WU03:FS02:0x21:Completed 2700000 out of 5000000 steps (54%)
06:39:40:WU03:FS02:0x21:Completed 2750000 out of 5000000 steps (55%)
06:42:03:WU03:FS02:0x21:Completed 2800000 out of 5000000 steps (56%)
06:44:23:WU03:FS02:0x21:Completed 2850000 out of 5000000 steps (57%)
06:46:46:WU03:FS02:0x21:Completed 2900000 out of 5000000 steps (58%)
06:49:05:WU03:FS02:0x21:Completed 2950000 out of 5000000 steps (59%)
06:51:25:WU03:FS02:0x21:Completed 3000000 out of 5000000 steps (60%)
06:53:47:WU03:FS02:0x21:Completed 3050000 out of 5000000 steps (61%)
06:56:06:WU03:FS02:0x21:Completed 3100000 out of 5000000 steps (62%)
06:58:28:WU03:FS02:0x21:Completed 3150000 out of 5000000 steps (63%)
07:00:48:WU03:FS02:0x21:Completed 3200000 out of 5000000 steps (64%)
07:03:07:WU03:FS02:0x21:Completed 3250000 out of 5000000 steps (65%)
07:05:29:WU03:FS02:0x21:Completed 3300000 out of 5000000 steps (66%)
07:07:49:WU03:FS02:0x21:Completed 3350000 out of 5000000 steps (67%)
07:10:11:WU03:FS02:0x21:Completed 3400000 out of 5000000 steps (68%)
07:12:31:WU03:FS02:0x21:Completed 3450000 out of 5000000 steps (69%)
07:14:50:WU03:FS02:0x21:Completed 3500000 out of 5000000 steps (70%)
07:17:13:WU03:FS02:0x21:Completed 3550000 out of 5000000 steps (71%)
07:19:33:WU03:FS02:0x21:Completed 3600000 out of 5000000 steps (72%)
07:21:55:WU03:FS02:0x21:Completed 3650000 out of 5000000 steps (73%)
07:24:15:WU03:FS02:0x21:Completed 3700000 out of 5000000 steps (74%)
07:26:34:WU03:FS02:0x21:Completed 3750000 out of 5000000 steps (75%)
07:28:57:WU03:FS02:0x21:Completed 3800000 out of 5000000 steps (76%)
07:31:17:WU03:FS02:0x21:Completed 3850000 out of 5000000 steps (77%)
07:33:39:WU03:FS02:0x21:Completed 3900000 out of 5000000 steps (78%)
07:35:59:WU03:FS02:0x21:Completed 3950000 out of 5000000 steps (79%)
07:38:18:WU03:FS02:0x21:Completed 4000000 out of 5000000 steps (80%)
07:40:40:WU03:FS02:0x21:Completed 4050000 out of 5000000 steps (81%)
07:43:00:WU03:FS02:0x21:Completed 4100000 out of 5000000 steps (82%)
07:45:22:WU03:FS02:0x21:Completed 4150000 out of 5000000 steps (83%)
07:47:42:WU03:FS02:0x21:Completed 4200000 out of 5000000 steps (84%)
07:50:02:WU03:FS02:0x21:Completed 4250000 out of 5000000 steps (85%)
07:52:24:WU03:FS02:0x21:Completed 4300000 out of 5000000 steps (86%)
07:54:43:WU03:FS02:0x21:Completed 4350000 out of 5000000 steps (87%)
07:57:05:WU03:FS02:0x21:Completed 4400000 out of 5000000 steps (88%)
07:59:25:WU03:FS02:0x21:Completed 4450000 out of 5000000 steps (89%)
08:01:44:WU03:FS02:0x21:Completed 4500000 out of 5000000 steps (90%)
08:04:07:WU03:FS02:0x21:Completed 4550000 out of 5000000 steps (91%)
08:06:26:WU03:FS02:0x21:Completed 4600000 out of 5000000 steps (92%)
08:08:48:WU03:FS02:0x21:Completed 4650000 out of 5000000 steps (93%)
08:11:08:WU03:FS02:0x21:Completed 4700000 out of 5000000 steps (94%)
08:13:28:WU03:FS02:0x21:Completed 4750000 out of 5000000 steps (95%)
08:15:50:WU03:FS02:0x21:Completed 4800000 out of 5000000 steps (96%)
08:18:09:WU03:FS02:0x21:Completed 4850000 out of 5000000 steps (97%)
08:20:31:WU03:FS02:0x21:Completed 4900000 out of 5000000 steps (98%)
08:22:51:WU03:FS02:0x21:Completed 4950000 out of 5000000 steps (99%)
08:22:51:WU00:FS02:Connecting to 171.67.108.45:80
08:22:51:WU00:FS02:Assigned to work server 171.67.108.157
08:22:51:WU00:FS02:Requesting new work unit for slot 02: RUNNING gpu:4:GP104 [GeForce GTX 1080] from 171.67.108.157
08:22:51:WU00:FS02:Connecting to 171.67.108.157:8080
08:22:51:WU00:FS02:Downloading 5.17MiB
08:22:53:WU00:FS02:Download complete
08:22:53:WU00:FS02:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9414 run:75 clone:0 gen:4 core:0x21 unit:0x00000005ab436c9d585e06905a27280f
08:25:10:WU03:FS02:0x21:Completed 5000000 out of 5000000 steps (100%)
08:25:13:WU03:FS02:0x21:Saving result file logfile_01.txt
08:25:13:WU03:FS02:0x21:Saving result file checkpointState.xml
08:25:16:WU03:FS02:0x21:Saving result file checkpt.crc
08:25:16:WU03:FS02:0x21:Saving result file log.txt
08:25:16:WU03:FS02:0x21:Saving result file positions.xtc
08:25:18:WU03:FS02:0x21:Folding@home Core Shutdown: FINISHED_UNIT
08:25:18:WU03:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
08:25:18:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:11406 run:1 clone:46 gen:392 core:0x21 unit:0x0000020f8ca304f25686b1b1930ef57e
08:25:18:WU03:FS02:Uploading 14.01MiB to 140.163.4.242
08:25:18:WU03:FS02:Connecting to 140.163.4.242:8080
08:25:19:WU00:FS02:Starting
08:25:19:WU00:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 2 -gpu-vendor nvidia
08:25:19:WU00:FS02:Started FahCore on PID 10729
08:25:19:WU00:FS02:Core PID:10733
08:25:19:WU00:FS02:FahCore 0x21 started
08:25:19:WARNING:WU00:FS02:FahCore returned: CORE_OUTDATED (110 = 0x6e)
08:25:24:WU03:FS02:Upload 30.33%
08:25:30:WU03:FS02:Upload 65.13%
08:25:36:WU03:FS02:Upload 91.45%
08:25:47:WU03:FS02:Upload complete
08:25:47:WU03:FS02:Server responded WORK_ACK (400)
08:25:47:WU03:FS02:Final credit estimate, 153128.00 points
08:25:47:WU03:FS02:Cleaning up
10:50:05:WU00:FS02:Downloading core from http://fahwebx.stanford.edu/cores/Linux ... ore_21.fah
10:50:05:WU00:FS02:Connecting to fahwebx.stanford.edu:80
10:50:05:WU00:FS02:FahCore 21: Downloading 3.23MiB
10:50:06:WU00:FS02:FahCore 21: Download complete
10:50:06:WU00:FS02:Valid core signature
10:50:06:WU00:FS02:Unpacked 7.94MiB to cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21
10:50:06:WU00:FS02:Starting
10:50:06:WU00:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 2 -gpu-vendor nvidia
10:50:06:WU00:FS02:Started FahCore on PID 10914
10:50:06:WU00:FS02:Core PID:10918
10:50:06:WU00:FS02:FahCore 0x21 started
10:50:07:WU00:FS02:0x21:*********************** Log Started 2017-02-01T10:50:06Z ***********************
10:50:07:WU00:FS02:0x21:Project: 9414 (Run 75, Clone 0, Gen 4)
10:50:07:WU00:FS02:0x21:Unit: 0x00000005ab436c9d585e06905a27280f
10:50:07:WU00:FS02:0x21:CPU: 0x00000000000000000000000000000000
10:50:07:WU00:FS02:0x21:Machine: 2
10:50:07:WU00:FS02:0x21:Reading tar file core.xml
10:50:07:WU00:FS02:0x21:Reading tar file integrator.xml
10:50:07:WU00:FS02:0x21:Reading tar file state.xml
10:50:07:WU00:FS02:0x21:Reading tar file system.xml
10:50:07:WU00:FS02:0x21:Digital signatures verified
10:50:07:WU00:FS02:0x21:Folding@home GPU Core21 Folding@home Core
10:50:07:WU00:FS02:0x21:Version 0.0.18
10:50:09:WU00:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
10:50:09:WU00:FS02:0x21:Saving result file logfile_01.txt
10:50:09:WU00:FS02:0x21:Saving result file log.txt
10:50:09:WU00:FS02:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:50:10:WARNING:WU00:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:50:10:WU00:FS02:Sending unit results: id:00 state:SEND error:FAULTY project:9414 run:75 clone:0 gen:4 core:0x21 unit:0x00000005ab436c9d585e06905a27280f
10:50:10:WU00:FS02:Uploading 7.00KiB to 171.67.108.157
10:50:10:WU00:FS02:Connecting to 171.67.108.157:8080
10:50:10:WU00:FS02:Upload complete
10:50:10:WU05:FS02:Connecting to 171.67.108.45:80
10:50:11:WU00:FS02:Server responded WORK_ACK (400)
10:50:11:WU00:FS02:Cleaning up
10:50:11:WU05:FS02:Assigned to work server 171.67.108.105
10:50:11:WU05:FS02:Requesting new work unit for slot 02: READY gpu:4:GP104 [GeForce GTX 1080] from 171.67.108.105
10:50:11:WU05:FS02:Connecting to 171.67.108.105:8080
10:50:12:WU05:FS02:Downloading 18.92MiB
10:50:18:WU05:FS02:Download 24.11%
10:50:19:WU05:FS02:Download complete
10:50:19:WU05:FS02:Received Unit: id:05 state:DOWNLOAD error:NO_ERROR project:9176 run:19 clone:0 gen:181 core:0x21 unit:0x00000115ab436c6957b24c285565b4aa
10:50:19:WU05:FS02:Starting
10:50:19:WU05:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 05 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 2 -gpu-vendor nvidia
10:50:19:WU05:FS02:Started FahCore on PID 10957
10:50:19:WU05:FS02:Core PID:10961
10:50:19:WU05:FS02:FahCore 0x21 started
10:50:20:WU05:FS02:0x21:*********************** Log Started 2017-02-01T10:50:19Z ***********************
10:50:20:WU05:FS02:0x21:Project: 9176 (Run 19, Clone 0, Gen 181)
10:50:20:WU05:FS02:0x21:Unit: 0x00000115ab436c6957b24c285565b4aa
10:50:20:WU05:FS02:0x21:CPU: 0x00000000000000000000000000000000
10:50:20:WU05:FS02:0x21:Machine: 2
10:50:20:WU05:FS02:0x21:Reading tar file core.xml
10:50:20:WU05:FS02:0x21:Reading tar file integrator.xml
10:50:20:WU05:FS02:0x21:Reading tar file state.xml
10:50:20:WU05:FS02:0x21:Reading tar file system.xml
10:50:20:WU05:FS02:0x21:Digital signatures verified
10:50:20:WU05:FS02:0x21:Folding@home GPU Core21 Folding@home Core
10:50:20:WU05:FS02:0x21:Version 0.0.18
10:51:12:WU05:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
10:51:12:WU05:FS02:0x21:Saving result file logfile_01.txt
10:51:12:WU05:FS02:0x21:Saving result file log.txt
10:51:12:WU05:FS02:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:51:12:WARNING:WU05:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:51:12:WU05:FS02:Sending unit results: id:05 state:SEND error:FAULTY project:9176 run:19 clone:0 gen:181 core:0x21 unit:0x00000115ab436c6957b24c285565b4aa
10:51:12:WU05:FS02:Uploading 7.00KiB to 171.67.108.105
10:51:12:WU05:FS02:Connecting to 171.67.108.105:8080
10:51:12:WU05:FS02:Upload complete
10:51:12:WU05:FS02:Server responded WORK_ACK (400)
10:51:12:WU05:FS02:Cleaning up
10:51:12:WU01:FS02:Connecting to 171.67.108.45:80
10:51:12:WU01:FS02:Assigned to work server 140.163.4.231
10:51:12:WU01:FS02:Requesting new work unit for slot 02: READY gpu:4:GP104 [GeForce GTX 1080] from 140.163.4.231
10:51:12:WU01:FS02:Connecting to 140.163.4.231:8080
10:51:13:WU01:FS02:Downloading 16.73MiB
10:51:18:WU01:FS02:Download complete
10:51:19:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:11710 run:0 clone:241 gen:47 core:0x21 unit:0x000000558ca304e75814df306a984a54
10:51:19:WU01:FS02:Starting
10:51:19:WU01:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 01 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 2 -gpu-vendor nvidia
10:51:19:WU01:FS02:Started FahCore on PID 11038
10:51:19:WU01:FS02:Core PID:11042
10:51:19:WU01:FS02:FahCore 0x21 started
10:51:19:WU01:FS02:0x21:*********************** Log Started 2017-02-01T10:51:19Z ***********************
10:51:19:WU01:FS02:0x21:Project: 11710 (Run 0, Clone 241, Gen 47)
10:51:19:WU01:FS02:0x21:Unit: 0x000000558ca304e75814df306a984a54
10:51:19:WU01:FS02:0x21:CPU: 0x00000000000000000000000000000000
10:51:19:WU01:FS02:0x21:Machine: 2
10:51:19:WU01:FS02:0x21:Reading tar file core.xml
10:51:19:WU01:FS02:0x21:Reading tar file integrator.xml
10:51:19:WU01:FS02:0x21:Reading tar file state.xml
10:51:19:WU01:FS02:0x21:Reading tar file system.xml
10:51:19:WU01:FS02:0x21:Digital signatures verified
10:51:19:WU01:FS02:0x21:Folding@home GPU Core21 Folding@home Core
10:51:19:WU01:FS02:0x21:Version 0.0.18
10:51:45:WU01:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 1 0
10:51:45:WU01:FS02:0x21:Saving result file logfile_01.txt
10:51:45:WU01:FS02:0x21:Saving result file log.txt
10:51:45:WU01:FS02:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:51:45:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:51:45:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:11710 run:0 clone:241 gen:47 core:0x21 unit:0x000000558ca304e75814df306a984a54
10:51:45:WU01:FS02:Uploading 7.00KiB to 140.163.4.231
10:51:45:WU01:FS02:Connecting to 140.163.4.231:8080
10:51:46:WU03:FS02:Connecting to 171.67.108.45:80
10:51:46:WU01:FS02:Upload complete
10:51:46:WU01:FS02:Server responded WORK_ACK (400)
10:51:46:WU01:FS02:Cleaning up
10:51:46:WU03:FS02:Assigned to work server 140.163.4.243
10:51:46:WU03:FS02:Requesting new work unit for slot 02: READY gpu:4:GP104 [GeForce GTX 1080] from 140.163.4.243
10:51:46:WU03:FS02:Connecting to 140.163.4.243:8080
10:51:46:WU03:FS02:Downloading 2.67MiB
10:51:49:WU03:FS02:Download complete
10:51:49:WU03:FS02:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:11707 run:86 clone:2 gen:37 core:0x21 unit:0x0000002c8ca304f358702f82c202d580
10:51:49:WU03:FS02:Starting
10:51:49:WU03:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/fahwebx.stanford.edu/cores/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 03 -suffix 01 -version 704 -lifeline 1399 -checkpoint 30 -gpu 2 -gpu-vendor nvidia
10:51:49:WU03:FS02:Started FahCore on PID 11075
10:51:51:WU03:FS02:Core PID:11079
10:51:51:WU03:FS02:FahCore 0x21 started
10:51:52:WU03:FS02:0x21:*********************** Log Started 2017-02-01T10:51:51Z ***********************
10:51:52:WU03:FS02:0x21:Project: 11707 (Run 86, Clone 2, Gen 37)
10:51:52:WU03:FS02:0x21:Unit: 0x0000002c8ca304f358702f82c202d580
10:51:52:WU03:FS02:0x21:CPU: 0x00000000000000000000000000000000
10:51:52:WU03:FS02:0x21:Machine: 2
10:51:52:WU03:FS02:0x21:Reading tar file core.xml
10:51:52:WU03:FS02:0x21:Reading tar file system.xml
10:51:52:WU03:FS02:0x21:Reading tar file integrator.xml
10:51:52:WU03:FS02:0x21:Reading tar file state.xml
10:51:52:WU03:FS02:0x21:Digital signatures verified
10:51:52:WU03:FS02:0x21:Folding@home GPU Core21 Folding@home Core
10:51:52:WU03:FS02:0x21:Version 0.0.18
10:52:48:WU03:FS02:0x21:ERROR:Discrepancy: Forces are blowing up! 0 0
10:52:48:WU03:FS02:0x21:Saving result file logfile_01.txt
10:52:48:WU03:FS02:0x21:Saving result file log.txt
10:52:48:WU03:FS02:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
10:52:48:WARNING:WU03:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
11:01:20:WU03:FS02:0x21:ERROR:exception: Error downloading array interactionCount: clEnqueueReadBuffer (-5)
11:01:20:WU03:FS02:0x21:Saving result file logfile_01.txt
11:01:20:WU03:FS02:0x21:Saving result file log.txt
11:01:20:WU03:FS02:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
11:02:20:WARNING:WU03:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
11:02:20:WU03:FS02:Sending unit results: id:03 state:SEND error:FAULTY project:11406 run:5 clone:15 gen:266 core:0x21 unit:0x000001778ca304f25686b1e310b1a109
11:02:20:WU03:FS02:Uploading 2.54KiB to 140.163.4.242
11:02:20:WU03:FS02:Connecting to 140.163.4.242:8080
11:02:20:WU03:FS02:Upload complete
11:02:20:WU03:FS02:Server responded WORK_ACK (400)
11:02:20:WU03:FS02:Cleaning up
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-01 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-02 *******************************
******************************* Date: 2017-02-03 *******************************
******************************* Date: 2017-02-03 *******************************
10:29:59:FS02:Finishing

As you can see they didn't ALL start failing at the same time on 02/01/17. All my machines look similar, with minor details, but gather all that information, from machine to machine would take me hours. I did see something in one of them that said something about a core being out of date. Please let me know if you have any suggestions. I looked, and my driver for all is the newest I know of or can find, which is the 367.27.

There is a very long Topic: WARNING Do not upgrade to 375/376.xx drivers (for xx<48) which mostly involved Windows. It turns out that nVidia changed something in their compiler which broke FahCore21. FAH and nV have been working on the issue and both have issued revisions to their software. These changes have resulted in a lot of improvement for Windows Users but it's not certain that it's fully resolved for them.

Unfortunately the nVidia hot-fix driver has not been released for Linux and the FahCore_21 update to V0.0.18 does not completely resolve the problem for you. I'll bump this problem up to Development.

Unfortunately at this moment, all I can do is Thank You for the report while we wait for a better resolution for you.

So I'm just stuck at the moment, eh? I allowed all my machines to finish the cpu jobs and shut them down. I tried uninstall and reinstall by means of the terminal. I thought maybe it was some errors I was making during installation, but this was perfect for the first GPU, and I immediately got a job that started with the run, then not, then run and not again. Those always fail eventually. I suppose I'll save some money off my electricity bill for a while. Thanks Bruce, I really appreciate your help in this. RAB

I also use cinnamon 17.3 on one of my machines. 18 on the other one. Nvidia driver 370.28 on both. Running a mixture of 5 1080s, a 960 and a 750ti. Not having any of the problems you are having. Are your systems otherwise patched up to date?

The problem I described above may not be the only error that issues that same message. There's reason to suspect that overclocked GPUs MIGHT also issue the same message under specific situations. I can't promise that reducing the clock rate will help, but it's worth a try.

Both nV and FAH are trying to increase GPU performance, and I suppose a GPU that was previously stable might no longer be.

If you do test it, please report back.

As our team (#70335) started a 'folding' week because of world cancer day recently, my computer is only used for folding 24/7 and a little bit for browsing.
Last 11 WUs were fine w/o any errors or Bad States.

When this error happens then it seems that the GPU driver is reset. That's right.
But a reboot of the system is not necessary. FAH-client aborts this WU and then directly starts over by downloading the next WU and then processing it.

When this error appears, why doesn't the client jump back to the last correct checkpoint and then tries to continue to process the WU instead of aborting it?
Similar to the error with overclocking ('Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?').

This would keep frustration rate low. I don't mind when a WU is aborted after a few minutes. But after hours of processing it's really annoying. Waste of money.
Donors should have the chance to finish the WU

Edit:
Reset of driver takes 30sec max.

foldinghomealone wrote:When this error happens then it seems that the GPU driver is reset. That's right.
But a reboot of the system is not necessary. FAH-client aborts this WU and then directly starts over by downloading the next WU and then processing it.

Yes, but check the GPU clocks after the reset (with GPUZ), sometimes the GPU gets stuck in conservative clocks.

When this error appears, why doesn't the client jump back to the last correct checkpoint and then tries to continue to process the WU instead of aborting it?
Similar to the error with overclocking ('Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?').

This error is different : when the GPU is getting reset, the core doesn't see it anymore (this basically what the CL_OUT_OF_RESOURCES error means) for a short period. That why the client treat it as a serious error and dumps the WU.

The Bad State error is more soft : the software detects an inconsistency between the calculation done on the GPU and the reference calculation done periodically on the CPU. This error is recoverable because it assumes that's the computation error can be random, so it retries from last checkpoint. If the error is real (simulation leading to unstable state), you'll see 3 Bad states at the same point, and then the WU will be dumped as a bad one.

Be careful, because in the history of the failure of my 980 Ti, I first started to get random Bad States. Then I started to see some GPU resets with clEnqueueReadBuffer (-5) error. Then, I started to find my system being turned off automatically. And one day, after 9 months of operations, after powering it back on, the card blew up (VRM burned) when Windows activated it. The short circuit in the VRMs prevented the machine from even powering up.

Thanks for bringing this to our attention, bruce!

The NVIDIA hotfix issue is not relevant here---that was a thread synchronization issue that caused the core to refused to even start running on machines with the broken drivers. 0.0.18 works around that issue.

It looks like what is happening is that RABishop is running into very frequent NaN forces or energies, which is a different issue.

I see you're using GTX-1080s on Linux, RABishop. Are you still on 367.27? From what I understand, that driver (released 2016.6.13) was the first in which GTX-1080 support was introduced for Linux. I believe that we've seen this NaN issue elsewhere with GTX-1080s and OpenMM with early NVIDIA drivers, and I *believe* that more recent drivers clear that up.

The most recent driver series for the GTX-1080 on Linux is 375.26 (released 2016.12.14): http://www.nvidia.com/download/driverRe ... 2992/en-us

I'd suggest you try updating the NVIDIA driver and see if that clears up the issue. If not, we can take a really close look at those projects and try to reproduce the issue on GTX-1080s---it's possible (though unlikely) that something pathological is going on and you just got really unlucky here.

Thanks again for bearing with us, and apologies for the trouble you're having!

~ The F@h Core21 Team

Ah. Interesting. I had tried looking to see if there were any new driver available for the 10 series cards, but googling just kept leading me back to links for 367.27. I will, of course, try this new driver on this system, and get back about how well (or not) it works. It's been a while since I was in that putty screen, but that won't be a problem. Many Thanks. : )

toTOW wrote:Be careful, because in the history of the failure of my 980 Ti, I first started to get random Bad States. Then I started to see some GPU resets with clEnqueueReadBuffer (-5) error. Then, I started to find my system being turned off automatically. And one day, after 9 months of operations, after powering it back on, the card blew up (VRM burned) when Windows activated it. The short circuit in the VRMs prevented the machine from even powering up.

Thanks for your answer. I take this problem really serioiusly.
Are there any tests/SW/tools you can recommend that prove that something is wrong with my GPU?

bruce wrote:I did check, and you got partial credit for your efforts. It was reassigned and somebody else who completed it. I have no ideal what might be different about their system.

Hi foldinghomealone (team 70335),
Your WU (P10496 R144 C3 G5) was added to the stats database on 2017-01-31 21:07:04 for 4385.88 points of credit. (partial)
Hi ***** (team ******),
Your WU (P10496 R144 C3 G5) was added to the stats database on 2017-02-02 03:08:37 for 12183 points of credit.
...

toTOW wrote: 'Similar to the error with overclocking ('Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?'). '
This error is different : when the GPU is getting reset, the core doesn't see it anymore (this basically what the CL_OUT_OF_RESOURCES error means) for a short period. That why the client treat it as a serious error and dumps the WU.

Somehow there seem to be two different explainations about the same problem.
When someone else can finish the WU, why can't I do it myself?

I would understand that the client thinks that my GPU produces crap and therefore dumps the WU. Okay.
But why can someone else complete (as Bruce stated) the WU? Either my results are crap and nobody can use them and has to process the whole WU or my results are ok and I can finish it myself.

I don't need an answer to that question but I would be grateful if you'd considered my thoughts.

Please answer my question regarding GPU tests, though. Thank you.

SUCCESS!!! Other than some minor problems in the Putty Screen (a problem I've NEVER had before because, usually when I'm loading NVIDIA drivers in the Putty Screen FAH is NOT yet installed and trying to run), once I got 375.26 installed, and open the Advanced Control window, all my cards were loaded, and running jobs like nothing ever happened. I don't watch the forum much, except for when I have a problem, or I'd have probably seen something about this back in December when the new Driver came out. I'll have to get a calendar in here and mark it so I check here every few weeks. If I have any further problems, I'll be sure to post, but I'm extremely optimistic for the moment.

foldinghomealone wrote:Somehow there seem to be two different explainations about the same problem.
When someone else can finish the WU, why can't I do it myself?

I would understand that the client thinks that my GPU produces crap and therefore dumps the WU. Okay.
But why can someone else complete (as Bruce stated) the WU? Either my results are crap and nobody can use them and has to process the whole WU or my results are ok and I can finish it myself.

I don't need an answer to that question but I would be grateful if you'd considered my thoughts.

Please answer my question regarding GPU tests, though. Thank you.

There are probably several more explanations why a WU might have this error. We do try to sort out which might apply to you.

1) Maybe the data in the WU caused the problem.
The fact that somebody else could complete it seems to suggest this isn't the problem. Conclusion: there's probably something wrong on your system.

2) At this point, we have to ask you to provide answers to everything that might be different on your system compared to somebody else's.
a> Do you have a hardware defect or impending failure?
b> Is there a problem with the drivers you're running?
c> Is your system marginally stable due to overclocking, overheating, power limitations, etc.?
d> Was this a one-time event caused by, say, a cosmic ray flipping bits in memory?
e> Etc.

Note that <c> and <d> considered than your system produced crap ONCE and it might not be a frequent event. This is directly related to your question about software for testing. The generally understood recommendations surrounding reliability testing include
* You need to run the test for a long enough time
* Hardware is not monolithic. Different software focuses on different aspects of the hardware and saying something appears to be stable when tested with software-X does not assure it will pass testing with software-Y
* No matter how you test, you'll not have a guarantee that some combinations of factors may still lead to failures, so you have to add an additional margin of safety.

The GPU validation software that provides the best match for how FAH stresses each sub-component is FAHBench, which is based on actually running a segment of a FAH simulation.

Thanks Bruce for answering in detail.

The system was stable for weeks. A few weeks ago it startet that I lose 10-15% of WUs. For the last 26 WUs there was no fault at all.
Used driver is 372.70 which is considered stable.
As it was not a one time event and someone could complete my aborted WUs I now consider my system as 'damaged'.
So no need to waste your time on analyzing my problems in detail.

As my GPU is still in warranty period, I might consider returning it. But in day to day use there don't seem to be any faults.
Is there any diagnostic software that I can use as proof that the GPU is damaged in case the diagnostic software shows some errors?