Cannot finish Core 21 WU

^w^ing · Post by **^w^ing** » Sun Sep 06, 2015 11:33 am

Hello, today I got my first Core 21 Wor Unit and I cannot finish it - after a while of folding my PC halts to a crawl and the WU seems to be either stuck or progressing extremely slowly compared to the first couple of %. Happened as I left folding running over night to find my PC in the morning extremely unresponsive, but still seemingly folding.

Code: Select all

*********************** Log Started 2015-09-06T00:55:15Z ***********************
00:55:15:************************* Folding@home Client *************************
00:55:15:      Website: http://folding.stanford.edu/
00:55:15:    Copyright: (c) 2009-2014 Stanford University
00:55:15:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:55:15:         Args: 
00:55:15:       Config: D:/$TEMP$/fah/config.xml
00:55:15:******************************** Build ********************************
00:55:15:      Version: 7.4.4
00:55:15:         Date: Mar 4 2014
00:55:15:         Time: 20:26:54
00:55:15:      SVN Rev: 4130
00:55:15:       Branch: fah/trunk/client
00:55:15:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
00:55:15:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
00:55:15:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
00:55:15:     Platform: win32 XP
00:55:15:         Bits: 32
00:55:15:         Mode: Release
00:55:15:******************************* System ********************************
00:55:15:          CPU: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
00:55:15:       CPU ID: GenuineIntel Family 6 Model 26 Stepping 5
00:55:15:         CPUs: 8
00:55:15:       Memory: 17.99GiB
00:55:15:  Free Memory: 16.11GiB
00:55:15:      Threads: WINDOWS_THREADS
00:55:15:   OS Version: 6.1
00:55:15:  Has Battery: false
00:55:15:   On Battery: false
00:55:15:   UTC Offset: 2
00:55:15:          PID: 4340
00:55:15:          CWD: D:/$TEMP$/fah
00:55:15:           OS: Windows 7 Ultimate
00:55:15:      OS Arch: AMD64
00:55:15:         GPUs: 1
00:55:15:        GPU 0: NVIDIA:3 GK110 [GeForce GTX 780 Ti]
00:55:15:         CUDA: 3.5
00:55:15:  CUDA Driver: 7050
00:55:15:Win32 Service: false
00:55:15:***********************************************************************
00:55:15:<config>
00:55:15:  <!-- Folding Core -->
00:55:15:  <checkpoint v='10'/>
00:55:15:
00:55:15:  <!-- Folding Slot Configuration -->
00:55:15:  <extra-core-args v='-forceasm -verbose'/>
00:55:15:  <max-packet-size v='big'/>
00:55:15:
00:55:15:  <!-- HTTP Server -->
00:55:15:  <allow v='127.0.0.1 192.168.20.0/24'/>
00:55:15:
00:55:15:  <!-- Network -->
00:55:15:  <proxy v=':8080'/>
00:55:15:
00:55:15:  <!-- Remote Command Server -->
00:55:15:  <password v='*****'/>
00:55:15:
00:55:15:  <!-- Slot Control -->
00:55:15:  <power v='full'/>
00:55:15:
00:55:15:  <!-- User Information -->
00:55:15:  <passkey v='********************************'/>
00:55:15:  <user v='^w^ing'/>
00:55:15:
00:55:15:  <!-- Work Unit Control -->
00:55:15:  <next-unit-percentage v='100'/>
00:55:15:
00:55:15:  <!-- Folding Slots -->
00:55:15:  <slot id='0' type='GPU'>
00:55:15:    <client-type v='advanced'/>
00:55:15:    <core-priority v='low'/>
00:55:15:  </slot>
00:55:15:</config>
00:55:15:Trying to access database...
00:55:15:Successfully acquired database lock
00:55:15:Enabled folding slot 00: READY gpu:0:GK110 [GeForce GTX 780 Ti]
00:55:16:WU00:FS00:Connecting to 171.67.108.200:80
00:55:17:WU00:FS00:Assigned to work server 171.67.108.31
00:55:17:WU00:FS00:Requesting new work unit for slot 00: READY gpu:0:GK110 [GeForce GTX 780 Ti] from 171.67.108.31
00:55:17:WU00:FS00:Connecting to 171.67.108.31:8080
00:55:19:WU00:FS00:Downloading 1.39MiB
00:55:21:WU00:FS00:Download complete
00:55:21:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9615 run:6 clone:71 gen:50 core:0x18 unit:0x0000003a0a3b1e815546ef4639ab923f
00:55:21:WU00:FS00:Starting
00:55:21:WU00:FS00:Running FahCore: C:\FoldingAtHome\FAHClient/FAHCoreWrapper.exe D:/$TEMP$/fah/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 4340 -checkpoint 10 -gpu 0 -gpu-vendor nvidia -forceasm -verbose
00:55:21:WU00:FS00:Started FahCore on PID 4780
00:55:21:WU00:FS00:Core PID:2856
00:55:21:WU00:FS00:FahCore 0x18 started
00:55:24:WU00:FS00:0x18:*********************** Log Started 2015-09-06T00:55:23Z ***********************
00:55:24:WU00:FS00:0x18:Project: 9615 (Run 6, Clone 71, Gen 50)
00:55:24:WU00:FS00:0x18:Unit: 0x0000003a0a3b1e815546ef4639ab923f
00:55:24:WU00:FS00:0x18:CPU: 0x00000000000000000000000000000000
00:55:24:WU00:FS00:0x18:Machine: 0
00:55:24:WU00:FS00:0x18:Reading tar file core.xml
00:55:24:WU00:FS00:0x18:Reading tar file system.xml
00:55:24:WU00:FS00:0x18:Reading tar file integrator.xml
00:55:24:WU00:FS00:0x18:Reading tar file state.xml
00:55:24:WU00:FS00:0x18:Digital signatures verified
00:55:24:WU00:FS00:0x18:Folding@home GPU core18
00:55:24:WU00:FS00:0x18:Version 0.0.4
00:55:41:WU00:FS00:0x18:Completed 0 out of 2000000 steps (0%)
00:55:41:WU00:FS00:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
00:56:33:WU00:FS00:0x18:Completed 20000 out of 2000000 steps (1%)
00:57:20:WU00:FS00:0x18:Completed 40000 out of 2000000 steps (2%)
00:58:08:WU00:FS00:0x18:Completed 60000 out of 2000000 steps (3%)
00:58:59:WU00:FS00:0x18:Completed 80000 out of 2000000 steps (4%)
00:59:46:WU00:FS00:0x18:Completed 100000 out of 2000000 steps (5%)
01:00:38:WU00:FS00:0x18:Completed 120000 out of 2000000 steps (6%)
01:01:25:WU00:FS00:0x18:Completed 140000 out of 2000000 steps (7%)
01:02:11:WU00:FS00:0x18:Completed 160000 out of 2000000 steps (8%)
01:02:58:WU00:FS00:0x18:Completed 180000 out of 2000000 steps (9%)
01:03:44:WU00:FS00:0x18:Completed 200000 out of 2000000 steps (10%)
01:04:37:WU00:FS00:0x18:Completed 220000 out of 2000000 steps (11%)
01:05:24:WU00:FS00:0x18:Completed 240000 out of 2000000 steps (12%)
01:06:10:WU00:FS00:0x18:Completed 260000 out of 2000000 steps (13%)
01:06:57:WU00:FS00:0x18:Completed 280000 out of 2000000 steps (14%)
01:07:43:WU00:FS00:0x18:Completed 300000 out of 2000000 steps (15%)
01:08:36:WU00:FS00:0x18:Completed 320000 out of 2000000 steps (16%)
01:09:22:WU00:FS00:0x18:Completed 340000 out of 2000000 steps (17%)
01:10:09:WU00:FS00:0x18:Completed 360000 out of 2000000 steps (18%)
01:10:55:WU00:FS00:0x18:Completed 380000 out of 2000000 steps (19%)
01:11:41:WU00:FS00:0x18:Completed 400000 out of 2000000 steps (20%)
01:12:34:WU00:FS00:0x18:Completed 420000 out of 2000000 steps (21%)
01:13:21:WU00:FS00:0x18:Completed 440000 out of 2000000 steps (22%)
01:14:07:WU00:FS00:0x18:Completed 460000 out of 2000000 steps (23%)
01:14:53:WU00:FS00:0x18:Completed 480000 out of 2000000 steps (24%)
01:15:40:WU00:FS00:0x18:Completed 500000 out of 2000000 steps (25%)
01:16:33:WU00:FS00:0x18:Completed 520000 out of 2000000 steps (26%)
01:17:19:WU00:FS00:0x18:Completed 540000 out of 2000000 steps (27%)
01:18:05:WU00:FS00:0x18:Completed 560000 out of 2000000 steps (28%)
01:18:52:WU00:FS00:0x18:Completed 580000 out of 2000000 steps (29%)
01:19:38:WU00:FS00:0x18:Completed 600000 out of 2000000 steps (30%)
01:20:31:WU00:FS00:0x18:Completed 620000 out of 2000000 steps (31%)
01:21:17:WU00:FS00:0x18:Completed 640000 out of 2000000 steps (32%)
01:22:04:WU00:FS00:0x18:Completed 660000 out of 2000000 steps (33%)
01:22:50:WU00:FS00:0x18:Completed 680000 out of 2000000 steps (34%)
01:23:36:WU00:FS00:0x18:Completed 700000 out of 2000000 steps (35%)
01:24:29:WU00:FS00:0x18:Completed 720000 out of 2000000 steps (36%)
01:25:15:WU00:FS00:0x18:Completed 740000 out of 2000000 steps (37%)
01:26:02:WU00:FS00:0x18:Completed 760000 out of 2000000 steps (38%)
01:26:48:WU00:FS00:0x18:Completed 780000 out of 2000000 steps (39%)
01:27:35:WU00:FS00:0x18:Completed 800000 out of 2000000 steps (40%)
01:28:27:WU00:FS00:0x18:Completed 820000 out of 2000000 steps (41%)
01:29:14:WU00:FS00:0x18:Completed 840000 out of 2000000 steps (42%)
01:30:00:WU00:FS00:0x18:Completed 860000 out of 2000000 steps (43%)
01:30:46:WU00:FS00:0x18:Completed 880000 out of 2000000 steps (44%)
01:31:33:WU00:FS00:0x18:Completed 900000 out of 2000000 steps (45%)
01:32:25:WU00:FS00:0x18:Completed 920000 out of 2000000 steps (46%)
01:33:12:WU00:FS00:0x18:Completed 940000 out of 2000000 steps (47%)
01:33:58:WU00:FS00:0x18:Completed 960000 out of 2000000 steps (48%)
01:34:44:WU00:FS00:0x18:Completed 980000 out of 2000000 steps (49%)
01:35:31:WU00:FS00:0x18:Completed 1000000 out of 2000000 steps (50%)
01:36:24:WU00:FS00:0x18:Completed 1020000 out of 2000000 steps (51%)
01:37:10:WU00:FS00:0x18:Completed 1040000 out of 2000000 steps (52%)
01:37:56:WU00:FS00:0x18:Completed 1060000 out of 2000000 steps (53%)
01:38:43:WU00:FS00:0x18:Completed 1080000 out of 2000000 steps (54%)
01:39:29:WU00:FS00:0x18:Completed 1100000 out of 2000000 steps (55%)
01:40:22:WU00:FS00:0x18:Completed 1120000 out of 2000000 steps (56%)
01:41:08:WU00:FS00:0x18:Completed 1140000 out of 2000000 steps (57%)
01:41:55:WU00:FS00:0x18:Completed 1160000 out of 2000000 steps (58%)
01:42:41:WU00:FS00:0x18:Completed 1180000 out of 2000000 steps (59%)
01:43:27:WU00:FS00:0x18:Completed 1200000 out of 2000000 steps (60%)
01:44:20:WU00:FS00:0x18:Completed 1220000 out of 2000000 steps (61%)
01:45:07:WU00:FS00:0x18:Completed 1240000 out of 2000000 steps (62%)
01:45:53:WU00:FS00:0x18:Completed 1260000 out of 2000000 steps (63%)
01:46:39:WU00:FS00:0x18:Completed 1280000 out of 2000000 steps (64%)
01:47:25:WU00:FS00:0x18:Completed 1300000 out of 2000000 steps (65%)
01:48:18:WU00:FS00:0x18:Completed 1320000 out of 2000000 steps (66%)
01:49:05:WU00:FS00:0x18:Completed 1340000 out of 2000000 steps (67%)
01:49:51:WU00:FS00:0x18:Completed 1360000 out of 2000000 steps (68%)
01:50:37:WU00:FS00:0x18:Completed 1380000 out of 2000000 steps (69%)
01:51:24:WU00:FS00:0x18:Completed 1400000 out of 2000000 steps (70%)
01:52:16:WU00:FS00:0x18:Completed 1420000 out of 2000000 steps (71%)
01:53:03:WU00:FS00:0x18:Completed 1440000 out of 2000000 steps (72%)
01:53:49:WU00:FS00:0x18:Completed 1460000 out of 2000000 steps (73%)
01:54:35:WU00:FS00:0x18:Completed 1480000 out of 2000000 steps (74%)
01:55:22:WU00:FS00:0x18:Completed 1500000 out of 2000000 steps (75%)
01:56:14:WU00:FS00:0x18:Completed 1520000 out of 2000000 steps (76%)
01:57:01:WU00:FS00:0x18:Completed 1540000 out of 2000000 steps (77%)
01:57:47:WU00:FS00:0x18:Completed 1560000 out of 2000000 steps (78%)
01:58:34:WU00:FS00:0x18:Completed 1580000 out of 2000000 steps (79%)
01:59:20:WU00:FS00:0x18:Completed 1600000 out of 2000000 steps (80%)
02:00:13:WU00:FS00:0x18:Completed 1620000 out of 2000000 steps (81%)
02:00:59:WU00:FS00:0x18:Completed 1640000 out of 2000000 steps (82%)
02:01:46:WU00:FS00:0x18:Completed 1660000 out of 2000000 steps (83%)
02:02:32:WU00:FS00:0x18:Completed 1680000 out of 2000000 steps (84%)
02:03:18:WU00:FS00:0x18:Completed 1700000 out of 2000000 steps (85%)
02:04:11:WU00:FS00:0x18:Completed 1720000 out of 2000000 steps (86%)
02:04:57:WU00:FS00:0x18:Completed 1740000 out of 2000000 steps (87%)
02:05:43:WU00:FS00:0x18:Completed 1760000 out of 2000000 steps (88%)
02:06:30:WU00:FS00:0x18:Completed 1780000 out of 2000000 steps (89%)
02:07:16:WU00:FS00:0x18:Completed 1800000 out of 2000000 steps (90%)
02:08:09:WU00:FS00:0x18:Completed 1820000 out of 2000000 steps (91%)
02:08:55:WU00:FS00:0x18:Completed 1840000 out of 2000000 steps (92%)
02:09:42:WU00:FS00:0x18:Completed 1860000 out of 2000000 steps (93%)
02:10:28:WU00:FS00:0x18:Completed 1880000 out of 2000000 steps (94%)
02:11:14:WU00:FS00:0x18:Completed 1900000 out of 2000000 steps (95%)
02:12:07:WU00:FS00:0x18:Completed 1920000 out of 2000000 steps (96%)
02:12:54:WU00:FS00:0x18:Completed 1940000 out of 2000000 steps (97%)
02:13:40:WU00:FS00:0x18:Completed 1960000 out of 2000000 steps (98%)
02:14:26:WU00:FS00:0x18:Completed 1980000 out of 2000000 steps (99%)
02:15:12:WU00:FS00:0x18:Completed 2000000 out of 2000000 steps (100%)
02:15:14:WU01:FS00:Connecting to 171.67.108.200:80
02:15:15:WU01:FS00:Assigned to work server 171.64.65.93
02:15:15:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.93
02:15:15:WU01:FS00:Connecting to 171.64.65.93:8080
02:15:16:WU01:FS00:Downloading 3.47MiB
02:15:19:WU00:FS00:0x18:Saving result file logfile_01.txt
02:15:19:WU00:FS00:0x18:Saving result file checkpointState.xml
02:15:20:WU00:FS00:0x18:Saving result file checkpt.crc
02:15:20:WU00:FS00:0x18:Saving result file log.txt
02:15:20:WU00:FS00:0x18:Saving result file positions.xtc
02:15:20:WU00:FS00:0x18:Folding@home Core Shutdown: FINISHED_UNIT
02:15:20:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:15:21:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9615 run:6 clone:71 gen:50 core:0x18 unit:0x0000003a0a3b1e815546ef4639ab923f
02:15:21:WU01:FS00:Download complete
02:15:21:WU00:FS00:Uploading 2.66MiB to 171.67.108.31
02:15:21:WU00:FS00:Connecting to 171.67.108.31:8080
02:15:21:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9127 run:18 clone:4 gen:65 core:0x18 unit:0x000000540a3b1e815549367dc15117fe
02:15:21:WU01:FS00:Starting
02:15:21:WU01:FS00:Running FahCore: C:\FoldingAtHome\FAHClient/FAHCoreWrapper.exe D:/$TEMP$/fah/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 01 -suffix 01 -version 704 -lifeline 4340 -checkpoint 10 -gpu 0 -gpu-vendor nvidia -forceasm -verbose
02:15:21:WU01:FS00:Started FahCore on PID 4320
02:15:21:WU01:FS00:Core PID:4016
02:15:21:WU01:FS00:FahCore 0x18 started
02:15:22:WU01:FS00:0x18:*********************** Log Started 2015-09-06T02:15:21Z ***********************
02:15:22:WU01:FS00:0x18:Project: 9127 (Run 18, Clone 4, Gen 65)
02:15:22:WU01:FS00:0x18:Unit: 0x000000540a3b1e815549367dc15117fe
02:15:22:WU01:FS00:0x18:CPU: 0x00000000000000000000000000000000
02:15:22:WU01:FS00:0x18:Machine: 0
02:15:22:WU01:FS00:0x18:Reading tar file state.xml
02:15:22:WU01:FS00:0x18:Reading tar file system.xml
02:15:22:WU01:FS00:0x18:Reading tar file integrator.xml
02:15:22:WU01:FS00:0x18:Reading tar file core.xml
02:15:22:WU01:FS00:0x18:Digital signatures verified
02:15:22:WU01:FS00:0x18:Folding@home GPU core18
02:15:22:WU01:FS00:0x18:Version 0.0.4
02:15:27:WU00:FS00:Upload 16.42%
02:15:33:WU00:FS00:Upload 30.49%
02:15:39:WU00:FS00:Upload 44.56%
02:15:45:WU00:FS00:Upload 60.98%
02:15:50:WU01:FS00:0x18:Completed 0 out of 2500000 steps (0%)
02:15:50:WU01:FS00:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:15:51:WU00:FS00:Upload 75.05%
02:15:57:WU00:FS00:Upload 89.12%
02:16:05:WU00:FS00:Upload complete
02:16:05:WU00:FS00:Server responded WORK_ACK (400)
02:16:05:WU00:FS00:Final credit estimate, 14177.00 points
02:16:06:WU00:FS00:Cleaning up
02:17:52:WU01:FS00:0x18:Completed 25000 out of 2500000 steps (1%)
02:19:44:WU01:FS00:0x18:Completed 50000 out of 2500000 steps (2%)
02:21:35:WU01:FS00:0x18:Completed 75000 out of 2500000 steps (3%)
02:23:26:WU01:FS00:0x18:Completed 100000 out of 2500000 steps (4%)
02:25:30:WU01:FS00:0x18:Completed 125000 out of 2500000 steps (5%)
02:27:21:WU01:FS00:0x18:Completed 150000 out of 2500000 steps (6%)
02:29:13:WU01:FS00:0x18:Completed 175000 out of 2500000 steps (7%)
02:31:04:WU01:FS00:0x18:Completed 200000 out of 2500000 steps (8%)
02:33:08:WU01:FS00:0x18:Completed 225000 out of 2500000 steps (9%)
02:34:59:WU01:FS00:0x18:Completed 250000 out of 2500000 steps (10%)
02:36:51:WU01:FS00:0x18:Completed 275000 out of 2500000 steps (11%)
02:38:42:WU01:FS00:0x18:Completed 300000 out of 2500000 steps (12%)
02:40:46:WU01:FS00:0x18:Completed 325000 out of 2500000 steps (13%)
02:42:37:WU01:FS00:0x18:Completed 350000 out of 2500000 steps (14%)
02:44:29:WU01:FS00:0x18:Completed 375000 out of 2500000 steps (15%)
02:46:20:WU01:FS00:0x18:Completed 400000 out of 2500000 steps (16%)
02:48:24:WU01:FS00:0x18:Completed 425000 out of 2500000 steps (17%)
02:50:15:WU01:FS00:0x18:Completed 450000 out of 2500000 steps (18%)
02:52:06:WU01:FS00:0x18:Completed 475000 out of 2500000 steps (19%)
02:53:58:WU01:FS00:0x18:Completed 500000 out of 2500000 steps (20%)
02:56:02:WU01:FS00:0x18:Completed 525000 out of 2500000 steps (21%)
02:57:53:WU01:FS00:0x18:Completed 550000 out of 2500000 steps (22%)
02:59:45:WU01:FS00:0x18:Completed 575000 out of 2500000 steps (23%)
03:01:36:WU01:FS00:0x18:Completed 600000 out of 2500000 steps (24%)
03:03:40:WU01:FS00:0x18:Completed 625000 out of 2500000 steps (25%)
03:05:31:WU01:FS00:0x18:Completed 650000 out of 2500000 steps (26%)
03:07:22:WU01:FS00:0x18:Completed 675000 out of 2500000 steps (27%)
03:09:14:WU01:FS00:0x18:Completed 700000 out of 2500000 steps (28%)
03:11:18:WU01:FS00:0x18:Completed 725000 out of 2500000 steps (29%)
03:13:09:WU01:FS00:0x18:Completed 750000 out of 2500000 steps (30%)
03:15:00:WU01:FS00:0x18:Completed 775000 out of 2500000 steps (31%)
03:16:52:WU01:FS00:0x18:Completed 800000 out of 2500000 steps (32%)
03:18:56:WU01:FS00:0x18:Completed 825000 out of 2500000 steps (33%)
03:20:47:WU01:FS00:0x18:Completed 850000 out of 2500000 steps (34%)
03:22:39:WU01:FS00:0x18:Completed 875000 out of 2500000 steps (35%)
03:24:30:WU01:FS00:0x18:Completed 900000 out of 2500000 steps (36%)
03:26:34:WU01:FS00:0x18:Completed 925000 out of 2500000 steps (37%)
03:28:25:WU01:FS00:0x18:Completed 950000 out of 2500000 steps (38%)
03:30:17:WU01:FS00:0x18:Completed 975000 out of 2500000 steps (39%)
03:32:08:WU01:FS00:0x18:Completed 1000000 out of 2500000 steps (40%)
03:34:12:WU01:FS00:0x18:Completed 1025000 out of 2500000 steps (41%)
03:36:03:WU01:FS00:0x18:Completed 1050000 out of 2500000 steps (42%)
03:37:55:WU01:FS00:0x18:Completed 1075000 out of 2500000 steps (43%)
03:39:46:WU01:FS00:0x18:Completed 1100000 out of 2500000 steps (44%)
03:41:50:WU01:FS00:0x18:Completed 1125000 out of 2500000 steps (45%)
03:43:41:WU01:FS00:0x18:Completed 1150000 out of 2500000 steps (46%)
03:45:33:WU01:FS00:0x18:Completed 1175000 out of 2500000 steps (47%)
03:47:24:WU01:FS00:0x18:Completed 1200000 out of 2500000 steps (48%)
03:49:28:WU01:FS00:0x18:Completed 1225000 out of 2500000 steps (49%)
03:51:20:WU01:FS00:0x18:Completed 1250000 out of 2500000 steps (50%)
03:53:11:WU01:FS00:0x18:Completed 1275000 out of 2500000 steps (51%)
03:55:02:WU01:FS00:0x18:Completed 1300000 out of 2500000 steps (52%)
03:57:06:WU01:FS00:0x18:Completed 1325000 out of 2500000 steps (53%)
03:58:58:WU01:FS00:0x18:Completed 1350000 out of 2500000 steps (54%)
04:00:49:WU01:FS00:0x18:Completed 1375000 out of 2500000 steps (55%)
04:02:40:WU01:FS00:0x18:Completed 1400000 out of 2500000 steps (56%)
04:04:44:WU01:FS00:0x18:Completed 1425000 out of 2500000 steps (57%)
04:06:36:WU01:FS00:0x18:Completed 1450000 out of 2500000 steps (58%)
04:08:27:WU01:FS00:0x18:Completed 1475000 out of 2500000 steps (59%)
04:10:19:WU01:FS00:0x18:Completed 1500000 out of 2500000 steps (60%)
04:12:23:WU01:FS00:0x18:Completed 1525000 out of 2500000 steps (61%)
04:14:14:WU01:FS00:0x18:Completed 1550000 out of 2500000 steps (62%)
04:16:05:WU01:FS00:0x18:Completed 1575000 out of 2500000 steps (63%)
04:17:57:WU01:FS00:0x18:Completed 1600000 out of 2500000 steps (64%)
04:20:01:WU01:FS00:0x18:Completed 1625000 out of 2500000 steps (65%)
04:21:52:WU01:FS00:0x18:Completed 1650000 out of 2500000 steps (66%)
04:23:44:WU01:FS00:0x18:Completed 1675000 out of 2500000 steps (67%)
04:25:35:WU01:FS00:0x18:Completed 1700000 out of 2500000 steps (68%)
04:27:39:WU01:FS00:0x18:Completed 1725000 out of 2500000 steps (69%)
04:29:31:WU01:FS00:0x18:Completed 1750000 out of 2500000 steps (70%)
04:31:22:WU01:FS00:0x18:Completed 1775000 out of 2500000 steps (71%)
04:33:14:WU01:FS00:0x18:Completed 1800000 out of 2500000 steps (72%)
04:35:18:WU01:FS00:0x18:Completed 1825000 out of 2500000 steps (73%)
04:37:09:WU01:FS00:0x18:Completed 1850000 out of 2500000 steps (74%)
04:39:01:WU01:FS00:0x18:Completed 1875000 out of 2500000 steps (75%)
04:40:52:WU01:FS00:0x18:Completed 1900000 out of 2500000 steps (76%)
04:42:56:WU01:FS00:0x18:Completed 1925000 out of 2500000 steps (77%)
04:44:47:WU01:FS00:0x18:Completed 1950000 out of 2500000 steps (78%)
04:46:39:WU01:FS00:0x18:Completed 1975000 out of 2500000 steps (79%)
04:48:30:WU01:FS00:0x18:Completed 2000000 out of 2500000 steps (80%)
04:50:35:WU01:FS00:0x18:Completed 2025000 out of 2500000 steps (81%)
04:52:26:WU01:FS00:0x18:Completed 2050000 out of 2500000 steps (82%)
04:54:18:WU01:FS00:0x18:Completed 2075000 out of 2500000 steps (83%)
04:56:09:WU01:FS00:0x18:Completed 2100000 out of 2500000 steps (84%)
04:58:13:WU01:FS00:0x18:Completed 2125000 out of 2500000 steps (85%)
05:00:04:WU01:FS00:0x18:Completed 2150000 out of 2500000 steps (86%)
05:01:56:WU01:FS00:0x18:Completed 2175000 out of 2500000 steps (87%)
05:03:47:WU01:FS00:0x18:Completed 2200000 out of 2500000 steps (88%)
05:05:51:WU01:FS00:0x18:Completed 2225000 out of 2500000 steps (89%)
05:07:43:WU01:FS00:0x18:Completed 2250000 out of 2500000 steps (90%)
05:09:34:WU01:FS00:0x18:Completed 2275000 out of 2500000 steps (91%)
05:11:26:WU01:FS00:0x18:Completed 2300000 out of 2500000 steps (92%)
05:13:29:WU01:FS00:0x18:Completed 2325000 out of 2500000 steps (93%)
05:15:21:WU01:FS00:0x18:Completed 2350000 out of 2500000 steps (94%)
05:17:12:WU01:FS00:0x18:Completed 2375000 out of 2500000 steps (95%)
05:19:04:WU01:FS00:0x18:Completed 2400000 out of 2500000 steps (96%)
05:21:08:WU01:FS00:0x18:Completed 2425000 out of 2500000 steps (97%)
05:22:59:WU01:FS00:0x18:Completed 2450000 out of 2500000 steps (98%)
05:24:51:WU01:FS00:0x18:Completed 2475000 out of 2500000 steps (99%)
05:26:42:WU01:FS00:0x18:Completed 2500000 out of 2500000 steps (100%)
05:26:44:WU00:FS00:Connecting to 171.67.108.200:80
05:26:45:WU00:FS00:Assigned to work server 171.67.108.31
05:26:45:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GK110 [GeForce GTX 780 Ti] from 171.67.108.31
05:26:45:WU00:FS00:Connecting to 171.67.108.31:8080
05:26:46:WU00:FS00:Downloading 766.80KiB
05:26:48:WU00:FS00:Download complete
05:26:48:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:9612 run:5 clone:46 gen:63 core:0x18 unit:0x000000470a3b1e815546ebe9133dc906
05:26:55:WU01:FS00:0x18:Saving result file logfile_01.txt
05:26:55:WU01:FS00:0x18:Saving result file checkpointState.xml
05:26:56:WU01:FS00:0x18:Saving result file checkpt.crc
05:26:56:WU01:FS00:0x18:Saving result file log.txt
05:26:56:WU01:FS00:0x18:Saving result file positions.xtc
05:26:57:WU01:FS00:0x18:Folding@home Core Shutdown: FINISHED_UNIT
05:26:58:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:26:58:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:9127 run:18 clone:4 gen:65 core:0x18 unit:0x000000540a3b1e815549367dc15117fe
05:26:58:WU01:FS00:Uploading 6.59MiB to 171.64.65.93
05:26:58:WU00:FS00:Starting
05:26:58:WU01:FS00:Connecting to 171.64.65.93:8080
05:26:58:WU00:FS00:Running FahCore: C:\FoldingAtHome\FAHClient/FAHCoreWrapper.exe D:/$TEMP$/fah/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 4340 -checkpoint 10 -gpu 0 -gpu-vendor nvidia -forceasm -verbose
05:26:58:WU00:FS00:Started FahCore on PID 2084
05:26:58:WU00:FS00:Core PID:1484
05:26:58:WU00:FS00:FahCore 0x18 started
05:26:58:WU00:FS00:0x18:*********************** Log Started 2015-09-06T05:26:58Z ***********************
05:26:58:WU00:FS00:0x18:Project: 9612 (Run 5, Clone 46, Gen 63)
05:26:58:WU00:FS00:0x18:Unit: 0x000000470a3b1e815546ebe9133dc906
05:26:58:WU00:FS00:0x18:CPU: 0x00000000000000000000000000000000
05:26:58:WU00:FS00:0x18:Machine: 0
05:26:58:WU00:FS00:0x18:Reading tar file core.xml
05:26:58:WU00:FS00:0x18:Reading tar file system.xml
05:26:58:WU00:FS00:0x18:Reading tar file integrator.xml
05:26:58:WU00:FS00:0x18:Reading tar file state.xml
05:26:58:WU00:FS00:0x18:Digital signatures verified
05:26:58:WU00:FS00:0x18:Folding@home GPU core18
05:26:58:WU00:FS00:0x18:Version 0.0.4
05:27:04:WU01:FS00:Upload 8.54%
05:27:06:WU00:FS00:0x18:Completed 0 out of 2000000 steps (0%)
05:27:06:WU00:FS00:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
05:27:10:WU01:FS00:Upload 14.23%
05:27:16:WU01:FS00:Upload 19.92%
05:27:22:WU01:FS00:Upload 25.61%
05:27:28:WU01:FS00:Upload 31.30%
05:27:32:WU00:FS00:0x18:Completed 20000 out of 2000000 steps (1%)
05:27:34:WU01:FS00:Upload 36.99%
05:27:40:WU01:FS00:Upload 42.68%
05:27:46:WU01:FS00:Upload 47.42%
05:27:52:WU01:FS00:Upload 53.11%
05:27:55:WU00:FS00:0x18:Completed 40000 out of 2000000 steps (2%)
05:27:58:WU01:FS00:Upload 58.80%
05:28:04:WU01:FS00:Upload 63.54%
05:28:11:WU01:FS00:Upload 68.28%
05:28:17:WU01:FS00:Upload 73.97%
05:28:18:WU00:FS00:0x18:Completed 60000 out of 2000000 steps (3%)
05:28:23:WU01:FS00:Upload 78.72%
05:28:29:WU01:FS00:Upload 84.41%
05:28:35:WU01:FS00:Upload 90.10%
05:28:41:WU01:FS00:Upload 95.79%
05:28:42:WU00:FS00:0x18:Completed 80000 out of 2000000 steps (4%)
05:28:52:WU01:FS00:Upload complete
05:28:52:WU01:FS00:Server responded WORK_ACK (400)
05:28:52:WU01:FS00:Final credit estimate, 36899.00 points
05:28:52:WU01:FS00:Cleaning up
05:29:05:WU00:FS00:0x18:Completed 100000 out of 2000000 steps (5%)
05:29:32:WU00:FS00:0x18:Completed 120000 out of 2000000 steps (6%)
05:29:55:WU00:FS00:0x18:Completed 140000 out of 2000000 steps (7%)
05:30:18:WU00:FS00:0x18:Completed 160000 out of 2000000 steps (8%)
05:30:42:WU00:FS00:0x18:Completed 180000 out of 2000000 steps (9%)
05:31:06:WU00:FS00:0x18:Completed 200000 out of 2000000 steps (10%)
05:31:32:WU00:FS00:0x18:Completed 220000 out of 2000000 steps (11%)
05:31:55:WU00:FS00:0x18:Completed 240000 out of 2000000 steps (12%)
05:32:18:WU00:FS00:0x18:Completed 260000 out of 2000000 steps (13%)
05:32:42:WU00:FS00:0x18:Completed 280000 out of 2000000 steps (14%)
05:33:05:WU00:FS00:0x18:Completed 300000 out of 2000000 steps (15%)
05:33:31:WU00:FS00:0x18:Completed 320000 out of 2000000 steps (16%)
05:33:55:WU00:FS00:0x18:Completed 340000 out of 2000000 steps (17%)
05:34:18:WU00:FS00:0x18:Completed 360000 out of 2000000 steps (18%)
05:34:42:WU00:FS00:0x18:Completed 380000 out of 2000000 steps (19%)
05:35:05:WU00:FS00:0x18:Completed 400000 out of 2000000 steps (20%)
05:35:31:WU00:FS00:0x18:Completed 420000 out of 2000000 steps (21%)
05:35:55:WU00:FS00:0x18:Completed 440000 out of 2000000 steps (22%)
05:36:18:WU00:FS00:0x18:Completed 460000 out of 2000000 steps (23%)
05:36:42:WU00:FS00:0x18:Completed 480000 out of 2000000 steps (24%)
05:37:05:WU00:FS00:0x18:Completed 500000 out of 2000000 steps (25%)
05:37:31:WU00:FS00:0x18:Completed 520000 out of 2000000 steps (26%)
05:37:55:WU00:FS00:0x18:Completed 540000 out of 2000000 steps (27%)
05:38:18:WU00:FS00:0x18:Completed 560000 out of 2000000 steps (28%)
05:38:41:WU00:FS00:0x18:Completed 580000 out of 2000000 steps (29%)
05:39:05:WU00:FS00:0x18:Completed 600000 out of 2000000 steps (30%)
05:39:31:WU00:FS00:0x18:Completed 620000 out of 2000000 steps (31%)
05:39:54:WU00:FS00:0x18:Completed 640000 out of 2000000 steps (32%)
05:40:18:WU00:FS00:0x18:Completed 660000 out of 2000000 steps (33%)
05:40:41:WU00:FS00:0x18:Completed 680000 out of 2000000 steps (34%)
05:41:05:WU00:FS00:0x18:Completed 700000 out of 2000000 steps (35%)
05:41:31:WU00:FS00:0x18:Completed 720000 out of 2000000 steps (36%)
05:41:54:WU00:FS00:0x18:Completed 740000 out of 2000000 steps (37%)
05:42:18:WU00:FS00:0x18:Completed 760000 out of 2000000 steps (38%)
05:42:41:WU00:FS00:0x18:Completed 780000 out of 2000000 steps (39%)
05:43:05:WU00:FS00:0x18:Completed 800000 out of 2000000 steps (40%)
05:43:31:WU00:FS00:0x18:Completed 820000 out of 2000000 steps (41%)
05:43:54:WU00:FS00:0x18:Completed 840000 out of 2000000 steps (42%)
05:44:18:WU00:FS00:0x18:Completed 860000 out of 2000000 steps (43%)
05:44:41:WU00:FS00:0x18:Completed 880000 out of 2000000 steps (44%)
05:45:05:WU00:FS00:0x18:Completed 900000 out of 2000000 steps (45%)
05:45:31:WU00:FS00:0x18:Completed 920000 out of 2000000 steps (46%)
05:45:54:WU00:FS00:0x18:Completed 940000 out of 2000000 steps (47%)
05:46:18:WU00:FS00:0x18:Completed 960000 out of 2000000 steps (48%)
05:46:41:WU00:FS00:0x18:Completed 980000 out of 2000000 steps (49%)
05:47:05:WU00:FS00:0x18:Completed 1000000 out of 2000000 steps (50%)
05:47:31:WU00:FS00:0x18:Completed 1020000 out of 2000000 steps (51%)
05:47:54:WU00:FS00:0x18:Completed 1040000 out of 2000000 steps (52%)
05:48:18:WU00:FS00:0x18:Completed 1060000 out of 2000000 steps (53%)
05:48:41:WU00:FS00:0x18:Completed 1080000 out of 2000000 steps (54%)
05:49:05:WU00:FS00:0x18:Completed 1100000 out of 2000000 steps (55%)
05:49:31:WU00:FS00:0x18:Completed 1120000 out of 2000000 steps (56%)
05:49:54:WU00:FS00:0x18:Completed 1140000 out of 2000000 steps (57%)
05:50:18:WU00:FS00:0x18:Completed 1160000 out of 2000000 steps (58%)
05:50:41:WU00:FS00:0x18:Completed 1180000 out of 2000000 steps (59%)
05:51:04:WU00:FS00:0x18:Completed 1200000 out of 2000000 steps (60%)
05:51:31:WU00:FS00:0x18:Completed 1220000 out of 2000000 steps (61%)
05:51:54:WU00:FS00:0x18:Completed 1240000 out of 2000000 steps (62%)
05:52:17:WU00:FS00:0x18:Completed 1260000 out of 2000000 steps (63%)
05:52:41:WU00:FS00:0x18:Completed 1280000 out of 2000000 steps (64%)
05:53:05:WU00:FS00:0x18:Completed 1300000 out of 2000000 steps (65%)
05:53:30:WU00:FS00:0x18:Completed 1320000 out of 2000000 steps (66%)
05:53:54:WU00:FS00:0x18:Completed 1340000 out of 2000000 steps (67%)
05:54:18:WU00:FS00:0x18:Completed 1360000 out of 2000000 steps (68%)
05:54:41:WU00:FS00:0x18:Completed 1380000 out of 2000000 steps (69%)
05:55:04:WU00:FS00:0x18:Completed 1400000 out of 2000000 steps (70%)
05:55:30:WU00:FS00:0x18:Completed 1420000 out of 2000000 steps (71%)
05:55:54:WU00:FS00:0x18:Completed 1440000 out of 2000000 steps (72%)
05:56:17:WU00:FS00:0x18:Completed 1460000 out of 2000000 steps (73%)
05:56:41:WU00:FS00:0x18:Completed 1480000 out of 2000000 steps (74%)
05:57:04:WU00:FS00:0x18:Completed 1500000 out of 2000000 steps (75%)
05:57:30:WU00:FS00:0x18:Completed 1520000 out of 2000000 steps (76%)
05:57:54:WU00:FS00:0x18:Completed 1540000 out of 2000000 steps (77%)
05:58:17:WU00:FS00:0x18:Completed 1560000 out of 2000000 steps (78%)
05:58:41:WU00:FS00:0x18:Completed 1580000 out of 2000000 steps (79%)
05:59:04:WU00:FS00:0x18:Completed 1600000 out of 2000000 steps (80%)
05:59:30:WU00:FS00:0x18:Completed 1620000 out of 2000000 steps (81%)
05:59:54:WU00:FS00:0x18:Completed 1640000 out of 2000000 steps (82%)
06:00:17:WU00:FS00:0x18:Completed 1660000 out of 2000000 steps (83%)
06:00:41:WU00:FS00:0x18:Completed 1680000 out of 2000000 steps (84%)
06:01:04:WU00:FS00:0x18:Completed 1700000 out of 2000000 steps (85%)
06:01:30:WU00:FS00:0x18:Completed 1720000 out of 2000000 steps (86%)
06:01:53:WU00:FS00:0x18:Completed 1740000 out of 2000000 steps (87%)
06:02:17:WU00:FS00:0x18:Completed 1760000 out of 2000000 steps (88%)
06:02:40:WU00:FS00:0x18:Completed 1780000 out of 2000000 steps (89%)
06:03:04:WU00:FS00:0x18:Completed 1800000 out of 2000000 steps (90%)
06:03:30:WU00:FS00:0x18:Completed 1820000 out of 2000000 steps (91%)
06:03:53:WU00:FS00:0x18:Completed 1840000 out of 2000000 steps (92%)
06:04:17:WU00:FS00:0x18:Completed 1860000 out of 2000000 steps (93%)
06:04:40:WU00:FS00:0x18:Completed 1880000 out of 2000000 steps (94%)
06:05:04:WU00:FS00:0x18:Completed 1900000 out of 2000000 steps (95%)
06:05:30:WU00:FS00:0x18:Completed 1920000 out of 2000000 steps (96%)
06:05:53:WU00:FS00:0x18:Completed 1940000 out of 2000000 steps (97%)
06:06:16:WU00:FS00:0x18:Completed 1960000 out of 2000000 steps (98%)
06:06:40:WU00:FS00:0x18:Completed 1980000 out of 2000000 steps (99%)
06:07:03:WU00:FS00:0x18:Completed 2000000 out of 2000000 steps (100%)
06:07:05:WU01:FS00:Connecting to 171.67.108.200:80
06:07:05:WU01:FS00:Assigned to work server 171.64.65.98
06:07:05:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GK110 [GeForce GTX 780 Ti] from 171.64.65.98
06:07:05:WU01:FS00:Connecting to 171.64.65.98:8080
06:07:06:WU00:FS00:0x18:Saving result file logfile_01.txt
06:07:06:WU00:FS00:0x18:Saving result file checkpointState.xml
06:07:06:WU01:FS00:Downloading 7.49MiB
06:07:06:WU00:FS00:0x18:Saving result file checkpt.crc
06:07:06:WU00:FS00:0x18:Saving result file log.txt
06:07:06:WU00:FS00:0x18:Saving result file positions.xtc
06:07:07:WU00:FS00:0x18:Folding@home Core Shutdown: FINISHED_UNIT
06:07:07:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
06:07:07:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:9612 run:5 clone:46 gen:63 core:0x18 unit:0x000000470a3b1e815546ebe9133dc906
06:07:07:WU00:FS00:Uploading 1.45MiB to 171.67.108.31
06:07:07:WU00:FS00:Connecting to 171.67.108.31:8080
06:07:12:WU01:FS00:Download 56.78%
06:07:13:WU00:FS00:Upload 25.77%
06:07:14:WU01:FS00:Download complete
06:07:14:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:9712 run:6 clone:0 gen:14 core:0x21 unit:0x00000044ab40416255b9a71b4787447e
06:07:14:WU01:FS00:Downloading core from http://web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah
06:07:14:WU01:FS00:Connecting to web.stanford.edu:80
06:07:15:WU01:FS00:FahCore 21: Downloading 3.34MiB
06:07:19:WU00:FS00:Upload 51.55%
06:07:21:WU01:FS00:FahCore 21: 52.35%
06:07:25:WU01:FS00:FahCore 21: Download complete
06:07:25:WU00:FS00:Upload 77.32%
06:07:25:WU01:FS00:Valid core signature
06:07:25:WU01:FS00:Unpacked 11.43MiB to cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe
06:07:26:WU01:FS00:Starting
06:07:26:WU01:FS00:Running FahCore: C:\FoldingAtHome\FAHClient/FAHCoreWrapper.exe D:/$TEMP$/fah/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 4340 -checkpoint 10 -gpu 0 -gpu-vendor nvidia -forceasm -verbose
06:07:26:WU01:FS00:Started FahCore on PID 2848
06:07:26:WU01:FS00:Core PID:2624
06:07:26:WU01:FS00:FahCore 0x21 started
06:07:27:WU01:FS00:0x21:*********************** Log Started 2015-09-06T06:07:27Z ***********************
06:07:27:WU01:FS00:0x21:Project: 9712 (Run 6, Clone 0, Gen 14)
06:07:27:WU01:FS00:0x21:Unit: 0x00000044ab40416255b9a71b4787447e
06:07:27:WU01:FS00:0x21:CPU: 0x00000000000000000000000000000000
06:07:27:WU01:FS00:0x21:Machine: 0
06:07:27:WU01:FS00:0x21:Reading tar file core.xml
06:07:27:WU01:FS00:0x21:Reading tar file integrator.xml
06:07:27:WU01:FS00:0x21:Reading tar file system.xml
06:07:28:WU01:FS00:0x21:Reading tar file state.xml
06:07:29:WU01:FS00:0x21:Digital signatures verified
06:07:29:WU01:FS00:0x21:Folding@home GPU Core21 Folding@home Core
06:07:29:WU01:FS00:0x21:Version 0.0.11
06:07:33:WU00:FS00:Upload complete
06:07:33:WU00:FS00:Server responded WORK_ACK (400)
06:07:33:WU00:FS00:Final credit estimate, 6595.00 points
06:07:34:WU00:FS00:Cleaning up
06:08:28:WU01:FS00:0x21:Completed 0 out of 1280000 steps (0%)
06:08:28:WU01:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
06:10:19:WU01:FS00:0x21:Completed 12800 out of 1280000 steps (1%)
06:11:53:WU01:FS00:0x21:Completed 25600 out of 1280000 steps (2%)
06:13:26:WU01:FS00:0x21:Completed 38400 out of 1280000 steps (3%)
06:52:51:WU01:FS00:0x21:Completed 51200 out of 1280000 steps (4%)
******************************* Date: 2015-09-06 *******************************
07:46:14:WU01:FS00:0x21:Completed 64000 out of 1280000 steps (5%)
08:39:39:WU01:FS00:0x21:Completed 76800 out of 1280000 steps (6%)
08:53:00:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint
08:54:34:WU01:FS00:0x21:Completed 12800 out of 1280000 steps (1%)
09:08:11:WARNING:Console control signal 2 on PID 4340
09:08:11:Exiting, please wait. . .

when I found out in the morning, I deleted everything in the work unit folder except for wudata_01.dat, started folding and it happened again at 4%

Code: Select all

*********************** Log Started 2015-09-06T10:48:27Z ***********************
10:48:27:************************* Folding@home Client *************************
10:48:27:      Website: http://folding.stanford.edu/
10:48:27:    Copyright: (c) 2009-2014 Stanford University
10:48:27:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
10:48:27:         Args: 
10:48:27:       Config: D:/$TEMP$/fah/config.xml
10:48:27:******************************** Build ********************************
10:48:27:      Version: 7.4.4
10:48:27:         Date: Mar 4 2014
10:48:27:         Time: 20:26:54
10:48:27:      SVN Rev: 4130
10:48:27:       Branch: fah/trunk/client
10:48:27:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
10:48:27:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
10:48:27:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
10:48:27:     Platform: win32 XP
10:48:27:         Bits: 32
10:48:27:         Mode: Release
10:48:27:******************************* System ********************************
10:48:27:          CPU: Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
10:48:27:       CPU ID: GenuineIntel Family 6 Model 26 Stepping 5
10:48:27:         CPUs: 8
10:48:27:       Memory: 17.99GiB
10:48:27:  Free Memory: 16.07GiB
10:48:27:      Threads: WINDOWS_THREADS
10:48:27:   OS Version: 6.1
10:48:27:  Has Battery: false
10:48:27:   On Battery: false
10:48:27:   UTC Offset: 2
10:48:27:          PID: 4672
10:48:27:          CWD: D:/$TEMP$/fah
10:48:27:           OS: Windows 7 Ultimate
10:48:27:      OS Arch: AMD64
10:48:27:         GPUs: 1
10:48:27:        GPU 0: NVIDIA:3 GK110 [GeForce GTX 780 Ti]
10:48:27:         CUDA: 3.5
10:48:27:  CUDA Driver: 7050
10:48:27:Win32 Service: false
10:48:27:***********************************************************************
10:48:27:<config>
10:48:27:  <!-- Folding Core -->
10:48:27:  <checkpoint v='10'/>
10:48:27:
10:48:27:  <!-- Folding Slot Configuration -->
10:48:27:  <extra-core-args v='-forceasm -verbose'/>
10:48:27:  <max-packet-size v='big'/>
10:48:27:
10:48:27:  <!-- HTTP Server -->
10:48:27:  <allow v='127.0.0.1 192.168.20.0/24'/>
10:48:27:
10:48:27:  <!-- Network -->
10:48:27:  <proxy v=':8080'/>
10:48:27:
10:48:27:  <!-- Remote Command Server -->
10:48:27:  <password v='*****'/>
10:48:27:
10:48:27:  <!-- Slot Control -->
10:48:27:  <power v='full'/>
10:48:27:
10:48:27:  <!-- User Information -->
10:48:27:  <passkey v='********************************'/>
10:48:27:  <user v='^w^ing'/>
10:48:27:
10:48:27:  <!-- Work Unit Control -->
10:48:27:  <next-unit-percentage v='100'/>
10:48:27:
10:48:27:  <!-- Folding Slots -->
10:48:27:  <slot id='0' type='GPU'>
10:48:27:    <client-type v='advanced'/>
10:48:27:    <core-priority v='low'/>
10:48:27:  </slot>
10:48:27:</config>
10:48:27:Trying to access database...
10:48:27:Successfully acquired database lock
10:48:27:Enabled folding slot 00: READY gpu:0:GK110 [GeForce GTX 780 Ti]
10:48:27:WU01:FS00:Starting
10:48:27:WU01:FS00:Running FahCore: C:\FoldingAtHome\FAHClient/FAHCoreWrapper.exe D:/$TEMP$/fah/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 4672 -checkpoint 10 -gpu 0 -gpu-vendor nvidia -forceasm -verbose
10:48:27:WU01:FS00:Started FahCore on PID 2212
10:48:27:WU01:FS00:Core PID:4592
10:48:27:WU01:FS00:FahCore 0x21 started
10:48:30:WU01:FS00:0x21:*********************** Log Started 2015-09-06T10:48:29Z ***********************
10:48:30:WU01:FS00:0x21:Project: 9712 (Run 6, Clone 0, Gen 14)
10:48:30:WU01:FS00:0x21:Unit: 0x00000044ab40416255b9a71b4787447e
10:48:30:WU01:FS00:0x21:CPU: 0x00000000000000000000000000000000
10:48:30:WU01:FS00:0x21:Machine: 0
10:48:30:WU01:FS00:0x21:Reading tar file core.xml
10:48:30:WU01:FS00:0x21:Reading tar file integrator.xml
10:48:30:WU01:FS00:0x21:Reading tar file system.xml
10:48:31:WU01:FS00:0x21:Reading tar file state.xml
10:48:32:WU01:FS00:0x21:Digital signatures verified
10:48:32:WU01:FS00:0x21:Folding@home GPU Core21 Folding@home Core
10:48:32:WU01:FS00:0x21:Version 0.0.11
10:49:32:WU01:FS00:0x21:Completed 0 out of 1280000 steps (0%)
10:49:32:WU01:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
10:51:24:WU01:FS00:0x21:Completed 12800 out of 1280000 steps (1%)
10:52:57:WU01:FS00:0x21:Completed 25600 out of 1280000 steps (2%)
10:54:31:WU01:FS00:0x21:Completed 38400 out of 1280000 steps (3%)
10:56:04:WU01:FS00:0x21:Completed 51200 out of 1280000 steps (4%)
10:59:44:WU01:FS00:0x21:WARNING:Console control signal 2 on PID 4592
10:59:44:WU01:FS00:0x21:Exiting, please wait. . .
10:59:44:WARNING:Console control signal 2 on PID 4672
10:59:44:Exiting, please wait. . .

consulting MSI Afterburner graphs on the second run, GPU usage oscillates between 90% and 99% when the WU works normally, but becomes constant 99% when it breaks.

running on windows 7 64bit, gtx 780 Ti, driver version 355.84

Post by **bruce** » Sun Sep 06, 2015 11:56 pm

Code: Select all

08:39:39:WU01:FS00:0x21:Completed 76800 out of 1280000 steps (6%)
08:53:00:WU01:FS00:0x21:Bad State detected... attempting to resume from last good checkpoint
08:54:34:WU01:FS00:0x21:Completed 12800 out of 1280000 steps (1%)
09:08:11:WARNING:Console control signal 2 on PID 4340
09:08:11:Exiting, please wait.

The "Bas State detected..." is a known problem. Development is working to determine the cause (perhaps overheating or overclocking the GPU) and hopefully prevent it from happening in the future. After such an error, FahCore_21 resumes work from the previous checkpoint (in this case before 1%) and very likely would have run to completion if you had not interrupted it.

Deleting the work you had done and again interrupting the WU, this time after 4%, does NOT show the same "Bad State" error.

Code: Select all

10:56:04:WU01:FS00:0x21:Completed 51200 out of 1280000 steps (4%)
10:59:44:WU01:FS00:0x21:WARNING:Console control signal 2 on PID 4592
10:59:44:WU01:FS00:0x21:Exiting, please wait. . .

The GPU cores, including FahCore_21, ignore the <checkpoint v='10'/> directive. The checkpoint interval is predefined by the researcher.

Projects for Core_21 are significantly larger than past projects and are expected to run longer than many other GPU projects, which is why this one doesn't expire until 10.80 days have elapsed.

davidcoton · Post by **davidcoton** » Mon Sep 07, 2015 12:02 am

Try reducing the clocks a bit, even if they are currently "stock". Try 50MHz off processor and 500MHz off memory to start. Take more off until it runs stable. Then if you want increase by small amounts to find the limit.

It worked for my 780Ti, YMMV. Mine was not actually failing with Bad State, but various other errors, so the solution may or may not be the same.

^w^ing · Post by **^w^ing** » Mon Sep 07, 2015 7:45 am

bruce wrote: Development is working to determine the cause (perhaps overheating or overclocking the GPU)

overheating shouldn't be a problem, my GPU temps didn't go over 45°C

bruce wrote:...After such an error, FahCore_21 resumes work from the previous checkpoint (in this case before 1%) and very likely would have run to completion if you had not interrupted it.

Are you suggesting that the variance in % time from a couple of minutes in the first few % to almost an hour in the last few % is normal/expected? I killed it on the second run because my PC became once again unresponsive after those 4% and I assumed it had broke again. Is then the fact that my PC becomes unusable while running core 21 WUs also expected? That would make it completely unacceptable for me to run such work.

bruce wrote: The GPU cores, including FahCore_21, ignore the <checkpoint v='10'/> directive. The checkpoint interval is predefined by the researcher.

yeah I've noticed that pretty much no core after the old gromacs single cpu cores care about this setting.

davidcoton wrote:Try reducing the clocks a bit, even if they are currently "stock". Try 50MHz off processor and 500MHz off memory to start. Take more off until it runs stable. Then if you want increase by small amounts to find the limit.

It worked for my 780Ti, YMMV. Mine was not actually failing with Bad State, but various other errors, so the solution may or may not be the same.

there seems to be two issues here - that it restarted from last checkpoint due to Bad State in the first run and that it made my PC unusable on both runs, if the second issue turns out to be expected and considered normal, then the first is somewhat moot since I won't be running any core 21 WUs on my machine. I might try underclocking for the sake of Bad State, since I'm curious if these two issues are related, thanks for the suggestion.

davidcoton · Post by **davidcoton** » Mon Sep 07, 2015 8:29 am

It is not yet clear to me whether the two forms of failure (and several others I found) are related. I am just trying to determine if downclocking is generally a quick fix, as it was for me. My card appeared stable with Cores 17 and 18, but went loopy on one particular project on core 21. Others are reporting the same two problems as you, though not enough on this project to prevent its wider distribution. PG are working on it -- pushing the science forward always risks breaking the experimental system, but determining how to fix it is not so easy. Any fixes we can find "out here" will be useful to them.

billford · Post by **billford** » Mon Sep 07, 2015 9:55 am

I've had an oddity with Core_21 on my 780 Ti (Linux, 346.35 drivers, overclocked from NVidia stock but underclocked from factory setting), not sure if it's related.

Normally it runs at ~70-73ºC, but occasionally it will shoot up to the low 80's, stay there for 10 minutes or so then drop back to normal. It may or may not do the same again later on, can't predict it. No bad state errors and no discernible effect on TPF, so now I ignore it. Despite being sure from past experience that it would fall over within seconds at those temperatures on most WUs. Odd

Not all WUs display the effect and it doesn't happen on the Maxwells.

Post by **bruce** » Mon Sep 07, 2015 9:09 pm

billford wrote:Normally it runs at ~70-73ºC, but occasionally it will shoot up to the low 80's, stay there for 10 minutes or so then drop back to normal. It may or may not do the same again later on, can't predict it. No bad state errors and no discernible effect on TPF, so now I ignore it. Despite being sure from past experience that it would fall over within seconds at those temperatures on most WUs. Odd

Not all WUs display the effect and it doesn't happen on the Maxwells.

Interesting, to say the least. Do those temperature changes correlate to some non-FAH activity, such as screen update frequency or to running some other program that might use the GPU?

billford · Post by **billford** » Mon Sep 07, 2015 9:43 pm

bruce wrote:Do those temperature changes correlate to some non-FAH activity, such as screen update frequency or to running some other program that might use the GPU?

No, it's a dedicated machine, nothing else on it, and the display is in sleep mode most of the time unless I'm updating the OS or something similar. On the odd occasion I've managed to get to the machine whilst it's happening (it's remotely monitored but in another part of the house so it's luck if I spot it in time to get to it and wake it up) nothing untoward seems to be going on.

It does seem as if something else is running though, is it possible that some part of the GPU that's not being used for folding is somehow being kicked into activity? I don't know enough about how GPUs or folding cores work to have a valid opinion!

I can't be sure of this bit because, as I implied, I don't take any notice of it now, but it does seem to be occurring less often as the trajectories proceed. Maybe some odd interaction with the configuration of the atoms? I'm making wild guesses here

Just had a thought... it's also running a cpu:3 client, if I see it happening again I'll pause the smp client just to check on any possible interaction. Long shot, but worth a try.

^w^ing · Post by **^w^ing** » Tue Sep 08, 2015 4:30 am

I did try underclocking the GPU and the WU finished without either an increase in % time of around 2 minutes or halting my PC to a crawl. I mean my PC did become noticeably less responsive running the WU (no other GPU project/core so far did this on my PC), which is pretty annoying but nowhere near how it was in the first two runs. Also no Bad State error, obviously. So perhaps it really is a stability issue, pretty strange symptoms though.

Post by **Joe_H** » Tue Sep 08, 2015 5:35 am

Does your GPU come with a factory overclock? I did not see a mention of the brand, just that the card was a 780ti. A number of cards come already overclocked, so reducing them to the nVidia reference clocks is a first step in troubleshooting GPU processing problems.

^w^ing · Post by **^w^ing** » Tue Sep 08, 2015 6:41 am

no it is an asus card with nvidia reference design, connected to a custom waterloop. I had my own overclock on it that I failed to consider since it was stable on everything for almost two years since I got it. should have realized that could be the issue since I know fah tends to find a way to break even seemingly stable overclocks, but since it didn't give me any issues since I overclocked it two years ago, I guess I stopped think about it as overclocked, my bad.

Post by **bruce** » Tue Sep 08, 2015 4:30 pm

Stanford does their best to use ALL of the resources they can. That means building projects that are tougher than any benchmark designed by somebody else leaving no margin for overclocking.

billford · Post by **billford** » Tue Sep 08, 2015 8:07 pm

bruce wrote: Interesting, to say the least. Do those temperature changes correlate to some non-FAH activity, such as screen update frequency or to running some other program that might use the GPU?

Just had another one, Project: 9704 (Run 30, Clone 15, Gen 15) if it matters.

Happened to spot it just as it started, it barely completed one frame before remote monitoring showed the temperature up to 83ºC.

So went to the machine, woke it up, nothing abnormal. System monitor showed nothing else running, GPU utilisation 98-99%, PCIe bus utilisation ~30-35%, fans at 70%, clock speed dropping slightly now and then (fans at max?). All as expected.

Paused the smp client- no effect, also as expected.

About 10 minutes (~5 frames) later it all went back to normal.

Thought of another thing to test- the GPU hadn't got to the first checkpoint so I paused/unpaused it to force it to start the WU again. The pattern repeated exactly... which would seem to suggest that it's something inherent in the WU rather than in my system, I think.

I'm not unduly concerned as it doesn't cause me any problems (and may get "accidentally" fixed in a future version of Core_21, which I believe is still undergoing some development?), but just possibly it might give someone a clue to the problems some people are having?

edit- it's now somewhat over halfway through, there have been a couple more temperature excursions but very brief- less than one frame.

Post by **bruce** » Tue Sep 08, 2015 10:38 pm

Option 1: The power is exceeding TDP.
Option 2: The fan or fan-speed regulation circuitry is failing.
Option 3: The cooling subsystem is unable to handle the heat generated by TDP.

You'll have to tell us which.

billford · Post by **billford** » Tue Sep 08, 2015 11:02 pm

Option 1: The power is exceeding TDP.

Possible... not sure how I can tell, there aren't any diagnostic utilities for Linux (that I'm aware of). Suggestions? (FWIW it's a 600W PSU)

Option 2: The fan or fan-speed regulation circuitry is failing.

Unlikely- that seems to reacting correctly to the temperature excursions.

Option 3: The cooling subsystem is unable to handle the heat generated by TDP.

See above, it's also handled ambient temperatures considerably higher than current without any bother.

You missed one:

Option 4: Core_21's are demanding too much from the GPU, maybe specifically from 780 Ti's. (This includes your Option 1 if I understand TDP correctly)

Sure, they can be downclocked to cope (although mine can't, it's at the maximum underclock already); this would benefit the Core_21 projects but would be to the detriment of others and, quite possibly, to the science as a whole.

If option 4 is the case, the science might be better served by barring 780 Ti's from getting Core_21 WUs.

Folding Forum

Cannot finish Core 21 WU

Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU

Re: Cannot finish Core 21 WU