GPU WU completes but doesn't

[lyndwyrm]

I just started folding recently, but I haven't been able to turn in any GPU WUs. I've looked through possible issues, but none of the one's I've seen seem to quite match what's happening for me. Under the status tab of client v7.4.4 it lists the progress as 99.99% complete with an ETA of 1.00/2.00 secs and then sits there for hours. In contrast, the log lists the WU as completing, going through saving result files and then stops with no error reported. GPU activity also stops once 100% has been reached. The related log for the WU currently with the issue is below

Code: Select all

*********************** Log Started 2015-12-08T23:18:07Z ***********************
23:32:54:WU00:FS01:Connecting to 171.67.108.45:80
23:32:55:WU00:FS01:Assigned to work server 140.163.4.235
23:32:55:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GM200 [GeForce GTX 980 Ti] from 140.163.4.235
23:32:55:WU00:FS01:Connecting to 140.163.4.235:8080
23:32:56:WU00:FS01:Downloading 4.23MiB
23:32:58:WU00:FS01:Download complete
23:32:58:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10472 run:0 clone:69 gen:264 core:0x18 unit:0x00000156538b3dbb53beb4f52046b226
23:32:58:WU00:FS01:Starting
23:32:58:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/$$$$/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 5188 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
23:32:58:WU00:FS01:Started FahCore on PID 1336
23:32:59:WU00:FS01:Core PID:10840
23:32:59:WU00:FS01:FahCore 0x18 started
23:32:59:WU00:FS01:0x18:*********************** Log Started 2015-12-08T23:32:59Z ***********************
23:32:59:WU00:FS01:0x18:Project: 10472 (Run 0, Clone 69, Gen 264)
23:32:59:WU00:FS01:0x18:Unit: 0x00000156538b3dbb53beb4f52046b226
23:32:59:WU00:FS01:0x18:CPU: 0x00000000000000000000000000000000
23:32:59:WU00:FS01:0x18:Machine: 1
23:32:59:WU00:FS01:0x18:Reading tar file state.xml
23:32:59:WU00:FS01:0x18:Reading tar file system.xml
23:33:00:WU00:FS01:0x18:Reading tar file integrator.xml
23:33:00:WU00:FS01:0x18:Reading tar file core.xml
23:33:00:WU00:FS01:0x18:Digital signatures verified
23:33:00:WU00:FS01:0x18:Folding@home GPU core18
23:33:00:WU00:FS01:0x18:Version 0.0.4
23:33:16:WU00:FS01:0x18:Completed 0 out of 5000000 steps (0%)
23:33:16:WU00:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
23:36:13:WU00:FS01:0x18:Completed 50000 out of 5000000 steps (1%)
23:39:03:WU00:FS01:0x18:Completed 100000 out of 5000000 steps (2%)
23:42:01:WU00:FS01:0x18:Completed 150000 out of 5000000 steps (3%)
23:44:55:WU00:FS01:0x18:Completed 200000 out of 5000000 steps (4%)
23:47:49:WU00:FS01:0x18:Completed 250000 out of 5000000 steps (5%)
23:50:49:WU00:FS01:0x18:Completed 300000 out of 5000000 steps (6%)
23:53:42:WU00:FS01:0x18:Completed 350000 out of 5000000 steps (7%)
23:56:42:WU00:FS01:0x18:Completed 400000 out of 5000000 steps (8%)
23:59:33:WU00:FS01:0x18:Completed 450000 out of 5000000 steps (9%)
00:02:22:WU00:FS01:0x18:Completed 500000 out of 5000000 steps (10%)
00:05:17:WU00:FS01:0x18:Completed 550000 out of 5000000 steps (11%)
00:08:06:WU00:FS01:0x18:Completed 600000 out of 5000000 steps (12%)
00:11:02:WU00:FS01:0x18:Completed 650000 out of 5000000 steps (13%)
00:13:51:WU00:FS01:0x18:Completed 700000 out of 5000000 steps (14%)
00:16:40:WU00:FS01:0x18:Completed 750000 out of 5000000 steps (15%)
00:19:36:WU00:FS01:0x18:Completed 800000 out of 5000000 steps (16%)
00:22:26:WU00:FS01:0x18:Completed 850000 out of 5000000 steps (17%)
00:25:23:WU00:FS01:0x18:Completed 900000 out of 5000000 steps (18%)
00:28:15:WU00:FS01:0x18:Completed 950000 out of 5000000 steps (19%)
00:31:06:WU00:FS01:0x18:Completed 1000000 out of 5000000 steps (20%)
00:34:04:WU00:FS01:0x18:Completed 1050000 out of 5000000 steps (21%)
00:36:54:WU00:FS01:0x18:Completed 1100000 out of 5000000 steps (22%)
00:39:51:WU00:FS01:0x18:Completed 1150000 out of 5000000 steps (23%)
00:42:41:WU00:FS01:0x18:Completed 1200000 out of 5000000 steps (24%)
00:45:31:WU00:FS01:0x18:Completed 1250000 out of 5000000 steps (25%)
00:48:29:WU00:FS01:0x18:Completed 1300000 out of 5000000 steps (26%)
00:51:19:WU00:FS01:0x18:Completed 1350000 out of 5000000 steps (27%)
00:54:16:WU00:FS01:0x18:Completed 1400000 out of 5000000 steps (28%)
00:57:07:WU00:FS01:0x18:Completed 1450000 out of 5000000 steps (29%)
00:59:56:WU00:FS01:0x18:Completed 1500000 out of 5000000 steps (30%)
01:02:54:WU00:FS01:0x18:Completed 1550000 out of 5000000 steps (31%)
01:05:44:WU00:FS01:0x18:Completed 1600000 out of 5000000 steps (32%)
01:08:42:WU00:FS01:0x18:Completed 1650000 out of 5000000 steps (33%)
01:11:32:WU00:FS01:0x18:Completed 1700000 out of 5000000 steps (34%)
01:14:23:WU00:FS01:0x18:Completed 1750000 out of 5000000 steps (35%)
01:17:22:WU00:FS01:0x18:Completed 1800000 out of 5000000 steps (36%)
01:20:19:WU00:FS01:0x18:Completed 1850000 out of 5000000 steps (37%)
01:23:22:WU00:FS01:0x18:Completed 1900000 out of 5000000 steps (38%)
01:26:18:WU00:FS01:0x18:Completed 1950000 out of 5000000 steps (39%)
01:29:13:WU00:FS01:0x18:Completed 2000000 out of 5000000 steps (40%)
01:32:16:WU00:FS01:0x18:Completed 2050000 out of 5000000 steps (41%)
01:35:11:WU00:FS01:0x18:Completed 2100000 out of 5000000 steps (42%)
01:38:15:WU00:FS01:0x18:Completed 2150000 out of 5000000 steps (43%)
01:41:11:WU00:FS01:0x18:Completed 2200000 out of 5000000 steps (44%)
01:44:07:WU00:FS01:0x18:Completed 2250000 out of 5000000 steps (45%)
01:47:09:WU00:FS01:0x18:Completed 2300000 out of 5000000 steps (46%)
01:50:00:WU00:FS01:0x18:Completed 2350000 out of 5000000 steps (47%)
01:53:01:WU00:FS01:0x18:Completed 2400000 out of 5000000 steps (48%)
01:55:52:WU00:FS01:0x18:Completed 2450000 out of 5000000 steps (49%)
01:58:45:WU00:FS01:0x18:Completed 2500000 out of 5000000 steps (50%)
02:01:45:WU00:FS01:0x18:Completed 2550000 out of 5000000 steps (51%)
02:04:38:WU00:FS01:0x18:Completed 2600000 out of 5000000 steps (52%)
02:07:39:WU00:FS01:0x18:Completed 2650000 out of 5000000 steps (53%)
02:10:33:WU00:FS01:0x18:Completed 2700000 out of 5000000 steps (54%)
02:13:27:WU00:FS01:0x18:Completed 2750000 out of 5000000 steps (55%)
02:16:29:WU00:FS01:0x18:Completed 2800000 out of 5000000 steps (56%)
02:19:22:WU00:FS01:0x18:Completed 2850000 out of 5000000 steps (57%)
02:22:22:WU00:FS01:0x18:Completed 2900000 out of 5000000 steps (58%)
02:25:11:WU00:FS01:0x18:Completed 2950000 out of 5000000 steps (59%)
02:28:01:WU00:FS01:0x18:Completed 3000000 out of 5000000 steps (60%)
02:30:58:WU00:FS01:0x18:Completed 3050000 out of 5000000 steps (61%)
02:33:50:WU00:FS01:0x18:Completed 3100000 out of 5000000 steps (62%)
02:36:52:WU00:FS01:0x18:Completed 3150000 out of 5000000 steps (63%)
02:39:46:WU00:FS01:0x18:Completed 3200000 out of 5000000 steps (64%)
02:42:40:WU00:FS01:0x18:Completed 3250000 out of 5000000 steps (65%)
02:45:41:WU00:FS01:0x18:Completed 3300000 out of 5000000 steps (66%)
02:48:36:WU00:FS01:0x18:Completed 3350000 out of 5000000 steps (67%)
02:51:41:WU00:FS01:0x18:Completed 3400000 out of 5000000 steps (68%)
02:54:33:WU00:FS01:0x18:Completed 3450000 out of 5000000 steps (69%)
02:57:28:WU00:FS01:0x18:Completed 3500000 out of 5000000 steps (70%)
03:00:33:WU00:FS01:0x18:Completed 3550000 out of 5000000 steps (71%)
03:03:23:WU00:FS01:0x18:Completed 3600000 out of 5000000 steps (72%)
03:06:22:WU00:FS01:0x18:Completed 3650000 out of 5000000 steps (73%)
03:08:10:WU00:FS01:0x18:WARNING:Console control signal 1 on PID 10840
03:08:10:WU00:FS01:0x18:Exiting, please wait. . .
03:08:10:WU00:FS01:0x18:Lost lifeline PID 1336, exiting
03:08:10:WU00:FS01:0x18:ERROR:103: Lost client lifeline
03:08:10:WU00:FS01:0x18:Folding@home Core Shutdown: CLIENT_DIED
03:08:11:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
03:10:13:WU00:FS01:Starting
03:10:13:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/$$$$/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 5188 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
03:10:13:WU00:FS01:Started FahCore on PID 9652
03:10:13:WU00:FS01:Core PID:11340
03:10:13:WU00:FS01:FahCore 0x18 started
03:10:14:WU00:FS01:0x18:*********************** Log Started 2015-12-09T03:10:13Z ***********************
03:10:14:WU00:FS01:0x18:Project: 10472 (Run 0, Clone 69, Gen 264)
03:10:14:WU00:FS01:0x18:Unit: 0x00000156538b3dbb53beb4f52046b226
03:10:14:WU00:FS01:0x18:CPU: 0x00000000000000000000000000000000
03:10:14:WU00:FS01:0x18:Machine: 1
03:10:14:WU00:FS01:0x18:Digital signatures verified
03:10:14:WU00:FS01:0x18:Folding@home GPU core18
03:10:14:WU00:FS01:0x18:Version 0.0.4
03:10:14:WU00:FS01:0x18:  Found a checkpoint file
03:10:28:WU00:FS01:0x18:Completed 3625000 out of 5000000 steps (72%)
03:10:28:WU00:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
03:12:00:WU00:FS01:0x18:Completed 3650000 out of 5000000 steps (73%)
03:14:50:WU00:FS01:0x18:Completed 3700000 out of 5000000 steps (74%)
03:17:39:WU00:FS01:0x18:Completed 3750000 out of 5000000 steps (75%)
03:20:36:WU00:FS01:0x18:Completed 3800000 out of 5000000 steps (76%)
03:23:26:WU00:FS01:0x18:Completed 3850000 out of 5000000 steps (77%)
03:26:22:WU00:FS01:0x18:Completed 3900000 out of 5000000 steps (78%)
03:29:12:WU00:FS01:0x18:Completed 3950000 out of 5000000 steps (79%)
03:32:02:WU00:FS01:0x18:Completed 4000000 out of 5000000 steps (80%)
03:35:00:WU00:FS01:0x18:Completed 4050000 out of 5000000 steps (81%)
03:37:50:WU00:FS01:0x18:Completed 4100000 out of 5000000 steps (82%)
03:40:47:WU00:FS01:0x18:Completed 4150000 out of 5000000 steps (83%)
03:43:36:WU00:FS01:0x18:Completed 4200000 out of 5000000 steps (84%)
03:46:26:WU00:FS01:0x18:Completed 4250000 out of 5000000 steps (85%)
03:49:23:WU00:FS01:0x18:Completed 4300000 out of 5000000 steps (86%)
03:52:14:WU00:FS01:0x18:Completed 4350000 out of 5000000 steps (87%)
03:55:12:WU00:FS01:0x18:Completed 4400000 out of 5000000 steps (88%)
03:58:04:WU00:FS01:0x18:Completed 4450000 out of 5000000 steps (89%)
04:00:55:WU00:FS01:0x18:Completed 4500000 out of 5000000 steps (90%)
04:03:52:WU00:FS01:0x18:Completed 4550000 out of 5000000 steps (91%)
04:06:42:WU00:FS01:0x18:Completed 4600000 out of 5000000 steps (92%)
04:09:39:WU00:FS01:0x18:Completed 4650000 out of 5000000 steps (93%)
04:12:28:WU00:FS01:0x18:Completed 4700000 out of 5000000 steps (94%)
04:15:18:WU00:FS01:0x18:Completed 4750000 out of 5000000 steps (95%)
04:18:16:WU00:FS01:0x18:Completed 4800000 out of 5000000 steps (96%)
04:21:06:WU00:FS01:0x18:Completed 4850000 out of 5000000 steps (97%)
04:24:02:WU00:FS01:0x18:Completed 4900000 out of 5000000 steps (98%)
04:26:52:WU00:FS01:0x18:Completed 4950000 out of 5000000 steps (99%)
04:29:41:WU00:FS01:0x18:Completed 5000000 out of 5000000 steps (100%)
04:29:49:WU00:FS01:0x18:Saving result file logfile_01.txt
04:29:49:WU00:FS01:0x18:Saving result file checkpointState.xml
04:29:51:WU00:FS01:0x18:Saving result file checkpt.crc
04:29:51:WU00:FS01:0x18:Saving result file log.txt
04:29:51:WU00:FS01:0x18:Saving result file positions.xtc
******************************* Date: 2015-12-09 *******************************

Further, here's the start of the log (was going to post the whole thing, but ran out of characters )

Code: Select all

*********************** Log Started 2015-12-08T23:18:07Z ***********************
23:18:07:************************* Folding@home Client *************************
23:18:07:      Website: http://folding.stanford.edu/
23:18:07:    Copyright: (c) 2009-2014 Stanford University
23:18:07:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
23:18:07:         Args: 
23:18:07:       Config: C:/Users/$$$$/AppData/Roaming/FAHClient/config.xml
23:18:07:******************************** Build ********************************
23:18:07:      Version: 7.4.4
23:18:07:         Date: Mar 4 2014
23:18:07:         Time: 20:26:54
23:18:07:      SVN Rev: 4130
23:18:07:       Branch: fah/trunk/client
23:18:07:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
23:18:07:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
23:18:07:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
23:18:07:     Platform: win32 XP
23:18:07:         Bits: 32
23:18:07:         Mode: Release
23:18:07:******************************* System ********************************
23:18:07:          CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
23:18:07:       CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
23:18:07:         CPUs: 8
23:18:07:       Memory: 31.92GiB
23:18:07:  Free Memory: 28.70GiB
23:18:07:      Threads: WINDOWS_THREADS
23:18:07:   OS Version: 6.1
23:18:07:  Has Battery: false
23:18:07:   On Battery: false
23:18:07:   UTC Offset: -5
23:18:07:          PID: 5188
23:18:07:          CWD: C:/Users/$$$$AppData/Roaming/FAHClient
23:18:07:           OS: Windows 7 Ultimate
23:18:07:      OS Arch: AMD64
23:18:07:         GPUs: 1
23:18:07:        GPU 0: NVIDIA:5 GM200 [GeForce GTX 980 Ti]
23:18:07:         CUDA: 5.2
23:18:07:  CUDA Driver: 7050
23:18:07:Win32 Service: false
23:18:07:***********************************************************************

I've tried removing the GPU slot, restarting the computer, and re-adding it later, but the same issue is happening with a different assigned WU. I believe I have the most recent CUDA drivers, and graphics driver 359.06 installed. The 980 Ti is factory overclocked to 1102Mhz, about 10%. My CPU is not overclocked, but my RAM is to 3000MHz, which is within manufacturer specifications. Stability isn't my suspicion, but I've only ever tested the CPU stability.

Any suggestions?

[toTOW]

You cut your first log extract too early ... the WU completed fine and is writing results, but we don't see what happened next (it should be exiting the core and uploading results).

[lyndwyrm]

Well, The first log isn't really cut. I've only filtered the log to Unit 00. Right now the same log reads:

Code: Select all

*********************** Log Started 2015-12-08T23:18:07Z ***********************
23:32:54:WU00:FS01:Connecting to 171.67.108.45:80
23:32:55:WU00:FS01:Assigned to work server 140.163.4.235
23:32:55:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GM200 [GeForce GTX 980 Ti] from 140.163.4.235
23:32:55:WU00:FS01:Connecting to 140.163.4.235:8080
23:32:56:WU00:FS01:Downloading 4.23MiB
23:32:58:WU00:FS01:Download complete
23:32:58:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10472 run:0 clone:69 gen:264 core:0x18 unit:0x00000156538b3dbb53beb4f52046b226
23:32:58:WU00:FS01:Starting
23:32:58:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/$$$$/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 5188 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
23:32:58:WU00:FS01:Started FahCore on PID 1336
23:32:59:WU00:FS01:Core PID:10840
23:32:59:WU00:FS01:FahCore 0x18 started
23:32:59:WU00:FS01:0x18:*********************** Log Started 2015-12-08T23:32:59Z ***********************
23:32:59:WU00:FS01:0x18:Project: 10472 (Run 0, Clone 69, Gen 264)
23:32:59:WU00:FS01:0x18:Unit: 0x00000156538b3dbb53beb4f52046b226
23:32:59:WU00:FS01:0x18:CPU: 0x00000000000000000000000000000000
23:32:59:WU00:FS01:0x18:Machine: 1
23:32:59:WU00:FS01:0x18:Reading tar file state.xml
23:32:59:WU00:FS01:0x18:Reading tar file system.xml
23:33:00:WU00:FS01:0x18:Reading tar file integrator.xml
23:33:00:WU00:FS01:0x18:Reading tar file core.xml
23:33:00:WU00:FS01:0x18:Digital signatures verified
23:33:00:WU00:FS01:0x18:Folding@home GPU core18
23:33:00:WU00:FS01:0x18:Version 0.0.4
23:33:16:WU00:FS01:0x18:Completed 0 out of 5000000 steps (0%)
23:33:16:WU00:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
23:36:13:WU00:FS01:0x18:Completed 50000 out of 5000000 steps (1%)
23:39:03:WU00:FS01:0x18:Completed 100000 out of 5000000 steps (2%)
23:42:01:WU00:FS01:0x18:Completed 150000 out of 5000000 steps (3%)
23:44:55:WU00:FS01:0x18:Completed 200000 out of 5000000 steps (4%)
23:47:49:WU00:FS01:0x18:Completed 250000 out of 5000000 steps (5%)
23:50:49:WU00:FS01:0x18:Completed 300000 out of 5000000 steps (6%)
23:53:42:WU00:FS01:0x18:Completed 350000 out of 5000000 steps (7%)
23:56:42:WU00:FS01:0x18:Completed 400000 out of 5000000 steps (8%)
23:59:33:WU00:FS01:0x18:Completed 450000 out of 5000000 steps (9%)
00:02:22:WU00:FS01:0x18:Completed 500000 out of 5000000 steps (10%)
00:05:17:WU00:FS01:0x18:Completed 550000 out of 5000000 steps (11%)
00:08:06:WU00:FS01:0x18:Completed 600000 out of 5000000 steps (12%)
00:11:02:WU00:FS01:0x18:Completed 650000 out of 5000000 steps (13%)
00:13:51:WU00:FS01:0x18:Completed 700000 out of 5000000 steps (14%)
00:16:40:WU00:FS01:0x18:Completed 750000 out of 5000000 steps (15%)
00:19:36:WU00:FS01:0x18:Completed 800000 out of 5000000 steps (16%)
00:22:26:WU00:FS01:0x18:Completed 850000 out of 5000000 steps (17%)
00:25:23:WU00:FS01:0x18:Completed 900000 out of 5000000 steps (18%)
00:28:15:WU00:FS01:0x18:Completed 950000 out of 5000000 steps (19%)
00:31:06:WU00:FS01:0x18:Completed 1000000 out of 5000000 steps (20%)
00:34:04:WU00:FS01:0x18:Completed 1050000 out of 5000000 steps (21%)
00:36:54:WU00:FS01:0x18:Completed 1100000 out of 5000000 steps (22%)
00:39:51:WU00:FS01:0x18:Completed 1150000 out of 5000000 steps (23%)
00:42:41:WU00:FS01:0x18:Completed 1200000 out of 5000000 steps (24%)
00:45:31:WU00:FS01:0x18:Completed 1250000 out of 5000000 steps (25%)
00:48:29:WU00:FS01:0x18:Completed 1300000 out of 5000000 steps (26%)
00:51:19:WU00:FS01:0x18:Completed 1350000 out of 5000000 steps (27%)
00:54:16:WU00:FS01:0x18:Completed 1400000 out of 5000000 steps (28%)
00:57:07:WU00:FS01:0x18:Completed 1450000 out of 5000000 steps (29%)
00:59:56:WU00:FS01:0x18:Completed 1500000 out of 5000000 steps (30%)
01:02:54:WU00:FS01:0x18:Completed 1550000 out of 5000000 steps (31%)
01:05:44:WU00:FS01:0x18:Completed 1600000 out of 5000000 steps (32%)
01:08:42:WU00:FS01:0x18:Completed 1650000 out of 5000000 steps (33%)
01:11:32:WU00:FS01:0x18:Completed 1700000 out of 5000000 steps (34%)
01:14:23:WU00:FS01:0x18:Completed 1750000 out of 5000000 steps (35%)
01:17:22:WU00:FS01:0x18:Completed 1800000 out of 5000000 steps (36%)
01:20:19:WU00:FS01:0x18:Completed 1850000 out of 5000000 steps (37%)
01:23:22:WU00:FS01:0x18:Completed 1900000 out of 5000000 steps (38%)
01:26:18:WU00:FS01:0x18:Completed 1950000 out of 5000000 steps (39%)
01:29:13:WU00:FS01:0x18:Completed 2000000 out of 5000000 steps (40%)
01:32:16:WU00:FS01:0x18:Completed 2050000 out of 5000000 steps (41%)
01:35:11:WU00:FS01:0x18:Completed 2100000 out of 5000000 steps (42%)
01:38:15:WU00:FS01:0x18:Completed 2150000 out of 5000000 steps (43%)
01:41:11:WU00:FS01:0x18:Completed 2200000 out of 5000000 steps (44%)
01:44:07:WU00:FS01:0x18:Completed 2250000 out of 5000000 steps (45%)
01:47:09:WU00:FS01:0x18:Completed 2300000 out of 5000000 steps (46%)
01:50:00:WU00:FS01:0x18:Completed 2350000 out of 5000000 steps (47%)
01:53:01:WU00:FS01:0x18:Completed 2400000 out of 5000000 steps (48%)
01:55:52:WU00:FS01:0x18:Completed 2450000 out of 5000000 steps (49%)
01:58:45:WU00:FS01:0x18:Completed 2500000 out of 5000000 steps (50%)
02:01:45:WU00:FS01:0x18:Completed 2550000 out of 5000000 steps (51%)
02:04:38:WU00:FS01:0x18:Completed 2600000 out of 5000000 steps (52%)
02:07:39:WU00:FS01:0x18:Completed 2650000 out of 5000000 steps (53%)
02:10:33:WU00:FS01:0x18:Completed 2700000 out of 5000000 steps (54%)
02:13:27:WU00:FS01:0x18:Completed 2750000 out of 5000000 steps (55%)
02:16:29:WU00:FS01:0x18:Completed 2800000 out of 5000000 steps (56%)
02:19:22:WU00:FS01:0x18:Completed 2850000 out of 5000000 steps (57%)
02:22:22:WU00:FS01:0x18:Completed 2900000 out of 5000000 steps (58%)
02:25:11:WU00:FS01:0x18:Completed 2950000 out of 5000000 steps (59%)
02:28:01:WU00:FS01:0x18:Completed 3000000 out of 5000000 steps (60%)
02:30:58:WU00:FS01:0x18:Completed 3050000 out of 5000000 steps (61%)
02:33:50:WU00:FS01:0x18:Completed 3100000 out of 5000000 steps (62%)
02:36:52:WU00:FS01:0x18:Completed 3150000 out of 5000000 steps (63%)
02:39:46:WU00:FS01:0x18:Completed 3200000 out of 5000000 steps (64%)
02:42:40:WU00:FS01:0x18:Completed 3250000 out of 5000000 steps (65%)
02:45:41:WU00:FS01:0x18:Completed 3300000 out of 5000000 steps (66%)
02:48:36:WU00:FS01:0x18:Completed 3350000 out of 5000000 steps (67%)
02:51:41:WU00:FS01:0x18:Completed 3400000 out of 5000000 steps (68%)
02:54:33:WU00:FS01:0x18:Completed 3450000 out of 5000000 steps (69%)
02:57:28:WU00:FS01:0x18:Completed 3500000 out of 5000000 steps (70%)
03:00:33:WU00:FS01:0x18:Completed 3550000 out of 5000000 steps (71%)
03:03:23:WU00:FS01:0x18:Completed 3600000 out of 5000000 steps (72%)
03:06:22:WU00:FS01:0x18:Completed 3650000 out of 5000000 steps (73%)
03:08:10:WU00:FS01:0x18:WARNING:Console control signal 1 on PID 10840
03:08:10:WU00:FS01:0x18:Exiting, please wait. . .
03:08:10:WU00:FS01:0x18:Lost lifeline PID 1336, exiting
03:08:10:WU00:FS01:0x18:ERROR:103: Lost client lifeline
03:08:10:WU00:FS01:0x18:Folding@home Core Shutdown: CLIENT_DIED
03:08:11:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
03:10:13:WU00:FS01:Starting
03:10:13:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/$$$$/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 5188 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
03:10:13:WU00:FS01:Started FahCore on PID 9652
03:10:13:WU00:FS01:Core PID:11340
03:10:13:WU00:FS01:FahCore 0x18 started
03:10:14:WU00:FS01:0x18:*********************** Log Started 2015-12-09T03:10:13Z ***********************
03:10:14:WU00:FS01:0x18:Project: 10472 (Run 0, Clone 69, Gen 264)
03:10:14:WU00:FS01:0x18:Unit: 0x00000156538b3dbb53beb4f52046b226
03:10:14:WU00:FS01:0x18:CPU: 0x00000000000000000000000000000000
03:10:14:WU00:FS01:0x18:Machine: 1
03:10:14:WU00:FS01:0x18:Digital signatures verified
03:10:14:WU00:FS01:0x18:Folding@home GPU core18
03:10:14:WU00:FS01:0x18:Version 0.0.4
03:10:14:WU00:FS01:0x18:  Found a checkpoint file
03:10:28:WU00:FS01:0x18:Completed 3625000 out of 5000000 steps (72%)
03:10:28:WU00:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
03:12:00:WU00:FS01:0x18:Completed 3650000 out of 5000000 steps (73%)
03:14:50:WU00:FS01:0x18:Completed 3700000 out of 5000000 steps (74%)
03:17:39:WU00:FS01:0x18:Completed 3750000 out of 5000000 steps (75%)
03:20:36:WU00:FS01:0x18:Completed 3800000 out of 5000000 steps (76%)
03:23:26:WU00:FS01:0x18:Completed 3850000 out of 5000000 steps (77%)
03:26:22:WU00:FS01:0x18:Completed 3900000 out of 5000000 steps (78%)
03:29:12:WU00:FS01:0x18:Completed 3950000 out of 5000000 steps (79%)
03:32:02:WU00:FS01:0x18:Completed 4000000 out of 5000000 steps (80%)
03:35:00:WU00:FS01:0x18:Completed 4050000 out of 5000000 steps (81%)
03:37:50:WU00:FS01:0x18:Completed 4100000 out of 5000000 steps (82%)
03:40:47:WU00:FS01:0x18:Completed 4150000 out of 5000000 steps (83%)
03:43:36:WU00:FS01:0x18:Completed 4200000 out of 5000000 steps (84%)
03:46:26:WU00:FS01:0x18:Completed 4250000 out of 5000000 steps (85%)
03:49:23:WU00:FS01:0x18:Completed 4300000 out of 5000000 steps (86%)
03:52:14:WU00:FS01:0x18:Completed 4350000 out of 5000000 steps (87%)
03:55:12:WU00:FS01:0x18:Completed 4400000 out of 5000000 steps (88%)
03:58:04:WU00:FS01:0x18:Completed 4450000 out of 5000000 steps (89%)
04:00:55:WU00:FS01:0x18:Completed 4500000 out of 5000000 steps (90%)
04:03:52:WU00:FS01:0x18:Completed 4550000 out of 5000000 steps (91%)
04:06:42:WU00:FS01:0x18:Completed 4600000 out of 5000000 steps (92%)
04:09:39:WU00:FS01:0x18:Completed 4650000 out of 5000000 steps (93%)
04:12:28:WU00:FS01:0x18:Completed 4700000 out of 5000000 steps (94%)
04:15:18:WU00:FS01:0x18:Completed 4750000 out of 5000000 steps (95%)
04:18:16:WU00:FS01:0x18:Completed 4800000 out of 5000000 steps (96%)
04:21:06:WU00:FS01:0x18:Completed 4850000 out of 5000000 steps (97%)
04:24:02:WU00:FS01:0x18:Completed 4900000 out of 5000000 steps (98%)
04:26:52:WU00:FS01:0x18:Completed 4950000 out of 5000000 steps (99%)
04:29:41:WU00:FS01:0x18:Completed 5000000 out of 5000000 steps (100%)
04:29:49:WU00:FS01:0x18:Saving result file logfile_01.txt
04:29:49:WU00:FS01:0x18:Saving result file checkpointState.xml
04:29:51:WU00:FS01:0x18:Saving result file checkpt.crc
04:29:51:WU00:FS01:0x18:Saving result file log.txt
04:29:51:WU00:FS01:0x18:Saving result file positions.xtc
******************************* Date: 2015-12-09 *******************************
******************************* Date: 2015-12-09 *******************************
******************************* Date: 2015-12-09 *******************************
******************************* Date: 2015-12-09 *******************************

While I have little idea why the date stamp has been added 3 more times since I last checked (it is the number of WUs I have right now, one from the CPU is having trouble uploading, one currently on the CPU, this one, and one more downloaded and ready for the GPU), it's been sitting at the positions.xtc step for nearing 24 hours at this point. Is there something that would prevent it from exiting without throwing an error? Are there errors that wouldn't show up if I filter by unit or by the GPU's slot, ie. should I look elsewhere for an error? I can't imagine it would actually take this long to save a file.

My CPU on the other hand has still been completing WUs and has completed about 10 more since last night.

[bruce]

The extra date stamp are a bad sign. They happen on a regular basis, whether anything else is happening or not.

I'm not sure what's causing your problem, but from what you posted, there is something wrong. The final message: Saving result file positions.xtc is commonly found after a WU reaches 100% and the messages that SHOULD follow that message are normally logged within a very short period of time. In your case, it's taking a VERY long time to finish writing that file (It's hung and nothing is being accomplished.) Also, the likelihood of the files being incomplete or corrupt is unfortunately high.

I recommend you do whatever you can to identify a potential problem with your disk, though that may not show anything. Also, I'd look in the Windows Event Log for a report of something happening around 04:29:51 UTC which would be 11:29 PM in your time zone.

In any case, I'd reboot in hopes of clearing the problem and see if you can complete the next GPU assignment. (You may want to do a FINISH and let the current CPU assignment upload.)

[lyndwyrm]

Alright, so I'll let the CPU finish what I've got right now and then reboot. Would it be of any use to increase the log verbosity? In the case it is the disk, can I change where work units are saved?

On issues with the disk and the Event Log, S.M.A.R.T. doesn't show anything wrong and error checking the disk didn't return any problems. Nothing that I would expect to be relevant shows up in the Event Viewer near that time, just some information level events about Chrome and a Windows diagnostic module.

[bruce]

Do not increase the client log verbosity. It adds details that are not useful.

To move FAH's data directory, I'd recommend a full custom reinstall of the client. That location is used many places in the installation.

[toTOW]

Are the $$$$ in your install location for real, or did you replace username ?

The core might be messing them with some kind of batch variable definition which does not exist, and doesn't work ...

[lyndwyrm]

I replaced my real name with $$$$ when I copied the log into the post. I don't think "$" is normally allowed in file names.

[lyndwyrm]

So I rebooted my computer this morning and have let it run all day, so there shouldn't have been any interruptions. It picked up the same work unit from yesterday, and is still having the same problem.

Code: Select all

*********************** Log Started 2015-12-10T12:43:31Z ***********************
12:44:51:FS01:Unpaused
12:44:51:WU00:FS01:Starting
12:44:51:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/$$$$/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 00 -suffix 01 -version 704 -lifeline 6832 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
12:44:53:WU00:FS01:Started FahCore on PID 7252
12:44:55:WU00:FS01:Core PID:9444
12:44:55:WU00:FS01:FahCore 0x18 started
12:44:55:WU00:FS01:0x18:*********************** Log Started 2015-12-10T12:44:55Z ***********************
12:44:55:WU00:FS01:0x18:Project: 10472 (Run 0, Clone 69, Gen 264)
12:44:55:WU00:FS01:0x18:Unit: 0x00000156538b3dbb53beb4f52046b226
12:44:55:WU00:FS01:0x18:CPU: 0x00000000000000000000000000000000
12:44:55:WU00:FS01:0x18:Machine: 1
12:44:55:WU00:FS01:0x18:Reading tar file state.xml
12:44:56:WU00:FS01:0x18:Reading tar file system.xml
12:44:56:WU00:FS01:0x18:Reading tar file integrator.xml
12:44:56:WU00:FS01:0x18:Reading tar file core.xml
12:44:56:WU00:FS01:0x18:Digital signatures verified
12:44:56:WU00:FS01:0x18:Folding@home GPU core18
12:44:56:WU00:FS01:0x18:Version 0.0.4
12:45:09:WU00:FS01:0x18:Completed 0 out of 5000000 steps (0%)
12:45:09:WU00:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
12:48:03:WU00:FS01:0x18:Completed 50000 out of 5000000 steps (1%)
12:50:53:WU00:FS01:0x18:Completed 100000 out of 5000000 steps (2%)
12:53:48:WU00:FS01:0x18:Completed 150000 out of 5000000 steps (3%)
12:56:36:WU00:FS01:0x18:Completed 200000 out of 5000000 steps (4%)
12:59:24:WU00:FS01:0x18:Completed 250000 out of 5000000 steps (5%)
13:02:17:WU00:FS01:0x18:Completed 300000 out of 5000000 steps (6%)
13:05:04:WU00:FS01:0x18:Completed 350000 out of 5000000 steps (7%)
13:07:59:WU00:FS01:0x18:Completed 400000 out of 5000000 steps (8%)
13:10:46:WU00:FS01:0x18:Completed 450000 out of 5000000 steps (9%)
13:13:33:WU00:FS01:0x18:Completed 500000 out of 5000000 steps (10%)
13:16:25:WU00:FS01:0x18:Completed 550000 out of 5000000 steps (11%)
13:19:12:WU00:FS01:0x18:Completed 600000 out of 5000000 steps (12%)
13:22:08:WU00:FS01:0x18:Completed 650000 out of 5000000 steps (13%)
13:25:01:WU00:FS01:0x18:Completed 700000 out of 5000000 steps (14%)
13:27:48:WU00:FS01:0x18:Completed 750000 out of 5000000 steps (15%)
13:30:39:WU00:FS01:0x18:Completed 800000 out of 5000000 steps (16%)
13:33:25:WU00:FS01:0x18:Completed 850000 out of 5000000 steps (17%)
13:36:17:WU00:FS01:0x18:Completed 900000 out of 5000000 steps (18%)
13:39:03:WU00:FS01:0x18:Completed 950000 out of 5000000 steps (19%)
13:41:50:WU00:FS01:0x18:Completed 1000000 out of 5000000 steps (20%)
13:44:41:WU00:FS01:0x18:Completed 1050000 out of 5000000 steps (21%)
13:47:28:WU00:FS01:0x18:Completed 1100000 out of 5000000 steps (22%)
13:50:19:WU00:FS01:0x18:Completed 1150000 out of 5000000 steps (23%)
13:53:05:WU00:FS01:0x18:Completed 1200000 out of 5000000 steps (24%)
13:55:52:WU00:FS01:0x18:Completed 1250000 out of 5000000 steps (25%)
13:58:43:WU00:FS01:0x18:Completed 1300000 out of 5000000 steps (26%)
14:01:29:WU00:FS01:0x18:Completed 1350000 out of 5000000 steps (27%)
14:04:20:WU00:FS01:0x18:Completed 1400000 out of 5000000 steps (28%)
14:07:06:WU00:FS01:0x18:Completed 1450000 out of 5000000 steps (29%)
14:09:51:WU00:FS01:0x18:Completed 1500000 out of 5000000 steps (30%)
14:12:42:WU00:FS01:0x18:Completed 1550000 out of 5000000 steps (31%)
14:15:28:WU00:FS01:0x18:Completed 1600000 out of 5000000 steps (32%)
14:18:19:WU00:FS01:0x18:Completed 1650000 out of 5000000 steps (33%)
14:21:05:WU00:FS01:0x18:Completed 1700000 out of 5000000 steps (34%)
14:23:51:WU00:FS01:0x18:Completed 1750000 out of 5000000 steps (35%)
14:26:42:WU00:FS01:0x18:Completed 1800000 out of 5000000 steps (36%)
14:29:28:WU00:FS01:0x18:Completed 1850000 out of 5000000 steps (37%)
14:32:19:WU00:FS01:0x18:Completed 1900000 out of 5000000 steps (38%)
14:35:05:WU00:FS01:0x18:Completed 1950000 out of 5000000 steps (39%)
14:37:51:WU00:FS01:0x18:Completed 2000000 out of 5000000 steps (40%)
14:40:42:WU00:FS01:0x18:Completed 2050000 out of 5000000 steps (41%)
14:43:28:WU00:FS01:0x18:Completed 2100000 out of 5000000 steps (42%)
14:46:19:WU00:FS01:0x18:Completed 2150000 out of 5000000 steps (43%)
14:49:05:WU00:FS01:0x18:Completed 2200000 out of 5000000 steps (44%)
14:51:51:WU00:FS01:0x18:Completed 2250000 out of 5000000 steps (45%)
14:54:42:WU00:FS01:0x18:Completed 2300000 out of 5000000 steps (46%)
14:57:29:WU00:FS01:0x18:Completed 2350000 out of 5000000 steps (47%)
15:00:24:WU00:FS01:0x18:Completed 2400000 out of 5000000 steps (48%)
15:03:11:WU00:FS01:0x18:Completed 2450000 out of 5000000 steps (49%)
15:05:59:WU00:FS01:0x18:Completed 2500000 out of 5000000 steps (50%)
15:08:54:WU00:FS01:0x18:Completed 2550000 out of 5000000 steps (51%)
15:11:42:WU00:FS01:0x18:Completed 2600000 out of 5000000 steps (52%)
15:14:37:WU00:FS01:0x18:Completed 2650000 out of 5000000 steps (53%)
15:17:25:WU00:FS01:0x18:Completed 2700000 out of 5000000 steps (54%)
15:20:13:WU00:FS01:0x18:Completed 2750000 out of 5000000 steps (55%)
15:23:08:WU00:FS01:0x18:Completed 2800000 out of 5000000 steps (56%)
15:25:56:WU00:FS01:0x18:Completed 2850000 out of 5000000 steps (57%)
15:28:51:WU00:FS01:0x18:Completed 2900000 out of 5000000 steps (58%)
15:31:39:WU00:FS01:0x18:Completed 2950000 out of 5000000 steps (59%)
15:34:26:WU00:FS01:0x18:Completed 3000000 out of 5000000 steps (60%)
15:37:21:WU00:FS01:0x18:Completed 3050000 out of 5000000 steps (61%)
15:40:09:WU00:FS01:0x18:Completed 3100000 out of 5000000 steps (62%)
15:43:04:WU00:FS01:0x18:Completed 3150000 out of 5000000 steps (63%)
15:45:52:WU00:FS01:0x18:Completed 3200000 out of 5000000 steps (64%)
15:48:40:WU00:FS01:0x18:Completed 3250000 out of 5000000 steps (65%)
15:51:35:WU00:FS01:0x18:Completed 3300000 out of 5000000 steps (66%)
15:54:23:WU00:FS01:0x18:Completed 3350000 out of 5000000 steps (67%)
15:57:18:WU00:FS01:0x18:Completed 3400000 out of 5000000 steps (68%)
16:00:06:WU00:FS01:0x18:Completed 3450000 out of 5000000 steps (69%)
16:02:54:WU00:FS01:0x18:Completed 3500000 out of 5000000 steps (70%)
16:05:49:WU00:FS01:0x18:Completed 3550000 out of 5000000 steps (71%)
16:08:37:WU00:FS01:0x18:Completed 3600000 out of 5000000 steps (72%)
16:11:32:WU00:FS01:0x18:Completed 3650000 out of 5000000 steps (73%)
16:14:20:WU00:FS01:0x18:Completed 3700000 out of 5000000 steps (74%)
16:17:07:WU00:FS01:0x18:Completed 3750000 out of 5000000 steps (75%)
16:20:02:WU00:FS01:0x18:Completed 3800000 out of 5000000 steps (76%)
16:22:50:WU00:FS01:0x18:Completed 3850000 out of 5000000 steps (77%)
16:25:45:WU00:FS01:0x18:Completed 3900000 out of 5000000 steps (78%)
16:28:33:WU00:FS01:0x18:Completed 3950000 out of 5000000 steps (79%)
16:31:21:WU00:FS01:0x18:Completed 4000000 out of 5000000 steps (80%)
16:34:16:WU00:FS01:0x18:Completed 4050000 out of 5000000 steps (81%)
16:37:04:WU00:FS01:0x18:Completed 4100000 out of 5000000 steps (82%)
16:39:59:WU00:FS01:0x18:Completed 4150000 out of 5000000 steps (83%)
16:42:47:WU00:FS01:0x18:Completed 4200000 out of 5000000 steps (84%)
16:45:35:WU00:FS01:0x18:Completed 4250000 out of 5000000 steps (85%)
16:48:30:WU00:FS01:0x18:Completed 4300000 out of 5000000 steps (86%)
16:51:18:WU00:FS01:0x18:Completed 4350000 out of 5000000 steps (87%)
16:54:13:WU00:FS01:0x18:Completed 4400000 out of 5000000 steps (88%)
16:57:01:WU00:FS01:0x18:Completed 4450000 out of 5000000 steps (89%)
16:59:49:WU00:FS01:0x18:Completed 4500000 out of 5000000 steps (90%)
17:02:44:WU00:FS01:0x18:Completed 4550000 out of 5000000 steps (91%)
17:05:32:WU00:FS01:0x18:Completed 4600000 out of 5000000 steps (92%)
17:08:27:WU00:FS01:0x18:Completed 4650000 out of 5000000 steps (93%)
17:11:15:WU00:FS01:0x18:Completed 4700000 out of 5000000 steps (94%)
17:14:02:WU00:FS01:0x18:Completed 4750000 out of 5000000 steps (95%)
17:16:57:WU00:FS01:0x18:Completed 4800000 out of 5000000 steps (96%)
17:19:45:WU00:FS01:0x18:Completed 4850000 out of 5000000 steps (97%)
17:22:40:WU00:FS01:0x18:Completed 4900000 out of 5000000 steps (98%)
17:25:28:WU00:FS01:0x18:Completed 4950000 out of 5000000 steps (99%)
17:28:16:WU00:FS01:0x18:Completed 5000000 out of 5000000 steps (100%)
17:28:23:WU00:FS01:0x18:Saving result file logfile_01.txt
17:28:23:WU00:FS01:0x18:Saving result file checkpointState.xml
17:28:25:WU00:FS01:0x18:Saving result file checkpt.crc
17:28:25:WU00:FS01:0x18:Saving result file log.txt
17:28:25:WU00:FS01:0x18:Saving result file positions.xtc
******************************* Date: 2015-12-10 *******************************

Should all of those files be in the same folder as the logfile_01.txt, ie. ...\AppData\Roaming\FAHClient\work\00? or will they be saved elsewhere? Because these are all I can see right now in that folder



Should I try reinstalling? And if I do so, are there any other options I should switch besides the install/saved-data location?

[mmonnin]

Those are the data files of your Work Units. I have logs in my work directories as well.

[lyndwyrm]

But should checkpointState.xml, checkpt.crc, and positions.xtc be in the same directory? If they should, then they aren't getting written at all.

[mmonnin]

Mine are inside the 2nd '01' folder and 01 is my GPU Slot.

..FAHClient\work\01\01\checkpt.crc

[lyndwyrm]

In that folder I just have a bunch of viewerFrame##.json and viewerTop.json, but no checkpoint files.

[mmonnin]

Where are they? They aren't in the screenshot above.

[lyndwyrm]

The .json files are in the "01" folder that is visible, like



but the only things in this folder are .json files, no checkpoints.