About 1 in 5 times, I get a job that completes, but then crashes with unknown error.
This is the log file from the advanced control
Code: Select all
11:24:08:WU01:FS00:Starting
11:24:08:WU01:FS00:Running FahCore: D:\C_Alt\programs\FAHClient/FAHCoreWrapper.exe D:\C_Alt\data\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 11432 -checkpoint 5 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
11:24:08:WU01:FS00:Started FahCore on PID 13736
11:24:08:WU01:FS00:Core PID:12508
11:24:08:WU01:FS00:FahCore 0x22 started
11:24:08:WU01:FS00:0x22:*********************** Log Started 2020-06-04T11:24:08Z ***********************
11:24:08:WU01:FS00:0x22:*************************** Core22 Folding@home Core ***************************
11:24:08:WU01:FS00:0x22: Type: 0x22
11:24:08:WU01:FS00:0x22: Core: Core22
11:24:08:WU01:FS00:0x22: Website: https://foldingathome.org/
11:24:08:WU01:FS00:0x22: Copyright: (c) 2009-2018 foldingathome.org
11:24:08:WU01:FS00:0x22: Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
11:24:08:WU01:FS00:0x22: <rafal.wiewiora@choderalab.org>
11:24:08:WU01:FS00:0x22: Args: -dir 01 -suffix 01 -version 706 -lifeline 13736 -checkpoint 5
11:24:08:WU01:FS00:0x22: -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
11:24:08:WU01:FS00:0x22: Config: <none>
11:24:08:WU01:FS00:0x22:************************************ Build *************************************
11:24:08:WU01:FS00:0x22: Version: 0.0.5
11:24:08:WU01:FS00:0x22: Date: Apr 22 2020
11:24:08:WU01:FS00:0x22: Time: 04:42:59
11:24:08:WU01:FS00:0x22: Repository: Git
11:24:08:WU01:FS00:0x22: Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
11:24:08:WU01:FS00:0x22: Branch: HEAD
11:24:08:WU01:FS00:0x22: Compiler: Visual C++ 2008
11:24:08:WU01:FS00:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
11:24:08:WU01:FS00:0x22: Platform: win32 10
11:24:08:WU01:FS00:0x22: Bits: 64
11:24:08:WU01:FS00:0x22: Mode: Release
11:24:08:WU01:FS00:0x22:************************************ System ************************************
11:24:08:WU01:FS00:0x22: CPU: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
11:24:08:WU01:FS00:0x22: CPU ID: GenuineIntel Family 6 Model 158 Stepping 9
11:24:08:WU01:FS00:0x22: CPUs: 8
11:24:08:WU01:FS00:0x22: Memory: 31.96GiB
11:24:08:WU01:FS00:0x22:Free Memory: 25.68GiB
11:24:08:WU01:FS00:0x22: Threads: WINDOWS_THREADS
11:24:08:WU01:FS00:0x22: OS Version: 6.2
11:24:08:WU01:FS00:0x22:Has Battery: false
11:24:08:WU01:FS00:0x22: On Battery: false
11:24:08:WU01:FS00:0x22: UTC Offset: -4
11:24:08:WU01:FS00:0x22: PID: 12508
11:24:08:WU01:FS00:0x22: CWD: D:\C_Alt\data\FAHClient\work
11:24:08:WU01:FS00:0x22: OS: Windows 10 Pro
11:24:08:WU01:FS00:0x22: OS Arch: AMD64
11:24:08:WU01:FS00:0x22:********************************************************************************
11:24:08:WU01:FS00:0x22:Project: 13409 (Run 540, Clone 45, Gen 0)
11:24:08:WU01:FS00:0x22:Unit: 0x0000000112bc7d9a5ed4a88d03bcb4cf
11:24:08:WU01:FS00:0x22:Reading tar file core.xml
11:24:08:WU01:FS00:0x22:Reading tar file integrator.xml
11:24:08:WU01:FS00:0x22:Reading tar file state.xml
11:24:08:WU01:FS00:0x22:Reading tar file system.xml
11:24:08:WU01:FS00:0x22:Digital signatures verified
11:24:08:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
11:24:08:WU01:FS00:0x22:Version 0.0.5
11:24:14:WU01:FS00:0x22:Completed 0 out of 1000000 steps (0%)
11:24:14:WU01:FS00:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
11:25:12:WU01:FS00:0x22:Completed 10000 out of 1000000 steps (1%)
11:26:10:WU01:FS00:0x22:Completed 20000 out of 1000000 steps (2%)
11:27:07:WU01:FS00:0x22:Completed 30000 out of 1000000 steps (3%)
11:28:04:WU01:FS00:0x22:Completed 40000 out of 1000000 steps (4%)
11:29:02:WU01:FS00:0x22:Completed 50000 out of 1000000 steps (5%)
11:30:00:WU01:FS00:0x22:Completed 60000 out of 1000000 steps (6%)
11:30:58:WU01:FS00:0x22:Completed 70000 out of 1000000 steps (7%)
11:31:55:WU01:FS00:0x22:Completed 80000 out of 1000000 steps (8%)
11:32:53:WU01:FS00:0x22:Completed 90000 out of 1000000 steps (9%)
11:33:51:WU01:FS00:0x22:Completed 100000 out of 1000000 steps (10%)
11:34:50:WU01:FS00:0x22:Completed 110000 out of 1000000 steps (11%)
11:35:48:WU01:FS00:0x22:Completed 120000 out of 1000000 steps (12%)
11:36:45:WU01:FS00:0x22:Completed 130000 out of 1000000 steps (13%)
11:37:43:WU01:FS00:0x22:Completed 140000 out of 1000000 steps (14%)
11:38:40:WU01:FS00:0x22:Completed 150000 out of 1000000 steps (15%)
11:39:38:WU01:FS00:0x22:Completed 160000 out of 1000000 steps (16%)
11:40:38:WU01:FS00:0x22:Completed 170000 out of 1000000 steps (17%)
11:41:35:WU01:FS00:0x22:Completed 180000 out of 1000000 steps (18%)
11:42:33:WU01:FS00:0x22:Completed 190000 out of 1000000 steps (19%)
11:43:30:WU01:FS00:0x22:Completed 200000 out of 1000000 steps (20%)
11:44:27:WU01:FS00:0x22:Completed 210000 out of 1000000 steps (21%)
11:45:25:WU01:FS00:0x22:Completed 220000 out of 1000000 steps (22%)
11:46:22:WU01:FS00:0x22:Completed 230000 out of 1000000 steps (23%)
11:47:22:WU01:FS00:0x22:Completed 240000 out of 1000000 steps (24%)
11:48:20:WU01:FS00:0x22:Completed 250000 out of 1000000 steps (25%)
11:49:20:WU01:FS00:0x22:Completed 260000 out of 1000000 steps (26%)
11:50:24:WU01:FS00:0x22:Completed 270000 out of 1000000 steps (27%)
11:51:23:WU01:FS00:0x22:Completed 280000 out of 1000000 steps (28%)
11:52:21:WU01:FS00:0x22:Completed 290000 out of 1000000 steps (29%)
11:53:20:WU01:FS00:0x22:Completed 300000 out of 1000000 steps (30%)
11:54:21:WU01:FS00:0x22:Completed 310000 out of 1000000 steps (31%)
11:55:21:WU01:FS00:0x22:Completed 320000 out of 1000000 steps (32%)
11:56:19:WU01:FS00:0x22:Completed 330000 out of 1000000 steps (33%)
11:57:18:WU01:FS00:0x22:Completed 340000 out of 1000000 steps (34%)
11:58:18:WU01:FS00:0x22:Completed 350000 out of 1000000 steps (35%)
11:59:16:WU01:FS00:0x22:Completed 360000 out of 1000000 steps (36%)
12:00:15:WU01:FS00:0x22:Completed 370000 out of 1000000 steps (37%)
12:01:14:WU01:FS00:0x22:Completed 380000 out of 1000000 steps (38%)
12:02:12:WU01:FS00:0x22:Completed 390000 out of 1000000 steps (39%)
12:03:10:WU01:FS00:0x22:Completed 400000 out of 1000000 steps (40%)
12:04:09:WU01:FS00:0x22:Completed 410000 out of 1000000 steps (41%)
12:05:07:WU01:FS00:0x22:Completed 420000 out of 1000000 steps (42%)
12:06:05:WU01:FS00:0x22:Completed 430000 out of 1000000 steps (43%)
12:07:03:WU01:FS00:0x22:Completed 440000 out of 1000000 steps (44%)
12:08:01:WU01:FS00:0x22:Completed 450000 out of 1000000 steps (45%)
12:09:00:WU01:FS00:0x22:Completed 460000 out of 1000000 steps (46%)
12:09:58:WU01:FS00:0x22:Completed 470000 out of 1000000 steps (47%)
12:10:56:WU01:FS00:0x22:Completed 480000 out of 1000000 steps (48%)
12:11:54:WU01:FS00:0x22:Completed 490000 out of 1000000 steps (49%)
12:12:53:WU01:FS00:0x22:Completed 500000 out of 1000000 steps (50%)
12:13:50:WU01:FS00:0x22:Completed 510000 out of 1000000 steps (51%)
12:14:46:WU01:FS00:0x22:Completed 520000 out of 1000000 steps (52%)
12:15:43:WU01:FS00:0x22:Completed 530000 out of 1000000 steps (53%)
12:16:40:WU01:FS00:0x22:Completed 540000 out of 1000000 steps (54%)
12:17:36:WU01:FS00:0x22:Completed 550000 out of 1000000 steps (55%)
12:18:33:WU01:FS00:0x22:Completed 560000 out of 1000000 steps (56%)
12:19:30:WU01:FS00:0x22:Completed 570000 out of 1000000 steps (57%)
12:20:27:WU01:FS00:0x22:Completed 580000 out of 1000000 steps (58%)
12:21:24:WU01:FS00:0x22:Completed 590000 out of 1000000 steps (59%)
12:22:20:WU01:FS00:0x22:Completed 600000 out of 1000000 steps (60%)
12:23:17:WU01:FS00:0x22:Completed 610000 out of 1000000 steps (61%)
12:24:14:WU01:FS00:0x22:Completed 620000 out of 1000000 steps (62%)
12:25:11:WU01:FS00:0x22:Completed 630000 out of 1000000 steps (63%)
12:26:07:WU01:FS00:0x22:Completed 640000 out of 1000000 steps (64%)
12:27:04:WU01:FS00:0x22:Completed 650000 out of 1000000 steps (65%)
12:28:01:WU01:FS00:0x22:Completed 660000 out of 1000000 steps (66%)
12:28:58:WU01:FS00:0x22:Completed 670000 out of 1000000 steps (67%)
12:29:54:WU01:FS00:0x22:Completed 680000 out of 1000000 steps (68%)
12:30:51:WU01:FS00:0x22:Completed 690000 out of 1000000 steps (69%)
12:31:48:WU01:FS00:0x22:Completed 700000 out of 1000000 steps (70%)
12:32:45:WU01:FS00:0x22:Completed 710000 out of 1000000 steps (71%)
12:33:41:WU01:FS00:0x22:Completed 720000 out of 1000000 steps (72%)
12:34:38:WU01:FS00:0x22:Completed 730000 out of 1000000 steps (73%)
12:35:35:WU01:FS00:0x22:Completed 740000 out of 1000000 steps (74%)
12:36:32:WU01:FS00:0x22:Completed 750000 out of 1000000 steps (75%)
12:37:30:WU01:FS00:0x22:Completed 760000 out of 1000000 steps (76%)
12:38:28:WU01:FS00:0x22:Completed 770000 out of 1000000 steps (77%)
12:39:26:WU01:FS00:0x22:Completed 780000 out of 1000000 steps (78%)
12:40:24:WU01:FS00:0x22:Completed 790000 out of 1000000 steps (79%)
12:41:23:WU01:FS00:0x22:Completed 800000 out of 1000000 steps (80%)
12:42:21:WU01:FS00:0x22:Completed 810000 out of 1000000 steps (81%)
12:43:19:WU01:FS00:0x22:Completed 820000 out of 1000000 steps (82%)
12:44:17:WU01:FS00:0x22:Completed 830000 out of 1000000 steps (83%)
12:45:15:WU01:FS00:0x22:Completed 840000 out of 1000000 steps (84%)
12:46:13:WU01:FS00:0x22:Completed 850000 out of 1000000 steps (85%)
12:47:12:WU01:FS00:0x22:Completed 860000 out of 1000000 steps (86%)
12:48:10:WU01:FS00:0x22:Completed 870000 out of 1000000 steps (87%)
12:49:08:WU01:FS00:0x22:Completed 880000 out of 1000000 steps (88%)
12:50:06:WU01:FS00:0x22:Completed 890000 out of 1000000 steps (89%)
12:51:04:WU01:FS00:0x22:Completed 900000 out of 1000000 steps (90%)
12:52:02:WU01:FS00:0x22:Completed 910000 out of 1000000 steps (91%)
12:53:00:WU01:FS00:0x22:Completed 920000 out of 1000000 steps (92%)
12:53:59:WU01:FS00:0x22:Completed 930000 out of 1000000 steps (93%)
12:54:57:WU01:FS00:0x22:Completed 940000 out of 1000000 steps (94%)
12:55:55:WU01:FS00:0x22:Completed 950000 out of 1000000 steps (95%)
12:56:53:WU01:FS00:0x22:Completed 960000 out of 1000000 steps (96%)
12:57:51:WU01:FS00:0x22:Completed 970000 out of 1000000 steps (97%)
12:58:50:WU01:FS00:0x22:Completed 980000 out of 1000000 steps (98%)
12:59:48:WU01:FS00:0x22:Completed 990000 out of 1000000 steps (99%)
13:00:46:WU01:FS00:0x22:Completed 1000000 out of 1000000 steps (100%)
13:00:47:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
13:00:47:WU01:FS00:0x22:Saving result file checkpointState.xml
13:00:47:WU01:FS00:0x22:Saving result file checkpt.crc
13:00:47:WU01:FS00:0x22:Saving result file globals.csv
13:00:47:WU01:FS00:0x22:Saving result file positions.xtc
13:00:47:WU01:FS00:0x22:Saving result file science.log
13:00:47:WU01:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
13:00:49:WARNING:WU01:FS00:FahCore returned an unknown error code which probably indicates that it crashed
13:00:49:WARNING:WU01:FS00:FahCore returned: UNKNOWN_ENUM (-1073740940 = 0xc0000374)
Code: Select all
*********************** Log Started 2020-06-04T11:24:08Z ***********************
*************************** Core22 Folding@home Core ***************************
Type: 0x22
Core: Core22
Website: https://foldingathome.org/
Copyright: (c) 2009-2018 foldingathome.org
Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
<rafal.wiewiora@choderalab.org>
Args: -dir 01 -suffix 01 -version 706 -lifeline 13736 -checkpoint 5
-gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
Config: <none>
************************************ Build *************************************
Version: 0.0.5
Date: Apr 22 2020
Time: 04:42:59
Repository: Git
Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
Branch: HEAD
Compiler: Visual C++ 2008
Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
Platform: win32 10
Bits: 64
Mode: Release
************************************ System ************************************
CPU: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
CPU ID: GenuineIntel Family 6 Model 158 Stepping 9
CPUs: 8
Memory: 31.96GiB
Free Memory: 25.68GiB
Threads: WINDOWS_THREADS
OS Version: 6.2
Has Battery: false
On Battery: false
UTC Offset: -4
PID: 12508
CWD: D:\C_Alt\data\FAHClient\work
OS: Windows 10 Pro
OS Arch: AMD64
********************************************************************************
Project: 13409 (Run 540, Clone 45, Gen 0)
Unit: 0x0000000112bc7d9a5ed4a88d03bcb4cf
Reading tar file core.xml
Reading tar file integrator.xml
Reading tar file state.xml
Reading tar file system.xml
Digital signatures verified
Folding@home GPU Core22 Folding@home Core
Version 0.0.5
Completed 0 out of 1000000 steps (0%)
Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
Completed 10000 out of 1000000 steps (1%)
Completed 20000 out of 1000000 steps (2%)
Completed 30000 out of 1000000 steps (3%)
Completed 40000 out of 1000000 steps (4%)
Completed 50000 out of 1000000 steps (5%)
Completed 60000 out of 1000000 steps (6%)
Completed 70000 out of 1000000 steps (7%)
Completed 80000 out of 1000000 steps (8%)
Completed 90000 out of 1000000 steps (9%)
Completed 100000 out of 1000000 steps (10%)
Completed 110000 out of 1000000 steps (11%)
Completed 120000 out of 1000000 steps (12%)
Completed 130000 out of 1000000 steps (13%)
Completed 140000 out of 1000000 steps (14%)
Completed 150000 out of 1000000 steps (15%)
Completed 160000 out of 1000000 steps (16%)
Completed 170000 out of 1000000 steps (17%)
Completed 180000 out of 1000000 steps (18%)
Completed 190000 out of 1000000 steps (19%)
Completed 200000 out of 1000000 steps (20%)
Completed 210000 out of 1000000 steps (21%)
Completed 220000 out of 1000000 steps (22%)
Completed 230000 out of 1000000 steps (23%)
Completed 240000 out of 1000000 steps (24%)
Completed 250000 out of 1000000 steps (25%)
Completed 260000 out of 1000000 steps (26%)
Completed 270000 out of 1000000 steps (27%)
Completed 280000 out of 1000000 steps (28%)
Completed 290000 out of 1000000 steps (29%)
Completed 300000 out of 1000000 steps (30%)
Completed 310000 out of 1000000 steps (31%)
Completed 320000 out of 1000000 steps (32%)
Completed 330000 out of 1000000 steps (33%)
Completed 340000 out of 1000000 steps (34%)
Completed 350000 out of 1000000 steps (35%)
Completed 360000 out of 1000000 steps (36%)
Completed 370000 out of 1000000 steps (37%)
Completed 380000 out of 1000000 steps (38%)
Completed 390000 out of 1000000 steps (39%)
Completed 400000 out of 1000000 steps (40%)
Completed 410000 out of 1000000 steps (41%)
Completed 420000 out of 1000000 steps (42%)
Completed 430000 out of 1000000 steps (43%)
Completed 440000 out of 1000000 steps (44%)
Completed 450000 out of 1000000 steps (45%)
Completed 460000 out of 1000000 steps (46%)
Completed 470000 out of 1000000 steps (47%)
Completed 480000 out of 1000000 steps (48%)
Completed 490000 out of 1000000 steps (49%)
Completed 500000 out of 1000000 steps (50%)
Completed 510000 out of 1000000 steps (51%)
Completed 520000 out of 1000000 steps (52%)
Completed 530000 out of 1000000 steps (53%)
Completed 540000 out of 1000000 steps (54%)
Completed 550000 out of 1000000 steps (55%)
Completed 560000 out of 1000000 steps (56%)
Completed 570000 out of 1000000 steps (57%)
Completed 580000 out of 1000000 steps (58%)
Completed 590000 out of 1000000 steps (59%)
Completed 600000 out of 1000000 steps (60%)
Completed 610000 out of 1000000 steps (61%)
Completed 620000 out of 1000000 steps (62%)
Completed 630000 out of 1000000 steps (63%)
Completed 640000 out of 1000000 steps (64%)
Completed 650000 out of 1000000 steps (65%)
Completed 660000 out of 1000000 steps (66%)
Completed 670000 out of 1000000 steps (67%)
Completed 680000 out of 1000000 steps (68%)
Completed 690000 out of 1000000 steps (69%)
Completed 700000 out of 1000000 steps (70%)
Completed 710000 out of 1000000 steps (71%)
Completed 720000 out of 1000000 steps (72%)
Completed 730000 out of 1000000 steps (73%)
Completed 740000 out of 1000000 steps (74%)
Completed 750000 out of 1000000 steps (75%)
Completed 760000 out of 1000000 steps (76%)
Completed 770000 out of 1000000 steps (77%)
Completed 780000 out of 1000000 steps (78%)
Completed 790000 out of 1000000 steps (79%)
Completed 800000 out of 1000000 steps (80%)
Completed 810000 out of 1000000 steps (81%)
Completed 820000 out of 1000000 steps (82%)
Completed 830000 out of 1000000 steps (83%)
Completed 840000 out of 1000000 steps (84%)
Completed 850000 out of 1000000 steps (85%)
Completed 860000 out of 1000000 steps (86%)
Completed 870000 out of 1000000 steps (87%)
Completed 880000 out of 1000000 steps (88%)
Completed 890000 out of 1000000 steps (89%)
Completed 900000 out of 1000000 steps (90%)
Completed 910000 out of 1000000 steps (91%)
Completed 920000 out of 1000000 steps (92%)
Completed 930000 out of 1000000 steps (93%)
Completed 940000 out of 1000000 steps (94%)
Completed 950000 out of 1000000 steps (95%)
Completed 960000 out of 1000000 steps (96%)
Completed 970000 out of 1000000 steps (97%)
Completed 980000 out of 1000000 steps (98%)
Completed 990000 out of 1000000 steps (99%)
Completed 1000000 out of 1000000 steps (100%)
Saving result file ..\logfile_01.txt
Saving result file checkpointState.xml
Saving result file checkpt.crc
Saving result file globals.csv
Saving result file positions.xtc
Saving result file science.log
Folding@home Core Shutdown: FINISHED_UNIT
Once this happens, the work unit is left in the advanced control at 100% complete. Another Work unit will be processed, then the system goes back to the start of the one that failed. That failed one will process the second time. The work eventually gets done, but it has to happen twice.
Anybody had any ideas?