CUDA_ERROR_LAUNCH_FAILED

Tuna_Ertemalp · Post by **Tuna_Ertemalp** » Thu Oct 22, 2020 6:10 pm

This is on one of my 9 hosts,

running Win10 on AMD Ryzen TR 1950X (16C/32T using 28T for WUs)
with 32G memory and
with 4x NVidia 1080Ti
each with 11G memory
using the latest driver 456.71
all on Hybrid cooling, therefore always staying between 41C-58C temperature,
with FAH WUs usually keeping the GPUs busy anywhere between 60-90%, but never really hitting 90%+. (In comparison, many BOINC projects I ran in the past used to keep them at 99-100% 24/7)
running the latest FAH 7.6.20

Lately, I have been randomly getting CUDA_ERROR_LAUNCH_FAILED on different GPUs at random WU progress points, for variety of WUs. When the log shows this error, I can see a Windows dialog that indicates that FahClient_22.exe associated with that GPU has crashed. Sometimes it is just 1 GPU, sometimes 2, sometimes 3. If so, I have that many crash dialogs up on my screen. If I OK that dialog, the WU either continues and finishes later, or continues and crashes again after a random amount of further progress, and if it crashes enough times, some restart threshold is hit and FAH terminates the WU. Therefore, one of my best machines has been not been fulfilling its PPD capacity. And that has slowed down my cracking into the top-400 users...

Thoughts?

If this is something that needs to be debugged, feel free to point me to instrumented EXEs/DLLs and/or instructions to give you what you need. This happens irregularly, yet fairly often on this machine; we would have some detailed debug logs in short order.

Here is the log file:

Code: Select all

******************************* Date: 2020-10-22 *******************************
16:01:45:WU01:FS04:Connecting to assign1.foldingathome.org:80
16:01:45:WU01:FS04:Assigned to work server 129.213.157.105
16:01:45:WU01:FS04:Requesting new work unit for slot 04: gpu:67:0 GP102 [GeForce GTX 1080 Ti] 11380 - RUNNING from 129.213.157.105
16:01:45:WU01:FS04:Connecting to 129.213.157.105:8080
16:01:49:WU01:FS04:Downloading 10.63MiB
16:01:55:WU01:FS04:Download 5.29%
16:02:01:WU01:FS04:Download 7.06%
16:02:07:WU01:FS04:Download 11.17%
16:02:14:WU01:FS04:Download 15.29%
16:02:20:WU01:FS04:Download 17.64%
16:02:26:WU01:FS04:Download 20.00%
16:02:32:WU01:FS04:Download 22.94%
16:02:38:WU01:FS04:Download 25.88%
16:02:44:WU01:FS04:Download 28.82%
16:02:50:WU01:FS04:Download 31.17%
16:02:56:WU01:FS04:Download 34.70%
16:03:03:WU01:FS04:Download 37.64%
16:03:10:WU01:FS04:Download 40.58%
16:03:17:WU01:FS04:Download 43.52%
16:03:23:WU01:FS04:Download 45.28%
16:03:30:WU01:FS04:Download 48.81%
16:03:36:WU01:FS04:Download 52.93%
16:03:42:WU01:FS04:Download 57.05%
16:03:48:WU01:FS04:Download 59.99%
16:03:54:WU01:FS04:Download 63.52%
16:04:01:WU01:FS04:Download 67.04%
16:04:08:WU01:FS04:Download 69.40%
16:04:15:WU01:FS04:Download 72.93%
16:04:21:WU01:FS04:Download 78.22%
16:04:27:WU01:FS04:Download 80.57%
16:04:34:WU01:FS04:Download 84.69%
16:04:40:WU01:FS04:Download 87.63%
16:04:47:WU01:FS04:Download 90.57%
16:04:53:WU01:FS04:Download 93.51%
16:04:59:WU01:FS04:Download 97.63%
16:05:02:WU01:FS04:Download complete
16:05:02:WU01:FS04:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14904 run:417 clone:0 gen:62 core:0x22 unit:0x0000005981d59d695f4ec9e340f2570a
16:05:02:WU01:FS04:Starting
16:05:02:WU01:FS04:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Master\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 13572 -checkpoint 5 -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor nvidia -gpu 3 -gpu-usage 100
16:05:02:WU01:FS04:Started FahCore on PID 14572
16:05:02:WU01:FS04:Core PID:11896
16:05:02:WU01:FS04:FahCore 0x22 started
16:05:02:WU01:FS04:0x22:*********************** Log Started 2020-10-22T16:05:02Z ***********************
16:05:02:WU01:FS04:0x22:*************************** Core22 Folding@home Core ***************************
16:05:03:WU01:FS04:0x22:       Core: Core22
16:05:03:WU01:FS04:0x22:       Type: 0x22
16:05:03:WU01:FS04:0x22:    Version: 0.0.13
16:05:03:WU01:FS04:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:05:03:WU01:FS04:0x22:  Copyright: 2020 foldingathome.org
16:05:03:WU01:FS04:0x22:   Homepage: https://foldingathome.org/
16:05:03:WU01:FS04:0x22:       Date: Sep 19 2020
16:05:03:WU01:FS04:0x22:       Time: 02:35:58
16:05:03:WU01:FS04:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:05:03:WU01:FS04:0x22:     Branch: core22-0.0.13
16:05:03:WU01:FS04:0x22:   Compiler: Visual C++ 2015
16:05:03:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:05:03:WU01:FS04:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:05:03:WU01:FS04:0x22:   Platform: win32 10
16:05:03:WU01:FS04:0x22:       Bits: 64
16:05:03:WU01:FS04:0x22:       Mode: Release
16:05:03:WU01:FS04:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:05:03:WU01:FS04:0x22:             <peastman@stanford.edu>
16:05:03:WU01:FS04:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 14572 -checkpoint 5
16:05:03:WU01:FS04:0x22:             -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor
16:05:03:WU01:FS04:0x22:             nvidia -gpu 3 -gpu-usage 100
16:05:03:WU01:FS04:0x22:************************************ libFAH ************************************
16:05:03:WU01:FS04:0x22:       Date: Sep 7 2020
16:05:03:WU01:FS04:0x22:       Time: 19:09:56
16:05:03:WU01:FS04:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:05:03:WU01:FS04:0x22:     Branch: HEAD
16:05:03:WU01:FS04:0x22:   Compiler: Visual C++ 2015
16:05:03:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:05:03:WU01:FS04:0x22:   Platform: win32 10
16:05:03:WU01:FS04:0x22:       Bits: 64
16:05:03:WU01:FS04:0x22:       Mode: Release
16:05:03:WU01:FS04:0x22:************************************ CBang *************************************
16:05:03:WU01:FS04:0x22:       Date: Sep 7 2020
16:05:03:WU01:FS04:0x22:       Time: 19:08:30
16:05:03:WU01:FS04:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:05:03:WU01:FS04:0x22:     Branch: HEAD
16:05:03:WU01:FS04:0x22:   Compiler: Visual C++ 2015
16:05:03:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:05:03:WU01:FS04:0x22:   Platform: win32 10
16:05:03:WU01:FS04:0x22:       Bits: 64
16:05:03:WU01:FS04:0x22:       Mode: Release
16:05:03:WU01:FS04:0x22:************************************ System ************************************
16:05:03:WU01:FS04:0x22:        CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
16:05:03:WU01:FS04:0x22:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
16:05:03:WU01:FS04:0x22:       CPUs: 32
16:05:03:WU01:FS04:0x22:     Memory: 31.88GiB
16:05:03:WU01:FS04:0x22:Free Memory: 25.04GiB
16:05:03:WU01:FS04:0x22:    Threads: WINDOWS_THREADS
16:05:03:WU01:FS04:0x22: OS Version: 6.2
16:05:03:WU01:FS04:0x22:Has Battery: false
16:05:03:WU01:FS04:0x22: On Battery: false
16:05:03:WU01:FS04:0x22: UTC Offset: -7
16:05:03:WU01:FS04:0x22:        PID: 11896
16:05:03:WU01:FS04:0x22:        CWD: C:\Users\Master\AppData\Roaming\FAHClient\work
16:05:03:WU01:FS04:0x22:************************************ OpenMM ************************************
16:05:03:WU01:FS04:0x22:   Revision: 189320d0
16:05:03:WU01:FS04:0x22:********************************************************************************
16:05:03:WU01:FS04:0x22:Project: 14904 (Run 417, Clone 0, Gen 62)
16:05:03:WU01:FS04:0x22:Unit: 0x0000005981d59d695f4ec9e340f2570a
16:05:03:WU01:FS04:0x22:Reading tar file core.xml
16:05:03:WU01:FS04:0x22:Reading tar file integrator.xml
16:05:03:WU01:FS04:0x22:Reading tar file state.xml
16:05:03:WU01:FS04:0x22:Reading tar file system.xml
16:05:05:WU01:FS04:0x22:Digital signatures verified
16:05:05:WU01:FS04:0x22:Folding@home GPU Core22 Folding@home Core
16:05:05:WU01:FS04:0x22:Version 0.0.13
16:05:05:WU01:FS04:0x22:  Checkpoint write interval: 100000 steps (5%) [20 total]
16:05:05:WU01:FS04:0x22:  JSON viewer frame write interval: 20000 steps (1%) [100 total]
16:05:05:WU01:FS04:0x22:  XTC frame write interval: 50000 steps (2.5%) [40 total]
16:05:05:WU01:FS04:0x22:  Global context and integrator variables write interval: disabled
16:05:05:WU01:FS04:0x22:There are 4 platforms available.
16:05:05:WU01:FS04:0x22:Platform 0: Reference
16:05:05:WU01:FS04:0x22:Platform 1: CPU
16:05:05:WU01:FS04:0x22:Platform 2: OpenCL
16:05:05:WU01:FS04:0x22:  opencl-device 3 specified
16:05:05:WU01:FS04:0x22:Platform 3: CUDA
16:05:05:WU01:FS04:0x22:  cuda-device 3 specified
16:05:17:WU01:FS04:0x22:Attempting to create CUDA context:
16:05:17:WU01:FS04:0x22:  Configuring platform CUDA
16:05:23:WU01:FS04:0x22:  Using CUDA and gpu 3
16:05:23:WU01:FS04:0x22:Completed 0 out of 2000000 steps (0%)
16:05:24:WU01:FS04:0x22:Checkpoint completed at step 0
16:06:45:WU01:FS04:0x22:An exception occurred at step 18957: Error invoking kernel: CUDA_ERROR_LAUNCH_FAILED (719)
16:06:45:WU01:FS04:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
16:06:45:WU01:FS04:0x22:Folding@home Core Shutdown: CORE_RESTART
17:33:52:WARNING:WU01:FS04:FahCore returned an unknown error code which probably indicates that it crashed
17:33:52:WARNING:WU01:FS04:FahCore returned: UNKNOWN_ENUM (-1073740791 = 0xc0000409)
17:33:53:WU01:FS04:Starting
17:33:53:WU01:FS04:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Master\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 13572 -checkpoint 5 -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor nvidia -gpu 3 -gpu-usage 100
17:33:53:WU01:FS04:Started FahCore on PID 4460
17:33:53:WU01:FS04:Core PID:4012
17:33:53:WU01:FS04:FahCore 0x22 started
17:33:53:WU01:FS04:0x22:*********************** Log Started 2020-10-22T17:33:53Z ***********************
17:33:53:WU01:FS04:0x22:*************************** Core22 Folding@home Core ***************************
17:33:53:WU01:FS04:0x22:       Core: Core22
17:33:53:WU01:FS04:0x22:       Type: 0x22
17:33:53:WU01:FS04:0x22:    Version: 0.0.13
17:33:53:WU01:FS04:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:33:53:WU01:FS04:0x22:  Copyright: 2020 foldingathome.org
17:33:53:WU01:FS04:0x22:   Homepage: https://foldingathome.org/
17:33:53:WU01:FS04:0x22:       Date: Sep 19 2020
17:33:53:WU01:FS04:0x22:       Time: 02:35:58
17:33:53:WU01:FS04:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
17:33:53:WU01:FS04:0x22:     Branch: core22-0.0.13
17:33:53:WU01:FS04:0x22:   Compiler: Visual C++ 2015
17:33:53:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
17:33:53:WU01:FS04:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
17:33:53:WU01:FS04:0x22:   Platform: win32 10
17:33:53:WU01:FS04:0x22:       Bits: 64
17:33:53:WU01:FS04:0x22:       Mode: Release
17:33:53:WU01:FS04:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
17:33:53:WU01:FS04:0x22:             <peastman@stanford.edu>
17:33:53:WU01:FS04:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 4460 -checkpoint 5
17:33:53:WU01:FS04:0x22:             -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor
17:33:53:WU01:FS04:0x22:             nvidia -gpu 3 -gpu-usage 100
17:33:53:WU01:FS04:0x22:************************************ libFAH ************************************
17:33:53:WU01:FS04:0x22:       Date: Sep 7 2020
17:33:53:WU01:FS04:0x22:       Time: 19:09:56
17:33:53:WU01:FS04:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
17:33:53:WU01:FS04:0x22:     Branch: HEAD
17:33:53:WU01:FS04:0x22:   Compiler: Visual C++ 2015
17:33:53:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
17:33:53:WU01:FS04:0x22:   Platform: win32 10
17:33:53:WU01:FS04:0x22:       Bits: 64
17:33:53:WU01:FS04:0x22:       Mode: Release
17:33:53:WU01:FS04:0x22:************************************ CBang *************************************
17:33:53:WU01:FS04:0x22:       Date: Sep 7 2020
17:33:53:WU01:FS04:0x22:       Time: 19:08:30
17:33:53:WU01:FS04:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
17:33:53:WU01:FS04:0x22:     Branch: HEAD
17:33:53:WU01:FS04:0x22:   Compiler: Visual C++ 2015
17:33:53:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
17:33:53:WU01:FS04:0x22:   Platform: win32 10
17:33:53:WU01:FS04:0x22:       Bits: 64
17:33:53:WU01:FS04:0x22:       Mode: Release
17:33:53:WU01:FS04:0x22:************************************ System ************************************
17:33:53:WU01:FS04:0x22:        CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
17:33:53:WU01:FS04:0x22:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
17:33:53:WU01:FS04:0x22:       CPUs: 32
17:33:53:WU01:FS04:0x22:     Memory: 31.88GiB
17:33:53:WU01:FS04:0x22:Free Memory: 25.61GiB
17:33:53:WU01:FS04:0x22:    Threads: WINDOWS_THREADS
17:33:53:WU01:FS04:0x22: OS Version: 6.2
17:33:53:WU01:FS04:0x22:Has Battery: false
17:33:53:WU01:FS04:0x22: On Battery: false
17:33:53:WU01:FS04:0x22: UTC Offset: -7
17:33:53:WU01:FS04:0x22:        PID: 4012
17:33:53:WU01:FS04:0x22:        CWD: C:\Users\Master\AppData\Roaming\FAHClient\work
17:33:53:WU01:FS04:0x22:************************************ OpenMM ************************************
17:33:53:WU01:FS04:0x22:   Revision: 189320d0
17:33:53:WU01:FS04:0x22:********************************************************************************
17:33:54:WU01:FS04:0x22:Project: 14904 (Run 417, Clone 0, Gen 62)
17:33:54:WU01:FS04:0x22:Unit: 0x0000005981d59d695f4ec9e340f2570a
17:33:54:WU01:FS04:0x22:Digital signatures verified
17:33:54:WU01:FS04:0x22:Folding@home GPU Core22 Folding@home Core
17:33:54:WU01:FS04:0x22:Version 0.0.13
17:33:54:WU01:FS04:0x22:  Checkpoint write interval: 100000 steps (5%) [20 total]
17:33:54:WU01:FS04:0x22:  JSON viewer frame write interval: 20000 steps (1%) [100 total]
17:33:54:WU01:FS04:0x22:  XTC frame write interval: 50000 steps (2.5%) [40 total]
17:33:54:WU01:FS04:0x22:  Global context and integrator variables write interval: disabled
17:33:54:WU01:FS04:0x22:There are 4 platforms available.
17:33:54:WU01:FS04:0x22:Platform 0: Reference
17:33:54:WU01:FS04:0x22:Platform 1: CPU
17:33:54:WU01:FS04:0x22:Platform 2: OpenCL
17:33:54:WU01:FS04:0x22:  opencl-device 3 specified
17:33:54:WU01:FS04:0x22:Platform 3: CUDA
17:33:54:WU01:FS04:0x22:  cuda-device 3 specified
17:34:04:WU01:FS04:0x22:Attempting to create CUDA context:
17:34:04:WU01:FS04:0x22:  Configuring platform CUDA
17:34:10:WU01:FS04:0x22:  Using CUDA and gpu 3
17:34:10:WU01:FS04:0x22:Completed 0 out of 2000000 steps (0%)
17:35:25:WU01:FS04:0x22:Completed 20000 out of 2000000 steps (1%)
17:36:42:WU01:FS04:0x22:Completed 40000 out of 2000000 steps (2%)
17:37:52:WU01:FS04:0x22:Completed 60000 out of 2000000 steps (3%)
17:39:02:WU01:FS04:0x22:Completed 80000 out of 2000000 steps (4%)
17:40:11:WU01:FS04:0x22:Completed 100000 out of 2000000 steps (5%)
17:40:12:WU01:FS04:0x22:Checkpoint completed at step 100000
17:41:21:WU01:FS04:0x22:Completed 120000 out of 2000000 steps (6%)
17:42:30:WU01:FS04:0x22:Completed 140000 out of 2000000 steps (7%)
17:43:39:WU01:FS04:0x22:Completed 160000 out of 2000000 steps (8%)
17:44:53:WU01:FS04:0x22:Completed 180000 out of 2000000 steps (9%)
17:46:12:WU01:FS04:0x22:Completed 200000 out of 2000000 steps (10%)
17:46:13:WU01:FS04:0x22:Checkpoint completed at step 200000
17:47:33:WU01:FS04:0x22:Completed 220000 out of 2000000 steps (11%)
17:48:53:WU01:FS04:0x22:Completed 240000 out of 2000000 steps (12%)
17:50:12:WU01:FS04:0x22:Completed 260000 out of 2000000 steps (13%)
17:51:32:WU01:FS04:0x22:Completed 280000 out of 2000000 steps (14%)
17:52:51:WU01:FS04:0x22:Completed 300000 out of 2000000 steps (15%)
17:52:52:WU01:FS04:0x22:Checkpoint completed at step 300000
17:54:10:WU01:FS04:0x22:Completed 320000 out of 2000000 steps (16%)
17:55:28:WU01:FS04:0x22:Completed 340000 out of 2000000 steps (17%)
17:56:47:WU01:FS04:0x22:Completed 360000 out of 2000000 steps (18%)
17:58:06:WU01:FS04:0x22:Completed 380000 out of 2000000 steps (19%)
17:59:25:WU01:FS04:0x22:Completed 400000 out of 2000000 steps (20%)
17:59:26:WU01:FS04:0x22:Checkpoint completed at step 400000
18:00:35:WU01:FS04:0x22:Completed 420000 out of 2000000 steps (21%)
18:01:39:WU01:FS04:0x22:Completed 440000 out of 2000000 steps (22%)
18:02:39:WU01:FS04:0x22:Completed 460000 out of 2000000 steps (23%)
18:03:38:WU01:FS04:0x22:Completed 480000 out of 2000000 steps (24%)
18:04:38:WU01:FS04:0x22:Completed 500000 out of 2000000 steps (25%)
18:04:39:WU01:FS04:0x22:Checkpoint completed at step 500000
18:05:38:WU01:FS04:0x22:Completed 520000 out of 2000000 steps (26%)
18:06:37:WU01:FS04:0x22:Completed 540000 out of 2000000 steps (27%)

Thanks!
Tuna

Here

JimboPalmer · Post by **JimboPalmer** » Thu Oct 22, 2020 6:50 pm

The first 200 lines of the log show the PC's configuration.

It would help to advise what is wrong with your configuration if we could see it.

Tuna_Ertemalp · Post by **Tuna_Ertemalp** » Thu Oct 22, 2020 7:16 pm

Of course. See below.

And, when I posted my original question just a wee bit ago, the machine was progressing 27% into a WU after I had dismissed a crash dialog, and right now it has 3 "FahCore_22.exe has stopped working .... [Close program]" dialogs up, and therefore three stuck slots. If there is anything else you'd like me to do while in this state, let me know.

Thanks!
Tuna

Code: Select all

*********************** Log Started 2020-10-21T12:09:45Z ***********************
12:09:45:******************************* libFAH ********************************
12:09:45:           Date: Oct 9 2020
12:09:45:           Time: 10:37:10
12:09:45:       Revision: 06b99f7701e0d3f883dd14a78b459ad27da23809
12:09:45:         Branch: master
12:09:45:       Compiler: Visual C++ 2015
12:09:45:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:09:45:       Platform: win32 10
12:09:45:           Bits: 32
12:09:45:           Mode: Release
12:09:45:****************************** FAHClient ******************************
12:09:45:        Version: 7.6.20
12:09:45:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
12:09:45:      Copyright: 2020 foldingathome.org
12:09:45:       Homepage: https://foldingathome.org/
12:09:45:           Date: Oct 12 2020
12:09:45:           Time: 15:02:06
12:09:45:       Revision: c858fe2a8342bfa3e116e00b394d8dfa322ecd18
12:09:45:         Branch: master
12:09:45:       Compiler: Visual C++ 2015
12:09:45:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:09:45:       Platform: win32 10
12:09:45:           Bits: 32
12:09:45:           Mode: Release
12:09:45:         Config: C:\Users\Master\AppData\Roaming\FAHClient\config.xml
12:09:45:******************************** CBang ********************************
12:09:45:           Date: Oct 9 2020
12:09:45:           Time: 10:29:47
12:09:45:       Revision: ab0a6d9e35982b831a74cb2706c569fe46bac2af
12:09:45:         Branch: master
12:09:45:       Compiler: Visual C++ 2015
12:09:45:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:09:45:       Platform: win32 10
12:09:45:           Bits: 32
12:09:45:           Mode: Release
12:09:45:******************************* System ********************************
12:09:45:            CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
12:09:45:         CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
12:09:45:           CPUs: 32
12:09:45:         Memory: 31.88GiB
12:09:45:    Free Memory: 29.30GiB
12:09:45:        Threads: WINDOWS_THREADS
12:09:45:     OS Version: 6.2
12:09:45:    Has Battery: false
12:09:45:     On Battery: false
12:09:45:     UTC Offset: -7
12:09:45:            PID: 13572
12:09:45:            CWD: C:\Users\Master\AppData\Roaming\FAHClient
12:09:45:  Win32 Service: false
12:09:45:             OS: Windows 10 Pro
12:09:45:        OS Arch: AMD64
12:09:45:           GPUs: 4
12:09:45:          GPU 0: Bus:8 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
12:09:45:          GPU 1: Bus:9 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
12:09:45:          GPU 2: Bus:66 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
12:09:45:          GPU 3: Bus:67 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
12:09:45:  CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:11.1
12:09:45:  CUDA Device 1: Platform:0 Device:1 Bus:9 Slot:0 Compute:6.1 Driver:11.1
12:09:45:  CUDA Device 2: Platform:0 Device:2 Bus:66 Slot:0 Compute:6.1 Driver:11.1
12:09:45:  CUDA Device 3: Platform:0 Device:3 Bus:67 Slot:0 Compute:6.1 Driver:11.1
12:09:45:OpenCL Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:1.2 Driver:456.71
12:09:45:OpenCL Device 1: Platform:0 Device:1 Bus:9 Slot:0 Compute:1.2 Driver:456.71
12:09:45:OpenCL Device 2: Platform:0 Device:2 Bus:66 Slot:0 Compute:1.2 Driver:456.71
12:09:45:OpenCL Device 3: Platform:0 Device:3 Bus:67 Slot:0 Compute:1.2 Driver:456.71
12:09:45:***********************************************************************
12:09:45:<config>
12:09:45:  <!-- Client Control -->
12:09:45:  <client-threads v='6'/>
12:09:45:  <cycle-rate v='4'/>
12:09:45:  <cycles v='-1'/>
12:09:45:  <disable-sleep-when-active v='true'/>
12:09:45:  <exit-when-done v='false'/>
12:09:45:  <fold-anon v='false'/>
12:09:45:  <open-web-control v='false'/>
12:09:45:  <update-gpus-txt v='true'/>
12:09:45:
12:09:45:  <!-- Configuration -->
12:09:45:  <config-rotate v='true'/>
12:09:45:  <config-rotate-dir v='configs'/>
12:09:45:  <config-rotate-max v='16'/>
12:09:45:
12:09:45:  <!-- Debugging -->
12:09:45:  <assignment-servers>
12:09:45:    assign1.foldingathome.org assign2.foldingathome.org assign3.foldingathome.org assign4.foldingathome.org 
12:09:45:  </assignment-servers>
12:09:45:  <auth-as v='true'/>
12:09:45:  <capture-directory v='capture'/>
12:09:45:  <capture-on-error v='false'/>
12:09:45:  <capture-packets v='false'/>
12:09:45:  <capture-requests v='false'/>
12:09:45:  <capture-responses v='false'/>
12:09:45:  <capture-sockets v='false'/>
12:09:45:  <debug-sockets v='false'/>
12:09:45:  <exception-locations v='true'/>
12:09:45:  <stack-traces v='false'/>
12:09:45:
12:09:45:  <!-- Error Handling -->
12:09:45:  <max-slot-errors v='10'/>
12:09:45:  <max-unit-errors v='5'/>
12:09:45:
12:09:45:  <!-- Folding Core -->
12:09:45:  <checkpoint v='5'/>
12:09:45:  <core-priority v='low'/>
12:09:45:  <cpu-usage v='100'/>
12:09:45:  <gpu-usage v='100'/>
12:09:45:  <no-assembly v='false'/>
12:09:45:
12:09:45:  <!-- Folding Slot Configuration -->
12:09:45:  <cause v='HIGH_PRIORITY'/>
12:09:45:  <client-subtype v='STDCLI'/>
12:09:45:  <client-type v='normal'/>
12:09:45:  <cpu-species v='X86_AMD'/>
12:09:45:  <cpu-type v='AMD64'/>
12:09:45:  <cpus v='-1'/>
12:09:45:  <disable-viz v='false'/>
12:09:45:  <gpu v='true'/>
12:09:45:  <gpu-beta v='false'/>
12:09:45:  <max-packet-size v='normal'/>
12:09:45:  <os-species v='WIN_8'/>
12:09:45:  <os-type v='WIN32'/>
12:09:45:  <project-key v='0'/>
12:09:45:  <smp v='true'/>
12:09:45:
12:09:45:  <!-- GUI -->
12:09:45:  <gui-enabled v='true'/>
12:09:45:
12:09:45:  <!-- HTTP Server -->
12:09:45:  <allow v='127.0.0.1 192.168.1.0/24'/>
12:09:45:  <connection-timeout v='60'/>
12:09:45:  <deny v='0/0'/>
12:09:45:  <http-addresses v='0:7396'/>
12:09:45:  <https-addresses v=''/>
12:09:45:  <max-connect-time v='900'/>
12:09:45:  <max-connections v='800'/>
12:09:45:  <max-request-length v='52428800'/>
12:09:45:  <min-connect-time v='300'/>
12:09:45:
12:09:45:  <!-- Logging -->
12:09:45:  <log v='log.txt'/>
12:09:45:  <log-color v='false'/>
12:09:45:  <log-crlf v='true'/>
12:09:45:  <log-date v='false'/>
12:09:45:  <log-date-periodically v='21600'/>
12:09:45:  <log-domain v='false'/>
12:09:45:  <log-header v='true'/>
12:09:45:  <log-level v='true'/>
12:09:45:  <log-no-info-header v='true'/>
12:09:45:  <log-redirect v='false'/>
12:09:45:  <log-rotate v='true'/>
12:09:45:  <log-rotate-dir v='logs'/>
12:09:45:  <log-rotate-max v='16'/>
12:09:45:  <log-short-level v='false'/>
12:09:45:  <log-simple-domains v='true'/>
12:09:45:  <log-thread-id v='false'/>
12:09:45:  <log-thread-prefix v='true'/>
12:09:45:  <log-time v='true'/>
12:09:45:  <log-to-screen v='true'/>
12:09:45:  <log-truncate v='false'/>
12:09:45:  <verbosity v='5'/>
12:09:45:
12:09:45:  <!-- Network -->
12:09:45:  <proxy v=':8080'/>
12:09:45:  <proxy-enable v='false'/>
12:09:45:  <proxy-pass v='*****'/>
12:09:45:  <proxy-user v=''/>
12:09:45:
12:09:45:  <!-- Process Control -->
12:09:45:  <child v='false'/>
12:09:45:  <daemon v='false'/>
12:09:45:  <pid v='false'/>
12:09:45:  <pid-file v='FAHClient.pid'/>
12:09:45:  <respawn v='false'/>
12:09:45:  <service v='false'/>
12:09:45:
12:09:45:  <!-- Remote Command Server -->
12:09:45:  <command-address v='0.0.0.0'/>
12:09:45:  <command-allow-no-pass v='127.0.0.1'/>
12:09:45:  <command-deny-no-pass v='0/0'/>
12:09:45:  <command-enable v='true'/>
12:09:45:  <command-port v='36330'/>
12:09:45:  <password v='*****'/>
12:09:45:
12:09:45:  <!-- Slot Control -->
12:09:45:  <auto-conf v='true'/>
12:09:45:  <idle v='false'/>
12:09:45:  <max-shutdown-wait v='60'/>
12:09:45:  <pause-on-battery v='true'/>
12:09:45:  <pause-on-start v='false'/>
12:09:45:  <paused v='false'/>
12:09:45:  <power v='full'/>
12:09:45:
12:09:45:  <!-- User Information -->
12:09:45:  <machine-id v='0'/>
12:09:45:  <passkey v='*****'/>
12:09:45:  <team v='999'/>
12:09:45:  <user v='Tuna_Ertemalp'/>
12:09:45:
12:09:45:  <!-- Web Server -->
12:09:45:  <web-allow v='127.0.0.1'/>
12:09:45:  <web-deny v='0/0'/>
12:09:45:  <web-enable v='true'/>
12:09:45:
12:09:45:  <!-- Web Server Sessions -->
12:09:45:  <session-cookie v='sid'/>
12:09:45:  <session-lifetime v='86400'/>
12:09:45:  <session-timeout v='3600'/>
12:09:45:
12:09:45:  <!-- Windows Service -->
12:09:45:  <service-description v='FAHClient'/>
12:09:45:  <service-restart v='true'/>
12:09:45:  <service-restart-delay v='5000'/>
12:09:45:
12:09:45:  <!-- Work Unit Control -->
12:09:45:  <dump-after-deadline v='true'/>
12:09:45:  <max-queue v='16'/>
12:09:45:  <max-units v='0'/>
12:09:45:  <next-unit-percentage v='99'/>
12:09:45:  <stall-detection-enabled v='false'/>
12:09:45:  <stall-percent v='5'/>
12:09:45:  <stall-timeout v='1800'/>
12:09:45:
12:09:45:  <!-- Folding Slots -->
12:09:45:  <slot id='0' type='CPU'/>
12:09:45:  <slot id='1' type='GPU'>
12:09:45:    <pci-bus v='8'/>
12:09:45:    <pci-slot v='0'/>
12:09:45:  </slot>
12:09:45:  <slot id='2' type='GPU'>
12:09:45:    <pci-bus v='9'/>
12:09:45:    <pci-slot v='0'/>
12:09:45:  </slot>
12:09:45:  <slot id='3' type='GPU'>
12:09:45:    <pci-bus v='66'/>
12:09:45:    <pci-slot v='0'/>
12:09:45:  </slot>
12:09:45:  <slot id='4' type='GPU'>
12:09:45:    <pci-bus v='67'/>
12:09:45:    <pci-slot v='0'/>
12:09:45:  </slot>
12:09:45:</config>
12:09:45:Trying to access database...
12:09:46:Successfully acquired database lock
12:09:46:FS00:Initialized folding slot 00: cpu:28 - READY
12:09:46:FS01:Initialized folding slot 01: gpu:8:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
12:09:46:FS02:Initialized folding slot 02: gpu:9:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
12:09:46:FS03:Initialized folding slot 03: gpu:66:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
12:09:46:FS04:Initialized folding slot 04: gpu:67:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
12:09:46:Started thread 4 on PID 13572
12:09:46:Started thread 7 on PID 13572
12:09:46:Started thread 5 on PID 13572
12:09:46:Started thread 6 on PID 13572
12:09:46:Started thread 9 on PID 13572
12:09:46:Started thread 8 on PID 13572
12:09:46:WU06:FS01:Starting

JimboPalmer · Post by **JimboPalmer** » Thu Oct 22, 2020 9:04 pm

If you can set
12:09:45: <verbosity v='5'/>
back to
12:09:45: <verbosity v='3'/>
so someone can read your log.

Tuna_Ertemalp · Post by **Tuna_Ertemalp** » Thu Oct 22, 2020 9:29 pm

Really??

I guess I'll do that, restart the machine to start from a clean(er) state, and report back the next time this happens.

Tuna

Tuna_Ertemalp · Post by **Tuna_Ertemalp** » Fri Oct 23, 2020 5:17 am

Even with Verbosity=3, the full log is 274K. So, I am not going to put the whole thing here. If anybody needs it, tell me how you'd like to receive it. Instead, here is the config part from the top, which is way less than the 200 lines in my previous post on this thread that was generated with verbosity=5. If anybody needs more details on the config, please refer to that post. And, I am also including the log as filtered for FS04 for all WUs since reboot since Slot #4 is the one that crashed, after having processes & completed a number of other WUs.

Configuration:

Code: Select all

*********************** Log Started 2020-10-22T21:31:28Z ***********************
21:31:28:******************************* libFAH ********************************
21:31:28:           Date: Oct 9 2020
21:31:28:           Time: 10:37:10
21:31:28:       Revision: 06b99f7701e0d3f883dd14a78b459ad27da23809
21:31:28:         Branch: master
21:31:28:       Compiler: Visual C++ 2015
21:31:28:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
21:31:28:       Platform: win32 10
21:31:28:           Bits: 32
21:31:28:           Mode: Release
21:31:28:****************************** FAHClient ******************************
21:31:28:        Version: 7.6.20
21:31:28:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:31:28:      Copyright: 2020 foldingathome.org
21:31:28:       Homepage: https://foldingathome.org/
21:31:28:           Date: Oct 12 2020
21:31:28:           Time: 15:02:06
21:31:28:       Revision: c858fe2a8342bfa3e116e00b394d8dfa322ecd18
21:31:28:         Branch: master
21:31:28:       Compiler: Visual C++ 2015
21:31:28:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
21:31:28:       Platform: win32 10
21:31:28:           Bits: 32
21:31:28:           Mode: Release
21:31:28:         Config: C:\Users\Master\AppData\Roaming\FAHClient\config.xml
21:31:28:******************************** CBang ********************************
21:31:28:           Date: Oct 9 2020
21:31:28:           Time: 10:29:47
21:31:28:       Revision: ab0a6d9e35982b831a74cb2706c569fe46bac2af
21:31:28:         Branch: master
21:31:28:       Compiler: Visual C++ 2015
21:31:28:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
21:31:28:       Platform: win32 10
21:31:28:           Bits: 32
21:31:28:           Mode: Release
21:31:28:******************************* System ********************************
21:31:28:            CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
21:31:28:         CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
21:31:28:           CPUs: 32
21:31:28:         Memory: 31.88GiB
21:31:28:    Free Memory: 29.28GiB
21:31:28:        Threads: WINDOWS_THREADS
21:31:28:     OS Version: 6.2
21:31:28:    Has Battery: false
21:31:28:     On Battery: false
21:31:28:     UTC Offset: -7
21:31:28:            PID: 14024
21:31:28:            CWD: C:\Users\Master\AppData\Roaming\FAHClient
21:31:28:  Win32 Service: false
21:31:28:             OS: Windows 10 Pro
21:31:28:        OS Arch: AMD64
21:31:28:           GPUs: 4
21:31:28:          GPU 0: Bus:8 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
21:31:28:          GPU 1: Bus:9 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
21:31:28:          GPU 2: Bus:66 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
21:31:28:          GPU 3: Bus:67 Slot:0 Func:0 NVIDIA:8 GP102 [GeForce GTX 1080 Ti] 11380
21:31:28:  CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:11.1
21:31:28:  CUDA Device 1: Platform:0 Device:1 Bus:9 Slot:0 Compute:6.1 Driver:11.1
21:31:28:  CUDA Device 2: Platform:0 Device:2 Bus:66 Slot:0 Compute:6.1 Driver:11.1
21:31:28:  CUDA Device 3: Platform:0 Device:3 Bus:67 Slot:0 Compute:6.1 Driver:11.1
21:31:28:OpenCL Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:1.2 Driver:456.71
21:31:28:OpenCL Device 1: Platform:0 Device:1 Bus:9 Slot:0 Compute:1.2 Driver:456.71
21:31:28:OpenCL Device 2: Platform:0 Device:2 Bus:66 Slot:0 Compute:1.2 Driver:456.71
21:31:28:OpenCL Device 3: Platform:0 Device:3 Bus:67 Slot:0 Compute:1.2 Driver:456.71
21:31:28:***********************************************************************
21:31:28:<config>
21:31:28:  <!-- Folding Core -->
21:31:28:  <checkpoint v='5'/>
21:31:28:  <core-priority v='low'/>
21:31:28:
21:31:28:  <!-- Folding Slot Configuration -->
21:31:28:  <cause v='HIGH_PRIORITY'/>
21:31:28:
21:31:28:  <!-- HTTP Server -->
21:31:28:  <allow v='127.0.0.1 192.168.1.0/24'/>
21:31:28:
21:31:28:  <!-- Network -->
21:31:28:  <proxy v=':8080'/>
21:31:28:
21:31:28:  <!-- Remote Command Server -->
21:31:28:  <password v='*****'/>
21:31:28:
21:31:28:  <!-- Slot Control -->
21:31:28:  <power v='full'/>
21:31:28:
21:31:28:  <!-- User Information -->
21:31:28:  <passkey v='*****'/>
21:31:28:  <team v='999'/>
21:31:28:  <user v='Tuna_Ertemalp'/>
21:31:28:
21:31:28:  <!-- Folding Slots -->
21:31:28:  <slot id='0' type='CPU'/>
21:31:28:  <slot id='1' type='GPU'>
21:31:28:    <pci-bus v='8'/>
21:31:28:    <pci-slot v='0'/>
21:31:28:  </slot>
21:31:28:  <slot id='2' type='GPU'>
21:31:28:    <pci-bus v='9'/>
21:31:28:    <pci-slot v='0'/>
21:31:28:  </slot>
21:31:28:  <slot id='3' type='GPU'>
21:31:28:    <pci-bus v='66'/>
21:31:28:    <pci-slot v='0'/>
21:31:28:  </slot>
21:31:28:  <slot id='4' type='GPU'>
21:31:28:    <pci-bus v='67'/>
21:31:28:    <pci-slot v='0'/>
21:31:28:  </slot>
21:31:28:</config>
21:31:28:Trying to access database...
21:31:28:Successfully acquired database lock
21:31:28:FS00:Initialized folding slot 00: cpu:28 - READY
21:31:28:FS01:Initialized folding slot 01: gpu:8:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
21:31:28:FS02:Initialized folding slot 02: gpu:9:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
21:31:28:FS03:Initialized folding slot 03: gpu:66:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
21:31:28:FS04:Initialized folding slot 04: gpu:67:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
21:31:28:WU01:FS04:Starting
21:31:28:WU01:FS04:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Master\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 14024 -checkpoint 5 -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor nvidia -gpu 3 -gpu-usage 100
21:31:28:WU01:FS04:Started FahCore on PID 14136
21:31:28:WU01:FS04:Core PID:14160
21:31:28:WU01:FS04:FahCore 0x22 started

Slot #4 log:

Code: Select all

*********************** Log Started 2020-10-22T21:31:28Z ***********************
21:31:28:FS04:Initialized folding slot 04: gpu:67:0 GP102 [GeForce GTX 1080 Ti] 11380 - READY
21:31:28:WU01:FS04:Starting
21:31:28:WU01:FS04:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Master\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 14024 -checkpoint 5 -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor nvidia -gpu 3 -gpu-usage 100
21:31:28:WU01:FS04:Started FahCore on PID 14136
21:31:28:WU01:FS04:Core PID:14160
21:31:28:WU01:FS04:FahCore 0x22 started
21:31:29:WU01:FS04:0x22:*********************** Log Started 2020-10-22T21:31:29Z ***********************
21:31:29:WU01:FS04:0x22:*************************** Core22 Folding@home Core ***************************
21:31:29:WU01:FS04:0x22:       Core: Core22
21:31:29:WU01:FS04:0x22:       Type: 0x22
21:31:29:WU01:FS04:0x22:    Version: 0.0.13
21:31:29:WU01:FS04:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:31:29:WU01:FS04:0x22:  Copyright: 2020 foldingathome.org
21:31:29:WU01:FS04:0x22:   Homepage: https://foldingathome.org/
21:31:29:WU01:FS04:0x22:       Date: Sep 19 2020
21:31:29:WU01:FS04:0x22:       Time: 02:35:58
21:31:29:WU01:FS04:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
21:31:29:WU01:FS04:0x22:     Branch: core22-0.0.13
21:31:29:WU01:FS04:0x22:   Compiler: Visual C++ 2015
21:31:29:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
21:31:29:WU01:FS04:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
21:31:29:WU01:FS04:0x22:   Platform: win32 10
21:31:29:WU01:FS04:0x22:       Bits: 64
21:31:29:WU01:FS04:0x22:       Mode: Release
21:31:29:WU01:FS04:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
21:31:29:WU01:FS04:0x22:             <peastman@stanford.edu>
21:31:29:WU01:FS04:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 14136 -checkpoint 5
21:31:29:WU01:FS04:0x22:             -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor
21:31:29:WU01:FS04:0x22:             nvidia -gpu 3 -gpu-usage 100
21:31:29:WU01:FS04:0x22:************************************ libFAH ************************************
21:31:29:WU01:FS04:0x22:       Date: Sep 7 2020
21:31:29:WU01:FS04:0x22:       Time: 19:09:56
21:31:29:WU01:FS04:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
21:31:29:WU01:FS04:0x22:     Branch: HEAD
21:31:29:WU01:FS04:0x22:   Compiler: Visual C++ 2015
21:31:29:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
21:31:29:WU01:FS04:0x22:   Platform: win32 10
21:31:29:WU01:FS04:0x22:       Bits: 64
21:31:29:WU01:FS04:0x22:       Mode: Release
21:31:29:WU01:FS04:0x22:************************************ CBang *************************************
21:31:29:WU01:FS04:0x22:       Date: Sep 7 2020
21:31:29:WU01:FS04:0x22:       Time: 19:08:30
21:31:29:WU01:FS04:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
21:31:29:WU01:FS04:0x22:     Branch: HEAD
21:31:29:WU01:FS04:0x22:   Compiler: Visual C++ 2015
21:31:29:WU01:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
21:31:30:WU01:FS04:0x22:   Platform: win32 10
21:31:30:WU01:FS04:0x22:       Bits: 64
21:31:30:WU01:FS04:0x22:       Mode: Release
21:31:30:WU01:FS04:0x22:************************************ System ************************************
21:31:30:WU01:FS04:0x22:        CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
21:31:30:WU01:FS04:0x22:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
21:31:30:WU01:FS04:0x22:       CPUs: 32
21:31:30:WU01:FS04:0x22:     Memory: 31.88GiB
21:31:30:WU01:FS04:0x22:Free Memory: 29.26GiB
21:31:30:WU01:FS04:0x22:    Threads: WINDOWS_THREADS
21:31:30:WU01:FS04:0x22: OS Version: 6.2
21:31:30:WU01:FS04:0x22:Has Battery: false
21:31:30:WU01:FS04:0x22: On Battery: false
21:31:30:WU01:FS04:0x22: UTC Offset: -7
21:31:30:WU01:FS04:0x22:        PID: 14160
21:31:30:WU01:FS04:0x22:        CWD: C:\Users\Master\AppData\Roaming\FAHClient\work
21:31:30:WU01:FS04:0x22:************************************ OpenMM ************************************
21:31:30:WU01:FS04:0x22:   Revision: 189320d0
21:31:30:WU01:FS04:0x22:********************************************************************************
21:31:30:WU01:FS04:0x22:Project: 14904 (Run 417, Clone 0, Gen 62)
21:31:30:WU01:FS04:0x22:Unit: 0x0000005981d59d695f4ec9e340f2570a
21:31:30:WU01:FS04:0x22:Digital signatures verified
21:31:30:WU01:FS04:0x22:Folding@home GPU Core22 Folding@home Core
21:31:30:WU01:FS04:0x22:Version 0.0.13
21:31:30:WU01:FS04:0x22:  Checkpoint write interval: 100000 steps (5%) [20 total]
21:31:30:WU01:FS04:0x22:  JSON viewer frame write interval: 20000 steps (1%) [100 total]
21:31:30:WU01:FS04:0x22:  XTC frame write interval: 50000 steps (2.5%) [40 total]
21:31:30:WU01:FS04:0x22:  Global context and integrator variables write interval: disabled
21:31:30:WU01:FS04:0x22:There are 4 platforms available.
21:31:30:WU01:FS04:0x22:Platform 0: Reference
21:31:30:WU01:FS04:0x22:Platform 1: CPU
21:31:30:WU01:FS04:0x22:Platform 2: OpenCL
21:31:30:WU01:FS04:0x22:  opencl-device 3 specified
21:31:30:WU01:FS04:0x22:Platform 3: CUDA
21:31:30:WU01:FS04:0x22:  cuda-device 3 specified
21:31:41:WU01:FS04:0x22:Attempting to create CUDA context:
21:31:41:WU01:FS04:0x22:  Configuring platform CUDA
21:31:47:WU01:FS04:0x22:  Using CUDA and gpu 3
21:31:48:WU01:FS04:0x22:Completed 800000 out of 2000000 steps (40%)
21:33:01:WU01:FS04:0x22:Completed 820000 out of 2000000 steps (41%)
21:34:14:WU01:FS04:0x22:Completed 840000 out of 2000000 steps (42%)
21:35:27:WU01:FS04:0x22:Completed 860000 out of 2000000 steps (43%)
21:36:39:WU01:FS04:0x22:Completed 880000 out of 2000000 steps (44%)
21:37:51:WU01:FS04:0x22:Completed 900000 out of 2000000 steps (45%)
21:37:52:WU01:FS04:0x22:Checkpoint completed at step 900000
21:39:04:WU01:FS04:0x22:Completed 920000 out of 2000000 steps (46%)
21:40:17:WU01:FS04:0x22:Completed 940000 out of 2000000 steps (47%)
21:41:29:WU01:FS04:0x22:Completed 960000 out of 2000000 steps (48%)
21:42:40:WU01:FS04:0x22:Completed 980000 out of 2000000 steps (49%)
21:43:52:WU01:FS04:0x22:Completed 1000000 out of 2000000 steps (50%)
21:43:53:WU01:FS04:0x22:Checkpoint completed at step 1000000
21:45:05:WU01:FS04:0x22:Completed 1020000 out of 2000000 steps (51%)
21:46:17:WU01:FS04:0x22:Completed 1040000 out of 2000000 steps (52%)
21:47:28:WU01:FS04:0x22:Completed 1060000 out of 2000000 steps (53%)
21:48:39:WU01:FS04:0x22:Completed 1080000 out of 2000000 steps (54%)
21:49:51:WU01:FS04:0x22:Completed 1100000 out of 2000000 steps (55%)
21:49:53:WU01:FS04:0x22:Checkpoint completed at step 1100000
21:51:06:WU01:FS04:0x22:Completed 1120000 out of 2000000 steps (56%)
21:52:19:WU01:FS04:0x22:Completed 1140000 out of 2000000 steps (57%)
21:53:32:WU01:FS04:0x22:Completed 1160000 out of 2000000 steps (58%)
21:54:45:WU01:FS04:0x22:Completed 1180000 out of 2000000 steps (59%)
21:55:57:WU01:FS04:0x22:Completed 1200000 out of 2000000 steps (60%)
21:55:58:WU01:FS04:0x22:Checkpoint completed at step 1200000
21:57:09:WU01:FS04:0x22:Completed 1220000 out of 2000000 steps (61%)
21:58:21:WU01:FS04:0x22:Completed 1240000 out of 2000000 steps (62%)
21:59:31:WU01:FS04:0x22:Completed 1260000 out of 2000000 steps (63%)
22:00:57:WU01:FS04:0x22:Completed 1280000 out of 2000000 steps (64%)
22:02:29:WU01:FS04:0x22:Completed 1300000 out of 2000000 steps (65%)
22:02:31:WU01:FS04:0x22:Checkpoint completed at step 1300000
22:04:02:WU01:FS04:0x22:Completed 1320000 out of 2000000 steps (66%)
22:05:33:WU01:FS04:0x22:Completed 1340000 out of 2000000 steps (67%)
22:07:05:WU01:FS04:0x22:Completed 1360000 out of 2000000 steps (68%)
22:08:36:WU01:FS04:0x22:Completed 1380000 out of 2000000 steps (69%)
22:10:06:WU01:FS04:0x22:Completed 1400000 out of 2000000 steps (70%)
22:10:08:WU01:FS04:0x22:Checkpoint completed at step 1400000
22:11:39:WU01:FS04:0x22:Completed 1420000 out of 2000000 steps (71%)
22:13:09:WU01:FS04:0x22:Completed 1440000 out of 2000000 steps (72%)
22:14:40:WU01:FS04:0x22:Completed 1460000 out of 2000000 steps (73%)
22:16:10:WU01:FS04:0x22:Completed 1480000 out of 2000000 steps (74%)
22:17:41:WU01:FS04:0x22:Completed 1500000 out of 2000000 steps (75%)
22:17:43:WU01:FS04:0x22:Checkpoint completed at step 1500000
22:19:14:WU01:FS04:0x22:Completed 1520000 out of 2000000 steps (76%)
22:20:44:WU01:FS04:0x22:Completed 1540000 out of 2000000 steps (77%)
22:22:15:WU01:FS04:0x22:Completed 1560000 out of 2000000 steps (78%)
22:23:45:WU01:FS04:0x22:Completed 1580000 out of 2000000 steps (79%)
22:25:14:WU01:FS04:0x22:Completed 1600000 out of 2000000 steps (80%)
22:25:17:WU01:FS04:0x22:Checkpoint completed at step 1600000
22:26:47:WU01:FS04:0x22:Completed 1620000 out of 2000000 steps (81%)
22:28:17:WU01:FS04:0x22:Completed 1640000 out of 2000000 steps (82%)
22:29:48:WU01:FS04:0x22:Completed 1660000 out of 2000000 steps (83%)
22:31:18:WU01:FS04:0x22:Completed 1680000 out of 2000000 steps (84%)
22:32:47:WU01:FS04:0x22:Completed 1700000 out of 2000000 steps (85%)
22:32:50:WU01:FS04:0x22:Checkpoint completed at step 1700000
22:34:20:WU01:FS04:0x22:Completed 1720000 out of 2000000 steps (86%)
22:35:51:WU01:FS04:0x22:Completed 1740000 out of 2000000 steps (87%)
22:37:21:WU01:FS04:0x22:Completed 1760000 out of 2000000 steps (88%)
22:38:52:WU01:FS04:0x22:Completed 1780000 out of 2000000 steps (89%)
22:40:21:WU01:FS04:0x22:Completed 1800000 out of 2000000 steps (90%)
22:40:24:WU01:FS04:0x22:Checkpoint completed at step 1800000
22:41:54:WU01:FS04:0x22:Completed 1820000 out of 2000000 steps (91%)
22:43:24:WU01:FS04:0x22:Completed 1840000 out of 2000000 steps (92%)
22:44:53:WU01:FS04:0x22:Completed 1860000 out of 2000000 steps (93%)
22:46:24:WU01:FS04:0x22:Completed 1880000 out of 2000000 steps (94%)
22:47:54:WU01:FS04:0x22:Completed 1900000 out of 2000000 steps (95%)
22:47:56:WU01:FS04:0x22:Checkpoint completed at step 1900000
22:49:27:WU01:FS04:0x22:Completed 1920000 out of 2000000 steps (96%)
22:50:57:WU01:FS04:0x22:Completed 1940000 out of 2000000 steps (97%)
22:52:28:WU01:FS04:0x22:Completed 1960000 out of 2000000 steps (98%)
22:53:58:WU01:FS04:0x22:Completed 1980000 out of 2000000 steps (99%)
22:53:59:WU05:FS04:Connecting to assign1.foldingathome.org:80
22:53:59:WU05:FS04:Assigned to work server 18.188.125.154
22:53:59:WU05:FS04:Requesting new work unit for slot 04: gpu:67:0 GP102 [GeForce GTX 1080 Ti] 11380 - RUNNING from 18.188.125.154
22:53:59:WU05:FS04:Connecting to 18.188.125.154:8080
22:54:01:WU05:FS04:Downloading 12.01MiB
22:54:04:WU05:FS04:Download complete
22:54:04:WU05:FS04:Received Unit: id:05 state:DOWNLOAD error:NO_ERROR project:17308 run:0 clone:5567 gen:12 core:0x22 unit:0x0000000e12bc7d9a5f8f11cd821d1f53
22:55:28:WU01:FS04:0x22:Completed 2000000 out of 2000000 steps (100%)
22:55:28:WU01:FS04:0x22:Average performance: 19.115 ns/day
22:55:30:WU01:FS04:0x22:Checkpoint completed at step 2000000
22:55:37:WU01:FS04:0x22:Saving result file ..\logfile_01.txt
22:55:37:WU01:FS04:0x22:Saving result file checkpointIntegrator.xml
22:55:37:WU01:FS04:0x22:Saving result file checkpointState.xml
22:55:40:WU01:FS04:0x22:Saving result file positions.xtc
22:55:40:WU01:FS04:0x22:Saving result file science.log
22:55:40:WU01:FS04:0x22:Folding@home Core Shutdown: FINISHED_UNIT
22:55:41:WU01:FS04:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:55:41:WU01:FS04:Sending unit results: id:01 state:SEND error:NO_ERROR project:14904 run:417 clone:0 gen:62 core:0x22 unit:0x0000005981d59d695f4ec9e340f2570a
22:55:41:WU01:FS04:Uploading 11.28MiB to 129.213.157.105
22:55:41:WU01:FS04:Connecting to 129.213.157.105:8080
22:55:41:WU05:FS04:Starting
22:55:41:WU05:FS04:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Master\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 05 -suffix 01 -version 706 -lifeline 14024 -checkpoint 5 -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor nvidia -gpu 3 -gpu-usage 100
22:55:41:WU05:FS04:Started FahCore on PID 8120
22:55:41:WU05:FS04:Core PID:3896
22:55:41:WU05:FS04:FahCore 0x22 started
22:55:42:WU05:FS04:0x22:*********************** Log Started 2020-10-22T22:55:41Z ***********************
22:55:42:WU05:FS04:0x22:*************************** Core22 Folding@home Core ***************************
22:55:42:WU05:FS04:0x22:       Core: Core22
22:55:42:WU05:FS04:0x22:       Type: 0x22
22:55:42:WU05:FS04:0x22:    Version: 0.0.13
22:55:42:WU05:FS04:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:55:42:WU05:FS04:0x22:  Copyright: 2020 foldingathome.org
22:55:42:WU05:FS04:0x22:   Homepage: https://foldingathome.org/
22:55:42:WU05:FS04:0x22:       Date: Sep 19 2020
22:55:42:WU05:FS04:0x22:       Time: 02:35:58
22:55:42:WU05:FS04:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
22:55:42:WU05:FS04:0x22:     Branch: core22-0.0.13
22:55:42:WU05:FS04:0x22:   Compiler: Visual C++ 2015
22:55:42:WU05:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:55:42:WU05:FS04:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
22:55:42:WU05:FS04:0x22:   Platform: win32 10
22:55:42:WU05:FS04:0x22:       Bits: 64
22:55:42:WU05:FS04:0x22:       Mode: Release
22:55:42:WU05:FS04:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
22:55:42:WU05:FS04:0x22:             <peastman@stanford.edu>
22:55:42:WU05:FS04:0x22:       Args: -dir 05 -suffix 01 -version 706 -lifeline 8120 -checkpoint 5
22:55:42:WU05:FS04:0x22:             -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor
22:55:42:WU05:FS04:0x22:             nvidia -gpu 3 -gpu-usage 100
22:55:42:WU05:FS04:0x22:************************************ libFAH ************************************
22:55:42:WU05:FS04:0x22:       Date: Sep 7 2020
22:55:42:WU05:FS04:0x22:       Time: 19:09:56
22:55:42:WU05:FS04:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
22:55:42:WU05:FS04:0x22:     Branch: HEAD
22:55:42:WU05:FS04:0x22:   Compiler: Visual C++ 2015
22:55:42:WU05:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:55:42:WU05:FS04:0x22:   Platform: win32 10
22:55:42:WU05:FS04:0x22:       Bits: 64
22:55:42:WU05:FS04:0x22:       Mode: Release
22:55:42:WU05:FS04:0x22:************************************ CBang *************************************
22:55:42:WU05:FS04:0x22:       Date: Sep 7 2020
22:55:42:WU05:FS04:0x22:       Time: 19:08:30
22:55:42:WU05:FS04:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
22:55:42:WU05:FS04:0x22:     Branch: HEAD
22:55:42:WU05:FS04:0x22:   Compiler: Visual C++ 2015
22:55:42:WU05:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:55:42:WU05:FS04:0x22:   Platform: win32 10
22:55:42:WU05:FS04:0x22:       Bits: 64
22:55:42:WU05:FS04:0x22:       Mode: Release
22:55:42:WU05:FS04:0x22:************************************ System ************************************
22:55:42:WU05:FS04:0x22:        CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
22:55:42:WU05:FS04:0x22:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
22:55:42:WU05:FS04:0x22:       CPUs: 32
22:55:42:WU05:FS04:0x22:     Memory: 31.88GiB
22:55:42:WU05:FS04:0x22:Free Memory: 24.71GiB
22:55:42:WU05:FS04:0x22:    Threads: WINDOWS_THREADS
22:55:42:WU05:FS04:0x22: OS Version: 6.2
22:55:42:WU05:FS04:0x22:Has Battery: false
22:55:42:WU05:FS04:0x22: On Battery: false
22:55:42:WU05:FS04:0x22: UTC Offset: -7
22:55:42:WU05:FS04:0x22:        PID: 3896
22:55:42:WU05:FS04:0x22:        CWD: C:\Users\Master\AppData\Roaming\FAHClient\work
22:55:42:WU05:FS04:0x22:************************************ OpenMM ************************************
22:55:42:WU05:FS04:0x22:   Revision: 189320d0
22:55:42:WU05:FS04:0x22:********************************************************************************
22:55:42:WU05:FS04:0x22:Project: 17308 (Run 0, Clone 5567, Gen 12)
22:55:42:WU05:FS04:0x22:Unit: 0x0000000e12bc7d9a5f8f11cd821d1f53
22:55:42:WU05:FS04:0x22:Reading tar file core.xml
22:55:42:WU05:FS04:0x22:Reading tar file integrator.xml.bz2
22:55:42:WU05:FS04:0x22:Reading tar file state.xml.bz2
22:55:42:WU05:FS04:0x22:Reading tar file system.xml.bz2
22:55:42:WU05:FS04:0x22:Digital signatures verified
22:55:42:WU05:FS04:0x22:Folding@home GPU Core22 Folding@home Core
22:55:42:WU05:FS04:0x22:Version 0.0.13
22:55:42:WU05:FS04:0x22:  Checkpoint write interval: 62500 steps (5%) [20 total]
22:55:42:WU05:FS04:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
22:55:42:WU05:FS04:0x22:  XTC frame write interval: 125000 steps (10%) [10 total]
22:55:42:WU05:FS04:0x22:  Global context and integrator variables write interval: disabled
22:55:42:WU05:FS04:0x22:There are 4 platforms available.
22:55:42:WU05:FS04:0x22:Platform 0: Reference
22:55:42:WU05:FS04:0x22:Platform 1: CPU
22:55:42:WU05:FS04:0x22:Platform 2: OpenCL
22:55:42:WU05:FS04:0x22:  opencl-device 3 specified
22:55:42:WU05:FS04:0x22:Platform 3: CUDA
22:55:42:WU05:FS04:0x22:  cuda-device 3 specified
22:55:47:WU01:FS04:Upload 32.68%
22:55:53:WU01:FS04:Upload 68.13%
22:55:58:WU01:FS04:Upload complete
22:55:59:WU01:FS04:Server responded WORK_ACK (400)
22:55:59:WU01:FS04:Final credit estimate, 76545.00 points
22:55:59:WU01:FS04:Cleaning up
22:56:02:WU05:FS04:0x22:Attempting to create CUDA context:
22:56:02:WU05:FS04:0x22:  Configuring platform CUDA
22:56:10:WU05:FS04:0x22:  Using CUDA and gpu 3
22:56:10:WU05:FS04:0x22:Completed 0 out of 1250000 steps (0%)
22:56:11:WU05:FS04:0x22:Checkpoint completed at step 0
22:57:20:WU05:FS04:0x22:Completed 12500 out of 1250000 steps (1%)
22:58:28:WU05:FS04:0x22:Completed 25000 out of 1250000 steps (2%)
22:59:37:WU05:FS04:0x22:Completed 37500 out of 1250000 steps (3%)
23:00:46:WU05:FS04:0x22:Completed 50000 out of 1250000 steps (4%)
23:01:55:WU05:FS04:0x22:Completed 62500 out of 1250000 steps (5%)
23:01:57:WU05:FS04:0x22:Checkpoint completed at step 62500
23:03:05:WU05:FS04:0x22:Completed 75000 out of 1250000 steps (6%)
23:04:14:WU05:FS04:0x22:Completed 87500 out of 1250000 steps (7%)
23:05:21:WU05:FS04:0x22:Completed 100000 out of 1250000 steps (8%)
23:06:28:WU05:FS04:0x22:Completed 112500 out of 1250000 steps (9%)
23:07:35:WU05:FS04:0x22:Completed 125000 out of 1250000 steps (10%)
23:07:36:WU05:FS04:0x22:Checkpoint completed at step 125000
23:08:44:WU05:FS04:0x22:Completed 137500 out of 1250000 steps (11%)
23:09:51:WU05:FS04:0x22:Completed 150000 out of 1250000 steps (12%)
23:10:59:WU05:FS04:0x22:Completed 162500 out of 1250000 steps (13%)
23:12:06:WU05:FS04:0x22:Completed 175000 out of 1250000 steps (14%)
23:13:14:WU05:FS04:0x22:Completed 187500 out of 1250000 steps (15%)
23:13:15:WU05:FS04:0x22:Checkpoint completed at step 187500
23:14:23:WU05:FS04:0x22:Completed 200000 out of 1250000 steps (16%)
23:15:30:WU05:FS04:0x22:Completed 212500 out of 1250000 steps (17%)
23:16:38:WU05:FS04:0x22:Completed 225000 out of 1250000 steps (18%)
23:17:46:WU05:FS04:0x22:Completed 237500 out of 1250000 steps (19%)
23:18:53:WU05:FS04:0x22:Completed 250000 out of 1250000 steps (20%)
23:18:55:WU05:FS04:0x22:Checkpoint completed at step 250000
23:20:03:WU05:FS04:0x22:Completed 262500 out of 1250000 steps (21%)
23:21:10:WU05:FS04:0x22:Completed 275000 out of 1250000 steps (22%)
23:22:17:WU05:FS04:0x22:Completed 287500 out of 1250000 steps (23%)
23:23:25:WU05:FS04:0x22:Completed 300000 out of 1250000 steps (24%)
23:24:32:WU05:FS04:0x22:Completed 312500 out of 1250000 steps (25%)
23:24:34:WU05:FS04:0x22:Checkpoint completed at step 312500
23:25:40:WU05:FS04:0x22:Completed 325000 out of 1250000 steps (26%)
23:26:47:WU05:FS04:0x22:Completed 337500 out of 1250000 steps (27%)
23:27:53:WU05:FS04:0x22:Completed 350000 out of 1250000 steps (28%)
23:29:00:WU05:FS04:0x22:Completed 362500 out of 1250000 steps (29%)
23:30:07:WU05:FS04:0x22:Completed 375000 out of 1250000 steps (30%)
23:30:09:WU05:FS04:0x22:Checkpoint completed at step 375000
23:31:15:WU05:FS04:0x22:Completed 387500 out of 1250000 steps (31%)
23:32:22:WU05:FS04:0x22:Completed 400000 out of 1250000 steps (32%)
23:33:29:WU05:FS04:0x22:Completed 412500 out of 1250000 steps (33%)
23:34:36:WU05:FS04:0x22:Completed 425000 out of 1250000 steps (34%)
23:35:43:WU05:FS04:0x22:Completed 437500 out of 1250000 steps (35%)
23:35:44:WU05:FS04:0x22:Checkpoint completed at step 437500
23:36:51:WU05:FS04:0x22:Completed 450000 out of 1250000 steps (36%)
23:37:58:WU05:FS04:0x22:Completed 462500 out of 1250000 steps (37%)
23:39:05:WU05:FS04:0x22:Completed 475000 out of 1250000 steps (38%)
23:40:13:WU05:FS04:0x22:Completed 487500 out of 1250000 steps (39%)
23:41:20:WU05:FS04:0x22:Completed 500000 out of 1250000 steps (40%)
23:41:22:WU05:FS04:0x22:Checkpoint completed at step 500000
23:42:29:WU05:FS04:0x22:Completed 512500 out of 1250000 steps (41%)
23:43:37:WU05:FS04:0x22:Completed 525000 out of 1250000 steps (42%)
23:44:44:WU05:FS04:0x22:Completed 537500 out of 1250000 steps (43%)
23:45:47:WU05:FS04:0x22:Completed 550000 out of 1250000 steps (44%)
23:46:45:WU05:FS04:0x22:Completed 562500 out of 1250000 steps (45%)
23:46:46:WU05:FS04:0x22:Checkpoint completed at step 562500
23:47:43:WU05:FS04:0x22:Completed 575000 out of 1250000 steps (46%)
23:48:41:WU05:FS04:0x22:Completed 587500 out of 1250000 steps (47%)
23:49:39:WU05:FS04:0x22:Completed 600000 out of 1250000 steps (48%)
23:50:37:WU05:FS04:0x22:Completed 612500 out of 1250000 steps (49%)
23:51:36:WU05:FS04:0x22:Completed 625000 out of 1250000 steps (50%)
23:51:37:WU05:FS04:0x22:Checkpoint completed at step 625000
23:52:35:WU05:FS04:0x22:Completed 637500 out of 1250000 steps (51%)
23:53:34:WU05:FS04:0x22:Completed 650000 out of 1250000 steps (52%)
23:54:32:WU05:FS04:0x22:Completed 662500 out of 1250000 steps (53%)
23:55:31:WU05:FS04:0x22:Completed 675000 out of 1250000 steps (54%)
23:56:29:WU05:FS04:0x22:Completed 687500 out of 1250000 steps (55%)
23:56:31:WU05:FS04:0x22:Checkpoint completed at step 687500
23:57:30:WU05:FS04:0x22:Completed 700000 out of 1250000 steps (56%)
23:58:29:WU05:FS04:0x22:Completed 712500 out of 1250000 steps (57%)
23:59:29:WU05:FS04:0x22:Completed 725000 out of 1250000 steps (58%)
00:00:28:WU05:FS04:0x22:Completed 737500 out of 1250000 steps (59%)
00:01:27:WU05:FS04:0x22:Completed 750000 out of 1250000 steps (60%)
00:01:29:WU05:FS04:0x22:Checkpoint completed at step 750000
00:02:28:WU05:FS04:0x22:Completed 762500 out of 1250000 steps (61%)
00:03:27:WU05:FS04:0x22:Completed 775000 out of 1250000 steps (62%)
00:04:27:WU05:FS04:0x22:Completed 787500 out of 1250000 steps (63%)
00:05:27:WU05:FS04:0x22:Completed 800000 out of 1250000 steps (64%)
00:06:27:WU05:FS04:0x22:Completed 812500 out of 1250000 steps (65%)
00:06:28:WU05:FS04:0x22:Checkpoint completed at step 812500
00:07:27:WU05:FS04:0x22:Completed 825000 out of 1250000 steps (66%)
00:08:27:WU05:FS04:0x22:Completed 837500 out of 1250000 steps (67%)
00:09:27:WU05:FS04:0x22:Completed 850000 out of 1250000 steps (68%)
00:10:26:WU05:FS04:0x22:Completed 862500 out of 1250000 steps (69%)
00:11:26:WU05:FS04:0x22:Completed 875000 out of 1250000 steps (70%)
00:11:27:WU05:FS04:0x22:Checkpoint completed at step 875000
00:12:27:WU05:FS04:0x22:Completed 887500 out of 1250000 steps (71%)
00:13:27:WU05:FS04:0x22:Completed 900000 out of 1250000 steps (72%)
00:14:26:WU05:FS04:0x22:Completed 912500 out of 1250000 steps (73%)
00:15:26:WU05:FS04:0x22:Completed 925000 out of 1250000 steps (74%)
00:16:25:WU05:FS04:0x22:Completed 937500 out of 1250000 steps (75%)
00:16:27:WU05:FS04:0x22:Checkpoint completed at step 937500
00:17:26:WU05:FS04:0x22:Completed 950000 out of 1250000 steps (76%)
00:18:26:WU05:FS04:0x22:Completed 962500 out of 1250000 steps (77%)
00:19:25:WU05:FS04:0x22:Completed 975000 out of 1250000 steps (78%)
00:20:25:WU05:FS04:0x22:Completed 987500 out of 1250000 steps (79%)
00:21:23:WU05:FS04:0x22:Completed 1000000 out of 1250000 steps (80%)
00:21:25:WU05:FS04:0x22:Checkpoint completed at step 1000000
00:22:23:WU05:FS04:0x22:Completed 1012500 out of 1250000 steps (81%)
00:23:21:WU05:FS04:0x22:Completed 1025000 out of 1250000 steps (82%)
00:24:20:WU05:FS04:0x22:Completed 1037500 out of 1250000 steps (83%)
00:25:18:WU05:FS04:0x22:Completed 1050000 out of 1250000 steps (84%)
00:26:17:WU05:FS04:0x22:Completed 1062500 out of 1250000 steps (85%)
00:26:18:WU05:FS04:0x22:Checkpoint completed at step 1062500
00:27:17:WU05:FS04:0x22:Completed 1075000 out of 1250000 steps (86%)
00:28:15:WU05:FS04:0x22:Completed 1087500 out of 1250000 steps (87%)
00:29:14:WU05:FS04:0x22:Completed 1100000 out of 1250000 steps (88%)
00:30:12:WU05:FS04:0x22:Completed 1112500 out of 1250000 steps (89%)
00:31:11:WU05:FS04:0x22:Completed 1125000 out of 1250000 steps (90%)
00:31:12:WU05:FS04:0x22:Checkpoint completed at step 1125000
00:32:10:WU05:FS04:0x22:Completed 1137500 out of 1250000 steps (91%)
00:33:09:WU05:FS04:0x22:Completed 1150000 out of 1250000 steps (92%)
00:34:08:WU05:FS04:0x22:Completed 1162500 out of 1250000 steps (93%)
00:35:06:WU05:FS04:0x22:Completed 1175000 out of 1250000 steps (94%)
00:36:05:WU05:FS04:0x22:Completed 1187500 out of 1250000 steps (95%)
00:36:06:WU05:FS04:0x22:Checkpoint completed at step 1187500
00:37:05:WU05:FS04:0x22:Completed 1200000 out of 1250000 steps (96%)
00:38:03:WU05:FS04:0x22:Completed 1212500 out of 1250000 steps (97%)
00:39:01:WU05:FS04:0x22:Completed 1225000 out of 1250000 steps (98%)
00:39:59:WU05:FS04:0x22:Completed 1237500 out of 1250000 steps (99%)
00:39:59:WU02:FS04:Connecting to assign1.foldingathome.org:80
00:39:59:WU02:FS04:Assigned to work server 129.213.157.105
00:39:59:WU02:FS04:Requesting new work unit for slot 04: gpu:67:0 GP102 [GeForce GTX 1080 Ti] 11380 - RUNNING from 129.213.157.105
00:39:59:WU02:FS04:Connecting to 129.213.157.105:8080
00:40:05:WU02:FS04:Downloading 20.17MiB
00:40:10:WU02:FS04:Download complete
00:40:11:WU02:FS04:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14907 run:229 clone:5 gen:91 core:0x22 unit:0x0000007681d59d695f52603285fa6f2b
00:40:56:WU05:FS04:0x22:Completed 1250000 out of 1250000 steps (100%)
00:40:56:WU05:FS04:0x22:Average performance: 148.966 ns/day
00:40:58:WU05:FS04:0x22:Checkpoint completed at step 1250000
00:41:04:WU05:FS04:0x22:Saving result file ..\logfile_01.txt
00:41:04:WU05:FS04:0x22:Saving result file checkpointIntegrator.xml.bz2
00:41:04:WU05:FS04:0x22:Saving result file checkpointState.xml.bz2
00:41:05:WU05:FS04:0x22:Saving result file positions.xtc
00:41:05:WU05:FS04:0x22:Saving result file science.log
00:41:05:WU05:FS04:0x22:Folding@home Core Shutdown: FINISHED_UNIT
00:41:06:WU05:FS04:FahCore returned: FINISHED_UNIT (100 = 0x64)
00:41:06:WU05:FS04:Sending unit results: id:05 state:SEND error:NO_ERROR project:17308 run:0 clone:5567 gen:12 core:0x22 unit:0x0000000e12bc7d9a5f8f11cd821d1f53
00:41:06:WU05:FS04:Uploading 11.56MiB to 18.188.125.154
00:41:06:WU05:FS04:Connecting to 18.188.125.154:8080
00:41:06:WU02:FS04:Starting
00:41:06:WU02:FS04:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Master\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 14024 -checkpoint 5 -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor nvidia -gpu 3 -gpu-usage 100
00:41:06:WU02:FS04:Started FahCore on PID 11616
00:41:06:WU02:FS04:Core PID:15220
00:41:06:WU02:FS04:FahCore 0x22 started
00:41:06:WU02:FS04:0x22:*********************** Log Started 2020-10-23T00:41:06Z ***********************
00:41:06:WU02:FS04:0x22:*************************** Core22 Folding@home Core ***************************
00:41:06:WU02:FS04:0x22:       Core: Core22
00:41:06:WU02:FS04:0x22:       Type: 0x22
00:41:06:WU02:FS04:0x22:    Version: 0.0.13
00:41:06:WU02:FS04:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:41:06:WU02:FS04:0x22:  Copyright: 2020 foldingathome.org
00:41:06:WU02:FS04:0x22:   Homepage: https://foldingathome.org/
00:41:06:WU02:FS04:0x22:       Date: Sep 19 2020
00:41:06:WU02:FS04:0x22:       Time: 02:35:58
00:41:06:WU02:FS04:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
00:41:06:WU02:FS04:0x22:     Branch: core22-0.0.13
00:41:06:WU02:FS04:0x22:   Compiler: Visual C++ 2015
00:41:06:WU02:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
00:41:06:WU02:FS04:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
00:41:06:WU02:FS04:0x22:   Platform: win32 10
00:41:06:WU02:FS04:0x22:       Bits: 64
00:41:06:WU02:FS04:0x22:       Mode: Release
00:41:06:WU02:FS04:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
00:41:06:WU02:FS04:0x22:             <peastman@stanford.edu>
00:41:06:WU02:FS04:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 11616 -checkpoint 5
00:41:06:WU02:FS04:0x22:             -opencl-platform 0 -opencl-device 3 -cuda-device 3 -gpu-vendor
00:41:06:WU02:FS04:0x22:             nvidia -gpu 3 -gpu-usage 100
00:41:06:WU02:FS04:0x22:************************************ libFAH ************************************
00:41:06:WU02:FS04:0x22:       Date: Sep 7 2020
00:41:06:WU02:FS04:0x22:       Time: 19:09:56
00:41:06:WU02:FS04:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
00:41:06:WU02:FS04:0x22:     Branch: HEAD
00:41:06:WU02:FS04:0x22:   Compiler: Visual C++ 2015
00:41:06:WU02:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
00:41:06:WU02:FS04:0x22:   Platform: win32 10
00:41:06:WU02:FS04:0x22:       Bits: 64
00:41:06:WU02:FS04:0x22:       Mode: Release
00:41:06:WU02:FS04:0x22:************************************ CBang *************************************
00:41:06:WU02:FS04:0x22:       Date: Sep 7 2020
00:41:06:WU02:FS04:0x22:       Time: 19:08:30
00:41:06:WU02:FS04:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
00:41:06:WU02:FS04:0x22:     Branch: HEAD
00:41:06:WU02:FS04:0x22:   Compiler: Visual C++ 2015
00:41:06:WU02:FS04:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
00:41:06:WU02:FS04:0x22:   Platform: win32 10
00:41:06:WU02:FS04:0x22:       Bits: 64
00:41:06:WU02:FS04:0x22:       Mode: Release
00:41:06:WU02:FS04:0x22:************************************ System ************************************
00:41:06:WU02:FS04:0x22:        CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
00:41:06:WU02:FS04:0x22:     CPU ID: AuthenticAMD Family 23 Model 1 Stepping 1
00:41:06:WU02:FS04:0x22:       CPUs: 32
00:41:06:WU02:FS04:0x22:     Memory: 31.88GiB
00:41:06:WU02:FS04:0x22:Free Memory: 25.96GiB
00:41:06:WU02:FS04:0x22:    Threads: WINDOWS_THREADS
00:41:06:WU02:FS04:0x22: OS Version: 6.2
00:41:06:WU02:FS04:0x22:Has Battery: false
00:41:06:WU02:FS04:0x22: On Battery: false
00:41:06:WU02:FS04:0x22: UTC Offset: -7
00:41:06:WU02:FS04:0x22:        PID: 15220
00:41:06:WU02:FS04:0x22:        CWD: C:\Users\Master\AppData\Roaming\FAHClient\work
00:41:06:WU02:FS04:0x22:************************************ OpenMM ************************************
00:41:06:WU02:FS04:0x22:   Revision: 189320d0
00:41:06:WU02:FS04:0x22:********************************************************************************
00:41:06:WU02:FS04:0x22:Project: 14907 (Run 229, Clone 5, Gen 91)
00:41:06:WU02:FS04:0x22:Unit: 0x0000007681d59d695f52603285fa6f2b
00:41:06:WU02:FS04:0x22:Reading tar file core.xml
00:41:06:WU02:FS04:0x22:Reading tar file integrator.xml
00:41:06:WU02:FS04:0x22:Reading tar file state.xml
00:41:07:WU02:FS04:0x22:Reading tar file system.xml
00:41:09:WU02:FS04:0x22:Digital signatures verified
00:41:09:WU02:FS04:0x22:Folding@home GPU Core22 Folding@home Core
00:41:09:WU02:FS04:0x22:Version 0.0.13
00:41:09:WU02:FS04:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
00:41:09:WU02:FS04:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
00:41:09:WU02:FS04:0x22:  XTC frame write interval: 50000 steps (5%) [20 total]
00:41:09:WU02:FS04:0x22:  Global context and integrator variables write interval: disabled
00:41:09:WU02:FS04:0x22:There are 4 platforms available.
00:41:09:WU02:FS04:0x22:Platform 0: Reference
00:41:09:WU02:FS04:0x22:Platform 1: CPU
00:41:09:WU02:FS04:0x22:Platform 2: OpenCL
00:41:09:WU02:FS04:0x22:  opencl-device 3 specified
00:41:09:WU02:FS04:0x22:Platform 3: CUDA
00:41:09:WU02:FS04:0x22:  cuda-device 3 specified
00:41:12:WU05:FS04:Upload 31.35%
00:41:18:WU05:FS04:Upload 63.79%
00:41:20:WU02:FS04:0x22:Attempting to create CUDA context:
00:41:20:WU02:FS04:0x22:  Configuring platform CUDA
00:41:24:WU05:FS04:Upload 96.76%
00:41:25:WU05:FS04:Upload complete
00:41:25:WU05:FS04:Server responded WORK_ACK (400)
00:41:25:WU05:FS04:Final credit estimate, 138878.00 points
00:41:25:WU05:FS04:Cleaning up
00:41:26:WU02:FS04:0x22:  Using CUDA and gpu 3
00:41:26:WU02:FS04:0x22:Completed 0 out of 1000000 steps (0%)
00:41:27:WU02:FS04:0x22:Checkpoint completed at step 0
00:42:39:WU02:FS04:0x22:Completed 10000 out of 1000000 steps (1%)
00:43:56:WU02:FS04:0x22:Completed 20000 out of 1000000 steps (2%)
00:45:12:WU02:FS04:0x22:Completed 30000 out of 1000000 steps (3%)
00:46:28:WU02:FS04:0x22:Completed 40000 out of 1000000 steps (4%)
00:47:44:WU02:FS04:0x22:Completed 50000 out of 1000000 steps (5%)
00:47:47:WU02:FS04:0x22:Checkpoint completed at step 50000
00:49:04:WU02:FS04:0x22:Completed 60000 out of 1000000 steps (6%)
00:50:20:WU02:FS04:0x22:Completed 70000 out of 1000000 steps (7%)
00:51:37:WU02:FS04:0x22:Completed 80000 out of 1000000 steps (8%)
00:52:53:WU02:FS04:0x22:Completed 90000 out of 1000000 steps (9%)
00:54:10:WU02:FS04:0x22:Completed 100000 out of 1000000 steps (10%)
00:54:12:WU02:FS04:0x22:Checkpoint completed at step 100000
00:55:29:WU02:FS04:0x22:Completed 110000 out of 1000000 steps (11%)
00:56:45:WU02:FS04:0x22:Completed 120000 out of 1000000 steps (12%)
00:58:01:WU02:FS04:0x22:Completed 130000 out of 1000000 steps (13%)
00:59:18:WU02:FS04:0x22:Completed 140000 out of 1000000 steps (14%)
01:00:34:WU02:FS04:0x22:Completed 150000 out of 1000000 steps (15%)
01:00:37:WU02:FS04:0x22:Checkpoint completed at step 150000
01:01:53:WU02:FS04:0x22:Completed 160000 out of 1000000 steps (16%)
01:03:09:WU02:FS04:0x22:Completed 170000 out of 1000000 steps (17%)
01:04:26:WU02:FS04:0x22:Completed 180000 out of 1000000 steps (18%)
01:05:42:WU02:FS04:0x22:Completed 190000 out of 1000000 steps (19%)
01:06:59:WU02:FS04:0x22:Completed 200000 out of 1000000 steps (20%)
01:07:02:WU02:FS04:0x22:Checkpoint completed at step 200000
01:08:18:WU02:FS04:0x22:Completed 210000 out of 1000000 steps (21%)
01:09:35:WU02:FS04:0x22:Completed 220000 out of 1000000 steps (22%)
01:10:51:WU02:FS04:0x22:Completed 230000 out of 1000000 steps (23%)
01:12:07:WU02:FS04:0x22:Completed 240000 out of 1000000 steps (24%)
01:12:29:WU02:FS04:0x22:An exception occurred at step 242511: Error invoking kernel: CUDA_ERROR_LAUNCH_TIMEOUT (702)
01:12:29:WU02:FS04:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
01:12:29:WU02:FS04:0x22:Folding@home Core Shutdown: CORE_RESTART
******************************* Date: 2020-10-23 *******************************

JimboPalmer · Post by **JimboPalmer** » Fri Oct 23, 2020 3:26 pm

OK, I read up on CUDA_ERROR_LAUNCH_TIMEOUT, and it seems to happen on the GPU with a Windows Desktop when CUDA takes over 2 seconds to compute. The Windows OS generates the error when it can't update the desktop. You may wish to try to run the desktop on the first card.

Here is an explanation using MATHLAB from 2011:

In general, this error occurs when a long running gpuArray or CUDA code is run on a GPU that is used for both graphics rendering and CUDA computations. The error is triggered by the Operating System, which limits the time that the GPU can dedicate to computations, vs. rendering the user desktop.

This limit does not apply to GPUs which do not have the desktop extended to them, such as standalone GPU accelerators, or cards running under TCC driver on Windows.
For this reason it is recommended that gpuArray and CUDAkernel evaluations be run on a GPU that is NOT attached to a display and does not have the Windows or Linux desktop extended onto it.

On the Windows Vista, Windows 7 operating systems individual GPU kernels running on GPUs that are responsible for desktop rendering are limited to a 2 second runtime.
If your system has GPUs without a display attached then you can manually modify the system configuration to disable the TDR mechanism. Disabling the TDR timeout will allow kernels to run for extended periods of time without triggering an error. In addition, on Windows, Tesla cards running under Tesla Compute Cluster drivers are not subject to this limitation.

For information on how to modify or disable the TDR timeout on Windows Vista and Windows 7 refer to:
http://www.microsoft.com/whdc/device/di ... meout.mspx http://developer.nvidia.com/cuda-faq

Tuna_Ertemalp · Post by **Tuna_Ertemalp** » Fri Oct 23, 2020 4:40 pm

Well... As I had mentioned before, this has been happening on all of the 4 GPUs on this machine (and sometimes on a few other machines, too) randomly but frequently, therefore it is not happening just on the GPU attached to the monitor. Plus, the one I reported below happened on #4 while #1 is the GPU attached to the monitor. And, that monitor is off 95% of the time, and you'd hope Win10 doesn't do any desktop updating when there is no local/remote viewer for that desktop. And, since then, i.e. overnight, it further crashed on two other GPUs, #1 and #2. So, right now, only #3 is churning, but #3 has also been known to crash in the past. Another thing worth noting is that all my machines had been running BOINC for many many years with very intensive GPU use at much higher usage percentages until my switch to FAH this past March, and never had this issue before. And, it started roughly when FAH started using CUDA on NVidia instead of OpenCL.

Sooooo....

I am REALLY thinking that this has something to do with the CUDA code FAH is using, and this machine is just (un)lucky enough to run into it frequently. We could use it to debug the issue instead of trying to get rid of the symptoms. I guess my past 40 years of software engineering is kicking in, wanting to fix the root cause.

Is there any chance there is a FAH developer interested in using my machine via instrumented binaries to get to the bottom of this? Or, should I be reporting it on FAH GitHub, where the initial answer is usually to direct people to this forum.

Thanks
Tuna

Post by **PantherX** » Fri Oct 23, 2020 9:00 pm

You currently have a situation where you're encountering a bug. However, it isn't being able to be reproduced on demand but happens on that particular system so that's a good start. We can continue troubleshooting here on the Forum so see what the issue.

As a first step, I would finish all Slots and then then only fold on 1 GPU. Assuming you initially encountered the CUDA_ERROR_LAUNCH_TIMEOUT (702) issue once a week, see if you can encounter this issue again while folding on a single GPU. If you can fold an entire week without having that issue, then start up a second GPU. Continue until you encounter that issue.

BTW, I am assuming that you have disabled SLI in Nvidia Control Panel? F@H doesn't use SLI but maybe you're using other applications that do use SLI? Your system is slightly unique since you have quad GPUs while most have single, few have dual.

Tuna_Ertemalp · Post by **Tuna_Ertemalp** » Fri Oct 23, 2020 9:25 pm

Thanks, @PantherX. I will go the route of GPU elimination. Maybe, that will reveal something. However, I have run into this problem in single GPU machines, as well, although with no consistency. This machine just happens to be able to run into it multiple times a week, on different combinations of GPUs.

And, no, SLI on all my machines are disabled. I don't do any gaming or anything. I just get computers built for home with the latest hardware I can afford when I build them, and then use them 100% 24/7 for BOINC/FAH. Right now, they are all purely FAH. That's been my hobby since retirement...

What is the easiest way to disable a sliding group of 3 GPUs for FAH? Disabling the devices in Windows Device Manager? Removing three and modifying the PCI-BUS of the remaining one under FAHControl's SLOTS tab? Turning CUDA off in NVidia Control Pane for them Something else?

Tuna

Post by **PantherX** » Fri Oct 23, 2020 9:36 pm

The easiest way within F@H to stop folding would be to set the Slot to finish and then set <pause-on-start v='true'/> for that GPU Slot. Thus, it won't fond unless the GPU Slot was specifically started. No need to modify any F@H settings.

If you're taking about disabling it outside F@H, then I would personally either disable the GPU in Device Manager to pull it out, whatever is easiest. I don't think that disabling CUDA from within Nvidia Control panel would work.

BTW, since you're running Windows, please update to the latest version V7.6.21 since it resolves an issue with auto-startup

Tuna_Ertemalp · Post by **Tuna_Ertemalp** » Fri Oct 23, 2020 11:35 pm

Yup. I know all about 7.6.21. I reported the problem in 7.6.20, was given a beta within hours, and I tested it.

Thanks for the tip about pause-on-start per GPU. Updated FAH, started GPU #1, muted others. I'll report back about the CUDA error here when I have something new...

This is happening with no VERBOSITY flag, therefore with the default setting (3, I am guessing). If that is too much or too little, let me know.

Tuna

Post by **Joe_H** » Sat Oct 24, 2020 12:49 am

The log information that is affected by the log verbosity setting being set higher is rarely useful for troubleshooting. What error messages that get passed through to the client from the OS or driver get echoed in the log on the default setting anyways.

Post by **bruce** » Sat Oct 24, 2020 7:44 am

Comparing the performance of FAH to BOINC isn't useful. The error you're encountering has everything to do with a total time that a single GPU Kernel remains busy. that time depends on the number of operations processed in a single kernel which is direcly related to the problem being run and it's impossible to evaluate that without more information from both BOINC and FAH.

Most older GPUs process kernels in a FirstComeFirstServed sequenece and Windows has every right to assume something is wrong with the GPU if a kernel containing a screen update cannot be processed promptly, where promptly is defined by a number they put in the registry.

Why should a FAH Kernel run slower on your GPU than it does on arbitrary GPUs running arbitrary FAH assignments?

NVidia has announced some new features in some new GPUs allowing certain kernels to run concurrently, interrupting other kernels and altering the strice FIFO ordering. I don't think those features are present in your system but it's something you might want to reference if you happen to be chatting with FAH's developers.

Tuna_Ertemalp · Post by **Tuna_Ertemalp** » Sat Oct 24, 2020 3:14 pm

Thank you, @Bruce!

bruce wrote:Windows has every right to assume something is wrong with the GPU if a kernel containing a screen update cannot be processed promptly, where promptly is defined by a number they put in the registry.

But why would Windows do that on GPUs that are not connected to the monitor, therefore presumably are not involved in screen updates? That's making me scratch my head on this one...

bruce wrote:Why should a FAH Kernel run slower on your GPU than it does on arbitrary GPUs running arbitrary FAH assignments?

Yes, that's the question. It is also worth noting that the problem is not limited to the GPUs on this host, that I have seen it on some of my other hosts as well, although not in any frequency to attract my attention or to recognize any memorable pattern regarding single/dual GPU, connected to the monitor or not, etc.

bruce wrote:NVidia has announced some new features in some new GPUs allowing certain kernels to run concurrently, interrupting other kernels and altering the strice FIFO ordering. I don't think those features are present in your system but it's something you might want to reference if you happen to be chatting with FAH's developers.

I didn't yet get any 30xx GPUs, if those are what you are referring to. My forum signature contains a short list of my hardware, and they are all running the latest NVidia drivers that apply to them.

Thanks!
Tuna

Folding Forum

CUDA_ERROR_LAUNCH_FAILED

CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

CUDA_ERROR_LAUNCH_TIMEOUT

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED

Re: CUDA_ERROR_LAUNCH_FAILED