Error: Application FahCore_21.exe has been blocked

[Jimboc]

Hi everyone,

Happy New Year.

Earlier today I returned to my Folding@Home desktop system which had its dual screens in power save mode and they would not power back on. After waiting a short while , a BSoD appeared and the system restarted. Upon boot up Windows 10 displayed an error in the Action Center "Application FahCore_21.exe has been blocked from accessing Graphics hardware".

It would appear this sudden block to FahCore_21.exe caused the BSoD.

Please find below the screenshots of the Windows event logs from the relevant time and the FAH log for that time (it doesn't show anything out of the ordinary). I truncated the log to show the start and the relevant time period. I followed the suggestions in an older thread related to this error message of allowing FahCore_21 , FahCore_22 and FahCore_a7 through the firewall and whitelisting them with my AV software. I don't believe either was suddenly at fault:

viewtopic.php?f=106&t=30846&start=15

I believe this system is stable, its almost 18 months old and has never shown a BSoD before. I stress tested the CPU and RAM when I built it and again in early December 2019 just to check everything was still working fine and it was. I have been using Nvidia Studio Driver 441.66 since its release on the 10th of December without any issues.

Has anyone seen this error before or is it an isolated incident to my system only? Please let me know if you require any further details. Thanks in advance for any suggestions.

====================
For your reference my PCs specifications are as follows:

Processor: Intel Core i9 Extreme 7980XE @ 2.6 GHz (18 cores, 36 threads)
Motherboard: Asus Rampage VI Apex X299 (BIOS 1401)
Memory: 64GB Corsair Dominator Platinum DDR4 RAM + 64 GB Intel Optane M.2 Memory Accelerator (64 GB + 64 GB)
Video Cards: 2x Nvidia Titan RTX “T-Rex” (NVLink Enabled)(Nvidia Studio Driver 441.66)
Power Supply: Corsair AX1600i Titanium Plus PSU
OS: Windows 10 Pro for Workstations 64 Bit (Version 1909, Build 18363.535)

====================
https://imgur.com/JGfK2iz

https://imgur.com/wq7zKec

https://imgur.com/M5YB3H6

====================

Code: Select all

*********************** Log Started 2020-01-03T21:12:57Z ***********************
21:12:57:************************* Folding@home Client *************************
21:12:57:        Website: https://foldingathome.org/
21:12:57:      Copyright: (c) 2009-2018 foldingathome.org
21:12:57:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:12:57:           Args: --open-web-control
21:12:57:         Config: C:\Programs\FAHData\config.xml
21:12:57:******************************** Build ********************************
21:12:57:        Version: 7.5.1
21:12:57:           Date: May 11 2018
21:12:57:           Time: 13:06:32
21:12:57:     Repository: Git
21:12:57:       Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
21:12:57:         Branch: master
21:12:57:       Compiler: Visual C++ 2008
21:12:57:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
21:12:57:       Platform: win32 10
21:12:57:           Bits: 32
21:12:57:           Mode: Release
21:12:57:******************************* System ********************************
21:12:57:            CPU: Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz
21:12:57:         CPU ID: GenuineIntel Family 6 Model 85 Stepping 4
21:12:57:           CPUs: 32
21:12:57:         Memory: 63.68GiB
21:12:57:    Free Memory: 51.19GiB
21:12:57:        Threads: WINDOWS_THREADS
21:12:57:     OS Version: 6.2
21:12:57:    Has Battery: false
21:12:57:     On Battery: false
21:12:57:     UTC Offset: 0
21:12:57:            PID: 29020
21:12:57:            CWD: C:\Programs\FAHData
21:12:57:             OS: Windows 10 Pro for Workstations
21:12:57:        OS Arch: AMD64
21:12:57:           GPUs: 2
21:12:57:          GPU 0: Bus:101 Slot:0 Func:0 NVIDIA:8 TU102 [TITAN RTX]
21:12:57:          GPU 1: Bus:23 Slot:0 Func:0 NVIDIA:8 TU102 [TITAN RTX]
21:12:57:  CUDA Device 0: Platform:0 Device:0 Bus:23 Slot:0 Compute:7.5 Driver:10.2
21:12:57:  CUDA Device 1: Platform:0 Device:1 Bus:101 Slot:0 Compute:7.5 Driver:10.2
21:12:57:OpenCL Device 0: Platform:0 Device:0 Bus:23 Slot:0 Compute:1.2 Driver:441.66
21:12:57:OpenCL Device 1: Platform:0 Device:1 Bus:101 Slot:0 Compute:1.2 Driver:441.66
21:12:57:  Win32 Service: false
21:12:57:***********************************************************************
21:12:57:<config>
21:12:57:  <!-- Network -->
21:12:57:  <proxy v=':8080'/>
21:12:57:
21:12:57:  <!-- Remote Command Server -->
21:12:57:  <password v='**************************'/>
21:12:57:
21:12:57:  <!-- Slot Control -->
21:12:57:  <power v='full'/>
21:12:57:
21:12:57:  <!-- User Information -->
21:12:57:  <passkey v='********************************'/>
21:12:57:  <team v='142900'/>
21:12:57:  <user v='Jimboc'/>
21:12:57:
21:12:57:  <!-- Folding Slots -->
21:12:57:  <slot id='0' type='CPU'>
21:12:57:    <cpus v='20'/>
21:12:57:  </slot>
21:12:57:  <slot id='1' type='GPU'>
21:12:57:    <client-type v='advanced'/>
21:12:57:  </slot>
21:12:57:  <slot id='2' type='GPU'>
21:12:57:    <client-type v='advanced'/>
21:12:57:  </slot>
21:12:57:</config>
21:12:57:Trying to access database...
21:12:57:Successfully acquired database lock
21:12:57:Enabled folding slot 00: READY cpu:20
21:12:57:Enabled folding slot 01: READY gpu:0:TU102 [TITAN RTX]
21:12:57:Enabled folding slot 02: READY gpu:1:TU102 [TITAN RTX]
21:12:58:WU00:FS00:Connecting to 65.254.110.245:8080
21:12:58:WU01:FS01:Connecting to 65.254.110.245:8080
21:12:58:WU02:FS02:Connecting to 65.254.110.245:8080
21:12:58:WU00:FS00:Assigned to work server 128.252.203.9
21:12:58:WU00:FS00:Requesting new work unit for slot 00: READY cpu:20 from 128.252.203.9
21:12:58:WU00:FS00:Connecting to 128.252.203.9:8080
21:12:58:WU01:FS01:Assigned to work server 155.247.166.220
21:12:58:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [TITAN RTX] from 155.247.166.220
21:12:58:WU01:FS01:Connecting to 155.247.166.220:8080
21:12:58:WU02:FS02:Assigned to work server 128.252.203.10
21:12:58:WU02:FS02:Requesting new work unit for slot 02: READY gpu:1:TU102 [TITAN RTX] from 128.252.203.10
21:12:58:WU02:FS02:Connecting to 128.252.203.10:8080
21:12:59:WU01:FS01:Downloading 5.81MiB
21:12:59:WU00:FS00:Downloading 7.29MiB
21:12:59:WU02:FS02:Downloading 77.39MiB
21:13:05:WU01:FS01:Download 73.12%
21:13:05:WU02:FS02:Download 0.48%
21:13:05:WU00:FS00:Download 6.86%
21:13:07:WU01:FS01:Download complete
21:13:07:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14320 run:182 clone:1 gen:10 core:0x21 unit:0x0000000d0002894c5dfebae47e3a6512
21:13:07:WU01:FS01:Starting
21:13:07:WU01:FS01:Running FahCore: C:\Programs\FAHClient/FAHCoreWrapper.exe C:\Programs\FAHData\cores/cores.foldingathome.org/v7/win/64bit/nvidia/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 29020 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
21:13:07:WU01:FS01:Started FahCore on PID 9364
21:13:07:WU01:FS01:Core PID:36640
21:13:07:WU01:FS01:FahCore 0x21 started
21:13:08:WU01:FS01:0x21:*********************** Log Started 2020-01-03T21:13:07Z ***********************
21:13:08:WU01:FS01:0x21:Project: 14320 (Run 182, Clone 1, Gen 10)
21:13:08:WU01:FS01:0x21:Unit: 0x0000000d0002894c5dfebae47e3a6512
21:13:08:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
21:13:08:WU01:FS01:0x21:Machine: 1
21:13:08:WU01:FS01:0x21:Reading tar file core.xml
21:13:08:WU01:FS01:0x21:Reading tar file integrator.xml
21:13:08:WU01:FS01:0x21:Reading tar file state.xml
21:13:08:WU01:FS01:0x21:Reading tar file system.xml
21:13:08:WU01:FS01:0x21:Digital signatures verified
21:13:08:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
21:13:08:WU01:FS01:0x21:Version 0.0.20
21:13:09:WU01:FS01:0x21:Completed 0 out of 25000000 steps (0%)
21:13:09:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
21:13:11:WU00:FS00:Download 33.44%
21:13:11:WU02:FS02:Download 4.12%
21:13:17:WU00:FS00:Download 57.45%
21:13:17:WU02:FS02:Download 8.08%
21:13:19:FS00:Finishing
21:13:23:WU02:FS02:Download 10.50%
21:13:23:WU00:FS00:Download 82.31%
21:13:26:WU00:FS00:Download complete
21:13:26:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14400 run:0 clone:1759 gen:16 core:0xa7 unit:0x0000001280fccb095dcad8989d8763bd
21:13:26:WU00:FS00:Starting
21:13:26:WU00:FS00:Running FahCore: C:\Programs\FAHClient/FAHCoreWrapper.exe C:\Programs\FAHData\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 705 -lifeline 29020 -checkpoint 15 -np 20
21:13:26:WU00:FS00:Started FahCore on PID 25236
21:13:26:WU00:FS00:Core PID:28212
21:13:26:WU00:FS00:FahCore 0xa7 started
21:13:27:WU00:FS00:0xa7:*********************** Log Started 2020-01-03T21:13:26Z ***********************
21:13:27:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
21:13:27:WU00:FS00:0xa7:       Type: 0xa7
21:13:27:WU00:FS00:0xa7:       Core: Gromacs
21:13:27:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 25236 -checkpoint 15 -np
21:13:27:WU00:FS00:0xa7:             20
21:13:27:WU00:FS00:0xa7:************************************ CBang *************************************
21:13:27:WU00:FS00:0xa7:       Date: Oct 26 2019
21:13:27:WU00:FS00:0xa7:       Time: 01:38:25
21:13:27:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
21:13:27:WU00:FS00:0xa7:     Branch: master
21:13:27:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
21:13:27:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
21:13:27:WU00:FS00:0xa7:   Platform: win32 10
21:13:27:WU00:FS00:0xa7:       Bits: 64
21:13:27:WU00:FS00:0xa7:       Mode: Release
21:13:27:WU00:FS00:0xa7:************************************ System ************************************
21:13:27:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz
21:13:27:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 85 Stepping 4
21:13:27:WU00:FS00:0xa7:       CPUs: 36
21:13:27:WU00:FS00:0xa7:     Memory: 63.68GiB
21:13:27:WU00:FS00:0xa7:Free Memory: 51.11GiB
21:13:27:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
21:13:27:WU00:FS00:0xa7: OS Version: 6.2
21:13:27:WU00:FS00:0xa7:Has Battery: false
21:13:27:WU00:FS00:0xa7: On Battery: false
21:13:27:WU00:FS00:0xa7: UTC Offset: 0
21:13:27:WU00:FS00:0xa7:        PID: 28212
21:13:27:WU00:FS00:0xa7:        CWD: C:\Programs\FAHData\work
21:13:27:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
21:13:27:WU00:FS00:0xa7:    Version: 0.0.18
21:13:27:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:13:27:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
21:13:27:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
21:13:27:WU00:FS00:0xa7:       Date: Oct 26 2019
21:13:27:WU00:FS00:0xa7:       Time: 01:52:30
21:13:27:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
21:13:27:WU00:FS00:0xa7:     Branch: master
21:13:27:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
21:13:27:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
21:13:27:WU00:FS00:0xa7:   Platform: win32 10
21:13:27:WU00:FS00:0xa7:       Bits: 64
21:13:27:WU00:FS00:0xa7:       Mode: Release
21:13:27:WU00:FS00:0xa7:************************************ Build *************************************
21:13:27:WU00:FS00:0xa7:       SIMD: avx_256
21:13:27:WU00:FS00:0xa7:********************************************************************************
21:13:27:WU00:FS00:0xa7:Project: 14400 (Run 0, Clone 1759, Gen 16)
21:13:27:WU00:FS00:0xa7:Unit: 0x0000001280fccb095dcad8989d8763bd
21:13:27:WU00:FS00:0xa7:Reading tar file core.xml
21:13:27:WU00:FS00:0xa7:Reading tar file frame16.tpr
21:13:27:WU00:FS00:0xa7:Digital signatures verified
21:13:27:WU00:FS00:0xa7:Calling: mdrun -s frame16.tpr -o frame16.trr -x frame16.xtc -cpt 15 -nt 20
21:13:27:WU00:FS00:0xa7:Steps: first=2000000 total=125000
21:13:28:WU00:FS00:0xa7:Completed 1 out of 125000 steps (0%)
21:13:29:WU02:FS02:Download 14.62%
21:13:35:WU02:FS02:Download 21.00%
21:13:41:WU02:FS02:Download 28.27%
21:13:47:WU02:FS02:Download 36.42%
21:13:53:WU02:FS02:Download 45.47%
21:13:59:WU02:FS02:Download 54.51%
21:14:03:WU00:FS00:0xa7:Completed 1250 out of 125000 steps (1%)
21:14:05:WU02:FS02:Download 63.48%
21:14:11:WU02:FS02:Download 71.55%
21:14:17:WU02:FS02:Download 77.37%
21:14:23:WU02:FS02:Download 86.01%
21:14:29:WU02:FS02:Download 94.97%
21:14:32:WU02:FS02:Download complete
21:14:32:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14268 run:0 clone:20 gen:102 core:0x21 unit:0x0000007580fccb0a5daa000a5db256d5
21:14:32:WU02:FS02:Starting
21:14:32:WU02:FS02:Running FahCore: C:\Programs\FAHClient/FAHCoreWrapper.exe C:\Programs\FAHData\cores/cores.foldingathome.org/v7/win/64bit/nvidia/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 705 -lifeline 29020 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
21:14:32:WU02:FS02:Started FahCore on PID 32436
21:14:32:WU02:FS02:Core PID:23020
21:14:32:WU02:FS02:FahCore 0x21 started
21:14:32:WU02:FS02:0x21:*********************** Log Started 2020-01-03T21:14:32Z ***********************
21:14:32:WU02:FS02:0x21:Project: 14268 (Run 0, Clone 20, Gen 102)
21:14:32:WU02:FS02:0x21:Unit: 0x0000007580fccb0a5daa000a5db256d5
21:14:32:WU02:FS02:0x21:CPU: 0x00000000000000000000000000000000
21:14:32:WU02:FS02:0x21:Machine: 2
21:14:32:WU02:FS02:0x21:Reading tar file core.xml
21:14:32:WU02:FS02:0x21:Reading tar file integrator.xml
21:14:32:WU02:FS02:0x21:Reading tar file state.xml
21:14:33:WU02:FS02:0x21:Reading tar file system.xml
21:14:33:WU02:FS02:0x21:Digital signatures verified
21:14:33:WU02:FS02:0x21:Folding@home GPU Core21 Folding@home Core
21:14:33:WU02:FS02:0x21:Version 0.0.20
21:14:38:WU00:FS00:0xa7:Completed 2500 out of 125000 steps (2%)
21:14:55:WU02:FS02:0x21:Completed 0 out of 1000000 steps (0%)
21:14:55:WU02:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
21:15:14:WU00:FS00:0xa7:Completed 3750 out of 125000 steps (3%)
21:15:16:WU01:FS01:0x21:Completed 250000 out of 25000000 steps (1%)
21:15:39:WU02:FS02:0x21:Completed 10000 out of 1000000 steps (1%)
21:15:50:WU00:FS00:0xa7:Completed 5000 out of 125000 steps (4%)
21:16:20:WU02:FS02:0x21:Completed 20000 out of 1000000 steps (2%)
21:16:26:WU00:FS00:0xa7:Completed 6250 out of 125000 steps (5%)
21:17:02:WU00:FS00:0xa7:Completed 7500 out of 125000 steps (6%)

12:43:23:WU01:FS01:0x21:Completed 130000 out of 1000000 steps (13%)
12:43:28:WU00:FS01:Upload 57.24%
12:43:34:WU00:FS01:Upload 57.87%
12:43:40:WU00:FS01:Upload 58.41%
12:43:46:WU00:FS01:Upload 58.95%
12:43:52:WU00:FS01:Upload 59.49%
12:43:58:WU00:FS01:Upload 60.03%
12:44:02:WU01:FS01:0x21:Completed 140000 out of 1000000 steps (14%)
12:44:04:WU00:FS01:Upload 60.66%
12:44:10:WU00:FS01:Upload 61.20%
12:44:16:WU00:FS01:Upload 61.74%
12:44:22:WU00:FS01:Upload 62.37%
12:44:28:WU00:FS01:Upload 62.91%
12:44:34:WU00:FS01:Upload 63.54%
12:44:40:WU00:FS01:Upload 64.08%
12:44:46:WU00:FS01:Upload 64.62%
12:44:46:WU01:FS01:0x21:Completed 150000 out of 1000000 steps (15%)
12:44:52:WU00:FS01:Upload 65.25%
12:44:58:WU00:FS01:Upload 65.79%
12:45:04:WU00:FS01:Upload 66.42%
12:45:10:WU00:FS01:Upload 66.96%
12:45:16:WU00:FS01:Upload 67.59%
12:45:22:WU00:FS01:Upload 68.13%
12:45:25:WU01:FS01:0x21:Completed 160000 out of 1000000 steps (16%)
12:45:28:WU00:FS01:Upload 68.67%
12:45:34:WU00:FS01:Upload 69.30%
12:45:40:WU00:FS01:Upload 69.84%
12:45:46:WU00:FS01:Upload 70.47%
12:45:52:WU00:FS01:Upload 71.01%
12:45:59:WU00:FS01:Upload 71.64%
12:46:05:WU00:FS01:Upload 72.27%
12:46:10:WU01:FS01:0x21:Completed 170000 out of 1000000 steps (17%)
12:46:11:WU00:FS01:Upload 72.81%
12:46:17:WU00:FS01:Upload 73.44%
12:46:23:WU00:FS01:Upload 73.98%
12:46:29:WU00:FS01:Upload 74.61%
12:46:35:WU00:FS01:Upload 75.15%
12:46:41:WU00:FS01:Upload 75.69%
12:46:47:WU00:FS01:Upload 76.32%
12:46:49:WU01:FS01:0x21:Completed 180000 out of 1000000 steps (18%)
12:46:53:WU00:FS01:Upload 76.86%
12:46:59:WU00:FS01:Upload 77.49%
12:47:05:WU00:FS01:Upload 78.03%
12:47:11:WU00:FS01:Upload 78.66%
12:47:17:WU00:FS01:Upload 79.20%
12:47:24:WU00:FS01:Upload 79.83%
12:47:30:WU00:FS01:Upload 80.46%
12:47:33:WU01:FS01:0x21:Completed 190000 out of 1000000 steps (19%)
12:47:36:WU00:FS01:Upload 81.00%
12:47:42:WU00:FS01:Upload 81.63%
12:47:48:WU00:FS01:Upload 82.17%
12:47:54:WU00:FS01:Upload 82.80%
12:48:00:WU00:FS01:Upload 83.34%
12:48:06:WU00:FS01:Upload 83.97%
12:48:12:WU01:FS01:0x21:Completed 200000 out of 1000000 steps (20%)
12:48:12:WU00:FS01:Upload 84.51%
12:48:18:WU00:FS01:Upload 85.05%
12:48:24:WU00:FS01:Upload 85.68%
12:48:30:WU00:FS01:Upload 86.22%
12:48:36:WU00:FS01:Upload 86.85%
12:48:42:WU00:FS01:Upload 87.39%
12:48:48:WU00:FS01:Upload 87.93%
12:48:54:WU00:FS01:Upload 88.56%
12:48:56:WU01:FS01:0x21:Completed 210000 out of 1000000 steps (21%)
12:49:00:WU00:FS01:Upload 89.10%
12:49:06:WU00:FS01:Upload 89.73%
12:49:12:WU00:FS01:Upload 90.27%
12:49:19:WU00:FS01:Upload 90.90%
12:49:25:WU00:FS01:Upload 91.53%
12:49:31:WU00:FS01:Upload 92.07%
12:49:35:WU01:FS01:0x21:Completed 220000 out of 1000000 steps (22%)
12:49:37:WU00:FS01:Upload 92.70%
12:49:43:WU00:FS01:Upload 93.24%
12:49:49:WU00:FS01:Upload 93.87%
12:49:55:WU00:FS01:Upload 94.41%
12:50:01:WU00:FS01:Upload 95.04%
12:50:07:WU00:FS01:Upload 95.58%
12:50:13:WU00:FS01:Upload 96.12%
12:50:19:WU01:FS01:0x21:Completed 230000 out of 1000000 steps (23%)
12:50:20:WU00:FS01:Upload 96.39%
12:50:26:WU00:FS01:Upload 96.93%
12:50:33:WU00:FS01:Upload 97.56%
12:50:39:WU00:FS01:Upload 98.19%
12:50:45:WU00:FS01:Upload 98.82%
12:50:51:WU00:FS01:Upload 99.36%
12:50:57:WU00:FS01:Upload 99.90%
12:50:58:WU01:FS01:0x21:Completed 240000 out of 1000000 steps (24%)
12:50:59:WU00:FS01:Upload complete
12:50:59:WU00:FS01:Server responded WORK_ACK (400)
12:50:59:WU00:FS01:Final credit estimate, 120571.00 points
12:50:59:WU00:FS01:Cleaning up
12:51:43:WU01:FS01:0x21:Completed 250000 out of 1000000 steps (25%)
12:52:22:WU01:FS01:0x21:Completed 260000 out of 1000000 steps (26%)
12:53:06:WU01:FS01:0x21:Completed 270000 out of 1000000 steps (27%)
12:53:45:WU01:FS01:0x21:Completed 280000 out of 1000000 steps (28%)
12:54:29:WU01:FS01:0x21:Completed 290000 out of 1000000 steps (29%)
12:55:08:WU01:FS01:0x21:Completed 300000 out of 1000000 steps (30%)
12:55:52:WU01:FS01:0x21:Completed 310000 out of 1000000 steps (31%)
12:56:31:WU01:FS01:0x21:Completed 320000 out of 1000000 steps (32%)
12:57:15:WU01:FS01:0x21:Completed 330000 out of 1000000 steps (33%)
12:57:54:WU01:FS01:0x21:Completed 340000 out of 1000000 steps (34%)
12:58:46:WU01:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
12:59:26:WU01:FS01:0x21:Completed 360000 out of 1000000 steps (36%)

[toTOW]

Maybe I'm blind, but I don't see an error in FAHLog ...

Your third screenshot indicates that the display driver crashed and has been restarted ... this is a Windows feature called TDR and is usually caused by an unstable GPU ...

I never saw the first error (your second screenshot), but since it happened after many of the errors above, it probably a consequence and I wouldn't worry to much about it.

The driver reset error should be accompanied with errors in FAHLog talking about a dumped WU, and an OpenCL error (typically -5).

[bruce]

Is the power-saving feature of your OS enabled to let your CPU system sleep or hibernate? There used to be a (rare) problem caused by a suspended system attempting to resume processing a GPU assignment after the power to the GPU was interrupted.

[Jimboc]

Maybe I'm blind, but I don't see an error in FAHLog ...

Your third screenshot indicates that the display driver crashed and has been restarted ... this is a Windows feature called TDR and is usually caused by an unstable GPU ...

I never saw the first error (your second screenshot), but since it happened after many of the errors above, it probably a consequence and I wouldn't worry to much about it.

The driver reset error should be accompanied with errors in FAHLog talking about a dumped WU, and an OpenCL error (typically -5).

Hi toTOW,

Many thanks for your prompt response.

You are not blind, there are no error messages in the folding log. That is the strange thing. I expected some myself but it appears FAH was working fine until a TDR happened again and again until Windows then had a BSoD. If my assumption is not correct, I am unsure why those errors messages (dumped WU etc.) are not present in the log.

Agreed, that is an excellent idea about the GPUs being unstable. I will stability test them and return with the results. Those results may not be until next weekend since its unlikely I will have the time to do this during the working week. I will NOT be using FurMark for this testing.

Is the power-saving feature of your OS enabled to let your CPU system sleep or hibernate? There used to be a (rare) problem caused by a suspended system attempting to resume processing a GPU assignment after the power to the GPU was interrupted.

Hi bruce,

Thanks for your response.

No, all power saving for the CPU and all other aspects of the system (except my dual monitors) are turned off. Agreed, such suspended modes can cause such issues but I only had my dual screens in power saving mode.

@toTOW and bruce,
I will stability test the GPUs and return with my results as soon as possible. Thanks very much to you both.

[toTOW]

Since you have two GPUs, maybe you can run just one at once first and see how it goes ...

Maybe you can try to play with the CPU slot too ...

I see that you have 64 GB Intel Optane M.2 Memory Accelerator : what is it and how does it work ?

[Jimboc]

Since you have two GPUs, maybe you can run just one at once first and see how it goes ...

Maybe you can try to play with the CPU slot too ...

I see that you have 64 GB Intel Optane M.2 Memory Accelerator : what is it and how does it work ?

Hi toTOW,

You are correct. I will first test the GPUs combined since NVLink works to combine the two GPUs into one larger GPU. The older SLI standard had them share the load as close to 50% each as possible.

I will then test each GPU separately by removing the NVLink bridge and swapping the GPUs from their PCI Express slots. This was my reason for asking for more time. It will break my heart (and possibly my wallet again ) if these Titan RTX GPUs are now faulty.

I don’t see much benefit to stress testing the CPU again since I did in early December. However, if you feel I should I will.

I have PM’ed you with an answer to the Intel Optane question so as to keep the thread on topic. In short, the second video at the following link explains it the best: https://www.intel.com/content/www/us/en ... emory.html

Thanks again.

[toTOW]

No, keep your setup as it is now, just disable/enable GPU slots alternately ... if you remove physically one GPU from the system, you might miss something if the issue is related to power ...

I'm still wondering how the Optane caching system could affect the client or explain the crash ...

Did you find some details about the BSOD in Windows event viewer ?

P.S : NVLink is useless to FAH like SLI used to be, it considers the two GPU as two independant devices, whether you use the NVLink bridge or not.

[bruce]

I will then test each GPU separately by removing the NVLink bridge and swapping the GPUs from their PCI Express slots. This was my reason for asking for more time. It will break my heart (and possibly my wallet again ) if these Titan RTX GPUs are now faulty.

This doesn't sound right, but maybe I'm just operating on old information.

My understanding is that NVLink allows a pair of GPUs to combine their video resources to increase the video frame rate. FAH doesn't use those video resources but treats the computational resources of GPUs independently whether or not NVLink is in use. Do you have updated information that I don't have?

[Jimboc]

No, keep your setup as it is now, just disable/enable GPU slots alternately ... if you remove physically one GPU from the system, you might miss something if the issue is related to power ...

I'm still wondering how the Optane caching system could affect the client or explain the crash ...

Did you find some details about the BSOD in Windows event viewer ?

P.S : NVLink is useless to FAH like SLI used to be, it considers the two GPU as two independant devices, whether you use the NVLink bridge or not.

Hi toTOW,

Understood. I will leave the GPUs where they are. I can assure you this 1600 W PSU can handle the load.

The Optane should not have any affect since FAH is installed on my C drive (a separate SSD drive). The Optane is optimising my secondary mechanical 5 TB drive. I can disable the Optane drive if you wish? This is easily done using its settings application. Ironically, the BSoD is a DPC Watchdog BSoD which usually has to do with the CPU. The CPU was only be using by the GPUs for folding, only 2 of the 18 cores were being used. The system was otherwise idle. By the way, the Event Viewer does not contain details of BSoDs.

I realise NVLink is not used by FAH. Apologies I was skipping ahead to how I intend to test the GPUs during the stability test I have in mind (outside of FAH).

I will then test each GPU separately by removing the NVLink bridge and swapping the GPUs from their PCI Express slots. This was my reason for asking for more time. It will break my heart (and possibly my wallet again ) if these Titan RTX GPUs are now faulty.

This doesn't sound right, but maybe I'm just operating on old information.

My understanding is that NVLink allows a pair of GPUs to combine their video resources to increase the video frame rate. FAH doesn't use those video resources but treats the computational resources of GPUs independently whether or not NVLink is in use. Do you have updated information that I don't have?

Hi bruce,

Similar to my reply to toTOW, you are correct NVLink works the way you described. Again, apologies I was skipping ahead to how I intend to test the GPUs during the stability test I have in mind (outside of FAH).

I will carry out the stability test this weekend and come back with the results. Sorry for any confusion caused by my previous reply.

[Jimboc]

Hi bruce and toTOW,

Sorry for the delay in updating this thread.

I have stress tested the GPUs within my system using PrimeGrid and the benchmarks Unigine Heaven and Unigine Superposition, each for 8 hours without any issues.

https://unigine.com/en/products/benchmarks

https://www.primegrid.com/

I’m eager to return to F@H unless there any other troubleshooting steps you can suggest? Thank you.

[foldy]

google says call some Windows cleanup tasks in admin cmd shell

sfc /scannow
DISM.exe /Online /Cleanup-image /Scanhealth

[Jimboc]

google says call some Windows cleanup tasks in admin cmd shell

sfc /scannow
DISM.exe /Online /Cleanup-image /Scanhealth

Hi foldy,

You’re correct, I should have mentioned I ran the following commands as admin before I started this thread:

Code: Select all

sfc /scannow

DISM.exe /Online /Cleanup-image /Scanhealth

DISM.exe /Online /Cleanup-image /Restorehealth

No issues were found upon completion of these commands. Thanks.

[foldy]

Did you try increase TDR delay?

[Jimboc]

Did you try increase TDR delay?

No since I didn't experience any TDRs during the stress tests. Also isn't increasing that delay simply ignoring the issue why the TDR is occurring in the first instance? I'm going to resume folding on Friday evening and see how things go. I have a feeling things will be back to normal.

Thanks.

[Jimboc]

Hi toTOW, bruce and foldy,

Good news! I have had my system running F@H for 36 hours and it completed 55 WUs which included 7x Core 22 WUs. The Core 22 WUs made the GPUs run a little hotter usually at 69 or 70 C rather than 64 or 65 C. Boost clock speeds were unaffected at 1950 Mhz. GPUs usage was 92 to 94%. During this time there was no instability (the Windows event log showed no further TDRs or crashes).

The only changes I made to the system since the error that day has been to install the Windows 10 January 2020 security update bringing the system to version 18363.592 and the installation of a new Intel Ethernet driver v24.5 (previously v24.3). I decided not to adjust the TDR delay. Thanks for your help guys and I hope this is useful to someone