[Windows] RX 590: Particle coordinate is nan (Resolved)

[Talgeeze]

Hello.
I've been encountering an issue where most (if not all) GPU [XFX RX 590] WUs fail. Always with a series of

Code: Select all

An exception occurred at step XXXXX: Particle coordinate is nan

ending with

Code: Select all

Max number of attempts to resume from last checkpoint (X) reached. Aborting.

A few WUs reach 70 / 80 %, some fail at ~30% but most have trouble going past 10 %. Started happening a few days ago, regardless of CPU [Ryzen 5 2600] folding, over multiple projects. GPU Temp when folding is stable at 65-70 °C. Drivers were a bit outdated (four month old) but updating to the latest version (21.4.1) had no visible effect. I tried running FAHBenchmark for a good hour, it didn't return any error but I'm unsure if I used the correct settings.
I'm not having issues wit this GPU on other uses, except very occasionally crashes with a very short list of games.

Log:

Code: Select all

*********************** Log Started 2021-04-26T15:20:23Z ***********************
15:20:23:FS01:Initialized folding slot 01: gpu:6:0 Ellesmere XT [Radeon RX 470/480/570/580/590]
15:30:44:FS01:Unpaused
15:30:54:WU02:FS01:Downloading core from http://cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah
15:30:54:WU02:FS01:Connecting to cores.foldingathome.org:80
15:30:55:WU02:FS01:FahCore 22: Downloading 77.67MiB
15:31:01:WU02:FS01:FahCore 22: 3.38%
15:31:07:WU02:FS01:FahCore 22: 7.08%
15:31:13:WU02:FS01:FahCore 22: 10.94%
15:31:19:WU02:FS01:FahCore 22: 14.57%
15:31:25:WU02:FS01:FahCore 22: 18.27%
15:31:31:WU02:FS01:FahCore 22: 21.41%
15:31:37:WU02:FS01:FahCore 22: 25.11%
15:31:43:WU02:FS01:FahCore 22: 28.89%
15:31:49:WU02:FS01:FahCore 22: 32.59%
15:31:55:WU02:FS01:FahCore 22: 36.37%
15:32:01:WU02:FS01:FahCore 22: 39.83%
15:32:07:WU02:FS01:FahCore 22: 43.21%
15:32:13:WU02:FS01:FahCore 22: 46.75%
15:32:19:WU02:FS01:FahCore 22: 50.46%
15:32:25:WU02:FS01:FahCore 22: 54.24%
15:32:31:WU02:FS01:FahCore 22: 57.78%
15:32:37:WU02:FS01:FahCore 22: 61.56%
15:32:43:WU02:FS01:FahCore 22: 63.33%
15:32:49:WU02:FS01:FahCore 22: 63.73%
15:32:55:WU02:FS01:FahCore 22: 66.07%
15:33:01:WU02:FS01:FahCore 22: 69.69%
15:33:07:WU02:FS01:FahCore 22: 73.39%
15:33:13:WU02:FS01:FahCore 22: 77.09%
15:33:19:WU02:FS01:FahCore 22: 80.39%
15:33:25:WU02:FS01:FahCore 22: 83.85%
15:33:31:WU02:FS01:FahCore 22: 87.07%
15:33:37:WU02:FS01:FahCore 22: 90.45%
15:33:43:WU02:FS01:FahCore 22: 94.23%
15:33:49:WU02:FS01:FahCore 22: 97.77%
15:33:55:WU02:FS01:FahCore 22: 99.95%
15:33:58:WU02:FS01:FahCore 22: Download complete
15:33:58:WU02:FS01:Valid core signature
15:33:58:WU02:FS01:Unpacked 8.33MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe
15:33:58:WU02:FS01:Unpacked 3.06MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/nvrtc-builtins64_92.dll
15:33:58:WU02:FS01:Unpacked 3.13MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMM.dll
15:33:58:WU02:FS01:Unpacked 2.07MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMMOpenCL.dll
15:33:58:WU02:FS01:Unpacked 14.82MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/nvrtc64_92.dll
15:33:59:WU02:FS01:Unpacked 82.99MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/cufft64_92.dll
15:33:59:WU02:FS01:Unpacked 275.00KiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMMCudaCompiler.dll
15:33:59:WU02:FS01:Unpacked 1.95MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMMCUDA.dll
15:33:59:WU02:FS01:Unpacked 786.00KiB to cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/OpenMMCPU.dll
15:33:59:WU02:FS01:Starting
15:33:59:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 29100 -checkpoint 5 -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0 -gpu-usage 100
15:33:59:WU02:FS01:Started FahCore on PID 34632
15:33:59:WU02:FS01:Core PID:34784
15:33:59:WU02:FS01:FahCore 0x22 started
15:34:00:WU02:FS01:0x22:*********************** Log Started 2021-04-26T15:33:59Z ***********************
15:34:00:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
15:34:00:WU02:FS01:0x22:       Core: Core22
15:34:00:WU02:FS01:0x22:       Type: 0x22
15:34:00:WU02:FS01:0x22:    Version: 0.0.13
15:34:00:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:34:00:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
15:34:00:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
15:34:00:WU02:FS01:0x22:       Date: Sep 19 2020
15:34:00:WU02:FS01:0x22:       Time: 02:35:58
15:34:00:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
15:34:00:WU02:FS01:0x22:     Branch: core22-0.0.13
15:34:00:WU02:FS01:0x22:   Compiler: Visual C++ 2015
15:34:00:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
15:34:00:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
15:34:00:WU02:FS01:0x22:   Platform: win32 10
15:34:00:WU02:FS01:0x22:       Bits: 64
15:34:00:WU02:FS01:0x22:       Mode: Release
15:34:00:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
15:34:00:WU02:FS01:0x22:             <peastman@stanford.edu>
15:34:00:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 34632 -checkpoint 5
15:34:00:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0
15:34:00:WU02:FS01:0x22:             -gpu-usage 100
15:34:00:WU02:FS01:0x22:************************************ libFAH ************************************
15:34:00:WU02:FS01:0x22:       Date: Sep 7 2020
15:34:00:WU02:FS01:0x22:       Time: 19:09:56
15:34:00:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
15:34:00:WU02:FS01:0x22:     Branch: HEAD
15:34:00:WU02:FS01:0x22:   Compiler: Visual C++ 2015
15:34:00:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
15:34:00:WU02:FS01:0x22:   Platform: win32 10
15:34:00:WU02:FS01:0x22:       Bits: 64
15:34:00:WU02:FS01:0x22:       Mode: Release
15:34:00:WU02:FS01:0x22:************************************ CBang *************************************
15:34:00:WU02:FS01:0x22:       Date: Sep 7 2020
15:34:00:WU02:FS01:0x22:       Time: 19:08:30
15:34:00:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
15:34:00:WU02:FS01:0x22:     Branch: HEAD
15:34:00:WU02:FS01:0x22:   Compiler: Visual C++ 2015
15:34:00:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
15:34:00:WU02:FS01:0x22:   Platform: win32 10
15:34:00:WU02:FS01:0x22:       Bits: 64
15:34:00:WU02:FS01:0x22:       Mode: Release
15:34:00:WU02:FS01:0x22:************************************ System ************************************
15:34:00:WU02:FS01:0x22:        CPU: AMD Ryzen 5 2600 Six-Core Processor
15:34:00:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
15:34:00:WU02:FS01:0x22:       CPUs: 12
15:34:00:WU02:FS01:0x22:     Memory: 15.95GiB
15:34:00:WU02:FS01:0x22:Free Memory: 5.20GiB
15:34:00:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
15:34:00:WU02:FS01:0x22: OS Version: 6.2
15:34:00:WU02:FS01:0x22:Has Battery: false
15:34:00:WU02:FS01:0x22: On Battery: false
15:34:00:WU02:FS01:0x22: UTC Offset: 2
15:34:00:WU02:FS01:0x22:        PID: 34784
15:34:00:WU02:FS01:0x22:        CWD: C:\ProgramData\FAHClient\work
15:34:00:WU02:FS01:0x22:************************************ OpenMM ************************************
15:34:00:WU02:FS01:0x22:   Revision: 189320d0
15:34:00:WU02:FS01:0x22:********************************************************************************
15:34:00:WU02:FS01:0x22:Project: 17727 (Run 41, Clone 2, Gen 61)
15:34:00:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
15:34:00:WU02:FS01:0x22:Digital signatures verified
15:34:00:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
15:34:00:WU02:FS01:0x22:Version 0.0.13
15:34:00:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
15:34:00:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
15:34:00:WU02:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
15:34:00:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
15:34:00:WU02:FS01:0x22:There are 3 platforms available.
15:34:00:WU02:FS01:0x22:Platform 0: Reference
15:34:00:WU02:FS01:0x22:Platform 1: CPU
15:34:00:WU02:FS01:0x22:Platform 2: OpenCL
15:34:00:WU02:FS01:0x22:  opencl-device 0 specified
15:34:17:WU02:FS01:0x22:Attempting to create OpenCL context:
15:34:17:WU02:FS01:0x22:  Configuring platform OpenCL
15:34:34:WU02:FS01:0x22:  Using OpenCL on platformId 0 and gpu 0
15:34:35:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
15:36:03:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
15:37:28:WU02:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
15:38:54:WU02:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
15:40:20:WU02:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
15:41:47:WU02:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
15:41:50:WU02:FS01:0x22:Checkpoint completed at step 50000
15:43:15:WU02:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
15:44:37:WU02:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
15:45:21:WU02:FS01:0x22:An exception occurred at step 74797: Particle coordinate is nan
15:45:21:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
15:45:21:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
15:45:21:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
15:45:22:WU02:FS01:Starting
15:45:22:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 29100 -checkpoint 5 -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0 -gpu-usage 100
15:45:22:WU02:FS01:Started FahCore on PID 36080
15:45:22:WU02:FS01:Core PID:23136
15:45:22:WU02:FS01:FahCore 0x22 started
15:45:22:WU02:FS01:0x22:*********************** Log Started 2021-04-26T15:45:22Z ***********************
15:45:22:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
15:45:22:WU02:FS01:0x22:       Core: Core22
15:45:22:WU02:FS01:0x22:       Type: 0x22
15:45:22:WU02:FS01:0x22:    Version: 0.0.13
15:45:22:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:45:22:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
15:45:22:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
15:45:22:WU02:FS01:0x22:       Date: Sep 19 2020
15:45:22:WU02:FS01:0x22:       Time: 02:35:58
15:45:22:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
15:45:22:WU02:FS01:0x22:     Branch: core22-0.0.13
15:45:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
15:45:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
15:45:22:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
15:45:22:WU02:FS01:0x22:   Platform: win32 10
15:45:22:WU02:FS01:0x22:       Bits: 64
15:45:22:WU02:FS01:0x22:       Mode: Release
15:45:22:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
15:45:22:WU02:FS01:0x22:             <peastman@stanford.edu>
15:45:22:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 36080 -checkpoint 5
15:45:22:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0
15:45:22:WU02:FS01:0x22:             -gpu-usage 100
15:45:22:WU02:FS01:0x22:************************************ libFAH ************************************
15:45:22:WU02:FS01:0x22:       Date: Sep 7 2020
15:45:22:WU02:FS01:0x22:       Time: 19:09:56
15:45:22:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
15:45:22:WU02:FS01:0x22:     Branch: HEAD
15:45:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
15:45:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
15:45:22:WU02:FS01:0x22:   Platform: win32 10
15:45:22:WU02:FS01:0x22:       Bits: 64
15:45:22:WU02:FS01:0x22:       Mode: Release
15:45:22:WU02:FS01:0x22:************************************ CBang *************************************
15:45:22:WU02:FS01:0x22:       Date: Sep 7 2020
15:45:22:WU02:FS01:0x22:       Time: 19:08:30
15:45:22:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
15:45:22:WU02:FS01:0x22:     Branch: HEAD
15:45:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
15:45:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
15:45:22:WU02:FS01:0x22:   Platform: win32 10
15:45:22:WU02:FS01:0x22:       Bits: 64
15:45:22:WU02:FS01:0x22:       Mode: Release
15:45:22:WU02:FS01:0x22:************************************ System ************************************
15:45:22:WU02:FS01:0x22:        CPU: AMD Ryzen 5 2600 Six-Core Processor
15:45:22:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
15:45:22:WU02:FS01:0x22:       CPUs: 12
15:45:22:WU02:FS01:0x22:     Memory: 15.95GiB
15:45:22:WU02:FS01:0x22:Free Memory: 4.95GiB
15:45:22:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
15:45:22:WU02:FS01:0x22: OS Version: 6.2
15:45:22:WU02:FS01:0x22:Has Battery: false
15:45:22:WU02:FS01:0x22: On Battery: false
15:45:22:WU02:FS01:0x22: UTC Offset: 2
15:45:22:WU02:FS01:0x22:        PID: 23136
15:45:22:WU02:FS01:0x22:        CWD: C:\ProgramData\FAHClient\work
15:45:22:WU02:FS01:0x22:************************************ OpenMM ************************************
15:45:22:WU02:FS01:0x22:   Revision: 189320d0
15:45:22:WU02:FS01:0x22:********************************************************************************
15:45:22:WU02:FS01:0x22:Project: 17727 (Run 41, Clone 2, Gen 61)
15:45:22:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
15:45:22:WU02:FS01:0x22:Digital signatures verified
15:45:22:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
15:45:22:WU02:FS01:0x22:Version 0.0.13
15:45:22:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
15:45:22:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
15:45:22:WU02:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
15:45:22:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
15:45:22:WU02:FS01:0x22:There are 3 platforms available.
15:45:22:WU02:FS01:0x22:Platform 0: Reference
15:45:22:WU02:FS01:0x22:Platform 1: CPU
15:45:22:WU02:FS01:0x22:Platform 2: OpenCL
15:45:22:WU02:FS01:0x22:  opencl-device 0 specified
15:45:37:WU02:FS01:0x22:Attempting to create OpenCL context:
15:45:37:WU02:FS01:0x22:  Configuring platform OpenCL
15:45:53:WU02:FS01:0x22:  Using OpenCL on platformId 0 and gpu 0
15:45:54:WU02:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
15:47:22:WU02:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
15:48:50:WU02:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
15:50:15:WU02:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
15:51:43:WU02:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
15:53:09:WU02:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
15:53:12:WU02:FS01:0x22:Checkpoint completed at step 100000
15:54:39:WU02:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
15:56:06:WU02:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
15:57:32:WU02:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
15:58:57:WU02:FS01:0x22:Completed 140000 out of 1000000 steps (14%)
16:00:20:WU02:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
16:00:23:WU02:FS01:0x22:Checkpoint completed at step 150000
16:01:44:WU02:FS01:0x22:An exception occurred at step 158631: Particle coordinate is nan
16:01:44:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
16:01:44:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
16:01:45:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
16:01:45:WU02:FS01:Starting
16:01:45:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 29100 -checkpoint 5 -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0 -gpu-usage 100
16:01:45:WU02:FS01:Started FahCore on PID 39588
16:01:45:WU02:FS01:Core PID:7340
16:01:45:WU02:FS01:FahCore 0x22 started
16:01:45:WU02:FS01:0x22:*********************** Log Started 2021-04-26T16:01:45Z ***********************
16:01:45:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
16:01:45:WU02:FS01:0x22:       Core: Core22
16:01:45:WU02:FS01:0x22:       Type: 0x22
16:01:45:WU02:FS01:0x22:    Version: 0.0.13
16:01:45:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:01:45:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
16:01:45:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
16:01:45:WU02:FS01:0x22:       Date: Sep 19 2020
16:01:45:WU02:FS01:0x22:       Time: 02:35:58
16:01:45:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:01:45:WU02:FS01:0x22:     Branch: core22-0.0.13
16:01:45:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:01:45:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:45:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:01:45:WU02:FS01:0x22:   Platform: win32 10
16:01:45:WU02:FS01:0x22:       Bits: 64
16:01:45:WU02:FS01:0x22:       Mode: Release
16:01:45:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:01:45:WU02:FS01:0x22:             <peastman@stanford.edu>
16:01:45:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 39588 -checkpoint 5
16:01:45:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0
16:01:45:WU02:FS01:0x22:             -gpu-usage 100
16:01:45:WU02:FS01:0x22:************************************ libFAH ************************************
16:01:45:WU02:FS01:0x22:       Date: Sep 7 2020
16:01:45:WU02:FS01:0x22:       Time: 19:09:56
16:01:45:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:01:45:WU02:FS01:0x22:     Branch: HEAD
16:01:45:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:01:45:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:45:WU02:FS01:0x22:   Platform: win32 10
16:01:45:WU02:FS01:0x22:       Bits: 64
16:01:45:WU02:FS01:0x22:       Mode: Release
16:01:45:WU02:FS01:0x22:************************************ CBang *************************************
16:01:45:WU02:FS01:0x22:       Date: Sep 7 2020
16:01:45:WU02:FS01:0x22:       Time: 19:08:30
16:01:45:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:01:45:WU02:FS01:0x22:     Branch: HEAD
16:01:45:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:01:45:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:45:WU02:FS01:0x22:   Platform: win32 10
16:01:45:WU02:FS01:0x22:       Bits: 64
16:01:45:WU02:FS01:0x22:       Mode: Release
16:01:45:WU02:FS01:0x22:************************************ System ************************************
16:01:45:WU02:FS01:0x22:        CPU: AMD Ryzen 5 2600 Six-Core Processor
16:01:45:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
16:01:45:WU02:FS01:0x22:       CPUs: 12
16:01:45:WU02:FS01:0x22:     Memory: 15.95GiB
16:01:45:WU02:FS01:0x22:Free Memory: 5.41GiB
16:01:45:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
16:01:45:WU02:FS01:0x22: OS Version: 6.2
16:01:45:WU02:FS01:0x22:Has Battery: false
16:01:45:WU02:FS01:0x22: On Battery: false
16:01:45:WU02:FS01:0x22: UTC Offset: 2
16:01:45:WU02:FS01:0x22:        PID: 7340
16:01:45:WU02:FS01:0x22:        CWD: C:\ProgramData\FAHClient\work
16:01:45:WU02:FS01:0x22:************************************ OpenMM ************************************
16:01:45:WU02:FS01:0x22:   Revision: 189320d0
16:01:45:WU02:FS01:0x22:********************************************************************************
16:01:45:WU02:FS01:0x22:Project: 17727 (Run 41, Clone 2, Gen 61)
16:01:45:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
16:01:45:WU02:FS01:0x22:Digital signatures verified
16:01:45:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:01:45:WU02:FS01:0x22:Version 0.0.13
16:01:45:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
16:01:45:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
16:01:45:WU02:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
16:01:45:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
16:01:45:WU02:FS01:0x22:There are 3 platforms available.
16:01:46:WU02:FS01:0x22:Platform 0: Reference
16:01:46:WU02:FS01:0x22:Platform 1: CPU
16:01:46:WU02:FS01:0x22:Platform 2: OpenCL
16:01:46:WU02:FS01:0x22:  opencl-device 0 specified
16:02:01:WU02:FS01:0x22:Attempting to create OpenCL context:
16:02:01:WU02:FS01:0x22:  Configuring platform OpenCL
16:02:20:WU02:FS01:0x22:  Using OpenCL on platformId 0 and gpu 0
16:02:20:WU02:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
16:03:48:WU02:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
16:05:13:WU02:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
16:06:41:WU02:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
16:07:03:WU02:FS01:0x22:An exception occurred at step 182225: Particle coordinate is nan
16:07:03:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
16:07:03:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
16:07:04:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
16:07:04:WU02:FS01:Starting
16:07:04:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 29100 -checkpoint 5 -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0 -gpu-usage 100
16:07:04:WU02:FS01:Started FahCore on PID 38840
16:07:04:WU02:FS01:Core PID:27792
16:07:04:WU02:FS01:FahCore 0x22 started
16:07:04:WU02:FS01:0x22:*********************** Log Started 2021-04-26T16:07:04Z ***********************
16:07:04:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
16:07:04:WU02:FS01:0x22:       Core: Core22
16:07:04:WU02:FS01:0x22:       Type: 0x22
16:07:04:WU02:FS01:0x22:    Version: 0.0.13
16:07:04:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:07:04:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
16:07:04:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
16:07:04:WU02:FS01:0x22:       Date: Sep 19 2020
16:07:04:WU02:FS01:0x22:       Time: 02:35:58
16:07:04:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:07:04:WU02:FS01:0x22:     Branch: core22-0.0.13
16:07:04:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:07:04:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:07:04:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:07:04:WU02:FS01:0x22:   Platform: win32 10
16:07:04:WU02:FS01:0x22:       Bits: 64
16:07:04:WU02:FS01:0x22:       Mode: Release
16:07:04:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:07:04:WU02:FS01:0x22:             <peastman@stanford.edu>
16:07:04:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 38840 -checkpoint 5
16:07:04:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0
16:07:04:WU02:FS01:0x22:             -gpu-usage 100
16:07:04:WU02:FS01:0x22:************************************ libFAH ************************************
16:07:04:WU02:FS01:0x22:       Date: Sep 7 2020
16:07:04:WU02:FS01:0x22:       Time: 19:09:56
16:07:04:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:07:05:WU02:FS01:0x22:     Branch: HEAD
16:07:05:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:07:05:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:07:05:WU02:FS01:0x22:   Platform: win32 10
16:07:05:WU02:FS01:0x22:       Bits: 64
16:07:05:WU02:FS01:0x22:       Mode: Release
16:07:05:WU02:FS01:0x22:************************************ CBang *************************************
16:07:05:WU02:FS01:0x22:       Date: Sep 7 2020
16:07:05:WU02:FS01:0x22:       Time: 19:08:30
16:07:05:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:07:05:WU02:FS01:0x22:     Branch: HEAD
16:07:05:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:07:05:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:07:05:WU02:FS01:0x22:   Platform: win32 10
16:07:05:WU02:FS01:0x22:       Bits: 64
16:07:05:WU02:FS01:0x22:       Mode: Release
16:07:05:WU02:FS01:0x22:************************************ System ************************************
16:07:05:WU02:FS01:0x22:        CPU: AMD Ryzen 5 2600 Six-Core Processor
16:07:05:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
16:07:05:WU02:FS01:0x22:       CPUs: 12
16:07:05:WU02:FS01:0x22:     Memory: 15.95GiB
16:07:05:WU02:FS01:0x22:Free Memory: 4.56GiB
16:07:05:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
16:07:05:WU02:FS01:0x22: OS Version: 6.2
16:07:05:WU02:FS01:0x22:Has Battery: false
16:07:05:WU02:FS01:0x22: On Battery: false
16:07:05:WU02:FS01:0x22: UTC Offset: 2
16:07:05:WU02:FS01:0x22:        PID: 27792
16:07:05:WU02:FS01:0x22:        CWD: C:\ProgramData\FAHClient\work
16:07:05:WU02:FS01:0x22:************************************ OpenMM ************************************
16:07:05:WU02:FS01:0x22:   Revision: 189320d0
16:07:05:WU02:FS01:0x22:********************************************************************************
16:07:05:WU02:FS01:0x22:Project: 17727 (Run 41, Clone 2, Gen 61)
16:07:05:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
16:07:05:WU02:FS01:0x22:Digital signatures verified
16:07:05:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:07:05:WU02:FS01:0x22:Version 0.0.13
16:07:05:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
16:07:05:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
16:07:05:WU02:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
16:07:05:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
16:07:05:WU02:FS01:0x22:There are 3 platforms available.
16:07:05:WU02:FS01:0x22:Platform 0: Reference
16:07:05:WU02:FS01:0x22:Platform 1: CPU
16:07:05:WU02:FS01:0x22:Platform 2: OpenCL
16:07:05:WU02:FS01:0x22:  opencl-device 0 specified
16:07:20:WU02:FS01:0x22:Attempting to create OpenCL context:
16:07:20:WU02:FS01:0x22:  Configuring platform OpenCL
16:07:37:WU02:FS01:0x22:  Using OpenCL on platformId 0 and gpu 0
16:07:37:WU02:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
16:09:02:WU02:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
16:10:28:WU02:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
16:11:54:WU02:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
16:13:19:WU02:FS01:0x22:Completed 190000 out of 1000000 steps (19%)
16:14:43:WU02:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
16:14:46:WU02:FS01:0x22:Checkpoint completed at step 200000
16:16:14:WU02:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
16:17:16:WU02:FS01:0x22:An exception occurred at step 216863: Particle coordinate is nan
16:17:16:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
16:17:16:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
16:17:17:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
16:17:17:WU02:FS01:Starting
16:17:17:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 29100 -checkpoint 5 -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0 -gpu-usage 100
16:17:17:WU02:FS01:Started FahCore on PID 39144
16:17:17:WU02:FS01:Core PID:27352
16:17:17:WU02:FS01:FahCore 0x22 started
16:17:18:WU02:FS01:0x22:*********************** Log Started 2021-04-26T16:17:17Z ***********************
16:17:18:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
16:17:18:WU02:FS01:0x22:       Core: Core22
16:17:18:WU02:FS01:0x22:       Type: 0x22
16:17:18:WU02:FS01:0x22:    Version: 0.0.13
16:17:18:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:17:18:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
16:17:18:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
16:17:18:WU02:FS01:0x22:       Date: Sep 19 2020
16:17:18:WU02:FS01:0x22:       Time: 02:35:58
16:17:18:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:17:18:WU02:FS01:0x22:     Branch: core22-0.0.13
16:17:18:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:17:18:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:17:18:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:17:18:WU02:FS01:0x22:   Platform: win32 10
16:17:18:WU02:FS01:0x22:       Bits: 64
16:17:18:WU02:FS01:0x22:       Mode: Release
16:17:18:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:17:18:WU02:FS01:0x22:             <peastman@stanford.edu>
16:17:18:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 39144 -checkpoint 5
16:17:18:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0
16:17:18:WU02:FS01:0x22:             -gpu-usage 100
16:17:18:WU02:FS01:0x22:************************************ libFAH ************************************
16:17:18:WU02:FS01:0x22:       Date: Sep 7 2020
16:17:18:WU02:FS01:0x22:       Time: 19:09:56
16:17:18:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:17:18:WU02:FS01:0x22:     Branch: HEAD
16:17:18:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:17:18:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:17:18:WU02:FS01:0x22:   Platform: win32 10
16:17:18:WU02:FS01:0x22:       Bits: 64
16:17:18:WU02:FS01:0x22:       Mode: Release
16:17:18:WU02:FS01:0x22:************************************ CBang *************************************
16:17:18:WU02:FS01:0x22:       Date: Sep 7 2020
16:17:18:WU02:FS01:0x22:       Time: 19:08:30
16:17:18:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:17:18:WU02:FS01:0x22:     Branch: HEAD
16:17:18:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:17:18:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:17:18:WU02:FS01:0x22:   Platform: win32 10
16:17:18:WU02:FS01:0x22:       Bits: 64
16:17:18:WU02:FS01:0x22:       Mode: Release
16:17:18:WU02:FS01:0x22:************************************ System ************************************
16:17:18:WU02:FS01:0x22:        CPU: AMD Ryzen 5 2600 Six-Core Processor
16:17:18:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
16:17:18:WU02:FS01:0x22:       CPUs: 12
16:17:18:WU02:FS01:0x22:     Memory: 15.95GiB
16:17:18:WU02:FS01:0x22:Free Memory: 4.68GiB
16:17:18:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
16:17:18:WU02:FS01:0x22: OS Version: 6.2
16:17:18:WU02:FS01:0x22:Has Battery: false
16:17:18:WU02:FS01:0x22: On Battery: false
16:17:18:WU02:FS01:0x22: UTC Offset: 2
16:17:18:WU02:FS01:0x22:        PID: 27352
16:17:18:WU02:FS01:0x22:        CWD: C:\ProgramData\FAHClient\work
16:17:18:WU02:FS01:0x22:************************************ OpenMM ************************************
16:17:18:WU02:FS01:0x22:   Revision: 189320d0
16:17:18:WU02:FS01:0x22:********************************************************************************
16:17:18:WU02:FS01:0x22:Project: 17727 (Run 41, Clone 2, Gen 61)
16:17:18:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
16:17:18:WU02:FS01:0x22:Digital signatures verified
16:17:18:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:17:18:WU02:FS01:0x22:Version 0.0.13
16:17:18:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
16:17:18:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
16:17:18:WU02:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
16:17:18:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
16:17:18:WU02:FS01:0x22:There are 3 platforms available.
16:17:18:WU02:FS01:0x22:Platform 0: Reference
16:17:18:WU02:FS01:0x22:Platform 1: CPU
16:17:18:WU02:FS01:0x22:Platform 2: OpenCL
16:17:18:WU02:FS01:0x22:  opencl-device 0 specified
16:17:33:WU02:FS01:0x22:Attempting to create OpenCL context:
16:17:33:WU02:FS01:0x22:  Configuring platform OpenCL
16:17:50:WU02:FS01:0x22:  Using OpenCL on platformId 0 and gpu 0
16:17:50:WU02:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
16:19:18:WU02:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
16:20:40:WU02:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
16:21:13:WU02:FS01:0x22:An exception occurred at step 223640: Particle coordinate is nan
16:21:13:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
16:21:13:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
16:21:14:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
16:21:14:WU02:FS01:Starting
16:21:14:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 29100 -checkpoint 5 -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0 -gpu-usage 100
16:21:14:WU02:FS01:Started FahCore on PID 25604
16:21:14:WU02:FS01:Core PID:40472
16:21:14:WU02:FS01:FahCore 0x22 started
16:21:15:WU02:FS01:0x22:*********************** Log Started 2021-04-26T16:21:14Z ***********************
16:21:15:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
16:21:15:WU02:FS01:0x22:       Core: Core22
16:21:15:WU02:FS01:0x22:       Type: 0x22
16:21:15:WU02:FS01:0x22:    Version: 0.0.13
16:21:15:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:21:15:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
16:21:15:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
16:21:15:WU02:FS01:0x22:       Date: Sep 19 2020
16:21:15:WU02:FS01:0x22:       Time: 02:35:58
16:21:15:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:21:15:WU02:FS01:0x22:     Branch: core22-0.0.13
16:21:15:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:21:15:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:21:15:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:21:15:WU02:FS01:0x22:   Platform: win32 10
16:21:15:WU02:FS01:0x22:       Bits: 64
16:21:15:WU02:FS01:0x22:       Mode: Release
16:21:15:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:21:15:WU02:FS01:0x22:             <peastman@stanford.edu>
16:21:15:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 25604 -checkpoint 5
16:21:15:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0
16:21:15:WU02:FS01:0x22:             -gpu-usage 100
16:21:15:WU02:FS01:0x22:************************************ libFAH ************************************
16:21:15:WU02:FS01:0x22:       Date: Sep 7 2020
16:21:15:WU02:FS01:0x22:       Time: 19:09:56
16:21:15:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:21:15:WU02:FS01:0x22:     Branch: HEAD
16:21:15:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:21:15:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:21:15:WU02:FS01:0x22:   Platform: win32 10
16:21:15:WU02:FS01:0x22:       Bits: 64
16:21:15:WU02:FS01:0x22:       Mode: Release
16:21:15:WU02:FS01:0x22:************************************ CBang *************************************
16:21:15:WU02:FS01:0x22:       Date: Sep 7 2020
16:21:15:WU02:FS01:0x22:       Time: 19:08:30
16:21:15:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:21:15:WU02:FS01:0x22:     Branch: HEAD
16:21:15:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:21:15:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:21:15:WU02:FS01:0x22:   Platform: win32 10
16:21:15:WU02:FS01:0x22:       Bits: 64
16:21:15:WU02:FS01:0x22:       Mode: Release
16:21:15:WU02:FS01:0x22:************************************ System ************************************
16:21:15:WU02:FS01:0x22:        CPU: AMD Ryzen 5 2600 Six-Core Processor
16:21:15:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
16:21:15:WU02:FS01:0x22:       CPUs: 12
16:21:15:WU02:FS01:0x22:     Memory: 15.95GiB
16:21:15:WU02:FS01:0x22:Free Memory: 4.81GiB
16:21:15:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
16:21:15:WU02:FS01:0x22: OS Version: 6.2
16:21:15:WU02:FS01:0x22:Has Battery: false
16:21:15:WU02:FS01:0x22: On Battery: false
16:21:15:WU02:FS01:0x22: UTC Offset: 2
16:21:15:WU02:FS01:0x22:        PID: 40472
16:21:15:WU02:FS01:0x22:        CWD: C:\ProgramData\FAHClient\work
16:21:15:WU02:FS01:0x22:************************************ OpenMM ************************************
16:21:15:WU02:FS01:0x22:   Revision: 189320d0
16:21:15:WU02:FS01:0x22:********************************************************************************
16:21:15:WU02:FS01:0x22:Project: 17727 (Run 41, Clone 2, Gen 61)
16:21:15:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
16:21:15:WU02:FS01:0x22:Digital signatures verified
16:21:15:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:21:15:WU02:FS01:0x22:Version 0.0.13
16:21:15:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
16:21:15:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
16:21:15:WU02:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
16:21:15:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
16:21:15:WU02:FS01:0x22:There are 3 platforms available.
16:21:15:WU02:FS01:0x22:Platform 0: Reference
16:21:15:WU02:FS01:0x22:Platform 1: CPU
16:21:15:WU02:FS01:0x22:Platform 2: OpenCL
16:21:15:WU02:FS01:0x22:  opencl-device 0 specified
16:21:30:WU02:FS01:0x22:Attempting to create OpenCL context:
16:21:30:WU02:FS01:0x22:  Configuring platform OpenCL
16:21:46:WU02:FS01:0x22:  Using OpenCL on platformId 0 and gpu 0
16:21:46:WU02:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
16:23:15:WU02:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
16:24:42:WU02:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
16:24:48:WU02:FS01:0x22:An exception occurred at step 220126: Particle coordinate is nan
16:24:48:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
16:24:48:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
16:24:49:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
16:24:49:WU02:FS01:Starting
16:24:49:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 29100 -checkpoint 5 -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0 -gpu-usage 100
16:24:49:WU02:FS01:Started FahCore on PID 16468
16:24:49:WU02:FS01:Core PID:24284
16:24:49:WU02:FS01:FahCore 0x22 started
16:24:50:WU02:FS01:0x22:*********************** Log Started 2021-04-26T16:24:49Z ***********************
16:24:50:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
16:24:50:WU02:FS01:0x22:       Core: Core22
16:24:50:WU02:FS01:0x22:       Type: 0x22
16:24:50:WU02:FS01:0x22:    Version: 0.0.13
16:24:50:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:24:50:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
16:24:50:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
16:24:50:WU02:FS01:0x22:       Date: Sep 19 2020
16:24:50:WU02:FS01:0x22:       Time: 02:35:58
16:24:50:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:24:50:WU02:FS01:0x22:     Branch: core22-0.0.13
16:24:50:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:24:50:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:24:50:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:24:50:WU02:FS01:0x22:   Platform: win32 10
16:24:50:WU02:FS01:0x22:       Bits: 64
16:24:50:WU02:FS01:0x22:       Mode: Release
16:24:50:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:24:50:WU02:FS01:0x22:             <peastman@stanford.edu>
16:24:50:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 16468 -checkpoint 5
16:24:50:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -gpu-vendor amd -gpu 0
16:24:50:WU02:FS01:0x22:             -gpu-usage 100
16:24:50:WU02:FS01:0x22:************************************ libFAH ************************************
16:24:50:WU02:FS01:0x22:       Date: Sep 7 2020
16:24:50:WU02:FS01:0x22:       Time: 19:09:56
16:24:50:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:24:50:WU02:FS01:0x22:     Branch: HEAD
16:24:50:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:24:50:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:24:50:WU02:FS01:0x22:   Platform: win32 10
16:24:50:WU02:FS01:0x22:       Bits: 64
16:24:50:WU02:FS01:0x22:       Mode: Release
16:24:50:WU02:FS01:0x22:************************************ CBang *************************************
16:24:50:WU02:FS01:0x22:       Date: Sep 7 2020
16:24:50:WU02:FS01:0x22:       Time: 19:08:30
16:24:50:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:24:50:WU02:FS01:0x22:     Branch: HEAD
16:24:50:WU02:FS01:0x22:   Compiler: Visual C++ 2015
16:24:50:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:24:50:WU02:FS01:0x22:   Platform: win32 10
16:24:50:WU02:FS01:0x22:       Bits: 64
16:24:50:WU02:FS01:0x22:       Mode: Release
16:24:50:WU02:FS01:0x22:************************************ System ************************************
16:24:50:WU02:FS01:0x22:        CPU: AMD Ryzen 5 2600 Six-Core Processor
16:24:50:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
16:24:50:WU02:FS01:0x22:       CPUs: 12
16:24:50:WU02:FS01:0x22:     Memory: 15.95GiB
16:24:50:WU02:FS01:0x22:Free Memory: 4.77GiB
16:24:50:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
16:24:50:WU02:FS01:0x22: OS Version: 6.2
16:24:50:WU02:FS01:0x22:Has Battery: false
16:24:50:WU02:FS01:0x22: On Battery: false
16:24:50:WU02:FS01:0x22: UTC Offset: 2
16:24:50:WU02:FS01:0x22:        PID: 24284
16:24:50:WU02:FS01:0x22:        CWD: C:\ProgramData\FAHClient\work
16:24:50:WU02:FS01:0x22:************************************ OpenMM ************************************
16:24:50:WU02:FS01:0x22:   Revision: 189320d0
16:24:50:WU02:FS01:0x22:********************************************************************************
16:24:50:WU02:FS01:0x22:Project: 17727 (Run 41, Clone 2, Gen 61)
16:24:50:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
16:24:50:WU02:FS01:0x22:Digital signatures verified
16:24:50:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:24:50:WU02:FS01:0x22:Version 0.0.13
16:24:50:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
16:24:50:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
16:24:50:WU02:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
16:24:50:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
16:24:50:WU02:FS01:0x22:There are 3 platforms available.
16:24:50:WU02:FS01:0x22:Platform 0: Reference
16:24:50:WU02:FS01:0x22:Platform 1: CPU
16:24:50:WU02:FS01:0x22:Platform 2: OpenCL
16:24:50:WU02:FS01:0x22:  opencl-device 0 specified
16:25:05:WU02:FS01:0x22:Attempting to create OpenCL context:
16:25:05:WU02:FS01:0x22:  Configuring platform OpenCL
16:25:21:WU02:FS01:0x22:  Using OpenCL on platformId 0 and gpu 0
16:25:21:WU02:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
16:26:45:WU02:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
16:28:08:WU02:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
16:29:34:WU02:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
16:31:05:WU02:FS01:0x22:An exception occurred at step 239955: Particle coordinate is nan
16:31:05:WU02:FS01:0x22:Max number of attempts to resume from last checkpoint (4) reached. Aborting.
16:31:05:WU02:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
16:31:05:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
16:31:05:WU02:FS01:0x22:Saving result file positions.xtc
16:31:06:WU02:FS01:0x22:Saving result file science.log
16:31:06:WU02:FS01:0x22:Saving result file state.xml
16:31:07:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:31:07:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:31:07:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:17727 run:41 clone:2 gen:61 core:0x22 unit:0x000000020000003d0000453f00000029
16:31:07:WU02:FS01:Uploading 8.23MiB to 128.174.73.74
16:31:07:WU02:FS01:Connecting to 128.174.73.74:8080
16:31:09:WU00:FS01:Connecting to assign1.foldingathome.org:80
16:31:13:WU02:FS01:Upload 5.32%
16:31:19:WU02:FS01:Upload 10.63%

[bruce]

NaNs can be caused by faulty hardware (including overclocking), corrupt software, or they can be inherent in the initial characteristics of the WU, itself. Ordinarily, the inherently bad WUs are weeded out during or before beta testing but that's not always true. The only good way to tell the difference is to reassign the WU to someone else and see if they can complete it. Keep checking https://apps.foldingathome.org/wu#proje ... e=2&gen=61

[Talgeeze]

Well I don't do overclocking though I suspect my GPU might be factory-overclocked (I have a small doubt about the exact model). I also tried a full FAH client reinstall (that's why there's FahCore download in the log I posted) and a full clean GPU drivers install.
When it comes to the WU status after being reassigned, of all the WUs that I got in the last few days: 5 were faulty, 4 couldn't be found in the database, 6 haven't been reassigned yet and 29 were completed. All of them returned as failed on my computer (100 % fail rate).

[JimboPalmer]

Msi afterburner is billed as overclocking software, but you can use it to underclock as well.

https://www.msi.com/Landing/afterburner/graphics-cards

[Talgeeze]

Issue is I don't know how/what to change without damaging my hardware.

[gunnarre]

The exact same WU failing at different points also suggests that system instability is the issue, rather than something wrong with the WU itself. And when other folders are able to successfully fold the WU, then it's more definitively your system that is unstable in some way.

You can use the settings in the Radeon software from AMD, or you can use MSI's Afterburner software, to adjust the frequency and power limit of the GPU down to a stable level. General advice for instability is to try to reduce the max frequency and power limit. Increasing the voltage might help stability, but it might also just lead to more power usage for no benefit.

Issue is I don't know how/what to change without damaging my hardware.

There are limits in the firmware that you typically can't override without a special VBIOS, so it's usually safe to change the settings in the AMD Radeon softare or MSI Afterburner.

Note that the actual problem might be with your power supply or motherboard, so if you have spare parts to swap out, or perhaps try the GPU in a different machine, that might uncover that. Some AIB cards have higher quality power stages than others, and the cheaper variants might be more sensitive to power quality than others.

[Talgeeze]

You can use the settings in the Radeon software from AMD, or you can use MSI's Afterburner software, to adjust the frequency and power limit of the GPU down to a stable level. General advice for instability is to try to reduce the max frequency and power limit. Increasing the voltage might help stability, but it might also just lead to more power usage for no benefit.

Any idea by how much I should lower the max frequency?

[gunnarre]

Try the stock RX 590 frequency first - 1545 MHz max boost, and 2000 MHz max memory frequency. If it's still unstable, try to go even lower.

It might be a good idea to install GPU-Z to see what the GPU is actually doing. Even if you request a particular max frequency, the GPU typically sets the frequency in steps, so e.g. 1545 MHz vs. 1546 MHz has no practical difference.

[Talgeeze]

I've set frequency of highest P-state to 1545 Mhz. Been folding for ~ two hours (80% progress on current WU as I'm writing this) without a single error yet.
Call me crazy, but after comparing progress times I think it might be folding slightly faster than with the original settings.

[gunnarre]

There is variability in folding speed between different projects, so you need to compare the same project to itself. Sometimes WUs from the same project has different performance as well. Also, there is a quick-return bonus, so even if you run the exact same WU twice, the second run will estimate a lower quick return bonus. There are tools for more exact benchmarking of GPU folding outside of the F@H client.

That said, in some GPUs, setting the memory frequency too high shows itself in slower performance, because the error-detection in the memory protocol in those cards will re-transmit the data rather than crash immediately. The GDDR5 standard used by the RX 590 uses this, so if we assume that the WUs were equal, then could be a symptom of a bad memory overclock. The crashes were likely due to a bad GPU overclock, that might have worked in the AIB designer's machines, but not in yours. (Or they didn't test their overclock with anything as hard-working as Folding@Home - only a few benchmarks and games maybe.)

[Talgeeze]

The crashes were likely due to a bad GPU overclock, that might have worked in the AIB designer's machines, but not in yours. (Or they didn't test their overclock with anything as hard-working as Folding@Home - only a few benchmarks and games maybe.)

What's weird is FAHBench didn't return any error, but I could have not set it up correctly. On the other hand after double-checking I noticed I didn't set it at 1545 but at 1515 Mhz. Might try to up it and see how it goes.
Might worth a try to to check the 2 or 3 games I was having crashes with.

The fun part: I was 100 % sure I had a stock 1545 Mhz RX 590 (the sticker on the box the GPU came has the stock model reference) but somehow I got an overclocked model.

[Joe_H]

The current version of FAHBench is based on the code used for the previous GPU folding core. Core_22 is known for more fully utilizing the GPU, so that might be part of the difference.

[Talgeeze]

In any case it looks like the issue is resolved.

[JimboPalmer]

In any case it looks like the issue is resolved.

And I thank you for coming back to confirm that. Some times we give advice but never know if it was good advice or not.

[Talgeeze]

The only odd thing I noticed is, for some WUs, the average GPU usage is quite low (~45%) but errors are all gone.