16600 consistently crashing on AMD Radeon VII

[NormalDiffusion]

I would like to report that project 16600 is almost always crashing on Radeon VII.
This behaviour can be seen on 2 different cards running in 2 different machines. Four (4) units out of 51 were completed on the first PC, the bad work units stopping anywhere between 0 and 93 frames. On the second machine, eight (8) out of 54 could be completed. On a side note: on a Radeon 290x (running with the Radeon VII in the second machine), one (1) unit out of seven (7) could be completed.

Here is the (partial) log for one unit (machine 1, running only the RVII):

Code: Select all

12:37:02:WU00:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
12:37:02:WU00:FS01:Started FahCore on PID 12056
12:37:02:WU00:FS01:Core PID:14896
12:37:02:WU00:FS01:FahCore 0x22 started
12:37:03:WU00:FS01:0x22:*********************** Log Started 2020-08-05T12:37:02Z ***********************
12:37:03:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
12:37:03:WU00:FS01:0x22:       Core: Core22
12:37:03:WU00:FS01:0x22:       Type: 0x22
12:37:03:WU00:FS01:0x22:    Version: 0.0.11
12:37:03:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
12:37:03:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
12:37:03:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
12:37:03:WU00:FS01:0x22:       Date: Jun 26 2020
12:37:03:WU00:FS01:0x22:       Time: 19:49:16
12:37:03:WU00:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
12:37:03:WU00:FS01:0x22:     Branch: core22-0.0.11
12:37:03:WU00:FS01:0x22:   Compiler: Visual C++ 2015
12:37:03:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:37:03:WU00:FS01:0x22:   Platform: win32 10
12:37:03:WU00:FS01:0x22:       Bits: 64
12:37:03:WU00:FS01:0x22:       Mode: Release
12:37:03:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
12:37:03:WU00:FS01:0x22:             <peastman@stanford.edu>
12:37:03:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 12056 -checkpoint 15
12:37:03:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
12:37:03:WU00:FS01:0x22:************************************ libFAH ************************************
12:37:03:WU00:FS01:0x22:       Date: Jun 26 2020
12:37:03:WU00:FS01:0x22:       Time: 19:47:12
12:37:03:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
12:37:03:WU00:FS01:0x22:     Branch: HEAD
12:37:03:WU00:FS01:0x22:   Compiler: Visual C++ 2015
12:37:03:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:37:03:WU00:FS01:0x22:   Platform: win32 10
12:37:03:WU00:FS01:0x22:       Bits: 64
12:37:03:WU00:FS01:0x22:       Mode: Release
12:37:03:WU00:FS01:0x22:************************************ CBang *************************************
12:37:03:WU00:FS01:0x22:       Date: Jun 26 2020
12:37:03:WU00:FS01:0x22:       Time: 19:46:11
12:37:03:WU00:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
12:37:03:WU00:FS01:0x22:     Branch: master
12:37:03:WU00:FS01:0x22:   Compiler: Visual C++ 2015
12:37:03:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:37:03:WU00:FS01:0x22:   Platform: win32 10
12:37:03:WU00:FS01:0x22:       Bits: 64
12:37:03:WU00:FS01:0x22:       Mode: Release
12:37:03:WU00:FS01:0x22:************************************ System ************************************
12:37:03:WU00:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
12:37:03:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
12:37:03:WU00:FS01:0x22:       CPUs: 12
12:37:03:WU00:FS01:0x22:     Memory: 31.92GiB
12:37:03:WU00:FS01:0x22:Free Memory: 26.75GiB
12:37:03:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
12:37:03:WU00:FS01:0x22: OS Version: 6.2
12:37:03:WU00:FS01:0x22:Has Battery: false
12:37:03:WU00:FS01:0x22: On Battery: false
12:37:03:WU00:FS01:0x22: UTC Offset: 2
12:37:03:WU00:FS01:0x22:        PID: 14896
12:37:03:WU00:FS01:0x22:        CWD: C:\FAHClient\work
12:37:03:WU00:FS01:0x22:********************************************************************************
12:37:03:WU00:FS01:0x22:Project: 16448 (Run 0, Clone 2285, Gen 188)
12:37:03:WU00:FS01:0x22:Unit: 0x0000012f42aa6f325ed18ce28c2619b5
12:37:03:WU00:FS01:0x22:Reading tar file core.xml
12:37:03:WU00:FS01:0x22:Reading tar file integrator.xml
12:37:03:WU00:FS01:0x22:Reading tar file state.xml
12:37:04:WU00:FS01:0x22:Reading tar file system.xml
12:37:05:WU00:FS01:0x22:Digital signatures verified
12:37:05:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
12:37:05:WU00:FS01:0x22:Version 0.0.11
12:37:05:WU00:FS01:0x22:  Checkpoint write interval: 30000 steps (5%) [20 total]
12:37:05:WU00:FS01:0x22:  JSON viewer frame write interval: 6000 steps (1%) [100 total]
12:37:05:WU00:FS01:0x22:  XTC frame write interval: 20000 steps (3.3%) [30 total]
12:37:05:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
12:37:28:WU00:FS01:0x22:Completed 0 out of 600000 steps (0%)
12:38:39:WU00:FS01:0x22:An exception occurred at step 5270: Particle coordinate is nan
12:38:39:WU00:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
12:38:39:WU00:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
12:38:39:WARNING:WU00:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
12:38:39:WU00:FS01:Starting
12:38:39:WU00:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
12:38:39:WU00:FS01:Started FahCore on PID 9236
12:38:39:WU00:FS01:Core PID:14720
12:38:39:WU00:FS01:FahCore 0x22 started
12:38:40:WU00:FS01:0x22:*********************** Log Started 2020-08-05T12:38:39Z ***********************
12:38:40:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
12:38:40:WU00:FS01:0x22:       Core: Core22
12:38:40:WU00:FS01:0x22:       Type: 0x22
12:38:40:WU00:FS01:0x22:    Version: 0.0.11
12:38:40:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
12:38:40:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
12:38:40:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
12:38:40:WU00:FS01:0x22:       Date: Jun 26 2020
12:38:40:WU00:FS01:0x22:       Time: 19:49:16
12:38:40:WU00:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
12:38:40:WU00:FS01:0x22:     Branch: core22-0.0.11
12:38:40:WU00:FS01:0x22:   Compiler: Visual C++ 2015
12:38:40:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:38:40:WU00:FS01:0x22:   Platform: win32 10
12:38:40:WU00:FS01:0x22:       Bits: 64
12:38:40:WU00:FS01:0x22:       Mode: Release
12:38:40:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
12:38:40:WU00:FS01:0x22:             <peastman@stanford.edu>
12:38:40:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 9236 -checkpoint 15
12:38:40:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
12:38:40:WU00:FS01:0x22:************************************ libFAH ************************************
12:38:40:WU00:FS01:0x22:       Date: Jun 26 2020
12:38:40:WU00:FS01:0x22:       Time: 19:47:12
12:38:40:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
12:38:40:WU00:FS01:0x22:     Branch: HEAD
12:38:40:WU00:FS01:0x22:   Compiler: Visual C++ 2015
12:38:40:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:38:40:WU00:FS01:0x22:   Platform: win32 10
12:38:40:WU00:FS01:0x22:       Bits: 64
12:38:40:WU00:FS01:0x22:       Mode: Release
12:38:40:WU00:FS01:0x22:************************************ CBang *************************************
12:38:40:WU00:FS01:0x22:       Date: Jun 26 2020
12:38:40:WU00:FS01:0x22:       Time: 19:46:11
12:38:40:WU00:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
12:38:40:WU00:FS01:0x22:     Branch: master
12:38:40:WU00:FS01:0x22:   Compiler: Visual C++ 2015
12:38:40:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
12:38:40:WU00:FS01:0x22:   Platform: win32 10
12:38:40:WU00:FS01:0x22:       Bits: 64
12:38:40:WU00:FS01:0x22:       Mode: Release
12:38:40:WU00:FS01:0x22:************************************ System ************************************
12:38:40:WU00:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
12:38:40:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
12:38:40:WU00:FS01:0x22:       CPUs: 12
12:38:40:WU00:FS01:0x22:     Memory: 31.92GiB
12:38:40:WU00:FS01:0x22:Free Memory: 26.78GiB
12:38:40:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
12:38:40:WU00:FS01:0x22: OS Version: 6.2
12:38:40:WU00:FS01:0x22:Has Battery: false
12:38:40:WU00:FS01:0x22: On Battery: false
12:38:40:WU00:FS01:0x22: UTC Offset: 2
12:38:40:WU00:FS01:0x22:        PID: 14720
12:38:40:WU00:FS01:0x22:        CWD: C:\FAHClient\work
12:38:40:WU00:FS01:0x22:********************************************************************************
12:38:40:WU00:FS01:0x22:Project: 16448 (Run 0, Clone 2285, Gen 188)
12:38:40:WU00:FS01:0x22:Unit: 0x0000012f42aa6f325ed18ce28c2619b5
12:38:40:WU00:FS01:0x22:Digital signatures verified
12:38:40:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
12:38:40:WU00:FS01:0x22:Version 0.0.11
12:38:40:WU00:FS01:0x22:  Checkpoint write interval: 30000 steps (5%) [20 total]
12:38:40:WU00:FS01:0x22:  JSON viewer frame write interval: 6000 steps (1%) [100 total]
12:38:40:WU00:FS01:0x22:  XTC frame write interval: 20000 steps (3.3%) [30 total]
12:38:40:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
12:39:03:WU00:FS01:0x22:Completed 0 out of 600000 steps (0%)
12:39:36:WU00:FS01:0x22:An exception occurred at step 4015: Particle coordinate is nan
12:39:36:WU00:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
12:39:36:WU00:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
12:39:36:WU00:FS01:0x22:Saving result file ..\logfile_01.txt
12:39:36:WU00:FS01:0x22:Saving result file science.log
12:39:36:WU00:FS01:0x22:Saving result file state.xml
12:39:39:WU00:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
12:39:39:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
12:39:39:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:16448 run:0 clone:2285 gen:188 core:0x22 unit:0x0000012f42aa6f325ed18ce28c2619b5

Just let me know if you like me to provide more data/information.

[muziqaz]

Log is showing p16448 failing, not 16600.
When 16448 was released to Beta it was one of those projects which would fail on GCN based AMD cards (including Radeon VII) due to fahcore_22 (v 0.0.2) having some openCL bug.
This bug has been squashed not long ago and fahcore_22 (v 0.0.11) does not contain it.
It is likely that simulation of that project is not stable.

Would be good to see full system specs

[NormalDiffusion]

Log is showing p16448 failing, not 16600.
When 16448 was released to Beta it was one of those projects which would fail on GCN based AMD cards (including Radeon VII) due to fahcore_22 (v 0.0.2) having some openCL bug.
This bug has been squashed not long ago and fahcore_22 (v 0.0.11) does not contain it.
It is likely that simulation of that project is not stable.

Would be good to see full system specs

Aouch, my bad! Sorry, didn't realized that it was one of the few not 16600 crashing (the log is full of failed 16600, so I just picked one failed WU).


Let's start with the system specs:

Code: Select all

*********************** Log Started 2020-07-21T09:06:25Z ***********************
09:06:25:Trying to access database...
09:06:25:Successfully acquired database lock
09:06:25:Downloading GPUs.txt from assign1.foldingathome.org:80
09:06:25:Connecting to assign1.foldingathome.org:80
09:06:25:Read GPUs.txt
09:06:26:Enabled folding slot 01: READY gpu:0:Vega 20 [Radeon VII]
09:06:26:****************************** FAHClient ******************************
09:06:26:        Version: 7.6.13
09:06:26:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
09:06:26:      Copyright: 2020 foldingathome.org
09:06:26:       Homepage: https://foldingathome.org/
09:06:26:           Date: Apr 27 2020
09:06:26:           Time: 21:21:01
09:06:26:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
09:06:26:         Branch: master
09:06:26:       Compiler: Visual C++ 2008
09:06:26:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:06:26:       Platform: win32 10
09:06:26:           Bits: 32
09:06:26:           Mode: Release
09:06:26:         Config: C:\FAHClient\config.xml
09:06:26:******************************** CBang ********************************
09:06:26:           Date: Apr 24 2020
09:06:26:           Time: 17:07:55
09:06:26:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
09:06:26:         Branch: master
09:06:26:       Compiler: Visual C++ 2008
09:06:26:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:06:26:       Platform: win32 10
09:06:26:           Bits: 32
09:06:26:           Mode: Release
09:06:26:******************************* System ********************************
09:06:26:            CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
09:06:26:         CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
09:06:26:           CPUs: 12
09:06:26:         Memory: 31.92GiB
09:06:26:    Free Memory: 26.97GiB
09:06:26:        Threads: WINDOWS_THREADS
09:06:26:     OS Version: 6.2
09:06:26:    Has Battery: false
09:06:26:     On Battery: false
09:06:26:     UTC Offset: 2
09:06:26:            PID: 12204
09:06:26:            CWD: C:\FAHClient
09:06:26:  Win32 Service: false
09:06:26:             OS: Windows 10 Enterprise
09:06:26:        OS Arch: AMD64
09:06:26:           GPUs: 1
09:06:26:          GPU 0: Bus:5 Slot:0 Func:0 AMD:5 Vega 20 [Radeon VII]
09:06:26:           CUDA: Not detected: Failed to open dynamic library 'nvcuda.dll': The
09:06:26:                 specified module could not be found.
09:06:26:
09:06:26:OpenCL Device 0: Platform:0 Device:0 Bus:5 Slot:0 Compute:1.2 Driver:2906.10
09:06:26:******************************* libFAH ********************************
09:06:26:           Date: Apr 15 2020
09:06:26:           Time: 14:53:14
09:06:26:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
09:06:26:         Branch: master
09:06:26:       Compiler: Visual C++ 2008
09:06:26:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
09:06:26:       Platform: win32 10
09:06:26:           Bits: 32
09:06:26:           Mode: Release
09:06:26:***********************************************************************
09:06:26:<config>
09:06:26:  <!-- Folding Slot Configuration -->
09:06:26:  <cause v='HIGH_PRIORITY'/>
09:06:26:
09:06:26:  <!-- Network -->
09:06:26:  <proxy v=':8080'/>
09:06:26:
09:06:26:  <!-- Slot Control -->
09:06:26:  <power v='full'/>
09:06:26:
09:06:26:  <!-- User Information -->
09:06:26:  <passkey v='*****'/>
09:06:26:  <user v='NormalDiffusion'/>
09:06:26:
09:06:26:  <!-- Folding Slots -->
09:06:26:  <slot id='1' type='GPU'>
09:06:26:    <client-type v='advanced'/>
09:06:26:  </slot>
09:06:26:</config>

Here we go with a fresh one from this morning:

Code: Select all

08:30:44:WU00:FS01:Starting
08:30:44:WU00:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
08:30:44:WU00:FS01:Started FahCore on PID 13300
08:30:44:WU00:FS01:Core PID:5056
08:30:44:WU00:FS01:FahCore 0x22 started
08:30:44:WU00:FS01:0x22:*********************** Log Started 2020-08-06T08:30:44Z ***********************
08:30:44:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
08:30:44:WU00:FS01:0x22:       Core: Core22
08:30:44:WU00:FS01:0x22:       Type: 0x22
08:30:44:WU00:FS01:0x22:    Version: 0.0.11
08:30:44:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:30:44:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
08:30:44:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
08:30:44:WU00:FS01:0x22:       Date: Jun 26 2020
08:30:44:WU00:FS01:0x22:       Time: 19:49:16
08:30:44:WU00:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
08:30:44:WU00:FS01:0x22:     Branch: core22-0.0.11
08:30:44:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:30:44:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:30:44:WU00:FS01:0x22:   Platform: win32 10
08:30:44:WU00:FS01:0x22:       Bits: 64
08:30:44:WU00:FS01:0x22:       Mode: Release
08:30:44:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
08:30:44:WU00:FS01:0x22:             <peastman@stanford.edu>
08:30:44:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 13300 -checkpoint 15
08:30:44:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
08:30:44:WU00:FS01:0x22:************************************ libFAH ************************************
08:30:44:WU00:FS01:0x22:       Date: Jun 26 2020
08:30:44:WU00:FS01:0x22:       Time: 19:47:12
08:30:44:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
08:30:44:WU00:FS01:0x22:     Branch: HEAD
08:30:44:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:30:44:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:30:44:WU00:FS01:0x22:   Platform: win32 10
08:30:44:WU00:FS01:0x22:       Bits: 64
08:30:44:WU00:FS01:0x22:       Mode: Release
08:30:44:WU00:FS01:0x22:************************************ CBang *************************************
08:30:44:WU00:FS01:0x22:       Date: Jun 26 2020
08:30:44:WU00:FS01:0x22:       Time: 19:46:11
08:30:44:WU00:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
08:30:44:WU00:FS01:0x22:     Branch: master
08:30:44:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:30:44:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:30:44:WU00:FS01:0x22:   Platform: win32 10
08:30:44:WU00:FS01:0x22:       Bits: 64
08:30:44:WU00:FS01:0x22:       Mode: Release
08:30:44:WU00:FS01:0x22:************************************ System ************************************
08:30:44:WU00:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
08:30:44:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
08:30:44:WU00:FS01:0x22:       CPUs: 12
08:30:44:WU00:FS01:0x22:     Memory: 31.92GiB
08:30:44:WU00:FS01:0x22:Free Memory: 24.25GiB
08:30:44:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
08:30:44:WU00:FS01:0x22: OS Version: 6.2
08:30:44:WU00:FS01:0x22:Has Battery: false
08:30:44:WU00:FS01:0x22: On Battery: false
08:30:44:WU00:FS01:0x22: UTC Offset: 2
08:30:44:WU00:FS01:0x22:        PID: 5056
08:30:44:WU00:FS01:0x22:        CWD: C:\FAHClient\work
08:30:44:WU00:FS01:0x22:********************************************************************************
08:30:44:WU00:FS01:0x22:Project: 16600 (Run 0, Clone 963, Gen 426)
08:30:44:WU00:FS01:0x22:Unit: 0x000001dd8f59f36f5ec36911585af816
08:30:44:WU00:FS01:0x22:Reading tar file core.xml
08:30:44:WU00:FS01:0x22:Reading tar file integrator.xml
08:30:44:WU00:FS01:0x22:Reading tar file state.xml
08:30:45:WU01:FS01:Upload 75.73%
08:30:45:WU00:FS01:0x22:Reading tar file system.xml
08:30:46:WU00:FS01:0x22:Digital signatures verified
08:30:46:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
08:30:46:WU00:FS01:0x22:Version 0.0.11
08:30:47:WU00:FS01:0x22:  Checkpoint write interval: 25000 steps (5%) [20 total]
08:30:47:WU00:FS01:0x22:  JSON viewer frame write interval: 5000 steps (1%) [100 total]
08:30:47:WU00:FS01:0x22:  XTC frame write interval: 20000 steps (4%) [25 total]
08:30:47:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
08:30:51:WU01:FS01:Upload 81.78%
08:30:57:WU01:FS01:Upload 87.50%
08:31:03:WU01:FS01:Upload 93.55%
08:31:07:WU00:FS01:0x22:Completed 0 out of 500000 steps (0%)
08:31:09:WU01:FS01:Upload 99.28%
08:31:11:WU01:FS01:Upload complete
08:31:11:WU01:FS01:Server responded WORK_ACK (400)
08:31:11:WU01:FS01:Cleaning up
08:31:46:WU00:FS01:0x22:Completed 5000 out of 500000 steps (1%)
08:31:51:FS01:Finishing
08:32:22:WU00:FS01:0x22:Completed 10000 out of 500000 steps (2%)
08:32:59:WU00:FS01:0x22:Completed 15000 out of 500000 steps (3%)
08:33:36:WU00:FS01:0x22:Completed 20000 out of 500000 steps (4%)
08:34:19:WU00:FS01:0x22:An exception occurred at step 20581: Particle coordinate is nan
08:34:19:WU00:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
08:34:19:WU00:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
08:34:19:WARNING:WU00:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
08:34:20:WU00:FS01:Starting
08:34:20:WU00:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
08:34:20:WU00:FS01:Started FahCore on PID 9524
08:34:20:WU00:FS01:Core PID:2656
08:34:20:WU00:FS01:FahCore 0x22 started
08:34:20:WU00:FS01:0x22:*********************** Log Started 2020-08-06T08:34:20Z ***********************
08:34:20:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
08:34:20:WU00:FS01:0x22:       Core: Core22
08:34:20:WU00:FS01:0x22:       Type: 0x22
08:34:20:WU00:FS01:0x22:    Version: 0.0.11
08:34:20:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:34:20:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
08:34:20:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
08:34:20:WU00:FS01:0x22:       Date: Jun 26 2020
08:34:20:WU00:FS01:0x22:       Time: 19:49:16
08:34:20:WU00:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
08:34:20:WU00:FS01:0x22:     Branch: core22-0.0.11
08:34:20:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:34:20:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:34:20:WU00:FS01:0x22:   Platform: win32 10
08:34:20:WU00:FS01:0x22:       Bits: 64
08:34:20:WU00:FS01:0x22:       Mode: Release
08:34:20:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
08:34:20:WU00:FS01:0x22:             <peastman@stanford.edu>
08:34:20:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 9524 -checkpoint 15
08:34:20:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
08:34:20:WU00:FS01:0x22:************************************ libFAH ************************************
08:34:20:WU00:FS01:0x22:       Date: Jun 26 2020
08:34:20:WU00:FS01:0x22:       Time: 19:47:12
08:34:20:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
08:34:20:WU00:FS01:0x22:     Branch: HEAD
08:34:20:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:34:20:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:34:20:WU00:FS01:0x22:   Platform: win32 10
08:34:20:WU00:FS01:0x22:       Bits: 64
08:34:20:WU00:FS01:0x22:       Mode: Release
08:34:20:WU00:FS01:0x22:************************************ CBang *************************************
08:34:20:WU00:FS01:0x22:       Date: Jun 26 2020
08:34:20:WU00:FS01:0x22:       Time: 19:46:11
08:34:20:WU00:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
08:34:20:WU00:FS01:0x22:     Branch: master
08:34:20:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:34:20:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:34:20:WU00:FS01:0x22:   Platform: win32 10
08:34:20:WU00:FS01:0x22:       Bits: 64
08:34:20:WU00:FS01:0x22:       Mode: Release
08:34:20:WU00:FS01:0x22:************************************ System ************************************
08:34:20:WU00:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
08:34:20:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
08:34:20:WU00:FS01:0x22:       CPUs: 12
08:34:20:WU00:FS01:0x22:     Memory: 31.92GiB
08:34:20:WU00:FS01:0x22:Free Memory: 24.36GiB
08:34:20:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
08:34:20:WU00:FS01:0x22: OS Version: 6.2
08:34:20:WU00:FS01:0x22:Has Battery: false
08:34:20:WU00:FS01:0x22: On Battery: false
08:34:20:WU00:FS01:0x22: UTC Offset: 2
08:34:20:WU00:FS01:0x22:        PID: 2656
08:34:20:WU00:FS01:0x22:        CWD: C:\FAHClient\work
08:34:20:WU00:FS01:0x22:********************************************************************************
08:34:20:WU00:FS01:0x22:Project: 16600 (Run 0, Clone 963, Gen 426)
08:34:20:WU00:FS01:0x22:Unit: 0x000001dd8f59f36f5ec36911585af816
08:34:20:WU00:FS01:0x22:Digital signatures verified
08:34:20:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
08:34:20:WU00:FS01:0x22:Version 0.0.11
08:34:20:WU00:FS01:0x22:  Checkpoint write interval: 25000 steps (5%) [20 total]
08:34:20:WU00:FS01:0x22:  JSON viewer frame write interval: 5000 steps (1%) [100 total]
08:34:20:WU00:FS01:0x22:  XTC frame write interval: 20000 steps (4%) [25 total]
08:34:20:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
08:34:41:WU00:FS01:0x22:Completed 0 out of 500000 steps (0%)
08:35:18:WU00:FS01:0x22:Completed 5000 out of 500000 steps (1%)
08:35:54:WU00:FS01:0x22:Completed 10000 out of 500000 steps (2%)
08:36:31:WU00:FS01:0x22:Completed 15000 out of 500000 steps (3%)
08:37:07:WU00:FS01:0x22:Completed 20000 out of 500000 steps (4%)
08:37:44:WU00:FS01:0x22:Completed 25000 out of 500000 steps (5%)
08:38:23:WU00:FS01:0x22:Completed 30000 out of 500000 steps (6%)
08:38:59:WU00:FS01:0x22:Completed 35000 out of 500000 steps (7%)
08:39:36:WU00:FS01:0x22:Completed 40000 out of 500000 steps (8%)
08:40:28:WU00:FS01:0x22:An exception occurred at step 44928: Particle coordinate is nan
08:40:28:WU00:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
08:40:28:WU00:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
08:40:28:WARNING:WU00:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
08:40:29:WU00:FS01:Starting
08:40:29:WU00:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
08:40:29:WU00:FS01:Started FahCore on PID 13540
08:40:29:WU00:FS01:Core PID:6836
08:40:29:WU00:FS01:FahCore 0x22 started
08:40:29:WU00:FS01:0x22:*********************** Log Started 2020-08-06T08:40:29Z ***********************
08:40:29:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
08:40:29:WU00:FS01:0x22:       Core: Core22
08:40:29:WU00:FS01:0x22:       Type: 0x22
08:40:29:WU00:FS01:0x22:    Version: 0.0.11
08:40:29:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:40:29:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
08:40:29:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
08:40:29:WU00:FS01:0x22:       Date: Jun 26 2020
08:40:29:WU00:FS01:0x22:       Time: 19:49:16
08:40:29:WU00:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
08:40:29:WU00:FS01:0x22:     Branch: core22-0.0.11
08:40:29:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:40:29:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:40:29:WU00:FS01:0x22:   Platform: win32 10
08:40:29:WU00:FS01:0x22:       Bits: 64
08:40:29:WU00:FS01:0x22:       Mode: Release
08:40:29:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
08:40:29:WU00:FS01:0x22:             <peastman@stanford.edu>
08:40:29:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 13540 -checkpoint 15
08:40:29:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
08:40:29:WU00:FS01:0x22:************************************ libFAH ************************************
08:40:29:WU00:FS01:0x22:       Date: Jun 26 2020
08:40:29:WU00:FS01:0x22:       Time: 19:47:12
08:40:29:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
08:40:29:WU00:FS01:0x22:     Branch: HEAD
08:40:29:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:40:29:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:40:29:WU00:FS01:0x22:   Platform: win32 10
08:40:29:WU00:FS01:0x22:       Bits: 64
08:40:29:WU00:FS01:0x22:       Mode: Release
08:40:29:WU00:FS01:0x22:************************************ CBang *************************************
08:40:29:WU00:FS01:0x22:       Date: Jun 26 2020
08:40:29:WU00:FS01:0x22:       Time: 19:46:11
08:40:29:WU00:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
08:40:29:WU00:FS01:0x22:     Branch: master
08:40:29:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:40:29:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:40:29:WU00:FS01:0x22:   Platform: win32 10
08:40:29:WU00:FS01:0x22:       Bits: 64
08:40:29:WU00:FS01:0x22:       Mode: Release
08:40:29:WU00:FS01:0x22:************************************ System ************************************
08:40:29:WU00:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
08:40:29:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
08:40:29:WU00:FS01:0x22:       CPUs: 12
08:40:29:WU00:FS01:0x22:     Memory: 31.92GiB
08:40:29:WU00:FS01:0x22:Free Memory: 24.17GiB
08:40:29:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
08:40:29:WU00:FS01:0x22: OS Version: 6.2
08:40:29:WU00:FS01:0x22:Has Battery: false
08:40:29:WU00:FS01:0x22: On Battery: false
08:40:29:WU00:FS01:0x22: UTC Offset: 2
08:40:29:WU00:FS01:0x22:        PID: 6836
08:40:29:WU00:FS01:0x22:        CWD: C:\FAHClient\work
08:40:29:WU00:FS01:0x22:********************************************************************************
08:40:29:WU00:FS01:0x22:Project: 16600 (Run 0, Clone 963, Gen 426)
08:40:29:WU00:FS01:0x22:Unit: 0x000001dd8f59f36f5ec36911585af816
08:40:29:WU00:FS01:0x22:Digital signatures verified
08:40:29:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
08:40:29:WU00:FS01:0x22:Version 0.0.11
08:40:29:WU00:FS01:0x22:  Checkpoint write interval: 25000 steps (5%) [20 total]
08:40:29:WU00:FS01:0x22:  JSON viewer frame write interval: 5000 steps (1%) [100 total]
08:40:29:WU00:FS01:0x22:  XTC frame write interval: 20000 steps (4%) [25 total]
08:40:29:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
08:40:50:WU00:FS01:0x22:Completed 25000 out of 500000 steps (5%)
08:41:27:WU00:FS01:0x22:Completed 30000 out of 500000 steps (6%)
08:42:04:WU00:FS01:0x22:Completed 35000 out of 500000 steps (7%)
08:42:41:WU00:FS01:0x22:Completed 40000 out of 500000 steps (8%)
08:43:18:WU00:FS01:0x22:Completed 45000 out of 500000 steps (9%)
08:43:55:WU00:FS01:0x22:Completed 50000 out of 500000 steps (10%)
08:44:33:WU00:FS01:0x22:Completed 55000 out of 500000 steps (11%)
08:45:10:WU00:FS01:0x22:Completed 60000 out of 500000 steps (12%)
08:45:46:WU00:FS01:0x22:An exception occurred at step 63000: Particle coordinate is nan
08:45:46:WU00:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
08:45:46:WU00:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
08:45:46:WARNING:WU00:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
08:45:46:WU00:FS01:Starting
08:45:46:WU00:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
08:45:46:WU00:FS01:Started FahCore on PID 14748
08:45:46:WU00:FS01:Core PID:12524
08:45:46:WU00:FS01:FahCore 0x22 started
08:45:47:WU00:FS01:0x22:*********************** Log Started 2020-08-06T08:45:46Z ***********************
08:45:47:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
08:45:47:WU00:FS01:0x22:       Core: Core22
08:45:47:WU00:FS01:0x22:       Type: 0x22
08:45:47:WU00:FS01:0x22:    Version: 0.0.11
08:45:47:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
08:45:47:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
08:45:47:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
08:45:47:WU00:FS01:0x22:       Date: Jun 26 2020
08:45:47:WU00:FS01:0x22:       Time: 19:49:16
08:45:47:WU00:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
08:45:47:WU00:FS01:0x22:     Branch: core22-0.0.11
08:45:47:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:45:47:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:45:47:WU00:FS01:0x22:   Platform: win32 10
08:45:47:WU00:FS01:0x22:       Bits: 64
08:45:47:WU00:FS01:0x22:       Mode: Release
08:45:47:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
08:45:47:WU00:FS01:0x22:             <peastman@stanford.edu>
08:45:47:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 14748 -checkpoint 15
08:45:47:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
08:45:47:WU00:FS01:0x22:************************************ libFAH ************************************
08:45:47:WU00:FS01:0x22:       Date: Jun 26 2020
08:45:47:WU00:FS01:0x22:       Time: 19:47:12
08:45:47:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
08:45:47:WU00:FS01:0x22:     Branch: HEAD
08:45:47:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:45:47:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:45:47:WU00:FS01:0x22:   Platform: win32 10
08:45:47:WU00:FS01:0x22:       Bits: 64
08:45:47:WU00:FS01:0x22:       Mode: Release
08:45:47:WU00:FS01:0x22:************************************ CBang *************************************
08:45:47:WU00:FS01:0x22:       Date: Jun 26 2020
08:45:47:WU00:FS01:0x22:       Time: 19:46:11
08:45:47:WU00:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
08:45:47:WU00:FS01:0x22:     Branch: master
08:45:47:WU00:FS01:0x22:   Compiler: Visual C++ 2015
08:45:47:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
08:45:47:WU00:FS01:0x22:   Platform: win32 10
08:45:47:WU00:FS01:0x22:       Bits: 64
08:45:47:WU00:FS01:0x22:       Mode: Release
08:45:47:WU00:FS01:0x22:************************************ System ************************************
08:45:47:WU00:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
08:45:47:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
08:45:47:WU00:FS01:0x22:       CPUs: 12
08:45:47:WU00:FS01:0x22:     Memory: 31.92GiB
08:45:47:WU00:FS01:0x22:Free Memory: 24.21GiB
08:45:47:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
08:45:47:WU00:FS01:0x22: OS Version: 6.2
08:45:47:WU00:FS01:0x22:Has Battery: false
08:45:47:WU00:FS01:0x22: On Battery: false
08:45:47:WU00:FS01:0x22: UTC Offset: 2
08:45:47:WU00:FS01:0x22:        PID: 12524
08:45:47:WU00:FS01:0x22:        CWD: C:\FAHClient\work
08:45:47:WU00:FS01:0x22:********************************************************************************
08:45:47:WU00:FS01:0x22:Project: 16600 (Run 0, Clone 963, Gen 426)
08:45:47:WU00:FS01:0x22:Unit: 0x000001dd8f59f36f5ec36911585af816
08:45:47:WU00:FS01:0x22:Digital signatures verified
08:45:47:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
08:45:47:WU00:FS01:0x22:Version 0.0.11
08:45:47:WU00:FS01:0x22:  Checkpoint write interval: 25000 steps (5%) [20 total]
08:45:47:WU00:FS01:0x22:  JSON viewer frame write interval: 5000 steps (1%) [100 total]
08:45:47:WU00:FS01:0x22:  XTC frame write interval: 20000 steps (4%) [25 total]
08:45:47:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
08:46:07:WU00:FS01:0x22:Completed 50000 out of 500000 steps (10%)
08:46:45:WU00:FS01:0x22:Completed 55000 out of 500000 steps (11%)
08:47:22:WU00:FS01:0x22:Completed 60000 out of 500000 steps (12%)
08:47:58:WU00:FS01:0x22:Completed 65000 out of 500000 steps (13%)
08:48:35:WU00:FS01:0x22:Completed 70000 out of 500000 steps (14%)
08:49:12:WU00:FS01:0x22:Completed 75000 out of 500000 steps (15%)
08:49:12:WU00:FS01:0x22:An exception occurred at step 75000: Discrepancy: Velocities are blowing up! 18548 1 18.8656
08:49:12:WU00:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
08:49:12:WU00:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.[code]

with the corresponding science.log (sadly not complete as it got deleted to fast...)

[code]*************************** Core22 Folding@home Core ***************************
       Core: Core22
       Type: 0x22
    Version: 0.0.11
     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
  Copyright: 2020 foldingathome.org
   Homepage: https://foldingathome.org/
       Date: Jun 26 2020
       Time: 19:49:16
   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
     Branch: core22-0.0.11
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
             <peastman@stanford.edu>
       Args: -dir 00 -suffix 01 -version 706 -lifeline 13300 -checkpoint 15
             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
************************************ libFAH ************************************
       Date: Jun 26 2020
       Time: 19:47:12
   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
     Branch: HEAD
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
************************************ CBang *************************************
       Date: Jun 26 2020
       Time: 19:46:11
   Revision: f8529962055b0e7bde23e429f5072ff758089dee
     Branch: master
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
************************************ System ************************************
        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
       CPUs: 12
     Memory: 31.92GiB
Free Memory: 24.25GiB
    Threads: WINDOWS_THREADS
 OS Version: 6.2
Has Battery: false
 On Battery: false
 UTC Offset: 2
        PID: 5056
        CWD: C:\FAHClient\work
********************************************************************************
Folding@home GPU Core22 Folding@home Core

Version 0.0.11

[1] compatible platform(s):
  -- 0 --
  PROFILE = FULL_PROFILE
  VERSION = OpenCL 2.1 AMD-APP (2906.10)
  NAME = AMD Accelerated Parallel Processing
  VENDOR = Advanced Micro Devices, Inc.

(1) device(s) found on platform 0:
  -- 0 --
  DEVICE_NAME = gfx906
  DEVICE_VENDOR = Advanced Micro Devices, Inc.
  DEVICE_VERSION = OpenCL 2.0 AMD-APP (2906.10)

[ Entering Init ]
  Launch time: 2020-08-06T08:30:46Z
  Arguments passed: -dir 00 -suffix 01 -version 706 -lifeline 13300 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0 
  For testState comparison of CPU and GPU, will use:
    forceTolerance: 5 kJ/mol/nm
    energyTolerance: 10 kJ/mol
[ Leaving  Init ]
[ Entering Main ]
  Reading core settings...
  Total number of steps: 500000
  Checkpoint write interval: 25000 steps (5%) [20 total]

  JSON viewer frame write interval: 5000 steps (1%) [100 total]

  XTC frame write interval: 20000 steps (4%) [25 total]

  Global context and integrator variables write interval: disabled

[ Initializing Core Contexts ]
  Using platform OpenCL
  Looking for vendor: amd...found on platformId 0
  Setting platform precision to mixed
  Setting DisablePmeStream to 1
    Checking for integrator.xml
    Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
    Checking for integrator.xml
    Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
    Checking for system.xml
    Found system.xml
  Deserializing System...successful.
  Found 226545 atoms, 4 forces.
  Finding State XML file...
    Checking for state.xml
    Found state.xml
  Deserializing State...successful.
    Ewald error tolerance in force 0 is 0.0005
    Ewald parameters: alpha 2.62826 nx 120 ny 120 nz 84
    Integrator Type: class OpenMM::LangevinIntegrator
    Constraint Tolerance: 1e-05
    Time Step in PS: 0.002
    Using CPU platform for reference calculations.
  Performing initial sanity checks before starting work...
  Comparing forces and energies between initial State and CPU...
  Comparing forces and energies between GPU and CPU...
[ Initialized Core Contexts... ]
[ Finding non-water atoms ]
  10083 non-water atoms found / 226545 particles
[ Finished finiding non-water atoms ]
  10083 atoms specified to write to XTC files
[ Writing viewer topology ]
  10083 atoms written
[ Viewer topology written ]
  Using OpenCL on platformId 0 and gpu 0
  Creating a list of all Context parameters:

  v(^_^)v  MD ready starting from step 0

  Checking for existence of retries file (numRetries) ...
  Starting watchdog...
Completed 0 out of 500000 steps (0%)

  Running tests
  All tests passed.
  Writing binary checkpoint
  Binary checkpoint complete. Cleared numRetries file.
Completed 5000 out of 500000 steps (1%)

Completed 10000 out of 500000 steps (2%)

Completed 15000 out of 500000 steps (3%)

Completed 20000 out of 500000 steps (4%)

  Appending to XTC file positions.xtc
An exception occurred at step 20581: Particle coordinate is nan

  Updating retries file (numRetries) with 1 retries.
ERROR:98: Attempting to restart from last good checkpoint by restarting core.

Folding@home Core Shutdown: CORE_RESTART

*************************** Core22 Folding@home Core ***************************
       Core: Core22
       Type: 0x22
    Version: 0.0.11
     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
  Copyright: 2020 foldingathome.org
   Homepage: https://foldingathome.org/
       Date: Jun 26 2020
       Time: 19:49:16
   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
     Branch: core22-0.0.11
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
             <peastman@stanford.edu>
       Args: -dir 00 -suffix 01 -version 706 -lifeline 9524 -checkpoint 15
             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
************************************ libFAH ************************************
       Date: Jun 26 2020
       Time: 19:47:12
   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
     Branch: HEAD
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
************************************ CBang *************************************
       Date: Jun 26 2020
       Time: 19:46:11
   Revision: f8529962055b0e7bde23e429f5072ff758089dee
     Branch: master
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
************************************ System ************************************
        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
       CPUs: 12
     Memory: 31.92GiB
Free Memory: 24.36GiB
    Threads: WINDOWS_THREADS
 OS Version: 6.2
Has Battery: false
 On Battery: false
 UTC Offset: 2
        PID: 2656
        CWD: C:\FAHClient\work
********************************************************************************
Folding@home GPU Core22 Folding@home Core

Version 0.0.11

[1] compatible platform(s):
  -- 0 --
  PROFILE = FULL_PROFILE
  VERSION = OpenCL 2.1 AMD-APP (2906.10)
  NAME = AMD Accelerated Parallel Processing
  VENDOR = Advanced Micro Devices, Inc.

(1) device(s) found on platform 0:
  -- 0 --
  DEVICE_NAME = gfx906
  DEVICE_VENDOR = Advanced Micro Devices, Inc.
  DEVICE_VERSION = OpenCL 2.0 AMD-APP (2906.10)

[ Entering Init ]
  Launch time: 2020-08-06T08:34:20Z
  Arguments passed: -dir 00 -suffix 01 -version 706 -lifeline 9524 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0 
  For testState comparison of CPU and GPU, will use:
    forceTolerance: 5 kJ/mol/nm
    energyTolerance: 10 kJ/mol
[ Leaving  Init ]
[ Entering Main ]
  Reading core settings...
  Total number of steps: 500000
  Checkpoint write interval: 25000 steps (5%) [20 total]

  JSON viewer frame write interval: 5000 steps (1%) [100 total]

  XTC frame write interval: 20000 steps (4%) [25 total]

  Global context and integrator variables write interval: disabled

[ Initializing Core Contexts ]
  Using platform OpenCL
  Looking for vendor: amd...found on platformId 0
  Setting platform precision to mixed
  Setting DisablePmeStream to 1
    Checking for integrator.xml
    Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
    Checking for integrator.xml
    Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
    Checking for system.xml
    Found system.xml
  Deserializing System...successful.
  Found 226545 atoms, 4 forces.
  Finding State XML file...
    Checking for state.xml
    Found state.xml
  Deserializing State...successful.
    Ewald error tolerance in force 0 is 0.0005
    Ewald parameters: alpha 2.62826 nx 120 ny 120 nz 84
    Integrator Type: class OpenMM::LangevinIntegrator
    Constraint Tolerance: 1e-05
    Time Step in PS: 0.002
    Using CPU platform for reference calculations.
  Performing initial sanity checks before starting work...
  Comparing forces and energies between initial State and CPU...
  Comparing forces and energies between GPU and CPU...
[ Initialized Core Contexts... ]
[ Finding non-water atoms ]
  10083 non-water atoms found / 226545 particles
[ Finished finiding non-water atoms ]
  10083 atoms specified to write to XTC files
[ Writing viewer topology ]
  10083 atoms written
[ Viewer topology written ]
  Using OpenCL on platformId 0 and gpu 0
  Creating a list of all Context parameters:

  Binary checkpoint file found. Restoring context from checkpoint...
  Binary checkpoint read. Resuming from step 0
  v(^_^)v  MD ready starting from step 0

  Checking for existence of retries file (numRetries) ...
  Opening retries file (numRetries) ...
  There have been 1 retries already.
  Starting watchdog...
Completed 0 out of 500000 steps (0%)

Completed 5000 out of 500000 steps (1%)

Completed 10000 out of 500000 steps (2%)

Completed 15000 out of 500000 steps (3%)

Completed 20000 out of 500000 steps (4%)

  Appending to XTC file positions.xtc
Completed 25000 out of 500000 steps (5%)

  Running tests
  All tests passed.
  Writing binary checkpoint
  Binary checkpoint complete. Cleared numRetries file.
Completed 30000 out of 500000 steps (6%)

Completed 35000 out of 500000 steps (7%)

Completed 40000 out of 500000 steps (8%)

  Appending to XTC file positions.xtc
An exception occurred at step 44928: Particle coordinate is nan

  Updating retries file (numRetries) with 1 retries.
ERROR:98: Attempting to restart from last good checkpoint by restarting core.

Folding@home Core Shutdown: CORE_RESTART

*************************** Core22 Folding@home Core ***************************
       Core: Core22
       Type: 0x22
    Version: 0.0.11
     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
  Copyright: 2020 foldingathome.org
   Homepage: https://foldingathome.org/
       Date: Jun 26 2020
       Time: 19:49:16
   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
     Branch: core22-0.0.11
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
             <peastman@stanford.edu>
       Args: -dir 00 -suffix 01 -version 706 -lifeline 13540 -checkpoint 15
             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
************************************ libFAH ************************************
       Date: Jun 26 2020
       Time: 19:47:12
   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
     Branch: HEAD
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
************************************ CBang *************************************
       Date: Jun 26 2020
       Time: 19:46:11
   Revision: f8529962055b0e7bde23e429f5072ff758089dee
     Branch: master
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
************************************ System ************************************
        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
       CPUs: 12
     Memory: 31.92GiB
Free Memory: 24.17GiB
    Threads: WINDOWS_THREADS
 OS Version: 6.2
Has Battery: false
 On Battery: false
 UTC Offset: 2
        PID: 6836
        CWD: C:\FAHClient\work
********************************************************************************
Folding@home GPU Core22 Folding@home Core

Version 0.0.11

[1] compatible platform(s):
  -- 0 --
  PROFILE = FULL_PROFILE
  VERSION = OpenCL 2.1 AMD-APP (2906.10)
  NAME = AMD Accelerated Parallel Processing
  VENDOR = Advanced Micro Devices, Inc.

(1) device(s) found on platform 0:
  -- 0 --
  DEVICE_NAME = gfx906
  DEVICE_VENDOR = Advanced Micro Devices, Inc.
  DEVICE_VERSION = OpenCL 2.0 AMD-APP (2906.10)

[ Entering Init ]
  Launch time: 2020-08-06T08:40:29Z
  Arguments passed: -dir 00 -suffix 01 -version 706 -lifeline 13540 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0 
  For testState comparison of CPU and GPU, will use:
    forceTolerance: 5 kJ/mol/nm
    energyTolerance: 10 kJ/mol
[ Leaving  Init ]
[ Entering Main ]
  Reading core settings...
  Total number of steps: 500000
  Checkpoint write interval: 25000 steps (5%) [20 total]

  JSON viewer frame write interval: 5000 steps (1%) [100 total]

  XTC frame write interval: 20000 steps (4%) [25 total]

  Global context and integrator variables write interval: disabled

[ Initializing Core Contexts ]
  Using platform OpenCL
  Looking for vendor: amd...found on platformId 0
  Setting platform precision to mixed
  Setting DisablePmeStream to 1
    Checking for integrator.xml
    Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
    Checking for integrator.xml
    Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
    Checking for system.xml
    Found system.xml
  Deserializing System...successful.
  Found 226545 atoms, 4 forces.
  Finding State XML file...
    Checking for state.xml
    Found state.xml
  Deserializing State...successful.
    Ewald error tolerance in force 0 is 0.0005
    Ewald parameters: alpha 2.62826 nx 120 ny 120 nz 84
    Integrator Type: class OpenMM::LangevinIntegrator
    Constraint Tolerance: 1e-05
    Time Step in PS: 0.002
    Using CPU platform for reference calculations.
  Performing initial sanity checks before starting work...
  Comparing forces and energies between initial State and CPU...
  Comparing forces and energies between GPU and CPU...
[ Initialized Core Contexts... ]
[ Finding non-water atoms ]
  10083 non-water atoms found / 226545 particles
[ Finished finiding non-water atoms ]
  10083 atoms specified to write to XTC files
[ Writing viewer topology ]
  10083 atoms written
[ Viewer topology written ]
  Using OpenCL on platformId 0 and gpu 0
  Creating a list of all Context parameters:

  Binary checkpoint file found. Restoring context from checkpoint...
  Binary checkpoint read. Resuming from step 25000
  v(^_^)v  MD ready starting from step 25000

  Checking for existence of retries file (numRetries) ...
  Opening retries file (numRetries) ...
  There have been 1 retries already.
  Starting watchdog...
Completed 25000 out of 500000 steps (5%)

Completed 30000 out of 500000 steps (6%)

Completed 35000 out of 500000 steps (7%)

Completed 40000 out of 500000 steps (8%)

  Appending to XTC file positions.xtc
Completed 45000 out of 500000 steps (9%)

Completed 50000 out of 500000 steps (10%)

  Running tests
  All tests passed.
  Writing binary checkpoint
  Binary checkpoint complete. Cleared numRetries file.
Completed 55000 out of 500000 steps (11%)

Completed 60000 out of 500000 steps (12%)

  Appending to XTC file positions.xtc
An exception occurred at step 63000: Particle coordinate is nan

  Updating retries file (numRetries) with 1 retries.
ERROR:98: Attempting to restart from last good checkpoint by restarting core.

Folding@home Core Shutdown: CORE_RESTART

*************************** Core22 Folding@home Core ***************************
       Core: Core22
       Type: 0x22
    Version: 0.0.11
     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
  Copyright: 2020 foldingathome.org
   Homepage: https://foldingathome.org/
       Date: Jun 26 2020
       Time: 19:49:16
   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
     Branch: core22-0.0.11
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
             <peastman@stanford.edu>
       Args: -dir 00 -suffix 01 -version 706 -lifeline 14748 -checkpoint 15
             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
************************************ libFAH ************************************
       Date: Jun 26 2020
       Time: 19:47:12
   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
     Branch: HEAD
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
************************************ CBang *************************************
       Date: Jun 26 2020
       Time: 19:46:11
   Revision: f8529962055b0e7bde23e429f5072ff758089dee
     Branch: master
   Compiler: Visual C++ 2015
    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
   Platform: win32 10
       Bits: 64
       Mode: Release
************************************ System ************************************
        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
       CPUs: 12
     Memory: 31.92GiB
Free Memory: 24.21GiB
    Threads: WINDOWS_THREADS
 OS Version: 6.2
Has Battery: false
 On Battery: false
 UTC Offset: 2
        PID: 12524
        CWD: C:\FAHClient\work
********************************************************************************
Folding@home GPU Core22 Folding@home Core

Version 0.0.11

[1] compatible platform(s):
  -- 0 --
  PROFILE = FULL_PROFILE
  VERSION = OpenCL 2.1 AMD-APP (2906.10)
  NAME = AMD Accelerated Parallel Processing
  VENDOR = Advanced Micro Devices, Inc.

(1) device(s) found on platform 0:
  -- 0 --
  DEVICE_NAME = gfx906
  DEVICE_VENDOR = Advanced Micro Devices, Inc.
  DEVICE_VERSION = OpenCL 2.0 AMD-APP (2906.10)

[ Entering Init ]
  Launch time: 2020-08-06T08:45:46Z
  Arguments passed: -dir 00 -suffix 01 -version 706 -lifeline 14748 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0 
  For testState comparison of CPU and GPU, will use:
    forceTolerance: 5 kJ/mol/nm
    energyTolerance: 10 kJ/mol
[ Leaving  Init ]
[ Entering Main ]
  Reading core settings...
  Total number of steps: 500000
  Checkpoint write interval: 25000 steps (5%) [20 total]

  JSON viewer frame write interval: 5000 steps (1%) [100 total]

  XTC frame write interval: 20000 steps (4%) [25 total]

  Global context and integrator variables write interval: disabled

[ Initializing Core Contexts ]
  Using platform OpenCL
  Looking for vendor: amd...found on platformId 0
  Setting platform precision to mixed
  Setting DisablePmeStream to 1
    Checking for integrator.xml
    Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
    Checking for integrator.xml
    Found integrator.xml
Loading integrator from integrator.xml
Stream copied, deserializing...
    Checking for system.xml
    Found system.xml
  Deserializing System...successful.
  Found 226545 atoms, 4 forces.
  Finding State XML file...
    Checking for state.xml
    Found state.xml
  Deserializing State...successful.
    Ewald error tolerance in force 0 is 0.0005
    Ewald parameters: alpha 2.62826 nx 120 ny 120 nz 84
    Integrator Type: class OpenMM::LangevinIntegrator
    Constraint Tolerance: 1e-05
    Time Step in PS: 0.002
    Using CPU platform for reference calculations.
  Performing initial sanity checks before starting work...
  Comparing forces and energies between initial State and CPU...
  Comparing forces and energies between GPU and CPU...
[ Initialized Core Contexts... ]
[ Finding non-water atoms ]
  10083 non-water atoms found / 226545 particles
[ Finished finiding non-water atoms ]
  10083 atoms specified to write to XTC files
[ Writing viewer topology ]
  10083 atoms written
[ Viewer topology written ]
  Using OpenCL on platformId 0 and gpu 0
  Creating a list of all Context parameters:

  Binary checkpoint file found. Restoring context from checkpoint...
  Binary checkpoint read. Resuming from step 50000
  v(^_^)v  MD ready starting from step 50000

  Checking for existence of retries file (numRetries) ...
  Opening retries file (numRetries) ...
  There have been 1 retries already.
  Starting watchdog...
Completed 50000 out of 500000 steps (10%)

Completed 55000 out of 500000 steps (11%)

Completed 60000 out of 500000 steps (12%)

  Appending to XTC file positions.xtc

[NormalDiffusion]

Or here, another 16600 failing live:

Code: Select all

14:26:14:WU01:FS01:Starting
14:26:14:WU01:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
14:26:14:WU01:FS01:Started FahCore on PID 14016
14:26:14:WU01:FS01:Core PID:4628
14:26:14:WU01:FS01:FahCore 0x22 started
14:26:14:WU01:FS01:0x22:*********************** Log Started 2020-08-06T14:26:14Z ***********************
14:26:14:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
14:26:14:WU01:FS01:0x22:       Core: Core22
14:26:14:WU01:FS01:0x22:       Type: 0x22
14:26:14:WU01:FS01:0x22:    Version: 0.0.11
14:26:14:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
14:26:14:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
14:26:14:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
14:26:14:WU01:FS01:0x22:       Date: Jun 26 2020
14:26:14:WU01:FS01:0x22:       Time: 19:49:16
14:26:14:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
14:26:14:WU01:FS01:0x22:     Branch: core22-0.0.11
14:26:14:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:26:14:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:26:14:WU01:FS01:0x22:   Platform: win32 10
14:26:14:WU01:FS01:0x22:       Bits: 64
14:26:14:WU01:FS01:0x22:       Mode: Release
14:26:14:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
14:26:14:WU01:FS01:0x22:             <peastman@stanford.edu>
14:26:14:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 14016 -checkpoint 15
14:26:14:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
14:26:14:WU01:FS01:0x22:************************************ libFAH ************************************
14:26:14:WU01:FS01:0x22:       Date: Jun 26 2020
14:26:14:WU01:FS01:0x22:       Time: 19:47:12
14:26:14:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
14:26:14:WU01:FS01:0x22:     Branch: HEAD
14:26:14:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:26:14:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:26:14:WU01:FS01:0x22:   Platform: win32 10
14:26:14:WU01:FS01:0x22:       Bits: 64
14:26:14:WU01:FS01:0x22:       Mode: Release
14:26:14:WU01:FS01:0x22:************************************ CBang *************************************
14:26:14:WU01:FS01:0x22:       Date: Jun 26 2020
14:26:14:WU01:FS01:0x22:       Time: 19:46:11
14:26:14:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
14:26:14:WU01:FS01:0x22:     Branch: master
14:26:14:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:26:14:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:26:14:WU01:FS01:0x22:   Platform: win32 10
14:26:14:WU01:FS01:0x22:       Bits: 64
14:26:14:WU01:FS01:0x22:       Mode: Release
14:26:14:WU01:FS01:0x22:************************************ System ************************************
14:26:14:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
14:26:14:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
14:26:14:WU01:FS01:0x22:       CPUs: 12
14:26:14:WU01:FS01:0x22:     Memory: 31.92GiB
14:26:14:WU01:FS01:0x22:Free Memory: 22.74GiB
14:26:14:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
14:26:14:WU01:FS01:0x22: OS Version: 6.2
14:26:14:WU01:FS01:0x22:Has Battery: false
14:26:14:WU01:FS01:0x22: On Battery: false
14:26:14:WU01:FS01:0x22: UTC Offset: 2
14:26:14:WU01:FS01:0x22:        PID: 4628
14:26:14:WU01:FS01:0x22:        CWD: C:\FAHClient\work
14:26:14:WU01:FS01:0x22:********************************************************************************
14:26:14:WU01:FS01:0x22:Project: 16600 (Run 0, Clone 231, Gen 221)
14:26:14:WU01:FS01:0x22:Unit: 0x000001008f59f36f5ec369129c7fb88e
14:26:14:WU01:FS01:0x22:Reading tar file core.xml
14:26:14:WU01:FS01:0x22:Reading tar file integrator.xml
14:26:14:WU01:FS01:0x22:Reading tar file state.xml
14:26:15:WU01:FS01:0x22:Reading tar file system.xml
14:26:17:WU01:FS01:0x22:Digital signatures verified
14:26:17:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
14:26:17:WU01:FS01:0x22:Version 0.0.11
14:26:17:WU01:FS01:0x22:  Checkpoint write interval: 25000 steps (5%) [20 total]
14:26:17:WU01:FS01:0x22:  JSON viewer frame write interval: 5000 steps (1%) [100 total]
14:26:17:WU01:FS01:0x22:  XTC frame write interval: 20000 steps (4%) [25 total]
14:26:17:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
14:26:37:WU01:FS01:0x22:Completed 0 out of 500000 steps (0%)
14:27:15:WU01:FS01:0x22:Completed 5000 out of 500000 steps (1%)
14:27:52:WU01:FS01:0x22:Completed 10000 out of 500000 steps (2%)
14:28:29:WU01:FS01:0x22:Completed 15000 out of 500000 steps (3%)
14:29:05:WU01:FS01:0x22:Completed 20000 out of 500000 steps (4%)
14:29:42:WU01:FS01:0x22:Completed 25000 out of 500000 steps (5%)
14:30:21:WU01:FS01:0x22:Completed 30000 out of 500000 steps (6%)
14:30:58:WU01:FS01:0x22:Completed 35000 out of 500000 steps (7%)
14:31:35:WU01:FS01:0x22:Completed 40000 out of 500000 steps (8%)
14:32:12:WU01:FS01:0x22:Completed 45000 out of 500000 steps (9%)
14:32:49:WU01:FS01:0x22:Completed 50000 out of 500000 steps (10%)
14:33:28:WU01:FS01:0x22:Completed 55000 out of 500000 steps (11%)
14:34:26:WU01:FS01:0x22:An exception occurred at step 59988: Particle coordinate is nan
14:34:26:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
14:34:26:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
14:34:26:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
14:34:26:WU01:FS01:Starting
14:34:26:WU01:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
14:34:26:WU01:FS01:Started FahCore on PID 7980
14:34:26:WU01:FS01:Core PID:6956
14:34:26:WU01:FS01:FahCore 0x22 started
14:34:27:WU01:FS01:0x22:*********************** Log Started 2020-08-06T14:34:26Z ***********************
14:34:27:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
14:34:27:WU01:FS01:0x22:       Core: Core22
14:34:27:WU01:FS01:0x22:       Type: 0x22
14:34:27:WU01:FS01:0x22:    Version: 0.0.11
14:34:27:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
14:34:27:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
14:34:27:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
14:34:27:WU01:FS01:0x22:       Date: Jun 26 2020
14:34:27:WU01:FS01:0x22:       Time: 19:49:16
14:34:27:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
14:34:27:WU01:FS01:0x22:     Branch: core22-0.0.11
14:34:27:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:34:27:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:34:27:WU01:FS01:0x22:   Platform: win32 10
14:34:27:WU01:FS01:0x22:       Bits: 64
14:34:27:WU01:FS01:0x22:       Mode: Release
14:34:27:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
14:34:27:WU01:FS01:0x22:             <peastman@stanford.edu>
14:34:27:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 7980 -checkpoint 15
14:34:27:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
14:34:27:WU01:FS01:0x22:************************************ libFAH ************************************
14:34:27:WU01:FS01:0x22:       Date: Jun 26 2020
14:34:27:WU01:FS01:0x22:       Time: 19:47:12
14:34:27:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
14:34:27:WU01:FS01:0x22:     Branch: HEAD
14:34:27:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:34:27:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:34:27:WU01:FS01:0x22:   Platform: win32 10
14:34:27:WU01:FS01:0x22:       Bits: 64
14:34:27:WU01:FS01:0x22:       Mode: Release
14:34:27:WU01:FS01:0x22:************************************ CBang *************************************
14:34:27:WU01:FS01:0x22:       Date: Jun 26 2020
14:34:27:WU01:FS01:0x22:       Time: 19:46:11
14:34:27:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
14:34:27:WU01:FS01:0x22:     Branch: master
14:34:27:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:34:27:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:34:27:WU01:FS01:0x22:   Platform: win32 10
14:34:27:WU01:FS01:0x22:       Bits: 64
14:34:27:WU01:FS01:0x22:       Mode: Release
14:34:27:WU01:FS01:0x22:************************************ System ************************************
14:34:27:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
14:34:27:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
14:34:27:WU01:FS01:0x22:       CPUs: 12
14:34:27:WU01:FS01:0x22:     Memory: 31.92GiB
14:34:27:WU01:FS01:0x22:Free Memory: 22.78GiB
14:34:27:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
14:34:27:WU01:FS01:0x22: OS Version: 6.2
14:34:27:WU01:FS01:0x22:Has Battery: false
14:34:27:WU01:FS01:0x22: On Battery: false
14:34:27:WU01:FS01:0x22: UTC Offset: 2
14:34:27:WU01:FS01:0x22:        PID: 6956
14:34:27:WU01:FS01:0x22:        CWD: C:\FAHClient\work
14:34:27:WU01:FS01:0x22:********************************************************************************
14:34:27:WU01:FS01:0x22:Project: 16600 (Run 0, Clone 231, Gen 221)
14:34:27:WU01:FS01:0x22:Unit: 0x000001008f59f36f5ec369129c7fb88e
14:34:27:WU01:FS01:0x22:Digital signatures verified
14:34:27:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
14:34:27:WU01:FS01:0x22:Version 0.0.11
14:34:27:WU01:FS01:0x22:  Checkpoint write interval: 25000 steps (5%) [20 total]
14:34:27:WU01:FS01:0x22:  JSON viewer frame write interval: 5000 steps (1%) [100 total]
14:34:27:WU01:FS01:0x22:  XTC frame write interval: 20000 steps (4%) [25 total]
14:34:27:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
14:34:47:WU01:FS01:0x22:Completed 50000 out of 500000 steps (10%)
14:35:25:WU01:FS01:0x22:Completed 55000 out of 500000 steps (11%)
14:36:02:WU01:FS01:0x22:Completed 60000 out of 500000 steps (12%)
14:36:39:WU01:FS01:0x22:Completed 65000 out of 500000 steps (13%)
14:37:16:WU01:FS01:0x22:Completed 70000 out of 500000 steps (14%)
14:37:52:WU01:FS01:0x22:Completed 75000 out of 500000 steps (15%)
14:38:31:WU01:FS01:0x22:Completed 80000 out of 500000 steps (16%)
14:39:08:WU01:FS01:0x22:Completed 85000 out of 500000 steps (17%)
14:39:45:WU01:FS01:0x22:Completed 90000 out of 500000 steps (18%)
14:40:22:WU01:FS01:0x22:Completed 95000 out of 500000 steps (19%)
14:40:59:WU01:FS01:0x22:Completed 100000 out of 500000 steps (20%)
14:41:37:WU01:FS01:0x22:Completed 105000 out of 500000 steps (21%)
14:42:14:WU01:FS01:0x22:Completed 110000 out of 500000 steps (22%)
14:42:51:WU01:FS01:0x22:Completed 115000 out of 500000 steps (23%)
14:43:28:WU01:FS01:0x22:Completed 120000 out of 500000 steps (24%)
14:44:05:WU01:FS01:0x22:Completed 125000 out of 500000 steps (25%)
14:44:50:WU01:FS01:0x22:An exception occurred at step 125750: Particle coordinate is nan
14:44:50:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
14:44:50:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
14:44:50:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
14:44:51:WU01:FS01:Starting
14:44:51:WU01:FS01:Running FahCore: C:\FAHClient/FAHCoreWrapper.exe C:\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 12204 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
14:44:51:WU01:FS01:Started FahCore on PID 13916
14:44:51:WU01:FS01:Core PID:10400
14:44:51:WU01:FS01:FahCore 0x22 started
14:44:51:WU01:FS01:0x22:*********************** Log Started 2020-08-06T14:44:51Z ***********************
14:44:51:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
14:44:51:WU01:FS01:0x22:       Core: Core22
14:44:51:WU01:FS01:0x22:       Type: 0x22
14:44:51:WU01:FS01:0x22:    Version: 0.0.11
14:44:51:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
14:44:51:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
14:44:51:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
14:44:51:WU01:FS01:0x22:       Date: Jun 26 2020
14:44:51:WU01:FS01:0x22:       Time: 19:49:16
14:44:51:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
14:44:51:WU01:FS01:0x22:     Branch: core22-0.0.11
14:44:51:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:44:51:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:44:51:WU01:FS01:0x22:   Platform: win32 10
14:44:51:WU01:FS01:0x22:       Bits: 64
14:44:51:WU01:FS01:0x22:       Mode: Release
14:44:51:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
14:44:51:WU01:FS01:0x22:             <peastman@stanford.edu>
14:44:51:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 13916 -checkpoint 15
14:44:51:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
14:44:51:WU01:FS01:0x22:************************************ libFAH ************************************
14:44:51:WU01:FS01:0x22:       Date: Jun 26 2020
14:44:51:WU01:FS01:0x22:       Time: 19:47:12
14:44:51:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
14:44:51:WU01:FS01:0x22:     Branch: HEAD
14:44:51:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:44:51:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:44:51:WU01:FS01:0x22:   Platform: win32 10
14:44:51:WU01:FS01:0x22:       Bits: 64
14:44:51:WU01:FS01:0x22:       Mode: Release
14:44:51:WU01:FS01:0x22:************************************ CBang *************************************
14:44:51:WU01:FS01:0x22:       Date: Jun 26 2020
14:44:51:WU01:FS01:0x22:       Time: 19:46:11
14:44:51:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
14:44:51:WU01:FS01:0x22:     Branch: master
14:44:51:WU01:FS01:0x22:   Compiler: Visual C++ 2015
14:44:51:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
14:44:51:WU01:FS01:0x22:   Platform: win32 10
14:44:51:WU01:FS01:0x22:       Bits: 64
14:44:51:WU01:FS01:0x22:       Mode: Release
14:44:51:WU01:FS01:0x22:************************************ System ************************************
14:44:51:WU01:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz
14:44:51:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
14:44:51:WU01:FS01:0x22:       CPUs: 12
14:44:51:WU01:FS01:0x22:     Memory: 31.92GiB
14:44:51:WU01:FS01:0x22:Free Memory: 22.76GiB
14:44:51:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
14:44:51:WU01:FS01:0x22: OS Version: 6.2
14:44:51:WU01:FS01:0x22:Has Battery: false
14:44:51:WU01:FS01:0x22: On Battery: false
14:44:51:WU01:FS01:0x22: UTC Offset: 2
14:44:51:WU01:FS01:0x22:        PID: 10400
14:44:51:WU01:FS01:0x22:        CWD: C:\FAHClient\work
14:44:51:WU01:FS01:0x22:********************************************************************************
14:44:51:WU01:FS01:0x22:Project: 16600 (Run 0, Clone 231, Gen 221)
14:44:51:WU01:FS01:0x22:Unit: 0x000001008f59f36f5ec369129c7fb88e
14:44:51:WU01:FS01:0x22:Digital signatures verified
14:44:51:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
14:44:51:WU01:FS01:0x22:Version 0.0.11
14:44:51:WU01:FS01:0x22:  Checkpoint write interval: 25000 steps (5%) [20 total]
14:44:51:WU01:FS01:0x22:  JSON viewer frame write interval: 5000 steps (1%) [100 total]
14:44:51:WU01:FS01:0x22:  XTC frame write interval: 20000 steps (4%) [25 total]
14:44:51:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
14:45:11:WU01:FS01:0x22:Completed 125000 out of 500000 steps (25%)
14:45:49:WU01:FS01:0x22:Completed 130000 out of 500000 steps (26%)
14:46:26:WU01:FS01:0x22:Completed 135000 out of 500000 steps (27%)
14:47:03:WU01:FS01:0x22:Completed 140000 out of 500000 steps (28%)
14:47:40:WU01:FS01:0x22:Completed 145000 out of 500000 steps (29%)
14:48:17:WU01:FS01:0x22:Completed 150000 out of 500000 steps (30%)
14:48:56:WU01:FS01:0x22:Completed 155000 out of 500000 steps (31%)
14:49:33:WU01:FS01:0x22:Completed 160000 out of 500000 steps (32%)
14:50:10:WU01:FS01:0x22:Completed 165000 out of 500000 steps (33%)
14:50:47:WU01:FS01:0x22:Completed 170000 out of 500000 steps (34%)
14:51:24:WU01:FS01:0x22:Completed 175000 out of 500000 steps (35%)
14:52:03:WU01:FS01:0x22:Completed 180000 out of 500000 steps (36%)
14:52:40:WU01:FS01:0x22:Completed 185000 out of 500000 steps (37%)
14:53:17:WU01:FS01:0x22:Completed 190000 out of 500000 steps (38%)
14:53:54:WU01:FS01:0x22:Completed 195000 out of 500000 steps (39%)
14:54:31:WU01:FS01:0x22:Completed 200000 out of 500000 steps (40%)
14:55:10:WU01:FS01:0x22:Completed 205000 out of 500000 steps (41%)
14:55:46:WU01:FS01:0x22:Completed 210000 out of 500000 steps (42%)
14:56:23:WU01:FS01:0x22:Completed 215000 out of 500000 steps (43%)
14:57:01:WU01:FS01:0x22:Completed 220000 out of 500000 steps (44%)
14:57:38:WU01:FS01:0x22:Completed 225000 out of 500000 steps (45%)
14:58:16:WU01:FS01:0x22:Completed 230000 out of 500000 steps (46%)
14:58:53:WU01:FS01:0x22:Completed 235000 out of 500000 steps (47%)
14:59:29:WU01:FS01:0x22:Completed 240000 out of 500000 steps (48%)
15:00:06:WU01:FS01:0x22:Completed 245000 out of 500000 steps (49%)
15:00:43:WU01:FS01:0x22:Completed 250000 out of 500000 steps (50%)
15:01:21:WU01:FS01:0x22:Completed 255000 out of 500000 steps (51%)
15:01:58:WU01:FS01:0x22:Completed 260000 out of 500000 steps (52%)
15:02:34:WU01:FS01:0x22:Completed 265000 out of 500000 steps (53%)
15:03:11:WU01:FS01:0x22:Completed 270000 out of 500000 steps (54%)
15:03:48:WU01:FS01:0x22:Completed 275000 out of 500000 steps (55%)
15:04:27:WU01:FS01:0x22:Completed 280000 out of 500000 steps (56%)
15:05:04:WU01:FS01:0x22:Completed 285000 out of 500000 steps (57%)
15:05:41:WU01:FS01:0x22:Completed 290000 out of 500000 steps (58%)
15:06:18:WU01:FS01:0x22:Completed 295000 out of 500000 steps (59%)

[bruce]

Is this problem associated with a Linux vs. Wnidows difference?
Is this problem somehow associated with the particular driver version that is on those machines?

[NormalDiffusion]

It's about 16600 WUs crashing on 2 different machines. Both machines are running windows 10. They use different drivers, but both of them have huge problems with the 16600s.
I can post a log of the 2nd pc tomorrow if needed.

[muziqaz]

Bruce this one is all windows. Once I'm home I'll have a look how it ran on my end.

@NormalDiffusion, I see you have GPU driver from last year. Any reason for that? If there is no particular reason, would you be able to download latest stable driver please. Also it would be great if you disabled Windows Update for device drivers. Last thing we need is clueless Microsoft staff autodownloading and updating GPU drivers for us

[NormalDiffusion]

Yes, the reason is that the old drivers are a little bit faster and the front end. On the new drivers there is no history of the temperature and other parameters over time.
I could update on one machine to check (the one with rvii only, which is my everyday workstation), but would prefer not to do so on the one with the 2 cards (rvii and 290x)...

[muziqaz]

I would not recommend updating the drivers on dual card system. But single Radeon 7 breathes better with new drivers, and there are a lot of issues fixed as well
By any chance are you using adrenaline drivers or the pro version? The reason I recommend updating is due to OpenCL driver version. It might not even solve anything

[NormalDiffusion]

Using adrenaline version. The pro one just contain the last stable gaming driver and has been released just to ease the task if sys admin... Or did this change in the last months?

The 290x should work fine with the new drivers, but somehow I don't like them. Cards are slower (tested with fahbench and with the latest core (the unofficial version)) and the front end... No history, a look made for kids....

[muziqaz]

When I updated from 19.12.1 to 20.4.2 performance remained the same, but quite a few major issues have been fixed.

Ok, I asked about p16600 failure rate, will see what it is. I have a theory about the failures, but it needs some testing.

[NormalDiffusion]

When I updated from 19.12.1 to 20.4.2 performance remained the same, but quite a few major issues have been fixed.

On the machine with the 2 cards I have an even older version. I need to check tomorrow. But there is a performance difference, even if it is so slight...

Which issues have been fixed? The intermittent black screen with 2 monitors was still present when I checked :/

[muziqaz]

working with GPU accelerated content while folding does not crash he whole system anymore

And also, Vega64 plus Radeon VII are working fine on windows with 20.4 drivers. ah you said two monitors, I see

[NormalDiffusion]

working with GPU accelerated content while folding does not crash he whole system anymore

And also, Vega64 plus Radeon VII are working fine on windows with 20.4 drivers. ah you said two monitors, I see

That's interesting! May have to give it a try!
And I'm looking for a vega fe wc to replace the 290x. But the prices in the last months... oO

[muziqaz]

I have Vega64 liquid edition. Replaced stock fan with two noctua fans for push pull, and it is running out of its mind. temps are super low, and clock are insane