Failed GPU slot daily :(

[crimson1077]

Thank you, MeeLee! A good reminder for us all. I shall do that one day.
I've been riding with temps not exceeding 75c with folding nor gaming (with ultra high graphic settings) fortunately.

My main concern consist within the abnormally high rate of failures with the preparation of Sprint 3 compounds for the Covid Moonshot collaboration. Rolled back to sprint 2 compounds.

[bruce]

The next sprint should start tomorror or the next day. A bug has been identified and although it won't be fixed tomorrow, there's a work-around and the WUs are being rebuilt using it.

[crimson1077]

The next sprint should start tomorror or the next day. A bug has been identified and although it won't be fixed tomorrow, there's a work-around and the WUs are being rebuilt using it.

Great to hear bruce!

[crimson1077]

Code: Select all

*********************** Log Started 2020-08-30T02:30:46Z ***********************
02:30:46:Trying to access database...
02:30:46:Successfully acquired database lock
02:30:46:Downloading GPUs.txt from assign1.foldingathome.org:80
02:30:46:Connecting to assign1.foldingathome.org:80
02:30:46:Read GPUs.txt
02:30:46:Enabled folding slot 00: PAUSED cpu:2 (not configured)
02:30:47:Enabled folding slot 01: PAUSED gpu:0:Hawaii [Radeon R9 200/300 Series] (not configured)
02:30:47:****************************** FAHClient ******************************
02:30:47:        Version: 7.6.13
02:30:47:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
02:30:47:      Copyright: 2020 foldingathome.org
02:30:47:       Homepage: https://foldingathome.org/
02:30:47:           Date: Apr 27 2020
02:30:47:           Time: 21:21:01
02:30:47:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
02:30:47:         Branch: master
02:30:47:       Compiler: Visual C++ 2008
02:30:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
02:30:47:       Platform: win32 10
02:30:47:           Bits: 32
02:30:47:           Mode: Release
02:30:47:           Args: --open-web-control
02:30:47:******************************** CBang ********************************
02:30:47:           Date: Apr 24 2020
02:30:47:           Time: 17:07:55
02:30:47:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
02:30:47:         Branch: master
02:30:47:       Compiler: Visual C++ 2008
02:30:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
02:30:47:       Platform: win32 10
02:30:47:           Bits: 32
02:30:47:           Mode: Release
02:30:47:******************************* System ********************************
02:30:47:            CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
02:30:47:         CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
02:30:47:           CPUs: 4
02:30:47:         Memory: 14.97GiB
02:30:47:    Free Memory: 13.12GiB
02:30:47:        Threads: WINDOWS_THREADS
02:30:47:     OS Version: 6.2
02:30:47:    Has Battery: false
02:30:47:     On Battery: false
02:30:47:     UTC Offset: -5
02:30:47:            PID: 6532
02:30:47:            CWD: C:\Users\Crimson\AppData\Roaming\FAHClient
02:30:47:  Win32 Service: false
02:30:47:             OS: Windows 10 Enterprise LTSC 2019
02:30:47:        OS Arch: AMD64
02:30:47:           GPUs: 1
02:30:47:          GPU 0: Bus:1 Slot:0 Func:0 AMD:5 Hawaii [Radeon R9 200/300 Series]
02:30:47:           CUDA: Not detected: Failed to open dynamic library 'nvcuda.dll': The
02:30:47:                 specified module could not be found.
02:30:47:
02:30:47:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:3110.7
02:30:47:******************************* libFAH ********************************
02:30:47:           Date: Apr 15 2020
02:30:47:           Time: 14:53:14
02:30:47:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
02:30:47:         Branch: master
02:30:47:       Compiler: Visual C++ 2008
02:30:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
02:30:47:       Platform: win32 10
02:30:47:           Bits: 32
02:30:47:           Mode: Release
02:30:47:***********************************************************************
02:30:47:<config>
02:30:47:  <!-- Folding Slots -->
02:30:47:  <slot id='0' type='CPU'/>
02:30:47:  <slot id='1' type='GPU'/>
02:30:47:</config>
02:30:53:Set client configured
02:30:53:WU00:FS00:Connecting to assign1.foldingathome.org:80
02:30:53:WU00:FS00:Connecting to assign1.foldingathome.org:80
02:30:53:WU01:FS01:Connecting to assign1.foldingathome.org:80
02:30:53:WU00:FS00:Assigned to work server 3.21.157.11
02:30:53:WU00:FS00:Requesting new work unit for slot 00: READY cpu:3 from 3.21.157.11
02:30:53:WU00:FS00:Connecting to 3.21.157.11:8080
02:30:53:WU01:FS01:Assigned to work server 18.188.125.154
02:30:53:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:Hawaii [Radeon R9 200/300 Series] from 18.188.125.154
02:30:53:WU01:FS01:Connecting to 18.188.125.154:8080
02:30:54:WU00:FS00:Downloading 1.40MiB
02:30:54:17:127.0.0.1:New Web session
02:30:55:WU00:FS00:Download complete
02:30:55:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14726 run:2151 clone:1 gen:25 core:0xa7 unit:0x0000001a03159d0b5ebcaae3901909ce
02:30:55:WU00:FS00:Downloading core from http://cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah
02:30:55:WU00:FS00:Connecting to cores.foldingathome.org:80
02:30:55:WU00:FS00:FahCore a7: Downloading 6.71MiB
02:30:55:WU01:FS01:Downloading 6.19MiB
02:30:56:WU01:FS01:Download complete
02:30:56:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13424 run:787 clone:47 gen:2 core:0x22 unit:0x0000000412bc7d9a5f498aed479d8f25
02:30:56:WU01:FS01:Downloading core from http://cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah
02:30:56:WU01:FS01:Connecting to cores.foldingathome.org:80
02:30:57:WU01:FS01:FahCore 22: Downloading 4.39MiB
02:30:57:WU01:FS01:FahCore 22: Download complete
02:30:57:WU01:FS01:Valid core signature
02:30:57:WU01:FS01:Unpacked 14.46MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe
02:30:57:WU01:FS01:Starting
02:30:57:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6532 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:30:57:WU01:FS01:Started FahCore on PID 3712
02:30:57:WU01:FS01:Core PID:4084
02:30:57:WU01:FS01:FahCore 0x22 started
02:30:58:WU01:FS01:0x22:*********************** Log Started 2020-08-30T02:30:57Z ***********************
02:30:58:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
02:30:58:WU01:FS01:0x22:       Core: Core22
02:30:58:WU01:FS01:0x22:       Type: 0x22
02:30:58:WU01:FS01:0x22:    Version: 0.0.11
02:30:58:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
02:30:58:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
02:30:58:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
02:30:58:WU01:FS01:0x22:       Date: Jun 26 2020
02:30:58:WU01:FS01:0x22:       Time: 19:49:16
02:30:58:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
02:30:58:WU01:FS01:0x22:     Branch: core22-0.0.11
02:30:58:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:30:58:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:30:58:WU01:FS01:0x22:   Platform: win32 10
02:30:58:WU01:FS01:0x22:       Bits: 64
02:30:58:WU01:FS01:0x22:       Mode: Release
02:30:58:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
02:30:58:WU01:FS01:0x22:             <peastman@stanford.edu>
02:30:58:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 3712 -checkpoint 15
02:30:58:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:30:58:WU01:FS01:0x22:************************************ libFAH ************************************
02:30:58:WU01:FS01:0x22:       Date: Jun 26 2020
02:30:58:WU01:FS01:0x22:       Time: 19:47:12
02:30:58:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
02:30:58:WU01:FS01:0x22:     Branch: HEAD
02:30:58:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:30:58:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:30:58:WU01:FS01:0x22:   Platform: win32 10
02:30:58:WU01:FS01:0x22:       Bits: 64
02:30:58:WU01:FS01:0x22:       Mode: Release
02:30:58:WU01:FS01:0x22:************************************ CBang *************************************
02:30:58:WU01:FS01:0x22:       Date: Jun 26 2020
02:30:58:WU01:FS01:0x22:       Time: 19:46:11
02:30:58:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
02:30:58:WU01:FS01:0x22:     Branch: master
02:30:58:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:30:58:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:30:58:WU01:FS01:0x22:   Platform: win32 10
02:30:58:WU01:FS01:0x22:       Bits: 64
02:30:58:WU01:FS01:0x22:       Mode: Release
02:30:58:WU01:FS01:0x22:************************************ System ************************************
02:30:58:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
02:30:58:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
02:30:58:WU01:FS01:0x22:       CPUs: 4
02:30:58:WU01:FS01:0x22:     Memory: 14.97GiB
02:30:58:WU01:FS01:0x22:Free Memory: 12.61GiB
02:30:58:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
02:30:58:WU01:FS01:0x22: OS Version: 6.2
02:30:58:WU01:FS01:0x22:Has Battery: false
02:30:58:WU01:FS01:0x22: On Battery: false
02:30:58:WU01:FS01:0x22: UTC Offset: -5
02:30:58:WU01:FS01:0x22:        PID: 4084
02:30:58:WU01:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
02:30:58:WU01:FS01:0x22:********************************************************************************
02:30:58:WU01:FS01:0x22:Project: 13424 (Run 787, Clone 47, Gen 2)
02:30:58:WU01:FS01:0x22:Unit: 0x0000000412bc7d9a5f498aed479d8f25
02:30:58:WU01:FS01:0x22:Reading tar file core.xml
02:30:58:WU01:FS01:0x22:Reading tar file integrator.xml
02:30:58:WU01:FS01:0x22:Reading tar file state.xml.bz2
02:30:58:WU01:FS01:0x22:Reading tar file system.xml.bz2
02:30:58:WU01:FS01:0x22:Digital signatures verified
02:30:58:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
02:30:58:WU01:FS01:0x22:Version 0.0.11
02:30:58:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
02:30:58:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
02:30:58:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
02:30:58:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
02:31:01:WU00:FS00:FahCore a7: 87.51%
02:31:01:WU00:FS00:FahCore a7: Download complete
02:31:01:WU00:FS00:Valid core signature
02:31:01:WU00:FS00:Unpacked 19.85MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe
02:31:01:WU00:FS00:Unpacked 2.64MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/libfftw3f-3.dll
02:31:01:WU00:FS00:Starting
02:31:01:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 6532 -checkpoint 15 -np 3
02:31:01:WU00:FS00:Started FahCore on PID 4204
02:31:01:WU00:FS00:Core PID:6256
02:31:01:WU00:FS00:FahCore 0xa7 started
02:31:02:WU00:FS00:0xa7:*********************** Log Started 2020-08-30T02:31:01Z ***********************
02:31:02:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
02:31:02:WU00:FS00:0xa7:       Type: 0xa7
02:31:02:WU00:FS00:0xa7:       Core: Gromacs
02:31:02:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 4204 -checkpoint 15 -np 3
02:31:02:WU00:FS00:0xa7:************************************ CBang *************************************
02:31:02:WU00:FS00:0xa7:       Date: Nov 27 2019
02:31:02:WU00:FS00:0xa7:       Time: 03:40:09
02:31:02:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
02:31:02:WU00:FS00:0xa7:     Branch: master
02:31:02:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
02:31:02:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
02:31:02:WU00:FS00:0xa7:   Platform: win32 10
02:31:02:WU00:FS00:0xa7:       Bits: 64
02:31:02:WU00:FS00:0xa7:       Mode: Release
02:31:02:WU00:FS00:0xa7:************************************ System ************************************
02:31:02:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
02:31:02:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
02:31:02:WU00:FS00:0xa7:       CPUs: 4
02:31:02:WU00:FS00:0xa7:     Memory: 14.97GiB
02:31:02:WU00:FS00:0xa7:Free Memory: 12.30GiB
02:31:02:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
02:31:02:WU00:FS00:0xa7: OS Version: 6.2
02:31:02:WU00:FS00:0xa7:Has Battery: false
02:31:02:WU00:FS00:0xa7: On Battery: false
02:31:02:WU00:FS00:0xa7: UTC Offset: -5
02:31:02:WU00:FS00:0xa7:        PID: 6256
02:31:02:WU00:FS00:0xa7:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
02:31:02:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
02:31:02:WU00:FS00:0xa7:    Version: 0.0.19
02:31:02:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
02:31:02:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
02:31:02:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
02:31:02:WU00:FS00:0xa7:       Date: Nov 25 2019
02:31:02:WU00:FS00:0xa7:       Time: 17:12:41
02:31:02:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
02:31:02:WU00:FS00:0xa7:     Branch: master
02:31:02:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
02:31:02:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
02:31:02:WU00:FS00:0xa7:   Platform: win32 10
02:31:02:WU00:FS00:0xa7:       Bits: 64
02:31:02:WU00:FS00:0xa7:       Mode: Release
02:31:02:WU00:FS00:0xa7:************************************ Build *************************************
02:31:02:WU00:FS00:0xa7:       SIMD: avx_256
02:31:02:WU00:FS00:0xa7:********************************************************************************
02:31:02:WU00:FS00:0xa7:Project: 14726 (Run 2151, Clone 1, Gen 25)
02:31:02:WU00:FS00:0xa7:Unit: 0x0000001a03159d0b5ebcaae3901909ce
02:31:02:WU00:FS00:0xa7:Reading tar file core.xml
02:31:02:WU00:FS00:0xa7:Reading tar file frame25.tpr
02:31:02:WU00:FS00:0xa7:Digital signatures verified
02:31:02:WU00:FS00:0xa7:Calling: mdrun -s frame25.tpr -o frame25.trr -cpt 15 -nt 3
02:31:02:WU00:FS00:0xa7:Steps: first=0 total=250000
02:31:03:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
02:31:16:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
02:31:47:Saving configuration to config.xml
02:31:47:<config>
02:31:47:  <!-- Network -->
02:31:47:  <proxy v=':8080'/>
02:31:47:
02:31:47:  <!-- Slot Control -->
02:31:47:  <power v='FULL'/>
02:31:47:
02:31:47:  <!-- User Information -->
02:31:47:  <passkey v='*****'/>
02:31:47:  <team v='12072'/>
02:31:47:  <user v='crimson1077'/>
02:31:47:
02:31:47:  <!-- Folding Slots -->
02:31:47:  <slot id='0' type='CPU'/>
02:31:47:  <slot id='1' type='GPU'/>
02:31:47:</config>
02:31:48:40:127.0.0.1:New Web session
02:33:13:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
02:34:42:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
02:35:19:WU00:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
02:37:27:WU00:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
02:38:05:WU01:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
02:39:32:WU00:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
02:39:37:WU01:FS01:0x22:An exception occurred at step 24597: Particle coordinate is nan
02:39:37:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
02:39:37:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
02:39:37:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
02:39:37:WU01:FS01:Starting
02:39:37:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6532 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:39:37:WU01:FS01:Started FahCore on PID 3836
02:39:38:WU01:FS01:Core PID:4584
02:39:38:WU01:FS01:FahCore 0x22 started
02:39:38:WU01:FS01:0x22:*********************** Log Started 2020-08-30T02:39:38Z ***********************
02:39:38:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
02:39:38:WU01:FS01:0x22:       Core: Core22
02:39:38:WU01:FS01:0x22:       Type: 0x22
02:39:38:WU01:FS01:0x22:    Version: 0.0.11
02:39:38:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
02:39:38:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
02:39:38:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
02:39:38:WU01:FS01:0x22:       Date: Jun 26 2020
02:39:38:WU01:FS01:0x22:       Time: 19:49:16
02:39:38:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
02:39:38:WU01:FS01:0x22:     Branch: core22-0.0.11
02:39:38:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:39:38:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:39:38:WU01:FS01:0x22:   Platform: win32 10
02:39:38:WU01:FS01:0x22:       Bits: 64
02:39:38:WU01:FS01:0x22:       Mode: Release
02:39:38:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
02:39:38:WU01:FS01:0x22:             <peastman@stanford.edu>
02:39:38:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 3836 -checkpoint 15
02:39:38:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:39:38:WU01:FS01:0x22:************************************ libFAH ************************************
02:39:38:WU01:FS01:0x22:       Date: Jun 26 2020
02:39:38:WU01:FS01:0x22:       Time: 19:47:12
02:39:38:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
02:39:38:WU01:FS01:0x22:     Branch: HEAD
02:39:38:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:39:38:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:39:38:WU01:FS01:0x22:   Platform: win32 10
02:39:38:WU01:FS01:0x22:       Bits: 64
02:39:38:WU01:FS01:0x22:       Mode: Release
02:39:38:WU01:FS01:0x22:************************************ CBang *************************************
02:39:38:WU01:FS01:0x22:       Date: Jun 26 2020
02:39:38:WU01:FS01:0x22:       Time: 19:46:11
02:39:38:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
02:39:38:WU01:FS01:0x22:     Branch: master
02:39:38:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:39:38:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:39:38:WU01:FS01:0x22:   Platform: win32 10
02:39:38:WU01:FS01:0x22:       Bits: 64
02:39:38:WU01:FS01:0x22:       Mode: Release
02:39:38:WU01:FS01:0x22:************************************ System ************************************
02:39:38:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
02:39:38:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
02:39:38:WU01:FS01:0x22:       CPUs: 4
02:39:38:WU01:FS01:0x22:     Memory: 14.97GiB
02:39:38:WU01:FS01:0x22:Free Memory: 12.39GiB
02:39:38:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
02:39:38:WU01:FS01:0x22: OS Version: 6.2
02:39:38:WU01:FS01:0x22:Has Battery: false
02:39:38:WU01:FS01:0x22: On Battery: false
02:39:38:WU01:FS01:0x22: UTC Offset: -5
02:39:38:WU01:FS01:0x22:        PID: 4584
02:39:38:WU01:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
02:39:38:WU01:FS01:0x22:********************************************************************************
02:39:38:WU01:FS01:0x22:Project: 13424 (Run 787, Clone 47, Gen 2)
02:39:38:WU01:FS01:0x22:Unit: 0x0000000412bc7d9a5f498aed479d8f25
02:39:38:WU01:FS01:0x22:Digital signatures verified
02:39:38:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
02:39:38:WU01:FS01:0x22:Version 0.0.11
02:39:38:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
02:39:38:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
02:39:38:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
02:39:38:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
02:39:55:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
02:41:40:WU00:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
02:43:17:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
02:43:45:WU00:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
02:43:56:WU01:FS01:0x22:An exception occurred at step 12047: Particle coordinate is nan
02:43:56:WU01:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
02:43:56:WU01:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
02:43:56:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
02:43:56:WU01:FS01:0x22:Saving result file globals.csv
02:43:56:WU01:FS01:0x22:Saving result file science.log
02:43:56:WU01:FS01:0x22:Saving result file state.xml.bz2
02:43:57:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
02:43:58:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:43:58:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13424 run:787 clone:47 gen:2 core:0x22 unit:0x0000000412bc7d9a5f498aed479d8f25
02:43:58:WU01:FS01:Uploading 5.62MiB to 18.188.125.154
02:43:58:WU01:FS01:Connecting to 18.188.125.154:8080
02:43:58:WU02:FS01:Connecting to assign1.foldingathome.org:80
02:43:59:WU02:FS01:Assigned to work server 18.188.125.154
02:43:59:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:Hawaii [Radeon R9 200/300 Series] from 18.188.125.154
02:43:59:WU02:FS01:Connecting to 18.188.125.154:8080
02:44:00:WU02:FS01:Downloading 6.19MiB
02:44:01:WU02:FS01:Download complete
02:44:01:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13424 run:712 clone:44 gen:2 core:0x22 unit:0x0000000212bc7d9a5f498aefaa7598d6
02:44:01:WU02:FS01:Starting
02:44:01:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6532 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:44:01:WU02:FS01:Started FahCore on PID 5424
02:44:01:WU02:FS01:Core PID:1140
02:44:01:WU02:FS01:FahCore 0x22 started
02:44:02:WU02:FS01:0x22:*********************** Log Started 2020-08-30T02:44:01Z ***********************
02:44:02:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
02:44:02:WU02:FS01:0x22:       Core: Core22
02:44:02:WU02:FS01:0x22:       Type: 0x22
02:44:02:WU02:FS01:0x22:    Version: 0.0.11
02:44:02:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
02:44:02:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
02:44:02:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
02:44:02:WU02:FS01:0x22:       Date: Jun 26 2020
02:44:02:WU02:FS01:0x22:       Time: 19:49:16
02:44:02:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
02:44:02:WU02:FS01:0x22:     Branch: core22-0.0.11
02:44:02:WU02:FS01:0x22:   Compiler: Visual C++ 2015
02:44:02:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:44:02:WU02:FS01:0x22:   Platform: win32 10
02:44:02:WU02:FS01:0x22:       Bits: 64
02:44:02:WU02:FS01:0x22:       Mode: Release
02:44:02:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
02:44:02:WU02:FS01:0x22:             <peastman@stanford.edu>
02:44:02:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 5424 -checkpoint 15
02:44:02:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:44:02:WU02:FS01:0x22:************************************ libFAH ************************************
02:44:02:WU02:FS01:0x22:       Date: Jun 26 2020
02:44:02:WU02:FS01:0x22:       Time: 19:47:12
02:44:02:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
02:44:02:WU02:FS01:0x22:     Branch: HEAD
02:44:02:WU02:FS01:0x22:   Compiler: Visual C++ 2015
02:44:02:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:44:02:WU02:FS01:0x22:   Platform: win32 10
02:44:02:WU02:FS01:0x22:       Bits: 64
02:44:02:WU02:FS01:0x22:       Mode: Release
02:44:02:WU02:FS01:0x22:************************************ CBang *************************************
02:44:02:WU02:FS01:0x22:       Date: Jun 26 2020
02:44:02:WU02:FS01:0x22:       Time: 19:46:11
02:44:02:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
02:44:02:WU02:FS01:0x22:     Branch: master
02:44:02:WU02:FS01:0x22:   Compiler: Visual C++ 2015
02:44:02:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:44:02:WU02:FS01:0x22:   Platform: win32 10
02:44:02:WU02:FS01:0x22:       Bits: 64
02:44:02:WU02:FS01:0x22:       Mode: Release
02:44:02:WU02:FS01:0x22:************************************ System ************************************
02:44:02:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
02:44:02:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
02:44:02:WU02:FS01:0x22:       CPUs: 4
02:44:02:WU02:FS01:0x22:     Memory: 14.97GiB
02:44:02:WU02:FS01:0x22:Free Memory: 12.33GiB
02:44:02:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
02:44:02:WU02:FS01:0x22: OS Version: 6.2
02:44:02:WU02:FS01:0x22:Has Battery: false
02:44:02:WU02:FS01:0x22: On Battery: false
02:44:02:WU02:FS01:0x22: UTC Offset: -5
02:44:02:WU02:FS01:0x22:        PID: 1140
02:44:02:WU02:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
02:44:02:WU02:FS01:0x22:********************************************************************************
02:44:02:WU02:FS01:0x22:Project: 13424 (Run 712, Clone 44, Gen 2)
02:44:02:WU02:FS01:0x22:Unit: 0x0000000212bc7d9a5f498aefaa7598d6
02:44:02:WU02:FS01:0x22:Reading tar file core.xml
02:44:02:WU02:FS01:0x22:Reading tar file integrator.xml
02:44:02:WU02:FS01:0x22:Reading tar file state.xml.bz2
02:44:02:WU02:FS01:0x22:Reading tar file system.xml.bz2
02:44:02:WU02:FS01:0x22:Digital signatures verified
02:44:02:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
02:44:02:WU02:FS01:0x22:Version 0.0.11
02:44:02:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
02:44:02:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
02:44:02:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
02:44:02:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
02:44:04:WU01:FS01:Upload complete
02:44:04:WU01:FS01:Server responded WORK_ACK (400)
02:44:04:WU01:FS01:Cleaning up
02:44:20:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
02:45:00:Saving configuration to config.xml
02:45:00:<config>
02:45:00:  <!-- Folding Core -->
02:45:00:  <checkpoint v='3'/>
02:45:00:
02:45:00:  <!-- Network -->
02:45:00:  <proxy v=':8080'/>
02:45:00:
02:45:00:  <!-- Slot Control -->
02:45:00:  <power v='FULL'/>
02:45:00:
02:45:00:  <!-- User Information -->
02:45:00:  <passkey v='*****'/>
02:45:00:  <team v='12072'/>
02:45:00:  <user v='crimson1077'/>
02:45:00:
02:45:00:  <!-- Folding Slots -->
02:45:00:  <slot id='0' type='CPU'/>
02:45:00:  <slot id='1' type='GPU'/>
02:45:00:</config>
02:45:52:WU00:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
02:47:47:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
02:47:59:WU00:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
02:49:06:61:127.0.0.1:New Web session
02:50:03:WU02:FS01:0x22:An exception occurred at step 16816: Particle coordinate is nan
02:50:03:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
02:50:03:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
02:50:03:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
02:50:03:WU02:FS01:Starting
02:50:03:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6532 -checkpoint 3 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:50:03:WU02:FS01:Started FahCore on PID 6568
02:50:03:WU02:FS01:Core PID:3724
02:50:03:WU02:FS01:FahCore 0x22 started
02:50:04:WU02:FS01:0x22:*********************** Log Started 2020-08-30T02:50:03Z ***********************
02:50:04:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
02:50:04:WU02:FS01:0x22:       Core: Core22
02:50:04:WU02:FS01:0x22:       Type: 0x22
02:50:04:WU02:FS01:0x22:    Version: 0.0.11
02:50:04:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
02:50:04:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
02:50:04:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
02:50:04:WU02:FS01:0x22:       Date: Jun 26 2020
02:50:04:WU02:FS01:0x22:       Time: 19:49:16
02:50:04:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
02:50:04:WU02:FS01:0x22:     Branch: core22-0.0.11
02:50:04:WU02:FS01:0x22:   Compiler: Visual C++ 2015
02:50:04:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:50:04:WU02:FS01:0x22:   Platform: win32 10
02:50:04:WU02:FS01:0x22:       Bits: 64
02:50:04:WU02:FS01:0x22:       Mode: Release
02:50:04:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
02:50:04:WU02:FS01:0x22:             <peastman@stanford.edu>
02:50:04:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 6568 -checkpoint 3
02:50:04:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:50:04:WU02:FS01:0x22:************************************ libFAH ************************************
02:50:04:WU02:FS01:0x22:       Date: Jun 26 2020
02:50:04:WU02:FS01:0x22:       Time: 19:47:12
02:50:04:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
02:50:04:WU02:FS01:0x22:     Branch: HEAD
02:50:04:WU02:FS01:0x22:   Compiler: Visual C++ 2015
02:50:04:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:50:04:WU02:FS01:0x22:   Platform: win32 10
02:50:04:WU02:FS01:0x22:       Bits: 64
02:50:04:WU02:FS01:0x22:       Mode: Release
02:50:04:WU02:FS01:0x22:************************************ CBang *************************************
02:50:04:WU02:FS01:0x22:       Date: Jun 26 2020
02:50:04:WU02:FS01:0x22:       Time: 19:46:11
02:50:04:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
02:50:04:WU02:FS01:0x22:     Branch: master
02:50:04:WU02:FS01:0x22:   Compiler: Visual C++ 2015
02:50:04:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:50:04:WU02:FS01:0x22:   Platform: win32 10
02:50:04:WU02:FS01:0x22:       Bits: 64
02:50:04:WU02:FS01:0x22:       Mode: Release
02:50:04:WU02:FS01:0x22:************************************ System ************************************
02:50:04:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
02:50:04:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
02:50:04:WU02:FS01:0x22:       CPUs: 4
02:50:04:WU02:FS01:0x22:     Memory: 14.97GiB
02:50:04:WU02:FS01:0x22:Free Memory: 12.34GiB
02:50:04:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
02:50:04:WU02:FS01:0x22: OS Version: 6.2
02:50:04:WU02:FS01:0x22:Has Battery: false
02:50:04:WU02:FS01:0x22: On Battery: false
02:50:04:WU02:FS01:0x22: UTC Offset: -5
02:50:04:WU02:FS01:0x22:        PID: 3724
02:50:04:WU02:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
02:50:04:WU02:FS01:0x22:********************************************************************************
02:50:04:WU02:FS01:0x22:Project: 13424 (Run 712, Clone 44, Gen 2)
02:50:04:WU02:FS01:0x22:Unit: 0x0000000212bc7d9a5f498aefaa7598d6
02:50:04:WU02:FS01:0x22:Digital signatures verified
02:50:04:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
02:50:04:WU02:FS01:0x22:Version 0.0.11
02:50:04:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
02:50:04:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
02:50:04:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
02:50:04:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
02:50:04:WU00:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
02:50:21:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
02:52:17:WU00:FS00:0xa7:Completed 25000 out of 250000 steps (10%)
02:53:56:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
02:54:37:WU00:FS00:0xa7:Completed 27500 out of 250000 steps (11%)
02:55:39:WU02:FS01:0x22:An exception occurred at step 14808: Particle coordinate is nan
02:55:39:WU02:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
02:55:39:WU02:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
02:55:39:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
02:55:39:WU02:FS01:0x22:Saving result file globals.csv
02:55:39:WU02:FS01:0x22:Saving result file science.log
02:55:39:WU02:FS01:0x22:Saving result file state.xml.bz2
02:55:40:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
02:55:41:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:55:41:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13424 run:712 clone:44 gen:2 core:0x22 unit:0x0000000212bc7d9a5f498aefaa7598d6
02:55:41:WU02:FS01:Uploading 5.61MiB to 18.188.125.154
02:55:41:WU02:FS01:Connecting to 18.188.125.154:8080
02:55:41:WU01:FS01:Connecting to assign1.foldingathome.org:80
02:55:41:WU01:FS01:Assigned to work server 18.188.125.154
02:55:41:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:Hawaii [Radeon R9 200/300 Series] from 18.188.125.154
02:55:41:WU01:FS01:Connecting to 18.188.125.154:8080
02:55:43:WU01:FS01:Downloading 6.20MiB
02:55:47:WU02:FS01:Upload complete
02:55:47:WU02:FS01:Server responded WORK_ACK (400)
02:55:47:WU02:FS01:Cleaning up
02:55:48:WU01:FS01:Download complete
02:55:48:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13424 run:544 clone:11 gen:5 core:0x22 unit:0x0000000512bc7d9a5f498af58e512f96
02:55:48:WU01:FS01:Starting
02:55:48:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6532 -checkpoint 3 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:55:48:WU01:FS01:Started FahCore on PID 6816
02:55:48:WU01:FS01:Core PID:5056
02:55:48:WU01:FS01:FahCore 0x22 started
02:55:49:WU01:FS01:0x22:*********************** Log Started 2020-08-30T02:55:48Z ***********************
02:55:49:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
02:55:49:WU01:FS01:0x22:       Core: Core22
02:55:49:WU01:FS01:0x22:       Type: 0x22
02:55:49:WU01:FS01:0x22:    Version: 0.0.11
02:55:49:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
02:55:49:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
02:55:49:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
02:55:49:WU01:FS01:0x22:       Date: Jun 26 2020
02:55:49:WU01:FS01:0x22:       Time: 19:49:16
02:55:49:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
02:55:49:WU01:FS01:0x22:     Branch: core22-0.0.11
02:55:49:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:55:49:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:55:49:WU01:FS01:0x22:   Platform: win32 10
02:55:49:WU01:FS01:0x22:       Bits: 64
02:55:49:WU01:FS01:0x22:       Mode: Release
02:55:49:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
02:55:49:WU01:FS01:0x22:             <peastman@stanford.edu>
02:55:49:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 6816 -checkpoint 3
02:55:49:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
02:55:49:WU01:FS01:0x22:************************************ libFAH ************************************
02:55:49:WU01:FS01:0x22:       Date: Jun 26 2020
02:55:49:WU01:FS01:0x22:       Time: 19:47:12
02:55:49:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
02:55:49:WU01:FS01:0x22:     Branch: HEAD
02:55:49:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:55:49:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:55:49:WU01:FS01:0x22:   Platform: win32 10
02:55:49:WU01:FS01:0x22:       Bits: 64
02:55:49:WU01:FS01:0x22:       Mode: Release
02:55:49:WU01:FS01:0x22:************************************ CBang *************************************
02:55:49:WU01:FS01:0x22:       Date: Jun 26 2020
02:55:49:WU01:FS01:0x22:       Time: 19:46:11
02:55:49:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
02:55:49:WU01:FS01:0x22:     Branch: master
02:55:49:WU01:FS01:0x22:   Compiler: Visual C++ 2015
02:55:49:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
02:55:49:WU01:FS01:0x22:   Platform: win32 10
02:55:49:WU01:FS01:0x22:       Bits: 64
02:55:49:WU01:FS01:0x22:       Mode: Release
02:55:49:WU01:FS01:0x22:************************************ System ************************************
02:55:49:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
02:55:49:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
02:55:49:WU01:FS01:0x22:       CPUs: 4
02:55:49:WU01:FS01:0x22:     Memory: 14.97GiB
02:55:49:WU01:FS01:0x22:Free Memory: 12.60GiB
02:55:49:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
02:55:49:WU01:FS01:0x22: OS Version: 6.2
02:55:49:WU01:FS01:0x22:Has Battery: false
02:55:49:WU01:FS01:0x22: On Battery: false
02:55:49:WU01:FS01:0x22: UTC Offset: -5
02:55:49:WU01:FS01:0x22:        PID: 5056
02:55:49:WU01:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
02:55:49:WU01:FS01:0x22:********************************************************************************
02:55:49:WU01:FS01:0x22:Project: 13424 (Run 544, Clone 11, Gen 5)
02:55:49:WU01:FS01:0x22:Unit: 0x0000000512bc7d9a5f498af58e512f96
02:55:49:WU01:FS01:0x22:Reading tar file core.xml
02:55:49:WU01:FS01:0x22:Reading tar file integrator.xml
02:55:49:WU01:FS01:0x22:Reading tar file state.xml.bz2
02:55:49:WU01:FS01:0x22:Reading tar file system.xml.bz2
02:55:49:WU01:FS01:0x22:Digital signatures verified
02:55:49:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
02:55:49:WU01:FS01:0x22:Version 0.0.11
02:55:49:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
02:55:49:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
02:55:49:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
02:55:49:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
02:56:07:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
02:56:50:WU00:FS00:0xa7:Completed 30000 out of 250000 steps (12%)
02:58:56:WU00:FS00:0xa7:Completed 32500 out of 250000 steps (13%)
02:59:28:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
03:00:04:WU01:FS01:0x22:An exception occurred at step 11796: Particle coordinate is nan
03:00:04:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
03:00:04:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
03:00:04:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
03:00:05:WU01:FS01:Starting
03:00:05:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6532 -checkpoint 3 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
03:00:05:WU01:FS01:Started FahCore on PID 6680
03:00:05:WU01:FS01:Core PID:3136
03:00:05:WU01:FS01:FahCore 0x22 started
03:00:05:WU01:FS01:0x22:*********************** Log Started 2020-08-30T03:00:05Z ***********************
03:00:05:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
03:00:05:WU01:FS01:0x22:       Core: Core22
03:00:05:WU01:FS01:0x22:       Type: 0x22
03:00:05:WU01:FS01:0x22:    Version: 0.0.11
03:00:05:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
03:00:05:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
03:00:05:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
03:00:05:WU01:FS01:0x22:       Date: Jun 26 2020
03:00:05:WU01:FS01:0x22:       Time: 19:49:16
03:00:05:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
03:00:05:WU01:FS01:0x22:     Branch: core22-0.0.11
03:00:05:WU01:FS01:0x22:   Compiler: Visual C++ 2015
03:00:05:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:00:05:WU01:FS01:0x22:   Platform: win32 10
03:00:05:WU01:FS01:0x22:       Bits: 64
03:00:05:WU01:FS01:0x22:       Mode: Release
03:00:05:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
03:00:05:WU01:FS01:0x22:             <peastman@stanford.edu>
03:00:05:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 6680 -checkpoint 3
03:00:05:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
03:00:05:WU01:FS01:0x22:************************************ libFAH ************************************
03:00:05:WU01:FS01:0x22:       Date: Jun 26 2020
03:00:05:WU01:FS01:0x22:       Time: 19:47:12
03:00:05:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
03:00:05:WU01:FS01:0x22:     Branch: HEAD
03:00:05:WU01:FS01:0x22:   Compiler: Visual C++ 2015
03:00:05:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:00:05:WU01:FS01:0x22:   Platform: win32 10
03:00:05:WU01:FS01:0x22:       Bits: 64
03:00:05:WU01:FS01:0x22:       Mode: Release
03:00:05:WU01:FS01:0x22:************************************ CBang *************************************
03:00:05:WU01:FS01:0x22:       Date: Jun 26 2020
03:00:05:WU01:FS01:0x22:       Time: 19:46:11
03:00:05:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
03:00:05:WU01:FS01:0x22:     Branch: master
03:00:05:WU01:FS01:0x22:   Compiler: Visual C++ 2015
03:00:05:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:00:05:WU01:FS01:0x22:   Platform: win32 10
03:00:05:WU01:FS01:0x22:       Bits: 64
03:00:05:WU01:FS01:0x22:       Mode: Release
03:00:05:WU01:FS01:0x22:************************************ System ************************************
03:00:05:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
03:00:05:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
03:00:05:WU01:FS01:0x22:       CPUs: 4
03:00:05:WU01:FS01:0x22:     Memory: 14.97GiB
03:00:05:WU01:FS01:0x22:Free Memory: 12.97GiB
03:00:05:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
03:00:05:WU01:FS01:0x22: OS Version: 6.2
03:00:05:WU01:FS01:0x22:Has Battery: false
03:00:05:WU01:FS01:0x22: On Battery: false
03:00:05:WU01:FS01:0x22: UTC Offset: -5
03:00:05:WU01:FS01:0x22:        PID: 3136
03:00:05:WU01:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
03:00:05:WU01:FS01:0x22:********************************************************************************
03:00:05:WU01:FS01:0x22:Project: 13424 (Run 544, Clone 11, Gen 5)
03:00:05:WU01:FS01:0x22:Unit: 0x0000000512bc7d9a5f498af58e512f96
03:00:05:WU01:FS01:0x22:Digital signatures verified
03:00:05:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
03:00:05:WU01:FS01:0x22:Version 0.0.11
03:00:05:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
03:00:05:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
03:00:05:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
03:00:05:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
03:00:22:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
03:01:02:WU00:FS00:0xa7:Completed 35000 out of 250000 steps (14%)
03:01:16:Saving configuration to config.xml
03:01:16:<config>
03:01:16:  <!-- Folding Core -->
03:01:16:  <checkpoint v='10'/>
03:01:16:
03:01:16:  <!-- Network -->
03:01:16:  <proxy v=':8080'/>
03:01:16:
03:01:16:  <!-- Slot Control -->
03:01:16:  <power v='FULL'/>
03:01:16:
03:01:16:  <!-- User Information -->
03:01:16:  <passkey v='*****'/>
03:01:16:  <team v='12072'/>
03:01:16:  <user v='crimson1077'/>
03:01:16:
03:01:16:  <!-- Folding Slots -->
03:01:16:  <slot id='0' type='CPU'/>
03:01:16:  <slot id='1' type='GPU'/>
03:01:16:</config>
03:03:08:WU00:FS00:0xa7:Completed 37500 out of 250000 steps (15%)
03:03:42:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
03:04:35:WU01:FS01:0x22:An exception occurred at step 12549: Particle coordinate is nan
03:04:35:WU01:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
03:04:35:WU01:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
03:04:35:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
03:04:35:WU01:FS01:0x22:Saving result file globals.csv
03:04:35:WU01:FS01:0x22:Saving result file science.log
03:04:35:WU01:FS01:0x22:Saving result file state.xml.bz2
03:04:36:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
03:04:37:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:04:37:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13424 run:544 clone:11 gen:5 core:0x22 unit:0x0000000512bc7d9a5f498af58e512f96
03:04:37:WU01:FS01:Uploading 5.62MiB to 18.188.125.154
03:04:37:WU01:FS01:Connecting to 18.188.125.154:8080
03:04:37:WU02:FS01:Connecting to assign1.foldingathome.org:80
03:04:37:WU02:FS01:Assigned to work server 18.188.125.154
03:04:37:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:Hawaii [Radeon R9 200/300 Series] from 18.188.125.154
03:04:37:WU02:FS01:Connecting to 18.188.125.154:8080
03:04:39:WU02:FS01:Downloading 6.20MiB
03:04:42:WU02:FS01:Download complete
03:04:42:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13424 run:797 clone:28 gen:1 core:0x22 unit:0x0000000212bc7d9a5f498aed5fc73b7a
03:04:42:WU02:FS01:Starting
03:04:42:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6532 -checkpoint 10 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
03:04:42:WU02:FS01:Started FahCore on PID 2172
03:04:42:WU02:FS01:Core PID:6648
03:04:42:WU02:FS01:FahCore 0x22 started
03:04:43:WU01:FS01:Upload complete
03:04:43:WU02:FS01:0x22:*********************** Log Started 2020-08-30T03:04:42Z ***********************
03:04:43:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
03:04:43:WU02:FS01:0x22:       Core: Core22
03:04:43:WU02:FS01:0x22:       Type: 0x22
03:04:43:WU02:FS01:0x22:    Version: 0.0.11
03:04:43:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
03:04:43:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
03:04:43:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
03:04:43:WU02:FS01:0x22:       Date: Jun 26 2020
03:04:43:WU02:FS01:0x22:       Time: 19:49:16
03:04:43:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
03:04:43:WU02:FS01:0x22:     Branch: core22-0.0.11
03:04:43:WU02:FS01:0x22:   Compiler: Visual C++ 2015
03:04:43:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:04:43:WU02:FS01:0x22:   Platform: win32 10
03:04:43:WU02:FS01:0x22:       Bits: 64
03:04:43:WU02:FS01:0x22:       Mode: Release
03:04:43:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
03:04:43:WU02:FS01:0x22:             <peastman@stanford.edu>
03:04:43:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 2172 -checkpoint 10
03:04:43:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
03:04:43:WU02:FS01:0x22:************************************ libFAH ************************************
03:04:43:WU02:FS01:0x22:       Date: Jun 26 2020
03:04:43:WU02:FS01:0x22:       Time: 19:47:12
03:04:43:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
03:04:43:WU02:FS01:0x22:     Branch: HEAD
03:04:43:WU02:FS01:0x22:   Compiler: Visual C++ 2015
03:04:43:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:04:43:WU02:FS01:0x22:   Platform: win32 10
03:04:43:WU02:FS01:0x22:       Bits: 64
03:04:43:WU02:FS01:0x22:       Mode: Release
03:04:43:WU02:FS01:0x22:************************************ CBang *************************************
03:04:43:WU02:FS01:0x22:       Date: Jun 26 2020
03:04:43:WU02:FS01:0x22:       Time: 19:46:11
03:04:43:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
03:04:43:WU02:FS01:0x22:     Branch: master
03:04:43:WU02:FS01:0x22:   Compiler: Visual C++ 2015
03:04:43:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:04:43:WU02:FS01:0x22:   Platform: win32 10
03:04:43:WU02:FS01:0x22:       Bits: 64
03:04:43:WU02:FS01:0x22:       Mode: Release
03:04:43:WU02:FS01:0x22:************************************ System ************************************
03:04:43:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
03:04:43:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
03:04:43:WU02:FS01:0x22:       CPUs: 4
03:04:43:WU02:FS01:0x22:     Memory: 14.97GiB
03:04:43:WU02:FS01:0x22:Free Memory: 12.45GiB
03:04:43:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
03:04:43:WU02:FS01:0x22: OS Version: 6.2
03:04:43:WU02:FS01:0x22:Has Battery: false
03:04:43:WU02:FS01:0x22: On Battery: false
03:04:43:WU02:FS01:0x22: UTC Offset: -5
03:04:43:WU02:FS01:0x22:        PID: 6648
03:04:43:WU02:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
03:04:43:WU02:FS01:0x22:********************************************************************************
03:04:43:WU02:FS01:0x22:Project: 13424 (Run 797, Clone 28, Gen 1)
03:04:43:WU02:FS01:0x22:Unit: 0x0000000212bc7d9a5f498aed5fc73b7a
03:04:43:WU02:FS01:0x22:Reading tar file core.xml
03:04:43:WU02:FS01:0x22:Reading tar file integrator.xml
03:04:43:WU02:FS01:0x22:Reading tar file state.xml.bz2
03:04:43:WU02:FS01:0x22:Reading tar file system.xml.bz2
03:04:43:WU02:FS01:0x22:Digital signatures verified
03:04:43:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
03:04:43:WU02:FS01:0x22:Version 0.0.11
03:04:43:WU01:FS01:Server responded WORK_ACK (400)
03:04:43:WU01:FS01:Cleaning up
03:04:43:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
03:04:43:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
03:04:43:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
03:04:43:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
03:04:54:FS01:Paused
03:04:54:FS01:Shutting core down
03:04:54:WU02:FS01:0x22:WARNING:Console control signal 1 on PID 6648
03:04:54:WU02:FS01:0x22:Exiting, please wait. . .
03:05:01:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
03:05:01:WU02:FS01:0x22:Folding@home Core Shutdown: INTERRUPTED
03:05:01:WU02:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
03:05:20:Saving configuration to config.xml
03:05:20:<config>
03:05:20:  <!-- Folding Core -->
03:05:20:  <checkpoint v='10'/>
03:05:20:
03:05:20:  <!-- Network -->
03:05:20:  <proxy v=':8080'/>
03:05:20:
03:05:20:  <!-- Slot Control -->
03:05:20:  <power v='FULL'/>
03:05:20:
03:05:20:  <!-- User Information -->
03:05:20:  <passkey v='*****'/>
03:05:20:  <team v='12072'/>
03:05:20:  <user v='crimson1077'/>
03:05:20:
03:05:20:  <!-- Folding Slots -->
03:05:20:  <slot id='0' type='CPU'/>
03:05:20:  <slot id='1' type='GPU'>
03:05:20:    <paused v='true'/>
03:05:20:  </slot>
03:05:20:</config>
03:05:21:WU00:FS00:0xa7:Completed 40000 out of 250000 steps (16%)
03:05:37:FS01:Unpaused
03:05:37:FS01:Finishing
03:05:37:WU02:FS01:Starting
03:05:37:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6532 -checkpoint 10 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
03:05:37:WU02:FS01:Started FahCore on PID 6620
03:05:37:WU02:FS01:Core PID:5180
03:05:37:WU02:FS01:FahCore 0x22 started
03:05:37:WU02:FS01:0x22:*********************** Log Started 2020-08-30T03:05:37Z ***********************
03:05:37:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
03:05:37:WU02:FS01:0x22:       Core: Core22
03:05:37:WU02:FS01:0x22:       Type: 0x22
03:05:37:WU02:FS01:0x22:    Version: 0.0.11
03:05:37:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
03:05:37:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
03:05:37:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
03:05:37:WU02:FS01:0x22:       Date: Jun 26 2020
03:05:37:WU02:FS01:0x22:       Time: 19:49:16
03:05:37:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
03:05:37:WU02:FS01:0x22:     Branch: core22-0.0.11
03:05:37:WU02:FS01:0x22:   Compiler: Visual C++ 2015
03:05:37:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:05:37:WU02:FS01:0x22:   Platform: win32 10
03:05:37:WU02:FS01:0x22:       Bits: 64
03:05:37:WU02:FS01:0x22:       Mode: Release
03:05:37:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
03:05:37:WU02:FS01:0x22:             <peastman@stanford.edu>
03:05:37:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 6620 -checkpoint 10
03:05:37:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
03:05:37:WU02:FS01:0x22:************************************ libFAH ************************************
03:05:37:WU02:FS01:0x22:       Date: Jun 26 2020
03:05:37:WU02:FS01:0x22:       Time: 19:47:12
03:05:37:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
03:05:37:WU02:FS01:0x22:     Branch: HEAD
03:05:37:WU02:FS01:0x22:   Compiler: Visual C++ 2015
03:05:37:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:05:37:WU02:FS01:0x22:   Platform: win32 10
03:05:37:WU02:FS01:0x22:       Bits: 64
03:05:37:WU02:FS01:0x22:       Mode: Release
03:05:37:WU02:FS01:0x22:************************************ CBang *************************************
03:05:37:WU02:FS01:0x22:       Date: Jun 26 2020
03:05:37:WU02:FS01:0x22:       Time: 19:46:11
03:05:37:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
03:05:37:WU02:FS01:0x22:     Branch: master
03:05:37:WU02:FS01:0x22:   Compiler: Visual C++ 2015
03:05:37:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:05:37:WU02:FS01:0x22:   Platform: win32 10
03:05:37:WU02:FS01:0x22:       Bits: 64
03:05:37:WU02:FS01:0x22:       Mode: Release
03:05:37:WU02:FS01:0x22:************************************ System ************************************
03:05:37:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
03:05:37:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
03:05:37:WU02:FS01:0x22:       CPUs: 4
03:05:37:WU02:FS01:0x22:     Memory: 14.97GiB
03:05:37:WU02:FS01:0x22:Free Memory: 12.47GiB
03:05:37:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
03:05:37:WU02:FS01:0x22: OS Version: 6.2
03:05:37:WU02:FS01:0x22:Has Battery: false
03:05:37:WU02:FS01:0x22: On Battery: false
03:05:37:WU02:FS01:0x22: UTC Offset: -5
03:05:37:WU02:FS01:0x22:        PID: 5180
03:05:37:WU02:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
03:05:37:WU02:FS01:0x22:********************************************************************************
03:05:37:WU02:FS01:0x22:Project: 13424 (Run 797, Clone 28, Gen 1)
03:05:37:WU02:FS01:0x22:Unit: 0x0000000212bc7d9a5f498aed5fc73b7a
03:05:37:WU02:FS01:0x22:Digital signatures verified
03:05:37:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
03:05:37:WU02:FS01:0x22:Version 0.0.11
03:05:37:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
03:05:37:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
03:05:37:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
03:05:37:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
03:05:54:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
03:06:21:Saving configuration to config.xml
03:06:21:<config>
03:06:21:  <!-- Folding Core -->
03:06:21:  <checkpoint v='10'/>
03:06:21:
03:06:21:  <!-- Network -->
03:06:21:  <proxy v=':8080'/>
03:06:21:
03:06:21:  <!-- Slot Control -->
03:06:21:  <power v='FULL'/>
03:06:21:
03:06:21:  <!-- User Information -->
03:06:21:  <passkey v='*****'/>
03:06:21:  <team v='12072'/>
03:06:21:  <user v='crimson1077'/>
03:06:21:
03:06:21:  <!-- Folding Slots -->
03:06:21:  <slot id='0' type='CPU'/>
03:06:21:  <slot id='1' type='GPU'/>
03:06:21:</config>
03:07:31:WU00:FS00:0xa7:Completed 42500 out of 250000 steps (17%)
03:09:18:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
03:09:34:WU02:FS01:0x22:An exception occurred at step 10792: Particle coordinate is nan
03:09:34:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
03:09:34:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
03:09:34:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
03:09:35:WU02:FS01:Starting
03:09:35:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6532 -checkpoint 10 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
03:09:35:WU02:FS01:Started FahCore on PID 6688
03:09:35:WU02:FS01:Core PID:3556
03:09:35:WU02:FS01:FahCore 0x22 started

Mod Edit: Added Code Tags - PantherX

[crimson1077]

stop, start over, go, REPEAT! lol
I noticed a pattern of events from this nightmare. My client try's to send a WU that's less then 10% complete then go to a different portion of a 13424 project meanwhile GPU still not folding so sad.

[NormalDiffusion]

OK, as my 290x is folding without problems, a couple of questions :
1. Which driver are you using?
2. Which card is it exactly? (vendor + model)
3. Which clock speed is she running (in the amd software)?

[crimson1077]

I have driver version win10-radeon-software-adrenalin-2020-edition-20.8.2-aug17

Here's the card https://pcpartpicker.com/product/ymqbt6 ... 90gaming8g

And nothing is overclocked nor has it ever been before. She's running stock 1040mhz.

[JohnChodera]

@crimson1077: We're steadily making progress on fixing some AMD issues that may be responsible for your issues. These will go into core22 0.0.12, slated to roll out around Sep 14-18.

So sorry about this---we're hoping we can remedy your issues in that release!

Thanks for sticking with us.

~ John Chodera // MSKCC

[NormalDiffusion]

I have driver version win10-radeon-software-adrenalin-2020-edition-20.8.2-aug17

Here's the card https://pcpartpicker.com/product/ymqbt6 ... 90gaming8g

And nothing is overclocked nor has it ever been before. She's running stock 1040mhz.

She is factory overclocked and can NOT cope with it! It's the same on my 290x.
Bring her down to 1000 mhz and she should do fine. Also put the fans on manual at least at 2/3 of max speed. Mine doesn't like anything over 58°C.
If still not able to fold, 980 MHz will do it.
You may want to try older drivers as well, for me they are a little bit faster...

[crimson1077]

@crimson1077: We're steadily making progress on fixing some AMD issues that may be responsible for your issues. These will go into core22 0.0.12, slated to roll out around Sep 14-18.

So sorry about this---we're hoping we can remedy your issues in that release!

Thanks for sticking with us.

~ John Chodera // MSKCC

No worries! I will stay in touch! Fold on!

[crimson1077]

I have driver version win10-radeon-software-adrenalin-2020-edition-20.8.2-aug17

Here's the card https://pcpartpicker.com/product/ymqbt6 ... 90gaming8g

And nothing is overclocked nor has it ever been before. She's running stock 1040mhz.

She is factory overclocked and can NOT cope with it! It's the same on my 290x.
Bring her down to 1000 mhz and she should do fine. Also put the fans on manual at least at 2/3 of max speed. Mine doesn't like anything over 58°C.
If still not able to fold, 980 MHz will do it.
You may want to try older drivers as well, for me they are a little bit faster...

Hmm interesting! It's worth a shot.

[NormalDiffusion]

Hmm interesting! It's worth a shot.

About 2 min to change settings
The gpu of the 390x and the 290x are exactly the same. They just pushed the 390x further and added 4gb more ram.

Let me know if it works for you!

[Joe_H]

As mentioned, the card is factory overclocked. The GPU chip when first released as the R9 290 had a max reference clock of 947 MHz, that was increased to 1000 MHz as the reference clock for the R9 390. MSI pushed that another 40 MHz. That may be stable in games and normal usage, but might not be stable under the heavy compute load of folding. Some of this will depend on the exact chip on the card, some people win the chip lottery and get cards that will overclock more than others. The increase in RAM from 4 GB to 8 GB between the 290 to the 390 can also add to the heat load.

The 290X/390x Have a few more shaders than the 290/390 GPUs.

[crimson1077]

I have driver version win10-radeon-software-adrenalin-2020-edition-20.8.2-aug17

Here's the card https://pcpartpicker.com/product/ymqbt6 ... 90gaming8g

And nothing is overclocked nor has it ever been before. She's running stock 1040mhz.

She is factory overclocked and can NOT cope with it! It's the same on my 290x.
Bring her down to 1000 mhz and she should do fine. Also put the fans on manual at least at 2/3 of max speed. Mine doesn't like anything over 58°C.
If still not able to fold, 980 MHz will do it.
You may want to try older drivers as well, for me they are a little bit faster...

NormalDiffusion!! You say drop 40 MHz? I SAY GEEK IT TIL IT MHz! Thank you!! I am now back to climbing the charts for my beloved Team MajorGeeks.com!
That makes perfect sense now... I just had to underclock

A big thanks to everyone who shared and helped!! I learned quite a bit in the process from you great folks. I hope this helps the next person as much as it helped me.

[NormalDiffusion]

NormalDiffusion!! You say drop 40 MHz? I SAY GEEK IT TIL IT MHz! Thank you!! I am now back to climbing the charts for my beloved Team MajorGeeks.com!
That makes perfect sense now... I just had to underclock

A big thanks to everyone who shared and helped!! I learned quite a bit in the process from you great folks. I hope this helps the next person as much as it helped me.

I'm glad you're back folding and happy I could help!
Keep folding!