Failed GPU slot daily :(

It seems that a lot of GPU problems revolve around specific versions of drivers. Though AMD has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

Post Reply
crimson1077
Posts: 33
Joined: Sat Aug 22, 2020 4:37 pm

Re: Failed GPU slot daily :(

Post by crimson1077 »

Neil-B wrote:Project 13422 is the P part of PRCG ... R is run, C is clone and G is generation ... a PRCG identifies a specific WU within a Project ... hopefully you have actually got a new PRCG within Project 13422 ... related to the buggy PRCG - was the couple of days pause iirc you mentioned during the folding of that WU? - and did you pause the slot and exit the client before shutting down? - I am simply wondering if the initial failure of the WU may have been linked to some corruption caused at that point ... for a safe shutdown it can be worth pausing slots (there are threads discussing best time to do this relating to checkpoints) quit the client then wait a bit to ensure everything is saved properly before shutting down the system - this seems to minimise chances of issues from that respect.
Yes, indeed I always pause before shutting down client or computer. I normal use max-units 1for the most part as this my gaming rig so I like to get the GPU free for gaming. (And just to be safe, no I do not leave FAH on while gaming lol)
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Failed GPU slot daily :(

Post by bruce »

crimson1077 wrote:
bruce wrote:Oh, that Crimson 8-)
Thanks for your help bruce. I did as you said with 01 folder and backed up and deleted. As I fired up FAH I watched in windows file explore tab 01 reappear as it should but still on PRCG 13422.
Maybe too soon to tell as of now. We shall see!
You have successfully completed several members of Project 13422. Are you still getting the error?

As Neil-B suggested, you may have been assigned a different Run from the same project or your download had been corrupted and you got a fresh copy, either may have resolved your issue.
Top
crimson1077
Posts: 33
Joined: Sat Aug 22, 2020 4:37 pm

Re: Failed GPU slot daily :(

Post by crimson1077 »

bruce wrote:
crimson1077 wrote:
bruce wrote:Oh, that Crimson 8-)
Thanks for your help bruce. I did as you said with 01 folder and backed up and deleted. As I fired up FAH I watched in windows file explore tab 01 reappear as it should but still on PRCG 13422.
Maybe too soon to tell as of now. We shall see!
You have successfully completed several members of Project 13422. Are you still getting the error?

As Neil-B suggested, you may have been assigned a different Run from the same project or your download had been corrupted and you got a fresh copy, either may have resolved your issue.
Unfortunately it is still doing the same thing, The 13422 (3560, 92, 2) WU is stuck in a loop :e?:
Top
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Failed GPU slot daily :(

Post by Neil-B »

So different WU than original but same issue?
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070
(Green/Bold = Active)
Top
crimson1077
Posts: 33
Joined: Sat Aug 22, 2020 4:37 pm

Re: Failed GPU slot daily :(

Post by crimson1077 »

Same WU, even after dumping.
Top
crimson1077
Posts: 33
Joined: Sat Aug 22, 2020 4:37 pm

Re: Failed GPU slot daily :(

Post by crimson1077 »

FAILED. :(

Code: Select all

*********************** Log Started 2020-08-23T19:53:33Z ***********************
19:53:33:Trying to access database...
19:53:33:Successfully acquired database lock
19:53:33:Downloading GPUs.txt from assign1.foldingathome.org:80
19:53:33:Connecting to assign1.foldingathome.org:80
19:53:33:Read GPUs.txt
19:53:33:Enabled folding slot 00: PAUSED cpu:2 (not configured)
19:53:34:Enabled folding slot 01: PAUSED gpu:0:Hawaii [Radeon R9 200/300 Series] (not configured)
19:53:34:****************************** FAHClient ******************************
19:53:34:        Version: 7.6.13
19:53:34:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:53:34:      Copyright: 2020 foldingathome.org
19:53:34:       Homepage: https://foldingathome.org/
19:53:34:           Date: Apr 27 2020
19:53:34:           Time: 21:21:01
19:53:34:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
19:53:34:         Branch: master
19:53:34:       Compiler: Visual C++ 2008
19:53:34:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:53:34:       Platform: win32 10
19:53:34:           Bits: 32
19:53:34:           Mode: Release
19:53:34:           Args: --open-web-control
19:53:34:******************************** CBang ********************************
19:53:34:           Date: Apr 24 2020
19:53:34:           Time: 17:07:55
19:53:34:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
19:53:34:         Branch: master
19:53:34:       Compiler: Visual C++ 2008
19:53:34:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:53:34:       Platform: win32 10
19:53:34:           Bits: 32
19:53:34:           Mode: Release
19:53:34:******************************* System ********************************
19:53:34:            CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
19:53:34:         CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:53:34:           CPUs: 4
19:53:34:         Memory: 14.97GiB
19:53:34:    Free Memory: 13.12GiB
19:53:34:        Threads: WINDOWS_THREADS
19:53:34:     OS Version: 6.2
19:53:34:    Has Battery: false
19:53:34:     On Battery: false
19:53:34:     UTC Offset: -5
19:53:34:            PID: 6488
19:53:34:            CWD: C:\Users\Crimson\AppData\Roaming\FAHClient
19:53:34:  Win32 Service: false
19:53:34:             OS: Windows 10 Enterprise LTSC 2019
19:53:34:        OS Arch: AMD64
19:53:34:           GPUs: 1
19:53:34:          GPU 0: Bus:1 Slot:0 Func:0 AMD:5 Hawaii [Radeon R9 200/300 Series]
19:53:34:           CUDA: Not detected: Failed to open dynamic library 'nvcuda.dll': The
19:53:34:                 specified module could not be found.
19:53:34:
19:53:34:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:3110.7
19:53:34:******************************* libFAH ********************************
19:53:34:           Date: Apr 15 2020
19:53:34:           Time: 14:53:14
19:53:34:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
19:53:34:         Branch: master
19:53:34:       Compiler: Visual C++ 2008
19:53:34:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:53:34:       Platform: win32 10
19:53:34:           Bits: 32
19:53:34:           Mode: Release
19:53:34:***********************************************************************
19:53:34:<config>
19:53:34:  <!-- Folding Slots -->
19:53:34:  <slot id='0' type='CPU'/>
19:53:34:  <slot id='1' type='GPU'/>
19:53:34:</config>
19:53:40:17:127.0.0.1:New Web session
19:54:17:31:127.0.0.1:New Web session
19:54:18:33:127.0.0.1:New Web session
19:54:34:Saving configuration to config.xml
19:54:34:<config>
19:54:34:  <!-- Folding Slots -->
19:54:34:  <slot id='0' type='CPU'/>
19:54:34:  <slot id='1' type='GPU'/>
19:54:34:</config>
19:54:34:Set client configured
19:54:34:WU00:FS00:Connecting to assign1.foldingathome.org:80
19:54:34:WU00:FS00:Connecting to assign1.foldingathome.org:80
19:54:34:WU01:FS01:Connecting to assign1.foldingathome.org:80
19:54:34:WU00:FS00:Assigned to work server 150.136.14.110
19:54:34:WU00:FS00:Requesting new work unit for slot 00: READY cpu:2 from 150.136.14.110
19:54:34:WU00:FS00:Connecting to 150.136.14.110:8080
19:54:34:WU01:FS01:Assigned to work server 18.188.125.154
19:54:34:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:Hawaii [Radeon R9 200/300 Series] from 18.188.125.154
19:54:34:WU01:FS01:Connecting to 18.188.125.154:8080
19:54:34:WU00:FS00:Downloading 4.89MiB
19:54:35:WU01:FS01:Downloading 7.04MiB
19:54:35:WU00:FS00:Download complete
19:54:35:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16431 run:550 clone:3 gen:239 core:0xa7 unit:0x0000010e96880e6e5e97dbc54d927821
19:54:35:WU00:FS00:Downloading core from http://cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah
19:54:35:WU00:FS00:Connecting to cores.foldingathome.org:80
19:54:35:WU00:FS00:FahCore a7: Downloading 6.71MiB
19:54:36:WU01:FS01:Download complete
19:54:36:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13422 run:3597 clone:95 gen:2 core:0x22 unit:0x0000000212bc7d9a5f3993522f93f2f9
19:54:36:WU01:FS01:Downloading core from http://cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah
19:54:36:WU01:FS01:Connecting to cores.foldingathome.org:80
19:54:36:WU01:FS01:FahCore 22: Downloading 4.39MiB
19:54:37:WU01:FS01:FahCore 22: Download complete
19:54:37:WU01:FS01:Valid core signature
19:54:37:WU01:FS01:Unpacked 14.46MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe
19:54:37:WU01:FS01:Starting
19:54:37:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:54:37:WU01:FS01:Started FahCore on PID 4764
19:54:37:WU01:FS01:Core PID:4816
19:54:37:WU01:FS01:FahCore 0x22 started
19:54:37:WU01:FS01:0x22:*********************** Log Started 2020-08-23T19:54:37Z ***********************
19:54:37:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:54:37:WU01:FS01:0x22:       Core: Core22
19:54:37:WU01:FS01:0x22:       Type: 0x22
19:54:37:WU01:FS01:0x22:    Version: 0.0.11
19:54:37:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:54:37:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
19:54:37:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
19:54:37:WU01:FS01:0x22:       Date: Jun 26 2020
19:54:37:WU01:FS01:0x22:       Time: 19:49:16
19:54:37:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
19:54:37:WU01:FS01:0x22:     Branch: core22-0.0.11
19:54:37:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:54:37:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:54:37:WU01:FS01:0x22:   Platform: win32 10
19:54:37:WU01:FS01:0x22:       Bits: 64
19:54:37:WU01:FS01:0x22:       Mode: Release
19:54:37:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:54:37:WU01:FS01:0x22:             <peastman@stanford.edu>
19:54:37:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 4764 -checkpoint 15
19:54:37:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:54:37:WU01:FS01:0x22:************************************ libFAH ************************************
19:54:37:WU01:FS01:0x22:       Date: Jun 26 2020
19:54:37:WU01:FS01:0x22:       Time: 19:47:12
19:54:37:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
19:54:37:WU01:FS01:0x22:     Branch: HEAD
19:54:37:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:54:37:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:54:37:WU01:FS01:0x22:   Platform: win32 10
19:54:37:WU01:FS01:0x22:       Bits: 64
19:54:37:WU01:FS01:0x22:       Mode: Release
19:54:37:WU01:FS01:0x22:************************************ CBang *************************************
19:54:37:WU01:FS01:0x22:       Date: Jun 26 2020
19:54:37:WU01:FS01:0x22:       Time: 19:46:11
19:54:37:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
19:54:37:WU01:FS01:0x22:     Branch: master
19:54:37:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:54:37:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:54:37:WU01:FS01:0x22:   Platform: win32 10
19:54:37:WU01:FS01:0x22:       Bits: 64
19:54:37:WU01:FS01:0x22:       Mode: Release
19:54:37:WU01:FS01:0x22:************************************ System ************************************
19:54:37:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
19:54:37:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:54:37:WU01:FS01:0x22:       CPUs: 4
19:54:37:WU01:FS01:0x22:     Memory: 14.97GiB
19:54:37:WU01:FS01:0x22:Free Memory: 12.60GiB
19:54:37:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
19:54:37:WU01:FS01:0x22: OS Version: 6.2
19:54:37:WU01:FS01:0x22:Has Battery: false
19:54:37:WU01:FS01:0x22: On Battery: false
19:54:37:WU01:FS01:0x22: UTC Offset: -5
19:54:37:WU01:FS01:0x22:        PID: 4816
19:54:37:WU01:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
19:54:37:WU01:FS01:0x22:********************************************************************************
19:54:37:WU01:FS01:0x22:Project: 13422 (Run 3597, Clone 95, Gen 2)
19:54:37:WU01:FS01:0x22:Unit: 0x0000000212bc7d9a5f3993522f93f2f9
19:54:37:WU01:FS01:0x22:Reading tar file core.xml
19:54:37:WU01:FS01:0x22:Reading tar file integrator.xml
19:54:37:WU01:FS01:0x22:Reading tar file state.xml.bz2
19:54:37:WU01:FS01:0x22:Reading tar file system.xml.bz2
19:54:37:WU01:FS01:0x22:Digital signatures verified
19:54:37:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:54:37:WU01:FS01:0x22:Version 0.0.11
19:54:37:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:54:37:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:54:37:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
19:54:37:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
19:54:41:WU00:FS00:FahCore a7: 79.13%
19:54:41:WU00:FS00:FahCore a7: Download complete
19:54:41:WU00:FS00:Valid core signature
19:54:41:WU00:FS00:Unpacked 19.85MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe
19:54:41:WU00:FS00:Unpacked 2.64MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/libfftw3f-3.dll
19:54:41:WU00:FS00:Starting
19:54:41:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -np 2
19:54:41:WU00:FS00:Started FahCore on PID 6720
19:54:42:WU00:FS00:Core PID:2548
19:54:42:WU00:FS00:FahCore 0xa7 started
19:54:42:WU00:FS00:0xa7:*********************** Log Started 2020-08-23T19:54:42Z ***********************
19:54:42:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
19:54:42:WU00:FS00:0xa7:       Type: 0xa7
19:54:42:WU00:FS00:0xa7:       Core: Gromacs
19:54:42:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 6720 -checkpoint 15 -np 2
19:54:42:WU00:FS00:0xa7:************************************ CBang *************************************
19:54:42:WU00:FS00:0xa7:       Date: Nov 27 2019
19:54:42:WU00:FS00:0xa7:       Time: 03:40:09
19:54:42:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
19:54:42:WU00:FS00:0xa7:     Branch: master
19:54:42:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
19:54:42:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:54:42:WU00:FS00:0xa7:   Platform: win32 10
19:54:42:WU00:FS00:0xa7:       Bits: 64
19:54:42:WU00:FS00:0xa7:       Mode: Release
19:54:42:WU00:FS00:0xa7:************************************ System ************************************
19:54:42:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
19:54:42:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:54:42:WU00:FS00:0xa7:       CPUs: 4
19:54:42:WU00:FS00:0xa7:     Memory: 14.97GiB
19:54:42:WU00:FS00:0xa7:Free Memory: 12.14GiB
19:54:42:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
19:54:42:WU00:FS00:0xa7: OS Version: 6.2
19:54:42:WU00:FS00:0xa7:Has Battery: false
19:54:42:WU00:FS00:0xa7: On Battery: false
19:54:42:WU00:FS00:0xa7: UTC Offset: -5
19:54:42:WU00:FS00:0xa7:        PID: 2548
19:54:42:WU00:FS00:0xa7:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
19:54:42:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
19:54:42:WU00:FS00:0xa7:    Version: 0.0.19
19:54:42:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:54:42:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
19:54:42:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
19:54:42:WU00:FS00:0xa7:       Date: Nov 25 2019
19:54:42:WU00:FS00:0xa7:       Time: 17:12:41
19:54:42:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
19:54:42:WU00:FS00:0xa7:     Branch: master
19:54:42:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
19:54:42:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:54:42:WU00:FS00:0xa7:   Platform: win32 10
19:54:42:WU00:FS00:0xa7:       Bits: 64
19:54:42:WU00:FS00:0xa7:       Mode: Release
19:54:42:WU00:FS00:0xa7:************************************ Build *************************************
19:54:42:WU00:FS00:0xa7:       SIMD: avx_256
19:54:42:WU00:FS00:0xa7:********************************************************************************
19:54:42:WU00:FS00:0xa7:Project: 16431 (Run 550, Clone 3, Gen 239)
19:54:42:WU00:FS00:0xa7:Unit: 0x0000010e96880e6e5e97dbc54d927821
19:54:42:WU00:FS00:0xa7:Reading tar file core.xml
19:54:42:WU00:FS00:0xa7:Reading tar file frame239.tpr
19:54:42:WU00:FS00:0xa7:Digital signatures verified
19:54:42:WU00:FS00:0xa7:Calling: mdrun -s frame239.tpr -o frame239.trr -x frame239.xtc -cpt 15 -nt 2
19:54:42:WU00:FS00:0xa7:Steps: first=59750000 total=250000
19:54:43:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
19:54:54:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
19:55:35:Saving configuration to config.xml
19:55:35:<config>
19:55:35:  <!-- User Information -->
19:55:35:  <passkey v='*****'/>
19:55:35:  <team v='12072'/>
19:55:35:  <user v='crimson1077'/>
19:55:35:
19:55:35:  <!-- Folding Slots -->
19:55:35:  <slot id='0' type='CPU'/>
19:55:35:  <slot id='1' type='GPU'/>
19:55:35:</config>
19:56:55:42:127.0.0.1:New Web session
19:57:04:54:127.0.0.1:New Web session
19:57:06:61:127.0.0.1:New Web session
19:58:23:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
19:58:24:WU01:FS01:0x22:An exception occurred at step 10039: Particle coordinate is nan
19:58:24:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
19:58:24:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
19:58:24:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
19:58:24:WU01:FS01:Starting
19:58:24:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:58:24:WU01:FS01:Started FahCore on PID 3244
19:58:24:WU01:FS01:Core PID:4392
19:58:24:WU01:FS01:FahCore 0x22 started
19:58:25:WU01:FS01:0x22:*********************** Log Started 2020-08-23T19:58:24Z ***********************
19:58:25:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:58:25:WU01:FS01:0x22:       Core: Core22
19:58:25:WU01:FS01:0x22:       Type: 0x22
19:58:25:WU01:FS01:0x22:    Version: 0.0.11
19:58:25:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:58:25:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
19:58:25:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
19:58:25:WU01:FS01:0x22:       Date: Jun 26 2020
19:58:25:WU01:FS01:0x22:       Time: 19:49:16
19:58:25:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
19:58:25:WU01:FS01:0x22:     Branch: core22-0.0.11
19:58:25:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:58:25:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:58:25:WU01:FS01:0x22:   Platform: win32 10
19:58:25:WU01:FS01:0x22:       Bits: 64
19:58:25:WU01:FS01:0x22:       Mode: Release
19:58:25:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:58:25:WU01:FS01:0x22:             <peastman@stanford.edu>
19:58:25:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 3244 -checkpoint 15
19:58:25:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
19:58:25:WU01:FS01:0x22:************************************ libFAH ************************************
19:58:25:WU01:FS01:0x22:       Date: Jun 26 2020
19:58:25:WU01:FS01:0x22:       Time: 19:47:12
19:58:25:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
19:58:25:WU01:FS01:0x22:     Branch: HEAD
19:58:25:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:58:25:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:58:25:WU01:FS01:0x22:   Platform: win32 10
19:58:25:WU01:FS01:0x22:       Bits: 64
19:58:25:WU01:FS01:0x22:       Mode: Release
19:58:25:WU01:FS01:0x22:************************************ CBang *************************************
19:58:25:WU01:FS01:0x22:       Date: Jun 26 2020
19:58:25:WU01:FS01:0x22:       Time: 19:46:11
19:58:25:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
19:58:25:WU01:FS01:0x22:     Branch: master
19:58:25:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:58:25:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:58:25:WU01:FS01:0x22:   Platform: win32 10
19:58:25:WU01:FS01:0x22:       Bits: 64
19:58:25:WU01:FS01:0x22:       Mode: Release
19:58:25:WU01:FS01:0x22:************************************ System ************************************
19:58:25:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
19:58:25:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:58:25:WU01:FS01:0x22:       CPUs: 4
19:58:25:WU01:FS01:0x22:     Memory: 14.97GiB
19:58:25:WU01:FS01:0x22:Free Memory: 12.23GiB
19:58:25:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
19:58:25:WU01:FS01:0x22: OS Version: 6.2
19:58:25:WU01:FS01:0x22:Has Battery: false
19:58:25:WU01:FS01:0x22: On Battery: false
19:58:25:WU01:FS01:0x22: UTC Offset: -5
19:58:25:WU01:FS01:0x22:        PID: 4392
19:58:25:WU01:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
19:58:25:WU01:FS01:0x22:********************************************************************************
19:58:25:WU01:FS01:0x22:Project: 13422 (Run 3597, Clone 95, Gen 2)
19:58:25:WU01:FS01:0x22:Unit: 0x0000000212bc7d9a5f3993522f93f2f9
19:58:25:WU01:FS01:0x22:Digital signatures verified
19:58:25:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:58:25:WU01:FS01:0x22:Version 0.0.11
19:58:25:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:58:25:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:58:25:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
19:58:25:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
19:58:42:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
20:00:30:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
20:00:51:71:127.0.0.1:New Web session
20:01:50:79:127.0.0.1:New Web session
20:02:06:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
20:03:45:88:127.0.0.1:New Web session
20:05:30:WU01:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
20:06:12:WU00:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
20:08:54:WU01:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
20:10:23:WU01:FS01:0x22:An exception occurred at step 32629: Particle coordinate is nan
20:10:23:WU01:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
20:10:23:WU01:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
20:10:23:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
20:10:23:WU01:FS01:0x22:Saving result file globals.csv
20:10:23:WU01:FS01:0x22:Saving result file science.log
20:10:23:WU01:FS01:0x22:Saving result file state.xml.bz2
20:10:23:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
20:10:24:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
20:10:24:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:3597 clone:95 gen:2 core:0x22 unit:0x0000000212bc7d9a5f3993522f93f2f9
20:10:24:WU01:FS01:Uploading 5.61MiB to 18.188.125.154
20:10:24:WU01:FS01:Connecting to 18.188.125.154:8080
20:10:24:WU02:FS01:Connecting to assign1.foldingathome.org:80
20:10:25:WU02:FS01:Assigned to work server 18.188.125.154
20:10:25:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:Hawaii [Radeon R9 200/300 Series] from 18.188.125.154
20:10:25:WU02:FS01:Connecting to 18.188.125.154:8080
20:10:25:WU02:FS01:Downloading 7.04MiB
20:10:26:WU02:FS01:Download complete
20:10:26:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13422 run:3560 clone:92 gen:2 core:0x22 unit:0x0000000212bc7d9a5f39934df412be70
20:10:26:WU02:FS01:Starting
20:10:26:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:10:26:WU02:FS01:Started FahCore on PID 2464
20:10:26:WU02:FS01:Core PID:4904
20:10:26:WU02:FS01:FahCore 0x22 started
20:10:26:WU02:FS01:0x22:*********************** Log Started 2020-08-23T20:10:26Z ***********************
20:10:26:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
20:10:26:WU02:FS01:0x22:       Core: Core22
20:10:26:WU02:FS01:0x22:       Type: 0x22
20:10:26:WU02:FS01:0x22:    Version: 0.0.11
20:10:26:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:10:26:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
20:10:26:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
20:10:26:WU02:FS01:0x22:       Date: Jun 26 2020
20:10:26:WU02:FS01:0x22:       Time: 19:49:16
20:10:26:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
20:10:26:WU02:FS01:0x22:     Branch: core22-0.0.11
20:10:26:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:10:26:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:10:26:WU02:FS01:0x22:   Platform: win32 10
20:10:26:WU02:FS01:0x22:       Bits: 64
20:10:26:WU02:FS01:0x22:       Mode: Release
20:10:26:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
20:10:26:WU02:FS01:0x22:             <peastman@stanford.edu>
20:10:26:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 2464 -checkpoint 15
20:10:26:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:10:26:WU02:FS01:0x22:************************************ libFAH ************************************
20:10:26:WU02:FS01:0x22:       Date: Jun 26 2020
20:10:26:WU02:FS01:0x22:       Time: 19:47:12
20:10:26:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
20:10:26:WU02:FS01:0x22:     Branch: HEAD
20:10:26:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:10:26:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:10:26:WU02:FS01:0x22:   Platform: win32 10
20:10:26:WU02:FS01:0x22:       Bits: 64
20:10:26:WU02:FS01:0x22:       Mode: Release
20:10:26:WU02:FS01:0x22:************************************ CBang *************************************
20:10:26:WU02:FS01:0x22:       Date: Jun 26 2020
20:10:26:WU02:FS01:0x22:       Time: 19:46:11
20:10:26:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
20:10:27:WU02:FS01:0x22:     Branch: master
20:10:27:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:10:27:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:10:27:WU02:FS01:0x22:   Platform: win32 10
20:10:27:WU02:FS01:0x22:       Bits: 64
20:10:27:WU02:FS01:0x22:       Mode: Release
20:10:27:WU02:FS01:0x22:************************************ System ************************************
20:10:27:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
20:10:27:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
20:10:27:WU02:FS01:0x22:       CPUs: 4
20:10:27:WU02:FS01:0x22:     Memory: 14.97GiB
20:10:27:WU02:FS01:0x22:Free Memory: 12.20GiB
20:10:27:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
20:10:27:WU02:FS01:0x22: OS Version: 6.2
20:10:27:WU02:FS01:0x22:Has Battery: false
20:10:27:WU02:FS01:0x22: On Battery: false
20:10:27:WU02:FS01:0x22: UTC Offset: -5
20:10:27:WU02:FS01:0x22:        PID: 4904
20:10:27:WU02:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
20:10:27:WU02:FS01:0x22:********************************************************************************
20:10:27:WU02:FS01:0x22:Project: 13422 (Run 3560, Clone 92, Gen 2)
20:10:27:WU02:FS01:0x22:Unit: 0x0000000212bc7d9a5f39934df412be70
20:10:27:WU02:FS01:0x22:Reading tar file core.xml
20:10:27:WU02:FS01:0x22:Reading tar file integrator.xml
20:10:27:WU02:FS01:0x22:Reading tar file state.xml.bz2
20:10:27:WU02:FS01:0x22:Reading tar file system.xml.bz2
20:10:27:WU02:FS01:0x22:Digital signatures verified
20:10:27:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
20:10:27:WU02:FS01:0x22:Version 0.0.11
20:10:27:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
20:10:27:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
20:10:27:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
20:10:27:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
20:10:30:WU01:FS01:Upload 85.73%
20:10:30:WU01:FS01:Upload complete
20:10:30:WU01:FS01:Server responded WORK_ACK (400)
20:10:30:WU01:FS01:Cleaning up
20:10:43:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
20:12:01:WU00:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
20:14:08:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
20:16:25:FS01:Shutting core down
20:16:26:WU02:FS01:0x22:WARNING:Console control signal 1 on PID 4904
20:16:26:WU02:FS01:0x22:Exiting, please wait. . .
20:16:26:WU02:FS01:0x22:Folding@home Core Shutdown: INTERRUPTED
20:16:26:WU02:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
20:16:56:Saving configuration to config.xml
20:16:56:<config>
20:16:56:  <!-- User Information -->
20:16:56:  <passkey v='*****'/>
20:16:56:  <team v='12072'/>
20:16:56:  <user v='crimson1077'/>
20:16:56:
20:16:56:  <!-- Folding Slots -->
20:16:56:  <slot id='0' type='CPU'/>
20:16:56:  <slot id='1' type='GPU'>
20:16:56:    <idle v='true'/>
20:16:56:  </slot>
20:16:56:</config>
20:16:58:WU02:FS01:Starting
20:16:58:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:16:58:WU02:FS01:Started FahCore on PID 1720
20:16:58:WU02:FS01:Core PID:1776
20:16:58:WU02:FS01:FahCore 0x22 started
20:16:59:WU02:FS01:0x22:*********************** Log Started 2020-08-23T20:16:58Z ***********************
20:16:59:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
20:16:59:WU02:FS01:0x22:       Core: Core22
20:16:59:WU02:FS01:0x22:       Type: 0x22
20:16:59:WU02:FS01:0x22:    Version: 0.0.11
20:16:59:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:16:59:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
20:16:59:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
20:16:59:WU02:FS01:0x22:       Date: Jun 26 2020
20:16:59:WU02:FS01:0x22:       Time: 19:49:16
20:16:59:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
20:16:59:WU02:FS01:0x22:     Branch: core22-0.0.11
20:16:59:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:16:59:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:16:59:WU02:FS01:0x22:   Platform: win32 10
20:16:59:WU02:FS01:0x22:       Bits: 64
20:16:59:WU02:FS01:0x22:       Mode: Release
20:16:59:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
20:16:59:WU02:FS01:0x22:             <peastman@stanford.edu>
20:16:59:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 1720 -checkpoint 15
20:16:59:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:16:59:WU02:FS01:0x22:************************************ libFAH ************************************
20:16:59:WU02:FS01:0x22:       Date: Jun 26 2020
20:16:59:WU02:FS01:0x22:       Time: 19:47:12
20:16:59:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
20:16:59:WU02:FS01:0x22:     Branch: HEAD
20:16:59:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:16:59:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:16:59:WU02:FS01:0x22:   Platform: win32 10
20:16:59:WU02:FS01:0x22:       Bits: 64
20:16:59:WU02:FS01:0x22:       Mode: Release
20:16:59:WU02:FS01:0x22:************************************ CBang *************************************
20:16:59:WU02:FS01:0x22:       Date: Jun 26 2020
20:16:59:WU02:FS01:0x22:       Time: 19:46:11
20:16:59:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
20:16:59:WU02:FS01:0x22:     Branch: master
20:16:59:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:16:59:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:16:59:WU02:FS01:0x22:   Platform: win32 10
20:16:59:WU02:FS01:0x22:       Bits: 64
20:16:59:WU02:FS01:0x22:       Mode: Release
20:16:59:WU02:FS01:0x22:************************************ System ************************************
20:16:59:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
20:16:59:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
20:16:59:WU02:FS01:0x22:       CPUs: 4
20:16:59:WU02:FS01:0x22:     Memory: 14.97GiB
20:16:59:WU02:FS01:0x22:Free Memory: 12.29GiB
20:16:59:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
20:16:59:WU02:FS01:0x22: OS Version: 6.2
20:16:59:WU02:FS01:0x22:Has Battery: false
20:16:59:WU02:FS01:0x22: On Battery: false
20:16:59:WU02:FS01:0x22: UTC Offset: -5
20:16:59:WU02:FS01:0x22:        PID: 1776
20:16:59:WU02:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
20:16:59:WU02:FS01:0x22:********************************************************************************
20:16:59:WU02:FS01:0x22:Project: 13422 (Run 3560, Clone 92, Gen 2)
20:16:59:WU02:FS01:0x22:Unit: 0x0000000212bc7d9a5f39934df412be70
20:16:59:WU02:FS01:0x22:Digital signatures verified
20:16:59:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
20:16:59:WU02:FS01:0x22:Version 0.0.11
20:16:59:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
20:16:59:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
20:16:59:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
20:16:59:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
20:17:16:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
20:17:32:WU02:FS01:0x22:An exception occurred at step 752: Particle coordinate is nan
20:17:33:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
20:17:33:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
20:17:33:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
20:17:37:WU00:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
20:17:57:Saving configuration to config.xml
20:17:57:<config>
20:17:57:  <!-- User Information -->
20:17:57:  <passkey v='*****'/>
20:17:57:  <team v='12072'/>
20:17:57:  <user v='crimson1077'/>
20:17:57:
20:17:57:  <!-- Folding Slots -->
20:17:57:  <slot id='0' type='CPU'/>
20:17:57:  <slot id='1' type='GPU'/>
20:17:57:</config>
20:17:58:WU02:FS01:Starting
20:17:58:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:17:58:WU02:FS01:Started FahCore on PID 5372
20:17:58:WU02:FS01:Core PID:3776
20:17:58:WU02:FS01:FahCore 0x22 started
20:17:59:WU02:FS01:0x22:*********************** Log Started 2020-08-23T20:17:58Z ***********************
20:17:59:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
20:17:59:WU02:FS01:0x22:       Core: Core22
20:17:59:WU02:FS01:0x22:       Type: 0x22
20:17:59:WU02:FS01:0x22:    Version: 0.0.11
20:17:59:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:17:59:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
20:17:59:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
20:17:59:WU02:FS01:0x22:       Date: Jun 26 2020
20:17:59:WU02:FS01:0x22:       Time: 19:49:16
20:17:59:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
20:17:59:WU02:FS01:0x22:     Branch: core22-0.0.11
20:17:59:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:17:59:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:17:59:WU02:FS01:0x22:   Platform: win32 10
20:17:59:WU02:FS01:0x22:       Bits: 64
20:17:59:WU02:FS01:0x22:       Mode: Release
20:17:59:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
20:17:59:WU02:FS01:0x22:             <peastman@stanford.edu>
20:17:59:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 5372 -checkpoint 15
20:17:59:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:17:59:WU02:FS01:0x22:************************************ libFAH ************************************
20:17:59:WU02:FS01:0x22:       Date: Jun 26 2020
20:17:59:WU02:FS01:0x22:       Time: 19:47:12
20:17:59:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
20:17:59:WU02:FS01:0x22:     Branch: HEAD
20:17:59:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:17:59:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:17:59:WU02:FS01:0x22:   Platform: win32 10
20:17:59:WU02:FS01:0x22:       Bits: 64
20:17:59:WU02:FS01:0x22:       Mode: Release
20:17:59:WU02:FS01:0x22:************************************ CBang *************************************
20:17:59:WU02:FS01:0x22:       Date: Jun 26 2020
20:17:59:WU02:FS01:0x22:       Time: 19:46:11
20:17:59:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
20:17:59:WU02:FS01:0x22:     Branch: master
20:17:59:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:17:59:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:17:59:WU02:FS01:0x22:   Platform: win32 10
20:17:59:WU02:FS01:0x22:       Bits: 64
20:17:59:WU02:FS01:0x22:       Mode: Release
20:17:59:WU02:FS01:0x22:************************************ System ************************************
20:17:59:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
20:17:59:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
20:17:59:WU02:FS01:0x22:       CPUs: 4
20:17:59:WU02:FS01:0x22:     Memory: 14.97GiB
20:17:59:WU02:FS01:0x22:Free Memory: 12.28GiB
20:17:59:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
20:17:59:WU02:FS01:0x22: OS Version: 6.2
20:17:59:WU02:FS01:0x22:Has Battery: false
20:17:59:WU02:FS01:0x22: On Battery: false
20:17:59:WU02:FS01:0x22: UTC Offset: -5
20:17:59:WU02:FS01:0x22:        PID: 3776
20:17:59:WU02:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
20:17:59:WU02:FS01:0x22:********************************************************************************
20:17:59:WU02:FS01:0x22:Project: 13422 (Run 3560, Clone 92, Gen 2)
20:17:59:WU02:FS01:0x22:Unit: 0x0000000212bc7d9a5f39934df412be70
20:17:59:WU02:FS01:0x22:Digital signatures verified
20:17:59:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
20:17:59:WU02:FS01:0x22:Version 0.0.11
20:17:59:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
20:17:59:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
20:17:59:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
20:17:59:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
20:18:16:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
20:19:38:ERROR:Receive error: 10054: An existing connection was forcibly closed by the remote host.
20:21:39:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
20:23:16:WU00:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
20:25:01:WU02:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
20:28:07:Saving configuration to config.xml
20:28:07:<config>
20:28:07:  <!-- Network -->
20:28:07:  <proxy v=':8080'/>
20:28:07:
20:28:07:  <!-- User Information -->
20:28:07:  <passkey v='*****'/>
20:28:07:  <team v='12072'/>
20:28:07:  <user v='crimson1077'/>
20:28:07:
20:28:07:  <!-- Folding Slots -->
20:28:07:  <slot id='0' type='CPU'/>
20:28:07:  <slot id='1' type='GPU'/>
20:28:07:</config>
20:28:07:WU02:FS01:0x22:An exception occurred at step 29115: Particle coordinate is nan
20:28:07:WU02:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
20:28:07:WU02:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
20:28:07:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
20:28:07:WU02:FS01:0x22:Saving result file globals.csv
20:28:07:WU02:FS01:0x22:Saving result file science.log
20:28:07:WU02:FS01:0x22:Saving result file state.xml.bz2
20:28:07:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
20:28:07:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
20:28:08:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13422 run:3560 clone:92 gen:2 core:0x22 unit:0x0000000212bc7d9a5f39934df412be70
20:28:08:WU02:FS01:Uploading 5.62MiB to 18.188.125.154
20:28:08:WU02:FS01:Connecting to 18.188.125.154:8080
20:28:08:WU01:FS01:Connecting to assign1.foldingathome.org:80
20:28:08:WU01:FS01:Assigned to work server 18.188.125.154
20:28:08:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:Hawaii [Radeon R9 200/300 Series] from 18.188.125.154
20:28:08:WU01:FS01:Connecting to 18.188.125.154:8080
20:28:08:WU01:FS01:Downloading 7.04MiB
20:28:09:WU01:FS01:Download complete
20:28:09:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13422 run:3639 clone:50 gen:1 core:0x22 unit:0x0000000212bc7d9a5f39934fd990450c
20:28:09:WU01:FS01:Starting
20:28:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:28:09:WU01:FS01:Started FahCore on PID 2112
20:28:09:WU01:FS01:Core PID:1392
20:28:09:WU01:FS01:FahCore 0x22 started
20:28:10:WU01:FS01:0x22:*********************** Log Started 2020-08-23T20:28:09Z ***********************
20:28:10:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
20:28:10:WU01:FS01:0x22:       Core: Core22
20:28:10:WU01:FS01:0x22:       Type: 0x22
20:28:10:WU01:FS01:0x22:    Version: 0.0.11
20:28:10:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:28:10:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
20:28:10:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
20:28:10:WU01:FS01:0x22:       Date: Jun 26 2020
20:28:10:WU01:FS01:0x22:       Time: 19:49:16
20:28:10:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
20:28:10:WU01:FS01:0x22:     Branch: core22-0.0.11
20:28:10:WU01:FS01:0x22:   Compiler: Visual C++ 2015
20:28:10:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:28:10:WU01:FS01:0x22:   Platform: win32 10
20:28:10:WU01:FS01:0x22:       Bits: 64
20:28:10:WU01:FS01:0x22:       Mode: Release
20:28:10:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
20:28:10:WU01:FS01:0x22:             <peastman@stanford.edu>
20:28:10:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 2112 -checkpoint 15
20:28:10:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:28:10:WU01:FS01:0x22:************************************ libFAH ************************************
20:28:10:WU01:FS01:0x22:       Date: Jun 26 2020
20:28:10:WU01:FS01:0x22:       Time: 19:47:12
20:28:10:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
20:28:10:WU01:FS01:0x22:     Branch: HEAD
20:28:10:WU01:FS01:0x22:   Compiler: Visual C++ 2015
20:28:10:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:28:10:WU01:FS01:0x22:   Platform: win32 10
20:28:10:WU01:FS01:0x22:       Bits: 64
20:28:10:WU01:FS01:0x22:       Mode: Release
20:28:10:WU01:FS01:0x22:************************************ CBang *************************************
20:28:10:WU01:FS01:0x22:       Date: Jun 26 2020
20:28:10:WU01:FS01:0x22:       Time: 19:46:11
20:28:10:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
20:28:10:WU01:FS01:0x22:     Branch: master
20:28:10:WU01:FS01:0x22:   Compiler: Visual C++ 2015
20:28:10:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:28:10:WU01:FS01:0x22:   Platform: win32 10
20:28:10:WU01:FS01:0x22:       Bits: 64
20:28:10:WU01:FS01:0x22:       Mode: Release
20:28:10:WU01:FS01:0x22:************************************ System ************************************
20:28:10:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
20:28:10:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
20:28:10:WU01:FS01:0x22:       CPUs: 4
20:28:10:WU01:FS01:0x22:     Memory: 14.97GiB
20:28:10:WU01:FS01:0x22:Free Memory: 12.21GiB
20:28:10:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
20:28:10:WU01:FS01:0x22: OS Version: 6.2
20:28:10:WU01:FS01:0x22:Has Battery: false
20:28:10:WU01:FS01:0x22: On Battery: false
20:28:10:WU01:FS01:0x22: UTC Offset: -5
20:28:10:WU01:FS01:0x22:        PID: 1392
20:28:10:WU01:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
20:28:10:WU01:FS01:0x22:********************************************************************************
20:28:10:WU01:FS01:0x22:Project: 13422 (Run 3639, Clone 50, Gen 1)
20:28:10:WU01:FS01:0x22:Unit: 0x0000000212bc7d9a5f39934fd990450c
20:28:10:WU01:FS01:0x22:Reading tar file core.xml
20:28:10:WU01:FS01:0x22:Reading tar file integrator.xml
20:28:10:WU01:FS01:0x22:Reading tar file state.xml.bz2
20:28:10:WU01:FS01:0x22:Reading tar file system.xml.bz2
20:28:10:WU01:FS01:0x22:Digital signatures verified
20:28:10:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
20:28:10:WU01:FS01:0x22:Version 0.0.11
20:28:10:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
20:28:10:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
20:28:10:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
20:28:10:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
20:28:13:WU02:FS01:Upload complete
20:28:13:WU02:FS01:Server responded WORK_ACK (400)
20:28:13:WU02:FS01:Cleaning up
20:28:27:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
20:28:55:WU00:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
20:31:51:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
20:34:29:WU00:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
20:35:14:WU01:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
20:37:53:WU01:FS01:0x22:An exception occurred at step 27860: Particle coordinate is nan
20:37:53:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
20:37:53:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
20:37:54:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
20:37:54:WU01:FS01:Starting
20:37:54:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:37:54:WU01:FS01:Started FahCore on PID 3256
20:37:54:WU01:FS01:Core PID:1820
20:37:54:WU01:FS01:FahCore 0x22 started
20:37:54:WU01:FS01:0x22:*********************** Log Started 2020-08-23T20:37:54Z ***********************
20:37:54:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
20:37:54:WU01:FS01:0x22:       Core: Core22
20:37:54:WU01:FS01:0x22:       Type: 0x22
20:37:54:WU01:FS01:0x22:    Version: 0.0.11
20:37:54:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:37:54:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
20:37:54:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
20:37:54:WU01:FS01:0x22:       Date: Jun 26 2020
20:37:54:WU01:FS01:0x22:       Time: 19:49:16
20:37:54:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
20:37:54:WU01:FS01:0x22:     Branch: core22-0.0.11
20:37:54:WU01:FS01:0x22:   Compiler: Visual C++ 2015
20:37:54:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:37:55:WU01:FS01:0x22:   Platform: win32 10
20:37:55:WU01:FS01:0x22:       Bits: 64
20:37:55:WU01:FS01:0x22:       Mode: Release
20:37:55:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
20:37:55:WU01:FS01:0x22:             <peastman@stanford.edu>
20:37:55:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 3256 -checkpoint 15
20:37:55:WU01:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:37:55:WU01:FS01:0x22:************************************ libFAH ************************************
20:37:55:WU01:FS01:0x22:       Date: Jun 26 2020
20:37:55:WU01:FS01:0x22:       Time: 19:47:12
20:37:55:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
20:37:55:WU01:FS01:0x22:     Branch: HEAD
20:37:55:WU01:FS01:0x22:   Compiler: Visual C++ 2015
20:37:55:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:37:55:WU01:FS01:0x22:   Platform: win32 10
20:37:55:WU01:FS01:0x22:       Bits: 64
20:37:55:WU01:FS01:0x22:       Mode: Release
20:37:55:WU01:FS01:0x22:************************************ CBang *************************************
20:37:55:WU01:FS01:0x22:       Date: Jun 26 2020
20:37:55:WU01:FS01:0x22:       Time: 19:46:11
20:37:55:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
20:37:55:WU01:FS01:0x22:     Branch: master
20:37:55:WU01:FS01:0x22:   Compiler: Visual C++ 2015
20:37:55:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:37:55:WU01:FS01:0x22:   Platform: win32 10
20:37:55:WU01:FS01:0x22:       Bits: 64
20:37:55:WU01:FS01:0x22:       Mode: Release
20:37:55:WU01:FS01:0x22:************************************ System ************************************
20:37:55:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
20:37:55:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
20:37:55:WU01:FS01:0x22:       CPUs: 4
20:37:55:WU01:FS01:0x22:     Memory: 14.97GiB
20:37:55:WU01:FS01:0x22:Free Memory: 12.34GiB
20:37:55:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
20:37:55:WU01:FS01:0x22: OS Version: 6.2
20:37:55:WU01:FS01:0x22:Has Battery: false
20:37:55:WU01:FS01:0x22: On Battery: false
20:37:55:WU01:FS01:0x22: UTC Offset: -5
20:37:55:WU01:FS01:0x22:        PID: 1820
20:37:55:WU01:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
20:37:55:WU01:FS01:0x22:********************************************************************************
20:37:55:WU01:FS01:0x22:Project: 13422 (Run 3639, Clone 50, Gen 1)
20:37:55:WU01:FS01:0x22:Unit: 0x0000000212bc7d9a5f39934fd990450c
20:37:55:WU01:FS01:0x22:Digital signatures verified
20:37:55:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
20:37:55:WU01:FS01:0x22:Version 0.0.11
20:37:55:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
20:37:55:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
20:37:55:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
20:37:55:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
20:38:11:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
20:38:37:WU01:FS01:0x22:An exception occurred at step 1254: Particle coordinate is nan
20:38:37:WU01:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
20:38:37:WU01:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
20:38:37:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
20:38:37:WU01:FS01:0x22:Saving result file globals.csv
20:38:37:WU01:FS01:0x22:Saving result file science.log
20:38:37:WU01:FS01:0x22:Saving result file state.xml.bz2
20:38:37:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
20:38:38:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
20:38:38:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:13422 run:3639 clone:50 gen:1 core:0x22 unit:0x0000000212bc7d9a5f39934fd990450c
20:38:38:WU01:FS01:Uploading 5.61MiB to 18.188.125.154
20:38:38:WU01:FS01:Connecting to 18.188.125.154:8080
20:38:38:WU02:FS01:Connecting to assign1.foldingathome.org:80
20:38:39:WU02:FS01:Assigned to work server 18.188.125.154
20:38:39:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:Hawaii [Radeon R9 200/300 Series] from 18.188.125.154
20:38:39:WU02:FS01:Connecting to 18.188.125.154:8080
20:38:39:WU02:FS01:Downloading 7.04MiB
20:38:40:WU02:FS01:Download complete
20:38:40:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13422 run:3631 clone:7 gen:1 core:0x22 unit:0x0000000112bc7d9a5f39935267cd579d
20:38:40:WU02:FS01:Starting
20:38:40:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Crimson\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6488 -checkpoint 15 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:38:40:WU02:FS01:Started FahCore on PID 244
20:38:40:WU02:FS01:Core PID:2212
20:38:40:WU02:FS01:FahCore 0x22 started
20:38:40:WU02:FS01:0x22:*********************** Log Started 2020-08-23T20:38:40Z ***********************
20:38:40:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
20:38:40:WU02:FS01:0x22:       Core: Core22
20:38:40:WU02:FS01:0x22:       Type: 0x22
20:38:40:WU02:FS01:0x22:    Version: 0.0.11
20:38:40:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:38:40:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
20:38:40:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
20:38:40:WU02:FS01:0x22:       Date: Jun 26 2020
20:38:40:WU02:FS01:0x22:       Time: 19:49:16
20:38:40:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
20:38:40:WU02:FS01:0x22:     Branch: core22-0.0.11
20:38:40:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:38:40:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:38:40:WU02:FS01:0x22:   Platform: win32 10
20:38:40:WU02:FS01:0x22:       Bits: 64
20:38:40:WU02:FS01:0x22:       Mode: Release
20:38:40:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
20:38:40:WU02:FS01:0x22:             <peastman@stanford.edu>
20:38:40:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 244 -checkpoint 15
20:38:40:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
20:38:40:WU02:FS01:0x22:************************************ libFAH ************************************
20:38:40:WU02:FS01:0x22:       Date: Jun 26 2020
20:38:40:WU02:FS01:0x22:       Time: 19:47:12
20:38:40:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
20:38:40:WU02:FS01:0x22:     Branch: HEAD
20:38:40:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:38:40:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:38:40:WU02:FS01:0x22:   Platform: win32 10
20:38:40:WU02:FS01:0x22:       Bits: 64
20:38:40:WU02:FS01:0x22:       Mode: Release
20:38:40:WU02:FS01:0x22:************************************ CBang *************************************
20:38:40:WU02:FS01:0x22:       Date: Jun 26 2020
20:38:40:WU02:FS01:0x22:       Time: 19:46:11
20:38:40:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
20:38:40:WU02:FS01:0x22:     Branch: master
20:38:40:WU02:FS01:0x22:   Compiler: Visual C++ 2015
20:38:40:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
20:38:40:WU02:FS01:0x22:   Platform: win32 10
20:38:40:WU02:FS01:0x22:       Bits: 64
20:38:40:WU02:FS01:0x22:       Mode: Release
20:38:40:WU02:FS01:0x22:************************************ System ************************************
20:38:40:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
20:38:40:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
20:38:40:WU02:FS01:0x22:       CPUs: 4
20:38:40:WU02:FS01:0x22:     Memory: 14.97GiB
20:38:40:WU02:FS01:0x22:Free Memory: 12.34GiB
20:38:40:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
20:38:40:WU02:FS01:0x22: OS Version: 6.2
20:38:40:WU02:FS01:0x22:Has Battery: false
20:38:40:WU02:FS01:0x22: On Battery: false
20:38:40:WU02:FS01:0x22: UTC Offset: -5
20:38:40:WU02:FS01:0x22:        PID: 2212
20:38:40:WU02:FS01:0x22:        CWD: C:\Users\Crimson\AppData\Roaming\FAHClient\work
20:38:40:WU02:FS01:0x22:********************************************************************************
20:38:40:WU02:FS01:0x22:Project: 13422 (Run 3631, Clone 7, Gen 1)
20:38:40:WU02:FS01:0x22:Unit: 0x0000000112bc7d9a5f39935267cd579d
20:38:40:WU02:FS01:0x22:Reading tar file core.xml
20:38:40:WU02:FS01:0x22:Reading tar file integrator.xml
20:38:40:WU02:FS01:0x22:Reading tar file state.xml.bz2
20:38:40:WU02:FS01:0x22:Reading tar file system.xml.bz2
20:38:40:WU02:FS01:0x22:Digital signatures verified
20:38:40:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
20:38:40:WU02:FS01:0x22:Version 0.0.11
20:38:40:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
20:38:40:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
20:38:40:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
20:38:40:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
20:38:43:WU01:FS01:Upload complete
20:38:44:WU01:FS01:Server responded WORK_ACK (400)
20:38:44:WU01:FS01:Cleaning up
20:38:57:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
20:40:05:WU00:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
20:42:22:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
20:45:40:WU00:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
20:45:44:WU02:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
20:49:06:WU02:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
20:51:16:WU00:FS00:0xa7:Completed 25000 out of 250000 steps (10%)
20:52:27:WU02:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
20:55:28:FS01:Paused
20:55:28:FS01:Shutting core down
20:55:28:WU02:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
20:55:34:Saving configuration to config.xml
20:55:34:<config>
20:55:34:  <!-- Network -->
20:55:34:  <proxy v=':8080'/>
20:55:34:
20:55:34:  <!-- User Information -->
20:55:34:  <passkey v='*****'/>
20:55:34:  <team v='12072'/>
20:55:34:  <user v='crimson1077'/>
20:55:34:
20:55:34:  <!-- Folding Slots -->
20:55:34:  <slot id='0' type='CPU'/>
20:55:34:  <slot id='1' type='GPU'>
20:55:34:    <paused v='true'/>
20:55:34:  </slot>
20:55:34:</config>
20:55:51:FS00:Shutting core down
20:55:52:WU00:FS00:0xa7:WARNING:Console control signal 1 on PID 2548
20:55:52:WU00:FS00:0xa7:Exiting, please wait. . .
20:55:53:ERROR:Receive error: 10054: An existing connection was forcibly closed by the remote host.
20:55:54:Clean exit
Mod Edit: Added Code Tags - PantherX
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: Failed GPU slot daily :(

Post by PantherX »

The GPU Slot failed since you had 5 consecutive failures to fold a WU which would indicate a hardware/software problem instead of a bad WU.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
crimson1077
Posts: 33
Joined: Sat Aug 22, 2020 4:37 pm

Re: Failed GPU slot daily :(

Post by crimson1077 »

PantherX wrote:The GPU Slot failed since you had 5 consecutive failures to fold a WU which would indicate a hardware/software problem instead of a bad WU.
Hhm,I see. What should I do from here?
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: Failed GPU slot daily :(

Post by PantherX »

I would suggest that you pause the GPU slot for a while. If the person that bruce emailed responds, then follow their instructions :)
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
JohnChodera
Pande Group Member
Posts: 467
Joined: Fri Feb 22, 2013 9:59 pm

Re: Failed GPU slot daily :(

Post by JohnChodera »

@Crimson1077: Apologies for the difficulties encountered here. We're working on getting a debug core built that will do a lot more testing when you run into issues like the above:

Code: Select all

20:20:47:WU01:FS01:0x22:An exception occurred at step 1003: Particle coordinate is nan
Let me send a private message with a bit more debugging information

~ John Chodera // MSKCC
Top
crimson1077
Posts: 33
Joined: Sat Aug 22, 2020 4:37 pm

Re: Failed GPU slot daily :(

Post by crimson1077 »

I decided to just pump out a few WU's CPU crunching only for about 12hrs and this afternoon decided to add a GPU slot (crossed my fingers and walked away)
Now i'm at 20.42% on PRCG 16911 (1, 5, 66) (Cancer project) and my point's aren't showing stupid numbers like "670,000 point's per day" anymore! Now it's an accurate 208000 per day.

Still getting looping WU's. First project 11752 then it switches to 13422 then fails.
Last edited by crimson1077 on Wed Aug 26, 2020 8:23 am, edited 1 time in total.
Top
Foliant
Posts: 104
Joined: Wed May 13, 2020 4:39 pm
Location: Bavaria

Re: Failed GPU slot daily :(

Post by Foliant »

JohnChodera wrote:

Code: Select all

20:20:47:WU01:FS01:0x22:An exception occurred at step 1003: Particle coordinate is nan
Had that several times for the last two or three weeks on all my "slow" slots. (GTX 560Ti, 770, 760Ti, 660, 650...)
I dont remember the Project Numbers, but it was always a verry tiny download. (Under 1MB)
As it first appears i benchmarked the failing GPUs but couldnt find any weeknes.

I found out that by deleting/adding a slot -that is not the last added- sometimes the OCL information for other cards gets corrupted.
So i simply stop and start FAHClient, or reboot the System.
As my start/stop happens hours after the "Failed" i did never get the same WU and folding is just fine.

Regards
Patrick
24/7
1x i5 3470 @2Cores
1x GTX750 (GM107)
2x GTX750Ti (GM107)
Top
crimson1077
Posts: 33
Joined: Sat Aug 22, 2020 4:37 pm

Re: Failed GPU slot daily :(

Post by crimson1077 »

I'll try what you did because I got 13422 again and doing the same. I will let you know!
Top
JohnChodera
Pande Group Member
Posts: 467
Joined: Fri Feb 22, 2013 9:59 pm

Re: Failed GPU slot daily :(

Post by JohnChodera »

@crimson1077: Apologies for the persistent issues here. We're tracking a few AMD-specific issues in the OpenMM issue tracker:
https://github.com/openmm/openmm/issues/2817
https://github.com/openmm/openmm/issues/2813

For now, I've excluded GPUSpecies<6 from the 134xx projects (which includes the VEGA cards we've seen high error rates with) until we can solve this issue.

@crimson1077: I'll also PM you instructions on how to point your system toward a test project to help us gather more debug info.

Thanks so much for your patience and support!

~ John Chodera // MSKCC
Top
MeeLee
Posts: 1339
Joined: Tue Feb 19, 2019 10:16 pm

Re: Failed GPU slot daily :(

Post by MeeLee »

Perhaps not the solution to your problem, but considering the GPU is so old, I would suggest taking the cooler off, and cleaning it (brush/vacuum).
Then renew the thermal paste on them.
Thermal paste usually goes bad after about 4-5 years, it gets hard and doesn't conduct heat very well anymore.
While you're at it, perhaps also replace any thermal pads, or add them if it's your opinion that certain chips, caps, or VRMs aren't being cooled properly.

I doubt these GPUs had any thermal pads to begin with, but if they did, they're surely of inferior quality, as thermal pads have seen a big thermal improvements in the last year or two.
Top
Post Reply

Return to “Problems with AMD/ATI drivers”