Bad work unit??

Moderators: Site Moderators, FAHC Science Team

Post Reply
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Particle coordinate is nan???

Post by stratocastor »

i6700K, galax 980ti, Nvidia driver version 452.06, have tried newest 456.55 as well, just keeps crashing with Particle coordinate is nan error... eventually will continue to error out and stop attempting to fold. It will utilize my gpu for maybe a minute or two.. Any clarification or troubleshooting would be appreciated. I have been folding on and off for quite a few years with this rig.. Thought it might be the latest nvidia driver which is why I rolled back to 452.06. Thank you folding gurus!!

Code: Select all

*********************** Log Started 2020-10-01T18:59:09Z ***********************
18:59:21:FS01:Unpaused
18:59:22:WU02:FS01:Starting
18:59:22:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Beast\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 16336 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
18:59:22:WU02:FS01:Started FahCore on PID 14808
18:59:22:WU02:FS01:Core PID:15072
18:59:22:WU02:FS01:FahCore 0x22 started
18:59:22:WU02:FS01:0x22:*********************** Log Started 2020-10-01T18:59:22Z ***********************
18:59:22:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
18:59:22:WU02:FS01:0x22:       Core: Core22
18:59:22:WU02:FS01:0x22:       Type: 0x22
18:59:22:WU02:FS01:0x22:    Version: 0.0.13
18:59:22:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:59:22:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
18:59:22:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
18:59:22:WU02:FS01:0x22:       Date: Sep 19 2020
18:59:22:WU02:FS01:0x22:       Time: 02:35:58
18:59:22:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
18:59:22:WU02:FS01:0x22:     Branch: core22-0.0.13
18:59:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:59:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:59:22:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
18:59:22:WU02:FS01:0x22:   Platform: win32 10
18:59:22:WU02:FS01:0x22:       Bits: 64
18:59:22:WU02:FS01:0x22:       Mode: Release
18:59:22:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
18:59:22:WU02:FS01:0x22:             <peastman@stanford.edu>
18:59:22:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 14808 -checkpoint 15
18:59:22:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
18:59:22:WU02:FS01:0x22:             0 -gpu 0
18:59:22:WU02:FS01:0x22:************************************ libFAH ************************************
18:59:22:WU02:FS01:0x22:       Date: Sep 7 2020
18:59:22:WU02:FS01:0x22:       Time: 19:09:56
18:59:22:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
18:59:22:WU02:FS01:0x22:     Branch: HEAD
18:59:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:59:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:59:22:WU02:FS01:0x22:   Platform: win32 10
18:59:22:WU02:FS01:0x22:       Bits: 64
18:59:22:WU02:FS01:0x22:       Mode: Release
18:59:22:WU02:FS01:0x22:************************************ CBang *************************************
18:59:22:WU02:FS01:0x22:       Date: Sep 7 2020
18:59:22:WU02:FS01:0x22:       Time: 19:08:30
18:59:22:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
18:59:22:WU02:FS01:0x22:     Branch: HEAD
18:59:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:59:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:59:22:WU02:FS01:0x22:   Platform: win32 10
18:59:22:WU02:FS01:0x22:       Bits: 64
18:59:22:WU02:FS01:0x22:       Mode: Release
18:59:22:WU02:FS01:0x22:************************************ System ************************************
18:59:22:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
18:59:22:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
18:59:22:WU02:FS01:0x22:       CPUs: 8
18:59:22:WU02:FS01:0x22:     Memory: 15.94GiB
18:59:22:WU02:FS01:0x22:Free Memory: 11.87GiB
18:59:22:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
18:59:22:WU02:FS01:0x22: OS Version: 6.2
18:59:22:WU02:FS01:0x22:Has Battery: false
18:59:22:WU02:FS01:0x22: On Battery: false
18:59:22:WU02:FS01:0x22: UTC Offset: -6
18:59:22:WU02:FS01:0x22:        PID: 15072
18:59:22:WU02:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
18:59:22:WU02:FS01:0x22:************************************ OpenMM ************************************
18:59:22:WU02:FS01:0x22:   Revision: 189320d0
18:59:22:WU02:FS01:0x22:********************************************************************************
18:59:22:WU02:FS01:0x22:Project: 13426 (Run 7203, Clone 10, Gen 4)
18:59:22:WU02:FS01:0x22:Unit: 0x0000000812bc7d9a5f6650069675c6d3
18:59:22:WU02:FS01:0x22:Digital signatures verified
18:59:22:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
18:59:22:WU02:FS01:0x22:Version 0.0.13
18:59:23:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
18:59:23:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
18:59:23:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
18:59:23:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
18:59:24:WU02:FS01:0x22:There are 4 platforms available.
18:59:24:WU02:FS01:0x22:Platform 0: Reference
18:59:24:WU02:FS01:0x22:Platform 1: CPU
18:59:24:WU02:FS01:0x22:Platform 2: OpenCL
18:59:24:WU02:FS01:0x22:  opencl-device 0 specified
18:59:24:WU02:FS01:0x22:Platform 3: CUDA
18:59:24:WU02:FS01:0x22:  cuda-device 0 specified
18:59:35:WU02:FS01:0x22:Attempting to create CUDA context:
18:59:35:WU02:FS01:0x22:  Configuring platform CUDA
18:59:41:WU02:FS01:0x22:  Using CUDA and gpu 0
18:59:41:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
18:59:42:WU02:FS01:0x22:Checkpoint completed at step 0
19:00:15:WU02:FS01:0x22:An exception occurred at step 2258: Particle coordinate is nan
19:00:15:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
19:00:15:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
19:00:15:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
19:00:15:WU02:FS01:Starting
19:00:15:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Beast\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 16336 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
19:00:15:WU02:FS01:Started FahCore on PID 16652
19:00:15:WU02:FS01:Core PID:16676
19:00:15:WU02:FS01:FahCore 0x22 started
19:00:16:WU02:FS01:0x22:*********************** Log Started 2020-10-01T19:00:16Z ***********************
19:00:16:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:00:16:WU02:FS01:0x22:       Core: Core22
19:00:16:WU02:FS01:0x22:       Type: 0x22
19:00:16:WU02:FS01:0x22:    Version: 0.0.13
19:00:16:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:00:16:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
19:00:16:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
19:00:16:WU02:FS01:0x22:       Date: Sep 19 2020
19:00:16:WU02:FS01:0x22:       Time: 02:35:58
19:00:16:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
19:00:16:WU02:FS01:0x22:     Branch: core22-0.0.13
19:00:16:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:00:16:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:00:16:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
19:00:16:WU02:FS01:0x22:   Platform: win32 10
19:00:16:WU02:FS01:0x22:       Bits: 64
19:00:16:WU02:FS01:0x22:       Mode: Release
19:00:16:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:00:16:WU02:FS01:0x22:             <peastman@stanford.edu>
19:00:16:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 16652 -checkpoint 15
19:00:16:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
19:00:16:WU02:FS01:0x22:             0 -gpu 0
19:00:16:WU02:FS01:0x22:************************************ libFAH ************************************
19:00:16:WU02:FS01:0x22:       Date: Sep 7 2020
19:00:16:WU02:FS01:0x22:       Time: 19:09:56
19:00:16:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
19:00:16:WU02:FS01:0x22:     Branch: HEAD
19:00:16:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:00:16:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:00:16:WU02:FS01:0x22:   Platform: win32 10
19:00:16:WU02:FS01:0x22:       Bits: 64
19:00:16:WU02:FS01:0x22:       Mode: Release
19:00:16:WU02:FS01:0x22:************************************ CBang *************************************
19:00:16:WU02:FS01:0x22:       Date: Sep 7 2020
19:00:16:WU02:FS01:0x22:       Time: 19:08:30
19:00:16:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
19:00:16:WU02:FS01:0x22:     Branch: HEAD
19:00:16:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:00:16:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:00:16:WU02:FS01:0x22:   Platform: win32 10
19:00:16:WU02:FS01:0x22:       Bits: 64
19:00:16:WU02:FS01:0x22:       Mode: Release
19:00:16:WU02:FS01:0x22:************************************ System ************************************
19:00:16:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
19:00:16:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:00:16:WU02:FS01:0x22:       CPUs: 8
19:00:16:WU02:FS01:0x22:     Memory: 15.94GiB
19:00:16:WU02:FS01:0x22:Free Memory: 11.74GiB
19:00:16:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
19:00:16:WU02:FS01:0x22: OS Version: 6.2
19:00:16:WU02:FS01:0x22:Has Battery: false
19:00:16:WU02:FS01:0x22: On Battery: false
19:00:16:WU02:FS01:0x22: UTC Offset: -6
19:00:16:WU02:FS01:0x22:        PID: 16676
19:00:16:WU02:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
19:00:16:WU02:FS01:0x22:************************************ OpenMM ************************************
19:00:16:WU02:FS01:0x22:   Revision: 189320d0
19:00:16:WU02:FS01:0x22:********************************************************************************
19:00:16:WU02:FS01:0x22:Project: 13426 (Run 7203, Clone 10, Gen 4)
19:00:16:WU02:FS01:0x22:Unit: 0x0000000812bc7d9a5f6650069675c6d3
19:00:16:WU02:FS01:0x22:Digital signatures verified
19:00:16:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:00:16:WU02:FS01:0x22:Version 0.0.13
19:00:16:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:00:16:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:00:16:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
19:00:16:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
19:00:16:WU02:FS01:0x22:There are 4 platforms available.
19:00:16:WU02:FS01:0x22:Platform 0: Reference
19:00:16:WU02:FS01:0x22:Platform 1: CPU
19:00:16:WU02:FS01:0x22:Platform 2: OpenCL
19:00:16:WU02:FS01:0x22:  opencl-device 0 specified
19:00:16:WU02:FS01:0x22:Platform 3: CUDA
19:00:16:WU02:FS01:0x22:  cuda-device 0 specified
19:00:28:WU02:FS01:0x22:Attempting to create CUDA context:
19:00:28:WU02:FS01:0x22:  Configuring platform CUDA
19:00:32:WU02:FS01:0x22:  Using CUDA and gpu 0
19:00:32:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
19:00:43:WU02:FS01:0x22:An exception occurred at step 752: Particle coordinate is nan
19:00:43:WU02:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
19:00:43:WU02:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
19:00:43:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
19:00:43:WU02:FS01:0x22:Saving result file globals.csv
19:00:43:WU02:FS01:0x22:Saving result file science.log
19:00:43:WU02:FS01:0x22:Saving result file state.xml.bz2
19:00:43:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
19:00:43:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
19:00:43:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13426 run:7203 clone:10 gen:4 core:0x22 unit:0x0000000812bc7d9a5f6650069675c6d3
19:00:43:WU02:FS01:Uploading 6.36MiB to 18.188.125.154
19:00:43:WU02:FS01:Connecting to 18.188.125.154:8080
19:00:44:WU01:FS01:Connecting to assign1.foldingathome.org:80
19:00:44:WU01:FS01:Assigned to work server 129.213.157.105
19:00:44:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GM200 [GeForce GTX 980 Ti] 5632 from 129.213.157.105
19:00:44:WU01:FS01:Connecting to 129.213.157.105:8080
19:00:49:WU02:FS01:Upload 40.29%
19:00:50:WU01:FS01:Downloading 21.64MiB
19:00:55:WU02:FS01:Upload 67.81%
19:00:56:WU01:FS01:Download 14.73%
19:01:01:WU02:FS01:Upload 99.25%
19:01:01:WU02:FS01:Upload complete
19:01:01:WU02:FS01:Server responded WORK_ACK (400)
19:01:01:WU02:FS01:Cleaning up
19:01:02:WU01:FS01:Download 60.36%
19:01:05:WU01:FS01:Download complete
19:01:05:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14907 run:496 clone:2 gen:11 core:0x22 unit:0x0000000e81d59d695f52602d1f785722
19:01:05:WU01:FS01:Starting
19:01:05:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Beast\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 16336 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
19:01:05:WU01:FS01:Started FahCore on PID 15072
19:01:05:WU01:FS01:Core PID:9304
19:01:05:WU01:FS01:FahCore 0x22 started
19:01:06:WU01:FS01:0x22:*********************** Log Started 2020-10-01T19:01:05Z ***********************
19:01:06:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:01:06:WU01:FS01:0x22:       Core: Core22
19:01:06:WU01:FS01:0x22:       Type: 0x22
19:01:06:WU01:FS01:0x22:    Version: 0.0.13
19:01:06:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:01:06:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
19:01:06:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
19:01:06:WU01:FS01:0x22:       Date: Sep 19 2020
19:01:06:WU01:FS01:0x22:       Time: 02:35:58
19:01:06:WU01:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
19:01:06:WU01:FS01:0x22:     Branch: core22-0.0.13
19:01:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:01:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:01:06:WU01:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
19:01:06:WU01:FS01:0x22:   Platform: win32 10
19:01:06:WU01:FS01:0x22:       Bits: 64
19:01:06:WU01:FS01:0x22:       Mode: Release
19:01:06:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:01:06:WU01:FS01:0x22:             <peastman@stanford.edu>
19:01:06:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 15072 -checkpoint 15
19:01:06:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
19:01:06:WU01:FS01:0x22:             0 -gpu 0
19:01:06:WU01:FS01:0x22:************************************ libFAH ************************************
19:01:06:WU01:FS01:0x22:       Date: Sep 7 2020
19:01:06:WU01:FS01:0x22:       Time: 19:09:56
19:01:06:WU01:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
19:01:06:WU01:FS01:0x22:     Branch: HEAD
19:01:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:01:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:01:06:WU01:FS01:0x22:   Platform: win32 10
19:01:06:WU01:FS01:0x22:       Bits: 64
19:01:06:WU01:FS01:0x22:       Mode: Release
19:01:06:WU01:FS01:0x22:************************************ CBang *************************************
19:01:06:WU01:FS01:0x22:       Date: Sep 7 2020
19:01:06:WU01:FS01:0x22:       Time: 19:08:30
19:01:06:WU01:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
19:01:06:WU01:FS01:0x22:     Branch: HEAD
19:01:06:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:01:06:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:01:06:WU01:FS01:0x22:   Platform: win32 10
19:01:06:WU01:FS01:0x22:       Bits: 64
19:01:06:WU01:FS01:0x22:       Mode: Release
19:01:06:WU01:FS01:0x22:************************************ System ************************************
19:01:06:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
19:01:06:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:01:06:WU01:FS01:0x22:       CPUs: 8
19:01:06:WU01:FS01:0x22:     Memory: 15.94GiB
19:01:06:WU01:FS01:0x22:Free Memory: 11.75GiB
19:01:06:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
19:01:06:WU01:FS01:0x22: OS Version: 6.2
19:01:06:WU01:FS01:0x22:Has Battery: false
19:01:06:WU01:FS01:0x22: On Battery: false
19:01:06:WU01:FS01:0x22: UTC Offset: -6
19:01:06:WU01:FS01:0x22:        PID: 9304
19:01:06:WU01:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
19:01:06:WU01:FS01:0x22:************************************ OpenMM ************************************
19:01:06:WU01:FS01:0x22:   Revision: 189320d0
19:01:06:WU01:FS01:0x22:********************************************************************************
19:01:06:WU01:FS01:0x22:Project: 14907 (Run 496, Clone 2, Gen 11)
19:01:06:WU01:FS01:0x22:Unit: 0x0000000e81d59d695f52602d1f785722
19:01:06:WU01:FS01:0x22:Reading tar file core.xml
19:01:06:WU01:FS01:0x22:Reading tar file integrator.xml
19:01:06:WU01:FS01:0x22:Reading tar file state.xml
19:01:07:WU01:FS01:0x22:Reading tar file system.xml
19:01:08:WU01:FS01:0x22:Digital signatures verified
19:01:08:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:01:08:WU01:FS01:0x22:Version 0.0.13
19:01:08:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:01:08:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:01:08:WU01:FS01:0x22:  XTC frame write interval: 50000 steps (5%) [20 total]
19:01:08:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
19:01:08:WU01:FS01:0x22:There are 4 platforms available.
19:01:08:WU01:FS01:0x22:Platform 0: Reference
19:01:08:WU01:FS01:0x22:Platform 1: CPU
19:01:08:WU01:FS01:0x22:Platform 2: OpenCL
19:01:08:WU01:FS01:0x22:  opencl-device 0 specified
19:01:08:WU01:FS01:0x22:Platform 3: CUDA
19:01:08:WU01:FS01:0x22:  cuda-device 0 specified
19:01:24:WU01:FS01:0x22:Attempting to create CUDA context:
19:01:24:WU01:FS01:0x22:  Configuring platform CUDA
19:01:33:WU01:FS01:0x22:  Using CUDA and gpu 0
19:01:34:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
19:01:35:WU01:FS01:0x22:Checkpoint completed at step 0
19:02:19:WU01:FS01:0x22:An exception occurred at step 3513: Particle coordinate is nan
19:02:19:WU01:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
19:02:19:WU01:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
19:02:19:WARNING:WU01:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
19:02:19:WU01:FS01:Starting
19:02:19:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Beast\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 16336 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
19:02:19:WU01:FS01:Started FahCore on PID 12100
19:02:19:WU01:FS01:Core PID:13752
19:02:19:WU01:FS01:FahCore 0x22 started
19:02:20:WU01:FS01:0x22:*********************** Log Started 2020-10-01T19:02:19Z ***********************
19:02:20:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:02:20:WU01:FS01:0x22:       Core: Core22
19:02:20:WU01:FS01:0x22:       Type: 0x22
19:02:20:WU01:FS01:0x22:    Version: 0.0.13
19:02:20:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:02:20:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
19:02:20:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
19:02:20:WU01:FS01:0x22:       Date: Sep 19 2020
19:02:20:WU01:FS01:0x22:       Time: 02:35:58
19:02:20:WU01:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
19:02:20:WU01:FS01:0x22:     Branch: core22-0.0.13
19:02:20:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:02:20:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:02:20:WU01:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
19:02:20:WU01:FS01:0x22:   Platform: win32 10
19:02:20:WU01:FS01:0x22:       Bits: 64
19:02:20:WU01:FS01:0x22:       Mode: Release
19:02:20:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:02:20:WU01:FS01:0x22:             <peastman@stanford.edu>
19:02:20:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 12100 -checkpoint 15
19:02:20:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
19:02:20:WU01:FS01:0x22:             0 -gpu 0
19:02:20:WU01:FS01:0x22:************************************ libFAH ************************************
19:02:20:WU01:FS01:0x22:       Date: Sep 7 2020
19:02:20:WU01:FS01:0x22:       Time: 19:09:56
19:02:20:WU01:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
19:02:20:WU01:FS01:0x22:     Branch: HEAD
19:02:20:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:02:20:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:02:20:WU01:FS01:0x22:   Platform: win32 10
19:02:20:WU01:FS01:0x22:       Bits: 64
19:02:20:WU01:FS01:0x22:       Mode: Release
19:02:20:WU01:FS01:0x22:************************************ CBang *************************************
19:02:20:WU01:FS01:0x22:       Date: Sep 7 2020
19:02:20:WU01:FS01:0x22:       Time: 19:08:30
19:02:20:WU01:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
19:02:20:WU01:FS01:0x22:     Branch: HEAD
19:02:20:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:02:20:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:02:20:WU01:FS01:0x22:   Platform: win32 10
19:02:20:WU01:FS01:0x22:       Bits: 64
19:02:20:WU01:FS01:0x22:       Mode: Release
19:02:20:WU01:FS01:0x22:************************************ System ************************************
19:02:20:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
19:02:20:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:02:20:WU01:FS01:0x22:       CPUs: 8
19:02:20:WU01:FS01:0x22:     Memory: 15.94GiB
19:02:20:WU01:FS01:0x22:Free Memory: 11.80GiB
19:02:20:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
19:02:20:WU01:FS01:0x22: OS Version: 6.2
19:02:20:WU01:FS01:0x22:Has Battery: false
19:02:20:WU01:FS01:0x22: On Battery: false
19:02:20:WU01:FS01:0x22: UTC Offset: -6
19:02:20:WU01:FS01:0x22:        PID: 13752
19:02:20:WU01:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
19:02:20:WU01:FS01:0x22:************************************ OpenMM ************************************
19:02:20:WU01:FS01:0x22:   Revision: 189320d0
19:02:20:WU01:FS01:0x22:********************************************************************************
19:02:20:WU01:FS01:0x22:Project: 14907 (Run 496, Clone 2, Gen 11)
19:02:20:WU01:FS01:0x22:Unit: 0x0000000e81d59d695f52602d1f785722
19:02:20:WU01:FS01:0x22:Digital signatures verified
19:02:20:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:02:20:WU01:FS01:0x22:Version 0.0.13
19:02:20:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:02:20:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:02:20:WU01:FS01:0x22:  XTC frame write interval: 50000 steps (5%) [20 total]
19:02:20:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
19:02:20:WU01:FS01:0x22:There are 4 platforms available.
19:02:20:WU01:FS01:0x22:Platform 0: Reference
19:02:20:WU01:FS01:0x22:Platform 1: CPU
19:02:20:WU01:FS01:0x22:Platform 2: OpenCL
19:02:20:WU01:FS01:0x22:  opencl-device 0 specified
19:02:20:WU01:FS01:0x22:Platform 3: CUDA
19:02:20:WU01:FS01:0x22:  cuda-device 0 specified
19:02:36:WU01:FS01:0x22:Attempting to create CUDA context:
19:02:36:WU01:FS01:0x22:  Configuring platform CUDA
19:02:42:WU01:FS01:0x22:  Using CUDA and gpu 0
19:02:42:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
19:03:23:WU01:FS01:0x22:An exception occurred at step 3011: Particle coordinate is nan
19:03:23:WU01:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
19:03:23:WU01:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
19:03:23:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
19:03:23:WU01:FS01:0x22:Saving result file science.log
19:03:23:WU01:FS01:0x22:Saving result file state.xml
19:03:26:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
19:03:27:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
19:03:27:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:14907 run:496 clone:2 gen:11 core:0x22 unit:0x0000000e81d59d695f52602d1f785722
19:03:27:WU01:FS01:Uploading 18.67MiB to 129.213.157.105
19:03:27:WU01:FS01:Connecting to 129.213.157.105:8080
19:03:27:WU02:FS01:Connecting to assign1.foldingathome.org:80
19:03:28:WU02:FS01:Assigned to work server 18.188.125.154
19:03:28:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GM200 [GeForce GTX 980 Ti] 5632 from 18.188.125.154
19:03:28:WU02:FS01:Connecting to 18.188.125.154:8080
19:03:29:WU02:FS01:Downloading 6.91MiB
19:03:31:WU02:FS01:Download complete
19:03:31:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13426 run:6237 clone:25 gen:5 core:0x22 unit:0x0000000a12bc7d9a5f664ff1b2eb62b4
19:03:31:WU02:FS01:Starting
19:03:31:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Beast\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 16336 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
19:03:31:WU02:FS01:Started FahCore on PID 16508
19:03:31:WU02:FS01:Core PID:15624
19:03:31:WU02:FS01:FahCore 0x22 started
19:03:31:WU02:FS01:0x22:*********************** Log Started 2020-10-01T19:03:31Z ***********************
19:03:31:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:03:31:WU02:FS01:0x22:       Core: Core22
19:03:31:WU02:FS01:0x22:       Type: 0x22
19:03:31:WU02:FS01:0x22:    Version: 0.0.13
19:03:31:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:03:31:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
19:03:31:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
19:03:31:WU02:FS01:0x22:       Date: Sep 19 2020
19:03:31:WU02:FS01:0x22:       Time: 02:35:58
19:03:31:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
19:03:31:WU02:FS01:0x22:     Branch: core22-0.0.13
19:03:31:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:03:31:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:03:31:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
19:03:31:WU02:FS01:0x22:   Platform: win32 10
19:03:31:WU02:FS01:0x22:       Bits: 64
19:03:31:WU02:FS01:0x22:       Mode: Release
19:03:31:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:03:31:WU02:FS01:0x22:             <peastman@stanford.edu>
19:03:31:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 16508 -checkpoint 15
19:03:31:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
19:03:31:WU02:FS01:0x22:             0 -gpu 0
19:03:31:WU02:FS01:0x22:************************************ libFAH ************************************
19:03:31:WU02:FS01:0x22:       Date: Sep 7 2020
19:03:31:WU02:FS01:0x22:       Time: 19:09:56
19:03:31:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
19:03:31:WU02:FS01:0x22:     Branch: HEAD
19:03:31:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:03:31:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:03:31:WU02:FS01:0x22:   Platform: win32 10
19:03:31:WU02:FS01:0x22:       Bits: 64
19:03:31:WU02:FS01:0x22:       Mode: Release
19:03:31:WU02:FS01:0x22:************************************ CBang *************************************
19:03:31:WU02:FS01:0x22:       Date: Sep 7 2020
19:03:31:WU02:FS01:0x22:       Time: 19:08:30
19:03:31:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
19:03:31:WU02:FS01:0x22:     Branch: HEAD
19:03:31:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:03:31:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:03:31:WU02:FS01:0x22:   Platform: win32 10
19:03:31:WU02:FS01:0x22:       Bits: 64
19:03:31:WU02:FS01:0x22:       Mode: Release
19:03:31:WU02:FS01:0x22:************************************ System ************************************
19:03:31:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
19:03:31:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:03:31:WU02:FS01:0x22:       CPUs: 8
19:03:31:WU02:FS01:0x22:     Memory: 15.94GiB
19:03:31:WU02:FS01:0x22:Free Memory: 11.84GiB
19:03:31:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
19:03:31:WU02:FS01:0x22: OS Version: 6.2
19:03:31:WU02:FS01:0x22:Has Battery: false
19:03:31:WU02:FS01:0x22: On Battery: false
19:03:31:WU02:FS01:0x22: UTC Offset: -6
19:03:31:WU02:FS01:0x22:        PID: 15624
19:03:31:WU02:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
19:03:31:WU02:FS01:0x22:************************************ OpenMM ************************************
19:03:31:WU02:FS01:0x22:   Revision: 189320d0
19:03:31:WU02:FS01:0x22:********************************************************************************
19:03:31:WU02:FS01:0x22:Project: 13426 (Run 6237, Clone 25, Gen 5)
19:03:31:WU02:FS01:0x22:Unit: 0x0000000a12bc7d9a5f664ff1b2eb62b4
19:03:31:WU02:FS01:0x22:Reading tar file core.xml
19:03:31:WU02:FS01:0x22:Reading tar file integrator.xml
19:03:31:WU02:FS01:0x22:Reading tar file state.xml.bz2
19:03:31:WU02:FS01:0x22:Reading tar file system.xml.bz2
19:03:31:WU02:FS01:0x22:Digital signatures verified
19:03:31:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:03:31:WU02:FS01:0x22:Version 0.0.13
19:03:31:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:03:31:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:03:31:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
19:03:31:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
19:03:31:WU02:FS01:0x22:There are 4 platforms available.
19:03:31:WU02:FS01:0x22:Platform 0: Reference
19:03:31:WU02:FS01:0x22:Platform 1: CPU
19:03:31:WU02:FS01:0x22:Platform 2: OpenCL
19:03:31:WU02:FS01:0x22:  opencl-device 0 specified
19:03:31:WU02:FS01:0x22:Platform 3: CUDA
19:03:31:WU02:FS01:0x22:  cuda-device 0 specified
19:03:33:WU01:FS01:Upload 10.71%
19:03:39:WU01:FS01:Upload 32.47%
19:03:43:WU02:FS01:0x22:Attempting to create CUDA context:
19:03:43:WU02:FS01:0x22:  Configuring platform CUDA
19:03:45:WU01:FS01:Upload 50.54%
19:03:49:WU02:FS01:0x22:  Using CUDA and gpu 0
19:03:49:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
19:03:50:WU02:FS01:0x22:Checkpoint completed at step 0
19:03:51:WU01:FS01:Upload 61.59%
19:03:57:WU01:FS01:Upload 74.98%
19:04:03:WU01:FS01:Upload 87.70%
19:04:07:WU01:FS01:Upload complete
19:04:07:WU01:FS01:Server responded WORK_ACK (400)
19:04:07:WU01:FS01:Cleaning up
19:04:21:WU02:FS01:0x22:An exception occurred at step 1505: Particle coordinate is nan
19:04:21:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
19:04:21:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
19:04:21:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
19:04:22:WU02:FS01:Starting
19:04:22:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Beast\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 16336 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
19:04:22:WU02:FS01:Started FahCore on PID 16512
19:04:22:WU02:FS01:Core PID:1420
19:04:22:WU02:FS01:FahCore 0x22 started
19:04:22:WU02:FS01:0x22:*********************** Log Started 2020-10-01T19:04:22Z ***********************
19:04:22:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:04:22:WU02:FS01:0x22:       Core: Core22
19:04:22:WU02:FS01:0x22:       Type: 0x22
19:04:22:WU02:FS01:0x22:    Version: 0.0.13
19:04:22:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:04:22:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
19:04:22:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
19:04:22:WU02:FS01:0x22:       Date: Sep 19 2020
19:04:22:WU02:FS01:0x22:       Time: 02:35:58
19:04:22:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
19:04:22:WU02:FS01:0x22:     Branch: core22-0.0.13
19:04:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:04:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:04:22:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
19:04:22:WU02:FS01:0x22:   Platform: win32 10
19:04:22:WU02:FS01:0x22:       Bits: 64
19:04:22:WU02:FS01:0x22:       Mode: Release
19:04:22:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:04:22:WU02:FS01:0x22:             <peastman@stanford.edu>
19:04:22:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 16512 -checkpoint 15
19:04:22:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
19:04:22:WU02:FS01:0x22:             0 -gpu 0
19:04:22:WU02:FS01:0x22:************************************ libFAH ************************************
19:04:22:WU02:FS01:0x22:       Date: Sep 7 2020
19:04:22:WU02:FS01:0x22:       Time: 19:09:56
19:04:22:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
19:04:22:WU02:FS01:0x22:     Branch: HEAD
19:04:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:04:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:04:22:WU02:FS01:0x22:   Platform: win32 10
19:04:22:WU02:FS01:0x22:       Bits: 64
19:04:22:WU02:FS01:0x22:       Mode: Release
19:04:22:WU02:FS01:0x22:************************************ CBang *************************************
19:04:22:WU02:FS01:0x22:       Date: Sep 7 2020
19:04:22:WU02:FS01:0x22:       Time: 19:08:30
19:04:22:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
19:04:22:WU02:FS01:0x22:     Branch: HEAD
19:04:22:WU02:FS01:0x22:   Compiler: Visual C++ 2015
19:04:22:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:04:22:WU02:FS01:0x22:   Platform: win32 10
19:04:22:WU02:FS01:0x22:       Bits: 64
19:04:22:WU02:FS01:0x22:       Mode: Release
19:04:22:WU02:FS01:0x22:************************************ System ************************************
19:04:22:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
19:04:22:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:04:22:WU02:FS01:0x22:       CPUs: 8
19:04:22:WU02:FS01:0x22:     Memory: 15.94GiB
19:04:22:WU02:FS01:0x22:Free Memory: 11.86GiB
19:04:22:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
19:04:22:WU02:FS01:0x22: OS Version: 6.2
19:04:22:WU02:FS01:0x22:Has Battery: false
19:04:22:WU02:FS01:0x22: On Battery: false
19:04:22:WU02:FS01:0x22: UTC Offset: -6
19:04:22:WU02:FS01:0x22:        PID: 1420
19:04:22:WU02:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
19:04:22:WU02:FS01:0x22:************************************ OpenMM ************************************
19:04:22:WU02:FS01:0x22:   Revision: 189320d0
19:04:22:WU02:FS01:0x22:********************************************************************************
19:04:22:WU02:FS01:0x22:Project: 13426 (Run 6237, Clone 25, Gen 5)
19:04:22:WU02:FS01:0x22:Unit: 0x0000000a12bc7d9a5f664ff1b2eb62b4
19:04:22:WU02:FS01:0x22:Digital signatures verified
19:04:22:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:04:22:WU02:FS01:0x22:Version 0.0.13
19:04:22:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:04:22:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:04:22:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
19:04:22:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
19:04:22:WU02:FS01:0x22:There are 4 platforms available.
19:04:22:WU02:FS01:0x22:Platform 0: Reference
19:04:22:WU02:FS01:0x22:Platform 1: CPU
19:04:22:WU02:FS01:0x22:Platform 2: OpenCL
19:04:22:WU02:FS01:0x22:  opencl-device 0 specified
19:04:22:WU02:FS01:0x22:Platform 3: CUDA
19:04:22:WU02:FS01:0x22:  cuda-device 0 specified
Last edited by Joe_H on Thu Oct 01, 2020 7:17 pm, edited 1 time in total.
Reason: added Code tags to log
Top
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Re: Particle coordinate is nan???

Post by stratocastor »

Hmmm... as soon as I was done posting this... seems to be folding away now.... Any ideas???

Code: Select all

19:07:23:WU01:FS01:0x22:*********************** Log Started 2020-10-01T19:07:22Z ***********************
19:07:23:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:07:23:WU01:FS01:0x22:       Core: Core22
19:07:23:WU01:FS01:0x22:       Type: 0x22
19:07:23:WU01:FS01:0x22:    Version: 0.0.13
19:07:23:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:07:23:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
19:07:23:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
19:07:23:WU01:FS01:0x22:       Date: Sep 19 2020
19:07:23:WU01:FS01:0x22:       Time: 02:35:58
19:07:23:WU01:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
19:07:23:WU01:FS01:0x22:     Branch: core22-0.0.13
19:07:23:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:07:23:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:07:23:WU01:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
19:07:23:WU01:FS01:0x22:   Platform: win32 10
19:07:23:WU01:FS01:0x22:       Bits: 64
19:07:23:WU01:FS01:0x22:       Mode: Release
19:07:23:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:07:23:WU01:FS01:0x22:             <peastman@stanford.edu>
19:07:23:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 14412 -checkpoint 15
19:07:23:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
19:07:23:WU01:FS01:0x22:             0 -gpu 0
19:07:23:WU01:FS01:0x22:************************************ libFAH ************************************
19:07:23:WU01:FS01:0x22:       Date: Sep 7 2020
19:07:23:WU01:FS01:0x22:       Time: 19:09:56
19:07:23:WU01:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
19:07:23:WU01:FS01:0x22:     Branch: HEAD
19:07:23:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:07:23:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:07:23:WU01:FS01:0x22:   Platform: win32 10
19:07:23:WU01:FS01:0x22:       Bits: 64
19:07:23:WU01:FS01:0x22:       Mode: Release
19:07:23:WU01:FS01:0x22:************************************ CBang *************************************
19:07:23:WU01:FS01:0x22:       Date: Sep 7 2020
19:07:23:WU01:FS01:0x22:       Time: 19:08:30
19:07:23:WU01:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
19:07:23:WU01:FS01:0x22:     Branch: HEAD
19:07:23:WU01:FS01:0x22:   Compiler: Visual C++ 2015
19:07:23:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
19:07:23:WU01:FS01:0x22:   Platform: win32 10
19:07:23:WU01:FS01:0x22:       Bits: 64
19:07:23:WU01:FS01:0x22:       Mode: Release
19:07:23:WU01:FS01:0x22:************************************ System ************************************
19:07:23:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
19:07:23:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
19:07:23:WU01:FS01:0x22:       CPUs: 8
19:07:23:WU01:FS01:0x22:     Memory: 15.94GiB
19:07:23:WU01:FS01:0x22:Free Memory: 11.70GiB
19:07:23:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
19:07:23:WU01:FS01:0x22: OS Version: 6.2
19:07:23:WU01:FS01:0x22:Has Battery: false
19:07:23:WU01:FS01:0x22: On Battery: false
19:07:23:WU01:FS01:0x22: UTC Offset: -6
19:07:23:WU01:FS01:0x22:        PID: 7884
19:07:23:WU01:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
19:07:23:WU01:FS01:0x22:************************************ OpenMM ************************************
19:07:23:WU01:FS01:0x22:   Revision: 189320d0
19:07:23:WU01:FS01:0x22:********************************************************************************
19:07:23:WU01:FS01:0x22:Project: 14486 (Run 0, Clone 401, Gen 38)
19:07:23:WU01:FS01:0x22:Unit: 0x0000003f42aa6f325f45dea903158802
19:07:23:WU01:FS01:0x22:Reading tar file core.xml
19:07:23:WU01:FS01:0x22:Reading tar file integrator.xml.bz2
19:07:23:WU01:FS01:0x22:Reading tar file state.xml.bz2
19:07:23:WU01:FS01:0x22:Reading tar file system.xml.bz2
19:07:23:WU01:FS01:0x22:Digital signatures verified
19:07:23:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:07:23:WU01:FS01:0x22:Version 0.0.13
19:07:23:WU01:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
19:07:23:WU01:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
19:07:23:WU01:FS01:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
19:07:23:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
19:07:23:WU01:FS01:0x22:There are 4 platforms available.
19:07:23:WU01:FS01:0x22:Platform 0: Reference
19:07:23:WU01:FS01:0x22:Platform 1: CPU
19:07:23:WU01:FS01:0x22:Platform 2: OpenCL
19:07:23:WU01:FS01:0x22:  opencl-device 0 specified
19:07:23:WU01:FS01:0x22:Platform 3: CUDA
19:07:23:WU01:FS01:0x22:  cuda-device 0 specified
19:07:24:WU02:FS01:Upload 54.56%
19:07:29:WU02:FS01:Upload complete
19:07:29:WU02:FS01:Server responded WORK_ACK (400)
19:07:29:WU02:FS01:Cleaning up
19:07:34:WU01:FS01:0x22:Attempting to create CUDA context:
19:07:34:WU01:FS01:0x22:  Configuring platform CUDA
19:07:42:WU01:FS01:0x22:  Using CUDA and gpu 0
19:07:42:WU01:FS01:0x22:Completed 0 out of 1250000 steps (0%)
19:07:43:WU01:FS01:0x22:Checkpoint completed at step 0
19:08:42:WU01:FS01:0x22:Completed 12500 out of 1250000 steps (1%)
Last edited by Joe_H on Thu Oct 01, 2020 7:18 pm, edited 1 time in total.
Reason: added Code tags to log
Top
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Re: Particle coordinate is nan???

Post by stratocastor »

The above is still folding successfully on driver version 452.06 btw..
Top
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Re: Particle coordinate is nan???

Post by stratocastor »

Well messed with it some more this afternoon... now folding on 456.55, and doing fine.. who knows... Back at it!!
Top
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Re: Particle coordinate is nan???

Post by stratocastor »

stratocastor wrote:Well messed with it some more this afternoon... now folding on 456.55, and doing fine.. who knows... Back at it!!
I take that back, 456.55 and 456.38 both do it, 452.06 seems stable.
Top
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Bad work unit??

Post by stratocastor »

I need some help!! Thought this was driver related after latest update, but now still having problems despite driver rollback after DDU uninstall. Not quite sure whats gong on! Can someone help point me in the right direction? Multiple projects... Thank you all so much!!

Code: Select all

18:35:30:WU00:FS01:0x22:*********************** Log Started 2020-10-02T18:35:30Z ***********************
18:35:30:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
18:35:30:WU00:FS01:0x22:       Core: Core22
18:35:30:WU00:FS01:0x22:       Type: 0x22
18:35:30:WU00:FS01:0x22:    Version: 0.0.13
18:35:30:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:35:30:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
18:35:30:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
18:35:30:WU00:FS01:0x22:       Date: Sep 19 2020
18:35:30:WU00:FS01:0x22:       Time: 02:35:58
18:35:30:WU00:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
18:35:30:WU00:FS01:0x22:     Branch: core22-0.0.13
18:35:30:WU00:FS01:0x22:   Compiler: Visual C++ 2015
18:35:30:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:35:30:WU00:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
18:35:30:WU00:FS01:0x22:   Platform: win32 10
18:35:30:WU00:FS01:0x22:       Bits: 64
18:35:30:WU00:FS01:0x22:       Mode: Release
18:35:30:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
18:35:30:WU00:FS01:0x22:             <peastman@stanford.edu>
18:35:30:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 9360 -checkpoint 15
18:35:30:WU00:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
18:35:30:WU00:FS01:0x22:             0 -gpu 0
18:35:30:WU00:FS01:0x22:************************************ libFAH ************************************
18:35:30:WU00:FS01:0x22:       Date: Sep 7 2020
18:35:30:WU00:FS01:0x22:       Time: 19:09:56
18:35:30:WU00:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
18:35:30:WU00:FS01:0x22:     Branch: HEAD
18:35:30:WU00:FS01:0x22:   Compiler: Visual C++ 2015
18:35:30:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:35:30:WU00:FS01:0x22:   Platform: win32 10
18:35:30:WU00:FS01:0x22:       Bits: 64
18:35:30:WU00:FS01:0x22:       Mode: Release
18:35:30:WU00:FS01:0x22:************************************ CBang *************************************
18:35:30:WU00:FS01:0x22:       Date: Sep 7 2020
18:35:30:WU00:FS01:0x22:       Time: 19:08:30
18:35:30:WU00:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
18:35:30:WU00:FS01:0x22:     Branch: HEAD
18:35:30:WU00:FS01:0x22:   Compiler: Visual C++ 2015
18:35:30:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:35:30:WU00:FS01:0x22:   Platform: win32 10
18:35:30:WU00:FS01:0x22:       Bits: 64
18:35:30:WU00:FS01:0x22:       Mode: Release
18:35:30:WU00:FS01:0x22:************************************ System ************************************
18:35:30:WU00:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
18:35:30:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
18:35:30:WU00:FS01:0x22:       CPUs: 8
18:35:30:WU00:FS01:0x22:     Memory: 15.94GiB
18:35:30:WU00:FS01:0x22:Free Memory: 11.16GiB
18:35:30:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
18:35:30:WU00:FS01:0x22: OS Version: 6.2
18:35:30:WU00:FS01:0x22:Has Battery: false
18:35:30:WU00:FS01:0x22: On Battery: false
18:35:30:WU00:FS01:0x22: UTC Offset: -6
18:35:30:WU00:FS01:0x22:        PID: 9972
18:35:30:WU00:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
18:35:30:WU00:FS01:0x22:************************************ OpenMM ************************************
18:35:30:WU00:FS01:0x22:   Revision: 189320d0
18:35:30:WU00:FS01:0x22:********************************************************************************
18:35:30:WU00:FS01:0x22:Project: 16918 (Run 13, Clone 64, Gen 113)
18:35:30:WU00:FS01:0x22:Unit: 0x0000009c0002894c5f1761c81b4dfb30
18:35:30:WU00:FS01:0x22:Digital signatures verified
18:35:30:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
18:35:30:WU00:FS01:0x22:Version 0.0.13
18:35:30:WU00:FS01:0x22:  Checkpoint write interval: 100000 steps (2%) [50 total]
18:35:30:WU00:FS01:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
18:35:30:WU00:FS01:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
18:35:30:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
18:35:30:WU00:FS01:0x22:There are 4 platforms available.
18:35:30:WU00:FS01:0x22:Platform 0: Reference
18:35:30:WU00:FS01:0x22:Platform 1: CPU
18:35:30:WU00:FS01:0x22:Platform 2: OpenCL
18:35:30:WU00:FS01:0x22:  opencl-device 0 specified
18:35:30:WU00:FS01:0x22:Platform 3: CUDA
18:35:30:WU00:FS01:0x22:  cuda-device 0 specified
18:35:34:WU00:FS01:0x22:Attempting to create CUDA context:
18:35:34:WU00:FS01:0x22:  Configuring platform CUDA
18:35:36:WU00:FS01:0x22:  Using CUDA and gpu 0
18:35:36:WU00:FS01:0x22:Completed 0 out of 5000000 steps (0%)
18:36:14:WU00:FS01:0x22:An exception occurred at step 15059: Particle coordinate is nan
18:36:14:WU00:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
18:36:15:WU00:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
18:36:15:WU00:FS01:0x22:Saving result file ..\logfile_01.txt
18:36:15:WU00:FS01:0x22:Saving result file science.log
18:36:15:WU00:FS01:0x22:Saving result file state.xml
18:36:15:WU00:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
18:36:15:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
18:36:15:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:16918 run:13 clone:64 gen:113 core:0x22 unit:0x0000009c0002894c5f1761c81b4dfb30
18:36:15:WU00:FS01:Uploading 15.12MiB to 155.247.166.220
18:36:15:WU00:FS01:Connecting to 155.247.166.220:8080
18:36:15:WU02:FS01:Connecting to assign1.foldingathome.org:80
18:36:15:WU02:FS01:Assigned to work server 18.188.125.154
18:36:15:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GM200 [GeForce GTX 980 Ti] 5632 from 18.188.125.154
18:36:15:WU02:FS01:Connecting to 18.188.125.154:8080
18:36:17:WU02:FS01:Downloading 6.91MiB
18:36:18:WU02:FS01:Download complete
18:36:18:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13426 run:7220 clone:5 gen:4 core:0x22 unit:0x0000000812bc7d9a5f6650115527a24a
18:36:18:WU02:FS01:Starting
18:36:18:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Beast\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 12140 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
18:36:18:WU02:FS01:Started FahCore on PID 11060
18:36:18:WU02:FS01:Core PID:6524
18:36:18:WU02:FS01:FahCore 0x22 started
18:36:19:WU02:FS01:0x22:*********************** Log Started 2020-10-02T18:36:18Z ***********************
18:36:19:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
18:36:19:WU02:FS01:0x22:       Core: Core22
18:36:19:WU02:FS01:0x22:       Type: 0x22
18:36:19:WU02:FS01:0x22:    Version: 0.0.13
18:36:19:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:36:19:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
18:36:19:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
18:36:19:WU02:FS01:0x22:       Date: Sep 19 2020
18:36:19:WU02:FS01:0x22:       Time: 02:35:58
18:36:19:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
18:36:19:WU02:FS01:0x22:     Branch: core22-0.0.13
18:36:19:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:36:19:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:36:19:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
18:36:19:WU02:FS01:0x22:   Platform: win32 10
18:36:19:WU02:FS01:0x22:       Bits: 64
18:36:19:WU02:FS01:0x22:       Mode: Release
18:36:19:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
18:36:19:WU02:FS01:0x22:             <peastman@stanford.edu>
18:36:19:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 11060 -checkpoint 15
18:36:19:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
18:36:19:WU02:FS01:0x22:             0 -gpu 0
18:36:19:WU02:FS01:0x22:************************************ libFAH ************************************
18:36:19:WU02:FS01:0x22:       Date: Sep 7 2020
18:36:19:WU02:FS01:0x22:       Time: 19:09:56
18:36:19:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
18:36:19:WU02:FS01:0x22:     Branch: HEAD
18:36:19:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:36:19:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:36:19:WU02:FS01:0x22:   Platform: win32 10
18:36:19:WU02:FS01:0x22:       Bits: 64
18:36:19:WU02:FS01:0x22:       Mode: Release
18:36:19:WU02:FS01:0x22:************************************ CBang *************************************
18:36:19:WU02:FS01:0x22:       Date: Sep 7 2020
18:36:19:WU02:FS01:0x22:       Time: 19:08:30
18:36:19:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
18:36:19:WU02:FS01:0x22:     Branch: HEAD
18:36:19:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:36:19:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:36:19:WU02:FS01:0x22:   Platform: win32 10
18:36:19:WU02:FS01:0x22:       Bits: 64
18:36:19:WU02:FS01:0x22:       Mode: Release
18:36:19:WU02:FS01:0x22:************************************ System ************************************
18:36:19:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
18:36:19:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
18:36:19:WU02:FS01:0x22:       CPUs: 8
18:36:19:WU02:FS01:0x22:     Memory: 15.94GiB
18:36:19:WU02:FS01:0x22:Free Memory: 11.18GiB
18:36:19:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
18:36:19:WU02:FS01:0x22: OS Version: 6.2
18:36:19:WU02:FS01:0x22:Has Battery: false
18:36:19:WU02:FS01:0x22: On Battery: false
18:36:19:WU02:FS01:0x22: UTC Offset: -6
18:36:19:WU02:FS01:0x22:        PID: 6524
18:36:19:WU02:FS01:0x22:        CWD: C:\Users\Beast\AppData\Roaming\FAHClient\work
18:36:19:WU02:FS01:0x22:************************************ OpenMM ************************************
18:36:19:WU02:FS01:0x22:   Revision: 189320d0
18:36:19:WU02:FS01:0x22:********************************************************************************
18:36:19:WU02:FS01:0x22:Project: 13426 (Run 7220, Clone 5, Gen 4)
18:36:19:WU02:FS01:0x22:Unit: 0x0000000812bc7d9a5f6650115527a24a
18:36:19:WU02:FS01:0x22:Reading tar file core.xml
18:36:19:WU02:FS01:0x22:Reading tar file integrator.xml
18:36:19:WU02:FS01:0x22:Reading tar file state.xml.bz2
18:36:19:WU02:FS01:0x22:Reading tar file system.xml.bz2
18:36:19:WU02:FS01:0x22:Digital signatures verified
18:36:19:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
18:36:19:WU02:FS01:0x22:Version 0.0.13
18:36:19:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
18:36:19:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
18:36:19:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
18:36:19:WU02:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
18:36:19:WU02:FS01:0x22:There are 4 platforms available.
18:36:19:WU02:FS01:0x22:Platform 0: Reference
18:36:19:WU02:FS01:0x22:Platform 1: CPU
18:36:19:WU02:FS01:0x22:Platform 2: OpenCL
18:36:19:WU02:FS01:0x22:  opencl-device 0 specified
18:36:19:WU02:FS01:0x22:Platform 3: CUDA
18:36:19:WU02:FS01:0x22:  cuda-device 0 specified
18:36:21:WU00:FS01:Upload 18.18%
18:36:27:WU00:FS01:Upload 31.82%
18:36:30:WU02:FS01:0x22:Attempting to create CUDA context:
18:36:30:WU02:FS01:0x22:  Configuring platform CUDA
18:36:33:WU00:FS01:Upload 53.72%
18:36:35:WU02:FS01:0x22:  Using CUDA and gpu 0
18:36:35:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
18:36:36:WU02:FS01:0x22:Checkpoint completed at step 0
18:36:39:WU00:FS01:Upload 80.59%
18:36:45:WU00:FS01:Upload 99.60%
18:36:45:WU00:FS01:Upload complete
18:36:45:WU00:FS01:Server responded WORK_ACK (400)
18:36:45:WU00:FS01:Cleaning up
18:37:06:WU02:FS01:0x22:An exception occurred at step 1254: Particle coordinate is nan
18:37:06:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
18:37:06:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
18:37:07:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
Mod Edit: Added Code Tags - PantherX
Top
NormalDiffusion
Posts: 124
Joined: Sat Apr 18, 2020 1:50 pm

Re: Bad work unit??

Post by NormalDiffusion »

Gpu overclocked?
Top
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Re: Bad work unit??

Post by stratocastor »

Well that's what Im into now...

https://www.galax.com/en/galax-geforce- ... i-hof.html

says factory overclocked boost to 1291... Mine was going to 1404.. Have never messed with overclocking it manually... Set up afterburner, and reduced -90 tp 1314, and its going along right now.. Found default nvidia listed specs as follows: Base Clock
1000 MHz
Boost Clock
1076 MHz

Thats crazy. Its been running at this 1404mhz since I bought it years back....
Top
NormalDiffusion
Posts: 124
Joined: Sat Apr 18, 2020 1:50 pm

Re: Bad work unit??

Post by NormalDiffusion »

Yes, in games the OC may be stable, but f@h is more demanding on the hardware. And with the cuda update, even more from what I see on my cards (higher usage of the gpu).
also the parts and the heat transfer compound are not getting younger... If you can live with the noise, try a custom fan curve in afterburner.
Top
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Re: Bad work unit??

Post by stratocastor »

Additionally, and the root of the problem, I had to disassemble my water loop due to some buildup of gunk in the loop. Have been back on air cooling last two weeks. This problem started last week. When did the cuda update in folding drop again?? Temps are 73 C on the GPU, max 75C.. I have parts arriving in the next few days, and will rebuild. My fear is that from years of on and off folding, and gaming, my GPU might be about to the point of an upgrade. If I can squeak it out another month to where I can actually get my hands on an 3080, that would be the big payoff. But going back on the chill liquid might help out as well!! Thank you for the information @NormalDIffusion!!
Top
JohnChodera
Pande Group Member
Posts: 467
Joined: Fri Feb 22, 2013 9:59 pm

Re: Bad work unit??

Post by JohnChodera »

> When did the cuda update in folding drop again?? T

core22 0.0.13 with CUDA support rolled out on Mon 28 Sep for most folks (though BETA users had it more than a week earlier).

Is just this project giving you trouble, or all of them?

~ John Chodera // MSKCC
Top
NormalDiffusion
Posts: 124
Joined: Sat Apr 18, 2020 1:50 pm

Re: Bad work unit??

Post by NormalDiffusion »

stratocastor wrote:Additionally, and the root of the problem, I had to disassemble my water loop due to some buildup of gunk in the loop. Have been back on air cooling last two weeks. This problem started last week. When did the cuda update in folding drop again?? Temps are 73 C on the GPU, max 75C..
So you changed back your card from water to air? Check if the VRAM and VRM have good contact with the cooler and maybe renew the pads if you still have the originals (that I assume were sitting around unused on the air cooler for years).
Top
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:
Contact PantherX

Re: Particle coordinate is nan???

Post by PantherX »

Is your GPU overclocked (either factory or manually)? Have happens if you reduce the frequencies of the GPU?

Encountering a NAN once in a blue moon is fine. Encountering it consecutively, indicates an unstable GPU caused by; unstable GPU, PSU issues, failing GPU, overheating, etc.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Bad work unit??

Post by bruce »

stratocastor wrote:... my GPU might be about to the point of an upgrade....
We never argue with someone considering an upgrade ... but it's always going to be your choice, not ours.

If the increasing GPU demands of CUDA are causing temporary problems, you can "--disable-cuda" and it will go back to using OpenCL only (until the parts arrive) ... or a fan-curve adjustment may help.d Of course there's always the option of un-overclocking.

Temporary solutions abound ... and you might learn something that helps iyou n arriving at a new optimum setting.
Top
stratocastor
Posts: 20
Joined: Tue Aug 02, 2016 10:37 pm

Re: Particle coordinate is nan???

Post by stratocastor »

Is factory overclocked. Has worked flawlessly for years. Have tried Aida64 extreme, some mining with nice hash, and every other stress test for the GPU I could think of. Folding at home is the only program having issues currently. Have even tried downclocking the GPU... Still happens... Something about the cuda implementation...
Top
Post Reply

Return to “Issues with a specific WU”