BAD_WORK_UNIT on p13416

[uyaem]

Project 13416 (Run 276, Clone 82, Gen 0)

Full log below.

Code: Select all

18:13:34:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13416 run:276 clone:82 gen:0 core:0x22 unit:0x0000000412bc7d9a5f00a7f1a726a5ad
18:13:34:WU02:FS01:Starting
18:13:34:WU02:FS01:Running FahCore: \"C:\\Program Files (x86)\\FAHClient/FAHCoreWrapper.exe\" C:\\Users\\X\\AppData\\Roaming\\FAHClient\\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 11704 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
18:13:34:WU02:FS01:Started FahCore on PID 13364
18:13:34:WU02:FS01:Core PID:16128
18:13:34:WU02:FS01:FahCore 0x22 started
18:13:35:WU02:FS01:0x22:*********************** Log Started 2020-07-06T18:13:34Z ***********************
18:13:35:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
18:13:35:WU02:FS01:0x22:       Core: Core22
18:13:35:WU02:FS01:0x22:       Type: 0x22
18:13:35:WU02:FS01:0x22:    Version: 0.0.11
18:13:35:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:13:35:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
18:13:35:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
18:13:35:WU02:FS01:0x22:       Date: Jun 26 2020
18:13:35:WU02:FS01:0x22:       Time: 19:49:16
18:13:35:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
18:13:35:WU02:FS01:0x22:     Branch: core22-0.0.11
18:13:35:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:13:35:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:13:35:WU02:FS01:0x22:   Platform: win32 10
18:13:35:WU02:FS01:0x22:       Bits: 64
18:13:35:WU02:FS01:0x22:       Mode: Release
18:13:35:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
18:13:35:WU02:FS01:0x22:             <peastman@stanford.edu>
18:13:35:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 13364 -checkpoint 15
18:13:35:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
18:13:35:WU02:FS01:0x22:             0 -gpu 0
18:13:35:WU02:FS01:0x22:************************************ libFAH ************************************
18:13:35:WU02:FS01:0x22:       Date: Jun 26 2020
18:13:35:WU02:FS01:0x22:       Time: 19:47:12
18:13:35:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
18:13:35:WU02:FS01:0x22:     Branch: HEAD
18:13:35:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:13:35:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:13:35:WU02:FS01:0x22:   Platform: win32 10
18:13:35:WU02:FS01:0x22:       Bits: 64
18:13:35:WU02:FS01:0x22:       Mode: Release
18:13:35:WU02:FS01:0x22:************************************ CBang *************************************
18:13:35:WU02:FS01:0x22:       Date: Jun 26 2020
18:13:35:WU02:FS01:0x22:       Time: 19:46:11
18:13:35:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
18:13:35:WU02:FS01:0x22:     Branch: master
18:13:35:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:13:35:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:13:35:WU02:FS01:0x22:   Platform: win32 10
18:13:35:WU02:FS01:0x22:       Bits: 64
18:13:35:WU02:FS01:0x22:       Mode: Release
18:13:35:WU02:FS01:0x22:************************************ System ************************************
18:13:35:WU02:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
18:13:35:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
18:13:35:WU02:FS01:0x22:       CPUs: 24
18:13:35:WU02:FS01:0x22:     Memory: 31.95GiB
18:13:35:WU02:FS01:0x22:Free Memory: 20.54GiB
18:13:35:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
18:13:35:WU02:FS01:0x22: OS Version: 6.2
18:13:35:WU02:FS01:0x22:Has Battery: false
18:13:35:WU02:FS01:0x22: On Battery: false
18:13:35:WU02:FS01:0x22: UTC Offset: 2
18:13:35:WU02:FS01:0x22:        PID: 16128
18:13:35:WU02:FS01:0x22:        CWD: C:\\Users\\X\\AppData\\Roaming\\FAHClient\\work
18:13:35:WU02:FS01:0x22:********************************************************************************
18:13:35:WU02:FS01:0x22:Project: 13416 (Run 276, Clone 82, Gen 0)
18:13:35:WU02:FS01:0x22:Unit: 0x0000000412bc7d9a5f00a7f1a726a5ad
18:13:35:WU02:FS01:0x22:Reading tar file core.xml
18:13:35:WU02:FS01:0x22:Reading tar file integrator.xml
18:13:35:WU02:FS01:0x22:Reading tar file state.xml.bz2
18:13:35:WU02:FS01:0x22:Reading tar file system.xml.bz2
18:13:35:WU02:FS01:0x22:Digital signatures verified
18:13:35:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
18:13:35:WU02:FS01:0x22:Version 0.0.11
18:13:35:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
18:13:35:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
18:13:35:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
18:13:35:WU02:FS01:0x22:  Global context and integrator variables write interval: 2500 steps (0.25%) [400 total]
18:13:39:WU00:FS01:Upload complete
18:13:39:WU00:FS01:Server responded WORK_ACK (400)
18:13:39:WU00:FS01:Final credit estimate, 135127.00 points
18:13:39:WU00:FS01:Cleaning up
18:13:55:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
18:14:46:WU02:FS01:0x22:An exception occurred at step 1756: Particle coordinate is nan
18:14:46:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
18:14:46:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
18:14:47:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
18:14:47:WU02:FS01:Starting
18:14:47:WU02:FS01:Running FahCore: \"C:\\Program Files (x86)\\FAHClient/FAHCoreWrapper.exe\" C:\\Users\\X\\AppData\\Roaming\\FAHClient\\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 11704 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
18:14:47:WU02:FS01:Started FahCore on PID 16888
18:14:47:WU02:FS01:Core PID:15768
18:14:47:WU02:FS01:FahCore 0x22 started
18:14:48:WU02:FS01:0x22:*********************** Log Started 2020-07-06T18:14:47Z ***********************
18:14:48:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
18:14:48:WU02:FS01:0x22:       Core: Core22
18:14:48:WU02:FS01:0x22:       Type: 0x22
18:14:48:WU02:FS01:0x22:    Version: 0.0.11
18:14:48:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:14:48:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
18:14:48:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
18:14:48:WU02:FS01:0x22:       Date: Jun 26 2020
18:14:48:WU02:FS01:0x22:       Time: 19:49:16
18:14:48:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
18:14:48:WU02:FS01:0x22:     Branch: core22-0.0.11
18:14:48:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:14:48:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:14:48:WU02:FS01:0x22:   Platform: win32 10
18:14:48:WU02:FS01:0x22:       Bits: 64
18:14:48:WU02:FS01:0x22:       Mode: Release
18:14:48:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
18:14:48:WU02:FS01:0x22:             <peastman@stanford.edu>
18:14:48:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 16888 -checkpoint 15
18:14:48:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
18:14:48:WU02:FS01:0x22:             0 -gpu 0
18:14:48:WU02:FS01:0x22:************************************ libFAH ************************************
18:14:48:WU02:FS01:0x22:       Date: Jun 26 2020
18:14:48:WU02:FS01:0x22:       Time: 19:47:12
18:14:48:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
18:14:48:WU02:FS01:0x22:     Branch: HEAD
18:14:48:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:14:48:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:14:48:WU02:FS01:0x22:   Platform: win32 10
18:14:48:WU02:FS01:0x22:       Bits: 64
18:14:48:WU02:FS01:0x22:       Mode: Release
18:14:48:WU02:FS01:0x22:************************************ CBang *************************************
18:14:48:WU02:FS01:0x22:       Date: Jun 26 2020
18:14:48:WU02:FS01:0x22:       Time: 19:46:11
18:14:48:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
18:14:48:WU02:FS01:0x22:     Branch: master
18:14:48:WU02:FS01:0x22:   Compiler: Visual C++ 2015
18:14:48:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
18:14:48:WU02:FS01:0x22:   Platform: win32 10
18:14:48:WU02:FS01:0x22:       Bits: 64
18:14:48:WU02:FS01:0x22:       Mode: Release
18:14:48:WU02:FS01:0x22:************************************ System ************************************
18:14:48:WU02:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
18:14:48:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
18:14:48:WU02:FS01:0x22:       CPUs: 24
18:14:48:WU02:FS01:0x22:     Memory: 31.95GiB
18:14:48:WU02:FS01:0x22:Free Memory: 20.57GiB
18:14:48:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
18:14:48:WU02:FS01:0x22: OS Version: 6.2
18:14:48:WU02:FS01:0x22:Has Battery: false
18:14:48:WU02:FS01:0x22: On Battery: false
18:14:48:WU02:FS01:0x22: UTC Offset: 2
18:14:48:WU02:FS01:0x22:        PID: 15768
18:14:48:WU02:FS01:0x22:        CWD: C:\\Users\\X\\AppData\\Roaming\\FAHClient\\work
18:14:48:WU02:FS01:0x22:********************************************************************************
18:14:48:WU02:FS01:0x22:Project: 13416 (Run 276, Clone 82, Gen 0)
18:14:48:WU02:FS01:0x22:Unit: 0x0000000412bc7d9a5f00a7f1a726a5ad
18:14:48:WU02:FS01:0x22:Digital signatures verified
18:14:48:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
18:14:48:WU02:FS01:0x22:Version 0.0.11
18:14:48:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
18:14:48:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
18:14:48:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
18:14:48:WU02:FS01:0x22:  Global context and integrator variables write interval: 2500 steps (0.25%) [400 total]
18:15:04:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
18:15:53:WU02:FS01:0x22:An exception occurred at step 1756: Particle coordinate is nan
18:15:53:WU02:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
18:15:53:WU02:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
18:15:53:WU02:FS01:0x22:Saving result file ..\\logfile_01.txt
18:15:53:WU02:FS01:0x22:Saving result file globals.csv
18:15:53:WU02:FS01:0x22:Saving result file science.log
18:15:53:WU02:FS01:0x22:Saving result file state.xml.bz2
18:15:53:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
18:15:54:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
18:15:54:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13416 run:276 clone:82 gen:0 core:0x22 unit:0x0000000412bc7d9a5f00a7f1a726a5ad

[JimboPalmer]

You do not show the first 200 lines of your log, so we are unable to help much. You are trying to use Core_22, but do not show us your GPU or Driver versions, as examples. The configuration at the front of the log really helps.

viewtopic.php?f=24&t=26036

Code: Select all

18:15:53:WU02:FS01:0x22:An exception occurred at step 1756: Particle coordinate is nan

NaN is Not a Number.

https://en.wikipedia.org/wiki/NaN

[uyaem]

Hi Jimbo, I didn't necessarily post to seek help, but I was told that due to the experimental nature of 134xx, failures were to be expected, so wanted to report this one.
I'm 99.99% confident it has nothing to do with my setup, I've processed >250 GPU WUs fine in the past 2 months.

For the sake of completion:

Code: Select all

*********************** Log Started 2020-06-19T20:38:46Z ***********************
20:38:46:Trying to access database...
20:38:46:Successfully acquired database lock
20:38:46:Read GPUs.txt
20:38:46:Enabled folding slot 00: PAUSED cpu:21 (by user)
20:38:46:Enabled folding slot 01: PAUSED gpu:0:TU116 [GeForce GTX 1660 SUPER] (by user)
20:38:47:****************************** FAHClient ******************************
20:38:47:        Version: 7.6.13
20:38:47:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
20:38:47:      Copyright: 2020 foldingathome.org
20:38:47:       Homepage: https://foldingathome.org/
20:38:47:           Date: Apr 27 2020
20:38:47:           Time: 21:21:01
20:38:47:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
20:38:47:         Branch: master
20:38:47:       Compiler: Visual C++ 2008
20:38:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
20:38:47:       Platform: win32 10
20:38:47:           Bits: 32
20:38:47:           Mode: Release
20:38:47:           Args: --open-web-control
20:38:47:         Config: C:\\Users\\X\\AppData\\Roaming\\FAHClient\\config.xml
20:38:47:******************************** CBang ********************************
20:38:47:           Date: Apr 24 2020
20:38:47:           Time: 17:07:55
20:38:47:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
20:38:47:         Branch: master
20:38:47:       Compiler: Visual C++ 2008
20:38:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
20:38:47:       Platform: win32 10
20:38:47:           Bits: 32
20:38:47:           Mode: Release
20:38:47:******************************* System ********************************
20:38:47:            CPU: AMD Ryzen 9 3900X 12-Core Processor
20:38:47:         CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
20:38:47:           CPUs: 24
20:38:47:         Memory: 31.95GiB
20:38:47:    Free Memory: 28.15GiB
20:38:47:        Threads: WINDOWS_THREADS
20:38:47:     OS Version: 6.2
20:38:47:    Has Battery: false
20:38:47:     On Battery: false
20:38:47:     UTC Offset: 2
20:38:47:            PID: 11704
20:38:47:            CWD: C:\\Users\\X\\AppData\\Roaming\\FAHClient
20:38:47:  Win32 Service: false
20:38:47:             OS: Windows 10 Home
20:38:47:        OS Arch: AMD64
20:38:47:           GPUs: 1
20:38:47:          GPU 0: Bus:45 Slot:0 Func:0 NVIDIA:7 TU116 [GeForce GTX 1660 SUPER]
20:38:47:  CUDA Device 0: Platform:0 Device:0 Bus:45 Slot:0 Compute:7.5 Driver:11.0
20:38:47:OpenCL Device 0: Platform:0 Device:0 Bus:45 Slot:0 Compute:1.2 Driver:446.14
20:38:47:******************************* libFAH ********************************
20:38:47:           Date: Apr 15 2020
20:38:47:           Time: 14:53:14
20:38:47:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
20:38:47:         Branch: master
20:38:47:       Compiler: Visual C++ 2008
20:38:47:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
20:38:47:       Platform: win32 10
20:38:47:           Bits: 32
20:38:47:           Mode: Release
20:38:47:***********************************************************************
20:38:47:<config>
20:38:47:  <!-- Folding Slot Configuration -->
20:38:47:  <cause v='COVID_19'/>
20:38:47:  <client-type v='advanced'/>
20:38:47:
20:38:47:  <!-- HTTP Server -->
20:38:47:  <allow v='127.0.0.1 192.168.178.0/24 192.168.0.0/24'/>
20:38:47:
20:38:47:  <!-- Network -->
20:38:47:  <proxy v=':8080'/>
20:38:47:
20:38:47:  <!-- Slot Control -->
20:38:47:  <pause-on-battery v='false'/>
20:38:47:
20:38:47:  <!-- User Information -->
20:38:47:  <passkey v='*****'/>
20:38:47:  <team v='243774'/>
20:38:47:  <user v='Uyaem'/>
20:38:47:
20:38:47:  <!-- Folding Slots -->
20:38:47:  <slot id='0' type='CPU'>
20:38:47:    <cpus v='21'/>
20:38:47:    <next-unit-percentage v='100'/>
20:38:47:  </slot>
20:38:47:  <slot id='1' type='GPU'>
20:38:47:    <next-unit-percentage v='100'/>
20:38:47:  </slot>
20:38:47:</config>