Page 1 of 1

Project: 13415 (Run 4078, Clone 2, Gen 0)

Posted: Fri Jun 26, 2020 12:13 pm
by GregC
I was the fifth donor failing this work unit. Thought it would be useful for y'all to hold on to the logs.

Code: Select all

03:37:41:WU00:FS01:0x22:*********************** Log Started 2020-06-26T03:37:41Z ***********************
03:37:41:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
03:37:41:WU00:FS01:0x22:       Core: Core22
03:37:41:WU00:FS01:0x22:       Type: 0x22
03:37:41:WU00:FS01:0x22:    Version: 0.0.10
03:37:41:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
03:37:41:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
03:37:41:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
03:37:41:WU00:FS01:0x22:       Date: Jun 16 2020
03:37:41:WU00:FS01:0x22:       Time: 14:33:22
03:37:41:WU00:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
03:37:41:WU00:FS01:0x22:     Branch: core22-0.0.10
03:37:41:WU00:FS01:0x22:   Compiler: Visual C++ 2015
03:37:41:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:37:41:WU00:FS01:0x22:   Platform: win32 10
03:37:41:WU00:FS01:0x22:       Bits: 64
03:37:41:WU00:FS01:0x22:       Mode: Release
03:37:41:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
03:37:41:WU00:FS01:0x22:             <peastman@stanford.edu>
03:37:41:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 9748 -checkpoint 30
03:37:41:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
03:37:41:WU00:FS01:0x22:************************************ libFAH ************************************
03:37:41:WU00:FS01:0x22:       Date: Jun 15 2020
03:37:41:WU00:FS01:0x22:       Time: 18:05:04
03:37:41:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
03:37:41:WU00:FS01:0x22:     Branch: HEAD
03:37:41:WU00:FS01:0x22:   Compiler: Visual C++ 2015
03:37:41:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:37:41:WU00:FS01:0x22:   Platform: win32 10
03:37:41:WU00:FS01:0x22:       Bits: 64
03:37:41:WU00:FS01:0x22:       Mode: Release
03:37:41:WU00:FS01:0x22:************************************ CBang *************************************
03:37:41:WU00:FS01:0x22:       Date: Jun 16 2020
03:37:41:WU00:FS01:0x22:       Time: 14:31:33
03:37:41:WU00:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
03:37:41:WU00:FS01:0x22:     Branch: HEAD
03:37:41:WU00:FS01:0x22:   Compiler: Visual C++ 2015
03:37:41:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
03:37:41:WU00:FS01:0x22:   Platform: win32 10
03:37:41:WU00:FS01:0x22:       Bits: 64
03:37:41:WU00:FS01:0x22:       Mode: Release
03:37:41:WU00:FS01:0x22:************************************ System ************************************
03:37:41:WU00:FS01:0x22:        CPU: AMD Athlon(tm) II X2 245 Processor
03:37:41:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 16 Model 6 Stepping 3
03:37:41:WU00:FS01:0x22:       CPUs: 2
03:37:41:WU00:FS01:0x22:     Memory: 4.00GiB
03:37:41:WU00:FS01:0x22:Free Memory: 1.20GiB
03:37:41:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
03:37:41:WU00:FS01:0x22: OS Version: 6.2
03:37:41:WU00:FS01:0x22:Has Battery: false
03:37:41:WU00:FS01:0x22: On Battery: false
03:37:41:WU00:FS01:0x22: UTC Offset: -5
03:37:41:WU00:FS01:0x22:        PID: 5736
03:37:41:WU00:FS01:0x22:        CWD: C:\Users\BlueT\AppData\Roaming\FAHClient\work
03:37:41:WU00:FS01:0x22:********************************************************************************
03:37:41:WU00:FS01:0x22:Project: 13415 (Run 4078, Clone 2, Gen 0)
03:37:41:WU00:FS01:0x22:Unit: 0x0000000412bc7d9a5ef50d12bc7c7b7f
03:37:41:WU00:FS01:0x22:Reading tar file core.xml
03:37:41:WU00:FS01:0x22:Reading tar file integrator.xml
03:37:41:WU00:FS01:0x22:Reading tar file state.xml
03:37:41:WU00:FS01:0x22:Reading tar file system.xml
03:37:41:WU00:FS01:0x22:Digital signatures verified
03:37:41:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
03:37:41:WU00:FS01:0x22:Version 0.0.10
03:37:41:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
03:37:41:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
03:37:41:WU00:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
03:37:41:WU00:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
03:37:42:WU01:FS01:Upload complete
03:37:42:WU01:FS01:Server responded WORK_ACK (400)
03:37:42:WU01:FS01:Final credit estimate, 7229.00 points
03:37:42:WU01:FS01:Cleaning up
03:37:56:WU00:FS01:0x22:Completed 0 out of 1000000 steps (0%)
03:40:11:WU00:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
etc.etc.

03:57:28:WU00:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
******************************* Date: 2020-06-26 *******************************
03:59:37:WU00:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
etc.etc.

05:24:14:WU00:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
05:26:28:WU00:FS01:0x22:Completed 490000 out of 1000000 steps (49%)
05:26:44:WU00:FS01:0x22:An exception occurred at step 491206: Particle coordinate is nan
05:26:44:WU00:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
05:26:44:WU00:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
05:26:44:WARNING:WU00:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
05:26:45:WU00:FS01:Starting
05:26:45:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\BlueT\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 2004 -checkpoint 30 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
05:26:45:WU00:FS01:Started FahCore on PID 10108
05:26:45:WU00:FS01:Core PID:1212
05:26:45:WU00:FS01:FahCore 0x22 started
05:26:45:WU00:FS01:0x22:*********************** Log Started 2020-06-26T05:26:45Z ***********************
05:26:45:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
05:26:45:WU00:FS01:0x22:       Core: Core22
05:26:45:WU00:FS01:0x22:       Type: 0x22
05:26:45:WU00:FS01:0x22:    Version: 0.0.10
05:26:45:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
05:26:45:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
05:26:45:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
05:26:45:WU00:FS01:0x22:       Date: Jun 16 2020
05:26:45:WU00:FS01:0x22:       Time: 14:33:22
05:26:45:WU00:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
05:26:45:WU00:FS01:0x22:     Branch: core22-0.0.10
05:26:45:WU00:FS01:0x22:   Compiler: Visual C++ 2015
05:26:45:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
05:26:45:WU00:FS01:0x22:   Platform: win32 10
05:26:45:WU00:FS01:0x22:       Bits: 64
05:26:45:WU00:FS01:0x22:       Mode: Release
05:26:45:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
05:26:45:WU00:FS01:0x22:             <peastman@stanford.edu>
05:26:45:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 10108 -checkpoint 30
05:26:45:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
05:26:45:WU00:FS01:0x22:************************************ libFAH ************************************
05:26:45:WU00:FS01:0x22:       Date: Jun 15 2020
05:26:45:WU00:FS01:0x22:       Time: 18:05:04
05:26:45:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
05:26:45:WU00:FS01:0x22:     Branch: HEAD
05:26:45:WU00:FS01:0x22:   Compiler: Visual C++ 2015
05:26:45:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
05:26:45:WU00:FS01:0x22:   Platform: win32 10
05:26:45:WU00:FS01:0x22:       Bits: 64
05:26:45:WU00:FS01:0x22:       Mode: Release
05:26:45:WU00:FS01:0x22:************************************ CBang *************************************
05:26:45:WU00:FS01:0x22:       Date: Jun 16 2020
05:26:45:WU00:FS01:0x22:       Time: 14:31:33
05:26:45:WU00:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
05:26:45:WU00:FS01:0x22:     Branch: HEAD
05:26:45:WU00:FS01:0x22:   Compiler: Visual C++ 2015
05:26:45:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
05:26:45:WU00:FS01:0x22:   Platform: win32 10
05:26:45:WU00:FS01:0x22:       Bits: 64
05:26:45:WU00:FS01:0x22:       Mode: Release
05:26:45:WU00:FS01:0x22:************************************ System ************************************
05:26:45:WU00:FS01:0x22:        CPU: AMD Athlon(tm) II X2 245 Processor
05:26:45:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 16 Model 6 Stepping 3
05:26:45:WU00:FS01:0x22:       CPUs: 2
05:26:45:WU00:FS01:0x22:     Memory: 4.00GiB
05:26:45:WU00:FS01:0x22:Free Memory: 1.13GiB
05:26:45:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
05:26:45:WU00:FS01:0x22: OS Version: 6.2
05:26:45:WU00:FS01:0x22:Has Battery: false
05:26:45:WU00:FS01:0x22: On Battery: false
05:26:45:WU00:FS01:0x22: UTC Offset: -5
05:26:45:WU00:FS01:0x22:        PID: 1212
05:26:45:WU00:FS01:0x22:        CWD: C:\Users\BlueT\AppData\Roaming\FAHClient\work
05:26:45:WU00:FS01:0x22:********************************************************************************
05:26:45:WU00:FS01:0x22:Project: 13415 (Run 4078, Clone 2, Gen 0)
05:26:45:WU00:FS01:0x22:Unit: 0x0000000412bc7d9a5ef50d12bc7c7b7f
05:26:45:WU00:FS01:0x22:Digital signatures verified
05:26:45:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
05:26:45:WU00:FS01:0x22:Version 0.0.10
05:26:45:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
05:26:45:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
05:26:45:WU00:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
05:26:45:WU00:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
05:27:00:WU00:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
05:29:15:WU00:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
05:31:29:WU00:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
05:33:42:WU00:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
05:35:54:WU00:FS01:0x22:Completed 490000 out of 1000000 steps (49%)
05:38:08:WU00:FS01:0x22:Completed 500000 out of 1000000 steps (50%)
05:40:18:WU00:FS01:0x22:Completed 510000 out of 1000000 steps (51%)
05:42:28:WU00:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
05:42:42:WU00:FS01:0x22:An exception occurred at step 521075: Particle coordinate is nan
05:42:42:WU00:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
05:42:42:WU00:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
05:42:42:WARNING:WU00:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
05:42:42:WU00:FS01:Starting
05:42:42:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\BlueT\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 2004 -checkpoint 30 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
05:42:42:WU00:FS01:Started FahCore on PID 1396
05:42:42:WU00:FS01:Core PID:3320
05:42:42:WU00:FS01:FahCore 0x22 started
05:42:43:WU00:FS01:0x22:*********************** Log Started 2020-06-26T05:42:42Z ***********************
05:42:43:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
05:42:43:WU00:FS01:0x22:       Core: Core22
05:42:43:WU00:FS01:0x22:       Type: 0x22
05:42:43:WU00:FS01:0x22:    Version: 0.0.10
05:42:43:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
05:42:43:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
05:42:43:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
05:42:43:WU00:FS01:0x22:       Date: Jun 16 2020
05:42:43:WU00:FS01:0x22:       Time: 14:33:22
05:42:43:WU00:FS01:0x22:   Revision: 147051aad40bcbec7d4b25105bbedfab425f1dc2
05:42:43:WU00:FS01:0x22:     Branch: core22-0.0.10
05:42:43:WU00:FS01:0x22:   Compiler: Visual C++ 2015
05:42:43:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
05:42:43:WU00:FS01:0x22:   Platform: win32 10
05:42:43:WU00:FS01:0x22:       Bits: 64
05:42:43:WU00:FS01:0x22:       Mode: Release
05:42:43:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
05:42:43:WU00:FS01:0x22:             <peastman@stanford.edu>
05:42:43:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 1396 -checkpoint 30
05:42:43:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
05:42:43:WU00:FS01:0x22:************************************ libFAH ************************************
05:42:43:WU00:FS01:0x22:       Date: Jun 15 2020
05:42:43:WU00:FS01:0x22:       Time: 18:05:04
05:42:43:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
05:42:43:WU00:FS01:0x22:     Branch: HEAD
05:42:43:WU00:FS01:0x22:   Compiler: Visual C++ 2015
05:42:43:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
05:42:43:WU00:FS01:0x22:   Platform: win32 10
05:42:43:WU00:FS01:0x22:       Bits: 64
05:42:43:WU00:FS01:0x22:       Mode: Release
05:42:43:WU00:FS01:0x22:************************************ CBang *************************************
05:42:43:WU00:FS01:0x22:       Date: Jun 16 2020
05:42:43:WU00:FS01:0x22:       Time: 14:31:33
05:42:43:WU00:FS01:0x22:   Revision: 75fcee0b8e713cb47f5191a3689d5f4f07244c7f
05:42:43:WU00:FS01:0x22:     Branch: HEAD
05:42:43:WU00:FS01:0x22:   Compiler: Visual C++ 2015
05:42:43:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
05:42:43:WU00:FS01:0x22:   Platform: win32 10
05:42:43:WU00:FS01:0x22:       Bits: 64
05:42:43:WU00:FS01:0x22:       Mode: Release
05:42:43:WU00:FS01:0x22:************************************ System ************************************
05:42:43:WU00:FS01:0x22:        CPU: AMD Athlon(tm) II X2 245 Processor
05:42:43:WU00:FS01:0x22:     CPU ID: AuthenticAMD Family 16 Model 6 Stepping 3
05:42:43:WU00:FS01:0x22:       CPUs: 2
05:42:43:WU00:FS01:0x22:     Memory: 4.00GiB
05:42:43:WU00:FS01:0x22:Free Memory: 1.13GiB
05:42:43:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
05:42:43:WU00:FS01:0x22: OS Version: 6.2
05:42:43:WU00:FS01:0x22:Has Battery: false
05:42:43:WU00:FS01:0x22: On Battery: false
05:42:43:WU00:FS01:0x22: UTC Offset: -5
05:42:43:WU00:FS01:0x22:        PID: 3320
05:42:43:WU00:FS01:0x22:        CWD: C:\Users\BlueT\AppData\Roaming\FAHClient\work
05:42:43:WU00:FS01:0x22:********************************************************************************
05:42:43:WU00:FS01:0x22:Project: 13415 (Run 4078, Clone 2, Gen 0)
05:42:43:WU00:FS01:0x22:Unit: 0x0000000412bc7d9a5ef50d12bc7c7b7f
05:42:43:WU00:FS01:0x22:Digital signatures verified
05:42:43:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
05:42:43:WU00:FS01:0x22:Version 0.0.10
05:42:43:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
05:42:43:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
05:42:43:WU00:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
05:42:43:WU00:FS01:0x22:  Global context and integrator variables write interval: 250 steps (0.025%) [4000 total]
05:42:57:WU00:FS01:0x22:Completed 500000 out of 1000000 steps (50%)
05:45:08:WU00:FS01:0x22:Completed 510000 out of 1000000 steps (51%)
05:47:18:WU00:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
05:49:27:WU00:FS01:0x22:Completed 530000 out of 1000000 steps (53%)
05:50:17:WU00:FS01:0x22:An exception occurred at step 533876: Particle coordinate is nan
05:50:17:WU00:FS01:0x22:Max number of attempts to resume from last checkpoint (2) reached. Aborting.
05:50:17:WU00:FS01:0x22:ERROR:114: Max number of attempts to resume from last checkpoint reached.
05:50:17:WU00:FS01:0x22:Saving result file ..\logfile_01.txt
05:50:17:WU00:FS01:0x22:Saving result file globals.csv
05:50:17:WU00:FS01:0x22:Saving result file science.log
05:50:17:WU00:FS01:0x22:Saving result file state.xml
05:50:17:WU00:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
05:50:18:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
05:50:18:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:13415 run:4078 clone:2 gen:0 core:0x22 unit:0x0000000412bc7d9a5ef50d12bc7c7b7f
05:50:18:WU00:FS01:Uploading 450.04KiB to 18.188.125.154
05:50:18:WU00:FS01:Connecting to 18.188.125.154:8080
05:50:18:WU01:FS01:Connecting to assign1.foldingathome.org:80
05:50:18:WU01:FS01:Assigned to work server 18.188.125.154
05:50:18:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:Baffin [Polaris11] from 18.188.125.154
05:50:18:WU01:FS01:Connecting to 18.188.125.154:8080
05:50:19:WU00:FS01:Upload complete
05:50:19:WU00:FS01:Server responded WORK_ACK (400)
If there's a desire to capture anything in the debugger, I'm willing to help.

Re: Project: 13415 (Run 4078, Clone 2, Gen 0)

Posted: Fri Jun 26, 2020 6:05 pm
by Joe_H
If the WU error report uploaded okay, then they will be getting some data back about the error already.

I do know they are looking for debugger assistance on a different error related to this group of projects where the report is not getting uploaded, see this topic - viewtopic.php?f=19&t=35658 - and look for responses from JohnChodera.

The group of projects that 13415 is part of is using some new methods and is a bit experimental. So a higher rate of errors is expected. If your GPU is able to complete other projects without getting NaN errors you should be fine. Otherwise if the card is overclocked, including from the factory over the reference card, consider reducing that overclock a bit.