Page 1 of 1
13405 (156, 45, 0) failed at 48%
Posted: Thu May 07, 2020 8:47 pm
by STFC9F22
Hi,
Project 13405 (Run 156, Clone 45, Gen 0) has just failed at 48% due to 'Following exception occured: Particle coordinate is nan'.
My machine is not overclocked and has had no previous failures as far as I am aware. In addition, the WU Status page for this WU indicates that it is the third failure of this WU today.
The relevant section of the log follows:-
Code: Select all
20:01:15:WU02:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
20:03:58:WU02:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
20:06:42:WU02:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
20:09:30:WU02:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
20:09:30:WU02:FS01:0x22:Following exception occured: Particle coordinate is nan
20:09:32:FS01:Finishing
20:09:45:WU02:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
20:09:45:WU02:FS01:0x22:Following exception occured: Particle coordinate is nan
20:10:01:WU02:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
20:10:01:WU02:FS01:0x22:Following exception occured: Particle coordinate is nan
20:10:01:WU02:FS01:0x22:ERROR:114: Max Retries Reached
20:10:01:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
20:10:01:WU02:FS01:0x22:Saving result file badstate-0.xml
20:10:01:WU02:FS01:0x22:Saving result file badstate-1.xml
20:10:01:WU02:FS01:0x22:Saving result file badstate-2.xml
20:10:01:WU02:FS01:0x22:Saving result file checkpointState.xml
20:10:02:WU02:FS01:0x22:Saving result file checkpt.crc
20:10:02:WU02:FS01:0x22:Saving result file globals.csv
20:10:02:WU02:FS01:0x22:Saving result file positions.xtc
20:10:02:WU02:FS01:0x22:Saving result file science.log
20:10:02:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
20:10:03:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
20:10:03:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13405 run:156 clone:45 gen:0 core:0x22 unit:0x0000000212bc7d9a5eb3a38b3d948bc1
20:10:03:WU02:FS01:Uploading 4.97MiB to 18.188.125.154
20:10:03:WU02:FS01:Connecting to 18.188.125.154:8080
20:10:09:WU02:FS01:Upload 13.83%
20:10:15:WU02:FS01:Upload 27.67%
20:10:21:WU02:FS01:Upload 40.25%
20:10:27:WU02:FS01:Upload 54.08%
20:10:33:WU02:FS01:Upload 67.91%
20:10:39:WU02:FS01:Upload 81.75%
20:10:45:WU02:FS01:Upload 94.33%
20:10:48:WU02:FS01:Upload complete
20:10:48:WU02:FS01:Server responded WORK_ACK (400)
20:10:48:WU02:FS01:Cleaning up
Re: 13405 (156, 45, 0) failed at 48%
Posted: Thu May 07, 2020 11:02 pm
by Rel25917
Probably a bad unit, once it is returned multiple time as faulty (5 or so maybe, not sure the exact number) the server will stop sending it.
Re: 13405 (156, 45, 0) failed at 48%
Posted: Fri May 08, 2020 12:37 am
by JohnChodera
These simulations do have a nonzero failure rate due to their nature. We're trying to iteratively refine them and make them more robust, but right now we need to gather data, analyze it, and identify what the main causes of non-robustness are.
~ John Chodera // MSKCC
Re: 13405 (156, 45, 0) failed at 48%
Posted: Fri May 08, 2020 9:20 am
by STFC9F22
JohnChodera wrote:These simulations do have a nonzero failure rate due to their nature. We're trying to iteratively refine them and make them more robust, but right now we need to gather data, analyze it, and identify what the main causes of non-robustness are.
~ John Chodera // MSKCC
I don't know whether the details of my set-up are relevant or accessible to you, but here is the full log of the session in which the failure occurred -
Code: Select all
*********************** Log Started 2020-05-07T15:38:11Z ***********************
15:38:11:Trying to access database...
15:38:11:Successfully acquired database lock
15:38:11:Read GPUs.txt
15:38:11:Enabled folding slot 01: PAUSED gpu:0:GP104 [GeForce GTX 1070] 6463 (by user)
15:38:11:Enabled folding slot 00: PAUSED cpu:4 (by user)
15:38:11:****************************** FAHClient ******************************
15:38:11: Version: 7.6.13
15:38:11: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:38:11: Copyright: 2020 foldingathome.org
15:38:11: Homepage: https://foldingathome.org/
15:38:11: Date: Apr 27 2020
15:38:11: Time: 21:21:01
15:38:11: Revision: 5a652817f46116b6e135503af97f18e094414e3b
15:38:11: Branch: master
15:38:11: Compiler: Visual C++ 2008
15:38:11: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:38:11: Platform: win32 10
15:38:11: Bits: 32
15:38:11: Mode: Release
15:38:11: Config: C:\Users\Malcolm\AppData\Roaming\FAHClient\config.xml
15:38:11:******************************** CBang ********************************
15:38:11: Date: Apr 24 2020
15:38:11: Time: 17:07:55
15:38:11: Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
15:38:11: Branch: master
15:38:11: Compiler: Visual C++ 2008
15:38:11: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:38:11: Platform: win32 10
15:38:11: Bits: 32
15:38:11: Mode: Release
15:38:11:******************************* System ********************************
15:38:11: CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
15:38:11: CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
15:38:11: CPUs: 8
15:38:11: Memory: 15.86GiB
15:38:11: Free Memory: 11.82GiB
15:38:11: Threads: WINDOWS_THREADS
15:38:11: OS Version: 6.2
15:38:11: Has Battery: false
15:38:11: On Battery: false
15:38:11: UTC Offset: 1
15:38:11: PID: 29212
15:38:11: CWD: C:\Users\Malcolm\AppData\Roaming\FAHClient
15:38:11: Win32 Service: false
15:38:11: OS: Windows 10 Home
15:38:11: OS Arch: AMD64
15:38:11: GPUs: 1
15:38:11: GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070] 6463
15:38:11: CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:11.0
15:38:11:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:445.75
15:38:11:OpenCL Device 1: Platform:1 Device:0 Bus:NA Slot:NA Compute:1.2 Driver:20.19
15:38:11:******************************* libFAH ********************************
15:38:11: Date: Apr 15 2020
15:38:11: Time: 14:53:14
15:38:11: Revision: 216968bc7025029c841ed6e36e81a03a316890d3
15:38:11: Branch: master
15:38:11: Compiler: Visual C++ 2008
15:38:11: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:38:11: Platform: win32 10
15:38:11: Bits: 32
15:38:11: Mode: Release
15:38:11:***********************************************************************
15:38:11:<config>
15:38:11: <!-- Folding Core -->
15:38:11: <checkpoint v='30'/>
15:38:11:
15:38:11: <!-- Folding Slot Configuration -->
15:38:11: <cause v='COVID_19'/>
15:38:11:
15:38:11: <!-- Network -->
15:38:11: <proxy v=':8080'/>
15:38:11:
15:38:11: <!-- Slot Control -->
15:38:11: <pause-on-start v='true'/>
15:38:11:
15:38:11: <!-- User Information -->
15:38:11: <passkey v='*****'/>
15:38:11: <user v='STFC9F22'/>
15:38:11:
15:38:11: <!-- Work Unit Control -->
15:38:11: <next-unit-percentage v='100'/>
15:38:11:
15:38:11: <!-- Folding Slots -->
15:38:11: <slot id='1' type='GPU'/>
15:38:11: <slot id='0' type='CPU'>
15:38:11: <cpus v='4'/>
15:38:11: </slot>
15:38:11:</config>
15:38:49:FS01:Unpaused
15:38:49:FS00:Unpaused
15:38:50:WU00:FS01:Connecting to assign1.foldingathome.org:80
15:38:50:WU01:FS00:Connecting to assign1.foldingathome.org:80
15:38:50:WU00:FS01:Assigned to work server 3.133.76.19
15:38:50:WU00:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 3.133.76.19
15:38:50:WU00:FS01:Connecting to 3.133.76.19:8080
15:38:50:WU01:FS00:Assigned to work server 128.252.203.2
15:38:50:WU01:FS00:Requesting new work unit for slot 00: READY cpu:4 from 128.252.203.2
15:38:50:WU01:FS00:Connecting to 128.252.203.2:8080
15:39:11:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
15:39:11:WU00:FS01:Connecting to 3.133.76.19:80
15:39:43:ERROR:WU01:FS00:Exception: Server did not assign work unit
15:39:43:WU01:FS00:Connecting to assign1.foldingathome.org:80
15:39:44:WU01:FS00:Assigned to work server 40.114.52.201
15:39:44:WU01:FS00:Requesting new work unit for slot 00: READY cpu:4 from 40.114.52.201
15:39:44:WU01:FS00:Connecting to 40.114.52.201:8080
15:39:44:WU01:FS00:Downloading 2.14MiB
15:39:47:WU01:FS00:Download complete
15:39:47:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13850 run:0 clone:38659 gen:100 core:0xa7 unit:0x00000078287234c95e78894153dba994
15:39:47:WU01:FS00:Starting
15:39:47:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Malcolm\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/avx/Core_a7.fah/FahCore_a7.exe -dir 01 -suffix 01 -version 706 -lifeline 29212 -checkpoint 30 -np 4
15:39:49:WU01:FS00:Started FahCore on PID 30900
15:39:49:WU01:FS00:Core PID:31048
15:39:49:WU01:FS00:FahCore 0xa7 started
15:39:49:WU01:FS00:0xa7:*********************** Log Started 2020-05-07T15:39:49Z ***********************
15:39:49:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
15:39:49:WU01:FS00:0xa7: Type: 0xa7
15:39:49:WU01:FS00:0xa7: Core: Gromacs
15:39:49:WU01:FS00:0xa7: Args: -dir 01 -suffix 01 -version 706 -lifeline 30900 -checkpoint 30 -np
15:39:49:WU01:FS00:0xa7: 4
15:39:49:WU01:FS00:0xa7:************************************ CBang *************************************
15:39:49:WU01:FS00:0xa7: Date: Oct 26 2019
15:39:49:WU01:FS00:0xa7: Time: 01:38:25
15:39:49:WU01:FS00:0xa7: Revision: c46a1a011a24143739ac7218c5a435f66777f62f
15:39:49:WU01:FS00:0xa7: Branch: master
15:39:49:WU01:FS00:0xa7: Compiler: Visual C++ 2008
15:39:49:WU01:FS00:0xa7: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:39:49:WU01:FS00:0xa7: Platform: win32 10
15:39:49:WU01:FS00:0xa7: Bits: 64
15:39:49:WU01:FS00:0xa7: Mode: Release
15:39:49:WU01:FS00:0xa7:************************************ System ************************************
15:39:49:WU01:FS00:0xa7: CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
15:39:49:WU01:FS00:0xa7: CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
15:39:49:WU01:FS00:0xa7: CPUs: 8
15:39:49:WU01:FS00:0xa7: Memory: 15.86GiB
15:39:49:WU01:FS00:0xa7:Free Memory: 11.65GiB
15:39:49:WU01:FS00:0xa7: Threads: WINDOWS_THREADS
15:39:49:WU01:FS00:0xa7: OS Version: 6.2
15:39:49:WU01:FS00:0xa7:Has Battery: false
15:39:49:WU01:FS00:0xa7: On Battery: false
15:39:49:WU01:FS00:0xa7: UTC Offset: 1
15:39:49:WU01:FS00:0xa7: PID: 31048
15:39:49:WU01:FS00:0xa7: CWD: C:\Users\Malcolm\AppData\Roaming\FAHClient\work
15:39:49:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
15:39:49:WU01:FS00:0xa7: Version: 0.0.18
15:39:49:WU01:FS00:0xa7: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:39:49:WU01:FS00:0xa7: Copyright: 2019 foldingathome.org
15:39:49:WU01:FS00:0xa7: Homepage: https://foldingathome.org/
15:39:49:WU01:FS00:0xa7: Date: Oct 26 2019
15:39:49:WU01:FS00:0xa7: Time: 01:52:30
15:39:49:WU01:FS00:0xa7: Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
15:39:49:WU01:FS00:0xa7: Branch: master
15:39:49:WU01:FS00:0xa7: Compiler: Visual C++ 2008
15:39:49:WU01:FS00:0xa7: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:39:49:WU01:FS00:0xa7: Platform: win32 10
15:39:49:WU01:FS00:0xa7: Bits: 64
15:39:49:WU01:FS00:0xa7: Mode: Release
15:39:49:WU01:FS00:0xa7:************************************ Build *************************************
15:39:49:WU01:FS00:0xa7: SIMD: avx_256
15:39:49:WU01:FS00:0xa7:********************************************************************************
15:39:49:WU01:FS00:0xa7:Project: 13850 (Run 0, Clone 38659, Gen 100)
15:39:49:WU01:FS00:0xa7:Unit: 0x00000078287234c95e78894153dba994
15:39:49:WU01:FS00:0xa7:Reading tar file core.xml
15:39:49:WU01:FS00:0xa7:Reading tar file frame100.tpr
15:39:49:WU01:FS00:0xa7:Digital signatures verified
15:39:49:WU01:FS00:0xa7:Calling: mdrun -s frame100.tpr -o frame100.trr -x frame100.xtc -e frame100.edr -cpt 30 -nt 4
15:39:49:WU01:FS00:0xa7:Steps: first=50000000 total=500000
15:39:50:WU01:FS00:0xa7:Completed 1 out of 500000 steps (0%)
15:40:08:WU00:FS01:Downloading 9.20MiB
15:40:14:WU00:FS01:Download 55.03%
15:40:18:WU00:FS01:Download complete
15:40:18:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16433 run:893 clone:0 gen:30 core:0x22 unit:0x0000002603854c135e9a4f003e33ad50
15:40:18:WU00:FS01:Starting
15:40:18:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Malcolm\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 29212 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
15:40:18:WU00:FS01:Started FahCore on PID 3188
15:40:18:WU00:FS01:Core PID:31172
15:40:18:WU00:FS01:FahCore 0x22 started
15:40:18:WU00:FS01:0x22:*********************** Log Started 2020-05-07T15:40:18Z ***********************
15:40:18:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
15:40:18:WU00:FS01:0x22: Type: 0x22
15:40:18:WU00:FS01:0x22: Core: Core22
15:40:18:WU00:FS01:0x22: Website: https://foldingathome.org/
15:40:18:WU00:FS01:0x22: Copyright: (c) 2009-2018 foldingathome.org
15:40:18:WU00:FS01:0x22: Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
15:40:18:WU00:FS01:0x22: <rafal.wiewiora@choderalab.org>
15:40:18:WU00:FS01:0x22: Args: -dir 00 -suffix 01 -version 706 -lifeline 3188 -checkpoint 30
15:40:18:WU00:FS01:0x22: -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
15:40:18:WU00:FS01:0x22: 0 -gpu 0
15:40:18:WU00:FS01:0x22: Config: <none>
15:40:18:WU00:FS01:0x22:************************************ Build *************************************
15:40:18:WU00:FS01:0x22: Version: 0.0.5
15:40:18:WU00:FS01:0x22: Date: Apr 22 2020
15:40:18:WU00:FS01:0x22: Time: 04:42:59
15:40:18:WU00:FS01:0x22: Repository: Git
15:40:18:WU00:FS01:0x22: Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
15:40:18:WU00:FS01:0x22: Branch: HEAD
15:40:18:WU00:FS01:0x22: Compiler: Visual C++ 2008
15:40:18:WU00:FS01:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:40:18:WU00:FS01:0x22: Platform: win32 10
15:40:18:WU00:FS01:0x22: Bits: 64
15:40:18:WU00:FS01:0x22: Mode: Release
15:40:18:WU00:FS01:0x22:************************************ System ************************************
15:40:18:WU00:FS01:0x22: CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
15:40:18:WU00:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
15:40:18:WU00:FS01:0x22: CPUs: 8
15:40:18:WU00:FS01:0x22: Memory: 15.86GiB
15:40:18:WU00:FS01:0x22:Free Memory: 11.00GiB
15:40:18:WU00:FS01:0x22: Threads: WINDOWS_THREADS
15:40:18:WU00:FS01:0x22: OS Version: 6.2
15:40:18:WU00:FS01:0x22:Has Battery: false
15:40:18:WU00:FS01:0x22: On Battery: false
15:40:18:WU00:FS01:0x22: UTC Offset: 1
15:40:18:WU00:FS01:0x22: PID: 31172
15:40:18:WU00:FS01:0x22: CWD: C:\Users\Malcolm\AppData\Roaming\FAHClient\work
15:40:18:WU00:FS01:0x22: OS: Windows 10 Home
15:40:18:WU00:FS01:0x22: OS Arch: AMD64
15:40:18:WU00:FS01:0x22:********************************************************************************
15:40:18:WU00:FS01:0x22:Project: 16433 (Run 893, Clone 0, Gen 30)
15:40:18:WU00:FS01:0x22:Unit: 0x0000002603854c135e9a4f003e33ad50
15:40:18:WU00:FS01:0x22:Reading tar file core.xml
15:40:18:WU00:FS01:0x22:Reading tar file integrator.xml
15:40:18:WU00:FS01:0x22:Reading tar file state.xml
15:40:18:WU00:FS01:0x22:Reading tar file system.xml
15:40:18:WU00:FS01:0x22:Digital signatures verified
15:40:18:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
15:40:18:WU00:FS01:0x22:Version 0.0.5
15:40:20:WU00:FS01:0x22:Completed 0 out of 8000000 steps (0%)
15:40:20:WU00:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
15:41:44:WU00:FS01:0x22:Completed 80000 out of 8000000 steps (1%)
15:42:36:WU01:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
15:43:06:WU00:FS01:0x22:Completed 160000 out of 8000000 steps (2%)
15:44:28:WU00:FS01:0x22:Completed 240000 out of 8000000 steps (3%)
15:45:20:WU01:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
15:45:49:WU00:FS01:0x22:Completed 320000 out of 8000000 steps (4%)
15:47:11:WU00:FS01:0x22:Completed 400000 out of 8000000 steps (5%)
15:48:05:WU01:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
15:48:33:WU00:FS01:0x22:Completed 480000 out of 8000000 steps (6%)
15:49:55:WU00:FS01:0x22:Completed 560000 out of 8000000 steps (7%)
15:50:50:WU01:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
15:51:16:WU00:FS01:0x22:Completed 640000 out of 8000000 steps (8%)
15:52:38:WU00:FS01:0x22:Completed 720000 out of 8000000 steps (9%)
15:53:35:WU01:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
15:54:00:WU00:FS01:0x22:Completed 800000 out of 8000000 steps (10%)
15:55:22:WU00:FS01:0x22:Completed 880000 out of 8000000 steps (11%)
15:56:20:WU01:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
15:56:44:WU00:FS01:0x22:Completed 960000 out of 8000000 steps (12%)
15:58:05:WU00:FS01:0x22:Completed 1040000 out of 8000000 steps (13%)
15:59:05:WU01:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
15:59:27:WU00:FS01:0x22:Completed 1120000 out of 8000000 steps (14%)
16:00:49:WU00:FS01:0x22:Completed 1200000 out of 8000000 steps (15%)
16:01:50:WU01:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
16:02:11:WU00:FS01:0x22:Completed 1280000 out of 8000000 steps (16%)
16:03:33:WU00:FS01:0x22:Completed 1360000 out of 8000000 steps (17%)
16:04:35:WU01:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
16:04:54:WU00:FS01:0x22:Completed 1440000 out of 8000000 steps (18%)
16:06:16:WU00:FS01:0x22:Completed 1520000 out of 8000000 steps (19%)
16:07:20:WU01:FS00:0xa7:Completed 50000 out of 500000 steps (10%)
16:07:39:WU00:FS01:0x22:Completed 1600000 out of 8000000 steps (20%)
16:09:01:WU00:FS01:0x22:Completed 1680000 out of 8000000 steps (21%)
16:10:05:WU01:FS00:0xa7:Completed 55000 out of 500000 steps (11%)
16:10:23:WU00:FS01:0x22:Completed 1760000 out of 8000000 steps (22%)
16:11:45:WU00:FS01:0x22:Completed 1840000 out of 8000000 steps (23%)
16:12:50:WU01:FS00:0xa7:Completed 60000 out of 500000 steps (12%)
16:13:07:WU00:FS01:0x22:Completed 1920000 out of 8000000 steps (24%)
16:14:29:WU00:FS01:0x22:Completed 2000000 out of 8000000 steps (25%)
16:15:35:WU01:FS00:0xa7:Completed 65000 out of 500000 steps (13%)
16:15:51:WU00:FS01:0x22:Completed 2080000 out of 8000000 steps (26%)
16:17:14:WU00:FS01:0x22:Completed 2160000 out of 8000000 steps (27%)
16:18:20:WU01:FS00:0xa7:Completed 70000 out of 500000 steps (14%)
16:18:36:WU00:FS01:0x22:Completed 2240000 out of 8000000 steps (28%)
16:19:58:WU00:FS01:0x22:Completed 2320000 out of 8000000 steps (29%)
16:21:04:WU01:FS00:0xa7:Completed 75000 out of 500000 steps (15%)
16:21:20:WU00:FS01:0x22:Completed 2400000 out of 8000000 steps (30%)
16:22:41:WU00:FS01:0x22:Completed 2480000 out of 8000000 steps (31%)
16:23:48:WU01:FS00:0xa7:Completed 80000 out of 500000 steps (16%)
16:24:03:WU00:FS01:0x22:Completed 2560000 out of 8000000 steps (32%)
16:25:25:WU00:FS01:0x22:Completed 2640000 out of 8000000 steps (33%)
16:26:33:WU01:FS00:0xa7:Completed 85000 out of 500000 steps (17%)
16:26:47:WU00:FS01:0x22:Completed 2720000 out of 8000000 steps (34%)
16:28:09:WU00:FS01:0x22:Completed 2800000 out of 8000000 steps (35%)
16:29:18:WU01:FS00:0xa7:Completed 90000 out of 500000 steps (18%)
16:29:31:WU00:FS01:0x22:Completed 2880000 out of 8000000 steps (36%)
16:30:53:WU00:FS01:0x22:Completed 2960000 out of 8000000 steps (37%)
16:32:03:WU01:FS00:0xa7:Completed 95000 out of 500000 steps (19%)
16:32:15:WU00:FS01:0x22:Completed 3040000 out of 8000000 steps (38%)
16:33:37:WU00:FS01:0x22:Completed 3120000 out of 8000000 steps (39%)
16:34:48:WU01:FS00:0xa7:Completed 100000 out of 500000 steps (20%)
16:34:59:WU00:FS01:0x22:Completed 3200000 out of 8000000 steps (40%)
16:36:21:WU00:FS01:0x22:Completed 3280000 out of 8000000 steps (41%)
16:37:34:WU01:FS00:0xa7:Completed 105000 out of 500000 steps (21%)
16:37:43:WU00:FS01:0x22:Completed 3360000 out of 8000000 steps (42%)
16:39:06:WU00:FS01:0x22:Completed 3440000 out of 8000000 steps (43%)
16:40:19:WU01:FS00:0xa7:Completed 110000 out of 500000 steps (22%)
16:40:28:WU00:FS01:0x22:Completed 3520000 out of 8000000 steps (44%)
16:41:50:WU00:FS01:0x22:Completed 3600000 out of 8000000 steps (45%)
16:43:05:WU01:FS00:0xa7:Completed 115000 out of 500000 steps (23%)
16:43:12:WU00:FS01:0x22:Completed 3680000 out of 8000000 steps (46%)
16:44:34:WU00:FS01:0x22:Completed 3760000 out of 8000000 steps (47%)
16:45:50:WU01:FS00:0xa7:Completed 120000 out of 500000 steps (24%)
16:45:56:WU00:FS01:0x22:Completed 3840000 out of 8000000 steps (48%)
16:47:19:WU00:FS01:0x22:Completed 3920000 out of 8000000 steps (49%)
16:48:34:WU01:FS00:0xa7:Completed 125000 out of 500000 steps (25%)
16:48:41:WU00:FS01:0x22:Completed 4000000 out of 8000000 steps (50%)
16:50:03:WU00:FS01:0x22:Completed 4080000 out of 8000000 steps (51%)
16:51:20:WU01:FS00:0xa7:Completed 130000 out of 500000 steps (26%)
16:51:26:WU00:FS01:0x22:Completed 4160000 out of 8000000 steps (52%)
16:52:48:WU00:FS01:0x22:Completed 4240000 out of 8000000 steps (53%)
16:54:04:WU01:FS00:0xa7:Completed 135000 out of 500000 steps (27%)
16:54:11:WU00:FS01:0x22:Completed 4320000 out of 8000000 steps (54%)
16:55:33:WU00:FS01:0x22:Completed 4400000 out of 8000000 steps (55%)
16:56:50:WU01:FS00:0xa7:Completed 140000 out of 500000 steps (28%)
16:56:56:WU00:FS01:0x22:Completed 4480000 out of 8000000 steps (56%)
16:58:18:WU00:FS01:0x22:Completed 4560000 out of 8000000 steps (57%)
16:59:34:WU01:FS00:0xa7:Completed 145000 out of 500000 steps (29%)
16:59:41:WU00:FS01:0x22:Completed 4640000 out of 8000000 steps (58%)
17:01:03:WU00:FS01:0x22:Completed 4720000 out of 8000000 steps (59%)
17:02:17:WU01:FS00:0xa7:Completed 150000 out of 500000 steps (30%)
17:02:26:WU00:FS01:0x22:Completed 4800000 out of 8000000 steps (60%)
17:03:48:WU00:FS01:0x22:Completed 4880000 out of 8000000 steps (61%)
17:05:02:WU01:FS00:0xa7:Completed 155000 out of 500000 steps (31%)
17:05:10:WU00:FS01:0x22:Completed 4960000 out of 8000000 steps (62%)
17:06:34:WU00:FS01:0x22:Completed 5040000 out of 8000000 steps (63%)
17:07:46:WU01:FS00:0xa7:Completed 160000 out of 500000 steps (32%)
17:07:56:WU00:FS01:0x22:Completed 5120000 out of 8000000 steps (64%)
17:09:19:WU00:FS01:0x22:Completed 5200000 out of 8000000 steps (65%)
17:10:31:WU01:FS00:0xa7:Completed 165000 out of 500000 steps (33%)
17:10:41:WU00:FS01:0x22:Completed 5280000 out of 8000000 steps (66%)
17:12:04:WU00:FS01:0x22:Completed 5360000 out of 8000000 steps (67%)
17:13:16:WU01:FS00:0xa7:Completed 170000 out of 500000 steps (34%)
17:13:26:WU00:FS01:0x22:Completed 5440000 out of 8000000 steps (68%)
17:14:49:WU00:FS01:0x22:Completed 5520000 out of 8000000 steps (69%)
17:16:00:WU01:FS00:0xa7:Completed 175000 out of 500000 steps (35%)
17:16:12:WU00:FS01:0x22:Completed 5600000 out of 8000000 steps (70%)
17:17:34:WU00:FS01:0x22:Completed 5680000 out of 8000000 steps (71%)
17:18:44:WU01:FS00:0xa7:Completed 180000 out of 500000 steps (36%)
17:18:57:WU00:FS01:0x22:Completed 5760000 out of 8000000 steps (72%)
17:20:19:WU00:FS01:0x22:Completed 5840000 out of 8000000 steps (73%)
17:21:27:WU01:FS00:0xa7:Completed 185000 out of 500000 steps (37%)
17:21:42:WU00:FS01:0x22:Completed 5920000 out of 8000000 steps (74%)
17:23:04:WU00:FS01:0x22:Completed 6000000 out of 8000000 steps (75%)
17:24:11:WU01:FS00:0xa7:Completed 190000 out of 500000 steps (38%)
17:24:27:WU00:FS01:0x22:Completed 6080000 out of 8000000 steps (76%)
17:25:50:WU00:FS01:0x22:Completed 6160000 out of 8000000 steps (77%)
17:26:56:WU01:FS00:0xa7:Completed 195000 out of 500000 steps (39%)
17:27:12:WU00:FS01:0x22:Completed 6240000 out of 8000000 steps (78%)
17:28:35:WU00:FS01:0x22:Completed 6320000 out of 8000000 steps (79%)
17:29:40:WU01:FS00:0xa7:Completed 200000 out of 500000 steps (40%)
17:29:58:WU00:FS01:0x22:Completed 6400000 out of 8000000 steps (80%)
17:31:21:WU00:FS01:0x22:Completed 6480000 out of 8000000 steps (81%)
17:32:24:WU01:FS00:0xa7:Completed 205000 out of 500000 steps (41%)
17:32:43:WU00:FS01:0x22:Completed 6560000 out of 8000000 steps (82%)
17:34:06:WU00:FS01:0x22:Completed 6640000 out of 8000000 steps (83%)
17:35:08:WU01:FS00:0xa7:Completed 210000 out of 500000 steps (42%)
17:35:29:WU00:FS01:0x22:Completed 6720000 out of 8000000 steps (84%)
17:36:51:WU00:FS01:0x22:Completed 6800000 out of 8000000 steps (85%)
17:37:51:WU01:FS00:0xa7:Completed 215000 out of 500000 steps (43%)
17:38:14:WU00:FS01:0x22:Completed 6880000 out of 8000000 steps (86%)
17:39:37:WU00:FS01:0x22:Completed 6960000 out of 8000000 steps (87%)
17:40:36:WU01:FS00:0xa7:Completed 220000 out of 500000 steps (44%)
17:41:00:WU00:FS01:0x22:Completed 7040000 out of 8000000 steps (88%)
17:42:22:WU00:FS01:0x22:Completed 7120000 out of 8000000 steps (89%)
17:43:20:WU01:FS00:0xa7:Completed 225000 out of 500000 steps (45%)
17:43:45:WU00:FS01:0x22:Completed 7200000 out of 8000000 steps (90%)
17:45:08:WU00:FS01:0x22:Completed 7280000 out of 8000000 steps (91%)
17:46:05:WU01:FS00:0xa7:Completed 230000 out of 500000 steps (46%)
17:46:31:WU00:FS01:0x22:Completed 7360000 out of 8000000 steps (92%)
17:47:54:WU00:FS01:0x22:Completed 7440000 out of 8000000 steps (93%)
17:48:49:WU01:FS00:0xa7:Completed 235000 out of 500000 steps (47%)
17:49:17:WU00:FS01:0x22:Completed 7520000 out of 8000000 steps (94%)
17:50:40:WU00:FS01:0x22:Completed 7600000 out of 8000000 steps (95%)
17:51:33:WU01:FS00:0xa7:Completed 240000 out of 500000 steps (48%)
17:52:03:WU00:FS01:0x22:Completed 7680000 out of 8000000 steps (96%)
17:53:26:WU00:FS01:0x22:Completed 7760000 out of 8000000 steps (97%)
17:54:18:WU01:FS00:0xa7:Completed 245000 out of 500000 steps (49%)
17:54:49:WU00:FS01:0x22:Completed 7840000 out of 8000000 steps (98%)
17:56:12:WU00:FS01:0x22:Completed 7920000 out of 8000000 steps (99%)
17:57:02:WU01:FS00:0xa7:Completed 250000 out of 500000 steps (50%)
17:57:35:WU00:FS01:0x22:Completed 8000000 out of 8000000 steps (100%)
17:57:36:WU00:FS01:0x22:Saving result file ..\logfile_01.txt
17:57:36:WU00:FS01:0x22:Saving result file checkpointState.xml
17:57:36:WU00:FS01:0x22:Saving result file checkpt.crc
17:57:36:WU00:FS01:0x22:Saving result file positions.xtc
17:57:37:WU00:FS01:0x22:Saving result file science.log
17:57:37:WU00:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
17:57:38:WU02:FS01:Connecting to assign1.foldingathome.org:80
17:57:39:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:57:39:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:16433 run:893 clone:0 gen:30 core:0x22 unit:0x0000002603854c135e9a4f003e33ad50
17:57:39:WU00:FS01:Uploading 59.53MiB to 3.133.76.19
17:57:39:WU00:FS01:Connecting to 3.133.76.19:8080
17:57:40:WARNING:WU02:FS01:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration
17:57:40:WU02:FS01:Connecting to assign2.foldingathome.org:80
17:57:40:WU02:FS01:Assigned to work server 18.188.125.154
17:57:40:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1070] 6463 from 18.188.125.154
17:57:40:WU02:FS01:Connecting to 18.188.125.154:8080
17:57:42:WU02:FS01:Downloading 6.36MiB
17:57:48:WU02:FS01:Download 86.43%
17:57:48:WU02:FS01:Download complete
17:57:48:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13405 run:156 clone:45 gen:0 core:0x22 unit:0x0000000212bc7d9a5eb3a38b3d948bc1
17:57:48:WU02:FS01:Starting
17:57:48:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Malcolm\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 29212 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
17:57:48:WU02:FS01:Started FahCore on PID 27984
17:57:48:WU02:FS01:Core PID:20632
17:57:48:WU02:FS01:FahCore 0x22 started
17:57:49:WU02:FS01:0x22:*********************** Log Started 2020-05-07T17:57:48Z ***********************
17:57:49:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
17:57:49:WU02:FS01:0x22: Type: 0x22
17:57:49:WU02:FS01:0x22: Core: Core22
17:57:49:WU02:FS01:0x22: Website: https://foldingathome.org/
17:57:49:WU02:FS01:0x22: Copyright: (c) 2009-2018 foldingathome.org
17:57:49:WU02:FS01:0x22: Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
17:57:49:WU02:FS01:0x22: <rafal.wiewiora@choderalab.org>
17:57:49:WU02:FS01:0x22: Args: -dir 02 -suffix 01 -version 706 -lifeline 27984 -checkpoint 30
17:57:49:WU02:FS01:0x22: -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
17:57:49:WU02:FS01:0x22: 0 -gpu 0
17:57:49:WU02:FS01:0x22: Config: <none>
17:57:49:WU02:FS01:0x22:************************************ Build *************************************
17:57:49:WU02:FS01:0x22: Version: 0.0.5
17:57:49:WU02:FS01:0x22: Date: Apr 22 2020
17:57:49:WU02:FS01:0x22: Time: 04:42:59
17:57:49:WU02:FS01:0x22: Repository: Git
17:57:49:WU02:FS01:0x22: Revision: 2d69202c898bd9bb3e093f51cd32bf411c2a0388
17:57:49:WU02:FS01:0x22: Branch: HEAD
17:57:49:WU02:FS01:0x22: Compiler: Visual C++ 2008
17:57:49:WU02:FS01:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:57:49:WU02:FS01:0x22: Platform: win32 10
17:57:49:WU02:FS01:0x22: Bits: 64
17:57:49:WU02:FS01:0x22: Mode: Release
17:57:49:WU02:FS01:0x22:************************************ System ************************************
17:57:49:WU02:FS01:0x22: CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
17:57:49:WU02:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
17:57:49:WU02:FS01:0x22: CPUs: 8
17:57:49:WU02:FS01:0x22: Memory: 15.86GiB
17:57:49:WU02:FS01:0x22:Free Memory: 11.19GiB
17:57:49:WU02:FS01:0x22: Threads: WINDOWS_THREADS
17:57:49:WU02:FS01:0x22: OS Version: 6.2
17:57:49:WU02:FS01:0x22:Has Battery: false
17:57:49:WU02:FS01:0x22: On Battery: false
17:57:49:WU02:FS01:0x22: UTC Offset: 1
17:57:49:WU02:FS01:0x22: PID: 20632
17:57:49:WU02:FS01:0x22: CWD: C:\Users\Malcolm\AppData\Roaming\FAHClient\work
17:57:49:WU02:FS01:0x22: OS: Windows 10 Home
17:57:49:WU02:FS01:0x22: OS Arch: AMD64
17:57:49:WU02:FS01:0x22:********************************************************************************
17:57:49:WU02:FS01:0x22:Project: 13405 (Run 156, Clone 45, Gen 0)
17:57:49:WU02:FS01:0x22:Unit: 0x0000000212bc7d9a5eb3a38b3d948bc1
17:57:49:WU02:FS01:0x22:Reading tar file core.xml
17:57:49:WU02:FS01:0x22:Reading tar file integrator.xml
17:57:49:WU02:FS01:0x22:Reading tar file state.xml
17:57:49:WU02:FS01:0x22:Reading tar file system.xml
17:57:49:WU02:FS01:0x22:Digital signatures verified
17:57:49:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
17:57:49:WU02:FS01:0x22:Version 0.0.5
17:58:00:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
17:58:00:WU00:FS01:Connecting to 3.133.76.19:80
17:58:00:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
17:58:00:WU02:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
17:58:21:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 3.133.76.19:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
17:58:21:WU00:FS01:Trying to send results to collection server
17:58:21:WU00:FS01:Uploading 59.53MiB to 3.21.157.11
17:58:21:WU00:FS01:Connecting to 3.21.157.11:8080
17:58:27:WU00:FS01:Upload 1.15%
17:58:33:WU00:FS01:Upload 2.20%
17:58:39:WU00:FS01:Upload 3.25%
17:58:45:WU00:FS01:Upload 4.30%
17:58:51:WU00:FS01:Upload 5.46%
17:58:57:WU00:FS01:Upload 6.61%
17:59:03:WU00:FS01:Upload 7.66%
17:59:09:WU00:FS01:Upload 8.71%
17:59:15:WU00:FS01:Upload 9.87%
17:59:21:WU00:FS01:Upload 10.92%
17:59:27:WU00:FS01:Upload 12.07%
17:59:33:WU00:FS01:Upload 13.12%
17:59:39:WU00:FS01:Upload 14.17%
17:59:44:WU01:FS00:0xa7:Completed 255000 out of 500000 steps (51%)
17:59:45:WU00:FS01:Upload 15.33%
17:59:51:WU00:FS01:Upload 16.48%
17:59:57:WU00:FS01:Upload 17.53%
18:00:03:WU00:FS01:Upload 18.69%
18:00:09:WU00:FS01:Upload 19.74%
18:00:15:WU00:FS01:Upload 20.89%
18:00:21:WU00:FS01:Upload 21.94%
18:00:27:WU00:FS01:Upload 23.10%
18:00:33:WU00:FS01:Upload 24.25%
18:00:39:WU00:FS01:Upload 25.20%
18:00:40:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
18:00:45:WU00:FS01:Upload 26.35%
18:00:51:WU00:FS01:Upload 27.51%
18:00:57:WU00:FS01:Upload 28.56%
18:01:03:WU00:FS01:Upload 29.71%
18:01:09:WU00:FS01:Upload 30.76%
18:01:15:WU00:FS01:Upload 31.81%
18:01:21:WU00:FS01:Upload 32.97%
18:01:27:WU00:FS01:Upload 34.12%
18:01:33:WU00:FS01:Upload 35.17%
18:01:39:WU00:FS01:Upload 36.22%
18:01:45:WU00:FS01:Upload 37.38%
18:01:51:WU00:FS01:Upload 38.43%
18:01:57:WU00:FS01:Upload 39.58%
18:02:03:WU00:FS01:Upload 40.74%
18:02:09:WU00:FS01:Upload 41.79%
18:02:15:WU00:FS01:Upload 42.84%
18:02:21:WU00:FS01:Upload 43.99%
18:02:27:WU00:FS01:Upload 45.04%
18:02:28:WU01:FS00:0xa7:Completed 260000 out of 500000 steps (52%)
18:02:33:WU00:FS01:Upload 46.09%
18:02:39:WU00:FS01:Upload 47.25%
18:02:45:WU00:FS01:Upload 48.40%
18:02:51:WU00:FS01:Upload 49.45%
18:02:57:WU00:FS01:Upload 50.50%
18:03:03:WU00:FS01:Upload 51.66%
18:03:09:WU00:FS01:Upload 52.81%
18:03:15:WU00:FS01:Upload 53.86%
18:03:18:WU02:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
18:03:21:WU00:FS01:Upload 54.91%
18:03:27:WU00:FS01:Upload 56.07%
18:03:33:WU00:FS01:Upload 57.12%
18:03:39:WU00:FS01:Upload 58.27%
18:03:45:WU00:FS01:Upload 59.22%
18:03:51:WU00:FS01:Upload 60.26%
18:03:57:WU00:FS01:Upload 61.42%
18:04:03:WU00:FS01:Upload 62.57%
18:04:09:WU00:FS01:Upload 63.62%
18:04:15:WU00:FS01:Upload 64.78%
18:04:21:WU00:FS01:Upload 65.83%
18:04:27:WU00:FS01:Upload 66.98%
18:04:33:WU00:FS01:Upload 68.03%
18:04:41:WU00:FS01:Upload 69.08%
18:04:47:WU00:FS01:Upload 70.24%
18:04:53:WU00:FS01:Upload 71.39%
18:04:59:WU00:FS01:Upload 72.34%
18:05:05:WU00:FS01:Upload 73.49%
18:05:11:WU01:FS00:0xa7:Completed 265000 out of 500000 steps (53%)
18:05:11:WU00:FS01:Upload 74.65%
18:05:17:WU00:FS01:Upload 75.70%
18:05:23:WU00:FS01:Upload 76.85%
18:05:29:WU00:FS01:Upload 77.90%
18:05:35:WU00:FS01:Upload 79.06%
18:05:41:WU00:FS01:Upload 80.11%
18:05:47:WU00:FS01:Upload 81.26%
18:05:53:WU00:FS01:Upload 82.31%
18:05:56:WU02:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
18:05:59:WU00:FS01:Upload 83.36%
18:06:05:WU00:FS01:Upload 84.52%
18:06:11:WU00:FS01:Upload 85.67%
18:06:17:WU00:FS01:Upload 86.72%
18:06:23:WU00:FS01:Upload 87.88%
18:06:29:WU00:FS01:Upload 88.93%
18:06:35:WU00:FS01:Upload 89.98%
18:06:41:WU00:FS01:Upload 91.13%
18:06:47:WU00:FS01:Upload 92.18%
18:06:53:WU00:FS01:Upload 93.23%
18:06:59:WU00:FS01:Upload 94.39%
18:07:05:WU00:FS01:Upload 95.54%
18:07:11:WU00:FS01:Upload 96.59%
18:07:17:WU00:FS01:Upload 97.75%
18:07:23:WU00:FS01:Upload 98.80%
18:07:29:WU00:FS01:Upload 99.85%
18:07:31:WU00:FS01:Upload complete
18:07:31:WU00:FS01:Server responded WORK_ACK (400)
18:07:31:WU00:FS01:Final credit estimate, 75691.00 points
18:07:31:WU00:FS01:Cleaning up
18:07:54:WU01:FS00:0xa7:Completed 270000 out of 500000 steps (54%)
18:08:35:WU02:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
18:10:37:WU01:FS00:0xa7:Completed 275000 out of 500000 steps (55%)
18:11:13:WU02:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
18:13:21:WU01:FS00:0xa7:Completed 280000 out of 500000 steps (56%)
18:13:51:WU02:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
18:16:04:WU01:FS00:0xa7:Completed 285000 out of 500000 steps (57%)
18:16:30:WU02:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
18:18:48:WU01:FS00:0xa7:Completed 290000 out of 500000 steps (58%)
18:19:08:WU02:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
18:21:31:WU01:FS00:0xa7:Completed 295000 out of 500000 steps (59%)
18:21:46:WU02:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
18:24:15:WU01:FS00:0xa7:Completed 300000 out of 500000 steps (60%)
18:24:25:WU02:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
18:26:58:WU01:FS00:0xa7:Completed 305000 out of 500000 steps (61%)
18:27:03:WU02:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
18:29:41:WU02:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
18:29:42:WU01:FS00:0xa7:Completed 310000 out of 500000 steps (62%)
18:32:20:WU02:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
18:32:25:WU01:FS00:0xa7:Completed 315000 out of 500000 steps (63%)
18:34:58:WU02:FS01:0x22:Completed 140000 out of 1000000 steps (14%)
18:35:09:WU01:FS00:0xa7:Completed 320000 out of 500000 steps (64%)
18:37:36:WU02:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
18:37:52:WU01:FS00:0xa7:Completed 325000 out of 500000 steps (65%)
18:40:14:WU02:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
18:40:37:WU01:FS00:0xa7:Completed 330000 out of 500000 steps (66%)
18:42:53:WU02:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
18:43:20:WU01:FS00:0xa7:Completed 335000 out of 500000 steps (67%)
18:45:31:WU02:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
18:46:04:WU01:FS00:0xa7:Completed 340000 out of 500000 steps (68%)
18:48:09:WU02:FS01:0x22:Completed 190000 out of 1000000 steps (19%)
18:48:47:WU01:FS00:0xa7:Completed 345000 out of 500000 steps (69%)
18:50:47:WU02:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
18:51:32:WU01:FS00:0xa7:Completed 350000 out of 500000 steps (70%)
18:53:26:WU02:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
18:54:15:WU01:FS00:0xa7:Completed 355000 out of 500000 steps (71%)
18:56:04:WU02:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
18:56:59:WU01:FS00:0xa7:Completed 360000 out of 500000 steps (72%)
18:58:42:WU02:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
18:59:43:WU01:FS00:0xa7:Completed 365000 out of 500000 steps (73%)
19:01:21:WU02:FS01:0x22:Completed 240000 out of 1000000 steps (24%)
19:02:28:WU01:FS00:0xa7:Completed 370000 out of 500000 steps (74%)
19:03:59:WU02:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
19:05:12:WU01:FS00:0xa7:Completed 375000 out of 500000 steps (75%)
19:06:45:WU02:FS01:0x22:Completed 260000 out of 1000000 steps (26%)
19:07:55:WU01:FS00:0xa7:Completed 380000 out of 500000 steps (76%)
19:09:28:WU02:FS01:0x22:Completed 270000 out of 1000000 steps (27%)
19:10:39:WU01:FS00:0xa7:Completed 385000 out of 500000 steps (77%)
19:12:12:WU02:FS01:0x22:Completed 280000 out of 1000000 steps (28%)
19:13:23:WU01:FS00:0xa7:Completed 390000 out of 500000 steps (78%)
19:14:55:WU02:FS01:0x22:Completed 290000 out of 1000000 steps (29%)
19:16:06:WU01:FS00:0xa7:Completed 395000 out of 500000 steps (79%)
19:17:39:WU02:FS01:0x22:Completed 300000 out of 1000000 steps (30%)
19:18:50:WU01:FS00:0xa7:Completed 400000 out of 500000 steps (80%)
19:20:22:WU02:FS01:0x22:Completed 310000 out of 1000000 steps (31%)
19:21:34:WU01:FS00:0xa7:Completed 405000 out of 500000 steps (81%)
19:23:06:WU02:FS01:0x22:Completed 320000 out of 1000000 steps (32%)
19:24:17:WU01:FS00:0xa7:Completed 410000 out of 500000 steps (82%)
19:25:49:WU02:FS01:0x22:Completed 330000 out of 1000000 steps (33%)
19:27:01:WU01:FS00:0xa7:Completed 415000 out of 500000 steps (83%)
19:28:33:WU02:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
19:29:45:WU01:FS00:0xa7:Completed 420000 out of 500000 steps (84%)
19:31:16:WU02:FS01:0x22:Completed 350000 out of 1000000 steps (35%)
19:32:29:WU01:FS00:0xa7:Completed 425000 out of 500000 steps (85%)
19:34:00:WU02:FS01:0x22:Completed 360000 out of 1000000 steps (36%)
19:35:13:WU01:FS00:0xa7:Completed 430000 out of 500000 steps (86%)
19:36:43:WU02:FS01:0x22:Completed 370000 out of 1000000 steps (37%)
19:37:56:WU01:FS00:0xa7:Completed 435000 out of 500000 steps (87%)
19:39:27:WU02:FS01:0x22:Completed 380000 out of 1000000 steps (38%)
19:40:39:WU01:FS00:0xa7:Completed 440000 out of 500000 steps (88%)
19:42:10:WU02:FS01:0x22:Completed 390000 out of 1000000 steps (39%)
19:43:22:WU01:FS00:0xa7:Completed 445000 out of 500000 steps (89%)
19:44:54:WU02:FS01:0x22:Completed 400000 out of 1000000 steps (40%)
19:46:06:WU01:FS00:0xa7:Completed 450000 out of 500000 steps (90%)
19:47:37:WU02:FS01:0x22:Completed 410000 out of 1000000 steps (41%)
19:48:50:WU01:FS00:0xa7:Completed 455000 out of 500000 steps (91%)
19:50:21:WU02:FS01:0x22:Completed 420000 out of 1000000 steps (42%)
19:51:34:WU01:FS00:0xa7:Completed 460000 out of 500000 steps (92%)
19:53:04:WU02:FS01:0x22:Completed 430000 out of 1000000 steps (43%)
19:54:19:WU01:FS00:0xa7:Completed 465000 out of 500000 steps (93%)
19:55:48:WU02:FS01:0x22:Completed 440000 out of 1000000 steps (44%)
19:57:03:WU01:FS00:0xa7:Completed 470000 out of 500000 steps (94%)
19:58:31:WU02:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
19:59:47:WU01:FS00:0xa7:Completed 475000 out of 500000 steps (95%)
20:01:15:WU02:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
20:02:31:WU01:FS00:0xa7:Completed 480000 out of 500000 steps (96%)
20:03:58:WU02:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
20:05:15:WU01:FS00:0xa7:Completed 485000 out of 500000 steps (97%)
20:06:42:WU02:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
20:07:59:WU01:FS00:0xa7:Completed 490000 out of 500000 steps (98%)
20:09:30:WU02:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
20:09:30:WU02:FS01:0x22:Following exception occured: Particle coordinate is nan
20:09:32:FS01:Finishing
20:09:32:FS00:Finishing
20:09:45:WU02:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
20:09:45:WU02:FS01:0x22:Following exception occured: Particle coordinate is nan
20:10:01:WU02:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
20:10:01:WU02:FS01:0x22:Following exception occured: Particle coordinate is nan
20:10:01:WU02:FS01:0x22:ERROR:114: Max Retries Reached
20:10:01:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
20:10:01:WU02:FS01:0x22:Saving result file badstate-0.xml
20:10:01:WU02:FS01:0x22:Saving result file badstate-1.xml
20:10:01:WU02:FS01:0x22:Saving result file badstate-2.xml
20:10:01:WU02:FS01:0x22:Saving result file checkpointState.xml
20:10:02:WU02:FS01:0x22:Saving result file checkpt.crc
20:10:02:WU02:FS01:0x22:Saving result file globals.csv
20:10:02:WU02:FS01:0x22:Saving result file positions.xtc
20:10:02:WU02:FS01:0x22:Saving result file science.log
20:10:02:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
20:10:03:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
20:10:03:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13405 run:156 clone:45 gen:0 core:0x22 unit:0x0000000212bc7d9a5eb3a38b3d948bc1
20:10:03:WU02:FS01:Uploading 4.97MiB to 18.188.125.154
20:10:03:WU02:FS01:Connecting to 18.188.125.154:8080
20:10:09:WU02:FS01:Upload 13.83%
20:10:15:WU02:FS01:Upload 27.67%
20:10:21:WU02:FS01:Upload 40.25%
20:10:27:WU02:FS01:Upload 54.08%
20:10:33:WU02:FS01:Upload 67.91%
20:10:39:WU02:FS01:Upload 81.75%
20:10:40:WU01:FS00:0xa7:Completed 495000 out of 500000 steps (99%)
20:10:45:WU02:FS01:Upload 94.33%
20:10:48:WU02:FS01:Upload complete
20:10:48:WU02:FS01:Server responded WORK_ACK (400)
20:10:48:WU02:FS01:Cleaning up
20:13:07:WU01:FS00:0xa7:Completed 500000 out of 500000 steps (100%)
20:13:08:WU01:FS00:0xa7:Saving result file ..\logfile_01.txt
20:13:08:WU01:FS00:0xa7:Saving result file frame100.edr
20:13:08:WU01:FS00:0xa7:Saving result file frame100.trr
20:13:08:WU01:FS00:0xa7:Saving result file frame100.xtc
20:13:08:WU01:FS00:0xa7:Saving result file md.log
20:13:08:WU01:FS00:0xa7:Saving result file science.log
20:13:08:WU01:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
20:13:09:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
20:13:09:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:13850 run:0 clone:38659 gen:100 core:0xa7 unit:0x00000078287234c95e78894153dba994
20:13:09:WU01:FS00:Uploading 2.47MiB to 40.114.52.201
20:13:09:WU01:FS00:Connecting to 40.114.52.201:8080
20:13:15:WU01:FS00:Upload 30.33%
20:13:21:WU01:FS00:Upload 55.61%
20:13:27:WU01:FS00:Upload 83.41%
20:13:31:WU01:FS00:Upload complete
20:13:31:WU01:FS00:Server responded WORK_ACK (400)
20:13:31:WU01:FS00:Final credit estimate, 5254.00 points
20:13:31:WU01:FS00:Cleaning up
20:51:48:Clean exit
Re: 13405 (156, 45, 0) failed at 48%
Posted: Fri May 08, 2020 2:59 pm
by JohnChodera
Thanks so much, everyone! We get all this data on failures returned to us on the servers, so we're doing analytics on it already to understand how to make improvements as this project series continues!
~ John Chodera // MSKCC