Fahcore_23 v8.0.3 project:17116 run:5 clone:470 gen:0 core:0x23 Hangs at 100%
Posted: Wed Aug 16, 2023 8:19 am
Fahcore_23 v8.0.3 Hangs at 100%
project:17116 run:5 clone:470 gen:0 core:0x23
The particularly ugly thing about this core:0x23 hanging is that it disables your folding machine by hanging it. You cannot fold anything until the hung WU is cleared out.
I think there's an issue with Fahcore_23 v8.0.3 core:0x23 . My Nvidia 2070 Super GPU has been folding rock solid for many months, maybe 1+ years, less the occasional PG&E power failure....
project:17116 core:0x23 takes 7 seconds per frame, and completes to 99.9% / 100% and then the FahCore_23.exe process just hangs and never completes, exits or throws an error - just hangs...
The last log line from the core before it hangs forever is:
"06:58:25:WU00:FS01:0x23:Folding@home Core Shutdown: FINISHED_UNIT"
I had this happen to multiple project:17116 core:0x23 WUs. I rebooted and did some investigation on the new Core:0x23. Consequently, on this particular WU, I only waited about 15 minutes before pausing and starting to fold again. This WU failed to restart because prior FahCore_23.exe process never exited (I presume) and still help a lock on the files, preventing it from restarting after clicking "Fold". Even if it did restart, I assume it would again hang forever at the same point. I have a solid 1 Gbps down / 40 Mbps cable internet connection that my two other folding machines were running just fine on. This computer continued to fold perfectly with WUs that did not use FahCore_23 .
It appears that only 2 projects 17116, and 17114 are using core:0x23:
(Easy way to find them all is to search for "0x23" in the web page)
17116 Benchmark Memorial Sloan Kettering Cancer Center 0x23 50000 1000
17114 Benchmark Memorial Sloan Kettering Cancer Center 0x23 50000 1000
Source: https://folding.lar.systems/projects/
17116 54.157.202.86 50,000 1.00 2.00 1,000 0x23 john.chodera
Source: https://apps.foldingathome.org/psummary
Interesting https://folding.lar.systems/ shows only one of these WUs completed so far by someone with an Nvidia TU104 [GeForce RTX 2060]:
https://folding.lar.systems/projects/fo ... file/17116
I have searched and not been able to find any prior references or issues with the new project:17116 or the new core:0x23 / Fahcore_23 using multiple different search terms beyond their appearance in the project listing pages above.
Looks like the core was compiled on "Date: Aug 3 2023". I don't know exactly when the first WU using core:0x23 was released for folding. It's possible the first WUs were issued today or in the last few days, and certainly within the last two weeks, since the core was compiled less than two weeks ago.
project:17116 run:5 clone:470 gen:0 core:0x23
The particularly ugly thing about this core:0x23 hanging is that it disables your folding machine by hanging it. You cannot fold anything until the hung WU is cleared out.
I think there's an issue with Fahcore_23 v8.0.3 core:0x23 . My Nvidia 2070 Super GPU has been folding rock solid for many months, maybe 1+ years, less the occasional PG&E power failure....
project:17116 core:0x23 takes 7 seconds per frame, and completes to 99.9% / 100% and then the FahCore_23.exe process just hangs and never completes, exits or throws an error - just hangs...
The last log line from the core before it hangs forever is:
"06:58:25:WU00:FS01:0x23:Folding@home Core Shutdown: FINISHED_UNIT"
I had this happen to multiple project:17116 core:0x23 WUs. I rebooted and did some investigation on the new Core:0x23. Consequently, on this particular WU, I only waited about 15 minutes before pausing and starting to fold again. This WU failed to restart because prior FahCore_23.exe process never exited (I presume) and still help a lock on the files, preventing it from restarting after clicking "Fold". Even if it did restart, I assume it would again hang forever at the same point. I have a solid 1 Gbps down / 40 Mbps cable internet connection that my two other folding machines were running just fine on. This computer continued to fold perfectly with WUs that did not use FahCore_23 .
Code: Select all
07:14:18:FS01:Paused
07:14:18:FS01:Shutting core down
07:14:19:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
07:14:31:FS01:Unpaused
07:14:31:WU00:FS01:Starting
07:14:31:ERROR:WU00:FS01:Failed to start core: Failed to rename 'work/00/logfile_01.txt' to 'work/00/logfile_01-20230816-071431.txt': The process cannot access the file because it is being used by another process.
07:16:08:WU00:FS01:Starting
07:16:08:ERROR:WU00:FS01:Failed to start core: Failed to rename 'work/00/logfile_01.txt' to 'work/00/logfile_01-20230816-071608.txt': The process cannot access the file because it is being used by another process.
(Easy way to find them all is to search for "0x23" in the web page)
17116 Benchmark Memorial Sloan Kettering Cancer Center 0x23 50000 1000
17114 Benchmark Memorial Sloan Kettering Cancer Center 0x23 50000 1000
Source: https://folding.lar.systems/projects/
17116 54.157.202.86 50,000 1.00 2.00 1,000 0x23 john.chodera
Source: https://apps.foldingathome.org/psummary
Interesting https://folding.lar.systems/ shows only one of these WUs completed so far by someone with an Nvidia TU104 [GeForce RTX 2060]:
https://folding.lar.systems/projects/fo ... file/17116
I have searched and not been able to find any prior references or issues with the new project:17116 or the new core:0x23 / Fahcore_23 using multiple different search terms beyond their appearance in the project listing pages above.
Looks like the core was compiled on "Date: Aug 3 2023". I don't know exactly when the first WU using core:0x23 was released for folding. It's possible the first WUs were issued today or in the last few days, and certainly within the last two weeks, since the core was compiled less than two weeks ago.
Code: Select all
*********************** Log Started 2023-08-16T06:50:44Z ***********************
06:50:44:******************************* libFAH ********************************
06:50:44: Date: Oct 20 2020
06:50:44: Time: 13:36:55
06:50:44: Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
06:50:44: Branch: master
06:50:44: Compiler: Visual C++ 2015
06:50:44: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
06:50:44: Platform: win32 10
06:50:44: Bits: 32
06:50:44: Mode: Release
06:50:44:****************************** FAHClient ******************************
06:50:44: Version: 7.6.21
06:50:44: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
06:50:44: Copyright: 2020 foldingathome.org
06:50:44: Homepage: https://foldingathome.org/
06:50:44: Date: Oct 20 2020
06:50:44: Time: 13:41:04
06:50:44: Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
06:50:44: Branch: master
06:50:44: Compiler: Visual C++ 2015
06:50:44: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
06:50:44: Platform: win32 10
06:50:44: Bits: 32
06:50:44: Mode: Release
06:50:44: Args: --open-web-control
06:50:44: Config: C:\Users\MikHexa\AppData\Roaming\FAHClient\config.xml
06:50:44:******************************** CBang ********************************
06:50:44: Date: Oct 20 2020
06:50:44: Time: 11:36:18
06:50:44: Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
06:50:44: Branch: master
06:50:44: Compiler: Visual C++ 2015
06:50:44: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
06:50:44: Platform: win32 10
06:50:44: Bits: 32
06:50:44: Mode: Release
06:50:44:******************************* System ********************************
06:50:44: CPU: Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz
06:50:44: CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
06:50:44: CPUs: 12
06:50:44: Memory: 11.99GiB
06:50:44: Free Memory: 9.80GiB
06:50:44: Threads: WINDOWS_THREADS
06:50:44: OS Version: 6.1
06:50:44: Has Battery: false
06:50:44: On Battery: false
06:50:44: UTC Offset: -7
06:50:44: PID: 6708
06:50:44: CWD: C:\Users\MikHexa\AppData\Roaming\FAHClient
06:50:44: Win32 Service: false
06:50:44: OS: Windows 7 Ultimate
06:50:44: OS Arch: AMD64
06:50:44: GPUs: 1
06:50:44: GPU 0: Bus:3 Slot:0 Func:0 NVIDIA:8 TU104 [GeForce RTX 2070 SUPER]
06:50:44: 8218
06:50:44: CUDA Device 0: Platform:0 Device:0 Bus:3 Slot:0 Compute:7.5 Driver:11.4
06:50:44:OpenCL Device 0: Platform:0 Device:0 Bus:3 Slot:0 Compute:3.0 Driver:472.12
06:50:44:***********************************************************************
06:50:44:<config>
06:50:44: <!-- Folding Slot Configuration -->
06:50:44: <cause v='COVID_19'/>
06:50:44:
06:50:44: <!-- HTTP Server -->
06:50:44: <allow v='127.0.0.1 192.168.168.153 192.168.168.174 192.168.168.213'/>
06:50:44:
06:50:44: <!-- Network -->
06:50:44: <proxy v=':8080'/>
06:50:44:
06:50:44: <!-- Remote Command Server -->
06:50:44: <password v='*****'/>
06:50:44:
06:50:44: <!-- Slot Control -->
06:50:44: <power v='full'/>
06:50:44:
06:50:44: <!-- User Information -->
06:50:44: <passkey v='*****'/>
06:50:44: <team v='104636'/>
06:50:44: <user v='[EV]Tesla'/>
06:50:44:
06:50:44: <!-- Folding Slots -->
06:50:44: <slot id='1' type='GPU'>
06:50:44: <paused v='true'/>
06:50:44: <pci-bus v='3'/>
06:50:44: <pci-slot v='0'/>
06:50:44: </slot>
06:50:44:</config>
06:50:44:Trying to access database...
06:50:44:Successfully acquired database lock
06:50:44:FS01:Initialized folding slot 01: gpu:3:0 TU104 [GeForce RTX 2070 SUPER] 8218
06:50:44:Set client configured
06:50:57:18:127.0.0.1:New Web session
06:51:13:FS01:Unpaused
06:51:13:WU00:FS01:Connecting to assign1.foldingathome.org:80
06:51:14:WU00:FS01:Connecting to assign1.foldingathome.org:80
06:51:14:WU00:FS01:Assigned to work server 54.157.202.86
06:51:14:WU00:FS01:Requesting new work unit for slot 01: gpu:3:0 TU104 [GeForce RTX 2070 SUPER] 8218 from 54.157.202.86
06:51:14:WU00:FS01:Connecting to 54.157.202.86:8080
06:51:15:WU00:FS01:Downloading 7.41MiB
06:51:18:WU00:FS01:Download complete
06:51:18:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:17116 run:5 clone:470 gen:0 core:0x23 unit:0x000001d600000000000042dc00000005
06:51:18:WU00:FS01:Starting
06:51:18:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\MikHexa\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/openmm-core-23/windows-10-64bit/release/0x23-8.0.3/Core_23.fah/FahCore_23.exe -dir 00 -suffix 01 -version 706 -lifeline 6708 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
06:51:18:WU00:FS01:Started FahCore on PID 6040
06:51:18:WU00:FS01:Core PID:5248
06:51:18:WU00:FS01:FahCore 0x23 started
06:51:18:WU00:FS01:0x23:*********************** Log Started 2023-08-16T06:51:18Z ***********************
06:51:18:WU00:FS01:0x23:*************************** Core23 Folding@home Core ***************************
06:51:18:WU00:FS01:0x23: Core: Core23
06:51:18:WU00:FS01:0x23: Type: 0x23
06:51:18:WU00:FS01:0x23: Version: 8.0.3
06:51:18:WU00:FS01:0x23: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
06:51:18:WU00:FS01:0x23: Copyright: 2022 foldingathome.org
06:51:18:WU00:FS01:0x23: Homepage: https://foldingathome.org/
06:51:18:WU00:FS01:0x23: Date: Aug 3 2023
06:51:18:WU00:FS01:0x23: Time: 08:39:06
06:51:18:WU00:FS01:0x23: Compiler: Visual C++
06:51:18:WU00:FS01:0x23: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
06:51:18:WU00:FS01:0x23: -DOPENMM_VERSION="\"8.0.0\""
06:51:18:WU00:FS01:0x23: Platform: win32 10
06:51:18:WU00:FS01:0x23: Bits: 64
06:51:18:WU00:FS01:0x23: Mode: Release
06:51:18:WU00:FS01:0x23:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
06:51:18:WU00:FS01:0x23: <peastman@stanford.edu>
06:51:18:WU00:FS01:0x23: Args: -dir 00 -suffix 01 -version 706 -lifeline 6040 -checkpoint 15
06:51:18:WU00:FS01:0x23: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
06:51:18:WU00:FS01:0x23: nvidia -gpu 0 -gpu-usage 100
06:51:18:WU00:FS01:0x23:************************************ libFAH ************************************
06:51:18:WU00:FS01:0x23: Date: Aug 3 2023
06:51:18:WU00:FS01:0x23: Time: 08:37:55
06:51:18:WU00:FS01:0x23: Compiler: Visual C++
06:51:18:WU00:FS01:0x23: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
06:51:18:WU00:FS01:0x23: Platform: win32 10
06:51:18:WU00:FS01:0x23: Bits: 64
06:51:18:WU00:FS01:0x23: Mode: Release
06:51:18:WU00:FS01:0x23:************************************ CBang *************************************
06:51:18:WU00:FS01:0x23: Version: 1.7.2
06:51:18:WU00:FS01:0x23: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
06:51:18:WU00:FS01:0x23: Org: Cauldron Development LLC
06:51:18:WU00:FS01:0x23: Copyright: Cauldron Development LLC, 2003-2023
06:51:18:WU00:FS01:0x23: Homepage: https://cauldrondevelopment.com/
06:51:18:WU00:FS01:0x23: License: GPL 2+
06:51:18:WU00:FS01:0x23: Date: Aug 3 2023
06:51:18:WU00:FS01:0x23: Time: 08:37:14
06:51:18:WU00:FS01:0x23: Compiler: Visual C++
06:51:18:WU00:FS01:0x23: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
06:51:18:WU00:FS01:0x23: Platform: win32 10
06:51:18:WU00:FS01:0x23: Bits: 64
06:51:18:WU00:FS01:0x23: Mode: Release
06:51:18:WU00:FS01:0x23:************************************ System ************************************
06:51:18:WU00:FS01:0x23: CPU: Intel(R) Core(TM) i7 CPU X 980 @ 3.33GHz
06:51:18:WU00:FS01:0x23: CPU ID: GenuineIntel Family 6 Model 44 Stepping 2
06:51:18:WU00:FS01:0x23: CPUs: 12
06:51:18:WU00:FS01:0x23: Memory: 11.99GiB
06:51:18:WU00:FS01:0x23:Free Memory: 9.26GiB
06:51:18:WU00:FS01:0x23: Threads: WINDOWS_THREADS
06:51:18:WU00:FS01:0x23: OS Version: 6.1
06:51:18:WU00:FS01:0x23:Has Battery: false
06:51:18:WU00:FS01:0x23: On Battery: false
06:51:18:WU00:FS01:0x23: UTC Offset: -7
06:51:18:WU00:FS01:0x23: PID: 5248
06:51:18:WU00:FS01:0x23: CWD: C:\Users\MikHexa\AppData\Roaming\FAHClient\work
06:51:18:WU00:FS01:0x23: Exec: C:\Users\MikHexa\AppData\Roaming\FAHClient\cores\cores.foldingathome.org\openmm-core-23\windows-10-64bit\release\0x23-8.0.3\Core_23.fah\FahCore_23.exe
06:51:18:WU00:FS01:0x23:************************************ OpenMM ************************************
06:51:18:WU00:FS01:0x23: Version: 8.0.0
06:51:18:WU00:FS01:0x23:********************************************************************************
06:51:18:WU00:FS01:0x23:Project: 17116 (Run 5, Clone 470, Gen 0)
06:51:18:WU00:FS01:0x23:Reading tar file core.xml
06:51:18:WU00:FS01:0x23:Reading tar file integrator.xml.bz2
06:51:18:WU00:FS01:0x23:Reading tar file state.xml.bz2
06:51:18:WU00:FS01:0x23:Reading tar file system.xml.bz2
06:51:18:WU00:FS01:0x23:Digital signatures verified
06:51:18:WU00:FS01:0x23:Folding@home GPU Core23 Folding@home Core
06:51:18:WU00:FS01:0x23:Version 8.0.3
06:51:18:WU00:FS01:0x23: Checkpoint write interval: 95088 steps (50%) [2 total]
06:51:18:WU00:FS01:0x23: JSON viewer frame write interval: 1901 steps (1%) [100 total]
06:51:18:WU00:FS01:0x23: XTC frame write interval: 190176 steps (1e+02%) [1 total]
06:51:18:WU00:FS01:0x23: Global context and integrator variables write interval: disabled
06:51:18:WU00:FS01:0x23:There are 4 platforms available.
06:51:18:WU00:FS01:0x23:Platform 0: Reference
06:51:18:WU00:FS01:0x23:Platform 1: CPU
06:51:18:WU00:FS01:0x23:Platform 2: OpenCL
06:51:18:WU00:FS01:0x23: opencl-device 0 specified
06:51:18:WU00:FS01:0x23:Platform 3: CUDA
06:51:18:WU00:FS01:0x23: cuda-device 0 specified
06:51:28:WU00:FS01:0x23:Attempting to create CUDA context:
06:51:28:WU00:FS01:0x23: Configuring platform CUDA
06:51:35:WU00:FS01:0x23: Using CUDA on CUDA Platform and gpu 0
06:51:35:WU00:FS01:0x23: GPU info: Platform: CUDA
06:51:35:WU00:FS01:0x23: GPU info: PlatformIndex: 0
06:51:35:WU00:FS01:0x23: GPU info: Device: NVIDIA GeForce RTX 2070 SUPER
06:51:35:WU00:FS01:0x23: GPU info: DeviceIndex: 0
06:51:35:WU00:FS01:0x23: GPU info: Vendor: 0x10de
06:51:35:WU00:FS01:0x23: GPU info: PCI: 03:00:00
06:51:35:WU00:FS01:0x23: GPU info: Compute: 7.5
06:51:35:WU00:FS01:0x23: GPU info: Driver: 11.4
06:51:35:WU00:FS01:0x23: GPU info: GPU: true
06:51:36:WU00:FS01:0x23:Completed 0 out of 190176 steps (0%)
06:51:36:WU00:FS01:0x23:Checkpoint completed at step 0
06:51:40:WU00:FS01:0x23:Completed 1902 out of 190176 steps (1%)
06:51:44:WU00:FS01:0x23:Completed 3804 out of 190176 steps (2%)
06:51:45:Removing old file 'configs/config-20211120-081755.xml'
06:51:45:Saving configuration to config.xml
06:51:45:<config>
06:51:45: <!-- Folding Slot Configuration -->
06:51:45: <cause v='COVID_19'/>
06:51:45:
06:51:45: <!-- HTTP Server -->
06:51:45: <allow v='127.0.0.1 192.168.168.153 192.168.168.174 192.168.168.213'/>
06:51:45:
06:51:45: <!-- Network -->
06:51:45: <proxy v=':8080'/>
06:51:45:
06:51:45: <!-- Remote Command Server -->
06:51:45: <password v='*****'/>
06:51:45:
06:51:45: <!-- Slot Control -->
06:51:45: <power v='full'/>
06:51:45:
06:51:45: <!-- User Information -->
06:51:45: <passkey v='*****'/>
06:51:45: <team v='104636'/>
06:51:45: <user v='[EV]Tesla'/>
06:51:45:
06:51:45: <!-- Folding Slots -->
06:51:45: <slot id='1' type='GPU'>
06:51:45: <pci-bus v='3'/>
06:51:45: <pci-slot v='0'/>
06:51:45: </slot>
06:51:45:</config>
06:51:48:WU00:FS01:0x23:Completed 5706 out of 190176 steps (3%)
06:51:52:WU00:FS01:0x23:Completed 7608 out of 190176 steps (4%)
06:51:56:WU00:FS01:0x23:Completed 9509 out of 190176 steps (5%)
06:52:00:WU00:FS01:0x23:Completed 11411 out of 190176 steps (6%)
06:52:05:WU00:FS01:0x23:Completed 13313 out of 190176 steps (7%)
06:52:09:WU00:FS01:0x23:Completed 15215 out of 190176 steps (8%)
06:52:13:WU00:FS01:0x23:Completed 17116 out of 190176 steps (9%)
06:52:17:WU00:FS01:0x23:Completed 19018 out of 190176 steps (10%)
06:52:21:WU00:FS01:0x23:Completed 20920 out of 190176 steps (11%)
06:52:25:WU00:FS01:0x23:Completed 22822 out of 190176 steps (12%)
06:52:29:WU00:FS01:0x23:Completed 24723 out of 190176 steps (13%)
06:52:33:WU00:FS01:0x23:Completed 26625 out of 190176 steps (14%)
06:52:37:WU00:FS01:0x23:Completed 28527 out of 190176 steps (15%)
06:52:41:WU00:FS01:0x23:Completed 30429 out of 190176 steps (16%)
06:52:45:WU00:FS01:0x23:Completed 32330 out of 190176 steps (17%)
06:52:49:WU00:FS01:0x23:Completed 34232 out of 190176 steps (18%)
06:52:53:WU00:FS01:0x23:Completed 36134 out of 190176 steps (19%)
06:52:57:WU00:FS01:0x23:Completed 38036 out of 190176 steps (20%)
06:53:02:WU00:FS01:0x23:Completed 39937 out of 190176 steps (21%)
06:53:06:WU00:FS01:0x23:Completed 41839 out of 190176 steps (22%)
06:53:10:WU00:FS01:0x23:Completed 43741 out of 190176 steps (23%)
06:53:14:WU00:FS01:0x23:Completed 45643 out of 190176 steps (24%)
06:53:18:WU00:FS01:0x23:Completed 47544 out of 190176 steps (25%)
06:53:22:WU00:FS01:0x23:Completed 49446 out of 190176 steps (26%)
06:53:26:WU00:FS01:0x23:Completed 51348 out of 190176 steps (27%)
06:53:30:WU00:FS01:0x23:Completed 53250 out of 190176 steps (28%)
06:53:34:WU00:FS01:0x23:Completed 55152 out of 190176 steps (29%)
06:53:38:WU00:FS01:0x23:Completed 57053 out of 190176 steps (30%)
06:53:42:WU00:FS01:0x23:Completed 58955 out of 190176 steps (31%)
06:53:47:WU00:FS01:0x23:Completed 60857 out of 190176 steps (32%)
06:53:50:WU00:FS01:0x23:Completed 62759 out of 190176 steps (33%)
06:53:55:WU00:FS01:0x23:Completed 64660 out of 190176 steps (34%)
06:53:59:WU00:FS01:0x23:Completed 66562 out of 190176 steps (35%)
06:54:03:WU00:FS01:0x23:Completed 68464 out of 190176 steps (36%)
06:54:07:WU00:FS01:0x23:Completed 70366 out of 190176 steps (37%)
06:54:11:WU00:FS01:0x23:Completed 72267 out of 190176 steps (38%)
06:54:15:WU00:FS01:0x23:Completed 74169 out of 190176 steps (39%)
06:54:19:WU00:FS01:0x23:Completed 76071 out of 190176 steps (40%)
06:54:23:WU00:FS01:0x23:Completed 77973 out of 190176 steps (41%)
06:54:27:WU00:FS01:0x23:Completed 79874 out of 190176 steps (42%)
06:54:31:WU00:FS01:0x23:Completed 81776 out of 190176 steps (43%)
06:54:35:WU00:FS01:0x23:Completed 83678 out of 190176 steps (44%)
06:54:39:WU00:FS01:0x23:Completed 85580 out of 190176 steps (45%)
06:54:43:WU00:FS01:0x23:Completed 87481 out of 190176 steps (46%)
06:54:47:WU00:FS01:0x23:Completed 89383 out of 190176 steps (47%)
06:54:51:WU00:FS01:0x23:Completed 91285 out of 190176 steps (48%)
06:54:55:WU00:FS01:0x23:Completed 93187 out of 190176 steps (49%)
06:54:59:WU00:FS01:0x23:Completed 95088 out of 190176 steps (50%)
06:54:59:WU00:FS01:0x23:Checkpoint completed at step 95088
06:55:04:WU00:FS01:0x23:Completed 96990 out of 190176 steps (51%)
06:55:08:WU00:FS01:0x23:Completed 98892 out of 190176 steps (52%)
06:55:11:WU00:FS01:0x23:Completed 100794 out of 190176 steps (53%)
06:55:15:WU00:FS01:0x23:Completed 102696 out of 190176 steps (54%)
06:55:19:WU00:FS01:0x23:Completed 104597 out of 190176 steps (55%)
06:55:23:WU00:FS01:0x23:Completed 106499 out of 190176 steps (56%)
06:55:27:WU00:FS01:0x23:Completed 108401 out of 190176 steps (57%)
06:55:31:WU00:FS01:0x23:Completed 110303 out of 190176 steps (58%)
06:55:36:WU00:FS01:0x23:Completed 112204 out of 190176 steps (59%)
06:55:40:WU00:FS01:0x23:Completed 114106 out of 190176 steps (60%)
06:55:44:WU00:FS01:0x23:Completed 116008 out of 190176 steps (61%)
06:55:48:WU00:FS01:0x23:Completed 117910 out of 190176 steps (62%)
06:55:52:WU00:FS01:0x23:Completed 119811 out of 190176 steps (63%)
06:55:56:WU00:FS01:0x23:Completed 121713 out of 190176 steps (64%)
06:56:00:WU00:FS01:0x23:Completed 123615 out of 190176 steps (65%)
06:56:04:WU00:FS01:0x23:Completed 125517 out of 190176 steps (66%)
06:56:08:WU00:FS01:0x23:Completed 127418 out of 190176 steps (67%)
06:56:12:WU00:FS01:0x23:Completed 129320 out of 190176 steps (68%)
06:56:16:WU00:FS01:0x23:Completed 131222 out of 190176 steps (69%)
06:56:20:WU00:FS01:0x23:Completed 133124 out of 190176 steps (70%)
06:56:24:WU00:FS01:0x23:Completed 135025 out of 190176 steps (71%)
06:56:28:WU00:FS01:0x23:Completed 136927 out of 190176 steps (72%)
06:56:32:WU00:FS01:0x23:Completed 138829 out of 190176 steps (73%)
06:56:36:WU00:FS01:0x23:Completed 140731 out of 190176 steps (74%)
06:56:40:WU00:FS01:0x23:Completed 142632 out of 190176 steps (75%)
06:56:44:WU00:FS01:0x23:Completed 144534 out of 190176 steps (76%)
06:56:48:WU00:FS01:0x23:Completed 146436 out of 190176 steps (77%)
06:56:52:WU00:FS01:0x23:Completed 148338 out of 190176 steps (78%)
06:56:56:WU00:FS01:0x23:Completed 150240 out of 190176 steps (79%)
06:57:00:WU00:FS01:0x23:Completed 152141 out of 190176 steps (80%)
06:57:04:WU00:FS01:0x23:Completed 154043 out of 190176 steps (81%)
06:57:08:WU00:FS01:0x23:Completed 155945 out of 190176 steps (82%)
06:57:12:WU00:FS01:0x23:Completed 157847 out of 190176 steps (83%)
06:57:16:WU00:FS01:0x23:Completed 159748 out of 190176 steps (84%)
06:57:20:WU00:FS01:0x23:Completed 161650 out of 190176 steps (85%)
06:57:24:WU00:FS01:0x23:Completed 163552 out of 190176 steps (86%)
06:57:29:WU00:FS01:0x23:Completed 165454 out of 190176 steps (87%)
06:57:33:WU00:FS01:0x23:Completed 167355 out of 190176 steps (88%)
06:57:37:WU00:FS01:0x23:Completed 169257 out of 190176 steps (89%)
06:57:41:WU00:FS01:0x23:Completed 171159 out of 190176 steps (90%)
06:57:45:WU00:FS01:0x23:Completed 173061 out of 190176 steps (91%)
06:57:49:WU00:FS01:0x23:Completed 174962 out of 190176 steps (92%)
06:57:53:WU00:FS01:0x23:Completed 176864 out of 190176 steps (93%)
06:57:57:WU00:FS01:0x23:Completed 178766 out of 190176 steps (94%)
06:58:01:WU00:FS01:0x23:Completed 180668 out of 190176 steps (95%)
06:58:05:WU00:FS01:0x23:Completed 182569 out of 190176 steps (96%)
06:58:09:WU00:FS01:0x23:Completed 184471 out of 190176 steps (97%)
06:58:13:WU00:FS01:0x23:Completed 186373 out of 190176 steps (98%)
06:58:17:WU00:FS01:0x23:Completed 188275 out of 190176 steps (99%)
06:58:17:WU01:FS01:Connecting to assign1.foldingathome.org:80
06:58:18:WU01:FS01:Assigned to work server 128.174.73.74
06:58:18:WU01:FS01:Requesting new work unit for slot 01: gpu:3:0 TU104 [GeForce RTX 2070 SUPER] 8218 from 128.174.73.74
06:58:18:WU01:FS01:Connecting to 128.174.73.74:8080
06:58:21:WU00:FS01:0x23:Completed 190176 out of 190176 steps (100%)
06:58:21:WU00:FS01:0x23:Average performance: 163.495 ns/day
06:58:21:WU00:FS01:0x23:Checkpoint completed at step 190176
06:58:25:WU00:FS01:0x23:Saving result file ..\logfile_01.txt
06:58:25:WU00:FS01:0x23:Saving result file checkpointIntegrator.xml.bz2
06:58:25:WU00:FS01:0x23:Saving result file checkpointState.xml.bz2
06:58:25:WU00:FS01:0x23:Saving result file positions.xtc
06:58:25:WU00:FS01:0x23:Saving result file science.log
06:58:25:WU00:FS01:0x23:Saving result file xtcAtoms.csv.bz2
06:58:25:WU00:FS01:0x23:Folding@home Core Shutdown: FINISHED_UNIT
06:58:30:WU01:FS01:Downloading 37.73MiB
06:58:36:WU01:FS01:Download 52.51%
06:58:41:WU01:FS01:Download complete
06:58:41:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14955 run:0 clone:253 gen:246 core:0x22 unit:0xf6000000fd000000000000006b3a0000
07:14:18:FS01:Paused
07:14:18:FS01:Shutting core down
07:14:19:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
07:14:31:FS01:Unpaused
07:14:31:WU00:FS01:Starting
07:14:31:ERROR:WU00:FS01:Failed to start core: Failed to rename 'work/00/logfile_01.txt' to 'work/00/logfile_01-20230816-071431.txt': The process cannot access the file because it is being used by another process.
07:16:08:WU00:FS01:Starting
07:16:08:ERROR:WU00:FS01:Failed to start core: Failed to rename 'work/00/logfile_01.txt' to 'work/00/logfile_01-20230816-071608.txt': The process cannot access the file because it is being used by another process.