Page 1 of 2
Extremely Low PPD on Top Pascal Gpu's
Posted: Sat May 22, 2021 4:13 am
by Double J's
Hello, this is my first post on these forums. I'll provide a log later on during the day but I will explain my problem.
One of our folding rig, contains 3 Titan Xp's(Pascal top Gpu).
Highest PPD we have ever seen is almost 3m range, but normally about 2m-2.6m/2.7m range.
And we tried to do some folding today, the PPD went down as much as 450k ppd, Some where 1m, most we seen 1.6m, it is still very low.
This was using 20h2 win 10x64(19042.928) build. I then went back to the build we were originally using that we had much higher PPD (RS2 win 10x64(15063.2108) Build, and it was doing the same thing.
Next, we tried different nvidia drivers, Nope.
I also went out an tried my Sister's Pc, which also had a 1080ti, and the PPD was about 450k, very very low for 1080ti.
They type of projects in question: Covid-19 Related.
These projects have never given these low PPD, we have never seen this before.
Is anyone having this issue as well? Server related issue??
Again, I'll provide some log details later on during the day.
The drivers we used: 466.72, 466.27, and an old driver: 431.94, all behaving in similar fashion.
Before: 466,27 and 431.94 was hovering well over 2m ppd on each of the Titan Xp.
We normally hit all 3 gpu's: well over 6.5m/7m+, now its like 3.5m to way way less.
Hope to see a reply later on, thanks for reading.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Sat May 22, 2021 6:41 am
by Neil-B
Welcome to the forums... There is at least one project that is returning low ppd in play at the moment worth checking if they are all folding the same project ... my 3070 running latest win build drops from 3.6m ppd to 2.6m ppd or lower on the specific project I am thinking of but there are other ones that really don't utilise fast/wide gpus well
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Sat May 22, 2021 5:51 pm
by Double J's
Here is my current log thus far:
Code: Select all
*********************** Log Started 2021-05-22T11:07:48Z ***********************
11:07:48:******************************* libFAH ********************************
11:07:48: Date: Oct 20 2020
11:07:48: Time: 13:36:55
11:07:48: Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
11:07:48: Branch: master
11:07:48: Compiler: Visual C++ 2015
11:07:48: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
11:07:48: Platform: win32 10
11:07:48: Bits: 32
11:07:48: Mode: Release
11:07:48:****************************** FAHClient ******************************
11:07:48: Version: 7.6.21
11:07:48: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:07:48: Copyright: 2020 foldingathome.org
11:07:48: Homepage: https://foldingathome.org/
11:07:48: Date: Oct 20 2020
11:07:48: Time: 13:41:04
11:07:48: Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
11:07:48: Branch: master
11:07:48: Compiler: Visual C++ 2015
11:07:48: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
11:07:48: Platform: win32 10
11:07:48: Bits: 32
11:07:48: Mode: Release
11:07:48: Config: C:\ProgramData\FAHClient\config.xml
11:07:48:******************************** CBang ********************************
11:07:48: Date: Oct 20 2020
11:07:48: Time: 11:36:18
11:07:48: Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
11:07:48: Branch: master
11:07:48: Compiler: Visual C++ 2015
11:07:48: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
11:07:48: Platform: win32 10
11:07:48: Bits: 32
11:07:48: Mode: Release
11:07:48:******************************* System ********************************
11:07:48: CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
11:07:48: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
11:07:48: CPUs: 12
11:07:48: Memory: 15.91GiB
11:07:48: Free Memory: 12.42GiB
11:07:48: Threads: WINDOWS_THREADS
11:07:48: OS Version: 6.2
11:07:48: Has Battery: false
11:07:48: On Battery: false
11:07:48: UTC Offset: -7
11:07:48: PID: 13152
11:07:48: CWD: C:\ProgramData\FAHClient
11:07:48: Win32 Service: false
11:07:48: OS: Windows 10 Enterprise
11:07:48: OS Arch: AMD64
11:07:48: GPUs: 3
11:07:48: GPU 0: Bus:3 Slot:0 Func:0 NVIDIA:8 GP102 [TITAN Xp] 12150
11:07:48: GPU 1: Bus:1 Slot:0 Func:0 NVIDIA:8 GP102 [TITAN Xp] 12150
11:07:48: GPU 2: Bus:6 Slot:0 Func:0 NVIDIA:8 GP102 [TITAN Xp] 12150
11:07:48: CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:10.1
11:07:48: CUDA Device 1: Platform:0 Device:1 Bus:3 Slot:0 Compute:6.1 Driver:10.1
11:07:48: CUDA Device 2: Platform:0 Device:2 Bus:6 Slot:0 Compute:6.1 Driver:10.1
11:07:48:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:431.94
11:07:48:OpenCL Device 1: Platform:0 Device:1 Bus:3 Slot:0 Compute:1.2 Driver:431.94
11:07:48:OpenCL Device 2: Platform:0 Device:2 Bus:6 Slot:0 Compute:1.2 Driver:431.94
11:07:48:***********************************************************************
11:07:48:<config>
11:07:48: <!-- Folding Slot Configuration -->
11:07:48: <cause v='HIGH_PRIORITY'/>
11:07:48:
11:07:48: <!-- Network -->
11:07:48: <proxy v=':8080'/>
11:07:48:
11:07:48: <!-- Slot Control -->
11:07:48: <power v='full'/>
11:07:48:
11:07:48: <!-- User Information -->
11:07:48: <passkey v='*****'/>
11:07:48: <team v='69411'/>
11:07:48: <user v='4-Way-Sli-Disease-Neutralizer'/>
11:07:48:
11:07:48: <!-- Folding Slots -->
11:07:48: <slot id='1' type='GPU'>
11:07:48: <pci-bus v='3'/>
11:07:48: <pci-slot v='0'/>
11:07:48: </slot>
11:07:48: <slot id='2' type='GPU'>
11:07:48: <pci-bus v='1'/>
11:07:48: <pci-slot v='0'/>
11:07:48: </slot>
11:07:48: <slot id='3' type='GPU'>
11:07:48: <pci-bus v='6'/>
11:07:48: <pci-slot v='0'/>
11:07:48: </slot>
11:07:48:</config>
11:07:48:Trying to access database...
11:07:48:Successfully acquired database lock
11:07:48:FS01:Initialized folding slot 01: gpu:3:0 GP102 [TITAN Xp] 12150
11:07:48:FS02:Initialized folding slot 02: gpu:1:0 GP102 [TITAN Xp] 12150
11:07:48:FS03:Initialized folding slot 03: gpu:6:0 GP102 [TITAN Xp] 12150
11:07:48:WU00:FS01:Connecting to assign1.foldingathome.org:80
11:07:48:WU01:FS02:Connecting to assign1.foldingathome.org:80
11:07:48:WU02:FS03:Connecting to assign1.foldingathome.org:80
11:07:49:WU00:FS01:Assigned to work server 54.157.202.86
11:07:49:WU00:FS01:Requesting new work unit for slot 01: gpu:3:0 GP102 [TITAN Xp] 12150 from 54.157.202.86
11:07:49:WU02:FS03:Assigned to work server 54.157.202.86
11:07:49:WU00:FS01:Connecting to 54.157.202.86:8080
11:07:49:WU02:FS03:Requesting new work unit for slot 03: gpu:6:0 GP102 [TITAN Xp] 12150 from 54.157.202.86
11:07:49:WU01:FS02:Assigned to work server 128.174.73.74
11:07:49:WU02:FS03:Connecting to 54.157.202.86:8080
11:07:49:WU01:FS02:Requesting new work unit for slot 02: gpu:1:0 GP102 [TITAN Xp] 12150 from 128.174.73.74
11:07:49:WU01:FS02:Connecting to 128.174.73.74:8080
11:07:49:ERROR:WU00:FS01:Exception: Server did not assign work unit
11:07:50:WU00:FS01:Connecting to assign1.foldingathome.org:80
11:07:50:WU00:FS01:Assigned to work server 207.53.233.146
11:07:50:WU00:FS01:Requesting new work unit for slot 01: gpu:3:0 GP102 [TITAN Xp] 12150 from 207.53.233.146
11:07:50:WU00:FS01:Connecting to 207.53.233.146:8080
11:07:50:WU02:FS03:Downloading 7.71MiB
11:07:51:WU00:FS01:Downloading 6.14MiB
11:07:51:WU00:FS01:Download complete
11:07:51:WU02:FS03:Download complete
11:07:51:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:17804 run:22 clone:78 gen:30 core:0x22 unit:0x0000004e0000001e0000458c00000016
11:07:51:WU00:FS01:Starting
11:07:51:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 13152 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
11:07:51:WU00:FS01:Started FahCore on PID 13336
11:07:51:WU00:FS01:Core PID:12196
11:07:51:WU00:FS01:FahCore 0x22 started
11:07:52:WU02:FS03:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13448 run:1057 clone:96 gen:1 core:0x22 unit:0x00000060000000010000348800000421
11:07:52:WU02:FS03:Starting
11:07:52:WU02:FS03:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 13152 -checkpoint 15 -opencl-platform 0 -opencl-device 2 -cuda-device 2 -gpu-vendor nvidia -gpu 2 -gpu-usage 100
11:07:52:WU02:FS03:Started FahCore on PID 11656
11:07:52:WU02:FS03:Core PID:11896
11:07:52:WU02:FS03:FahCore 0x22 started
11:07:52:WU00:FS01:0x22:*********************** Log Started 2021-05-22T11:07:51Z ***********************
11:07:52:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
11:07:52:WU00:FS01:0x22: Core: Core22
11:07:52:WU00:FS01:0x22: Type: 0x22
11:07:52:WU00:FS01:0x22: Version: 0.0.13
11:07:52:WU00:FS01:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:07:52:WU00:FS01:0x22: Copyright: 2020 foldingathome.org
11:07:52:WU00:FS01:0x22: Homepage: https://foldingathome.org/
11:07:52:WU00:FS01:0x22: Date: Sep 19 2020
11:07:52:WU00:FS01:0x22: Time: 02:35:58
11:07:52:WU00:FS01:0x22: Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
11:07:52:WU00:FS01:0x22: Branch: core22-0.0.13
11:07:52:WU00:FS01:0x22: Compiler: Visual C++ 2015
11:07:52:WU00:FS01:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:07:52:WU00:FS01:0x22: -DOPENMM_GIT_HASH="\"189320d0\""
11:07:52:WU00:FS01:0x22: Platform: win32 10
11:07:52:WU00:FS01:0x22: Bits: 64
11:07:52:WU00:FS01:0x22: Mode: Release
11:07:52:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
11:07:52:WU00:FS01:0x22: <peastman@stanford.edu>
11:07:52:WU00:FS01:0x22: Args: -dir 00 -suffix 01 -version 706 -lifeline 13336 -checkpoint 15
11:07:52:WU00:FS01:0x22: -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
11:07:52:WU00:FS01:0x22: nvidia -gpu 1 -gpu-usage 100
11:07:52:WU00:FS01:0x22:************************************ libFAH ************************************
11:07:52:WU00:FS01:0x22: Date: Sep 7 2020
11:07:52:WU00:FS01:0x22: Time: 19:09:56
11:07:52:WU00:FS01:0x22: Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
11:07:52:WU00:FS01:0x22: Branch: HEAD
11:07:52:WU00:FS01:0x22: Compiler: Visual C++ 2015
11:07:52:WU00:FS01:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:07:52:WU00:FS01:0x22: Platform: win32 10
11:07:52:WU00:FS01:0x22: Bits: 64
11:07:52:WU00:FS01:0x22: Mode: Release
11:07:52:WU00:FS01:0x22:************************************ CBang *************************************
11:07:52:WU00:FS01:0x22: Date: Sep 7 2020
11:07:52:WU00:FS01:0x22: Time: 19:08:30
11:07:52:WU00:FS01:0x22: Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
11:07:52:WU00:FS01:0x22: Branch: HEAD
11:07:52:WU00:FS01:0x22: Compiler: Visual C++ 2015
11:07:52:WU00:FS01:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:07:52:WU00:FS01:0x22: Platform: win32 10
11:07:52:WU00:FS01:0x22: Bits: 64
11:07:52:WU00:FS01:0x22: Mode: Release
11:07:52:WU00:FS01:0x22:************************************ System ************************************
11:07:52:WU00:FS01:0x22: CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
11:07:52:WU00:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
11:07:52:WU00:FS01:0x22: CPUs: 12
11:07:52:WU00:FS01:0x22: Memory: 15.91GiB
11:07:52:WU00:FS01:0x22:Free Memory: 12.57GiB
11:07:52:WU00:FS01:0x22: Threads: WINDOWS_THREADS
11:07:52:WU00:FS01:0x22: OS Version: 6.2
11:07:52:WU00:FS01:0x22:Has Battery: false
11:07:52:WU00:FS01:0x22: On Battery: false
11:07:52:WU00:FS01:0x22: UTC Offset: -7
11:07:52:WU00:FS01:0x22: PID: 12196
11:07:52:WU00:FS01:0x22: CWD: C:\ProgramData\FAHClient\work
11:07:52:WU00:FS01:0x22:************************************ OpenMM ************************************
11:07:52:WU00:FS01:0x22: Revision: 189320d0
11:07:52:WU00:FS01:0x22:********************************************************************************
11:07:52:WU00:FS01:0x22:Project: 17804 (Run 22, Clone 78, Gen 30)
11:07:52:WU00:FS01:0x22:Unit: 0x00000000000000000000000000000000
11:07:52:WU00:FS01:0x22:Reading tar file core.xml
11:07:52:WU00:FS01:0x22:Reading tar file integrator.xml.bz2
11:07:52:WU00:FS01:0x22:Reading tar file state.xml.bz2
11:07:52:WU00:FS01:0x22:Reading tar file system.xml.bz2
11:07:52:WU00:FS01:0x22:Digital signatures verified
11:07:52:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
11:07:52:WU00:FS01:0x22:Version 0.0.13
11:07:52:WU00:FS01:0x22: Checkpoint write interval: 125000 steps (5%) [20 total]
11:07:52:WU00:FS01:0x22: JSON viewer frame write interval: 25000 steps (1%) [100 total]
11:07:52:WU00:FS01:0x22: XTC frame write interval: 25000 steps (1%) [100 total]
11:07:52:WU00:FS01:0x22: Global context and integrator variables write interval: disabled
11:07:52:WU00:FS01:0x22:There are 4 platforms available.
11:07:52:WU00:FS01:0x22:Platform 0: Reference
11:07:52:WU00:FS01:0x22:Platform 1: CPU
11:07:52:WU00:FS01:0x22:Platform 2: OpenCL
11:07:52:WU00:FS01:0x22: opencl-device 1 specified
11:07:52:WU00:FS01:0x22:Platform 3: CUDA
11:07:52:WU00:FS01:0x22: cuda-device 1 specified
11:07:52:WU02:FS03:0x22:*********************** Log Started 2021-05-22T11:07:52Z ***********************
11:07:52:WU02:FS03:0x22:*************************** Core22 Folding@home Core ***************************
11:07:52:WU02:FS03:0x22: Core: Core22
11:07:52:WU02:FS03:0x22: Type: 0x22
11:07:52:WU02:FS03:0x22: Version: 0.0.13
11:07:52:WU02:FS03:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:07:52:WU02:FS03:0x22: Copyright: 2020 foldingathome.org
11:07:52:WU02:FS03:0x22: Homepage: https://foldingathome.org/
11:07:52:WU02:FS03:0x22: Date: Sep 19 2020
11:07:52:WU02:FS03:0x22: Time: 02:35:58
11:07:52:WU02:FS03:0x22: Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
11:07:52:WU02:FS03:0x22: Branch: core22-0.0.13
11:07:52:WU02:FS03:0x22: Compiler: Visual C++ 2015
11:07:52:WU02:FS03:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:07:52:WU02:FS03:0x22: -DOPENMM_GIT_HASH="\"189320d0\""
11:07:52:WU02:FS03:0x22: Platform: win32 10
11:07:52:WU02:FS03:0x22: Bits: 64
11:07:52:WU02:FS03:0x22: Mode: Release
11:07:52:WU02:FS03:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
11:07:52:WU02:FS03:0x22: <peastman@stanford.edu>
11:07:52:WU02:FS03:0x22: Args: -dir 02 -suffix 01 -version 706 -lifeline 11656 -checkpoint 15
11:07:52:WU02:FS03:0x22: -opencl-platform 0 -opencl-device 2 -cuda-device 2 -gpu-vendor
11:07:52:WU02:FS03:0x22: nvidia -gpu 2 -gpu-usage 100
11:07:52:WU02:FS03:0x22:************************************ libFAH ************************************
11:07:52:WU02:FS03:0x22: Date: Sep 7 2020
11:07:52:WU02:FS03:0x22: Time: 19:09:56
11:07:52:WU02:FS03:0x22: Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
11:07:52:WU02:FS03:0x22: Branch: HEAD
11:07:52:WU02:FS03:0x22: Compiler: Visual C++ 2015
11:07:52:WU02:FS03:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:07:52:WU02:FS03:0x22: Platform: win32 10
11:07:52:WU02:FS03:0x22: Bits: 64
11:07:52:WU02:FS03:0x22: Mode: Release
11:07:52:WU02:FS03:0x22:************************************ CBang *************************************
11:07:52:WU02:FS03:0x22: Date: Sep 7 2020
11:07:52:WU02:FS03:0x22: Time: 19:08:30
11:07:52:WU02:FS03:0x22: Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
11:07:52:WU02:FS03:0x22: Branch: HEAD
11:07:52:WU02:FS03:0x22: Compiler: Visual C++ 2015
11:07:52:WU02:FS03:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:07:52:WU02:FS03:0x22: Platform: win32 10
11:07:52:WU02:FS03:0x22: Bits: 64
11:07:52:WU02:FS03:0x22: Mode: Release
11:07:52:WU02:FS03:0x22:************************************ System ************************************
11:07:52:WU02:FS03:0x22: CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
11:07:52:WU02:FS03:0x22: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
11:07:52:WU02:FS03:0x22: CPUs: 12
11:07:52:WU02:FS03:0x22: Memory: 15.91GiB
11:07:52:WU02:FS03:0x22:Free Memory: 12.51GiB
11:07:52:WU02:FS03:0x22: Threads: WINDOWS_THREADS
11:07:52:WU02:FS03:0x22: OS Version: 6.2
11:07:52:WU02:FS03:0x22:Has Battery: false
11:07:52:WU02:FS03:0x22: On Battery: false
11:07:52:WU02:FS03:0x22: UTC Offset: -7
11:07:52:WU02:FS03:0x22: PID: 11896
11:07:52:WU02:FS03:0x22: CWD: C:\ProgramData\FAHClient\work
11:07:52:WU02:FS03:0x22:************************************ OpenMM ************************************
11:07:52:WU02:FS03:0x22: Revision: 189320d0
11:07:52:WU02:FS03:0x22:********************************************************************************
11:07:52:WU02:FS03:0x22:Project: 13448 (Run 1057, Clone 96, Gen 1)
11:07:52:WU02:FS03:0x22:Unit: 0x00000000000000000000000000000000
11:07:52:WU02:FS03:0x22:Reading tar file core.xml
11:07:52:WU02:FS03:0x22:Reading tar file integrator.xml.bz2
11:07:52:WU02:FS03:0x22:Reading tar file state.xml.bz2
11:07:52:WU02:FS03:0x22:Reading tar file system.xml.bz2
11:07:52:WU02:FS03:0x22:Digital signatures verified
11:07:52:WU02:FS03:0x22:Folding@home GPU Core22 Folding@home Core
11:07:52:WU02:FS03:0x22:Version 0.0.13
11:07:52:WU02:FS03:0x22: Checkpoint write interval: 50000 steps (5%) [20 total]
11:07:52:WU02:FS03:0x22: JSON viewer frame write interval: 10000 steps (1%) [100 total]
11:07:52:WU02:FS03:0x22: XTC frame write interval: 250000 steps (25%) [4 total]
11:07:52:WU02:FS03:0x22: Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
11:07:52:WU02:FS03:0x22:There are 4 platforms available.
11:07:52:WU02:FS03:0x22:Platform 0: Reference
11:07:52:WU02:FS03:0x22:Platform 1: CPU
11:07:52:WU02:FS03:0x22:Platform 2: OpenCL
11:07:52:WU02:FS03:0x22: opencl-device 2 specified
11:07:52:WU02:FS03:0x22:Platform 3: CUDA
11:07:52:WU02:FS03:0x22: cuda-device 2 specified
11:07:57:WU00:FS01:0x22:Attempting to create CUDA context:
11:07:57:WU00:FS01:0x22: Configuring platform CUDA
11:07:59:WU00:FS01:0x22: Using CUDA and gpu 1
11:07:59:WU00:FS01:0x22:Completed 0 out of 2500000 steps (0%)
11:08:00:WU00:FS01:0x22:Checkpoint completed at step 0
11:08:01:WU01:FS02:Downloading 11.49MiB
11:08:02:WU01:FS02:Download complete
11:08:03:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:17716 run:5 clone:2 gen:192 core:0x22 unit:0x00000002000000c00000453400000005
11:08:03:WU01:FS02:Starting
11:08:03:WU01:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 13152 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
11:08:03:WU01:FS02:Started FahCore on PID 13592
11:08:03:WU01:FS02:Core PID:9420
11:08:03:WU01:FS02:FahCore 0x22 started
11:08:03:WU02:FS03:0x22:Attempting to create CUDA context:
11:08:03:WU02:FS03:0x22: Configuring platform CUDA
11:08:03:WU01:FS02:0x22:*********************** Log Started 2021-05-22T11:08:03Z ***********************
11:08:03:WU01:FS02:0x22:*************************** Core22 Folding@home Core ***************************
11:08:03:WU01:FS02:0x22: Core: Core22
11:08:03:WU01:FS02:0x22: Type: 0x22
11:08:03:WU01:FS02:0x22: Version: 0.0.13
11:08:03:WU01:FS02:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:08:03:WU01:FS02:0x22: Copyright: 2020 foldingathome.org
11:08:03:WU01:FS02:0x22: Homepage: https://foldingathome.org/
11:08:03:WU01:FS02:0x22: Date: Sep 19 2020
11:08:03:WU01:FS02:0x22: Time: 02:35:58
11:08:03:WU01:FS02:0x22: Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
11:08:03:WU01:FS02:0x22: Branch: core22-0.0.13
11:08:03:WU01:FS02:0x22: Compiler: Visual C++ 2015
11:08:03:WU01:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:08:03:WU01:FS02:0x22: -DOPENMM_GIT_HASH="\"189320d0\""
11:08:03:WU01:FS02:0x22: Platform: win32 10
11:08:03:WU01:FS02:0x22: Bits: 64
11:08:03:WU01:FS02:0x22: Mode: Release
11:08:03:WU01:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
11:08:03:WU01:FS02:0x22: <peastman@stanford.edu>
11:08:03:WU01:FS02:0x22: Args: -dir 01 -suffix 01 -version 706 -lifeline 13592 -checkpoint 15
11:08:03:WU01:FS02:0x22: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
11:08:03:WU01:FS02:0x22: nvidia -gpu 0 -gpu-usage 100
11:08:03:WU01:FS02:0x22:************************************ libFAH ************************************
11:08:03:WU01:FS02:0x22: Date: Sep 7 2020
11:08:03:WU01:FS02:0x22: Time: 19:09:56
11:08:03:WU01:FS02:0x22: Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
11:08:03:WU01:FS02:0x22: Branch: HEAD
11:08:03:WU01:FS02:0x22: Compiler: Visual C++ 2015
11:08:03:WU01:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:08:03:WU01:FS02:0x22: Platform: win32 10
11:08:03:WU01:FS02:0x22: Bits: 64
11:08:03:WU01:FS02:0x22: Mode: Release
11:08:03:WU01:FS02:0x22:************************************ CBang *************************************
11:08:03:WU01:FS02:0x22: Date: Sep 7 2020
11:08:03:WU01:FS02:0x22: Time: 19:08:30
11:08:03:WU01:FS02:0x22: Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
11:08:03:WU01:FS02:0x22: Branch: HEAD
11:08:03:WU01:FS02:0x22: Compiler: Visual C++ 2015
11:08:03:WU01:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:08:03:WU01:FS02:0x22: Platform: win32 10
11:08:03:WU01:FS02:0x22: Bits: 64
11:08:03:WU01:FS02:0x22: Mode: Release
11:08:03:WU01:FS02:0x22:************************************ System ************************************
11:08:03:WU01:FS02:0x22: CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
11:08:03:WU01:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
11:08:03:WU01:FS02:0x22: CPUs: 12
11:08:03:WU01:FS02:0x22: Memory: 15.91GiB
11:08:03:WU01:FS02:0x22:Free Memory: 11.64GiB
11:08:03:WU01:FS02:0x22: Threads: WINDOWS_THREADS
11:08:03:WU01:FS02:0x22: OS Version: 6.2
11:08:03:WU01:FS02:0x22:Has Battery: false
11:08:03:WU01:FS02:0x22: On Battery: false
11:08:03:WU01:FS02:0x22: UTC Offset: -7
11:08:03:WU01:FS02:0x22: PID: 9420
11:08:03:WU01:FS02:0x22: CWD: C:\ProgramData\FAHClient\work
11:08:03:WU01:FS02:0x22:************************************ OpenMM ************************************
11:08:03:WU01:FS02:0x22: Revision: 189320d0
11:08:03:WU01:FS02:0x22:********************************************************************************
11:08:03:WU01:FS02:0x22:Project: 17716 (Run 5, Clone 2, Gen 192)
11:08:03:WU01:FS02:0x22:Unit: 0x00000000000000000000000000000000
11:08:03:WU01:FS02:0x22:Reading tar file core.xml
11:08:03:WU01:FS02:0x22:Reading tar file integrator.xml
11:08:03:WU01:FS02:0x22:Reading tar file state.xml
11:08:03:WU01:FS02:0x22:Reading tar file system.xml
11:08:04:WU01:FS02:0x22:Digital signatures verified
11:08:04:WU01:FS02:0x22:Folding@home GPU Core22 Folding@home Core
11:08:04:WU01:FS02:0x22:Version 0.0.13
11:08:04:WU01:FS02:0x22: Checkpoint write interval: 50000 steps (5%) [20 total]
11:08:04:WU01:FS02:0x22: JSON viewer frame write interval: 10000 steps (1%) [100 total]
11:08:04:WU01:FS02:0x22: XTC frame write interval: 25000 steps (2.5%) [40 total]
11:08:04:WU01:FS02:0x22: Global context and integrator variables write interval: disabled
11:08:04:WU01:FS02:0x22:There are 4 platforms available.
11:08:04:WU01:FS02:0x22:Platform 0: Reference
11:08:04:WU01:FS02:0x22:Platform 1: CPU
11:08:04:WU01:FS02:0x22:Platform 2: OpenCL
11:08:04:WU01:FS02:0x22: opencl-device 0 specified
11:08:04:WU01:FS02:0x22:Platform 3: CUDA
11:08:04:WU01:FS02:0x22: cuda-device 0 specified
11:08:15:WU02:FS03:0x22: Using CUDA and gpu 2
11:08:16:WU02:FS03:0x22:Completed 0 out of 1000000 steps (0%)
11:08:16:WU02:FS03:0x22:Checkpoint completed at step 0
11:08:18:WU01:FS02:0x22:Attempting to create CUDA context:
11:08:18:WU01:FS02:0x22: Configuring platform CUDA
11:08:36:WU01:FS02:0x22: Using CUDA and gpu 0
11:08:36:WU01:FS02:0x22:Completed 0 out of 1000000 steps (0%)
11:08:39:WU01:FS02:0x22:Checkpoint completed at step 0
11:09:07:WU01:FS02:0x22:Completed 10000 out of 1000000 steps (1%)
11:09:16:WU00:FS01:0x22:Completed 25000 out of 2500000 steps (1%)
11:09:36:WU01:FS02:0x22:Completed 20000 out of 1000000 steps (2%)
11:10:04:WU02:FS03:0x22:Completed 10000 out of 1000000 steps (1%)
11:10:06:WU01:FS02:0x22:Completed 30000 out of 1000000 steps (3%)
11:10:31:WU00:FS01:0x22:Completed 50000 out of 2500000 steps (2%)
11:10:35:WU01:FS02:0x22:Completed 40000 out of 1000000 steps (4%)
11:11:05:WU01:FS02:0x22:Completed 50000 out of 1000000 steps (5%)
11:11:08:WU01:FS02:0x22:Checkpoint completed at step 50000
11:11:37:WU01:FS02:0x22:Completed 60000 out of 1000000 steps (6%)
11:11:47:WU00:FS01:0x22:Completed 75000 out of 2500000 steps (3%)
11:11:54:WU02:FS03:0x22:Completed 20000 out of 1000000 steps (2%)
11:12:07:WU01:FS02:0x22:Completed 70000 out of 1000000 steps (7%)
11:12:36:WU01:FS02:0x22:Completed 80000 out of 1000000 steps (8%)
11:13:02:WU00:FS01:0x22:Completed 100000 out of 2500000 steps (4%)
11:13:06:WU01:FS02:0x22:Completed 90000 out of 1000000 steps (9%)
11:13:36:WU01:FS02:0x22:Completed 100000 out of 1000000 steps (10%)
11:13:38:WU01:FS02:0x22:Checkpoint completed at step 100000
11:13:43:WU02:FS03:0x22:Completed 30000 out of 1000000 steps (3%)
11:14:08:WU01:FS02:0x22:Completed 110000 out of 1000000 steps (11%)
11:14:18:WU00:FS01:0x22:Completed 125000 out of 2500000 steps (5%)
11:14:19:WU00:FS01:0x22:Checkpoint completed at step 125000
11:14:38:WU01:FS02:0x22:Completed 120000 out of 1000000 steps (12%)
11:15:07:WU01:FS02:0x22:Completed 130000 out of 1000000 steps (13%)
11:15:30:WU02:FS03:0x22:Completed 40000 out of 1000000 steps (4%)
11:15:34:WU00:FS01:0x22:Completed 150000 out of 2500000 steps (6%)
11:15:37:WU01:FS02:0x22:Completed 140000 out of 1000000 steps (14%)
11:16:07:WU01:FS02:0x22:Completed 150000 out of 1000000 steps (15%)
11:16:10:WU01:FS02:0x22:Checkpoint completed at step 150000
11:16:39:WU01:FS02:0x22:Completed 160000 out of 1000000 steps (16%)
11:16:50:WU00:FS01:0x22:Completed 175000 out of 2500000 steps (7%)
11:17:09:WU01:FS02:0x22:Completed 170000 out of 1000000 steps (17%)
11:17:18:WU02:FS03:0x22:Completed 50000 out of 1000000 steps (5%)
11:17:19:WU02:FS03:0x22:Checkpoint completed at step 50000
11:17:38:WU01:FS02:0x22:Completed 180000 out of 1000000 steps (18%)
11:18:06:WU00:FS01:0x22:Completed 200000 out of 2500000 steps (8%)
11:18:08:WU01:FS02:0x22:Completed 190000 out of 1000000 steps (19%)
11:18:37:WU01:FS02:0x22:Completed 200000 out of 1000000 steps (20%)
11:18:40:WU01:FS02:0x22:Checkpoint completed at step 200000
11:19:08:WU02:FS03:0x22:Completed 60000 out of 1000000 steps (6%)
11:19:10:WU01:FS02:0x22:Completed 210000 out of 1000000 steps (21%)
11:19:22:WU00:FS01:0x22:Completed 225000 out of 2500000 steps (9%)
11:19:39:WU01:FS02:0x22:Completed 220000 out of 1000000 steps (22%)
11:20:08:WU01:FS02:0x22:Completed 230000 out of 1000000 steps (23%)
11:20:37:WU00:FS01:0x22:Completed 250000 out of 2500000 steps (10%)
11:20:38:WU00:FS01:0x22:Checkpoint completed at step 250000
11:20:38:WU01:FS02:0x22:Completed 240000 out of 1000000 steps (24%)
11:20:55:WU02:FS03:0x22:Completed 70000 out of 1000000 steps (7%)
11:21:08:WU01:FS02:0x22:Completed 250000 out of 1000000 steps (25%)
11:21:11:WU01:FS02:0x22:Checkpoint completed at step 250000
11:21:40:WU01:FS02:0x22:Completed 260000 out of 1000000 steps (26%)
11:21:54:WU00:FS01:0x22:Completed 275000 out of 2500000 steps (11%)
11:22:10:WU01:FS02:0x22:Completed 270000 out of 1000000 steps (27%)
11:22:39:WU01:FS02:0x22:Completed 280000 out of 1000000 steps (28%)
11:22:43:WU02:FS03:0x22:Completed 80000 out of 1000000 steps (8%)
11:23:09:WU01:FS02:0x22:Completed 290000 out of 1000000 steps (29%)
11:23:09:WU00:FS01:0x22:Completed 300000 out of 2500000 steps (12%)
11:23:39:WU01:FS02:0x22:Completed 300000 out of 1000000 steps (30%)
11:23:41:WU01:FS02:0x22:Checkpoint completed at step 300000
11:24:11:WU01:FS02:0x22:Completed 310000 out of 1000000 steps (31%)
11:24:25:WU00:FS01:0x22:Completed 325000 out of 2500000 steps (13%)
11:24:32:WU02:FS03:0x22:Completed 90000 out of 1000000 steps (9%)
11:24:40:WU01:FS02:0x22:Completed 320000 out of 1000000 steps (32%)
11:25:10:WU01:FS02:0x22:Completed 330000 out of 1000000 steps (33%)
11:25:40:WU01:FS02:0x22:Completed 340000 out of 1000000 steps (34%)
11:25:41:WU00:FS01:0x22:Completed 350000 out of 2500000 steps (14%)
11:26:09:WU01:FS02:0x22:Completed 350000 out of 1000000 steps (35%)
11:26:12:WU01:FS02:0x22:Checkpoint completed at step 350000
11:26:20:WU02:FS03:0x22:Completed 100000 out of 1000000 steps (10%)
11:26:21:WU02:FS03:0x22:Checkpoint completed at step 100000
11:26:42:WU01:FS02:0x22:Completed 360000 out of 1000000 steps (36%)
11:26:57:WU00:FS01:0x22:Completed 375000 out of 2500000 steps (15%)
11:26:58:WU00:FS01:0x22:Checkpoint completed at step 375000
11:27:12:WU01:FS02:0x22:Completed 370000 out of 1000000 steps (37%)
11:27:42:WU01:FS02:0x22:Completed 380000 out of 1000000 steps (38%)
11:28:10:WU02:FS03:0x22:Completed 110000 out of 1000000 steps (11%)
11:28:12:WU01:FS02:0x22:Completed 390000 out of 1000000 steps (39%)
11:28:13:WU00:FS01:0x22:Completed 400000 out of 2500000 steps (16%)
11:28:42:WU01:FS02:0x22:Completed 400000 out of 1000000 steps (40%)
11:28:44:WU01:FS02:0x22:Checkpoint completed at step 400000
11:29:14:WU01:FS02:0x22:Completed 410000 out of 1000000 steps (41%)
11:29:29:WU00:FS01:0x22:Completed 425000 out of 2500000 steps (17%)
11:29:44:WU01:FS02:0x22:Completed 420000 out of 1000000 steps (42%)
11:30:00:WU02:FS03:0x22:Completed 120000 out of 1000000 steps (12%)
11:30:13:WU01:FS02:0x22:Completed 430000 out of 1000000 steps (43%)
11:30:42:WU01:FS02:0x22:Completed 440000 out of 1000000 steps (44%)
11:30:45:WU00:FS01:0x22:Completed 450000 out of 2500000 steps (18%)
11:31:12:WU01:FS02:0x22:Completed 450000 out of 1000000 steps (45%)
11:31:15:WU01:FS02:0x22:Checkpoint completed at step 450000
11:31:44:WU01:FS02:0x22:Completed 460000 out of 1000000 steps (46%)
11:31:49:WU02:FS03:0x22:Completed 130000 out of 1000000 steps (13%)
11:32:01:WU00:FS01:0x22:Completed 475000 out of 2500000 steps (19%)
11:32:14:WU01:FS02:0x22:Completed 470000 out of 1000000 steps (47%)
11:32:43:WU01:FS02:0x22:Completed 480000 out of 1000000 steps (48%)
11:33:13:WU01:FS02:0x22:Completed 490000 out of 1000000 steps (49%)
11:33:17:WU00:FS01:0x22:Completed 500000 out of 2500000 steps (20%)
11:33:18:WU00:FS01:0x22:Checkpoint completed at step 500000
11:33:37:WU02:FS03:0x22:Completed 140000 out of 1000000 steps (14%)
11:33:42:WU01:FS02:0x22:Completed 500000 out of 1000000 steps (50%)
11:33:45:WU01:FS02:0x22:Checkpoint completed at step 500000
11:34:15:WU01:FS02:0x22:Completed 510000 out of 1000000 steps (51%)
11:34:34:WU00:FS01:0x22:Completed 525000 out of 2500000 steps (21%)
11:34:44:WU01:FS02:0x22:Completed 520000 out of 1000000 steps (52%)
11:35:13:WU01:FS02:0x22:Completed 530000 out of 1000000 steps (53%)
11:35:24:WU02:FS03:0x22:Completed 150000 out of 1000000 steps (15%)
11:35:25:WU02:FS03:0x22:Checkpoint completed at step 150000
11:35:42:WU01:FS02:0x22:Completed 540000 out of 1000000 steps (54%)
11:35:49:WU00:FS01:0x22:Completed 550000 out of 2500000 steps (22%)
11:36:12:WU01:FS02:0x22:Completed 550000 out of 1000000 steps (55%)
11:36:14:WU01:FS02:0x22:Checkpoint completed at step 550000
11:36:44:WU01:FS02:0x22:Completed 560000 out of 1000000 steps (56%)
11:37:04:WU00:FS01:0x22:Completed 575000 out of 2500000 steps (23%)
11:37:13:WU01:FS02:0x22:Completed 570000 out of 1000000 steps (57%)
11:37:13:WU02:FS03:0x22:Completed 160000 out of 1000000 steps (16%)
11:37:43:WU01:FS02:0x22:Completed 580000 out of 1000000 steps (58%)
11:38:13:WU01:FS02:0x22:Completed 590000 out of 1000000 steps (59%)
11:38:20:WU00:FS01:0x22:Completed 600000 out of 2500000 steps (24%)
11:38:42:WU01:FS02:0x22:Completed 600000 out of 1000000 steps (60%)
11:38:45:WU01:FS02:0x22:Checkpoint completed at step 600000
11:39:02:WU02:FS03:0x22:Completed 170000 out of 1000000 steps (17%)
11:39:15:WU01:FS02:0x22:Completed 610000 out of 1000000 steps (61%)
11:39:36:WU00:FS01:0x22:Completed 625000 out of 2500000 steps (25%)
11:39:37:WU00:FS01:0x22:Checkpoint completed at step 625000
11:39:45:WU01:FS02:0x22:Completed 620000 out of 1000000 steps (62%)
11:40:14:WU01:FS02:0x22:Completed 630000 out of 1000000 steps (63%)
11:40:44:WU01:FS02:0x22:Completed 640000 out of 1000000 steps (64%)
11:40:50:WU02:FS03:0x22:Completed 180000 out of 1000000 steps (18%)
11:40:52:WU00:FS01:0x22:Completed 650000 out of 2500000 steps (26%)
11:41:13:WU01:FS02:0x22:Completed 650000 out of 1000000 steps (65%)
11:41:16:WU01:FS02:0x22:Checkpoint completed at step 650000
11:41:46:WU01:FS02:0x22:Completed 660000 out of 1000000 steps (66%)
11:42:08:WU00:FS01:0x22:Completed 675000 out of 2500000 steps (27%)
11:42:16:WU01:FS02:0x22:Completed 670000 out of 1000000 steps (67%)
11:42:39:WU02:FS03:0x22:Completed 190000 out of 1000000 steps (19%)
11:42:45:WU01:FS02:0x22:Completed 680000 out of 1000000 steps (68%)
11:43:15:WU01:FS02:0x22:Completed 690000 out of 1000000 steps (69%)
11:43:24:WU00:FS01:0x22:Completed 700000 out of 2500000 steps (28%)
11:43:44:WU01:FS02:0x22:Completed 700000 out of 1000000 steps (70%)
11:43:47:WU01:FS02:0x22:Checkpoint completed at step 700000
11:44:17:WU01:FS02:0x22:Completed 710000 out of 1000000 steps (71%)
11:44:28:WU02:FS03:0x22:Completed 200000 out of 1000000 steps (20%)
11:44:30:WU02:FS03:0x22:Checkpoint completed at step 200000
11:44:40:WU00:FS01:0x22:Completed 725000 out of 2500000 steps (29%)
11:44:46:WU01:FS02:0x22:Completed 720000 out of 1000000 steps (72%)
11:45:16:WU01:FS02:0x22:Completed 730000 out of 1000000 steps (73%)
11:45:45:WU01:FS02:0x22:Completed 740000 out of 1000000 steps (74%)
11:45:55:WU00:FS01:0x22:Completed 750000 out of 2500000 steps (30%)
11:45:56:WU00:FS01:0x22:Checkpoint completed at step 750000
11:46:15:WU01:FS02:0x22:Completed 750000 out of 1000000 steps (75%)
11:46:18:WU02:FS03:0x22:Completed 210000 out of 1000000 steps (21%)
11:46:18:WU01:FS02:0x22:Checkpoint completed at step 750000
11:46:47:WU01:FS02:0x22:Completed 760000 out of 1000000 steps (76%)
11:47:12:WU00:FS01:0x22:Completed 775000 out of 2500000 steps (31%)
11:47:17:WU01:FS02:0x22:Completed 770000 out of 1000000 steps (77%)
11:47:47:WU01:FS02:0x22:Completed 780000 out of 1000000 steps (78%)
11:48:05:WU02:FS03:0x22:Completed 220000 out of 1000000 steps (22%)
11:48:16:WU01:FS02:0x22:Completed 790000 out of 1000000 steps (79%)
11:48:27:WU00:FS01:0x22:Completed 800000 out of 2500000 steps (32%)
11:48:46:WU01:FS02:0x22:Completed 800000 out of 1000000 steps (80%)
11:48:49:WU01:FS02:0x22:Checkpoint completed at step 800000
11:49:18:WU01:FS02:0x22:Completed 810000 out of 1000000 steps (81%)
11:49:43:WU00:FS01:0x22:Completed 825000 out of 2500000 steps (33%)
11:49:48:WU01:FS02:0x22:Completed 820000 out of 1000000 steps (82%)
11:49:52:WU02:FS03:0x22:Completed 230000 out of 1000000 steps (23%)
11:50:17:WU01:FS02:0x22:Completed 830000 out of 1000000 steps (83%)
11:50:47:WU01:FS02:0x22:Completed 840000 out of 1000000 steps (84%)
11:50:58:WU00:FS01:0x22:Completed 850000 out of 2500000 steps (34%)
11:51:17:WU01:FS02:0x22:Completed 850000 out of 1000000 steps (85%)
11:51:19:WU01:FS02:0x22:Checkpoint completed at step 850000
11:51:41:WU02:FS03:0x22:Completed 240000 out of 1000000 steps (24%)
11:51:49:WU01:FS02:0x22:Completed 860000 out of 1000000 steps (86%)
11:52:15:WU00:FS01:0x22:Completed 875000 out of 2500000 steps (35%)
11:52:15:WU00:FS01:0x22:Checkpoint completed at step 875000
11:52:19:WU01:FS02:0x22:Completed 870000 out of 1000000 steps (87%)
11:52:49:WU01:FS02:0x22:Completed 880000 out of 1000000 steps (88%)
11:53:18:WU01:FS02:0x22:Completed 890000 out of 1000000 steps (89%)
11:53:29:WU02:FS03:0x22:Completed 250000 out of 1000000 steps (25%)
11:53:31:WU02:FS03:0x22:Checkpoint completed at step 250000
11:53:32:WU00:FS01:0x22:Completed 900000 out of 2500000 steps (36%)
11:53:49:WU01:FS02:0x22:Completed 900000 out of 1000000 steps (90%)
11:53:51:WU01:FS02:0x22:Checkpoint completed at step 900000
11:54:21:WU01:FS02:0x22:Completed 910000 out of 1000000 steps (91%)
11:54:48:WU00:FS01:0x22:Completed 925000 out of 2500000 steps (37%)
11:54:51:WU01:FS02:0x22:Completed 920000 out of 1000000 steps (92%)
11:55:20:WU01:FS02:0x22:Completed 930000 out of 1000000 steps (93%)
11:55:22:WU02:FS03:0x22:Completed 260000 out of 1000000 steps (26%)
11:55:50:WU01:FS02:0x22:Completed 940000 out of 1000000 steps (94%)
11:56:03:WU00:FS01:0x22:Completed 950000 out of 2500000 steps (38%)
11:56:19:WU01:FS02:0x22:Completed 950000 out of 1000000 steps (95%)
11:56:22:WU01:FS02:0x22:Checkpoint completed at step 950000
11:56:52:WU01:FS02:0x22:Completed 960000 out of 1000000 steps (96%)
11:57:15:WU02:FS03:0x22:Completed 270000 out of 1000000 steps (27%)
11:57:20:WU00:FS01:0x22:Completed 975000 out of 2500000 steps (39%)
11:57:22:WU01:FS02:0x22:Completed 970000 out of 1000000 steps (97%)
11:57:51:WU01:FS02:0x22:Completed 980000 out of 1000000 steps (98%)
11:58:21:WU01:FS02:0x22:Completed 990000 out of 1000000 steps (99%)
11:58:22:WU03:FS02:Connecting to assign1.foldingathome.org:80
11:58:22:WU03:FS02:Assigned to work server 54.157.202.86
11:58:22:WU03:FS02:Requesting new work unit for slot 02: gpu:1:0 GP102 [TITAN Xp] 12150 from 54.157.202.86
11:58:22:WU03:FS02:Connecting to 54.157.202.86:8080
11:58:23:WU03:FS02:Downloading 6.93MiB
11:58:24:WU03:FS02:Download complete
11:58:24:WU03:FS02:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:13448 run:1430 clone:88 gen:0 core:0x22 unit:0x00000058000000000000348800000596
11:58:35:WU00:FS01:0x22:Completed 1000000 out of 2500000 steps (40%)
11:58:36:WU00:FS01:0x22:Checkpoint completed at step 1000000
11:58:51:WU01:FS02:0x22:Completed 1000000 out of 1000000 steps (100%)
11:58:51:WU01:FS02:0x22:Average performance: 57.9866 ns/day
11:58:54:WU01:FS02:0x22:Checkpoint completed at step 1000000
11:58:57:WU01:FS02:0x22:Saving result file ..\logfile_01.txt
11:58:57:WU01:FS02:0x22:Saving result file checkpointIntegrator.xml
11:58:57:WU01:FS02:0x22:Saving result file checkpointState.xml
11:58:59:WU01:FS02:0x22:Saving result file positions.xtc
11:58:59:WU01:FS02:0x22:Saving result file science.log
11:58:59:WU01:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
11:59:00:WU01:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:59:00:WU01:FS02:Sending unit results: id:01 state:SEND error:NO_ERROR project:17716 run:5 clone:2 gen:192 core:0x22 unit:0x00000002000000c00000453400000005
11:59:00:WU01:FS02:Uploading 12.67MiB to 128.174.73.74
11:59:00:WU03:FS02:Starting
11:59:00:WU01:FS02:Connecting to 128.174.73.74:8080
11:59:00:WU03:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 03 -suffix 01 -version 706 -lifeline 13152 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
11:59:00:WU03:FS02:Started FahCore on PID 5228
11:59:00:WU03:FS02:Core PID:5912
11:59:00:WU03:FS02:FahCore 0x22 started
11:59:01:WU03:FS02:0x22:*********************** Log Started 2021-05-22T11:59:00Z ***********************
11:59:01:WU03:FS02:0x22:*************************** Core22 Folding@home Core ***************************
11:59:01:WU03:FS02:0x22: Core: Core22
11:59:01:WU03:FS02:0x22: Type: 0x22
11:59:01:WU03:FS02:0x22: Version: 0.0.13
11:59:01:WU03:FS02:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:59:01:WU03:FS02:0x22: Copyright: 2020 foldingathome.org
11:59:01:WU03:FS02:0x22: Homepage: https://foldingathome.org/
11:59:01:WU03:FS02:0x22: Date: Sep 19 2020
11:59:01:WU03:FS02:0x22: Time: 02:35:58
11:59:01:WU03:FS02:0x22: Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
11:59:01:WU03:FS02:0x22: Branch: core22-0.0.13
11:59:01:WU03:FS02:0x22: Compiler: Visual C++ 2015
11:59:01:WU03:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:59:01:WU03:FS02:0x22: -DOPENMM_GIT_HASH="\"189320d0\""
11:59:01:WU03:FS02:0x22: Platform: win32 10
11:59:01:WU03:FS02:0x22: Bits: 64
11:59:01:WU03:FS02:0x22: Mode: Release
11:59:01:WU03:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
11:59:01:WU03:FS02:0x22: <peastman@stanford.edu>
11:59:01:WU03:FS02:0x22: Args: -dir 03 -suffix 01 -version 706 -lifeline 5228 -checkpoint 15
11:59:01:WU03:FS02:0x22: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
11:59:01:WU03:FS02:0x22: nvidia -gpu 0 -gpu-usage 100
11:59:01:WU03:FS02:0x22:************************************ libFAH ************************************
11:59:01:WU03:FS02:0x22: Date: Sep 7 2020
11:59:01:WU03:FS02:0x22: Time: 19:09:56
11:59:01:WU03:FS02:0x22: Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
11:59:01:WU03:FS02:0x22: Branch: HEAD
11:59:01:WU03:FS02:0x22: Compiler: Visual C++ 2015
11:59:01:WU03:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:59:01:WU03:FS02:0x22: Platform: win32 10
11:59:01:WU03:FS02:0x22: Bits: 64
11:59:01:WU03:FS02:0x22: Mode: Release
11:59:01:WU03:FS02:0x22:************************************ CBang *************************************
11:59:01:WU03:FS02:0x22: Date: Sep 7 2020
11:59:01:WU03:FS02:0x22: Time: 19:08:30
11:59:01:WU03:FS02:0x22: Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
11:59:01:WU03:FS02:0x22: Branch: HEAD
11:59:01:WU03:FS02:0x22: Compiler: Visual C++ 2015
11:59:01:WU03:FS02:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:59:01:WU03:FS02:0x22: Platform: win32 10
11:59:01:WU03:FS02:0x22: Bits: 64
11:59:01:WU03:FS02:0x22: Mode: Release
11:59:01:WU03:FS02:0x22:************************************ System ************************************
11:59:01:WU03:FS02:0x22: CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
11:59:01:WU03:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
11:59:01:WU03:FS02:0x22: CPUs: 12
11:59:01:WU03:FS02:0x22: Memory: 15.91GiB
11:59:01:WU03:FS02:0x22:Free Memory: 9.02GiB
11:59:01:WU03:FS02:0x22: Threads: WINDOWS_THREADS
11:59:01:WU03:FS02:0x22: OS Version: 6.2
11:59:01:WU03:FS02:0x22:Has Battery: false
11:59:01:WU03:FS02:0x22: On Battery: false
11:59:01:WU03:FS02:0x22: UTC Offset: -7
11:59:01:WU03:FS02:0x22: PID: 5912
11:59:01:WU03:FS02:0x22: CWD: C:\ProgramData\FAHClient\work
11:59:01:WU03:FS02:0x22:************************************ OpenMM ************************************
11:59:01:WU03:FS02:0x22: Revision: 189320d0
11:59:01:WU03:FS02:0x22:********************************************************************************
11:59:01:WU03:FS02:0x22:Project: 13448 (Run 1430, Clone 88, Gen 0)
11:59:01:WU03:FS02:0x22:Unit: 0x00000000000000000000000000000000
11:59:01:WU03:FS02:0x22:Reading tar file core.xml
11:59:01:WU03:FS02:0x22:Reading tar file integrator.xml.bz2
11:59:01:WU03:FS02:0x22:Reading tar file state.xml.bz2
11:59:01:WU03:FS02:0x22:Reading tar file system.xml.bz2
11:59:01:WU03:FS02:0x22:Digital signatures verified
11:59:01:WU03:FS02:0x22:Folding@home GPU Core22 Folding@home Core
11:59:01:WU03:FS02:0x22:Version 0.0.13
11:59:01:WU03:FS02:0x22: Checkpoint write interval: 50000 steps (5%) [20 total]
11:59:01:WU03:FS02:0x22: JSON viewer frame write interval: 10000 steps (1%) [100 total]
11:59:01:WU03:FS02:0x22: XTC frame write interval: 250000 steps (25%) [4 total]
11:59:01:WU03:FS02:0x22: Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
11:59:01:WU03:FS02:0x22:There are 4 platforms available.
11:59:01:WU03:FS02:0x22:Platform 0: Reference
11:59:01:WU03:FS02:0x22:Platform 1: CPU
11:59:01:WU03:FS02:0x22:Platform 2: OpenCL
11:59:01:WU03:FS02:0x22: opencl-device 0 specified
11:59:01:WU03:FS02:0x22:Platform 3: CUDA
11:59:01:WU03:FS02:0x22: cuda-device 0 specified
11:59:06:WU01:FS02:Upload 64.62%
11:59:07:WU02:FS03:0x22:Completed 280000 out of 1000000 steps (28%)
11:59:11:WU01:FS02:Upload complete
11:59:11:WU01:FS02:Server responded WORK_ACK (400)
11:59:11:WU01:FS02:Final credit estimate, 95645.00 points
11:59:11:WU01:FS02:Cleaning up
11:59:21:WU03:FS02:0x22:Attempting to create CUDA context:
11:59:21:WU03:FS02:0x22: Configuring platform CUDA
11:59:29:WU03:FS02:0x22: Using CUDA and gpu 0
11:59:29:WU03:FS02:0x22:Completed 0 out of 1000000 steps (0%)
11:59:30:WU03:FS02:0x22:Checkpoint completed at step 0
11:59:52:WU00:FS01:0x22:Completed 1025000 out of 2500000 steps (41%)
12:00:55:WU03:FS02:0x22:Completed 10000 out of 1000000 steps (1%)
12:00:58:WU02:FS03:0x22:Completed 290000 out of 1000000 steps (29%)
12:01:08:WU00:FS01:0x22:Completed 1050000 out of 2500000 steps (42%)
12:02:20:WU03:FS02:0x22:Completed 20000 out of 1000000 steps (2%)
12:02:23:WU00:FS01:0x22:Completed 1075000 out of 2500000 steps (43%)
12:02:48:WU02:FS03:0x22:Completed 300000 out of 1000000 steps (30%)
12:02:49:WU02:FS03:0x22:Checkpoint completed at step 300000
12:03:38:WU00:FS01:0x22:Completed 1100000 out of 2500000 steps (44%)
12:03:45:WU03:FS02:0x22:Completed 30000 out of 1000000 steps (3%)
12:04:39:WU02:FS03:0x22:Completed 310000 out of 1000000 steps (31%)
12:04:53:WU00:FS01:0x22:Completed 1125000 out of 2500000 steps (45%)
12:04:54:WU00:FS01:0x22:Checkpoint completed at step 1125000
12:05:10:WU03:FS02:0x22:Completed 40000 out of 1000000 steps (4%)
12:06:10:WU00:FS01:0x22:Completed 1150000 out of 2500000 steps (46%)
12:06:29:WU02:FS03:0x22:Completed 320000 out of 1000000 steps (32%)
12:06:35:WU03:FS02:0x22:Completed 50000 out of 1000000 steps (5%)
12:06:36:WU03:FS02:0x22:Checkpoint completed at step 50000
12:07:25:WU00:FS01:0x22:Completed 1175000 out of 2500000 steps (47%)
12:08:02:WU03:FS02:0x22:Completed 60000 out of 1000000 steps (6%)
12:08:20:WU02:FS03:0x22:Completed 330000 out of 1000000 steps (33%)
12:08:41:WU00:FS01:0x22:Completed 1200000 out of 2500000 steps (48%)
12:09:28:WU03:FS02:0x22:Completed 70000 out of 1000000 steps (7%)
12:09:56:WU00:FS01:0x22:Completed 1225000 out of 2500000 steps (49%)
12:10:11:WU02:FS03:0x22:Completed 340000 out of 1000000 steps (34%)
12:10:53:WU03:FS02:0x22:Completed 80000 out of 1000000 steps (8%)
12:11:11:WU00:FS01:0x22:Completed 1250000 out of 2500000 steps (50%)
12:11:12:WU00:FS01:0x22:Checkpoint completed at step 1250000
12:12:03:WU02:FS03:0x22:Completed 350000 out of 1000000 steps (35%)
12:12:04:WU02:FS03:0x22:Checkpoint completed at step 350000
12:12:19:WU03:FS02:0x22:Completed 90000 out of 1000000 steps (9%)
12:12:28:WU00:FS01:0x22:Completed 1275000 out of 2500000 steps (51%)
12:13:43:WU00:FS01:0x22:Completed 1300000 out of 2500000 steps (52%)
12:13:45:WU03:FS02:0x22:Completed 100000 out of 1000000 steps (10%)
12:13:46:WU03:FS02:0x22:Checkpoint completed at step 100000
12:13:55:WU02:FS03:0x22:Completed 360000 out of 1000000 steps (36%)
12:14:58:WU00:FS01:0x22:Completed 1325000 out of 2500000 steps (53%)
12:15:11:WU03:FS02:0x22:Completed 110000 out of 1000000 steps (11%)
12:15:46:WU02:FS03:0x22:Completed 370000 out of 1000000 steps (37%)
12:16:14:WU00:FS01:0x22:Completed 1350000 out of 2500000 steps (54%)
12:16:37:WU03:FS02:0x22:Completed 120000 out of 1000000 steps (12%)
12:17:29:WU00:FS01:0x22:Completed 1375000 out of 2500000 steps (55%)
12:17:30:WU00:FS01:0x22:Checkpoint completed at step 1375000
12:17:37:WU02:FS03:0x22:Completed 380000 out of 1000000 steps (38%)
12:18:01:WU03:FS02:0x22:Completed 130000 out of 1000000 steps (13%)
12:18:45:WU00:FS01:0x22:Completed 1400000 out of 2500000 steps (56%)
12:19:26:WU03:FS02:0x22:Completed 140000 out of 1000000 steps (14%)
12:19:27:WU02:FS03:0x22:Completed 390000 out of 1000000 steps (39%)
12:20:00:WU00:FS01:0x22:Completed 1425000 out of 2500000 steps (57%)
12:20:52:WU03:FS02:0x22:Completed 150000 out of 1000000 steps (15%)
12:20:53:WU03:FS02:0x22:Checkpoint completed at step 150000
12:21:16:WU00:FS01:0x22:Completed 1450000 out of 2500000 steps (58%)
12:21:17:WU02:FS03:0x22:Completed 400000 out of 1000000 steps (40%)
12:21:18:WU02:FS03:0x22:Checkpoint completed at step 400000
12:22:18:WU03:FS02:0x22:Completed 160000 out of 1000000 steps (16%)
12:22:31:WU00:FS01:0x22:Completed 1475000 out of 2500000 steps (59%)
12:23:09:WU02:FS03:0x22:Completed 410000 out of 1000000 steps (41%)
12:23:44:WU03:FS02:0x22:Completed 170000 out of 1000000 steps (17%)
12:23:47:WU00:FS01:0x22:Completed 1500000 out of 2500000 steps (60%)
12:23:48:WU00:FS01:0x22:Checkpoint completed at step 1500000
12:25:00:WU02:FS03:0x22:Completed 420000 out of 1000000 steps (42%)
12:25:03:WU00:FS01:0x22:Completed 1525000 out of 2500000 steps (61%)
12:25:10:WU03:FS02:0x22:Completed 180000 out of 1000000 steps (18%)
12:26:18:WU00:FS01:0x22:Completed 1550000 out of 2500000 steps (62%)
12:26:36:WU03:FS02:0x22:Completed 190000 out of 1000000 steps (19%)
12:26:51:WU02:FS03:0x22:Completed 430000 out of 1000000 steps (43%)
12:27:33:WU00:FS01:0x22:Completed 1575000 out of 2500000 steps (63%)
12:28:02:WU03:FS02:0x22:Completed 200000 out of 1000000 steps (20%)
12:28:03:WU03:FS02:0x22:Checkpoint completed at step 200000
12:28:42:WU02:FS03:0x22:Completed 440000 out of 1000000 steps (44%)
12:28:49:WU00:FS01:0x22:Completed 1600000 out of 2500000 steps (64%)
12:29:28:WU03:FS02:0x22:Completed 210000 out of 1000000 steps (21%)
12:30:04:WU00:FS01:0x22:Completed 1625000 out of 2500000 steps (65%)
12:30:05:WU00:FS01:0x22:Checkpoint completed at step 1625000
12:30:32:WU02:FS03:0x22:Completed 450000 out of 1000000 steps (45%)
12:30:33:WU02:FS03:0x22:Checkpoint completed at step 450000
12:30:54:WU03:FS02:0x22:Completed 220000 out of 1000000 steps (22%)
12:31:20:WU00:FS01:0x22:Completed 1650000 out of 2500000 steps (66%)
12:32:20:WU03:FS02:0x22:Completed 230000 out of 1000000 steps (23%)
12:32:24:WU02:FS03:0x22:Completed 460000 out of 1000000 steps (46%)
12:32:36:WU00:FS01:0x22:Completed 1675000 out of 2500000 steps (67%)
12:33:44:WU03:FS02:0x22:Completed 240000 out of 1000000 steps (24%)
12:33:51:WU00:FS01:0x22:Completed 1700000 out of 2500000 steps (68%)
12:34:14:WU02:FS03:0x22:Completed 470000 out of 1000000 steps (47%)
12:35:06:WU00:FS01:0x22:Completed 1725000 out of 2500000 steps (69%)
12:35:09:WU03:FS02:0x22:Completed 250000 out of 1000000 steps (25%)
12:35:11:WU03:FS02:0x22:Checkpoint completed at step 250000
12:36:05:WU02:FS03:0x22:Completed 480000 out of 1000000 steps (48%)
12:36:21:WU00:FS01:0x22:Completed 1750000 out of 2500000 steps (70%)
12:36:22:WU00:FS01:0x22:Checkpoint completed at step 1750000
12:36:39:WU03:FS02:0x22:Completed 260000 out of 1000000 steps (26%)
12:37:38:WU00:FS01:0x22:Completed 1775000 out of 2500000 steps (71%)
12:37:56:WU02:FS03:0x22:Completed 490000 out of 1000000 steps (49%)
12:38:06:WU03:FS02:0x22:Completed 270000 out of 1000000 steps (27%)
12:38:53:WU00:FS01:0x22:Completed 1800000 out of 2500000 steps (72%)
12:39:34:WU03:FS02:0x22:Completed 280000 out of 1000000 steps (28%)
12:39:45:WU02:FS03:0x22:Completed 500000 out of 1000000 steps (50%)
12:39:46:WU02:FS03:0x22:Checkpoint completed at step 500000
12:40:08:WU00:FS01:0x22:Completed 1825000 out of 2500000 steps (73%)
12:41:02:WU03:FS02:0x22:Completed 290000 out of 1000000 steps (29%)
12:41:23:WU00:FS01:0x22:Completed 1850000 out of 2500000 steps (74%)
12:41:33:WU02:FS03:0x22:Completed 510000 out of 1000000 steps (51%)
12:42:29:WU03:FS02:0x22:Completed 300000 out of 1000000 steps (30%)
12:42:30:WU03:FS02:0x22:Checkpoint completed at step 300000
12:42:39:WU00:FS01:0x22:Completed 1875000 out of 2500000 steps (75%)
12:42:39:WU00:FS01:0x22:Checkpoint completed at step 1875000
12:43:21:WU02:FS03:0x22:Completed 520000 out of 1000000 steps (52%)
12:43:55:WU00:FS01:0x22:Completed 1900000 out of 2500000 steps (76%)
12:43:58:WU03:FS02:0x22:Completed 310000 out of 1000000 steps (31%)
12:45:09:WU02:FS03:0x22:Completed 530000 out of 1000000 steps (53%)
12:45:10:WU00:FS01:0x22:Completed 1925000 out of 2500000 steps (77%)
12:45:27:WU03:FS02:0x22:Completed 320000 out of 1000000 steps (32%)
12:46:25:WU00:FS01:0x22:Completed 1950000 out of 2500000 steps (78%)
12:46:56:WU03:FS02:0x22:Completed 330000 out of 1000000 steps (33%)
12:46:58:WU02:FS03:0x22:Completed 540000 out of 1000000 steps (54%)
12:47:41:WU00:FS01:0x22:Completed 1975000 out of 2500000 steps (79%)
12:48:25:WU03:FS02:0x22:Completed 340000 out of 1000000 steps (34%)
12:48:45:WU02:FS03:0x22:Completed 550000 out of 1000000 steps (55%)
12:48:47:WU02:FS03:0x22:Checkpoint completed at step 550000
12:48:56:WU00:FS01:0x22:Completed 2000000 out of 2500000 steps (80%)
12:48:57:WU00:FS01:0x22:Checkpoint completed at step 2000000
12:49:53:WU03:FS02:0x22:Completed 350000 out of 1000000 steps (35%)
12:49:54:WU03:FS02:0x22:Checkpoint completed at step 350000
12:50:13:WU00:FS01:0x22:Completed 2025000 out of 2500000 steps (81%)
12:50:35:WU02:FS03:0x22:Completed 560000 out of 1000000 steps (56%)
12:51:22:WU03:FS02:0x22:Completed 360000 out of 1000000 steps (36%)
12:51:28:WU00:FS01:0x22:Completed 2050000 out of 2500000 steps (82%)
12:52:22:WU02:FS03:0x22:Completed 570000 out of 1000000 steps (57%)
12:52:43:WU00:FS01:0x22:Completed 2075000 out of 2500000 steps (83%)
12:52:50:WU03:FS02:0x22:Completed 370000 out of 1000000 steps (37%)
12:53:59:WU00:FS01:0x22:Completed 2100000 out of 2500000 steps (84%)
12:54:10:WU02:FS03:0x22:Completed 580000 out of 1000000 steps (58%)
12:54:18:WU03:FS02:0x22:Completed 380000 out of 1000000 steps (38%)
12:55:14:WU00:FS01:0x22:Completed 2125000 out of 2500000 steps (85%)
12:55:15:WU00:FS01:0x22:Checkpoint completed at step 2125000
12:55:46:WU03:FS02:0x22:Completed 390000 out of 1000000 steps (39%)
12:55:57:WU02:FS03:0x22:Completed 590000 out of 1000000 steps (59%)
12:56:30:WU00:FS01:0x22:Completed 2150000 out of 2500000 steps (86%)
12:57:13:WU03:FS02:0x22:Completed 400000 out of 1000000 steps (40%)
12:57:14:WU03:FS02:0x22:Checkpoint completed at step 400000
12:57:44:WU02:FS03:0x22:Completed 600000 out of 1000000 steps (60%)
12:57:46:WU00:FS01:0x22:Completed 2175000 out of 2500000 steps (87%)
12:57:46:WU02:FS03:0x22:Checkpoint completed at step 600000
12:58:43:WU03:FS02:0x22:Completed 410000 out of 1000000 steps (41%)
12:59:01:WU00:FS01:0x22:Completed 2200000 out of 2500000 steps (88%)
12:59:34:WU02:FS03:0x22:Completed 610000 out of 1000000 steps (61%)
13:00:11:WU03:FS02:0x22:Completed 420000 out of 1000000 steps (42%)
13:00:16:WU00:FS01:0x22:Completed 2225000 out of 2500000 steps (89%)
13:01:22:WU02:FS03:0x22:Completed 620000 out of 1000000 steps (62%)
13:01:32:WU00:FS01:0x22:Completed 2250000 out of 2500000 steps (90%)
13:01:33:WU00:FS01:0x22:Checkpoint completed at step 2250000
13:01:39:WU03:FS02:0x22:Completed 430000 out of 1000000 steps (43%)
13:02:48:WU00:FS01:0x22:Completed 2275000 out of 2500000 steps (91%)
13:03:07:WU03:FS02:0x22:Completed 440000 out of 1000000 steps (44%)
13:03:11:WU02:FS03:0x22:Completed 630000 out of 1000000 steps (63%)
13:04:03:WU00:FS01:0x22:Completed 2300000 out of 2500000 steps (92%)
13:04:35:WU03:FS02:0x22:Completed 450000 out of 1000000 steps (45%)
13:04:37:WU03:FS02:0x22:Checkpoint completed at step 450000
13:04:59:WU02:FS03:0x22:Completed 640000 out of 1000000 steps (64%)
13:05:19:WU00:FS01:0x22:Completed 2325000 out of 2500000 steps (93%)
13:06:04:WU03:FS02:0x22:Completed 460000 out of 1000000 steps (46%)
13:06:34:WU00:FS01:0x22:Completed 2350000 out of 2500000 steps (94%)
13:06:46:WU02:FS03:0x22:Completed 650000 out of 1000000 steps (65%)
13:06:47:WU02:FS03:0x22:Checkpoint completed at step 650000
13:07:32:WU03:FS02:0x22:Completed 470000 out of 1000000 steps (47%)
13:07:49:WU00:FS01:0x22:Completed 2375000 out of 2500000 steps (95%)
13:07:50:WU00:FS01:0x22:Checkpoint completed at step 2375000
13:08:35:WU02:FS03:0x22:Completed 660000 out of 1000000 steps (66%)
13:09:00:WU03:FS02:0x22:Completed 480000 out of 1000000 steps (48%)
13:09:06:WU00:FS01:0x22:Completed 2400000 out of 2500000 steps (96%)
13:10:21:WU00:FS01:0x22:Completed 2425000 out of 2500000 steps (97%)
13:10:22:WU02:FS03:0x22:Completed 670000 out of 1000000 steps (67%)
13:10:27:WU03:FS02:0x22:Completed 490000 out of 1000000 steps (49%)
13:11:36:WU00:FS01:0x22:Completed 2450000 out of 2500000 steps (98%)
13:11:54:WU03:FS02:0x22:Completed 500000 out of 1000000 steps (50%)
13:11:56:WU03:FS02:0x22:Checkpoint completed at step 500000
13:12:09:WU02:FS03:0x22:Completed 680000 out of 1000000 steps (68%)
13:12:51:WU00:FS01:0x22:Completed 2475000 out of 2500000 steps (99%)
13:12:52:WU01:FS01:Connecting to assign1.foldingathome.org:80
13:12:52:WU01:FS01:Assigned to work server 140.163.4.200
13:12:52:WU01:FS01:Requesting new work unit for slot 01: gpu:3:0 GP102 [TITAN Xp] 12150 from 140.163.4.200
13:12:52:WU01:FS01:Connecting to 140.163.4.200:8080
13:12:53:WU01:FS01:Downloading 20.25MiB
13:12:54:WU01:FS01:Download complete
13:12:54:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:17341 run:4 clone:389 gen:96 core:0x22 unit:0x0000018500000060000043bd00000004
13:13:21:WU03:FS02:0x22:Completed 510000 out of 1000000 steps (51%)
13:13:57:WU02:FS03:0x22:Completed 690000 out of 1000000 steps (69%)
13:14:07:WU00:FS01:0x22:Completed 2500000 out of 2500000 steps (100%)
13:14:07:WU00:FS01:0x22:Average performance: 22.9787 ns/day
13:14:08:WU00:FS01:0x22:Checkpoint completed at step 2500000
13:14:19:WU00:FS01:0x22:Saving result file ..\logfile_01.txt
13:14:19:WU00:FS01:0x22:Saving result file checkpointIntegrator.xml.bz2
13:14:19:WU00:FS01:0x22:Saving result file checkpointState.xml.bz2
13:14:19:WU00:FS01:0x22:Saving result file positions.xtc
13:14:19:WU00:FS01:0x22:Saving result file science.log
13:14:19:WU00:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
13:14:20:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
13:14:20:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:17804 run:22 clone:78 gen:30 core:0x22 unit:0x0000004e0000001e0000458c00000016
13:14:20:WU00:FS01:Uploading 8.57MiB to 207.53.233.146
13:14:20:WU00:FS01:Connecting to 207.53.233.146:8080
13:14:20:WU01:FS01:Starting
13:14:20:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 13152 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
13:14:20:WU01:FS01:Started FahCore on PID 8056
13:14:20:WU01:FS01:Core PID:7508
13:14:20:WU01:FS01:FahCore 0x22 started
This is partially of it, thanks again.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Sat May 22, 2021 6:49 pm
by JimboPalmer
Welcome to Folding@Home!
https://www.techpowerup.com/gpu-specs/titan-xp.c2948
Each card will want 16 lanes of PCI E 3.0 That would be an impressive motherboard.
It is worse than that, the CPU only supports 28 PCI-E lanes in total.
https://www.intel.com/content/www/us/en ... tions.html
Each card has a 250 watt TDP, that would be a large Power Supply. (over 1000 watts)
11:07:48:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:431.94
That is an older driver, but I am not sure a newer driver will perform better.
https://www.nvidia.com/en-us/drivers/results/175537/
The sensor tab of GPU-Z should show GPU utilization and Board Power Draw for each GPU
https://www.techpowerup.com/gpuz/
The main tab should give hints about the actual number of PCI-E lanes each card is using.
My GTX 1060 has PCIe x16 2.0 @ x16 2.0 So it can do PCIE 2.0 16 lanes and is.
My RTX 3060 has PCIe x16 4.0 @ x16 4.0 So it can do PCIE 4.0 16 lanes and is.
I am not sure any of this helps, most of your setup looks correct.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Sat May 22, 2021 8:12 pm
by iero
Given that it is an X99 chipset, i wager that the cards run at x8/x8/x8 with the 4th pcie slot(if there is one on the motherboard disabled). Anyhow, no matter the underlying hardware used, the problem is the sudden drop in performance, compared to earlier runs. Checking the log I saw that a WU accounted for 90K base points. That seems extremely low for a GPU of that power even if it runs in PCIE 3.0 x8 speeds. Could you think of any changes on your setup that might have happened between the expected performance and the drop? New drivers, updated OS, new apps, etc? If we eliminate the possibility of software changes affecting performance, then we can move on to potential hardware changes affecting the outcome.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Sat May 22, 2021 9:44 pm
by JimboPalmer
Could temperatures have risen? Throttling?
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Sat May 22, 2021 11:26 pm
by tictoc
Double J's wrote:Hello, this is my first post on these forums. I'll provide a log later on during the day but I will explain my problem.
One of our folding rig, contains 3 Titan Xp's(Pascal top Gpu).
Highest PPD we have ever seen is almost 3m range, but normally about 2m-2.6m/2.7m range.
And we tried to do some folding today, the PPD went down as much as 450k ppd, Some where 1m, most we seen 1.6m, it is still very low.
This was using 20h2 win 10x64(19042.928) build. I then went back to the build we were originally using that we had much higher PPD (RS2 win 10x64(15063.2108) Build, and it was doing the same thing.
Next, we tried different nvidia drivers, Nope.
I also went out an tried my Sister's Pc, which also had a 1080ti, and the PPD was about 450k, very very low for 1080ti.
They type of projects in question: Covid-19 Related.
These projects have never given these low PPD, we have never seen this before.
Is anyone having this issue as well? Server related issue??
Again, I'll provide some log details later on during the day.
The drivers we used: 466.72, 466.27, and an old driver: 431.94, all behaving in similar fashion.
Before: 466,27 and 431.94 was hovering well over 2m ppd on each of the Titan Xp.
We normally hit all 3 gpu's: well over 6.5m/7m+, now its like 3.5m to way way less.
Hope to see a reply later on, thanks for reading.
The GPU in slot 02 is running slower than the GPU in slot 03.
In the second part of your log two GPUs are running p13448. The GPU in slot 02 has an average tpf of 1:25 (≈2M ppd), compared to 1:47 for slot 03 (≈1.5M ppd). Either that card is throttling due to temps or power limit, or the clocks are set lower than his neighbor.
**Edit** Running at PCIe 3.0 x8 is not going to be a bottleneck on those GPUs.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Sun May 23, 2021 2:15 am
by Double J's
I have 5 gpu's on my other folding rig, 5 Gpu's in total, 4 Titan X's(Maxwell Based Gpu's), and 1 Titan Xp(High Class Pascal Based Gpu), Along with Supermicro x10drg-q, and the overall ppd performance in general, my father's folding rig(the one with the 3 titan Xp's is running on), is yielding better ppd, not only that, I understand that his system is X99 Based and running on intel 5820k(6 core), and Pci-e gen 2 to boot, it was still pumping serious ppd numbers, despite the 3rd slot running on paltry X4 pci-e speeds, and on that slot, I have seen high as almost 3m ppd on that slot, I was quite amazed to see a Pci-e X4, gen 2 slot pumping over 2m with this card, I thought it would be significantly less.
Now, when I had this ppd problem, I remember having this problem on my supermicro folding rig, but soon as I changed my graphic drivers around, it got better but with the other system, nope.
Again, I also tested this with my sisters pc, she has: Asus z10ped8 WS dual Xeon Cpu's with a 1080ti, and that card was pushing 450k tops, that is way to low for such of a card, should be close to 2m at least.
Again, running on X4 pci-e slot, yes, overall daily, it yielded less score then the other slots, BUT, I have seen well over 2m with this slot on a normal bases.
And my father's system, the one with 3 titan Xp's, Asus X99-a Mobo, is running on 16x8x4 configuration, due to the pci-e lanes on the cpu.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Mon May 24, 2021 8:13 pm
by bruce
A GPU gains it's speed by performing a large number of operations in parallel. Any program contains many operations which MUST operate serially.
In FAH's case, if a protein has a large number of atoms, the force calculations can be parallelized. If there is a relatively small number of atoms, the serial orations predominate.
The GPU also varies in how many parallel operations can be done in parallel. With a moderate number of shaders, those proteins with small numbers of atoms can be processed in a very short period of time. Having excess shaders doesn't change the time to process the parallizable operations changes very, very slightly. even if you have a gpu wit a small number of shades, that small number of atoms can be processed quite quickly, except for the serial operations, which does not change.
On the other hand, only proteins with large numbers of atoms will you get a measurable benefit from haveing more shaders.
Scientific considerations determine the complexity of the protein being studied. When we only have small proteins to assign, a "powerful" (wide) GPU will run just as slow as a "weak" (narrow) GPU. your 1080TI (or above) isn't going to be fully loaded. If, on the other hand, we happen to be distributing large proteins, the GPU will be more fully loaded. (We will be trying to issue only small proteins to the narrow WUs which are also looking for work.)
The number of atoms are shown
here listed by project number.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Mon May 24, 2021 8:28 pm
by bruce
As I said
elsewhere
bruce wrote:The smaller card will work up to its limits on any protein. The larger card won't run into it's internal limitations by a smaller protein but it'll still work better than the smaller card.
If you happen to get the most powerful card available, you'll be spending money on capabilities that might be used only part of the time. I vote for mid-range cards and more of them than top-of-the-line cards.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Tue May 25, 2021 10:34 am
by Double J's
I know alot of it depends on the WU(Work Unit).
Just for kicks, I tried another Project other then Covid-19, and the overall PPD was much worse then it currently is.
Projects I am currently doing:
13448
17341
17347
These were. pretty much the same exact projects that I did when the overall PPD was much higher(7M+ PPD).
I understand what your saying about the Atom and such, if the project doesn't call for extra power from the gpu's, its not going to give out full performance from the gpu's.
Some WU's I see, doesn't even hit full power targets from the gpu's, some will hit harder then the other.
And just heads up on Projects:
13448
13437
Power Targets coming from the Titan Xp', is hitting way over 100%, also on project 13448: its hitting about 80% power target range.
sometimes, some WU, the power target, will go fluctuate from way below 40% to above 100% but doesn't effect the overall PPD, I rarely see or any at all that I see very consistent Power Target, for instance 115% and stays there, very interesting how these WU's/Projects work.
Quick question for Bruce:
Project 13448. My current PPD(On the Titan Xp) is 1.2M/1.3M+, now a 980ti or Titan X(Maxwell Based Gpu's)will hit similar numbers, does this mean someone else with a 980ti will hit these Similar ppd numbers or will it be much lower?
And another question in regards about disease:
Which one will give me the overall best PPD in general in the long run, never really asked this till now, out of my curiosity, We been doing All Mostly: Covid-19 & High Priority for our selection of choice, sorry about that.
Thanks for the reply.
EDIT:
Just heads up, 1st Slot/Pci-e X16, Titan Xp, Project 13448 now dishing out, paltry 1.28M now, power target is hitting to its peak performance, my max power target is 120%, its hitting close to it but not always consistent.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Tue May 25, 2021 12:31 pm
by gunnarre
For those who have submitted data to Folding.LAR.systems, the Titan Xp gets around 2.8M PPD on project 13448 on average:
https://folding.lar.systems/gpu_ppd/bra ... n_xp_12150 The PPD is likely closer to 2.6M PPD on Windows due to Linux being more efficient for folding.
I think the 980 TI has a lower PPD, because the 980 TI is a GM200, while the Titan Xp is a GP102.
The Covid Moonshot, of which 13448 is part, is a non-typical project, so it is possible that variability in the work units themselves could be the issue, even though something to do with the software or hardware sounds more likely.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Tue May 25, 2021 6:06 pm
by bruce
Projects 134xx are not really single projects but rather groups of Moonshot projects. Typically there's one group of smaller projects which are only assigned to narrow GPUs and another group of larger projects which are only assigned to wider GPUs. It's an attempt to assign the right projects to the right GPUs but you can't expect your next WUs from the same project number to have identical PPD.
Obviously you're trying to compare one "wide" GPU with another "wide" GPU running an entirely different wide-oriented protein.
It's another innovation introduced by the unusual characteristics of Project Moonshot. (You can expect different innovations in the future.) A limited number of Intel iGPUs may also be participating in the "narrow" collections. Sure, they're slow GPUs, but it's better for them to be working on a WU than having that same WU sitting idle on the server waiting to be assigned.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Mon May 31, 2021 5:49 pm
by Double J's
Just to let everyone know:
Project 18202 is giving me close to 2m ppd.
Re: Extremely Low PPD on Top Pascal Gpu's
Posted: Mon May 31, 2021 6:36 pm
by iero
That means you are back up and running at normal speeds?