Poor performance with Multi-GPU in Windows

[ColonelRyzen]

I am having an issue with folding on my Windows 10 Desktop. I have a GTX 1080 (@ x8 3.0) and Quadro K4000 (x4 2.0) in my system. I have FAH client 7.5.1 installed and I have a CPU slot, and a GPU slot for each card. They have the OpenCL and CUDA indicies set to -1. The GPUs start out folding at full throttle and getting about 700K~800K PPD a piece. My problem is that after about 10 minutes of folding the Quadro drops to about 35K PPD. Each GPU has a CPU thread dedicated to it. I'm not sure what I am doing wrong. Has anyone ran into this before and possibly have a fix for it?

My system is:
Ryzen 2600X
32GB RAM
ASUS B450 Gaming-F
GTX 1080 (PCIe 3.0 x8)
Qaudro K4000 (PCIe 2.0 x4)

[JimboPalmer]

You may wish to get informed help by posting your log, the configuration section will clarify questions we have about your setup. viewtopic.php?f=24&t=26036
Have you uninstalled your F@H Client including data and reinstalled it. The client only looks for new hardware in a new install.

The K4000 may be as much as 20 times slower than a GTX 1080, so your numbers may not be wrong. https://gpu.userbenchmark.com/Compare/N ... 7730vs3603 https://versus.com/en/nvidia-geforce-gt ... adro-k4000

[bruce]

When you post your log, be sure to include the ***SYSTEM*** segment, followed by the config section.

Here's my ***SYSTEM*** segment. (yours will be different).

Code: Select all

05:30:38:******************************* System ********************************
05:30:38:            CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz
05:30:38:         CPU ID: GenuineIntel Family 6 Model 26 Stepping 4
05:30:38:           CPUs: 8
05:30:38:         Memory: 5.99GiB
05:30:38:    Free Memory: 4.10GiB
05:30:38:        Threads: WINDOWS_THREADS
05:30:38:     OS Version: 6.1
05:30:38:    Has Battery: false
05:30:38:     On Battery: false
05:30:38:     UTC Offset: -7
05:30:38:            PID: 6244
05:30:38:            CWD: C:\Users\bruce\AppData\Roaming\FAHClient
05:30:38:             OS: Windows 7 Home Premium
05:30:38:        OS Arch: AMD64
05:30:38:           GPUs: 2
05:30:38:          GPU 0: Bus:3 Slot:0 Func:0 NVIDIA:4 GM107 [GeForce GTX 750 Ti] 1306
05:30:38:          GPU 1: Bus:4 Slot:0 Func:0 NVIDIA:5 GM206 [GeForce GTX 960] 2308
05:30:38:  CUDA Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:5.2 Driver:10.2
05:30:38:  CUDA Device 1: Platform:0 Device:1 Bus:3 Slot:0 Compute:5.0 Driver:10.2
05:30:38:OpenCL Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:1.2 Driver:430.86
05:30:38:OpenCL Device 1: Platform:0 Device:1 Bus:3 Slot:0 Compute:1.2 Driver:430.86
05:30:38:  Win32 Service: false
05:30:38:***********************************************************************

While the index settings should be -1 when you define the slot, they will be configured during initialization. Note the the lines for CUDA Device and for OpenCL Device have various numbers.

[ColonelRyzen]

Here is my log. I would expect it to be slower, but I don't understand why it starts at the same speed as my 1080 then.

Also, it is now slower than my Ryzen 5 2600X (~50K PPD). That doesn't make sense to me. I would think it would be faster than a CPU.

Code: Select all

*********************** Log Started 2019-10-24T00:42:46Z ***********************
00:42:46:************************* Folding@home Client *************************
00:42:46:        Website: https://foldingathome.org/
00:42:46:      Copyright: (c) 2009-2018 foldingathome.org
00:42:46:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:42:46:           Args: --open-web-control
00:42:46:         Config: C:\Users\zacks\AppData\Roaming\FAHClient\config.xml
00:42:46:******************************** Build ********************************
00:42:46:        Version: 7.5.1
00:42:46:           Date: May 11 2018
00:42:46:           Time: 13:06:32
00:42:46:     Repository: Git
00:42:46:       Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
00:42:46:         Branch: master
00:42:46:       Compiler: Visual C++ 2008
00:42:46:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
00:42:46:       Platform: win32 10
00:42:46:           Bits: 32
00:42:46:           Mode: Release
00:42:46:******************************* System ********************************
00:42:46:            CPU: AMD Ryzen 5 2600X Six-Core Processor
00:42:46:         CPU ID: AuthenticAMD Family 23 Model 8 Stepping 2
00:42:46:           CPUs: 12
00:42:46:         Memory: 31.93GiB
00:42:46:    Free Memory: 26.99GiB
00:42:46:        Threads: WINDOWS_THREADS
00:42:46:     OS Version: 6.2
00:42:46:    Has Battery: false
00:42:46:     On Battery: false
00:42:46:     UTC Offset: -4
00:42:46:            PID: 5468
00:42:46:            CWD: C:\Users\zacks\AppData\Roaming\FAHClient
00:42:46:             OS: Windows 10 Enterprise
00:42:46:        OS Arch: AMD64
00:42:46:           GPUs: 2
00:42:46:          GPU 0: Bus:10 Slot:0 Func:0 NVIDIA:3 GK104 [Quadro K4000]
00:42:46:          GPU 1: Bus:9 Slot:0 Func:0 NVIDIA:8 GP104 [GeForce GTX 1080] 8873
00:42:46:  CUDA Device 0: Platform:0 Device:0 Bus:9 Slot:0 Compute:6.1 Driver:10.1
00:42:46:  CUDA Device 1: Platform:0 Device:1 Bus:10 Slot:0 Compute:3.0 Driver:10.1
00:42:46:OpenCL Device 0: Platform:0 Device:0 Bus:9 Slot:0 Compute:1.2 Driver:436.48
00:42:46:OpenCL Device 1: Platform:0 Device:1 Bus:10 Slot:0 Compute:1.2 Driver:436.48
00:42:46:  Win32 Service: false
00:42:46:***********************************************************************
00:42:46:<config>
00:42:46:  <!-- Network -->
00:42:46:  <proxy v=':8080'/>
00:42:46:
00:42:46:  <!-- User Information -->
00:42:46:  <passkey v='********************************'/>
00:42:46:  <team v='223518'/>
00:42:46:  <user v='ColonelRyzen'/>
00:42:46:
00:42:46:  <!-- Folding Slots -->
00:42:46:  <slot id='0' type='CPU'>
00:42:46:    <cpus v='6'/>
00:42:46:  </slot>
00:42:46:  <slot id='1' type='GPU'>
00:42:46:    <gpu-index v='0'/>
00:42:46:  </slot>
00:42:46:  <slot id='2' type='GPU'>
00:42:46:    <gpu-index v='1'/>
00:42:46:  </slot>
00:42:46:</config>
00:42:46:Trying to access database...
00:42:46:Successfully acquired database lock
00:42:46:Enabled folding slot 00: READY cpu:6
00:42:46:Enabled folding slot 01: READY gpu:0:GK104 [Quadro K4000]
00:42:46:Enabled folding slot 02: READY gpu:1:GP104 [GeForce GTX 1080] 8873
00:42:46:WU03:FS02:Starting
00:42:46:WU03:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 03 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
00:42:46:WU03:FS02:Started FahCore on PID 5772
00:42:46:WU03:FS02:Core PID:1764
00:42:46:WU03:FS02:FahCore 0x21 started
00:42:46:WU02:FS00:Starting
00:42:46:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/AVX/Core_a7.fah/FahCore_a7.exe -dir 02 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -np 6
00:42:46:WU02:FS00:Started FahCore on PID 11632
00:42:46:WU02:FS00:Core PID:5960
00:42:46:WU02:FS00:FahCore 0xa7 started
00:42:46:WU01:FS01:Starting
00:42:46:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
00:42:46:WU01:FS01:Started FahCore on PID 5388
00:42:46:WU01:FS01:Core PID:1360
00:42:46:WU01:FS01:FahCore 0x21 started
00:42:47:WU03:FS02:0x21:*********************** Log Started 2019-10-24T00:42:46Z ***********************
00:42:47:WU03:FS02:0x21:Project: 14196 (Run 19, Clone 78, Gen 8)
00:42:47:WU03:FS02:0x21:Unit: 0x0000000b0002894b5d9246b09714f006
00:42:47:WU03:FS02:0x21:CPU: 0x00000000000000000000000000000000
00:42:47:WU03:FS02:0x21:Machine: 2
00:42:47:WU03:FS02:0x21:Digital signatures verified
00:42:47:WU03:FS02:0x21:Folding@home GPU Core21 Folding@home Core
00:42:47:WU03:FS02:0x21:Version 0.0.20
00:42:47:WU02:FS00:0xa7:*********************** Log Started 2019-10-24T00:42:46Z ***********************
00:42:47:WU02:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
00:42:47:WU02:FS00:0xa7:       Type: 0xa7
00:42:47:WU02:FS00:0xa7:       Core: Gromacs
00:42:47:WU02:FS00:0xa7:    Website: https://foldingathome.org/
00:42:47:WU02:FS00:0xa7:  Copyright: (c) 2009-2018 foldingathome.org
00:42:47:WU02:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:42:47:WU02:FS00:0xa7:       Args: -dir 02 -suffix 01 -version 705 -lifeline 11632 -checkpoint 15 -np
00:42:47:WU02:FS00:0xa7:             6
00:42:47:WU02:FS00:0xa7:     Config: <none>
00:42:47:WU02:FS00:0xa7:************************************ Build *************************************
00:42:47:WU02:FS00:0xa7:    Version: 0.0.17
00:42:47:WU02:FS00:0xa7:       Date: Apr 27 2018
00:42:47:WU02:FS00:0xa7:       Time: 16:19:36
00:42:47:WU02:FS00:0xa7: Repository: Git
00:42:47:WU02:FS00:0xa7:   Revision: 21359963583d09ec2063ef946399441c4df4ccd7
00:42:47:WU02:FS00:0xa7:     Branch: master
00:42:47:WU02:FS00:0xa7:   Compiler: Visual C++ 2008
00:42:47:WU02:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
00:42:47:WU02:FS00:0xa7:   Platform: win32 10
00:42:47:WU02:FS00:0xa7:       Bits: 64
00:42:47:WU02:FS00:0xa7:       Mode: Release
00:42:47:WU02:FS00:0xa7:       SIMD: avx_256
00:42:47:WU02:FS00:0xa7:************************************ System ************************************
00:42:47:WU02:FS00:0xa7:        CPU: Unknown
00:42:47:WU02:FS00:0xa7:     CPU ID: 
00:42:47:WU02:FS00:0xa7:       CPUs: 12
00:42:47:WU01:FS01:0x21:*********************** Log Started 2019-10-24T00:42:46Z ***********************
00:42:47:WU02:FS00:0xa7:     Memory: 31.93GiB
00:42:47:WU01:FS01:0x21:Project: 14274 (Run 0, Clone 19, Gen 39)
00:42:47:WU02:FS00:0xa7:Free Memory: 26.97GiB
00:42:47:WU01:FS01:0x21:Unit: 0x0000003b80fccb0a5d98ab51533ed411
00:42:47:WU02:FS00:0xa7:    Threads: WINDOWS_THREADS
00:42:47:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
00:42:47:WU02:FS00:0xa7: OS Version: 6.2
00:42:47:WU01:FS01:0x21:Machine: 1
00:42:47:WU02:FS00:0xa7:Has Battery: false
00:42:47:WU01:FS01:0x21:Digital signatures verified
00:42:47:WU02:FS00:0xa7: On Battery: false
00:42:47:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
00:42:47:WU02:FS00:0xa7: UTC Offset: -4
00:42:47:WU01:FS01:0x21:Version 0.0.20
00:42:47:WU02:FS00:0xa7:        PID: 5960
00:42:47:WU01:FS01:0x21:  Found a checkpoint file
00:42:47:WU02:FS00:0xa7:        CWD: C:\Users\zacks\AppData\Roaming\FAHClient\work
00:42:47:WU02:FS00:0xa7:         OS: Windows 10 Pro
00:42:47:WU02:FS00:0xa7:    OS Arch: AMD64
00:42:47:WU02:FS00:0xa7:********************************************************************************
00:42:47:WU02:FS00:0xa7:Project: 14183 (Run 3, Clone 70, Gen 9)
00:42:47:WU02:FS00:0xa7:Unit: 0x0000000b0002894b5cf9362636419fd1
00:42:47:WU02:FS00:0xa7:Digital signatures verified
00:42:47:WU02:FS00:0xa7:Calling: mdrun -s frame9.tpr -o frame9.trr -cpi state.cpt -cpt 15 -nt 6
00:42:47:WU02:FS00:0xa7:Steps: first=22500000 total=2500000
00:42:47:WU02:FS00:0xa7:Completed 335422 out of 2500000 steps (13%)
00:42:48:10:127.0.0.1:New Web connection
00:42:54:31:127.0.0.1:New Web connection
00:42:56:WU03:FS02:0x21:Completed 0 out of 5000000 steps (0%)
00:42:56:WU03:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
00:43:13:WU01:FS01:0x21:Completed 300000 out of 1000000 steps (30%)
00:43:13:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
00:43:27:Removing old file 'configs/config-20191015-050628.xml'
00:43:27:Saving configuration to config.xml
00:43:27:<config>
00:43:27:  <!-- Network -->
00:43:27:  <proxy v=':8080'/>
00:43:27:
00:43:27:  <!-- User Information -->
00:43:27:  <passkey v='********************************'/>
00:43:27:  <team v='223518'/>
00:43:27:  <user v='ColonelRyzen'/>
00:43:27:
00:43:27:  <!-- Folding Slots -->
00:43:27:  <slot id='0' type='CPU'>
00:43:27:    <cpus v='8'/>
00:43:27:  </slot>
00:43:27:  <slot id='1' type='GPU'>
00:43:27:    <gpu-index v='0'/>
00:43:27:  </slot>
00:43:27:  <slot id='2' type='GPU'>
00:43:27:    <gpu-index v='1'/>
00:43:27:  </slot>
00:43:27:</config>
00:43:27:FS00:Shutting core down
00:43:28:WU02:FS00:0xa7:WARNING:Console control signal 1 on PID 5960
00:43:28:WU02:FS00:0xa7:Exiting, please wait. . .
00:43:28:WU00:FS00:Starting
00:43:28:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/AVX/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15
00:43:28:WU00:FS00:Started FahCore on PID 5480
00:43:28:WU00:FS00:Core PID:6944
00:43:28:WU00:FS00:FahCore 0xa7 started
00:43:28:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
00:43:29:WU00:FS00:0xa7:*********************** Log Started 2019-10-24T00:43:28Z ***********************
00:43:29:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
00:43:29:WU00:FS00:0xa7:       Type: 0xa7
00:43:29:WU00:FS00:0xa7:       Core: Gromacs
00:43:29:WU00:FS00:0xa7:    Website: https://foldingathome.org/
00:43:29:WU00:FS00:0xa7:  Copyright: (c) 2009-2018 foldingathome.org
00:43:29:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:43:29:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 5480 -checkpoint 15
00:43:29:WU00:FS00:0xa7:     Config: <none>
00:43:29:WU00:FS00:0xa7:************************************ Build *************************************
00:43:29:WU00:FS00:0xa7:    Version: 0.0.17
00:43:29:WU00:FS00:0xa7:       Date: Apr 27 2018
00:43:29:WU00:FS00:0xa7:       Time: 16:19:36
00:43:29:WU00:FS00:0xa7: Repository: Git
00:43:29:WU00:FS00:0xa7:   Revision: 21359963583d09ec2063ef946399441c4df4ccd7
00:43:29:WU00:FS00:0xa7:     Branch: master
00:43:29:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
00:43:29:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
00:43:29:WU00:FS00:0xa7:   Platform: win32 10
00:43:29:WU00:FS00:0xa7:       Bits: 64
00:43:29:WU00:FS00:0xa7:       Mode: Release
00:43:29:WU00:FS00:0xa7:       SIMD: avx_256
00:43:29:WU00:FS00:0xa7:************************************ System ************************************
00:43:29:WU00:FS00:0xa7:        CPU: Unknown
00:43:29:WU00:FS00:0xa7:     CPU ID: 
00:43:29:WU00:FS00:0xa7:       CPUs: 12
00:43:29:WU00:FS00:0xa7:     Memory: 31.93GiB
00:43:29:WU00:FS00:0xa7:Free Memory: 26.35GiB
00:43:29:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
00:43:29:WU00:FS00:0xa7: OS Version: 6.2
00:43:29:WU00:FS00:0xa7:Has Battery: false
00:43:29:WU00:FS00:0xa7: On Battery: false
00:43:29:WU00:FS00:0xa7: UTC Offset: -4
00:43:29:WU00:FS00:0xa7:        PID: 6944
00:43:29:WU00:FS00:0xa7:        CWD: C:\Users\zacks\AppData\Roaming\FAHClient\work
00:43:29:WU00:FS00:0xa7:         OS: Windows 10 Pro
00:43:29:WU00:FS00:0xa7:    OS Arch: AMD64
00:43:29:WU00:FS00:0xa7:********************************************************************************
00:43:29:WU00:FS00:0xa7:Project: 13829 (Run 969, Clone 2, Gen 19)
00:43:29:WU00:FS00:0xa7:Unit: 0x0000001680fccb095d66d29e396db3e9
00:43:29:WU00:FS00:0xa7:Digital signatures verified
00:43:29:WU00:FS00:0xa7:Calling: mdrun -s frame19.tpr -o frame19.trr -x frame19.xtc -cpi state.cpt -cpt 15 -nt 1
00:43:29:WU00:FS00:0xa7:Steps: first=2375000 total=125000
00:43:30:WU00:FS00:0xa7:Completed 472 out of 125000 steps (0%)
00:43:47:Removing old file 'configs/config-20191015-050721.xml'
00:43:47:Saving configuration to config.xml
00:43:47:<config>
00:43:47:  <!-- Network -->
00:43:47:  <proxy v=':8080'/>
00:43:47:
00:43:47:  <!-- User Information -->
00:43:47:  <passkey v='********************************'/>
00:43:47:  <team v='223518'/>
00:43:47:  <user v='ColonelRyzen'/>
00:43:47:
00:43:47:  <!-- Folding Slots -->
00:43:47:  <slot id='0' type='CPU'>
00:43:47:    <cpus v='8'/>
00:43:47:  </slot>
00:43:47:  <slot id='1' type='GPU'>
00:43:47:    <gpu-index v='0'/>
00:43:47:  </slot>
00:43:47:  <slot id='2' type='GPU'>
00:43:47:    <gpu-index v='1'/>
00:43:47:  </slot>
00:43:47:</config>
00:45:34:WU03:FS02:0x21:Completed 50000 out of 5000000 steps (1%)
00:47:55:WU00:FS00:0xa7:Completed 1250 out of 125000 steps (1%)
00:48:12:WU03:FS02:0x21:Completed 100000 out of 5000000 steps (2%)
00:50:50:WU03:FS02:0x21:Completed 150000 out of 5000000 steps (3%)
00:51:49:FS01:Paused
00:51:49:FS01:Shutting core down
00:51:49:WU01:FS01:0x21:WARNING:Console control signal 1 on PID 1360
00:51:49:WU01:FS01:0x21:Exiting, please wait. . .
00:51:52:WU01:FS01:0x21:Folding@home Core Shutdown: INTERRUPTED
00:51:53:WU01:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
00:51:55:Removing old file 'configs/config-20191015-054640.xml'
00:51:55:Saving configuration to config.xml
00:51:55:<config>
00:51:55:  <!-- Network -->
00:51:55:  <proxy v=':8080'/>
00:51:55:
00:51:55:  <!-- User Information -->
00:51:55:  <passkey v='********************************'/>
00:51:55:  <team v='223518'/>
00:51:55:  <user v='ColonelRyzen'/>
00:51:55:
00:51:55:  <!-- Folding Slots -->
00:51:55:  <slot id='0' type='CPU'>
00:51:55:    <cpus v='8'/>
00:51:55:  </slot>
00:51:55:  <slot id='1' type='GPU'>
00:51:55:    <gpu-index v='0'/>
00:51:55:    <paused v='true'/>
00:51:55:  </slot>
00:51:55:  <slot id='2' type='GPU'>
00:51:55:    <gpu-index v='1'/>
00:51:55:  </slot>
00:51:55:</config>
00:52:01:FS01:Unpaused
00:52:01:WU01:FS01:Starting
00:52:01:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
00:52:01:WU01:FS01:Started FahCore on PID 3328
00:52:01:WU01:FS01:Core PID:3628
00:52:01:WU01:FS01:FahCore 0x21 started
00:52:01:WU01:FS01:0x21:*********************** Log Started 2019-10-24T00:52:01Z ***********************
00:52:01:WU01:FS01:0x21:Project: 14274 (Run 0, Clone 19, Gen 39)
00:52:01:WU01:FS01:0x21:Unit: 0x0000003b80fccb0a5d98ab51533ed411
00:52:01:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
00:52:01:WU01:FS01:0x21:Machine: 1
00:52:01:WU01:FS01:0x21:Digital signatures verified
00:52:01:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
00:52:01:WU01:FS01:0x21:Version 0.0.20
00:52:01:WU01:FS01:0x21:  Found a checkpoint file
00:52:19:WU01:FS01:0x21:Completed 300000 out of 1000000 steps (30%)
00:52:19:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
00:52:56:Removing old file 'configs/config-20191015-054700.xml'
00:52:56:Saving configuration to config.xml
00:52:56:<config>
00:52:56:  <!-- Network -->
00:52:56:  <proxy v=':8080'/>
00:52:56:
00:52:56:  <!-- User Information -->
00:52:56:  <passkey v='********************************'/>
00:52:56:  <team v='223518'/>
00:52:56:  <user v='ColonelRyzen'/>
00:52:56:
00:52:56:  <!-- Folding Slots -->
00:52:56:  <slot id='0' type='CPU'>
00:52:56:    <cpus v='8'/>
00:52:56:  </slot>
00:52:56:  <slot id='1' type='GPU'>
00:52:56:    <gpu-index v='0'/>
00:52:56:  </slot>
00:52:56:  <slot id='2' type='GPU'>
00:52:56:    <gpu-index v='1'/>
00:52:56:  </slot>
00:52:56:</config>
00:53:30:WU03:FS02:0x21:Completed 200000 out of 5000000 steps (4%)
00:55:16:WU00:FS00:0xa7:Completed 2500 out of 125000 steps (2%)
00:56:09:WU03:FS02:0x21:Completed 250000 out of 5000000 steps (5%)
00:58:50:WU03:FS02:0x21:Completed 300000 out of 5000000 steps (6%)
01:01:27:WU03:FS02:0x21:Completed 350000 out of 5000000 steps (7%)
01:02:29:WU00:FS00:0xa7:Completed 3750 out of 125000 steps (3%)
01:02:38:WU01:FS01:0x21:Completed 310000 out of 1000000 steps (31%)
01:04:06:WU03:FS02:0x21:Completed 400000 out of 5000000 steps (8%)
01:06:45:WU03:FS02:0x21:Completed 450000 out of 5000000 steps (9%)
01:09:27:WU03:FS02:0x21:Completed 500000 out of 5000000 steps (10%)
01:09:52:WU00:FS00:0xa7:Completed 5000 out of 125000 steps (4%)
01:12:10:WU03:FS02:0x21:Completed 550000 out of 5000000 steps (11%)
01:12:56:WU01:FS01:0x21:Completed 320000 out of 1000000 steps (32%)
01:14:47:WU03:FS02:0x21:Completed 600000 out of 5000000 steps (12%)
01:17:10:WU00:FS00:0xa7:Completed 6250 out of 125000 steps (5%)
01:17:23:WU03:FS02:0x21:Completed 650000 out of 5000000 steps (13%)
01:20:03:WU03:FS02:0x21:Completed 700000 out of 5000000 steps (14%)
01:21:18:FS00:Paused
01:21:19:FS00:Shutting core down
01:21:19:WU00:FS00:0xa7:WARNING:Console control signal 1 on PID 6944
01:21:19:WU00:FS00:0xa7:Exiting, please wait. . .
01:21:24:Removing old file 'configs/config-20191015-054729.xml'
01:21:24:Saving configuration to config.xml
01:21:24:<config>
01:21:24:  <!-- Network -->
01:21:24:  <proxy v=':8080'/>
01:21:24:
01:21:24:  <!-- User Information -->
01:21:24:  <passkey v='********************************'/>
01:21:24:  <team v='223518'/>
01:21:24:  <user v='ColonelRyzen'/>
01:21:24:
01:21:24:  <!-- Folding Slots -->
01:21:24:  <slot id='0' type='CPU'>
01:21:24:    <cpus v='8'/>
01:21:24:    <paused v='true'/>
01:21:24:  </slot>
01:21:24:  <slot id='1' type='GPU'>
01:21:24:    <gpu-index v='0'/>
01:21:24:  </slot>
01:21:24:  <slot id='2' type='GPU'>
01:21:24:    <gpu-index v='1'/>
01:21:24:  </slot>
01:21:24:</config>
01:21:24:WU00:FS00:0xa7:Folding@home Core Shutdown: INTERRUPTED
01:21:24:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
01:22:39:WU03:FS02:0x21:Completed 750000 out of 5000000 steps (15%)
01:23:20:WU01:FS01:0x21:Completed 330000 out of 1000000 steps (33%)
01:25:14:WU03:FS02:0x21:Completed 800000 out of 5000000 steps (16%)
01:27:46:WU03:FS02:0x21:Completed 850000 out of 5000000 steps (17%)
01:30:18:WU03:FS02:0x21:Completed 900000 out of 5000000 steps (18%)
01:32:49:WU03:FS02:0x21:Completed 950000 out of 5000000 steps (19%)
01:33:38:WU01:FS01:0x21:Completed 340000 out of 1000000 steps (34%)
01:35:21:WU03:FS02:0x21:Completed 1000000 out of 5000000 steps (20%)
01:37:54:WU03:FS02:0x21:Completed 1050000 out of 5000000 steps (21%)
01:40:27:WU03:FS02:0x21:Completed 1100000 out of 5000000 steps (22%)
01:43:06:WU03:FS02:0x21:Completed 1150000 out of 5000000 steps (23%)
01:44:02:WU01:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
01:44:13:FS00:Unpaused
01:44:13:WU02:FS00:Starting
01:44:13:WARNING:WU02:FS00:Changed SMP threads from 6 to 8 this can cause some work units to fail
01:44:13:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/AVX/Core_a7.fah/FahCore_a7.exe -dir 02 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -np 8
01:44:13:WU02:FS00:Started FahCore on PID 7696
01:44:13:WU02:FS00:Core PID:10000
01:44:13:WU02:FS00:FahCore 0xa7 started
01:44:14:WU02:FS00:0xa7:*********************** Log Started 2019-10-24T01:44:13Z ***********************
01:44:14:WU02:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
01:44:14:WU02:FS00:0xa7:       Type: 0xa7
01:44:14:WU02:FS00:0xa7:       Core: Gromacs
01:44:14:WU02:FS00:0xa7:    Website: https://foldingathome.org/
01:44:14:WU02:FS00:0xa7:  Copyright: (c) 2009-2018 foldingathome.org
01:44:14:WU02:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:44:14:WU02:FS00:0xa7:       Args: -dir 02 -suffix 01 -version 705 -lifeline 7696 -checkpoint 15 -np 8
01:44:14:WU02:FS00:0xa7:     Config: <none>
01:44:14:WU02:FS00:0xa7:************************************ Build *************************************
01:44:14:WU02:FS00:0xa7:    Version: 0.0.17
01:44:14:WU02:FS00:0xa7:       Date: Apr 27 2018
01:44:14:WU02:FS00:0xa7:       Time: 16:19:36
01:44:14:WU02:FS00:0xa7: Repository: Git
01:44:14:WU02:FS00:0xa7:   Revision: 21359963583d09ec2063ef946399441c4df4ccd7
01:44:14:WU02:FS00:0xa7:     Branch: master
01:44:14:WU02:FS00:0xa7:   Compiler: Visual C++ 2008
01:44:14:WU02:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:44:14:WU02:FS00:0xa7:   Platform: win32 10
01:44:14:WU02:FS00:0xa7:       Bits: 64
01:44:14:WU02:FS00:0xa7:       Mode: Release
01:44:14:WU02:FS00:0xa7:       SIMD: avx_256
01:44:14:WU02:FS00:0xa7:************************************ System ************************************
01:44:14:WU02:FS00:0xa7:        CPU: Unknown
01:44:14:WU02:FS00:0xa7:     CPU ID: 
01:44:14:WU02:FS00:0xa7:       CPUs: 12
01:44:14:WU02:FS00:0xa7:     Memory: 31.93GiB
01:44:14:WU02:FS00:0xa7:Free Memory: 25.60GiB
01:44:14:WU02:FS00:0xa7:    Threads: WINDOWS_THREADS
01:44:14:WU02:FS00:0xa7: OS Version: 6.2
01:44:14:WU02:FS00:0xa7:Has Battery: false
01:44:14:WU02:FS00:0xa7: On Battery: false
01:44:14:WU02:FS00:0xa7: UTC Offset: -4
01:44:14:WU02:FS00:0xa7:        PID: 10000
01:44:14:WU02:FS00:0xa7:        CWD: C:\Users\zacks\AppData\Roaming\FAHClient\work
01:44:14:WU02:FS00:0xa7:         OS: Windows 10 Pro
01:44:14:WU02:FS00:0xa7:    OS Arch: AMD64
01:44:14:WU02:FS00:0xa7:********************************************************************************
01:44:14:WU02:FS00:0xa7:Project: 14183 (Run 3, Clone 70, Gen 9)
01:44:14:WU02:FS00:0xa7:Unit: 0x0000000b0002894b5cf9362636419fd1
01:44:14:WU02:FS00:0xa7:Digital signatures verified
01:44:14:WU02:FS00:0xa7:Calling: mdrun -s frame9.tpr -o frame9.trr -cpi state.cpt -cpt 15 -nt 8
01:44:14:WU02:FS00:0xa7:Steps: first=22500000 total=2500000
01:44:14:WU02:FS00:0xa7:Completed 338592 out of 2500000 steps (13%)
01:44:47:Removing old file 'configs/config-20191015-054801.xml'
01:44:47:Saving configuration to config.xml
01:44:47:<config>
01:44:47:  <!-- Network -->
01:44:47:  <proxy v=':8080'/>
01:44:47:
01:44:47:  <!-- User Information -->
01:44:47:  <passkey v='********************************'/>
01:44:47:  <team v='223518'/>
01:44:47:  <user v='ColonelRyzen'/>
01:44:47:
01:44:47:  <!-- Folding Slots -->
01:44:47:  <slot id='0' type='CPU'>
01:44:47:    <cpus v='8'/>
01:44:47:  </slot>
01:44:47:  <slot id='1' type='GPU'>
01:44:47:    <gpu-index v='0'/>
01:44:47:  </slot>
01:44:47:  <slot id='2' type='GPU'>
01:44:47:    <gpu-index v='1'/>
01:44:47:  </slot>
01:44:47:</config>
01:45:49:WU03:FS02:0x21:Completed 1200000 out of 5000000 steps (24%)
01:46:27:WU02:FS00:0xa7:Completed 350000 out of 2500000 steps (14%)
01:48:34:WU03:FS02:0x21:Completed 1250000 out of 5000000 steps (25%)
01:51:22:WU02:FS00:0xa7:Completed 375000 out of 2500000 steps (15%)
01:51:24:WU03:FS02:0x21:Completed 1300000 out of 5000000 steps (26%)
01:54:13:WU03:FS02:0x21:Completed 1350000 out of 5000000 steps (27%)
01:54:21:WU01:FS01:0x21:Completed 360000 out of 1000000 steps (36%)
01:56:50:WU02:FS00:0xa7:Completed 400000 out of 2500000 steps (16%)
01:57:00:WU03:FS02:0x21:Completed 1400000 out of 5000000 steps (28%)
01:57:46:Removing old file 'configs/config-20191023-234949.xml'
01:57:46:Saving configuration to config.xml
01:57:46:<config>
01:57:46:  <!-- Network -->
01:57:46:  <proxy v=':8080'/>
01:57:46:
01:57:46:  <!-- User Information -->
01:57:46:  <passkey v='********************************'/>
01:57:46:  <team v='223518'/>
01:57:46:  <user v='ColonelRyzen'/>
01:57:46:
01:57:46:  <!-- Folding Slots -->
01:57:46:  <slot id='0' type='CPU'/>
01:57:46:  <slot id='1' type='GPU'>
01:57:46:    <gpu-index v='0'/>
01:57:46:  </slot>
01:57:46:  <slot id='2' type='GPU'>
01:57:46:    <gpu-index v='1'/>
01:57:46:  </slot>
01:57:46:</config>
01:57:46:FS00:Shutting core down
01:57:46:WU02:FS00:0xa7:WARNING:Console control signal 1 on PID 10000
01:57:46:WU02:FS00:0xa7:Exiting, please wait. . .
01:57:47:WU02:FS00:0xa7:Folding@home Core Shutdown: INTERRUPTED
01:57:47:WU00:FS00:Starting
01:57:47:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/AVX/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15
01:57:47:WU00:FS00:Started FahCore on PID 12488
01:57:47:WU00:FS00:Core PID:4480
01:57:47:WU00:FS00:FahCore 0xa7 started
01:57:47:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
01:57:48:WU00:FS00:0xa7:*********************** Log Started 2019-10-24T01:57:47Z ***********************
01:57:48:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
01:57:48:WU00:FS00:0xa7:       Type: 0xa7
01:57:48:WU00:FS00:0xa7:       Core: Gromacs
01:57:48:WU00:FS00:0xa7:    Website: https://foldingathome.org/
01:57:48:WU00:FS00:0xa7:  Copyright: (c) 2009-2018 foldingathome.org
01:57:48:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:57:48:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 12488 -checkpoint 15
01:57:48:WU00:FS00:0xa7:     Config: <none>
01:57:48:WU00:FS00:0xa7:************************************ Build *************************************
01:57:48:WU00:FS00:0xa7:    Version: 0.0.17
01:57:48:WU00:FS00:0xa7:       Date: Apr 27 2018
01:57:48:WU00:FS00:0xa7:       Time: 16:19:36
01:57:48:WU00:FS00:0xa7: Repository: Git
01:57:48:WU00:FS00:0xa7:   Revision: 21359963583d09ec2063ef946399441c4df4ccd7
01:57:48:WU00:FS00:0xa7:     Branch: master
01:57:48:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
01:57:48:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
01:57:48:WU00:FS00:0xa7:   Platform: win32 10
01:57:48:WU00:FS00:0xa7:       Bits: 64
01:57:48:WU00:FS00:0xa7:       Mode: Release
01:57:48:WU00:FS00:0xa7:       SIMD: avx_256
01:57:48:WU00:FS00:0xa7:************************************ System ************************************
01:57:48:WU00:FS00:0xa7:        CPU: Unknown
01:57:48:WU00:FS00:0xa7:     CPU ID: 
01:57:48:WU00:FS00:0xa7:       CPUs: 12
01:57:48:WU00:FS00:0xa7:     Memory: 31.93GiB
01:57:48:WU00:FS00:0xa7:Free Memory: 25.21GiB
01:57:48:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
01:57:48:WU00:FS00:0xa7: OS Version: 6.2
01:57:48:WU00:FS00:0xa7:Has Battery: false
01:57:48:WU00:FS00:0xa7: On Battery: false
01:57:48:WU00:FS00:0xa7: UTC Offset: -4
01:57:48:WU00:FS00:0xa7:        PID: 4480
01:57:48:WU00:FS00:0xa7:        CWD: C:\Users\zacks\AppData\Roaming\FAHClient\work
01:57:48:WU00:FS00:0xa7:         OS: Windows 10 Pro
01:57:48:WU00:FS00:0xa7:    OS Arch: AMD64
01:57:48:WU00:FS00:0xa7:********************************************************************************
01:57:48:WU00:FS00:0xa7:Project: 13829 (Run 969, Clone 2, Gen 19)
01:57:48:WU00:FS00:0xa7:Unit: 0x0000001680fccb095d66d29e396db3e9
01:57:48:WU00:FS00:0xa7:Digital signatures verified
01:57:48:WU00:FS00:0xa7:Calling: mdrun -s frame19.tpr -o frame19.trr -x frame19.xtc -cpi state.cpt -cpt 15 -nt 1
01:57:48:WU00:FS00:0xa7:Steps: first=2375000 total=125000
01:57:50:WU00:FS00:0xa7:Completed 6922 out of 125000 steps (5%)
01:58:00:Removing old file 'configs/config-20191023-235039.xml'
01:58:00:Saving configuration to config.xml
01:58:00:<config>
01:58:00:  <!-- Network -->
01:58:00:  <proxy v=':8080'/>
01:58:00:
01:58:00:  <!-- User Information -->
01:58:00:  <passkey v='********************************'/>
01:58:00:  <team v='223518'/>
01:58:00:  <user v='ColonelRyzen'/>
01:58:00:
01:58:00:  <!-- Folding Slots -->
01:58:00:  <slot id='0' type='CPU'/>
01:58:00:  <slot id='1' type='GPU'>
01:58:00:    <gpu-index v='0'/>
01:58:00:  </slot>
01:58:00:  <slot id='2' type='GPU'>
01:58:00:    <gpu-index v='1'/>
01:58:00:  </slot>
01:58:00:</config>
01:59:44:WU03:FS02:0x21:Completed 1450000 out of 5000000 steps (29%)
02:01:31:WU00:FS00:0xa7:Completed 7500 out of 125000 steps (6%)
02:02:25:WU03:FS02:0x21:Completed 1500000 out of 5000000 steps (30%)
02:04:49:WU01:FS01:0x21:Completed 370000 out of 1000000 steps (37%)
02:05:10:WU03:FS02:0x21:Completed 1550000 out of 5000000 steps (31%)
02:07:54:WU03:FS02:0x21:Completed 1600000 out of 5000000 steps (32%)
02:09:53:WU00:FS00:0xa7:Completed 8750 out of 125000 steps (7%)
02:10:38:WU03:FS02:0x21:Completed 1650000 out of 5000000 steps (33%)
02:13:23:WU03:FS02:0x21:Completed 1700000 out of 5000000 steps (34%)
02:15:08:WU01:FS01:0x21:Completed 380000 out of 1000000 steps (38%)
02:16:06:WU03:FS02:0x21:Completed 1750000 out of 5000000 steps (35%)
02:18:10:WU00:FS00:0xa7:Completed 10000 out of 125000 steps (8%)
02:18:39:WU01:FS01:0x21:ERROR:exception: Error downloading array energyBuffer: clEnqueueReadBuffer (-5)
02:18:39:WU01:FS01:0x21:Saving result file logfile_01.txt
02:18:39:WU01:FS01:0x21:Saving result file log.txt
02:18:39:WU01:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
02:18:40:WU03:FS02:0x21:ERROR:exception: Error downloading array energyBuffer: clEnqueueReadBuffer (-5)
02:18:41:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:18:41:WU03:FS02:0x21:Saving result file logfile_01.txt
02:18:41:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:14274 run:0 clone:19 gen:39 core:0x21 unit:0x0000003b80fccb0a5d98ab51533ed411
02:18:41:WU03:FS02:0x21:Saving result file log.txt
02:18:41:WU03:FS02:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
02:18:41:WU01:FS01:Uploading 21.50KiB to 128.252.203.10
02:18:41:WU01:FS01:Connecting to 128.252.203.10:8080
02:18:41:WU01:FS01:Upload complete
02:18:41:WU04:FS01:Connecting to 65.254.110.245:8080
02:18:41:WU01:FS01:Server responded WORK_ACK (400)
02:18:43:WU01:FS01:Cleaning up
02:18:43:WARNING:WU03:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:18:43:WU03:FS02:Sending unit results: id:03 state:SEND error:FAULTY project:14196 run:19 clone:78 gen:8 core:0x21 unit:0x0000000b0002894b5d9246b09714f006
02:18:43:WU04:FS01:Assigned to work server 155.247.166.219
02:18:43:WU03:FS02:Uploading 18.50KiB to 155.247.166.219
02:18:43:WU04:FS01:Requesting new work unit for slot 01: READY gpu:0:GK104 [Quadro K4000] from 155.247.166.219
02:18:43:WU03:FS02:Connecting to 155.247.166.219:8080
02:18:43:WU04:FS01:Connecting to 155.247.166.219:8080
02:18:43:WU01:FS02:Connecting to 65.254.110.245:8080
02:18:43:WU03:FS02:Upload complete
02:18:44:WU01:FS02:Assigned to work server 128.252.203.10
02:18:44:WU01:FS02:Requesting new work unit for slot 02: READY gpu:1:GP104 [GeForce GTX 1080] 8873 from 128.252.203.10
02:18:44:WU01:FS02:Connecting to 128.252.203.10:8080
02:18:44:WU03:FS02:Server responded WORK_ACK (400)
02:18:44:WU03:FS02:Cleaning up
02:18:44:WU04:FS01:Downloading 27.48MiB
02:18:44:WU01:FS02:Downloading 78.65MiB
02:18:50:WU01:FS02:Download 62.30%
02:18:50:WU04:FS01:Download 34.79%
02:18:52:WU01:FS02:Download complete
02:18:53:WU01:FS02:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14275 run:0 clone:30 gen:73 core:0x21 unit:0x0000005980fccb0a5d98ab57bb217296
02:18:53:WU01:FS02:Starting
02:18:53:WU01:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
02:18:53:WU01:FS02:Started FahCore on PID 1008
02:18:53:WU01:FS02:Core PID:4504
02:18:53:WU01:FS02:FahCore 0x21 started
02:18:53:WU01:FS02:0x21:*********************** Log Started 2019-10-24T02:18:53Z ***********************
02:18:53:WU01:FS02:0x21:Project: 14275 (Run 0, Clone 30, Gen 73)
02:18:53:WU01:FS02:0x21:Unit: 0x0000005980fccb0a5d98ab57bb217296
02:18:53:WU01:FS02:0x21:CPU: 0x00000000000000000000000000000000
02:18:53:WU01:FS02:0x21:Machine: 2
02:18:53:WU01:FS02:0x21:Reading tar file core.xml
02:18:53:WU01:FS02:0x21:Reading tar file integrator.xml
02:18:53:WU01:FS02:0x21:Reading tar file state.xml
02:18:54:WU01:FS02:0x21:Reading tar file system.xml
02:18:55:WU01:FS02:0x21:Digital signatures verified
02:18:55:WU01:FS02:0x21:Folding@home GPU Core21 Folding@home Core
02:18:55:WU01:FS02:0x21:Version 0.0.20
02:18:56:WU04:FS01:Download 89.60%
02:18:56:WU04:FS01:Download complete
02:18:56:WU04:FS01:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:14196 run:16 clone:289 gen:17 core:0x21 unit:0x000000180002894b5d92469b8cef4d49
02:18:56:WU04:FS01:Starting
02:18:56:WU04:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 04 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
02:18:56:WU04:FS01:Started FahCore on PID 1300
02:18:56:WU04:FS01:Core PID:696
02:18:56:WU04:FS01:FahCore 0x21 started
02:18:57:WU04:FS01:0x21:*********************** Log Started 2019-10-24T02:18:56Z ***********************
02:18:57:WU04:FS01:0x21:Project: 14196 (Run 16, Clone 289, Gen 17)
02:18:57:WU04:FS01:0x21:Unit: 0x000000180002894b5d92469b8cef4d49
02:18:57:WU04:FS01:0x21:CPU: 0x00000000000000000000000000000000
02:18:57:WU04:FS01:0x21:Machine: 1
02:18:57:WU04:FS01:0x21:Reading tar file core.xml
02:18:57:WU04:FS01:0x21:Reading tar file integrator.xml
02:18:57:WU04:FS01:0x21:Reading tar file state.xml
02:18:57:WU04:FS01:0x21:Reading tar file system.xml
02:18:57:WU04:FS01:0x21:Digital signatures verified
02:18:57:WU04:FS01:0x21:Folding@home GPU Core21 Folding@home Core
02:18:57:WU04:FS01:0x21:Version 0.0.20
02:19:11:WU04:FS01:0x21:Completed 0 out of 5000000 steps (0%)
02:19:11:WU04:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:19:21:WU01:FS02:0x21:Completed 0 out of 1000000 steps (0%)
02:19:21:WU01:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:24:51:WU04:FS01:0x21:ERROR:exception: Error downloading array energyBuffer: clEnqueueReadBuffer (-5)
02:24:51:WU04:FS01:0x21:Saving result file logfile_01.txt
02:24:51:WU04:FS01:0x21:Saving result file log.txt
02:24:51:WU04:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
02:24:51:WU01:FS02:0x21:ERROR:exception: Error downloading array energyBuffer: clEnqueueReadBuffer (-5)
02:24:51:WU01:FS02:0x21:Saving result file logfile_01.txt
02:24:51:WU01:FS02:0x21:Saving result file log.txt
02:24:51:WU01:FS02:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
02:24:52:WARNING:WU04:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:24:53:WU04:FS01:Sending unit results: id:04 state:SEND error:FAULTY project:14196 run:16 clone:289 gen:17 core:0x21 unit:0x000000180002894b5d92469b8cef4d49
02:24:54:WU04:FS01:Uploading 8.00KiB to 155.247.166.219
02:24:54:WU04:FS01:Connecting to 155.247.166.219:8080
02:24:54:WU04:FS01:Upload complete
02:24:54:WARNING:WU01:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
02:24:54:WU01:FS02:Sending unit results: id:01 state:SEND error:FAULTY project:14275 run:0 clone:30 gen:73 core:0x21 unit:0x0000005980fccb0a5d98ab57bb217296
02:24:54:WU01:FS02:Uploading 7.50KiB to 128.252.203.10
02:24:54:WU01:FS02:Connecting to 128.252.203.10:8080
02:24:54:WU03:FS01:Connecting to 65.254.110.245:8080
02:24:54:WU05:FS02:Connecting to 65.254.110.245:8080
02:24:54:WU01:FS02:Upload complete
02:24:55:WU04:FS01:Server responded WORK_ACK (400)
02:24:55:WU04:FS01:Cleaning up
02:24:55:WU01:FS02:Server responded WORK_ACK (400)
02:24:55:WU01:FS02:Cleaning up
02:24:55:WU03:FS01:Assigned to work server 155.247.166.219
02:24:55:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GK104 [Quadro K4000] from 155.247.166.219
02:24:55:WU03:FS01:Connecting to 155.247.166.219:8080
02:24:55:WU05:FS02:Assigned to work server 155.247.166.219
02:24:55:WU05:FS02:Requesting new work unit for slot 02: READY gpu:1:GP104 [GeForce GTX 1080] 8873 from 155.247.166.219
02:24:55:WU05:FS02:Connecting to 155.247.166.219:8080
02:24:56:ERROR:WU03:FS01:Exception: Server did not assign work unit
02:24:56:WU05:FS02:Downloading 27.44MiB
02:24:56:WU03:FS01:Connecting to 65.254.110.245:8080
02:24:56:WU03:FS01:Assigned to work server 128.252.203.10
02:24:56:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GK104 [Quadro K4000] from 128.252.203.10
02:24:56:WU03:FS01:Connecting to 128.252.203.10:8080
02:24:57:ERROR:WU03:FS01:Exception: Server did not assign work unit
02:25:01:WU05:FS02:Download complete
02:25:01:WU05:FS02:Received Unit: id:05 state:DOWNLOAD error:NO_ERROR project:14196 run:2 clone:19 gen:44 core:0x21 unit:0x000000390002894b5d924646e3af489f
02:25:01:WU05:FS02:Starting
02:25:01:WU05:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 05 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
02:25:01:WU05:FS02:Started FahCore on PID 1368
02:25:01:WU05:FS02:Core PID:9356
02:25:01:WU05:FS02:FahCore 0x21 started
02:25:02:WU05:FS02:0x21:*********************** Log Started 2019-10-24T02:25:01Z ***********************
02:25:03:WU05:FS02:0x21:Project: 14196 (Run 2, Clone 19, Gen 44)
02:25:05:WU05:FS02:0x21:Unit: 0x000000390002894b5d924646e3af489f
02:25:05:WU05:FS02:0x21:CPU: 0x00000000000000000000000000000000
02:25:05:WU05:FS02:0x21:Machine: 2
02:25:05:WU05:FS02:0x21:Reading tar file core.xml
02:25:05:WU05:FS02:0x21:Reading tar file integrator.xml
02:25:05:WU05:FS02:0x21:Reading tar file state.xml
02:25:05:WU05:FS02:0x21:Reading tar file system.xml
02:25:05:WU05:FS02:0x21:Digital signatures verified
02:25:05:WU05:FS02:0x21:Folding@home GPU Core21 Folding@home Core
02:25:05:WU05:FS02:0x21:Version 0.0.20
02:25:18:WU05:FS02:0x21:Completed 0 out of 5000000 steps (0%)
02:25:18:WU05:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:25:56:WU03:FS01:Connecting to 65.254.110.245:8080
02:25:56:WU03:FS01:Assigned to work server 155.247.166.219
02:25:56:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GK104 [Quadro K4000] from 155.247.166.219
02:25:56:WU03:FS01:Connecting to 155.247.166.219:8080
02:25:57:WU03:FS01:Downloading 27.49MiB
02:26:02:WU03:FS01:Download complete
02:26:02:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14196 run:14 clone:75 gen:20 core:0x21 unit:0x0000002b0002894b5d92468b3700611a
02:26:02:WU03:FS01:Starting
02:26:02:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 03 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
02:26:02:WU03:FS01:Started FahCore on PID 612
02:26:02:WU03:FS01:Core PID:2052
02:26:02:WU03:FS01:FahCore 0x21 started
02:26:03:WU03:FS01:0x21:*********************** Log Started 2019-10-24T02:26:02Z ***********************
02:26:03:WU03:FS01:0x21:Project: 14196 (Run 14, Clone 75, Gen 20)
02:26:03:WU03:FS01:0x21:Unit: 0x0000002b0002894b5d92468b3700611a
02:26:03:WU03:FS01:0x21:CPU: 0x00000000000000000000000000000000
02:26:03:WU03:FS01:0x21:Machine: 1
02:26:03:WU03:FS01:0x21:Reading tar file core.xml
02:26:03:WU03:FS01:0x21:Reading tar file integrator.xml
02:26:03:WU03:FS01:0x21:Reading tar file state.xml
02:26:03:WU03:FS01:0x21:Reading tar file system.xml
02:26:03:WU03:FS01:0x21:Digital signatures verified
02:26:03:WU03:FS01:0x21:Folding@home GPU Core21 Folding@home Core
02:26:03:WU03:FS01:0x21:Version 0.0.20
02:26:13:WU03:FS01:0x21:Completed 0 out of 5000000 steps (0%)
02:26:13:WU03:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
02:26:30:WU00:FS00:0xa7:Completed 11250 out of 125000 steps (9%)
02:28:02:WU05:FS02:0x21:Completed 50000 out of 5000000 steps (1%)
02:30:43:WU05:FS02:0x21:Completed 100000 out of 5000000 steps (2%)
02:33:24:WU05:FS02:0x21:Completed 150000 out of 5000000 steps (3%)
02:34:48:WU00:FS00:0xa7:Completed 12500 out of 125000 steps (10%)
02:36:07:WU05:FS02:0x21:Completed 200000 out of 5000000 steps (4%)
02:38:49:WU05:FS02:0x21:Completed 250000 out of 5000000 steps (5%)
02:41:30:WU05:FS02:0x21:Completed 300000 out of 5000000 steps (6%)
02:43:22:WU00:FS00:0xa7:Completed 13750 out of 125000 steps (11%)
02:44:12:WU05:FS02:0x21:Completed 350000 out of 5000000 steps (7%)
02:46:50:WU05:FS02:0x21:Completed 400000 out of 5000000 steps (8%)
02:48:10:WU03:FS01:0x21:Completed 50000 out of 5000000 steps (1%)
02:49:27:WU05:FS02:0x21:Completed 450000 out of 5000000 steps (9%)
02:51:50:WU00:FS00:0xa7:Completed 15000 out of 125000 steps (12%)
02:52:03:WU05:FS02:0x21:Completed 500000 out of 5000000 steps (10%)
02:54:42:WU05:FS02:0x21:Completed 550000 out of 5000000 steps (11%)
02:57:18:WU05:FS02:0x21:Completed 600000 out of 5000000 steps (12%)
02:59:53:WU05:FS02:0x21:Completed 650000 out of 5000000 steps (13%)
02:59:59:WU00:FS00:0xa7:Completed 16250 out of 125000 steps (13%)
03:00:56:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
03:02:21:WU05:FS02:0x21:Completed 700000 out of 5000000 steps (14%)
03:04:49:WU05:FS02:0x21:Completed 750000 out of 5000000 steps (15%)
03:06:57:WU00:FS00:0xa7:Completed 17500 out of 125000 steps (14%)
03:07:19:WU05:FS02:0x21:Completed 800000 out of 5000000 steps (16%)
03:09:47:WU05:FS02:0x21:Completed 850000 out of 5000000 steps (17%)
03:12:15:WU05:FS02:0x21:Completed 900000 out of 5000000 steps (18%)
03:13:54:WU00:FS00:0xa7:Completed 18750 out of 125000 steps (15%)
03:14:32:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
03:14:43:WU05:FS02:0x21:Completed 950000 out of 5000000 steps (19%)
03:17:11:WU05:FS02:0x21:Completed 1000000 out of 5000000 steps (20%)
03:19:41:WU05:FS02:0x21:Completed 1050000 out of 5000000 steps (21%)
03:20:00:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
03:20:00:WU03:FS01:0x21:ERROR:114: Max Retries Reached
03:20:00:WU03:FS01:0x21:Saving result file logfile_01.txt
03:20:00:WU03:FS01:0x21:Saving result file badstate-0.xml
03:20:00:WU03:FS01:0x21:Saving result file badstate-1.xml
03:20:00:WU03:FS01:0x21:Saving result file badstate-2.xml
03:20:00:WU03:FS01:0x21:Saving result file log.txt
03:20:00:WU03:FS01:0x21:Folding@home Core Shutdown: BAD_WORK_UNIT
03:20:00:WARNING:WU03:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
03:20:00:WU03:FS01:Sending unit results: id:03 state:SEND error:FAULTY project:14196 run:14 clone:75 gen:20 core:0x21 unit:0x0000002b0002894b5d92468b3700611a
03:20:00:WU03:FS01:Uploading 20.50MiB to 155.247.166.219
03:20:00:WU03:FS01:Connecting to 155.247.166.219:8080
03:20:01:WU01:FS01:Connecting to 65.254.110.245:8080
03:20:01:WU01:FS01:Assigned to work server 155.247.166.219
03:20:01:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GK104 [Quadro K4000] from 155.247.166.219
03:20:01:WU01:FS01:Connecting to 155.247.166.219:8080
03:20:01:WU01:FS01:Downloading 27.50MiB
03:20:06:WU03:FS01:Upload 26.82%
03:20:06:WU01:FS01:Download complete
03:20:06:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14196 run:12 clone:150 gen:7 core:0x21 unit:0x0000000d0002894b5d9246807d2889a9
03:20:06:WU01:FS01:Starting
03:20:06:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\zacks\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 705 -lifeline 5468 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
03:20:06:WU01:FS01:Started FahCore on PID 10416
03:20:06:WU01:FS01:Core PID:9048
03:20:06:WU01:FS01:FahCore 0x21 started
03:20:06:WU01:FS01:0x21:*********************** Log Started 2019-10-24T03:20:06Z ***********************
03:20:06:WU01:FS01:0x21:Project: 14196 (Run 12, Clone 150, Gen 7)
03:20:06:WU01:FS01:0x21:Unit: 0x0000000d0002894b5d9246807d2889a9
03:20:06:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
03:20:06:WU01:FS01:0x21:Machine: 1
03:20:06:WU01:FS01:0x21:Reading tar file core.xml
03:20:06:WU01:FS01:0x21:Reading tar file integrator.xml
03:20:06:WU01:FS01:0x21:Reading tar file state.xml
03:20:06:WU01:FS01:0x21:Reading tar file system.xml
03:20:07:WU01:FS01:0x21:Digital signatures verified
03:20:07:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
03:20:07:WU01:FS01:0x21:Version 0.0.20
03:20:12:WU03:FS01:Upload 65.23%
03:20:16:WU01:FS01:0x21:Completed 0 out of 5000000 steps (0%)
03:20:16:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
03:20:18:WU03:FS01:Upload complete
03:20:18:WU03:FS01:Server responded WORK_ACK (400)
03:20:18:WU03:FS01:Cleaning up
03:20:55:WU00:FS00:0xa7:Completed 20000 out of 125000 steps (16%)
03:22:13:WU05:FS02:0x21:Completed 1100000 out of 5000000 steps (22%)
03:24:45:WU05:FS02:0x21:Completed 1150000 out of 5000000 steps (23%)
03:27:16:WU05:FS02:0x21:Completed 1200000 out of 5000000 steps (24%)
03:28:27:WU00:FS00:0xa7:Completed 21250 out of 125000 steps (17%)
03:29:49:WU05:FS02:0x21:Completed 1250000 out of 5000000 steps (25%)
03:32:23:WU05:FS02:0x21:Completed 1300000 out of 5000000 steps (26%)

[JimboPalmer]

Something is doing a lot of pause/unpause. The way the Quick Return Bonus works greatly encourages 24/7 folding.

Each time you unpause, it reverts to the last checkpoint.

[bruce]

It doesn't start at the same speed. The ESTIMATED speed is meaningless until you've completed a certain percentage of each new project so that a projected speed can be calculated based on real data.

[foldy]

Folding slot 01 GPU Quadro K4000 is bad
03:20:00:WU03:FS01:0x21:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
03:20:00:WU03:FS01:0x21:ERROR:114: Max Retries Reached

Folding slot 02 GPU GeForce GTX 1080 also gets errors
02:18:40:WU03:FS02:0x21:ERROR:exception: Error downloading array energyBuffer: clEnqueueReadBuffer (-5)

Is any CPU or GPU overclocked? Is your system stable?

[bruce]

Recently, at least one user discovered a NVidia driver version that doesn't work well. Avoid beta releases. See if you can find a standard WHQL version that works with both GPUs.

[HaloJones]

Also, try isolating each slot to see how first it goes on its own.

It is also a known bug that the slots are misreported (or was, maybe it's been fixed). So the slot that's slowing down could be the 1080 not the K4000 and if that's your main display card, that could explain why it's going so slowly.

[MeeLee]

I would check GPU temps. Ideally you'll want them to run at below 55C, but below 75C is still acceptable.
If they surpass 75C, something's wrong. Probably too much power allocated to the card, and too little cooling.