Low PPD from 4x 1080tis (~400k each) - help please!
Posted: Tue Sep 05, 2017 11:21 am
Morning all,
I was previously running a 4 1080ti pc that was pulling 3.8-4mil PPD successfully for a couple of months, using a Gigabyte X99-UD5 board. I needed to free this PC up for something else so I have setup a new rig and transfered the old 4 1080tis to the new rig but frustratingly am only pulling ~1.2 -2mil PPD.
Running windows 10 pro and have tried drivers 378.92, 384.94 and 385.41 and its not made a noticeable difference. I've also tried reinstalling FAH. GPU utilisation is low (65-75%) but cards are boosting nicely to 1850-1900 Mhz (no overclock). Temps are fine (~65c) and have tried 2 PSUs (1 of which is the 1600W I used in the old 4mil PPD rig).
The cpu is a 6850k (40 pcie lanes) and mobo is Gigabyte X99-SOC. GPU-Z confirms 3 of the GPUs are running 3.0 @ 8x and the other is running 3.0 @ 16x. I'm not cpu folding or doing anything else on the system so CPU usage is low (under 50% on each core/thread)
Looking at HFM download I have processed on 34 WUs for projects 9*** with an avergae of 407k PPD (highest 551k, lowest 208k). I have processed 12 WUs on projects 11*** with an average of 827k (highest 922k, lowest 549k).
It looks like a GPU driver issue to me but I've done a clean install twice for 3 different drivers and I'm still not getting anywhere. One of those times I used DDU (but not in safe mode, as I'm on teamviwer at work) but surely after 6 driver reinstalls you would have thought that would fix it if it was a driver issue.
Could it be something wrong with the mobo? Or the drivers of something other than the video card? or the windows 10 install?
Any help would be much appreciated!
I was previously running a 4 1080ti pc that was pulling 3.8-4mil PPD successfully for a couple of months, using a Gigabyte X99-UD5 board. I needed to free this PC up for something else so I have setup a new rig and transfered the old 4 1080tis to the new rig but frustratingly am only pulling ~1.2 -2mil PPD.
Running windows 10 pro and have tried drivers 378.92, 384.94 and 385.41 and its not made a noticeable difference. I've also tried reinstalling FAH. GPU utilisation is low (65-75%) but cards are boosting nicely to 1850-1900 Mhz (no overclock). Temps are fine (~65c) and have tried 2 PSUs (1 of which is the 1600W I used in the old 4mil PPD rig).
The cpu is a 6850k (40 pcie lanes) and mobo is Gigabyte X99-SOC. GPU-Z confirms 3 of the GPUs are running 3.0 @ 8x and the other is running 3.0 @ 16x. I'm not cpu folding or doing anything else on the system so CPU usage is low (under 50% on each core/thread)
Looking at HFM download I have processed on 34 WUs for projects 9*** with an avergae of 407k PPD (highest 551k, lowest 208k). I have processed 12 WUs on projects 11*** with an average of 827k (highest 922k, lowest 549k).
It looks like a GPU driver issue to me but I've done a clean install twice for 3 different drivers and I'm still not getting anywhere. One of those times I used DDU (but not in safe mode, as I'm on teamviwer at work) but surely after 6 driver reinstalls you would have thought that would fix it if it was a driver issue.
Could it be something wrong with the mobo? Or the drivers of something other than the video card? or the windows 10 install?
Any help would be much appreciated!
Code: Select all
*********************** Log Started 2017-09-05T09:25:40Z ***********************
09:25:40:************************* Folding@home Client *************************
09:25:40: Website: http://folding.stanford.edu/
09:25:40: Copyright: (c) 2009-2014 Stanford University
09:25:40: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
09:25:40: Args:
09:25:40: Config: C:/Users/ Gary/AppData/Roaming/FAHClient/config.xml
09:25:40:******************************** Build ********************************
09:25:40: Version: 7.4.4
09:25:40: Date: Mar 4 2014
09:25:40: Time: 20:26:54
09:25:40: SVN Rev: 4130
09:25:40: Branch: fah/trunk/client
09:25:40: Compiler: Intel(R) C++ MSVC 1500 mode 1200
09:25:40: Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
09:25:40: /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
09:25:40: Platform: win32 XP
09:25:40: Bits: 32
09:25:40: Mode: Release
09:25:40:******************************* System ********************************
09:25:40: CPU: Intel(R) Core(TM) i7-6850K CPU @ 3.60GHz
09:25:40: CPU ID: GenuineIntel Family 6 Model 79 Stepping 1
09:25:40: CPUs: 12
09:25:40: Memory: 15.83GiB
09:25:40: Free Memory: 12.71GiB
09:25:40: Threads: WINDOWS_THREADS
09:25:40: OS Version: 6.2
09:25:40: Has Battery: false
09:25:40: On Battery: false
09:25:40: UTC Offset: 1
09:25:40: PID: 11596
09:25:40: CWD: C:/Users/ Gary/AppData/Roaming/FAHClient
09:25:40: OS: Windows 10 Pro
09:25:40: OS Arch: AMD64
09:25:40: GPUs: 4
09:25:40: GPU 0: NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
09:25:40: GPU 1: NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
09:25:40: GPU 2: NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
09:25:40: GPU 3: NVIDIA:7 GP102 [GeForce GTX 1080 Ti] 11380
09:25:40: CUDA: 6.1
09:25:40: CUDA Driver: 9000
09:25:40:Win32 Service: false
09:25:40:***********************************************************************
09:25:41:<config>
09:25:41: <!-- Network -->
09:25:41: <proxy v=':8080'/>
09:25:41:
09:25:41: <!-- Slot Control -->
09:25:41: <power v='full'/>
09:25:41:
09:25:41: <!-- User Information -->
09:25:41: <passkey v='********************************'/>
09:25:41: <team v='224497'/>
09:25:41: <user v='Hughesy26_ALL_13HwtENuzNjjg2GoNiKwXSYt8Q1GqURCHT'/>
09:25:41:
09:25:41: <!-- Folding Slots -->
09:25:41: <slot id='1' type='GPU'>
09:25:41: <paused v='true'/>
09:25:41: </slot>
09:25:41: <slot id='2' type='GPU'>
09:25:41: <paused v='true'/>
09:25:41: </slot>
09:25:41: <slot id='3' type='GPU'>
09:25:41: <paused v='true'/>
09:25:41: </slot>
09:25:41: <slot id='4' type='GPU'>
09:25:41: <paused v='true'/>
09:25:41: </slot>
09:25:41:</config>
09:25:41:Trying to access database...
09:25:41:Successfully acquired database lock
09:25:41:Enabled folding slot 01: PAUSED gpu:0:GP102 [GeForce GTX 1080 Ti] 11380 (by user)
09:25:41:Enabled folding slot 02: PAUSED gpu:1:GP102 [GeForce GTX 1080 Ti] 11380 (by user)
09:25:41:Enabled folding slot 03: PAUSED gpu:2:GP102 [GeForce GTX 1080 Ti] 11380 (by user)
09:25:41:Enabled folding slot 04: PAUSED gpu:3:GP102 [GeForce GTX 1080 Ti] 11380 (by user)
09:26:14:FS01:Unpaused
09:26:14:FS02:Unpaused
09:26:14:FS03:Unpaused
09:26:14:FS04:Unpaused
09:26:14:WU00:FS01:Starting
09:26:14:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:/Users/ Gary/AppData/Roaming/FAHClient/cores/fahwebx.stanford.edu/cores/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe" -dir 00 -suffix 01 -version 704 -lifeline 11596 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
09:26:14:WU00:FS01:Started FahCore on PID 11836
09:26:14:WU00:FS01:Core PID:1596
09:26:14:WU00:FS01:FahCore 0x21 started
09:26:14:WU02:FS02:Starting
09:26:14:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:/Users/ Gary/AppData/Roaming/FAHClient/cores/fahwebx.stanford.edu/cores/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe" -dir 02 -suffix 01 -version 704 -lifeline 11596 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
09:26:14:WU02:FS02:Started FahCore on PID 4832
09:26:14:WU02:FS02:Core PID:5408
09:26:14:WU02:FS02:FahCore 0x21 started
09:26:14:WU05:FS03:Starting
09:26:14:WU05:FS03:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:/Users/ Gary/AppData/Roaming/FAHClient/cores/fahwebx.stanford.edu/cores/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe" -dir 05 -suffix 01 -version 704 -lifeline 11596 -checkpoint 15 -gpu 2 -gpu-vendor nvidia
09:26:14:WU05:FS03:Started FahCore on PID 6352
09:26:14:WU05:FS03:Core PID:1604
09:26:14:WU05:FS03:FahCore 0x21 started
09:26:14:WU07:FS04:Starting
09:26:14:WU07:FS04:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:/Users/ Gary/AppData/Roaming/FAHClient/cores/fahwebx.stanford.edu/cores/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe" -dir 07 -suffix 01 -version 704 -lifeline 11596 -checkpoint 15 -gpu 3 -gpu-vendor nvidia
09:26:14:WU07:FS04:Started FahCore on PID 7124
09:26:14:WU07:FS04:Core PID:6424
09:26:14:WU07:FS04:FahCore 0x21 started
09:26:15:WU00:FS01:0x21:*********************** Log Started 2017-09-05T09:26:14Z ***********************
09:26:15:WU00:FS01:0x21:Project: 9415 (Run 731, Clone 0, Gen 25)
09:26:15:WU00:FS01:0x21:Unit: 0x00000025ab436c9d585e06cf5226b00a
09:26:15:WU00:FS01:0x21:CPU: 0x00000000000000000000000000000000
09:26:15:WU00:FS01:0x21:Machine: 1
09:26:15:WU00:FS01:0x21:Digital signatures verified
09:26:15:WU00:FS01:0x21:Folding@home GPU Core21 Folding@home Core
09:26:15:WU00:FS01:0x21:Version 0.0.18
09:26:15:WU02:FS02:0x21:*********************** Log Started 2017-09-05T09:26:14Z ***********************
09:26:15:WU02:FS02:0x21:Project: 9431 (Run 1745, Clone 0, Gen 5)
09:26:15:WU02:FS02:0x21:Unit: 0x0000000aab436c9d586fdd4115a04fb6
09:26:15:WU05:FS03:0x21:*********************** Log Started 2017-09-05T09:26:14Z ***********************
09:26:15:WU02:FS02:0x21:CPU: 0x00000000000000000000000000000000
09:26:15:WU05:FS03:0x21:Project: 9415 (Run 1006, Clone 0, Gen 105)
09:26:15:WU02:FS02:0x21:Machine: 2
09:26:15:WU05:FS03:0x21:Unit: 0x0000007eab436c9d585e06d10e870c70
09:26:15:WU07:FS04:0x21:*********************** Log Started 2017-09-05T09:26:14Z ***********************
09:26:15:WU02:FS02:0x21:Digital signatures verified
09:26:15:WU05:FS03:0x21:CPU: 0x00000000000000000000000000000000
09:26:15:WU07:FS04:0x21:Project: 9414 (Run 2191, Clone 0, Gen 15)
09:26:15:WU02:FS02:0x21:Folding@home GPU Core21 Folding@home Core
09:26:15:WU05:FS03:0x21:Machine: 3
09:26:15:WU00:FS01:0x21: Found a checkpoint file
09:26:15:WU07:FS04:0x21:Unit: 0x00000012ab436c9d585e06a151554869
09:26:15:WU02:FS02:0x21:Version 0.0.18
09:26:15:WU05:FS03:0x21:Digital signatures verified
09:26:15:WU07:FS04:0x21:CPU: 0x00000000000000000000000000000000
09:26:15:WU05:FS03:0x21:Folding@home GPU Core21 Folding@home Core
09:26:15:WU07:FS04:0x21:Machine: 4
09:26:15:WU05:FS03:0x21:Version 0.0.18
09:26:15:WU07:FS04:0x21:Digital signatures verified
09:26:15:WU07:FS04:0x21:Folding@home GPU Core21 Folding@home Core
09:26:15:WU07:FS04:0x21:Version 0.0.18
09:26:15:WU02:FS02:0x21: Found a checkpoint file
09:26:15:WU05:FS03:0x21: Found a checkpoint file
09:26:15:WU07:FS04:0x21: Found a checkpoint file
09:26:19:WU05:FS03:0x21:Completed 1950000 out of 6250000 steps (31%)
09:26:19:WU05:FS03:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
09:26:20:WU07:FS04:0x21:Completed 1950000 out of 6250000 steps (31%)
09:26:20:WU07:FS04:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
09:26:20:WU00:FS01:0x21:Completed 1900000 out of 6250000 steps (30%)
09:26:20:WU00:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
09:26:20:WU02:FS02:0x21:Completed 2100000 out of 6250000 steps (33%)
09:26:20:WU02:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
09:26:42:Saving configuration to config.xml
09:26:42:<config>
09:26:42: <!-- Network -->
09:26:42: <proxy v=':8080'/>
09:26:42:
09:26:42: <!-- Slot Control -->
09:26:42: <power v='full'/>
09:26:42:
09:26:42: <!-- User Information -->
09:26:42: <passkey v='********************************'/>
09:26:42: <team v='224497'/>
09:26:42: <user v='Hughesy26_ALL_13HwtENuzNjjg2GoNiKwXSYt8Q1GqURCHT'/>
09:26:42:
09:26:42: <!-- Folding Slots -->
09:26:42: <slot id='1' type='GPU'/>
09:26:42: <slot id='2' type='GPU'/>
09:26:42: <slot id='3' type='GPU'/>
09:26:42: <slot id='4' type='GPU'/>
09:26:42:</config>
09:26:51:WU02:FS02:0x21:Completed 2125000 out of 6250000 steps (34%)
09:27:10:WU00:FS01:0x21:Completed 1937500 out of 6250000 steps (31%)
09:27:26:WU05:FS03:0x21:Completed 2000000 out of 6250000 steps (32%)
09:27:27:WU07:FS04:0x21:Completed 2000000 out of 6250000 steps (32%)
09:28:09:WU02:FS02:0x21:Completed 2187500 out of 6250000 steps (35%)
09:28:34:WU00:FS01:0x21:Completed 2000000 out of 6250000 steps (32%)
09:28:50:WU05:FS03:0x21:Completed 2062500 out of 6250000 steps (33%)
09:28:52:WU07:FS04:0x21:Completed 2062500 out of 6250000 steps (33%)
09:29:27:WU02:FS02:0x21:Completed 2250000 out of 6250000 steps (36%)
09:29:58:WU00:FS01:0x21:Completed 2062500 out of 6250000 steps (33%)
09:30:13:WU05:FS03:0x21:Completed 2125000 out of 6250000 steps (34%)
09:30:18:WU07:FS04:0x21:Completed 2125000 out of 6250000 steps (34%)
09:30:46:WU02:FS02:0x21:Completed 2312500 out of 6250000 steps (37%)
09:31:23:WU00:FS01:0x21:Completed 2125000 out of 6250000 steps (34%)
09:31:37:WU05:FS03:0x21:Completed 2187500 out of 6250000 steps (35%)
09:31:43:WU07:FS04:0x21:Completed 2187500 out of 6250000 steps (35%)
09:32:05:WU02:FS02:0x21:Completed 2375000 out of 6250000 steps (38%)
09:32:47:WU00:FS01:0x21:Completed 2187500 out of 6250000 steps (35%)
09:33:01:WU05:FS03:0x21:Completed 2250000 out of 6250000 steps (36%)
09:33:09:WU07:FS04:0x21:Completed 2250000 out of 6250000 steps (36%)
09:33:23:WU02:FS02:0x21:Completed 2437500 out of 6250000 steps (39%)
09:34:12:WU00:FS01:0x21:Completed 2250000 out of 6250000 steps (36%)
09:34:25:WU05:FS03:0x21:Completed 2312500 out of 6250000 steps (37%)
09:34:35:WU07:FS04:0x21:Completed 2312500 out of 6250000 steps (37%)
09:34:41:WU02:FS02:0x21:Completed 2500000 out of 6250000 steps (40%)
09:35:37:WU00:FS01:0x21:Completed 2312500 out of 6250000 steps (37%)
09:35:49:WU05:FS03:0x21:Completed 2375000 out of 6250000 steps (38%)
09:36:00:WU07:FS04:0x21:Completed 2375000 out of 6250000 steps (38%)
09:36:00:WU02:FS02:0x21:Completed 2562500 out of 6250000 steps (41%)
09:37:01:WU00:FS01:0x21:Completed 2375000 out of 6250000 steps (38%)
09:37:14:WU05:FS03:0x21:Completed 2437500 out of 6250000 steps (39%)
09:37:19:WU02:FS02:0x21:Completed 2625000 out of 6250000 steps (42%)
09:37:26:WU07:FS04:0x21:Completed 2437500 out of 6250000 steps (39%)
09:38:26:WU00:FS01:0x21:Completed 2437500 out of 6250000 steps (39%)
09:38:37:WU02:FS02:0x21:Completed 2687500 out of 6250000 steps (43%)
09:38:38:WU05:FS03:0x21:Completed 2500000 out of 6250000 steps (40%)
09:38:51:WU07:FS04:0x21:Completed 2500000 out of 6250000 steps (40%)
09:39:50:WU00:FS01:0x21:Completed 2500000 out of 6250000 steps (40%)
09:39:55:WU02:FS02:0x21:Completed 2750000 out of 6250000 steps (44%)
09:40:02:WU05:FS03:0x21:Completed 2562500 out of 6250000 steps (41%)
09:40:17:WU07:FS04:0x21:Completed 2562500 out of 6250000 steps (41%)
09:41:15:WU02:FS02:0x21:Completed 2812500 out of 6250000 steps (45%)
09:41:15:WU00:FS01:0x21:Completed 2562500 out of 6250000 steps (41%)
09:41:26:WU05:FS03:0x21:Completed 2625000 out of 6250000 steps (42%)
09:41:43:WU07:FS04:0x21:Completed 2625000 out of 6250000 steps (42%)
09:42:33:WU02:FS02:0x21:Completed 2875000 out of 6250000 steps (46%)
09:42:40:WU00:FS01:0x21:Completed 2625000 out of 6250000 steps (42%)
09:42:50:WU05:FS03:0x21:Completed 2687500 out of 6250000 steps (43%)
09:43:08:WU07:FS04:0x21:Completed 2687500 out of 6250000 steps (43%)
09:43:51:WU02:FS02:0x21:Completed 2937500 out of 6250000 steps (47%)
09:44:04:WU00:FS01:0x21:Completed 2687500 out of 6250000 steps (43%)
09:44:14:WU05:FS03:0x21:Completed 2750000 out of 6250000 steps (44%)
09:44:33:WU07:FS04:0x21:Completed 2750000 out of 6250000 steps (44%)
09:45:10:WU02:FS02:0x21:Completed 3000000 out of 6250000 steps (48%)
09:45:28:WU00:FS01:0x21:Completed 2750000 out of 6250000 steps (44%)
09:45:39:WU05:FS03:0x21:Completed 2812500 out of 6250000 steps (45%)
09:45:59:WU07:FS04:0x21:Completed 2812500 out of 6250000 steps (45%)
09:46:29:WU02:FS02:0x21:Completed 3062500 out of 6250000 steps (49%)
09:46:53:WU00:FS01:0x21:Completed 2812500 out of 6250000 steps (45%)
09:47:03:WU05:FS03:0x21:Completed 2875000 out of 6250000 steps (46%)
09:47:25:WU07:FS04:0x21:Completed 2875000 out of 6250000 steps (46%)
09:47:47:WU02:FS02:0x21:Completed 3125000 out of 6250000 steps (50%)
09:48:18:WU00:FS01:0x21:Completed 2875000 out of 6250000 steps (46%)
09:48:27:WU05:FS03:0x21:Completed 2937500 out of 6250000 steps (47%)
09:48:50:WU07:FS04:0x21:Completed 2937500 out of 6250000 steps (47%)
09:49:06:WU02:FS02:0x21:Completed 3187500 out of 6250000 steps (51%)
09:49:42:WU00:FS01:0x21:Completed 2937500 out of 6250000 steps (47%)
09:49:51:WU05:FS03:0x21:Completed 3000000 out of 6250000 steps (48%)
09:50:16:WU07:FS04:0x21:Completed 3000000 out of 6250000 steps (48%)
09:50:24:WU02:FS02:0x21:Completed 3250000 out of 6250000 steps (52%)
09:51:07:WU00:FS01:0x21:Completed 3000000 out of 6250000 steps (48%)
09:51:15:WU05:FS03:0x21:Completed 3062500 out of 6250000 steps (49%)
09:51:41:WU07:FS04:0x21:Completed 3062500 out of 6250000 steps (49%)
09:51:43:WU02:FS02:0x21:Completed 3312500 out of 6250000 steps (53%)
09:52:32:WU00:FS01:0x21:Completed 3062500 out of 6250000 steps (49%)
09:52:39:WU05:FS03:0x21:Completed 3125000 out of 6250000 steps (50%)
09:53:02:WU02:FS02:0x21:Completed 3375000 out of 6250000 steps (54%)
09:53:07:WU07:FS04:0x21:Completed 3125000 out of 6250000 steps (50%)
09:53:56:WU00:FS01:0x21:Completed 3125000 out of 6250000 steps (50%)
09:54:03:WU05:FS03:0x21:Completed 3187500 out of 6250000 steps (51%)
09:54:20:WU02:FS02:0x21:Completed 3437500 out of 6250000 steps (55%)
09:54:32:WU07:FS04:0x21:Completed 3187500 out of 6250000 steps (51%)
09:55:21:WU00:FS01:0x21:Completed 3187500 out of 6250000 steps (51%)
09:55:27:WU05:FS03:0x21:Completed 3250000 out of 6250000 steps (52%)
09:55:38:WU02:FS02:0x21:Completed 3500000 out of 6250000 steps (56%)
09:55:58:WU07:FS04:0x21:Completed 3250000 out of 6250000 steps (52%)
09:56:45:WU00:FS01:0x21:Completed 3250000 out of 6250000 steps (52%)
09:56:52:WU05:FS03:0x21:Completed 3312500 out of 6250000 steps (53%)
09:56:57:WU02:FS02:0x21:Completed 3562500 out of 6250000 steps (57%)
09:57:24:WU07:FS04:0x21:Completed 3312500 out of 6250000 steps (53%)
09:58:10:WU00:FS01:0x21:Completed 3312500 out of 6250000 steps (53%)
09:58:16:WU02:FS02:0x21:Completed 3625000 out of 6250000 steps (58%)
09:58:16:WU05:FS03:0x21:Completed 3375000 out of 6250000 steps (54%)
09:58:49:WU07:FS04:0x21:Completed 3375000 out of 6250000 steps (54%)
09:59:34:WU02:FS02:0x21:Completed 3687500 out of 6250000 steps (59%)
09:59:35:WU00:FS01:0x21:Completed 3375000 out of 6250000 steps (54%)
09:59:40:WU05:FS03:0x21:Completed 3437500 out of 6250000 steps (55%)
10:00:15:WU07:FS04:0x21:Completed 3437500 out of 6250000 steps (55%)
10:00:53:WU02:FS02:0x21:Completed 3750000 out of 6250000 steps (60%)
10:00:59:WU00:FS01:0x21:Completed 3437500 out of 6250000 steps (55%)
10:01:04:WU05:FS03:0x21:Completed 3500000 out of 6250000 steps (56%)
10:01:40:WU07:FS04:0x21:Completed 3500000 out of 6250000 steps (56%)
10:02:12:WU02:FS02:0x21:Completed 3812500 out of 6250000 steps (61%)
10:02:24:WU00:FS01:0x21:Completed 3500000 out of 6250000 steps (56%)
10:02:28:WU05:FS03:0x21:Completed 3562500 out of 6250000 steps (57%)
10:03:06:WU07:FS04:0x21:Completed 3562500 out of 6250000 steps (57%)
10:03:30:WU02:FS02:0x21:Completed 3875000 out of 6250000 steps (62%)
10:03:49:WU00:FS01:0x21:Completed 3562500 out of 6250000 steps (57%)
10:03:52:WU05:FS03:0x21:Completed 3625000 out of 6250000 steps (58%)
10:04:32:WU07:FS04:0x21:Completed 3625000 out of 6250000 steps (58%)
10:04:48:WU02:FS02:0x21:Completed 3937500 out of 6250000 steps (63%)
10:05:13:WU00:FS01:0x21:Completed 3625000 out of 6250000 steps (58%)
10:05:16:WU05:FS03:0x21:Completed 3687500 out of 6250000 steps (59%)
10:05:57:WU07:FS04:0x21:Completed 3687500 out of 6250000 steps (59%)
10:06:07:WU02:FS02:0x21:Completed 4000000 out of 6250000 steps (64%)
10:06:38:WU00:FS01:0x21:Completed 3687500 out of 6250000 steps (59%)
10:06:41:WU05:FS03:0x21:Completed 3750000 out of 6250000 steps (60%)
10:07:22:WU07:FS04:0x21:Completed 3750000 out of 6250000 steps (60%)
10:07:26:WU02:FS02:0x21:Completed 4062500 out of 6250000 steps (65%)
10:08:02:WU00:FS01:0x21:Completed 3750000 out of 6250000 steps (60%)
10:08:05:WU05:FS03:0x21:Completed 3812500 out of 6250000 steps (61%)
10:08:44:WU02:FS02:0x21:Completed 4125000 out of 6250000 steps (66%)
10:08:48:WU07:FS04:0x21:Completed 3812500 out of 6250000 steps (61%)
10:09:27:WU00:FS01:0x21:Completed 3812500 out of 6250000 steps (61%)
10:09:29:WU05:FS03:0x21:Completed 3875000 out of 6250000 steps (62%)
10:10:03:WU02:FS02:0x21:Completed 4187500 out of 6250000 steps (67%)
10:10:14:WU07:FS04:0x21:Completed 3875000 out of 6250000 steps (62%)
10:10:52:WU00:FS01:0x21:Completed 3875000 out of 6250000 steps (62%)
10:10:53:WU05:FS03:0x21:Completed 3937500 out of 6250000 steps (63%)
10:11:21:WU02:FS02:0x21:Completed 4250000 out of 6250000 steps (68%)
10:11:39:WU07:FS04:0x21:Completed 3937500 out of 6250000 steps (63%)
10:12:16:WU00:FS01:0x21:Completed 3937500 out of 6250000 steps (63%)
10:12:17:WU05:FS03:0x21:Completed 4000000 out of 6250000 steps (64%)
10:12:40:WU02:FS02:0x21:Completed 4312500 out of 6250000 steps (69%)
10:13:05:WU07:FS04:0x21:Completed 4000000 out of 6250000 steps (64%)
10:13:41:WU00:FS01:0x21:Completed 4000000 out of 6250000 steps (64%)
10:13:41:WU05:FS03:0x21:Completed 4062500 out of 6250000 steps (65%)