Folding Forum

Posted: **Sat Oct 12, 2019 4:29 pm**

If my memories are correct, dfgirl12 has slow upload speed and is having troubles uploading results of big WUs like p142xx ... so with 155.247.166.219 malfunctioning which serves smaller WUs, and his 2080 Ti completing p142xx WUs faster than he can upload one result, it is saturating his line.

Posted: **Sat Oct 12, 2019 8:45 pm**

When the servers are rebooted daily my connection has no issues keeping up with any of the uploads for the past 1-2 months that these larger WU sizes have been available. That is until the servers slow down after about 3 days of up-time and mostly during business hours in the US.

So, correct, I do have slower cable internet service, where I can typically upload 150KB/s during daytime business hours, and 300KB/s outside of those times, which would normally be good enough by a decent margin (like 100KB/s upload speed over what is needed for all my FAH PCs normally).

Bruce: I can open both of the WS IP addresses or sever names just fine in a browser. The current servers being rebooted every 1-3 days does alleviate this problem (without adding 1-2 more FAH collection severs, or me getting a second ISP {that might not fix the problem}). If you would like more log files, please let me know specifically what info you are looking for (Windows: all uploads fail in full saturation. Linux: Most uploads are very slow, and fail less often as I disconnect more machines in the slowdown periods), and in what state. I don't think more log files will help much though.

My connection was saturated, with 20-50+ uploads across all the connected machines, not much was getting through once the servers slow down, until I disconnect 50% of my PCs (current state, which is starting to catch up now). The previous log and screenshot are from the only Windows machine (the worst), where nothing is getting uploaded (so, I stopped folding on that machine, until the congestion clears in a few days, or after the next server reboot). I did start another forum topic about this: viewtopic.php?f=74&t=31882 where I was hoping for longer run times on larger file size completed WUs to reduce this type of network congestion, but this issue seems to still mostly be about server reboots (this topic) for the near term.

In the other topic, I was also hoping for a way to limit the number of WU uploads for these congestion periods, but the FAH client doesn't have a setting to do this (no ticket yet for it here either: https://github.com/FoldingAtHome/fah-co ... is%3Aissue). FAH client keeps trying 3-5 uploads until the uploads fail 10 times per WU slot on all the PCs in my network, which is like 50+ concurrent uploads. There is also no ticket on GitHub for a more robust uploading mechanism (like: retrying / resuming partial failed uploads. using a different upload software implementation. Or, doing a more peer-to-peer WU file distribution/collection, more like file torrents), or limiting the number of simultaneous uploads to a collection server in the FAH client and/or CS (and making the FAH Client wait to upload some other time or use a different CS, for better QOS).

Posted: **Sat Oct 12, 2019 11:00 pm**

Running FAHclient.exe --help lists various config options. One that looks interesting for this situation is max-queue, which appears to have a default of 16. Try adding max-queue with a value of 4 in Configure > Expert > Extra client options. Presumably, this would prevent the client from downloading a new work unit if the same slot has 4 work units that are unsent. You may have to restart the client or machine for the setting to take effect.

Posted: **Sat Oct 12, 2019 11:34 pm**

Thanks for the suggestion, but I've been running that setting for the past few weeks viewtopic.php?f=74&t=31882#p309004 and it turns out that max-queue=2 does nothing to limit the file uploads (just the downloaded WUs per slot??), there are still 3-5 uploads on all the PCs during the server slow-downs with upload queues of 4+ per folding slot. If you: 'pause' folding, and 'start' folding again with a deep (16+) WU backlog upload queue then the FAH Client typically stops folding until your upload queue clears, but it doesn't seem to be controlled by the 'max-queue' setting at all, as far as I can tell. I just changed all the PCs to max-queue=1, but I'm not expecting any difference.

Posted: **Sun Oct 13, 2019 10:53 am**

I have a similar issue with a Unit not uploading:

WS: 155.247.166.219
CS: 155.247.166.220

Other WUs for different servers have since finished and uploaded.

Code: Select all

08:47:26:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:14190 run:46 clone:64 gen:4 core:0xa7 unit:0x000000060002894b5d5d2af45defa614
08:47:26:WU00:FS00:Uploading 68.46MiB to 155.247.166.219
08:47:26:WU00:FS00:Connecting to 155.247.166.219:8080
08:47:27:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
08:47:27:WU00:FS00:Trying to send results to collection server
08:47:27:WU00:FS00:Uploading 68.46MiB to 155.247.166.220
08:47:27:WU00:FS00:Connecting to 155.247.166.220:8080
08:47:27:ERROR:WU00:FS00:Exception: Transfer failed

Unlike some of the other reports on this thread, this upload don't even look like it's starting as I never get a % report in the log.

Is this a manifestation of the same issue?

Configuration:

Code: Select all

*********************** Log Started 2019-10-13T06:45:58Z ***********************
06:45:58:************************* Folding@home Client *************************
06:45:58:        Website: https://foldingathome.org/
06:45:58:      Copyright: (c) 2009-2018 foldingathome.org
06:45:58:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
06:45:58:           Args: 
06:45:58:         Config: C:\Users\Doug\AppData\Roaming\FAHClient\config.xml
06:45:58:******************************** Build ********************************
06:45:58:        Version: 7.5.1
06:45:58:           Date: May 11 2018
06:45:58:           Time: 13:06:32
06:45:58:     Repository: Git
06:45:58:       Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
06:45:58:         Branch: master
06:45:58:       Compiler: Visual C++ 2008
06:45:58:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
06:45:58:       Platform: win32 10
06:45:58:           Bits: 32
06:45:58:           Mode: Release
06:45:58:******************************* System ********************************
06:45:58:            CPU: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
06:45:58:         CPU ID: GenuineIntel Family 6 Model 158 Stepping 12
06:45:58:           CPUs: 16
06:45:58:         Memory: 15.88GiB
06:45:58:    Free Memory: 13.81GiB
06:45:58:        Threads: WINDOWS_THREADS
06:45:58:     OS Version: 6.2
06:45:58:    Has Battery: false
06:45:58:     On Battery: false
06:45:58:     UTC Offset: 1
06:45:58:            PID: 8984
06:45:58:            CWD: C:\Users\Doug\AppData\Roaming\FAHClient
06:45:58:             OS: Windows 10 Home
06:45:58:        OS Arch: AMD64
06:45:58:           GPUs: 1
06:45:58:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:3 GK104 [GeForce GTX 770] 3213
06:45:58:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:3.0 Driver:9.1
06:45:58:OpenCL Device 0: Platform:0 Device:0 Bus:NA Slot:NA Compute:2.1 Driver:25.20
06:45:58:OpenCL Device 2: Platform:1 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:388.13
06:45:58:  Win32 Service: false
06:45:58:***********************************************************************
06:45:58:<config>
06:45:58:  <service-description v='Folding@home Client'/>
06:45:58:  <service-restart v='true'/>
06:45:58:  <service-restart-delay v='5000'/>
06:45:58:
06:45:58:  <!-- Client Control -->
06:45:58:  <client-threads v='6'/>
06:45:58:  <cycle-rate v='4'/>
06:45:58:  <cycles v='-1'/>
06:45:58:  <data-directory v='.'/>
06:45:58:  <disable-sleep-when-active v='true'/>
06:45:58:  <exec-directory v='C:\Program Files (x86)\FAHClient'/>
06:45:58:  <exit-when-done v='false'/>
06:45:58:  <fold-anon v='false'/>
06:45:58:  <open-web-control v='false'/>
06:45:58:
06:45:58:  <!-- Configuration -->
06:45:58:  <config-rotate v='true'/>
06:45:58:  <config-rotate-dir v='configs'/>
06:45:58:  <config-rotate-max v='16'/>
06:45:58:
06:45:58:  <!-- Debugging -->
06:45:58:  <assignment-servers>
06:45:58:    assign1.foldingathome.org:8080 assign2.foldingathome.org:80
06:45:58:  </assignment-servers>
06:45:58:  <auth-as v='true'/>
06:45:58:  <capture-directory v='capture'/>
06:45:58:  <capture-on-error v='false'/>
06:45:58:  <capture-packets v='false'/>
06:45:58:  <capture-requests v='false'/>
06:45:58:  <capture-responses v='false'/>
06:45:58:  <capture-sockets v='false'/>
06:45:58:  <core-exec v='FahCore_$type'/>
06:45:58:  <core-wrapper-exec v='FAHCoreWrapper'/>
06:45:58:  <debug-sockets v='false'/>
06:45:58:  <exception-locations v='true'/>
06:45:58:  <stack-traces v='false'/>
06:45:58:
06:45:58:  <!-- Error Handling -->
06:45:58:  <max-slot-errors v='10'/>
06:45:58:  <max-unit-errors v='5'/>
06:45:58:
06:45:58:  <!-- Folding Core -->
06:45:58:  <checkpoint v='15'/>
06:45:58:  <core-dir v='cores'/>
06:45:58:  <core-priority v='idle'/>
06:45:58:  <cpu-affinity v='false'/>
06:45:58:  <cpu-usage v='100'/>
06:45:58:  <gpu-usage v='100'/>
06:45:58:  <no-assembly v='false'/>
06:45:58:
06:45:58:  <!-- Folding Slot Configuration -->
06:45:58:  <cause v='CANCER'/>
06:45:58:  <client-subtype v='STDCLI'/>
06:45:58:  <client-type v='normal'/>
06:45:58:  <cpu-species v='X86_PENTIUM_II'/>
06:45:58:  <cpu-type v='AMD64'/>
06:45:58:  <cpus v='-1'/>
06:45:58:  <disable-viz v='false'/>
06:45:58:  <gpu v='true'/>
06:45:58:  <max-packet-size v='normal'/>
06:45:58:  <os-species v='WIN_8'/>
06:45:58:  <os-type v='WIN32'/>
06:45:58:  <project-key v='0'/>
06:45:58:  <smp v='true'/>
06:45:58:
06:45:58:  <!-- GUI -->
06:45:58:  <gui-enabled v='true'/>
06:45:58:
06:45:58:  <!-- HTTP Server -->
06:45:58:  <allow v='10.0.10.0/24'/>
06:45:58:  <connection-timeout v='60'/>
06:45:58:  <deny v='0/0'/>
06:45:58:  <http-addresses v='0:7396'/>
06:45:58:  <https-addresses v=''/>
06:45:58:  <max-connect-time v='900'/>
06:45:58:  <max-connections v='800'/>
06:45:58:  <max-request-length v='52428800'/>
06:45:58:  <min-connect-time v='300'/>
06:45:58:
06:45:58:  <!-- Logging -->
06:45:58:  <log v='log.txt'/>
06:45:58:  <log-color v='false'/>
06:45:58:  <log-crlf v='true'/>
06:45:58:  <log-date v='false'/>
06:45:58:  <log-date-periodically v='21600'/>
06:45:58:  <log-domain v='false'/>
06:45:58:  <log-header v='true'/>
06:45:58:  <log-level v='true'/>
06:45:58:  <log-no-info-header v='true'/>
06:45:58:  <log-redirect v='false'/>
06:45:58:  <log-rotate v='true'/>
06:45:58:  <log-rotate-dir v='logs'/>
06:45:58:  <log-rotate-max v='16'/>
06:45:58:  <log-short-level v='false'/>
06:45:58:  <log-simple-domains v='true'/>
06:45:58:  <log-thread-id v='false'/>
06:45:58:  <log-thread-prefix v='true'/>
06:45:58:  <log-time v='true'/>
06:45:58:  <log-to-screen v='true'/>
06:45:58:  <log-truncate v='false'/>
06:45:58:  <verbosity v='5'/>
06:45:58:
06:45:58:  <!-- Network -->
06:45:58:  <proxy v=':8080'/>
06:45:58:  <proxy-enable v='false'/>
06:45:58:  <proxy-pass v=''/>
06:45:58:  <proxy-user v=''/>
06:45:58:
06:45:58:  <!-- Process Control -->
06:45:58:  <child v='false'/>
06:45:58:  <daemon v='false'/>
06:45:58:  <pid v='false'/>
06:45:58:  <pid-file v='Folding@home Client.pid'/>
06:45:58:  <respawn v='false'/>
06:45:58:  <service v='false'/>
06:45:58:
06:45:58:  <!-- Remote Command Server -->
06:45:58:  <command-address v='0.0.0.0'/>
06:45:58:  <command-allow-no-pass v='127.0.0.1'/>
06:45:58:  <command-deny-no-pass v='0/0'/>
06:45:58:  <command-enable v='true'/>
06:45:58:  <command-port v='36330'/>
06:45:58:  <password v='*******'/>
06:45:58:
06:45:58:  <!-- Slot Control -->
06:45:58:  <idle v='false'/>
06:45:58:  <max-shutdown-wait v='60'/>
06:45:58:  <pause-on-battery v='true'/>
06:45:58:  <pause-on-start v='false'/>
06:45:58:  <paused v='false'/>
06:45:58:  <power v='full'/>
06:45:58:
06:45:58:  <!-- User Information -->
06:45:58:  <machine-id v='0'/>
06:45:58:  <passkey v='********************************'/>
06:45:58:  <team v='225070'/>
06:45:58:  <user v='Doug9900kFTW'/>
06:45:58:
06:45:58:  <!-- Web Server -->
06:45:58:  <web-allow v='127.0.0.1'/>
06:45:58:  <web-deny v='0/0'/>
06:45:58:  <web-enable v='true'/>
06:45:58:
06:45:58:  <!-- Web Server Sessions -->
06:45:58:  <session-cookie v='sid'/>
06:45:58:  <session-lifetime v='86400'/>
06:45:58:  <session-timeout v='3600'/>
06:45:58:
06:45:58:  <!-- Work Unit Control -->
06:45:58:  <dump-after-deadline v='true'/>
06:45:58:  <max-queue v='16'/>
06:45:58:  <max-units v='0'/>
06:45:58:  <next-unit-percentage v='99'/>
06:45:58:  <stall-detection-enabled v='false'/>
06:45:58:  <stall-percent v='5'/>
06:45:58:  <stall-timeout v='1800'/>
06:45:58:
06:45:58:  <!-- Folding Slots -->
06:45:58:  <slot id='0' type='CPU'/>
06:45:58:  <slot id='1' type='GPU'/>
06:45:58:</config>

Posted: **Sun Oct 13, 2019 3:23 pm**

Welcome to the folding support forum.

DougHunt wrote:Is this a manifestation of the same issue?

Maybe. Or it could be result of a network issue that sometimes shows up with the client. When sending a WU packets are sent, and then the client waits on ACK packets being returned before sending more. If enough of those ACK packets are lost through a number of reasons - dropped by an ISP under congestion, not routed by a wireless connection, and so on - the transfer fails. Most of the time the client will detect this and restart the upload after 15-30 minutes. When it doesn't, a restart of the client usually starts the upload again.

In this case a single retry is shown, so can't tell for sure.

P.S. Please return the logging verbosity to the default of 3. Higher levels do not help, and actually obscure information needed to troubleshoot problems. The higher levels were only useful in the early days of the version 7 client when it was in beta testing.

Posted: **Sun Oct 13, 2019 7:09 pm**

DougHunt wrote:Unlike some of the other reports on this thread, this upload don't even look like it's starting as I never get a % report in the log.

Is this a manifestation of the same issue?

I do think it's probably the same issue.

My current operational THEORY is that when a connection fails (upload or download starts but is interrupted ... resulting in a hang) that it never actually recovers. FAHClient can manage 2 or 3 independent connections, so you may never notice the first failure but after the second or third connection fails, the only way to clear the hung connections is to restart FAHClient. Also, there have been difficulties on the server side as well as on the client side and it's not easy to tell the difference.

At this moment, this is my theory, not a known fact

To restart FAHClient in Windows, either logout and logon again or simply reboot. I expect that will work, but please report our finding.

To restart on Linux sudo /etc/init.d/FAHClient stop (or perhaps sudo kill <PID>) followed by a sudo /etc/init.d/FAHClient start

Posted: **Sun Oct 13, 2019 8:01 pm**

Joe_H wrote:Welcome to the folding support forum.

Very Happy to be here!

Joe_H wrote: In this case a single retry is shown, so can't tell for sure.

P.S. Please return the logging verbosity to the default of 3. Higher levels do not help, and actually obscure information needed to troubleshoot problems. The higher levels were only useful in the early days of the version 7 client when it was in beta testing.

Apologies for the excessive verbosity, I've changed the setting.

The Unit has failed to upload around 40 times over the last 48 hours, I only included one log entry as they all show the same info. Perhaps the WU is simply corrupted and should be deleted?

bruce wrote: the only way to clear the hung connections is to restart FAHClient.

I've trried restarting the client and host numerous times, with and without pausing first. As I mentioned previously, WUs for different servers have since uploaded with no problems at all. I'm on a fairly idle 200/20mbps connection and have no issues with other services.

I'm really not that concerned with loosing credit for the WU in question, just want to help diagnose the issue if I can!

All the best,

Dx

Posted: **Sun Oct 13, 2019 8:22 pm**

I've just noticed that a different host on the same network successfully uploaded a completed WU to the same server:

Code: Select all

12:59:47:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
12:59:47:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:59:47:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:14182 run:9 clone:17 gen:5 core:0xa7 unit:0x000000060002894b5cf684b2cabefdf9
12:59:47:WU00:FS00:Uploading 5.43MiB to 155.247.166.219
12:59:47:WU00:FS00:Connecting to 155.247.166.219:8080
12:59:47:WU01:FS00:Starting
12:59:47:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/AVX/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 704 -lifeline 1151 -checkpoint 15 -np 3
12:59:47:WU01:FS00:Started FahCore on PID 8952
12:59:47:WU01:FS00:Core PID:8956
12:59:47:WU01:FS00:FahCore 0xa7 started
12:59:47:WU01:FS00:0xa7:*********************** Log Started 2019-10-13T12:59:47Z ***********************
12:59:47:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
... Lots of info on new WU ...
12:59:53:WU00:FS00:Upload complete

The above uploaded the 5.43MiB WU in around 6 seconds so bandwidth doesn't seem to be the issue.

Starting to think it's just an issue with the particular WU on the first host.

Posted: **Mon Oct 14, 2019 4:07 pm**

Well, my theory about restarting clearly doesn't work in this case. Apparently your theory that the upload package is somehow corrupt makes sense. Go ahead and delete the WU and we'll watch for other cases that hopefully will give us more information.

Posted: **Mon Oct 14, 2019 6:55 pm**

Update: My Linux machines uploaded the backlog of WUs from Friday by Saturday night, and the one Windows machine finished uploading the backlog by Sunday night. Now the backlog of WUs has begun again on Monday morning (Business hours). The servers were rebooted a few hours ago, but it was too late to prevent the saturation of my connection and backup of WUs. Hopefully the WUs clear out tonight now that the servers were rebooted (I don't need another full-time job)

.

Bruce for your theory: about the client getting hung up on failed uploads: it seems to be for a period of time (hours?), but I think it does release the hung resources once the WU slot does finally upload (or maybe event after a Pause/Resume). In some of my PC's log files once the WUs start piling up you start seeing 'Detected clock skew' warnings per WU slot on dedicated Linux PCs, where the FAH Client was too tied-up doing stuff (uploads & folding). Not all the PCs have the clock skew message, and It's not there once the WU upload backlog is gone. I'll probably reboot all my PCs tonight when the WUs have a better chance of uploading, but restarting the FAH client (and killing off any hung FAH PID) is typically all that is needed.

Linux logfile (search for 'Detected clock skew'):

Code: Select all

2019-10-14:17:21:08:WU06:FS01:0x21:Completed 640000 out of 1000000 steps (64%)
2019-10-14:17:21:10:WU05:FS00:Upload 2.05%
2019-10-14:17:21:11:WU03:FS01:Upload 49.93%
2019-10-14:17:21:11:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:21:11:WU03:FS01:Trying to send results to collection server
2019-10-14:17:21:11:WU03:FS01:Uploading 64.59MiB to 155.247.166.219
2019-10-14:17:21:11:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-14:17:21:12:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-14:17:21:12:WARNING:WU06:FS01:Detected clock skew (3 mins 31 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:21:12:WU07:FS00:Starting
2019-10-14:17:21:12:WU07:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 07 -suffix 01 -version 705 -lifeline 2081 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 1 -gpu 0
2019-10-14:17:21:12:WU07:FS00:Started FahCore on PID 20706
2019-10-14:17:21:12:WU07:FS00:Core PID:20710
2019-10-14:17:21:12:WU07:FS00:FahCore 0x21 started
2019-10-14:17:21:12:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:9 gen:53 core:0x21 unit:0x0000003580fccb0a5d98ab5040b3bced
2019-10-14:17:21:12:WU03:FS01:Uploading 64.59MiB to 128.252.203.10
2019-10-14:17:21:12:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:21:12:WU07:FS00:0x21:*********************** Log Started 2019-10-14T17:21:12Z ***********************
2019-10-14:17:21:12:WU07:FS00:0x21:Project: 14278 (Run 0, Clone 1, Gen 30)
2019-10-14:17:21:12:WU07:FS00:0x21:Unit: 0x0000001e80fccb0a5d98ab6585c3cef9
2019-10-14:17:21:12:WU07:FS00:0x21:CPU: 0x00000000000000000000000000000000
2019-10-14:17:21:12:WU07:FS00:0x21:Machine: 0
2019-10-14:17:21:12:WU07:FS00:0x21:Reading tar file core.xml
2019-10-14:17:21:12:WU07:FS00:0x21:Reading tar file integrator.xml
2019-10-14:17:21:12:WU07:FS00:0x21:Reading tar file state.xml
2019-10-14:17:21:12:WU07:FS00:0x21:Reading tar file system.xml
2019-10-14:17:21:12:WU07:FS00:0x21:Digital signatures verified
2019-10-14:17:21:12:WU07:FS00:0x21:Folding@home GPU Core21 Folding@home Core
2019-10-14:17:21:12:WU07:FS00:0x21:Version 0.0.20
2019-10-14:17:21:15:WU00:FS00:Upload 19.64%
2019-10-14:17:21:16:WU02:FS01:Upload 11.38%
2019-10-14:17:21:21:WU01:FS00:Upload 21.51%
2019-10-14:17:21:22:WU07:FS00:0x21:Completed 0 out of 1000000 steps (0%)
2019-10-14:17:21:22:WU07:FS00:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2019-10-14:17:21:22:WU04:FS01:Upload 81.21%
2019-10-14:17:21:23:WU05:FS00:Upload 2.14%
2019-10-14:17:21:28:WU04:FS01:Upload 81.30%
2019-10-14:17:21:29:WU01:FS00:Upload 21.60%
2019-10-14:17:21:31:WU00:FS00:Upload 19.81%
2019-10-14:17:21:31:WU03:FS01:Upload 0.39%
2019-10-14:17:21:37:WU03:FS01:Upload 0.48%
2019-10-14:17:21:37:WU05:FS00:Upload 2.40%
2019-10-14:17:21:38:WU02:FS01:Upload 11.46%
2019-10-14:17:21:39:WU01:FS00:Upload 21.69%
2019-10-14:17:21:42:WU00:FS00:Upload 19.97%
2019-10-14:17:21:47:WU03:FS01:Upload 0.58%
2019-10-14:17:21:47:WU05:FS00:Upload 2.48%
2019-10-14:17:21:49:WU06:FS01:0x21:Completed 650000 out of 1000000 steps (65%)
2019-10-14:17:21:52:WU02:FS01:Upload 11.63%
2019-10-14:17:21:54:WU00:FS00:Upload 20.06%
2019-10-14:17:21:56:WU05:FS00:Upload 2.74%
2019-10-14:17:21:58:WU04:FS01:Upload 81.48%
2019-10-14:17:22:00:WU03:FS01:Upload 0.68%
2019-10-14:17:22:01:WU02:FS01:Upload 11.80%
2019-10-14:17:22:04:WU04:FS01:Upload 81.57%
2019-10-14:17:22:04:WU07:FS00:0x21:Completed 10000 out of 1000000 steps (1%)
2019-10-14:17:22:06:WU00:FS00:Upload 20.14%
2019-10-14:17:22:09:WU02:FS01:Upload 11.89%
2019-10-14:17:22:10:WU05:FS00:Upload 2.82%
2019-10-14:17:22:12:WU04:FS01:Upload 81.66%
2019-10-14:17:22:14:WU03:FS01:Upload 0.97%
2019-10-14:17:22:14:WU01:FS00:Upload 21.77%
2019-10-14:17:22:15:WU00:FS00:Upload 20.22%
2019-10-14:17:22:17:WU02:FS01:Upload 12.06%
2019-10-14:17:22:18:WU04:FS01:Upload 81.75%
2019-10-14:17:22:22:WU01:FS00:Upload 21.95%
2019-10-14:17:22:24:WU00:FS00:Upload 20.39%
2019-10-14:17:22:24:WU05:FS00:Upload 2.91%
2019-10-14:17:22:27:WU06:FS01:0x21:Completed 660000 out of 1000000 steps (66%)
2019-10-14:17:22:28:WU01:FS00:Upload 22.04%
2019-10-14:17:22:29:WU04:FS01:Upload 81.94%
2019-10-14:17:22:31:WU00:FS00:Upload 20.47%
2019-10-14:17:22:33:WU05:FS00:Upload 2.99%
2019-10-14:17:22:38:WU00:FS00:Upload 20.64%
2019-10-14:17:22:39:WU05:FS00:Upload 3.17%
2019-10-14:17:22:40:WU04:FS01:Upload 82.12%
2019-10-14:17:22:45:WU07:FS00:0x21:Completed 20000 out of 1000000 steps (2%)
2019-10-14:17:22:47:WU05:FS00:Upload 3.25%
2019-10-14:17:22:47:WU04:FS01:Upload 82.21%
2019-10-14:17:22:53:WU05:FS00:Upload 3.34%
2019-10-14:17:22:53:WU01:FS00:Upload 22.13%
2019-10-14:17:22:59:WU05:FS00:Upload 3.42%
2019-10-14:17:22:59:WU03:FS01:Upload 1.06%
2019-10-14:17:23:00:WU04:FS01:Upload 82.30%
2019-10-14:17:23:02:WU00:FS00:Upload 20.72%
2019-10-14:17:23:05:WU03:FS01:Upload 1.16%
2019-10-14:17:23:08:WU06:FS01:0x21:Completed 670000 out of 1000000 steps (67%)
2019-10-14:17:23:09:WU05:FS00:Upload 3.59%
2019-10-14:17:23:13:WU00:FS00:Upload 20.81%
2019-10-14:17:23:15:WU02:FS01:Upload 12.23%
2019-10-14:17:23:23:WU04:FS01:Upload 82.57%
2019-10-14:17:23:25:WU05:FS00:Upload 3.68%
2019-10-14:17:23:26:WU00:FS00:Upload 20.89%
2019-10-14:17:23:29:WU04:FS01:Upload 82.67%
2019-10-14:17:23:31:WU07:FS00:0x21:Completed 30000 out of 1000000 steps (3%)
2019-10-14:17:23:32:WU00:FS00:Upload 20.97%
2019-10-14:17:23:35:WU03:FS01:Upload 1.35%
2019-10-14:17:23:35:WU02:FS01:Upload 12.32%
2019-10-14:17:23:35:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:23:35:WU02:FS01:Trying to send results to collection server
2019-10-14:17:23:35:WU02:FS01:Uploading 73.07MiB to 155.247.166.220
2019-10-14:17:23:35:WU02:FS01:Connecting to 155.247.166.220:8080
2019-10-14:17:23:35:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-14:17:23:35:WARNING:WU06:FS01:Detected clock skew (2 mins 23 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:23:36:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14272 run:0 clone:51 gen:40 core:0x21 unit:0x0000002a80fccb0a5d9cfc7006c7a817
2019-10-14:17:23:36:WU02:FS01:Uploading 73.07MiB to 128.252.203.10
2019-10-14:17:23:36:WU02:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:23:36:WU05:FS00:Upload 3.85%
2019-10-14:17:23:38:WU00:FS00:Upload 21.06%
2019-10-14:17:23:44:WU02:FS01:Upload 0.34%
2019-10-14:17:23:44:WU04:FS01:Upload 82.94%
2019-10-14:17:23:46:WU06:FS01:0x21:Completed 680000 out of 1000000 steps (68%)
2019-10-14:17:23:46:WU03:FS01:Upload 1.45%
2019-10-14:17:23:46:WU00:FS00:Upload 21.14%
2019-10-14:17:23:47:WU05:FS00:Upload 4.11%
2019-10-14:17:23:49:WU01:FS00:Upload 22.21%
2019-10-14:17:23:52:WU03:FS01:Upload 1.55%
2019-10-14:17:23:52:WU02:FS01:Upload 0.43%
2019-10-14:17:23:54:WU00:FS00:Upload 21.23%
2019-10-14:17:23:57:WU05:FS00:Upload 4.19%
2019-10-14:17:24:00:WU02:FS01:Upload 0.51%
2019-10-14:17:24:00:WU04:FS01:Upload 83.12%
2019-10-14:17:24:02:WU03:FS01:Upload 1.64%
2019-10-14:17:24:03:WU01:FS00:Upload 22.30%
2019-10-14:17:24:03:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:24:03:WU01:FS00:Trying to send results to collection server
2019-10-14:17:24:03:WU01:FS00:Uploading 71.19MiB to 155.247.166.220
2019-10-14:17:24:03:WU01:FS00:Connecting to 155.247.166.220:8080
2019-10-14:17:24:04:WU05:FS00:Upload 4.28%
2019-10-14:17:24:04:ERROR:WU01:FS00:Exception: Transfer failed
2019-10-14:17:24:04:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14277 run:0 clone:56 gen:33 core:0x21 unit:0x0000002180fccb0a5d98ab63c6c35655
2019-10-14:17:24:04:WU01:FS00:Uploading 71.19MiB to 128.252.203.10
2019-10-14:17:24:04:WU01:FS00:Connecting to 128.252.203.10:8080
2019-10-14:17:24:07:WU04:FS01:Upload 83.21%
2019-10-14:17:24:10:WU02:FS01:Upload 0.60%
2019-10-14:17:24:12:WU07:FS00:0x21:Completed 40000 out of 1000000 steps (4%)
2019-10-14:17:24:14:WU01:FS00:Upload 0.26%
2019-10-14:17:24:15:WU04:FS01:Upload 83.30%
2019-10-14:17:24:16:WU05:FS00:Upload 4.53%
2019-10-14:17:24:21:WU02:FS01:Upload 0.77%
2019-10-14:17:24:22:WU01:FS00:Upload 0.35%
2019-10-14:17:24:23:WU05:FS00:Upload 4.62%
2019-10-14:17:24:27:WU06:FS01:0x21:Completed 690000 out of 1000000 steps (69%)
2019-10-14:17:24:28:WU04:FS01:Upload 83.39%
2019-10-14:17:24:29:WU05:FS00:Upload 4.71%
2019-10-14:17:24:35:WU03:FS01:Upload 1.84%
2019-10-14:17:24:36:WU05:FS00:Upload 4.88%
2019-10-14:17:24:37:WU01:FS00:Upload 0.53%
2019-10-14:17:24:40:WU04:FS01:Upload 83.48%
2019-10-14:17:24:43:WU05:FS00:Upload 4.96%
2019-10-14:17:24:44:WU03:FS01:Upload 1.94%
2019-10-14:17:24:46:WU00:FS00:Upload 21.31%
2019-10-14:17:24:47:WU01:FS00:Upload 0.61%
2019-10-14:17:24:49:WU04:FS01:Upload 83.67%
2019-10-14:17:24:51:WU05:FS00:Upload 5.05%
2019-10-14:17:24:54:WU03:FS01:Upload 2.03%
2019-10-14:17:24:56:WU01:FS00:Upload 0.70%
2019-10-14:17:24:57:WU07:FS00:0x21:Completed 50000 out of 1000000 steps (5%)
2019-10-14:17:24:59:WU05:FS00:Upload 5.13%
2019-10-14:17:24:59:WU02:FS01:Upload 0.86%
2019-10-14:17:25:01:WU04:FS01:Upload 83.76%
2019-10-14:17:25:04:WU01:FS00:Upload 0.79%
2019-10-14:17:25:04:WU03:FS01:Upload 2.13%
2019-10-14:17:25:05:WU06:FS01:0x21:Completed 700000 out of 1000000 steps (70%)
2019-10-14:17:25:08:WU05:FS00:Upload 5.30%
2019-10-14:17:25:12:WU01:FS00:Upload 0.88%
2019-10-14:17:25:12:WU03:FS01:Upload 2.32%
2019-10-14:17:25:13:WU04:FS01:Upload 83.94%
2019-10-14:17:25:15:WU05:FS00:Upload 5.39%
2019-10-14:17:25:17:WU02:FS01:Upload 1.11%
2019-10-14:17:25:19:WU04:FS01:Upload 84.03%
2019-10-14:17:25:23:WU02:FS01:Upload 1.20%
2019-10-14:17:25:23:WU01:FS00:Upload 1.14%
2019-10-14:17:25:25:WU05:FS00:Upload 5.48%
2019-10-14:17:25:28:WU04:FS01:Upload 84.12%
2019-10-14:17:25:35:WU01:FS00:Upload 1.23%
2019-10-14:17:25:38:WU05:FS00:Upload 5.56%
2019-10-14:17:25:39:WU07:FS00:0x21:Completed 60000 out of 1000000 steps (6%)
2019-10-14:17:25:40:WU02:FS01:Upload 1.28%
2019-10-14:17:25:41:WU04:FS01:Upload 84.30%
2019-10-14:17:25:46:WU06:FS01:0x21:Completed 710000 out of 1000000 steps (71%)
2019-10-14:17:25:47:WU05:FS00:Upload 5.65%
2019-10-14:17:25:48:WU02:FS01:Upload 1.37%
2019-10-14:17:25:49:WU01:FS00:Upload 1.49%
2019-10-14:17:25:54:WU05:FS00:Upload 5.82%
2019-10-14:17:25:56:WU01:FS00:Upload 1.58%
2019-10-14:17:25:56:WU04:FS01:Upload 84.49%
2019-10-14:17:25:59:WU02:FS01:Upload 1.63%
2019-10-14:17:26:00:WU05:FS00:Upload 5.90%
2019-10-14:17:26:02:WU04:FS01:Upload 84.58%
2019-10-14:17:26:06:WU01:FS00:Upload 1.76%
2019-10-14:17:26:07:WU05:FS00:Upload 5.99%
2019-10-14:17:26:09:WU03:FS01:Upload 2.42%
2019-10-14:17:26:09:WU02:FS01:Upload 1.71%
2019-10-14:17:26:10:WU04:FS01:Upload 84.67%
2019-10-14:17:26:15:WU01:FS00:Upload 1.93%
2019-10-14:17:26:18:WU05:FS00:Upload 6.07%
2019-10-14:17:26:18:WU04:FS01:Upload 84.85%
2019-10-14:17:26:19:WU02:FS01:Upload 1.80%
2019-10-14:17:26:22:WU01:FS00:Upload 2.02%
2019-10-14:17:26:24:WU07:FS00:0x21:Completed 70000 out of 1000000 steps (7%)
2019-10-14:17:26:24:WU06:FS01:0x21:Completed 720000 out of 1000000 steps (72%)
2019-10-14:17:26:25:WU04:FS01:Upload 84.94%
2019-10-14:17:26:29:WU03:FS01:Upload 2.52%
2019-10-14:17:26:29:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:26:29:WU03:FS01:Trying to send results to collection server
2019-10-14:17:26:29:WU03:FS01:Uploading 64.59MiB to 155.247.166.219
2019-10-14:17:26:29:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-14:17:26:29:WU05:FS00:Upload 6.25%
2019-10-14:17:26:29:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-14:17:26:29:WARNING:WU06:FS01:Detected clock skew (2 mins 25 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:26:29:WARNING:WU07:FS00:Detected clock skew (2 mins 25 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:26:30:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:9 gen:53 core:0x21 unit:0x0000003580fccb0a5d98ab5040b3bced
2019-10-14:17:26:30:WU03:FS01:Uploading 64.59MiB to 128.252.203.10
2019-10-14:17:26:30:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:26:31:WU01:FS00:Upload 2.11%
2019-10-14:17:26:34:WU02:FS01:Upload 2.05%
2019-10-14:17:26:36:WU04:FS01:Upload 85.12%
2019-10-14:17:26:37:WU05:FS00:Upload 6.33%
2019-10-14:17:26:39:WU01:FS00:Upload 2.19%
2019-10-14:17:26:40:WU02:FS01:Upload 2.14%
2019-10-14:17:26:42:WU03:FS01:Upload 0.39%
2019-10-14:17:26:44:WU04:FS01:Upload 85.21%
2019-10-14:17:26:46:WU01:FS00:Upload 2.37%
2019-10-14:17:26:49:WU02:FS01:Upload 2.22%
2019-10-14:17:26:51:WU04:FS01:Upload 85.31%
2019-10-14:17:26:51:WU05:FS00:Upload 6.42%
2019-10-14:17:26:52:WU01:FS00:Upload 2.46%
2019-10-14:17:26:56:WU03:FS01:Upload 0.48%
2019-10-14:17:27:00:WU05:FS00:Upload 6.50%
2019-10-14:17:27:00:WU02:FS01:Upload 2.31%
2019-10-14:17:27:05:WU04:FS01:Upload 85.40%
2019-10-14:17:27:06:WU07:FS00:0x21:Completed 80000 out of 1000000 steps (8%)
2019-10-14:17:27:06:WU06:FS01:0x21:Completed 730000 out of 1000000 steps (73%)
2019-10-14:17:27:09:WU02:FS01:Upload 2.48%
2019-10-14:17:27:10:WU05:FS00:Upload 6.67%
2019-10-14:17:27:13:WU01:FS00:Upload 2.63%
2019-10-14:17:27:15:WU03:FS01:Upload 0.68%
2019-10-14:17:27:15:WU02:FS01:Upload 2.57%
2019-10-14:17:27:17:WU05:FS00:Upload 6.76%
2019-10-14:17:27:17:WU04:FS01:Upload 85.58%
2019-10-14:17:27:20:WU01:FS00:Upload 2.81%
2019-10-14:17:27:25:WU03:FS01:Upload 0.97%
2019-10-14:17:27:25:WU04:FS01:Upload 85.67%
2019-10-14:17:27:26:WU02:FS01:Upload 2.65%
2019-10-14:17:27:30:WU01:FS00:Upload 2.90%
2019-10-14:17:27:31:WU05:FS00:Upload 6.93%
2019-10-14:17:27:35:WU02:FS01:Upload 2.74%
2019-10-14:17:27:35:WU03:FS01:Upload 1.16%
2019-10-14:17:27:37:WU01:FS00:Upload 2.99%
2019-10-14:17:27:41:WU02:FS01:Upload 2.91%
2019-10-14:17:27:42:WU03:FS01:Upload 1.35%
2019-10-14:17:27:44:WU05:FS00:Upload 7.19%
2019-10-14:17:27:44:WU06:FS01:0x21:Completed 740000 out of 1000000 steps (74%)
2019-10-14:17:27:48:WU01:FS00:Upload 3.16%
2019-10-14:17:27:48:WU02:FS01:Upload 2.99%
2019-10-14:17:27:50:WU03:FS01:Upload 1.45%
2019-10-14:17:27:51:WU07:FS00:0x21:Completed 90000 out of 1000000 steps (9%)
2019-10-14:17:27:51:WU04:FS01:Upload 85.94%
2019-10-14:17:27:56:WU02:FS01:Upload 3.08%
2019-10-14:17:27:56:WU03:FS01:Upload 1.55%
2019-10-14:17:27:59:WU04:FS01:Upload 86.12%
2019-10-14:17:28:00:WU01:FS00:Upload 3.42%
2019-10-14:17:28:03:WU03:FS01:Upload 1.64%
2019-10-14:17:28:05:WU02:FS01:Upload 3.16%
2019-10-14:17:28:06:WU04:FS01:Upload 86.22%
2019-10-14:17:28:11:WU03:FS01:Upload 1.84%
2019-10-14:17:28:14:WU02:FS01:Upload 3.34%
2019-10-14:17:28:15:WU05:FS00:Upload 7.27%
2019-10-14:17:28:15:WU04:FS01:Upload 86.31%
2019-10-14:17:28:20:WU03:FS01:Upload 1.94%
2019-10-14:17:28:21:WU02:FS01:Upload 3.42%
2019-10-14:17:28:21:WU05:FS00:Upload 7.36%
2019-10-14:17:28:21:WU00:FS00:Upload 21.39%
2019-10-14:17:28:21:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:28:21:WU00:FS00:Trying to send results to collection server
2019-10-14:17:28:21:WU00:FS00:Uploading 74.79MiB to 155.247.166.220
2019-10-14:17:28:21:WU00:FS00:Connecting to 155.247.166.220:8080
2019-10-14:17:28:22:ERROR:WU00:FS00:Exception: Transfer failed
2019-10-14:17:28:22:WARNING:WU06:FS01:Detected clock skew (1 mins 53 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:28:22:WARNING:WU07:FS00:Detected clock skew (1 mins 53 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:28:22:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:14275 run:0 clone:30 gen:34 core:0x21 unit:0x0000002380fccb0a5d98ab57bb217296
2019-10-14:17:28:22:WU00:FS00:Uploading 74.79MiB to 128.252.203.10
2019-10-14:17:28:22:WU00:FS00:Connecting to 128.252.203.10:8080
2019-10-14:17:28:25:WU06:FS01:0x21:Completed 750000 out of 1000000 steps (75%)
2019-10-14:17:28:27:WU03:FS01:Upload 2.03%
2019-10-14:17:28:27:WU01:FS00:Upload 3.60%
2019-10-14:17:28:28:WU04:FS01:Upload 86.49%
2019-10-14:17:28:29:WU02:FS01:Upload 3.51%
2019-10-14:17:28:33:WU07:FS00:0x21:Completed 100000 out of 1000000 steps (10%)
2019-10-14:17:28:33:WU00:FS00:Upload 0.33%
2019-10-14:17:28:34:WU03:FS01:Upload 2.13%
2019-10-14:17:28:35:WU01:FS00:Upload 3.78%
2019-10-14:17:28:40:WU03:FS01:Upload 2.32%
2019-10-14:17:28:40:WU00:FS00:Upload 0.42%
2019-10-14:17:28:41:WU01:FS00:Upload 3.86%
2019-10-14:17:28:43:WU02:FS01:Upload 3.59%
2019-10-14:17:28:44:WU04:FS01:Upload 86.76%
2019-10-14:17:28:45:WU05:FS00:Upload 7.44%
2019-10-14:17:28:46:WU00:FS00:Upload 0.50%
2019-10-14:17:28:49:WU03:FS01:Upload 2.42%
2019-10-14:17:28:56:WU01:FS00:Upload 4.04%
2019-10-14:17:28:57:WU04:FS01:Upload 86.85%
2019-10-14:17:28:57:WU05:FS00:Upload 7.61%
2019-10-14:17:29:03:WU06:FS01:0x21:Completed 760000 out of 1000000 steps (76%)
2019-10-14:17:29:03:WU05:FS00:Upload 7.70%
2019-10-14:17:29:07:WU04:FS01:Upload 87.04%
2019-10-14:17:29:10:WU01:FS00:Upload 4.21%
2019-10-14:17:29:10:WU00:FS00:Upload 0.58%
2019-10-14:17:29:14:WU04:FS01:Upload 87.22%
2019-10-14:17:29:16:WU00:FS00:Upload 0.75%
2019-10-14:17:29:16:WU01:FS00:Upload 4.30%
2019-10-14:17:29:17:WU05:FS00:Upload 7.87%
2019-10-14:17:29:18:WU07:FS00:0x21:Completed 110000 out of 1000000 steps (11%)
2019-10-14:17:29:21:WU04:FS01:Upload 87.31%
2019-10-14:17:29:22:WU01:FS00:Upload 4.39%
2019-10-14:17:29:25:WU00:FS00:Upload 0.92%
2019-10-14:17:29:27:WU02:FS01:Upload 3.76%
2019-10-14:17:29:28:WU04:FS01:Upload 87.40%
2019-10-14:17:29:30:WU05:FS00:Upload 8.04%
2019-10-14:17:29:31:WU01:FS00:Upload 4.65%
2019-10-14:17:29:39:WU00:FS00:Upload 1.00%
2019-10-14:17:29:39:WU01:FS00:Upload 4.74%
2019-10-14:17:29:40:WU04:FS01:Upload 87.58%
2019-10-14:17:29:44:WU06:FS01:0x21:Completed 770000 out of 1000000 steps (77%)
2019-10-14:17:29:45:WU01:FS00:Upload 4.83%
2019-10-14:17:29:49:WU03:FS01:Upload 2.52%
2019-10-14:17:29:50:WU05:FS00:Upload 8.21%
2019-10-14:17:29:50:WU00:FS00:Upload 1.25%
2019-10-14:17:29:52:WU01:FS00:Upload 4.92%
2019-10-14:17:29:53:WU04:FS01:Upload 87.76%
2019-10-14:17:29:56:WU02:FS01:Upload 3.93%
2019-10-14:17:29:58:WU01:FS00:Upload 5.09%
2019-10-14:17:29:59:WU05:FS00:Upload 8.30%
2019-10-14:17:29:59:WU07:FS00:0x21:Completed 120000 out of 1000000 steps (12%)
2019-10-14:17:30:05:WU05:FS00:Upload 8.47%
2019-10-14:17:30:06:WU01:FS00:Upload 5.18%
2019-10-14:17:30:06:WU00:FS00:Upload 1.34%
2019-10-14:17:30:08:WU02:FS01:Upload 4.11%
2019-10-14:17:30:13:WU05:FS00:Upload 8.56%
2019-10-14:17:30:15:WU04:FS01:Upload 87.85%
2019-10-14:17:30:17:WU00:FS00:Upload 1.59%
2019-10-14:17:30:18:WU01:FS00:Upload 5.27%
2019-10-14:17:30:20:WU05:FS00:Upload 8.64%
2019-10-14:17:30:21:WU04:FS01:Upload 87.95%
2019-10-14:17:30:22:WU06:FS01:0x21:Completed 780000 out of 1000000 steps (78%)
2019-10-14:17:30:29:WU02:FS01:Upload 4.36%
2019-10-14:17:30:44:WU07:FS00:0x21:Completed 130000 out of 1000000 steps (13%)
2019-10-14:17:30:57:WU04:FS01:Upload 88.13%
2019-10-14:17:30:58:WU00:FS00:Upload 1.75%
2019-10-14:17:30:58:WU05:FS00:Upload 8.73%
2019-10-14:17:31:03:WU06:FS01:0x21:Completed 790000 out of 1000000 steps (79%)
2019-10-14:17:31:04:WU00:FS00:Upload 2.01%
2019-10-14:17:31:06:WU05:FS00:Upload 8.90%
2019-10-14:17:31:08:WU04:FS01:Upload 88.40%
2019-10-14:17:31:08:WU02:FS01:Upload 4.45%
2019-10-14:17:31:10:WU01:FS00:Upload 5.36%
2019-10-14:17:31:12:WU00:FS00:Upload 2.17%
2019-10-14:17:31:18:WU02:FS01:Upload 4.53%
2019-10-14:17:31:19:WU04:FS01:Upload 88.58%
2019-10-14:17:31:21:WU00:FS00:Upload 2.34%
2019-10-14:17:31:26:WU02:FS01:Upload 4.70%
2019-10-14:17:31:26:WU07:FS00:0x21:Completed 140000 out of 1000000 steps (14%)
2019-10-14:17:31:27:WU04:FS01:Upload 88.67%
2019-10-14:17:31:28:WU05:FS00:Upload 9.07%
2019-10-14:17:31:30:WU00:FS00:Upload 2.42%
2019-10-14:17:31:33:WU02:FS01:Upload 4.79%
2019-10-14:17:31:39:WU05:FS00:Upload 9.24%
2019-10-14:17:31:40:WU04:FS01:Upload 88.86%
2019-10-14:17:31:40:WU00:FS00:Upload 2.51%
2019-10-14:17:31:41:WU06:FS01:0x21:Completed 800000 out of 1000000 steps (80%)
2019-10-14:17:31:46:WU05:FS00:Upload 9.33%
2019-10-14:17:31:49:WU04:FS01:Upload 89.04%
2019-10-14:17:31:51:WU01:FS00:Upload 5.53%
2019-10-14:17:31:55:WU00:FS00:Upload 2.59%
2019-10-14:17:31:58:WU05:FS00:Upload 9.50%
2019-10-14:17:31:58:WU04:FS01:Upload 89.13%
2019-10-14:17:32:00:WU02:FS01:Upload 4.88%
2019-10-14:17:32:02:WU00:FS00:Upload 2.67%
2019-10-14:17:32:02:WU01:FS00:Upload 5.71%
2019-10-14:17:32:11:WU07:FS00:0x21:Completed 150000 out of 1000000 steps (15%)
2019-10-14:17:32:12:WU05:FS00:Upload 9.58%
2019-10-14:17:32:13:WU01:FS00:Upload 5.88%
2019-10-14:17:32:15:WU04:FS01:Upload 89.31%
2019-10-14:17:32:16:WU00:FS00:Upload 2.76%
2019-10-14:17:32:22:WU00:FS00:Upload 2.84%
2019-10-14:17:32:22:WU06:FS01:0x21:Completed 810000 out of 1000000 steps (81%)
2019-10-14:17:32:22:WU05:FS00:Upload 9.84%
2019-10-14:17:32:25:WU01:FS00:Upload 6.06%
2019-10-14:17:32:29:WU05:FS00:Upload 9.92%
2019-10-14:17:32:30:WU00:FS00:Upload 2.92%
2019-10-14:17:32:33:WU04:FS01:Upload 89.58%
2019-10-14:17:32:37:WU00:FS00:Upload 3.18%
2019-10-14:17:32:40:WU01:FS00:Upload 6.15%
2019-10-14:17:32:40:WU05:FS00:Upload 10.18%
2019-10-14:17:32:42:WU04:FS01:Upload 89.68%
2019-10-14:17:32:46:WU00:FS00:Upload 3.26%
2019-10-14:17:32:47:WU05:FS00:Upload 10.27%
2019-10-14:17:32:52:WU00:FS00:Upload 3.34%
2019-10-14:17:32:53:WU04:FS01:Upload 89.86%
2019-10-14:17:32:53:WU07:FS00:0x21:Completed 160000 out of 1000000 steps (16%)
2019-10-14:17:32:53:WU05:FS00:Upload 10.35%
2019-10-14:17:32:59:WU05:FS00:Upload 10.44%
2019-10-14:17:33:00:WU01:FS00:Upload 6.41%
2019-10-14:17:33:00:WU04:FS01:Upload 90.04%
2019-10-14:17:33:00:WU06:FS01:0x21:Completed 820000 out of 1000000 steps (82%)
2019-10-14:17:33:01:WU00:FS00:Upload 3.51%
2019-10-14:17:33:02:WU03:FS01:Upload 2.61%
2019-10-14:17:33:02:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:33:02:WU03:FS01:Trying to send results to collection server
2019-10-14:17:33:02:WU03:FS01:Uploading 64.59MiB to 155.247.166.219
2019-10-14:17:33:02:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-14:17:33:02:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-14:17:33:03:WARNING:WU06:FS01:Detected clock skew (4 mins 41 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:33:03:WARNING:WU07:FS00:Detected clock skew (4 mins 41 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:33:05:WU05:FS00:Upload 10.61%
2019-10-14:17:33:07:WU04:FS01:Upload 90.13%
2019-10-14:17:33:07:WU00:FS00:Upload 3.59%
2019-10-14:17:33:11:WU05:FS00:Upload 10.69%
2019-10-14:17:33:12:WU01:FS00:Upload 6.50%
2019-10-14:17:33:16:WU02:FS01:Upload 5.05%
2019-10-14:17:33:16:WU04:FS01:Upload 90.31%
2019-10-14:17:33:17:WU05:FS00:Upload 10.78%
2019-10-14:17:33:21:WU00:FS00:Upload 3.76%
2019-10-14:17:33:21:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:9 gen:53 core:0x21 unit:0x0000003580fccb0a5d98ab5040b3bced
2019-10-14:17:33:21:WU03:FS01:Uploading 64.59MiB to 128.252.203.10
2019-10-14:17:33:21:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:33:22:WU04:FS01:Upload 90.40%
2019-10-14:17:33:24:WU05:FS00:Upload 10.87%
2019-10-14:17:33:26:WU01:FS00:Upload 6.58%
2019-10-14:17:33:27:WU03:FS01:Upload 0.19%
2019-10-14:17:33:31:WU00:FS00:Upload 3.93%
2019-10-14:17:33:31:WU02:FS01:Upload 5.13%
2019-10-14:17:33:31:WU04:FS01:Upload 90.49%
2019-10-14:17:33:37:WU01:FS00:Upload 6.67%
2019-10-14:17:33:38:WU03:FS01:Upload 0.39%
2019-10-14:17:33:38:WU07:FS00:0x21:Completed 170000 out of 1000000 steps (17%)
2019-10-14:17:33:38:WU05:FS00:Upload 11.04%
2019-10-14:17:33:39:WU04:FS01:Upload 90.59%
2019-10-14:17:33:41:WU06:FS01:0x21:Completed 830000 out of 1000000 steps (83%)
2019-10-14:17:33:44:WU00:FS00:Upload 4.09%
2019-10-14:17:33:45:WU05:FS00:Upload 11.12%
2019-10-14:17:33:45:WU01:FS00:Upload 6.94%
2019-10-14:17:33:48:WU03:FS01:Upload 0.48%
2019-10-14:17:33:48:WU02:FS01:Upload 5.22%
2019-10-14:17:33:50:WU04:FS01:Upload 90.77%
2019-10-14:17:33:51:WU05:FS00:Upload 11.21%
2019-10-14:17:33:53:WU00:FS00:Upload 4.18%
2019-10-14:17:33:56:WU02:FS01:Upload 5.30%
2019-10-14:17:34:00:WU03:FS01:Upload 0.68%
2019-10-14:17:34:03:WU02:FS01:Upload 5.47%
2019-10-14:17:34:04:WU00:FS00:Upload 4.35%
2019-10-14:17:34:10:WU04:FS01:Upload 91.04%
2019-10-14:17:34:10:WU03:FS01:Upload 0.97%
2019-10-14:17:34:11:WU02:FS01:Upload 5.56%
2019-10-14:17:34:16:WU03:FS01:Upload 1.06%
2019-10-14:17:34:17:WU01:FS00:Upload 7.02%
2019-10-14:17:34:18:WU00:FS00:Upload 4.60%
2019-10-14:17:34:19:WU06:FS01:0x21:Completed 840000 out of 1000000 steps (84%)
2019-10-14:17:34:20:WU07:FS00:0x21:Completed 180000 out of 1000000 steps (18%)
2019-10-14:17:34:23:WU02:FS01:Upload 5.65%
2019-10-14:17:34:23:WU03:FS01:Upload 1.16%
2019-10-14:17:34:26:WU01:FS00:Upload 7.11%
2019-10-14:17:34:27:WU04:FS01:Upload 91.22%
2019-10-14:17:34:29:WU02:FS01:Upload 5.73%
2019-10-14:17:34:30:WU00:FS00:Upload 4.76%
2019-10-14:17:34:33:WU01:FS00:Upload 7.20%
2019-10-14:17:34:33:WU05:FS00:Upload 11.29%
2019-10-14:17:34:39:WU00:FS00:Upload 4.85%
2019-10-14:17:34:41:WU04:FS01:Upload 91.41%
2019-10-14:17:34:45:WU02:FS01:Upload 5.90%
2019-10-14:17:34:48:WU00:FS00:Upload 5.01%
2019-10-14:17:34:50:WU04:FS01:Upload 91.59%
2019-10-14:17:34:51:WU02:FS01:Upload 5.99%
2019-10-14:17:34:56:WU01:FS00:Upload 7.46%
2019-10-14:17:34:57:WU00:FS00:Upload 5.18%
2019-10-14:17:34:57:WU05:FS00:Upload 11.46%
2019-10-14:17:34:58:WU04:FS01:Upload 91.68%
2019-10-14:17:35:00:WU06:FS01:0x21:Completed 850000 out of 1000000 steps (85%)
2019-10-14:17:35:05:WU02:FS01:Upload 6.16%
2019-10-14:17:35:05:WU07:FS00:0x21:Completed 190000 out of 1000000 steps (19%)
2019-10-14:17:35:05:WU04:FS01:Upload 91.77%
2019-10-14:17:35:11:WU01:FS00:Upload 7.64%
2019-10-14:17:35:11:WU04:FS01:Upload 91.86%
2019-10-14:17:35:16:WU02:FS01:Upload 6.33%
2019-10-14:17:35:16:WU00:FS00:Upload 5.35%
2019-10-14:17:35:20:WU05:FS00:Upload 11.55%
2019-10-14:17:35:22:WU02:FS01:Upload 6.42%
2019-10-14:17:35:23:WU01:FS00:Upload 7.81%
2019-10-14:17:35:24:WU04:FS01:Upload 92.04%
2019-10-14:17:35:25:WU00:FS00:Upload 5.52%
2019-10-14:17:35:26:WU05:FS00:Upload 11.64%
2019-10-14:17:35:31:WU00:FS00:Upload 5.60%
2019-10-14:17:35:33:WU01:FS00:Upload 7.90%
2019-10-14:17:35:33:WU02:FS01:Upload 6.50%
2019-10-14:17:35:38:WU06:FS01:0x21:Completed 860000 out of 1000000 steps (86%)
2019-10-14:17:35:39:WU01:FS00:Upload 7.99%
2019-10-14:17:35:41:WU04:FS01:Upload 92.32%
2019-10-14:17:35:41:WU00:FS00:Upload 5.68%
2019-10-14:17:35:44:WU02:FS01:Upload 6.59%
2019-10-14:17:35:46:WU07:FS00:0x21:Completed 200000 out of 1000000 steps (20%)
2019-10-14:17:35:53:WU05:FS00:Upload 11.72%
2019-10-14:17:35:54:WU02:FS01:Upload 6.67%
2019-10-14:17:35:56:WU01:FS00:Upload 8.17%
2019-10-14:17:36:00:WU04:FS01:Upload 92.50%
2019-10-14:17:36:02:WU00:FS00:Upload 5.93%
2019-10-14:17:36:05:WU05:FS00:Upload 11.89%
2019-10-14:17:36:06:WU02:FS01:Upload 6.84%
2019-10-14:17:36:07:WU04:FS01:Upload 92.59%
2019-10-14:17:36:07:WU01:FS00:Upload 8.25%
2019-10-14:17:36:11:WU00:FS00:Upload 6.10%
2019-10-14:17:36:13:WU01:FS00:Upload 8.34%
2019-10-14:17:36:18:WU00:FS00:Upload 6.18%
2019-10-14:17:36:19:WU06:FS01:0x21:Completed 870000 out of 1000000 steps (87%)
2019-10-14:17:36:20:WU05:FS00:Upload 11.98%
2019-10-14:17:36:21:WU04:FS01:Upload 92.77%
2019-10-14:17:36:22:WU02:FS01:Upload 6.93%
2019-10-14:17:36:22:WU01:FS00:Upload 8.43%
2019-10-14:17:36:30:WU00:FS00:Upload 6.35%
2019-10-14:17:36:30:WU04:FS01:Upload 92.95%
2019-10-14:17:36:31:WU01:FS00:Upload 8.52%
2019-10-14:17:36:32:WU07:FS00:0x21:Completed 210000 out of 1000000 steps (21%)
2019-10-14:17:36:33:WU02:FS01:Upload 7.01%
2019-10-14:17:36:33:WU05:FS00:Upload 12.15%
2019-10-14:17:36:37:WU04:FS01:Upload 93.04%
2019-10-14:17:36:37:WU01:FS00:Upload 8.69%
2019-10-14:17:36:42:WU00:FS00:Upload 6.52%
2019-10-14:17:36:43:WU05:FS00:Upload 12.32%
2019-10-14:17:36:44:WU01:FS00:Upload 8.78%
2019-10-14:17:36:50:WU00:FS00:Upload 6.60%
2019-10-14:17:36:51:WU05:FS00:Upload 12.41%
2019-10-14:17:36:53:WU01:FS00:Upload 8.87%
2019-10-14:17:36:53:WU03:FS01:Upload 1.26%
2019-10-14:17:36:53:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:36:53:WU03:FS01:Trying to send results to collection server
2019-10-14:17:36:53:WU03:FS01:Uploading 64.59MiB to 155.247.166.219
2019-10-14:17:36:53:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-14:17:36:54:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-14:17:36:54:WARNING:WU06:FS01:Detected clock skew (3 mins 33 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:36:54:WU04:FS01:Upload 93.23%
2019-10-14:17:36:54:WARNING:WU07:FS00:Detected clock skew (3 mins 33 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:36:57:WU06:FS01:0x21:Completed 880000 out of 1000000 steps (88%)
2019-10-14:17:36:58:WU05:FS00:Upload 12.49%
2019-10-14:17:37:00:WU02:FS01:Upload 7.27%
2019-10-14:17:37:01:WU01:FS00:Upload 8.96%
2019-10-14:17:37:03:WU04:FS01:Upload 93.32%
2019-10-14:17:37:04:WU05:FS00:Upload 12.58%
2019-10-14:17:37:11:WU00:FS00:Upload 6.69%
2019-10-14:17:37:12:WU04:FS01:Upload 93.41%
2019-10-14:17:37:13:WU01:FS00:Upload 9.22%
2019-10-14:17:37:13:WU07:FS00:0x21:Completed 220000 out of 1000000 steps (22%)
2019-10-14:17:37:16:WU02:FS01:Upload 7.44%
2019-10-14:17:37:16:WU05:FS00:Upload 12.75%
2019-10-14:17:37:17:WU00:FS00:Upload 6.77%
2019-10-14:17:37:19:WU01:FS00:Upload 9.31%
2019-10-14:17:37:22:WU02:FS01:Upload 7.53%
2019-10-14:17:37:23:WU05:FS00:Upload 12.83%
2019-10-14:17:37:24:WU04:FS01:Upload 93.59%
2019-10-14:17:37:24:WU00:FS00:Upload 6.85%
2019-10-14:17:37:29:WU02:FS01:Upload 7.61%
2019-10-14:17:37:30:WU00:FS00:Upload 6.94%
2019-10-14:17:37:33:WU01:FS00:Upload 9.57%
2019-10-14:17:37:34:WU05:FS00:Upload 12.92%
2019-10-14:17:37:36:WU02:FS01:Upload 7.78%
2019-10-14:17:37:37:WU00:FS00:Upload 7.02%
2019-10-14:17:37:38:WU06:FS01:0x21:Completed 890000 out of 1000000 steps (89%)
2019-10-14:17:37:40:WU01:FS00:Upload 9.66%
2019-10-14:17:37:40:WU05:FS00:Upload 13.00%
2019-10-14:17:37:41:WU04:FS01:Upload 93.77%
2019-10-14:17:37:44:WU02:FS01:Upload 7.87%
2019-10-14:17:37:50:WU01:FS00:Upload 9.75%
2019-10-14:17:37:51:WU02:FS01:Upload 7.96%
2019-10-14:17:37:53:WU00:FS00:Upload 7.10%
2019-10-14:17:37:58:WU07:FS00:0x21:Completed 230000 out of 1000000 steps (23%)
2019-10-14:17:37:59:WU04:FS01:Upload 93.95%
2019-10-14:17:37:59:WU01:FS00:Upload 10.01%
2019-10-14:17:38:02:WU05:FS00:Upload 13.18%
2019-10-14:17:38:02:WU02:FS01:Upload 8.21%
2019-10-14:17:38:03:WU00:FS00:Upload 7.35%
2019-10-14:17:38:09:WU00:FS00:Upload 7.44%
2019-10-14:17:38:11:WU04:FS01:Upload 94.14%
2019-10-14:17:38:12:WU05:FS00:Upload 13.26%
2019-10-14:17:38:13:WU01:FS00:Upload 10.10%
2019-10-14:17:38:14:WU02:FS01:Upload 8.38%
2019-10-14:17:38:15:WU00:FS00:Upload 7.52%
2019-10-14:17:38:16:WU06:FS01:0x21:Completed 900000 out of 1000000 steps (90%)
2019-10-14:17:38:21:WU02:FS01:Upload 8.47%
2019-10-14:17:38:22:WU04:FS01:Upload 94.32%
2019-10-14:17:38:23:WU00:FS00:Upload 7.60%
2019-10-14:17:38:23:WU05:FS00:Upload 13.35%
2019-10-14:17:38:28:WU04:FS01:Upload 94.41%
2019-10-14:17:38:30:WU02:FS01:Upload 8.64%
2019-10-14:17:38:31:WU00:FS00:Upload 7.69%
2019-10-14:17:38:32:WU05:FS00:Upload 13.43%
2019-10-14:17:38:38:WU00:FS00:Upload 7.77%
2019-10-14:17:38:39:WU04:FS01:Upload 94.50%
2019-10-14:17:38:40:WU07:FS00:0x21:Completed 240000 out of 1000000 steps (24%)
2019-10-14:17:38:57:WU06:FS01:0x21:Completed 910000 out of 1000000 steps (91%)
2019-10-14:17:39:00:WU01:FS00:Upload 10.18%
2019-10-14:17:39:08:WU00:FS00:Upload 7.86%
2019-10-14:17:39:12:WU01:FS00:Upload 10.27%
2019-10-14:17:39:15:WU05:FS00:Upload 13.60%
2019-10-14:17:39:18:WU01:FS00:Upload 10.45%
2019-10-14:17:39:20:WU00:FS00:Upload 8.02%
2019-10-14:17:39:23:WU05:FS00:Upload 13.69%
2019-10-14:17:39:25:WU07:FS00:0x21:Completed 250000 out of 1000000 steps (25%)
2019-10-14:17:39:28:WU04:FS01:Upload 94.59%
2019-10-14:17:39:29:WU01:FS00:Upload 10.62%
2019-10-14:17:39:29:WU00:FS00:Upload 8.19%
2019-10-14:17:39:29:WU02:FS01:Upload 8.73%
2019-10-14:17:39:32:WU05:FS00:Upload 13.86%
2019-10-14:17:39:35:WU06:FS01:0x21:Completed 920000 out of 1000000 steps (92%)
2019-10-14:17:39:37:WU00:FS00:Upload 8.36%
2019-10-14:17:39:38:WU04:FS01:Upload 94.77%
2019-10-14:17:39:39:WU01:FS00:Upload 10.71%
2019-10-14:17:39:40:WU02:FS01:Upload 8.81%
2019-10-14:17:39:42:WU05:FS00:Upload 14.12%
2019-10-14:17:39:51:WU00:FS00:Upload 8.52%
2019-10-14:17:39:52:WU05:FS00:Upload 14.20%
2019-10-14:17:39:53:WU01:FS00:Upload 10.97%
2019-10-14:17:39:56:WU04:FS01:Upload 94.86%
2019-10-14:17:39:57:WU02:FS01:Upload 8.90%
2019-10-14:17:39:59:WU05:FS00:Upload 14.29%
2019-10-14:17:39:59:WU01:FS00:Upload 11.06%
2019-10-14:17:40:04:WU02:FS01:Upload 9.07%
2019-10-14:17:40:06:WU05:FS00:Upload 14.37%
2019-10-14:17:40:06:WU04:FS01:Upload 94.96%
2019-10-14:17:40:07:WU07:FS00:0x21:Completed 260000 out of 1000000 steps (26%)
2019-10-14:17:40:08:WU00:FS00:Upload 8.61%
2019-10-14:17:40:11:WU02:FS01:Upload 9.15%
2019-10-14:17:40:12:WU05:FS00:Upload 14.55%
2019-10-14:17:40:15:WU01:FS00:Upload 11.15%
2019-10-14:17:40:16:WU06:FS01:0x21:Completed 930000 out of 1000000 steps (93%)
2019-10-14:17:40:16:WU00:FS00:Upload 8.77%
2019-10-14:17:40:18:WU02:FS01:Upload 9.24%
2019-10-14:17:40:22:WU04:FS01:Upload 95.05%
2019-10-14:17:40:23:WU01:FS00:Upload 11.33%
2019-10-14:17:40:24:WU05:FS00:Upload 14.72%
2019-10-14:17:40:24:WU02:FS01:Upload 9.32%
2019-10-14:17:40:25:WU00:FS00:Upload 8.94%
2019-10-14:17:40:33:WU00:FS00:Upload 9.03%
2019-10-14:17:40:33:WU04:FS01:Upload 95.23%
2019-10-14:17:40:35:WU01:FS00:Upload 11.41%
2019-10-14:17:40:35:WU02:FS01:Upload 9.41%
2019-10-14:17:40:36:WU05:FS00:Upload 14.97%
2019-10-14:17:40:42:WU04:FS01:Upload 95.41%
2019-10-14:17:40:42:WU02:FS01:Upload 9.58%
2019-10-14:17:40:44:WU00:FS00:Upload 9.19%
2019-10-14:17:40:44:WU05:FS00:Upload 15.06%
2019-10-14:17:40:46:WU01:FS00:Upload 11.50%
2019-10-14:17:40:50:WU05:FS00:Upload 15.14%
2019-10-14:17:40:51:WU04:FS01:Upload 95.50%
2019-10-14:17:40:52:WU00:FS00:Upload 9.28%
2019-10-14:17:40:52:WU07:FS00:0x21:Completed 270000 out of 1000000 steps (27%)
2019-10-14:17:40:53:WU01:FS00:Upload 11.59%
2019-10-14:17:40:54:WU06:FS01:0x21:Completed 940000 out of 1000000 steps (94%)
2019-10-14:17:41:03:WU05:FS00:Upload 15.23%
2019-10-14:17:41:04:WU02:FS01:Upload 9.75%
2019-10-14:17:41:04:WU00:FS00:Upload 9.36%
2019-10-14:17:41:06:WU04:FS01:Upload 95.59%
2019-10-14:17:41:10:WU01:FS00:Upload 11.68%
2019-10-14:17:41:11:WU05:FS00:Upload 15.40%
2019-10-14:17:41:12:WU04:FS01:Upload 95.78%
2019-10-14:17:41:13:WU00:FS00:Upload 9.44%
2019-10-14:17:41:16:WU02:FS01:Upload 9.84%
2019-10-14:17:41:19:WU00:FS00:Upload 9.61%
2019-10-14:17:41:22:WU04:FS01:Upload 95.87%
2019-10-14:17:41:22:WU01:FS00:Upload 11.85%
2019-10-14:17:41:25:WU00:FS00:Upload 9.69%
2019-10-14:17:41:26:WU02:FS01:Upload 10.01%
2019-10-14:17:41:28:WU04:FS01:Upload 95.96%
2019-10-14:17:41:29:WU05:FS00:Upload 15.49%
2019-10-14:17:41:33:WU01:FS00:Upload 12.03%
2019-10-14:17:41:34:WU07:FS00:0x21:Completed 280000 out of 1000000 steps (28%)
2019-10-14:17:41:34:WU02:FS01:Upload 10.09%
2019-10-14:17:41:34:WU00:FS00:Upload 9.78%
2019-10-14:17:41:35:WU06:FS01:0x21:Completed 950000 out of 1000000 steps (95%)
2019-10-14:17:41:35:WU04:FS01:Upload 96.05%
2019-10-14:17:41:42:WU01:FS00:Upload 12.20%
2019-10-14:17:41:44:WU00:FS00:Upload 9.86%
2019-10-14:17:41:44:WU05:FS00:Upload 15.57%
2019-10-14:17:41:48:WU04:FS01:Upload 96.23%
2019-10-14:17:41:50:WU01:FS00:Upload 12.29%
2019-10-14:17:41:51:WU05:FS00:Upload 15.66%
2019-10-14:17:41:53:WU00:FS00:Upload 9.94%
2019-10-14:17:41:54:WU02:FS01:Upload 10.18%
2019-10-14:17:41:58:WU01:FS00:Upload 12.38%
2019-10-14:17:41:58:WU05:FS00:Upload 15.83%
2019-10-14:17:42:01:WU02:FS01:Upload 10.26%
2019-10-14:17:42:02:WU04:FS01:Upload 96.32%
2019-10-14:17:42:02:WU00:FS00:Upload 10.11%
2019-10-14:17:42:07:WU05:FS00:Upload 15.91%
2019-10-14:17:42:07:WU02:FS01:Upload 10.35%
2019-10-14:17:42:13:WU06:FS01:0x21:Completed 960000 out of 1000000 steps (96%)
2019-10-14:17:42:16:WU05:FS00:Upload 16.00%
2019-10-14:17:42:16:WU01:FS00:Upload 12.47%
2019-10-14:17:42:17:WU04:FS01:Upload 96.41%
2019-10-14:17:42:19:WU07:FS00:0x21:Completed 290000 out of 1000000 steps (29%)
2019-10-14:17:42:22:WU02:FS01:Upload 10.61%
2019-10-14:17:42:24:WU01:FS00:Upload 12.64%
2019-10-14:17:42:25:WU00:FS00:Upload 10.20%
2019-10-14:17:42:25:WU05:FS00:Upload 16.09%
2019-10-14:17:42:31:WU00:FS00:Upload 10.28%
2019-10-14:17:42:32:WU01:FS00:Upload 12.73%
2019-10-14:17:42:33:WU05:FS00:Upload 16.17%
2019-10-14:17:42:37:WU02:FS01:Upload 10.69%
2019-10-14:17:42:38:WU01:FS00:Upload 12.82%
2019-10-14:17:42:39:WU04:FS01:Upload 96.69%
2019-10-14:17:42:40:WU05:FS00:Upload 16.34%
2019-10-14:17:42:43:WU00:FS00:Upload 10.45%
2019-10-14:17:42:46:WU02:FS01:Upload 10.86%
2019-10-14:17:42:48:WU01:FS00:Upload 12.91%
2019-10-14:17:42:49:WU05:FS00:Upload 16.43%
2019-10-14:17:42:49:WU00:FS00:Upload 10.61%
2019-10-14:17:42:54:WU06:FS01:0x21:Completed 970000 out of 1000000 steps (97%)
2019-10-14:17:42:55:WU05:FS00:Upload 16.51%
2019-10-14:17:42:59:WU01:FS00:Upload 13.08%
2019-10-14:17:42:59:WU04:FS01:Upload 96.87%
2019-10-14:17:43:01:WU07:FS00:0x21:Completed 300000 out of 1000000 steps (30%)
2019-10-14:17:43:05:WU05:FS00:Upload 16.60%
2019-10-14:17:43:05:WU00:FS00:Upload 10.70%
2019-10-14:17:43:09:WU01:FS00:Upload 13.17%
2019-10-14:17:43:11:WU04:FS01:Upload 96.96%
2019-10-14:17:43:11:WU00:FS00:Upload 10.78%
2019-10-14:17:43:12:WU05:FS00:Upload 16.77%
2019-10-14:17:43:18:WU00:FS00:Upload 10.86%
2019-10-14:17:43:19:WU01:FS00:Upload 13.26%
2019-10-14:17:43:19:WU05:FS00:Upload 16.86%
2019-10-14:17:43:22:WU04:FS01:Upload 97.05%
2019-10-14:17:43:26:WU01:FS00:Upload 13.35%
2019-10-14:17:43:27:WU00:FS00:Upload 10.95%
2019-10-14:17:43:27:WU05:FS00:Upload 17.03%
2019-10-14:17:43:31:WU02:FS01:Upload 11.03%
2019-10-14:17:43:32:WU06:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
2019-10-14:17:43:34:WU05:FS00:Upload 17.20%
2019-10-14:17:43:37:WU01:FS00:Upload 13.61%
2019-10-14:17:43:37:WU00:FS00:Upload 11.03%
2019-10-14:17:43:45:WU05:FS00:Upload 17.28%
2019-10-14:17:43:46:WU07:FS00:0x21:Completed 310000 out of 1000000 steps (31%)
2019-10-14:17:43:47:WU00:FS00:Upload 11.20%
2019-10-14:17:43:52:WU01:FS00:Upload 13.70%
2019-10-14:17:43:54:WU02:FS01:Upload 11.12%
2019-10-14:17:43:57:WU05:FS00:Upload 17.37%
2019-10-14:17:44:00:WU01:FS00:Upload 13.78%
2019-10-14:17:44:01:WU02:FS01:Upload 11.21%
2019-10-14:17:44:02:WU00:FS00:Upload 11.36%
2019-10-14:17:44:06:WU01:FS00:Upload 13.96%
2019-10-14:17:44:10:WU05:FS00:Upload 17.45%
2019-10-14:17:44:13:WU02:FS01:Upload 11.38%
2019-10-14:17:44:13:WU06:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
2019-10-14:17:44:14:WU00:FS00:Upload 11.45%
2019-10-14:17:44:19:WU02:FS01:Upload 11.46%
2019-10-14:17:44:20:WU05:FS00:Upload 17.63%
2019-10-14:17:44:24:WU04:FS01:Upload 97.32%
2019-10-14:17:44:24:WU00:FS00:Upload 11.53%
2019-10-14:17:44:27:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:9 gen:53 core:0x21 unit:0x0000003580fccb0a5d98ab5040b3bced
2019-10-14:17:44:27:WU03:FS01:Uploading 64.59MiB to 128.252.203.10
2019-10-14:17:44:27:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:44:27:WU02:FS01:Upload 11.55%
2019-10-14:17:44:28:WU07:FS00:0x21:Completed 320000 out of 1000000 steps (32%)
2019-10-14:17:44:28:WU05:FS00:Upload 17.71%
2019-10-14:17:44:32:WU00:FS00:Upload 11.62%
2019-10-14:17:44:35:WU02:FS01:Upload 11.63%
2019-10-14:17:44:37:WU05:FS00:Upload 17.80%
2019-10-14:17:44:38:WU03:FS01:Upload 0.19%
2019-10-14:17:44:39:WU00:FS00:Upload 11.70%
2019-10-14:17:44:43:WU02:FS01:Upload 11.80%
2019-10-14:17:44:46:WU03:FS01:Upload 0.39%
2019-10-14:17:44:47:WU05:FS00:Upload 18.05%
2019-10-14:17:44:49:WU00:FS00:Upload 11.78%
2019-10-14:17:44:51:WU06:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
2019-10-14:17:44:54:WU06:FS01:0x21:Saving result file logfile_01.txt
2019-10-14:17:44:54:WU06:FS01:0x21:Saving result file checkpointState.xml
2019-10-14:17:44:54:WU06:FS01:0x21:Saving result file checkpt.crc
2019-10-14:17:44:54:WU06:FS01:0x21:Saving result file log.txt
2019-10-14:17:44:54:WU06:FS01:0x21:Saving result file positions.xtc
2019-10-14:17:44:54:WU06:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
2019-10-14:17:44:55:WU01:FS00:Upload 14.05%
2019-10-14:17:44:56:WU05:FS00:Upload 18.14%
2019-10-14:17:44:57:WU00:FS00:Upload 12.03%
2019-10-14:17:45:01:WU02:FS01:Upload 11.89%
2019-10-14:17:45:02:WU03:FS01:Upload 0.48%
2019-10-14:17:45:06:WU05:FS00:Upload 18.22%
2019-10-14:17:45:07:WU01:FS00:Upload 14.14%
2019-10-14:17:45:11:WU00:FS00:Upload 12.12%
2019-10-14:17:45:11:WU03:FS01:Upload 0.58%
2019-10-14:17:45:12:WU02:FS01:Upload 12.06%
2019-10-14:17:45:12:WU05:FS00:Upload 18.31%
2019-10-14:17:45:13:WU07:FS00:0x21:Completed 330000 out of 1000000 steps (33%)
2019-10-14:17:45:15:WU01:FS00:Upload 14.22%
2019-10-14:17:45:18:WU00:FS00:Upload 12.20%
2019-10-14:17:45:21:WU01:FS00:Upload 14.40%
2019-10-14:17:45:21:WU05:FS00:Upload 18.48%
2019-10-14:17:45:24:WU02:FS01:Upload 12.23%
2019-10-14:17:45:25:WU00:FS00:Upload 12.28%
2019-10-14:17:45:31:WU00:FS00:Upload 12.37%
2019-10-14:17:45:32:WU02:FS01:Upload 12.32%
2019-10-14:17:45:34:WU01:FS00:Upload 14.49%
2019-10-14:17:45:39:WU00:FS00:Upload 12.45%
2019-10-14:17:45:40:WU05:FS00:Upload 18.57%
2019-10-14:17:45:46:WU03:FS01:Upload 0.68%
2019-10-14:17:45:47:WU01:FS00:Upload 14.57%
2019-10-14:17:45:50:WU02:FS01:Upload 12.40%
2019-10-14:17:45:51:WU05:FS00:Upload 18.65%
2019-10-14:17:45:52:WU03:FS01:Upload 0.87%
2019-10-14:17:45:54:WU04:FS01:Upload 97.41%
2019-10-14:17:45:54:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:45:54:WU04:FS01:Trying to send results to collection server
2019-10-14:17:45:54:WU04:FS01:Uploading 68.65MiB to 155.247.166.220
2019-10-14:17:45:54:WU04:FS01:Connecting to 155.247.166.220:8080
2019-10-14:17:45:54:WU07:FS00:0x21:Completed 340000 out of 1000000 steps (34%)
2019-10-14:17:45:54:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-14:17:45:54:WARNING:WU06:FS01:Detected clock skew (1 mins 28 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:45:54:WU06:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
2019-10-14:17:45:55:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14277 run:0 clone:33 gen:30 core:0x21 unit:0x0000001e80fccb0a5d98ab6213df40a9
2019-10-14:17:45:55:WU06:FS01:Uploading 71.20MiB to 128.252.203.10
2019-10-14:17:45:55:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:45:58:WU00:FS00:Upload 12.62%
2019-10-14:17:45:58:WU05:FS00:Upload 18.74%
2019-10-14:17:46:00:WU03:FS01:Upload 0.97%
2019-10-14:17:46:01:WU06:FS01:Upload 0.18%
2019-10-14:17:46:09:WU00:FS00:Upload 12.79%
2019-10-14:17:46:11:WU05:FS00:Upload 18.91%
2019-10-14:17:46:11:WU03:FS01:Upload 1.16%
2019-10-14:17:46:14:WU02:FS01:Upload 12.49%
2019-10-14:17:46:16:WU00:FS00:Upload 12.87%
2019-10-14:17:46:17:WU03:FS01:Upload 1.35%
2019-10-14:17:46:20:WU05:FS00:Upload 18.99%
2019-10-14:17:46:21:WU06:FS01:Upload 0.44%
2019-10-14:17:46:25:WU03:FS01:Upload 1.45%
2019-10-14:17:46:27:WU01:FS00:Upload 14.66%
2019-10-14:17:46:27:WU00:FS00:Upload 13.04%
2019-10-14:17:46:29:WU02:FS01:Upload 12.75%
2019-10-14:17:46:32:WU03:FS01:Upload 1.55%
2019-10-14:17:46:34:WU06:FS01:Upload 0.61%
2019-10-14:17:46:38:WU00:FS00:Upload 13.12%
2019-10-14:17:46:39:WU07:FS00:0x21:Completed 350000 out of 1000000 steps (35%)
2019-10-14:17:46:43:WU06:FS01:Upload 0.79%
2019-10-14:17:46:43:WU01:FS00:Upload 14.84%
2019-10-14:17:46:43:WU03:FS01:Upload 1.64%
2019-10-14:17:46:45:WU02:FS01:Upload 12.83%
2019-10-14:17:46:49:WU01:FS00:Upload 14.93%
2019-10-14:17:46:50:WU03:FS01:Upload 1.84%
2019-10-14:17:46:51:WU00:FS00:Upload 13.37%
2019-10-14:17:46:52:WU05:FS00:Upload 19.17%
2019-10-14:17:46:54:WU02:FS01:Upload 13.00%
2019-10-14:17:46:57:WU01:FS00:Upload 15.01%
2019-10-14:17:46:57:WU06:FS01:Upload 0.97%
2019-10-14:17:46:57:WU03:FS01:Upload 1.94%
2019-10-14:17:47:00:WU02:FS01:Upload 13.09%
2019-10-14:17:47:03:WU05:FS00:Upload 19.34%
2019-10-14:17:47:04:WU01:FS00:Upload 15.10%
2019-10-14:17:47:04:WU06:FS01:Upload 1.14%
2019-10-14:17:47:06:WU00:FS00:Upload 13.54%
2019-10-14:17:47:08:WU03:FS01:Upload 2.03%
2019-10-14:17:47:11:WU02:FS01:Upload 13.17%
2019-10-14:17:47:11:WU01:FS00:Upload 15.28%
2019-10-14:17:47:12:WU05:FS00:Upload 19.42%
2019-10-14:17:47:14:WU00:FS00:Upload 13.62%
2019-10-14:17:47:15:WU06:FS01:Upload 1.23%
2019-10-14:17:47:17:WU01:FS00:Upload 15.36%
2019-10-14:17:47:18:WU03:FS01:Upload 2.13%
2019-10-14:17:47:21:WU07:FS00:0x21:Completed 360000 out of 1000000 steps (36%)
2019-10-14:17:47:22:WU06:FS01:Upload 1.32%
2019-10-14:17:47:22:WU00:FS00:Upload 13.70%
2019-10-14:17:47:23:WU05:FS00:Upload 19.51%
2019-10-14:17:47:26:WU02:FS01:Upload 13.34%
2019-10-14:17:47:27:WU01:FS00:Upload 15.45%
2019-10-14:17:47:28:WU00:FS00:Upload 13.79%
2019-10-14:17:47:31:WU03:FS01:Upload 2.23%
2019-10-14:17:47:32:WU05:FS00:Upload 19.59%
2019-10-14:17:47:34:WU06:FS01:Upload 1.49%
2019-10-14:17:47:38:WU03:FS01:Upload 2.42%
2019-10-14:17:47:41:WU05:FS00:Upload 19.76%
2019-10-14:17:47:42:WU02:FS01:Upload 13.52%
2019-10-14:17:47:43:WU00:FS00:Upload 13.87%
2019-10-14:17:47:45:WU01:FS00:Upload 15.54%
2019-10-14:17:47:47:WU03:FS01:Upload 2.52%
2019-10-14:17:47:48:WU02:FS01:Upload 13.60%
2019-10-14:17:47:48:WU06:FS01:Upload 1.67%
2019-10-14:17:47:57:WU00:FS00:Upload 13.96%
2019-10-14:17:47:58:WU02:FS01:Upload 13.69%
2019-10-14:17:48:04:WU02:FS01:Upload 13.86%
2019-10-14:17:48:05:WU00:FS00:Upload 14.04%
2019-10-14:17:48:06:WU05:FS00:Upload 19.94%
2019-10-14:17:48:06:WU07:FS00:0x21:Completed 370000 out of 1000000 steps (37%)
2019-10-14:17:48:11:WU01:FS00:Upload 15.63%
2019-10-14:17:48:11:WU03:FS01:Upload 2.71%
2019-10-14:17:48:16:WU02:FS01:Upload 13.94%
2019-10-14:17:48:16:WU06:FS01:Upload 1.84%
2019-10-14:17:48:17:WU00:FS00:Upload 14.21%
2019-10-14:17:48:18:WU05:FS00:Upload 20.02%
2019-10-14:17:48:20:WU03:FS01:Upload 2.90%
2019-10-14:17:48:22:WU01:FS00:Upload 15.80%
2019-10-14:17:48:25:WU00:FS00:Upload 14.29%
2019-10-14:17:48:26:WU02:FS01:Upload 14.03%
2019-10-14:17:48:28:WU03:FS01:Upload 3.00%
2019-10-14:17:48:30:WU06:FS01:Upload 2.11%
2019-10-14:17:48:33:WU00:FS00:Upload 14.37%
2019-10-14:17:48:34:WU02:FS01:Upload 14.11%
2019-10-14:17:48:39:WU03:FS01:Upload 3.10%
2019-10-14:17:48:41:WU00:FS00:Upload 14.46%
2019-10-14:17:48:45:WU06:FS01:Upload 2.19%
2019-10-14:17:48:45:WU05:FS00:Upload 20.19%
2019-10-14:17:48:46:WU01:FS00:Upload 15.98%
2019-10-14:17:48:47:WU03:FS01:Upload 3.19%
2019-10-14:17:48:48:WU00:FS00:Upload 14.54%
2019-10-14:17:48:48:WU07:FS00:0x21:Completed 380000 out of 1000000 steps (38%)
2019-10-14:17:48:54:WU00:FS00:Upload 14.71%
2019-10-14:17:48:56:WU01:FS00:Upload 16.07%
2019-10-14:17:48:57:WU03:FS01:Upload 3.39%
2019-10-14:17:49:02:WU01:FS00:Upload 16.24%
2019-10-14:17:49:10:WU02:FS01:Upload 14.29%
2019-10-14:17:49:10:WU01:FS00:Upload 16.33%
2019-10-14:17:49:12:WU03:FS01:Upload 3.48%
2019-10-14:17:49:13:WU06:FS01:Upload 2.28%
2019-10-14:17:49:16:WU02:FS01:Upload 14.37%
2019-10-14:17:49:18:WU01:FS00:Upload 16.42%
2019-10-14:17:49:23:WU03:FS01:Upload 3.58%
2019-10-14:17:49:32:WU02:FS01:Upload 14.46%
2019-10-14:17:49:33:WU03:FS01:Upload 3.68%
2019-10-14:17:49:33:WU07:FS00:0x21:Completed 390000 out of 1000000 steps (39%)
2019-10-14:17:49:34:WU00:FS00:Upload 14.79%
2019-10-14:17:49:39:WU06:FS01:Upload 2.37%
2019-10-14:17:49:43:WU00:FS00:Upload 14.87%
2019-10-14:17:49:46:WU01:FS00:Upload 16.68%
2019-10-14:17:49:46:WU03:FS01:Upload 3.87%
2019-10-14:17:49:58:WU06:FS01:Upload 2.63%
2019-10-14:17:49:59:WU00:FS00:Upload 15.04%
2019-10-14:17:50:02:WU05:FS00:Upload 20.28%
2019-10-14:17:50:02:WARNING:WU05:FS00:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:50:02:WU05:FS00:Trying to send results to collection server
2019-10-14:17:50:02:WU05:FS00:Uploading 73.05MiB to 155.247.166.220
2019-10-14:17:50:02:WU05:FS00:Connecting to 155.247.166.220:8080
2019-10-14:17:50:02:ERROR:WU05:FS00:Exception: Transfer failed
2019-10-14:17:50:02:WARNING:WU07:FS00:Detected clock skew (5 mins 36 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:50:03:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14280 run:0 clone:7 gen:18 core:0x21 unit:0x0000001280fccb0a5d9e115413dafbe9
2019-10-14:17:50:03:WU04:FS01:Uploading 68.65MiB to 128.252.203.10
2019-10-14:17:50:03:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:50:04:WU06:FS01:Upload 2.72%
2019-10-14:17:50:15:WU07:FS00:0x21:Completed 400000 out of 1000000 steps (40%)
2019-10-14:17:50:45:WU04:FS01:Upload 0.18%
2019-10-14:17:50:47:WU00:FS00:Upload 15.21%
2019-10-14:17:50:47:WU02:FS01:Upload 14.54%

Posted: **Mon Oct 14, 2019 7:00 pm**

Different Linux PC with the largest clock skew values at 20+ minutes:

Code: Select all

2019-10-14:17:59:26:WU08:FS01:0x21:Completed 340000 out of 1000000 steps (34%)
2019-10-14:17:59:32:WU05:FS00:Upload 46.03%
2019-10-14:17:59:32:WARNING:WU05:FS00:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:59:32:WU05:FS00:Trying to send results to collection server
2019-10-14:17:59:32:WU05:FS00:Uploading 77.67MiB to 155.247.166.220
2019-10-14:17:59:32:WU05:FS00:Connecting to 155.247.166.220:8080
2019-10-14:17:59:32:ERROR:WU05:FS00:Exception: Transfer failed
2019-10-14:17:59:32:WARNING:WU07:FS00:Detected clock skew (20 mins 59 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:17:59:33:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14280 run:0 clone:0 gen:26 core:0x21 unit:0x0000001a80fccb0a5d9e115325fe70c2
2019-10-14:17:59:33:WU03:FS01:Uploading 68.54MiB to 128.252.203.10
2019-10-14:17:59:33:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:59:34:WU01:FS00:Upload 14.69%
2019-10-14:17:59:36:WU06:FS01:Upload 8.75%
2019-10-14:17:59:38:WU00:FS01:Upload 20.94%
2019-10-14:17:59:38:WU07:FS00:0x21:Completed 960000 out of 1000000 steps (96%)
2019-10-14:17:59:40:WU01:FS00:Upload 14.89%
2019-10-14:17:59:45:WU02:FS01:Upload 1.94%
2019-10-14:17:59:45:WU06:FS01:Upload 8.84%
2019-10-14:17:59:48:WU00:FS01:Upload 21.02%
2019-10-14:17:59:52:WU03:FS01:Upload 0.36%
2019-10-14:17:59:53:WU06:FS01:Upload 8.93%
2019-10-14:17:59:55:WU02:FS01:Upload 2.24%
2019-10-14:17:59:56:WU01:FS00:Upload 14.98%
2019-10-14:17:59:58:WU03:FS01:Upload 0.46%
2019-10-14:18:00:00:WU00:FS01:Upload 21.11%
2019-10-14:18:00:02:WU01:FS00:Upload 15.08%
2019-10-14:18:00:03:WU08:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
2019-10-14:18:00:04:WU06:FS01:Upload 9.21%
2019-10-14:18:00:05:WU03:FS01:Upload 0.55%
2019-10-14:18:00:08:WU04:FS00:Upload 17.49%
2019-10-14:18:00:10:WU00:FS01:Upload 21.27%
2019-10-14:18:00:10:WU02:FS01:Upload 2.33%
2019-10-14:18:00:17:WU06:FS01:Upload 9.30%
2019-10-14:18:00:20:WU02:FS01:Upload 2.53%
2019-10-14:18:00:21:WU01:FS00:Upload 15.18%
2019-10-14:18:00:24:WU07:FS00:0x21:Completed 970000 out of 1000000 steps (97%)
2019-10-14:18:00:24:WU06:FS01:Upload 9.39%
2019-10-14:18:00:26:WU00:FS01:Upload 21.43%
2019-10-14:18:00:28:WU01:FS00:Upload 15.37%
2019-10-14:18:00:29:WU03:FS01:Upload 0.64%
2019-10-14:18:00:31:WU02:FS01:Upload 2.82%
2019-10-14:18:00:34:WU00:FS01:Upload 21.51%
2019-10-14:18:00:35:WU06:FS01:Upload 9.57%
2019-10-14:18:00:37:WU03:FS01:Upload 0.82%
2019-10-14:18:00:37:WU08:FS01:0x21:Completed 360000 out of 1000000 steps (36%)
2019-10-14:18:00:42:WU00:FS01:Upload 21.67%
2019-10-14:18:00:42:WU06:FS01:Upload 9.66%
2019-10-14:18:00:42:WU02:FS01:Upload 2.92%
2019-10-14:18:00:46:WU01:FS00:Upload 15.47%
2019-10-14:18:00:50:WU00:FS01:Upload 21.75%
2019-10-14:18:00:51:WU02:FS01:Upload 3.01%
2019-10-14:18:00:56:WU03:FS01:Upload 0.91%
2019-10-14:18:00:56:WU01:FS00:Upload 15.56%
2019-10-14:18:00:57:WU00:FS01:Upload 21.83%
2019-10-14:18:01:02:WU03:FS01:Upload 1.00%
2019-10-14:18:01:03:WU06:FS01:Upload 9.75%
2019-10-14:18:01:03:WU02:FS01:Upload 3.21%
2019-10-14:18:01:05:WU07:FS00:0x21:Completed 980000 out of 1000000 steps (98%)
2019-10-14:18:01:06:WU01:FS00:Upload 15.66%
2019-10-14:18:01:09:WU02:FS01:Upload 3.30%
2019-10-14:18:01:11:WU06:FS01:Upload 9.84%
2019-10-14:18:01:12:WU01:FS00:Upload 15.85%
2019-10-14:18:01:14:WU08:FS01:0x21:Completed 370000 out of 1000000 steps (37%)
2019-10-14:18:01:16:WU02:FS01:Upload 3.40%
2019-10-14:18:01:16:WU03:FS01:Upload 1.09%
2019-10-14:18:01:17:WU06:FS01:Upload 10.03%
2019-10-14:18:01:23:WU02:FS01:Upload 3.50%
2019-10-14:18:01:25:WU00:FS01:Upload 21.91%
2019-10-14:18:01:25:WU01:FS00:Upload 15.95%
2019-10-14:18:01:26:WU03:FS01:Upload 1.28%
2019-10-14:18:01:30:WU06:FS01:Upload 10.21%
2019-10-14:18:01:31:WU00:FS01:Upload 21.99%
2019-10-14:18:01:32:WU01:FS00:Upload 16.05%
2019-10-14:18:01:32:WU03:FS01:Upload 1.37%
2019-10-14:18:01:35:WU04:FS00:Upload 17.58%
2019-10-14:18:01:35:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
2019-10-14:18:01:35:WU04:FS00:Trying to send results to collection server
2019-10-14:18:01:35:WU04:FS00:Uploading 68.60MiB to 155.247.166.220
2019-10-14:18:01:35:WU04:FS00:Connecting to 155.247.166.220:8080
2019-10-14:18:01:35:WU02:FS01:Upload 3.69%
2019-10-14:18:01:35:ERROR:WU04:FS00:Exception: Transfer failed
2019-10-14:18:01:35:WU05:FS00:Sending unit results: id:05 state:SEND error:NO_ERROR project:14278 run:0 clone:59 gen:7 core:0x21 unit:0x0000000780fccb0a5d98ab68068c6f16
2019-10-14:18:01:35:WU05:FS00:Uploading 77.67MiB to 128.252.203.10
2019-10-14:18:01:35:WU05:FS00:Connecting to 128.252.203.10:8080
2019-10-14:18:01:36:WU06:FS01:Upload 10.30%
2019-10-14:18:01:39:WU03:FS01:Upload 1.46%
2019-10-14:18:01:40:WU01:FS00:Upload 16.14%
2019-10-14:18:01:44:WU02:FS01:Upload 3.79%
2019-10-14:18:01:45:WU00:FS01:Upload 22.15%
2019-10-14:18:01:45:WU03:FS01:Upload 1.55%
2019-10-14:18:01:48:WU06:FS01:Upload 10.48%
2019-10-14:18:01:48:WU08:FS01:0x21:Completed 380000 out of 1000000 steps (38%)
2019-10-14:18:01:50:WU07:FS00:0x21:Completed 990000 out of 1000000 steps (99%)
2019-10-14:18:01:51:WU01:FS00:Upload 16.34%
2019-10-14:18:01:51:WU00:FS01:Upload 22.23%
2019-10-14:18:01:56:WU06:FS01:Upload 10.57%
2019-10-14:18:01:58:WU01:FS00:Upload 16.43%
2019-10-14:18:01:58:WU03:FS01:Upload 1.82%
2019-10-14:18:02:00:WU00:FS01:Upload 22.31%
2019-10-14:18:02:02:WU06:FS01:Upload 10.66%
2019-10-14:18:02:02:WU05:FS00:Upload 0.32%
2019-10-14:18:02:03:WU02:FS01:Upload 3.89%
2019-10-14:18:02:06:WU03:FS01:Upload 1.91%
2019-10-14:18:02:09:WU01:FS00:Upload 16.53%
2019-10-14:18:02:10:WU02:FS01:Upload 3.99%
2019-10-14:18:02:12:WU06:FS01:Upload 10.76%
2019-10-14:18:02:12:WU03:FS01:Upload 2.01%
2019-10-14:18:02:13:WU00:FS01:Upload 22.39%
2019-10-14:18:02:16:WU02:FS01:Upload 4.18%
2019-10-14:18:02:18:WU03:FS01:Upload 2.19%
2019-10-14:18:02:20:WU01:FS00:Upload 16.82%
2019-10-14:18:02:23:WU02:FS01:Upload 4.28%
2019-10-14:18:02:23:WU00:FS01:Upload 22.56%
2019-10-14:18:02:25:WU03:FS01:Upload 2.28%
2019-10-14:18:02:26:WU08:FS01:0x21:Completed 390000 out of 1000000 steps (39%)
2019-10-14:18:02:31:WU03:FS01:Upload 2.37%
2019-10-14:18:02:32:WU06:FS01:Upload 10.85%
2019-10-14:18:02:32:WU07:FS00:0x21:Completed 1000000 out of 1000000 steps (100%)
2019-10-14:18:02:33:WU01:FS00:Upload 16.92%
2019-10-14:18:02:35:WU02:FS01:Upload 4.37%
2019-10-14:18:02:36:WU00:FS01:Upload 22.64%
2019-10-14:18:02:36:WU07:FS00:0x21:Saving result file logfile_01.txt
2019-10-14:18:02:36:WU07:FS00:0x21:Saving result file checkpointState.xml
2019-10-14:18:02:36:WU07:FS00:0x21:Saving result file checkpt.crc
2019-10-14:18:02:36:WU07:FS00:0x21:Saving result file log.txt
2019-10-14:18:02:36:WU07:FS00:0x21:Saving result file positions.xtc
2019-10-14:18:02:36:WU07:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
2019-10-14:18:02:38:WU06:FS01:Upload 11.03%
2019-10-14:18:02:39:WU01:FS00:Upload 17.01%
2019-10-14:18:02:39:WU03:FS01:Upload 2.46%
2019-10-14:18:02:42:WU00:FS01:Upload 22.72%
2019-10-14:18:02:42:WU05:FS00:Upload 0.40%
2019-10-14:18:02:45:WU03:FS01:Upload 2.64%
2019-10-14:18:02:48:WU05:FS00:Upload 0.48%
2019-10-14:18:02:49:WU01:FS00:Upload 17.11%
2019-10-14:18:02:55:WU05:FS00:Upload 0.56%
2019-10-14:18:02:56:WU06:FS01:Upload 11.21%
2019-10-14:18:02:57:WU02:FS01:Upload 4.47%
2019-10-14:18:02:57:WU01:FS00:Upload 17.30%
2019-10-14:18:03:00:WU08:FS01:0x21:Completed 400000 out of 1000000 steps (40%)
2019-10-14:18:03:02:WU00:FS01:Upload 22.80%
2019-10-14:18:03:06:WU05:FS00:Upload 0.72%
2019-10-14:18:03:10:WU01:FS00:Upload 17.50%
2019-10-14:18:03:12:WU05:FS00:Upload 0.80%
2019-10-14:18:03:13:WU00:FS01:Upload 22.96%
2019-10-14:18:03:19:WU00:FS01:Upload 23.04%
2019-10-14:18:03:20:WU06:FS01:Upload 11.30%
2019-10-14:18:03:21:WU05:FS00:Upload 0.89%
2019-10-14:18:03:25:WU01:FS00:Upload 17.79%
2019-10-14:18:03:25:WU00:FS01:Upload 23.12%
2019-10-14:18:03:26:WU02:FS01:Upload 4.67%
2019-10-14:18:03:26:WU06:FS01:Upload 11.49%
2019-10-14:18:03:32:WU05:FS00:Upload 0.97%
2019-10-14:18:03:33:WU02:FS01:Upload 4.76%
2019-10-14:18:03:33:WU06:FS01:Upload 11.58%
2019-10-14:18:03:35:WU00:FS01:Upload 23.20%
2019-10-14:18:03:35:WU01:FS00:Upload 17.88%
2019-10-14:18:03:37:WU08:FS01:0x21:Completed 410000 out of 1000000 steps (41%)
2019-10-14:18:03:41:WU00:FS01:Upload 23.36%
2019-10-14:18:03:43:WU02:FS01:Upload 4.96%
2019-10-14:18:03:45:WU06:FS01:Upload 11.67%
2019-10-14:18:03:46:WU01:FS00:Upload 17.98%
2019-10-14:18:03:50:WU02:FS01:Upload 5.15%
2019-10-14:18:03:52:WU00:FS01:Upload 23.44%
2019-10-14:18:03:53:WU06:FS01:Upload 11.76%
2019-10-14:18:03:57:WU01:FS00:Upload 18.17%
2019-10-14:18:03:57:WU05:FS00:Upload 1.13%
2019-10-14:18:03:58:WU02:FS01:Upload 5.25%
2019-10-14:18:04:03:WU01:FS00:Upload 18.27%
2019-10-14:18:04:05:WU05:FS00:Upload 1.29%
2019-10-14:18:04:06:WU00:FS01:Upload 23.52%
2019-10-14:18:04:06:WU06:FS01:Upload 11.94%
2019-10-14:18:04:06:WU02:FS01:Upload 5.35%
2019-10-14:18:04:11:WU08:FS01:0x21:Completed 420000 out of 1000000 steps (42%)
2019-10-14:18:04:13:WU01:FS00:Upload 18.37%
2019-10-14:18:04:14:WU02:FS01:Upload 5.44%
2019-10-14:18:04:15:WU05:FS00:Upload 1.37%
2019-10-14:18:04:20:WU00:FS01:Upload 23.60%
2019-10-14:18:04:22:WU06:FS01:Upload 12.03%
2019-10-14:18:04:23:WU01:FS00:Upload 18.46%
2019-10-14:18:04:28:WU00:FS01:Upload 23.76%
2019-10-14:18:04:31:WU05:FS00:Upload 1.53%
2019-10-14:18:04:31:WU06:FS01:Upload 12.12%
2019-10-14:18:04:38:WU02:FS01:Upload 5.64%
2019-10-14:18:04:39:WU06:FS01:Upload 12.21%
2019-10-14:18:04:39:WU05:FS00:Upload 1.61%
2019-10-14:18:04:42:WU00:FS01:Upload 23.84%
2019-10-14:18:04:45:WU05:FS00:Upload 1.69%
2019-10-14:18:04:46:WU01:FS00:Upload 18.75%
2019-10-14:18:04:48:WU08:FS01:0x21:Completed 430000 out of 1000000 steps (43%)
2019-10-14:18:04:54:WU06:FS01:Upload 12.40%
2019-10-14:18:04:55:WU00:FS01:Upload 23.92%
2019-10-14:18:04:58:WU01:FS00:Upload 18.95%
2019-10-14:18:04:58:WU05:FS00:Upload 1.85%
2019-10-14:18:05:04:WU02:FS01:Upload 5.73%
2019-10-14:18:05:07:WU05:FS00:Upload 2.01%
2019-10-14:18:05:11:WU01:FS00:Upload 19.04%
2019-10-14:18:05:12:WU02:FS01:Upload 5.83%
2019-10-14:18:05:13:WU00:FS01:Upload 24.09%
2019-10-14:18:05:14:WU03:FS01:Upload 2.83%
2019-10-14:18:05:14:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:18:05:14:WU03:FS01:Trying to send results to collection server
2019-10-14:18:05:14:WU03:FS01:Uploading 68.54MiB to 155.247.166.219
2019-10-14:18:05:14:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-14:18:05:14:WU06:FS01:Upload 12.58%
2019-10-14:18:05:14:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-14:18:05:14:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:14280 run:0 clone:21 gen:11 core:0x21 unit:0x0000000b80fccb0a5d9e1156ca2f2352
2019-10-14:18:05:14:WU04:FS00:Uploading 68.60MiB to 128.252.203.10
2019-10-14:18:05:14:WU04:FS00:Connecting to 128.252.203.10:8080
2019-10-14:18:05:18:WU01:FS00:Upload 19.24%
2019-10-14:18:05:21:WU02:FS01:Upload 5.93%
2019-10-14:18:05:22:WU04:FS00:Upload 0.27%
2019-10-14:18:05:23:WU08:FS01:0x21:Completed 440000 out of 1000000 steps (44%)
2019-10-14:18:05:26:WU05:FS00:Upload 2.09%
2019-10-14:18:05:27:WU01:FS00:Upload 19.33%
2019-10-14:18:05:32:WU02:FS01:Upload 6.12%
2019-10-14:18:05:34:WU04:FS00:Upload 0.55%
2019-10-14:18:05:36:WU05:FS00:Upload 2.17%
2019-10-14:18:05:38:WU02:FS01:Upload 6.22%
2019-10-14:18:05:39:WU00:FS01:Upload 24.25%
2019-10-14:18:05:40:WU01:FS00:Upload 19.43%
2019-10-14:18:05:44:WU04:FS00:Upload 0.73%
2019-10-14:18:05:45:WU06:FS01:Upload 12.67%
2019-10-14:18:05:48:WU05:FS00:Upload 2.33%
2019-10-14:18:05:52:WU00:FS01:Upload 24.33%
2019-10-14:18:05:53:WU04:FS00:Upload 0.91%
2019-10-14:18:05:54:WU02:FS01:Upload 6.32%
2019-10-14:18:05:54:WU05:FS00:Upload 2.41%
2019-10-14:18:05:56:WU01:FS00:Upload 19.72%
2019-10-14:18:05:59:WU00:FS01:Upload 24.41%
2019-10-14:18:06:00:WU08:FS01:0x21:Completed 450000 out of 1000000 steps (45%)
2019-10-14:18:06:00:WU05:FS00:Upload 2.49%
2019-10-14:18:06:00:WU02:FS01:Upload 6.42%
2019-10-14:18:06:01:WU06:FS01:Upload 12.94%
2019-10-14:18:06:02:WU01:FS00:Upload 19.82%
2019-10-14:18:06:07:WU04:FS00:Upload 1.18%
2019-10-14:18:06:10:WU01:FS00:Upload 19.91%
2019-10-14:18:06:18:WU04:FS00:Upload 1.46%
2019-10-14:18:06:18:WU05:FS00:Upload 2.57%
2019-10-14:18:06:23:WU02:FS01:Upload 6.71%
2019-10-14:18:06:24:WU05:FS00:Upload 2.66%
2019-10-14:18:06:31:WU06:FS01:Upload 13.03%
2019-10-14:18:06:34:WU08:FS01:0x21:Completed 460000 out of 1000000 steps (46%)
2019-10-14:18:07:01:WU05:FS00:Upload 2.82%
2019-10-14:18:07:02:WU00:FS01:Upload 24.49%
2019-10-14:18:07:03:WU04:FS00:Upload 1.64%
2019-10-14:18:07:09:WU01:FS00:Upload 20.01%
2019-10-14:18:07:09:WU00:FS01:Upload 24.65%
2019-10-14:18:07:10:WU02:FS01:Upload 6.80%
2019-10-14:18:07:11:WU08:FS01:0x21:Completed 470000 out of 1000000 steps (47%)
2019-10-14:18:07:14:WU05:FS00:Upload 3.06%
2019-10-14:18:07:15:WU06:FS01:Upload 13.13%
2019-10-14:18:07:15:WU00:FS01:Upload 24.73%
2019-10-14:18:07:16:WU04:FS00:Upload 1.73%
2019-10-14:18:07:21:WU05:FS00:Upload 3.22%
2019-10-14:18:07:21:WU00:FS01:Upload 24.81%
2019-10-14:18:07:21:WU01:FS00:Upload 20.30%
2019-10-14:18:07:25:WU06:FS01:Upload 13.31%
2019-10-14:18:07:28:WU05:FS00:Upload 3.30%
2019-10-14:18:07:28:WU04:FS00:Upload 2.00%
2019-10-14:18:07:34:WU02:FS01:Upload 6.90%
2019-10-14:18:07:36:WU05:FS00:Upload 3.38%
2019-10-14:18:07:41:WU04:FS00:Upload 2.10%
2019-10-14:18:07:42:WU05:FS00:Upload 3.46%
2019-10-14:18:07:43:WU01:FS00:Upload 20.50%
2019-10-14:18:07:45:WU08:FS01:0x21:Completed 480000 out of 1000000 steps (48%)
2019-10-14:18:07:45:WU00:FS01:Upload 24.89%
2019-10-14:18:07:46:WU02:FS01:Upload 7.19%
2019-10-14:18:07:46:WU06:FS01:Upload 13.40%
2019-10-14:18:07:50:WU04:FS00:Upload 2.28%
2019-10-14:18:07:52:WU06:FS01:Upload 13.49%
2019-10-14:18:07:53:WU02:FS01:Upload 7.29%
2019-10-14:18:07:53:WU00:FS01:Upload 25.05%
2019-10-14:18:07:54:WU05:FS00:Upload 3.70%
2019-10-14:18:08:00:WU06:FS01:Upload 13.58%
2019-10-14:18:08:01:WU04:FS00:Upload 2.46%
2019-10-14:18:08:02:WU00:FS01:Upload 25.13%
2019-10-14:18:08:07:WU02:FS01:Upload 7.39%
2019-10-14:18:08:08:WU06:FS01:Upload 13.76%
2019-10-14:18:08:08:WU01:FS00:Upload 20.79%
2019-10-14:18:08:09:WU05:FS00:Upload 3.86%
2019-10-14:18:08:09:WU04:FS00:Upload 2.55%
2019-10-14:18:08:13:WU02:FS01:Upload 7.48%
2019-10-14:18:08:15:WU06:FS01:Upload 13.85%
2019-10-14:18:08:17:WU01:FS00:Upload 20.88%
2019-10-14:18:08:17:WU04:FS00:Upload 2.64%
2019-10-14:18:08:18:WU05:FS00:Upload 4.02%
2019-10-14:18:08:21:WU02:FS01:Upload 7.68%
2019-10-14:18:08:22:WU08:FS01:0x21:Completed 490000 out of 1000000 steps (49%)
2019-10-14:18:08:23:WU00:FS01:Upload 25.21%
2019-10-14:18:08:23:WU06:FS01:Upload 13.95%
2019-10-14:18:08:28:WU04:FS00:Upload 2.92%
2019-10-14:18:08:30:WU01:FS00:Upload 20.98%
2019-10-14:18:08:30:WU06:FS01:Upload 14.04%
2019-10-14:18:08:31:WU05:FS00:Upload 4.18%
2019-10-14:18:08:32:WU00:FS01:Upload 25.29%
2019-10-14:18:08:36:WU06:FS01:Upload 14.22%
2019-10-14:18:08:36:WU01:FS00:Upload 21.17%
2019-10-14:18:08:37:WU02:FS01:Upload 7.78%
2019-10-14:18:08:40:WU04:FS00:Upload 3.10%
2019-10-14:18:08:44:WU00:FS01:Upload 25.37%
2019-10-14:18:08:47:WU01:FS00:Upload 21.27%
2019-10-14:18:08:48:WU02:FS01:Upload 7.87%
2019-10-14:18:08:48:WU06:FS01:Upload 14.31%
2019-10-14:18:08:49:WU04:FS00:Upload 3.19%
2019-10-14:18:08:55:WU05:FS00:Upload 4.26%
2019-10-14:18:08:55:WU06:FS01:Upload 14.40%
2019-10-14:18:08:56:WU04:FS00:Upload 3.37%
2019-10-14:18:08:56:WU08:FS01:0x21:Completed 500000 out of 1000000 steps (50%)
2019-10-14:18:08:59:WU01:FS00:Upload 21.37%
2019-10-14:18:08:59:WU00:FS01:Upload 25.62%
2019-10-14:18:09:02:WU04:FS00:Upload 3.46%
2019-10-14:18:09:02:WU02:FS01:Upload 8.16%
2019-10-14:18:09:03:WU05:FS00:Upload 4.35%
2019-10-14:18:09:06:WU00:FS01:Upload 25.70%
2019-10-14:18:09:09:WU06:FS01:Upload 14.49%
2019-10-14:18:09:13:WU01:FS00:Upload 21.46%
2019-10-14:18:09:14:WU00:FS01:Upload 25.78%
2019-10-14:18:09:15:WU04:FS00:Upload 3.55%
2019-10-14:18:09:16:WU02:FS01:Upload 8.26%
2019-10-14:18:09:17:WU05:FS00:Upload 4.59%
2019-10-14:18:09:21:WU04:FS00:Upload 3.64%
2019-10-14:18:09:21:WU01:FS00:Upload 21.56%
2019-10-14:18:09:25:WU06:FS01:Upload 14.77%
2019-10-14:18:09:25:WU05:FS00:Upload 4.67%
2019-10-14:18:09:27:WU02:FS01:Upload 8.36%
2019-10-14:18:09:30:WU04:FS00:Upload 3.92%
2019-10-14:18:09:30:WU01:FS00:Upload 21.75%
2019-10-14:18:09:31:WU06:FS01:Upload 14.86%
2019-10-14:18:09:33:WU05:FS00:Upload 4.75%
2019-10-14:18:09:34:WU08:FS01:0x21:Completed 510000 out of 1000000 steps (51%)
2019-10-14:18:09:35:WU02:FS01:Upload 8.46%
2019-10-14:18:09:38:WU04:FS00:Upload 4.01%
2019-10-14:18:09:39:WU05:FS00:Upload 4.91%
2019-10-14:18:09:40:WU06:FS01:Upload 14.95%
2019-10-14:18:09:40:WU01:FS00:Upload 21.85%
2019-10-14:18:09:45:WU04:FS00:Upload 4.10%
2019-10-14:18:09:45:WU00:FS01:Upload 26.02%
2019-10-14:18:09:47:WU02:FS01:Upload 8.75%
2019-10-14:18:09:51:WU04:FS00:Upload 4.28%
2019-10-14:18:09:51:WU01:FS00:Upload 21.95%
2019-10-14:18:09:51:WU06:FS01:Upload 15.22%
2019-10-14:18:09:52:WU05:FS00:Upload 4.99%
2019-10-14:18:09:53:WU00:FS01:Upload 26.10%
2019-10-14:18:09:57:WU04:FS00:Upload 4.37%
2019-10-14:18:09:57:WU02:FS01:Upload 8.85%
2019-10-14:18:09:58:WU06:FS01:Upload 15.31%
2019-10-14:18:10:01:WU05:FS00:Upload 5.07%
2019-10-14:18:10:03:WU00:FS01:Upload 26.34%
2019-10-14:18:10:05:WU02:FS01:Upload 8.94%
2019-10-14:18:10:05:WU01:FS00:Upload 22.04%
2019-10-14:18:10:08:WU06:FS01:Upload 15.40%
2019-10-14:18:10:08:WU08:FS01:0x21:Completed 520000 out of 1000000 steps (52%)
2019-10-14:18:10:09:WU05:FS00:Upload 5.15%
2019-10-14:18:10:13:WU00:FS01:Upload 26.42%
2019-10-14:18:10:15:WU04:FS00:Upload 4.56%
2019-10-14:18:10:15:WU05:FS00:Upload 5.31%
2019-10-14:18:10:16:WU06:FS01:Upload 15.59%
2019-10-14:18:10:19:WU00:FS01:Upload 26.50%
2019-10-14:18:10:19:WU01:FS00:Upload 22.24%
2019-10-14:18:10:22:WU06:FS01:Upload 15.68%
2019-10-14:18:10:25:WU04:FS00:Upload 4.83%
2019-10-14:18:10:26:WU00:FS01:Upload 26.58%
2019-10-14:18:10:26:WU05:FS00:Upload 5.39%
2019-10-14:18:10:28:WU01:FS00:Upload 22.33%
2019-10-14:18:10:32:WU06:FS01:Upload 15.77%
2019-10-14:18:10:33:WU02:FS01:Upload 9.04%
2019-10-14:18:10:33:WU05:FS00:Upload 5.47%
2019-10-14:18:10:35:WU04:FS00:Upload 5.01%
2019-10-14:18:10:35:WU00:FS01:Upload 26.74%
2019-10-14:18:10:41:WU00:FS01:Upload 26.82%
2019-10-14:18:10:43:WU06:FS01:Upload 15.86%
2019-10-14:18:10:44:WU04:FS00:Upload 5.19%
2019-10-14:18:10:45:WU08:FS01:0x21:Completed 530000 out of 1000000 steps (53%)
2019-10-14:18:10:47:WU02:FS01:Upload 9.23%
2019-10-14:18:10:50:WU00:FS01:Upload 26.90%
2019-10-14:18:10:51:WU06:FS01:Upload 16.04%
2019-10-14:18:10:52:WU04:FS00:Upload 5.28%
2019-10-14:18:10:52:WU05:FS00:Upload 5.71%
2019-10-14:18:10:53:WU02:FS01:Upload 9.33%
2019-10-14:18:10:58:WU06:FS01:Upload 16.13%
2019-10-14:18:10:59:WU01:FS00:Upload 22.43%
2019-10-14:18:11:00:WU05:FS00:Upload 5.87%
2019-10-14:18:11:03:WU04:FS00:Upload 5.47%
2019-10-14:18:11:05:WU01:FS00:Upload 22.53%
2019-10-14:18:11:07:WU05:FS00:Upload 5.95%
2019-10-14:18:11:10:WU00:FS01:Upload 27.15%
2019-10-14:18:11:11:WU06:FS01:Upload 16.22%
2019-10-14:18:11:13:WU04:FS00:Upload 5.74%
2019-10-14:18:11:16:WU05:FS00:Upload 6.12%
2019-10-14:18:11:19:WU06:FS01:Upload 16.32%
2019-10-14:18:11:19:WU08:FS01:0x21:Completed 540000 out of 1000000 steps (54%)
2019-10-14:18:11:19:WU01:FS00:Upload 22.72%
2019-10-14:18:11:20:WU02:FS01:Upload 9.43%
2019-10-14:18:11:20:WU00:FS01:Upload 27.23%
2019-10-14:18:11:26:WU04:FS00:Upload 5.92%
2019-10-14:18:11:26:WU02:FS01:Upload 9.53%
2019-10-14:18:11:29:WU05:FS00:Upload 6.20%
2019-10-14:18:11:29:WU01:FS00:Upload 22.82%
2019-10-14:18:11:33:WU02:FS01:Upload 9.72%
2019-10-14:18:11:35:WU06:FS01:Upload 16.50%
2019-10-14:18:11:36:WU00:FS01:Upload 27.31%
2019-10-14:18:11:38:WU05:FS00:Upload 6.28%
2019-10-14:18:11:38:WU01:FS00:Upload 22.91%
2019-10-14:18:11:39:WU02:FS01:Upload 9.82%
2019-10-14:18:11:40:WU04:FS00:Upload 6.20%
2019-10-14:18:11:44:WU06:FS01:Upload 16.68%
2019-10-14:18:11:45:WU05:FS00:Upload 6.52%
2019-10-14:18:11:46:WU02:FS01:Upload 9.91%
2019-10-14:18:11:48:WU04:FS00:Upload 6.47%
2019-10-14:18:11:49:WU01:FS00:Upload 23.20%
2019-10-14:18:11:53:WU05:FS00:Upload 6.60%
2019-10-14:18:11:53:WU06:FS01:Upload 16.77%
2019-10-14:18:11:56:WU02:FS01:Upload 10.21%
2019-10-14:18:11:56:WU08:FS01:0x21:Completed 550000 out of 1000000 steps (55%)
2019-10-14:18:11:57:WU04:FS00:Upload 6.65%
2019-10-14:18:12:01:WU01:FS00:Upload 23.30%
2019-10-14:18:12:01:WU05:FS00:Upload 6.76%
2019-10-14:18:12:04:WU04:FS00:Upload 6.74%
2019-10-14:18:12:04:WU02:FS01:Upload 10.30%
2019-10-14:18:12:05:WU06:FS01:Upload 16.86%
2019-10-14:18:12:05:WU00:FS01:Upload 27.39%
2019-10-14:18:12:08:WU01:FS00:Upload 23.40%
2019-10-14:18:12:11:WU02:FS01:Upload 10.40%
2019-10-14:18:12:11:WU05:FS00:Upload 6.84%
2019-10-14:18:12:11:WU06:FS01:Upload 17.05%
2019-10-14:18:12:12:WU04:FS00:Upload 7.02%
2019-10-14:18:12:14:WU00:FS01:Upload 27.55%
2019-10-14:18:12:16:WU01:FS00:Upload 23.49%
2019-10-14:18:12:17:WU05:FS00:Upload 7.00%
2019-10-14:18:12:18:WU04:FS00:Upload 7.20%
2019-10-14:18:12:19:WU02:FS01:Upload 10.50%
2019-10-14:18:12:24:WU00:FS01:Upload 27.63%
2019-10-14:18:12:27:WU01:FS00:Upload 23.69%
2019-10-14:18:12:27:WU02:FS01:Upload 10.69%
2019-10-14:18:12:28:WU06:FS01:Upload 17.23%
2019-10-14:18:12:30:WU05:FS00:Upload 7.16%
2019-10-14:18:12:31:WU08:FS01:0x21:Completed 560000 out of 1000000 steps (56%)
2019-10-14:18:12:32:WU04:FS00:Upload 7.47%
2019-10-14:18:12:34:WU02:FS01:Upload 10.79%
2019-10-14:18:12:34:WU01:FS00:Upload 23.78%
2019-10-14:18:12:35:WU06:FS01:Upload 17.41%
2019-10-14:18:12:41:WU04:FS00:Upload 7.56%
2019-10-14:18:12:41:WU06:FS01:Upload 17.50%
2019-10-14:18:12:41:WU01:FS00:Upload 23.88%
2019-10-14:18:12:43:WU05:FS00:Upload 7.24%
2019-10-14:18:12:43:WU02:FS01:Upload 10.98%
2019-10-14:18:12:44:WU00:FS01:Upload 27.79%
2019-10-14:18:12:48:WU06:FS01:Upload 17.59%
2019-10-14:18:12:50:WU05:FS00:Upload 7.40%
2019-10-14:18:12:50:WU02:FS01:Upload 11.18%
2019-10-14:18:12:50:WU01:FS00:Upload 23.98%
2019-10-14:18:12:52:WU00:FS01:Upload 27.95%
2019-10-14:18:12:54:WU04:FS00:Upload 7.74%
2019-10-14:18:12:58:WU02:FS01:Upload 11.28%
2019-10-14:18:12:59:WU01:FS00:Upload 24.17%
2019-10-14:18:13:02:WU04:FS00:Upload 8.02%
2019-10-14:18:13:02:WU06:FS01:Upload 17.87%
2019-10-14:18:13:03:WU00:FS01:Upload 28.03%
2019-10-14:18:13:04:WU05:FS00:Upload 7.48%
2019-10-14:18:13:07:WU01:FS00:Upload 24.27%
2019-10-14:18:13:08:WU08:FS01:0x21:Completed 570000 out of 1000000 steps (57%)
2019-10-14:18:13:12:WU05:FS00:Upload 7.56%
2019-10-14:18:13:15:WU01:FS00:Upload 24.36%
2019-10-14:18:13:17:WU06:FS01:Upload 17.96%
2019-10-14:18:13:19:WU05:FS00:Upload 7.72%
2019-10-14:18:13:21:WU00:FS01:Upload 28.11%
2019-10-14:18:13:23:WU02:FS01:Upload 11.47%
2019-10-14:18:13:24:WU01:FS00:Upload 24.65%
2019-10-14:18:13:25:WU05:FS00:Upload 7.81%
2019-10-14:18:13:25:WU06:FS01:Upload 18.05%
2019-10-14:18:13:30:WU00:FS01:Upload 28.19%
2019-10-14:18:13:34:WU02:FS01:Upload 11.66%
2019-10-14:18:13:34:WU01:FS00:Upload 24.75%
2019-10-14:18:13:34:WU05:FS00:Upload 7.97%
2019-10-14:18:13:38:WU00:FS01:Upload 28.35%
2019-10-14:18:13:40:WU01:FS00:Upload 24.85%
2019-10-14:18:13:41:WU05:FS00:Upload 8.13%
2019-10-14:18:13:41:WU06:FS01:Upload 18.32%
2019-10-14:18:13:42:WU08:FS01:0x21:Completed 580000 out of 1000000 steps (58%)
2019-10-14:18:13:44:WU02:FS01:Upload 11.76%
2019-10-14:18:13:47:WU01:FS00:Upload 24.94%
2019-10-14:18:13:48:WU06:FS01:Upload 18.41%
2019-10-14:18:13:48:WU05:FS00:Upload 8.21%
2019-10-14:18:13:48:WU00:FS01:Upload 28.44%
2019-10-14:18:13:55:WU02:FS01:Upload 11.96%
2019-10-14:18:14:02:WU01:FS00:Upload 25.14%
2019-10-14:18:14:03:WU06:FS01:Upload 18.50%
2019-10-14:18:14:07:WU05:FS00:Upload 8.37%
2019-10-14:18:14:08:WU00:FS01:Upload 28.60%
2019-10-14:18:14:11:WU02:FS01:Upload 12.25%
2019-10-14:18:14:16:WU01:FS00:Upload 25.23%
2019-10-14:18:14:16:WU06:FS01:Upload 18.59%
2019-10-14:18:14:18:WU05:FS00:Upload 8.61%
2019-10-14:18:14:19:WU08:FS01:0x21:Completed 590000 out of 1000000 steps (59%)
2019-10-14:18:14:19:WU02:FS01:Upload 12.34%
2019-10-14:18:14:20:WU00:FS01:Upload 28.76%
2019-10-14:18:14:23:WU06:FS01:Upload 18.78%
2019-10-14:18:14:23:WU01:FS00:Upload 25.33%
2019-10-14:18:14:28:WU05:FS00:Upload 8.77%
2019-10-14:18:14:31:WU02:FS01:Upload 12.44%
2019-10-14:18:14:31:WU01:FS00:Upload 25.43%
2019-10-14:18:14:35:WU05:FS00:Upload 8.93%
2019-10-14:18:14:37:WU00:FS01:Upload 28.84%
2019-10-14:18:14:38:WU02:FS01:Upload 12.64%
2019-10-14:18:14:38:WU01:FS00:Upload 25.52%
2019-10-14:18:14:39:WU06:FS01:Upload 18.87%
2019-10-14:18:14:41:WU05:FS00:Upload 9.01%
2019-10-14:18:14:44:WU02:FS01:Upload 12.73%
2019-10-14:18:14:49:WU06:FS01:Upload 19.05%
2019-10-14:18:14:50:WU00:FS01:Upload 28.92%
2019-10-14:18:14:50:WU05:FS00:Upload 9.25%
2019-10-14:18:14:53:WU08:FS01:0x21:Completed 600000 out of 1000000 steps (60%)
2019-10-14:18:14:55:WU01:FS00:Upload 25.72%
2019-10-14:18:14:56:WU00:FS01:Upload 29.00%
2019-10-14:18:14:56:WU06:FS01:Upload 19.23%
2019-10-14:18:14:59:WU05:FS00:Upload 9.41%
2019-10-14:18:15:02:WU06:FS01:Upload 19.32%
2019-10-14:18:15:07:WU01:FS00:Upload 25.81%
2019-10-14:18:15:08:WU05:FS00:Upload 9.49%
2019-10-14:18:15:09:WU02:FS01:Upload 12.83%
2019-10-14:18:15:14:WU00:FS01:Upload 29.24%
2019-10-14:18:15:17:WU05:FS00:Upload 9.74%
2019-10-14:18:15:21:WU00:FS01:Upload 29.32%
2019-10-14:18:15:23:WU02:FS01:Upload 13.02%
2019-10-14:18:15:24:WU01:FS00:Upload 25.91%
2019-10-14:18:15:24:WU04:FS00:Upload 8.11%
2019-10-14:18:15:24:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
2019-10-14:18:15:24:WU04:FS00:Trying to send results to collection server
2019-10-14:18:15:24:WU04:FS00:Uploading 68.60MiB to 155.247.166.220
2019-10-14:18:15:24:WU04:FS00:Connecting to 155.247.166.220:8080
2019-10-14:18:15:25:ERROR:WU04:FS00:Exception: Transfer failed
2019-10-14:18:15:25:WU06:FS01:Upload 19.42%
2019-10-14:18:15:25:WARNING:WU07:FS00:Detected clock skew (15 mins 53 secs), I/O delay, laptop hibernation or other slowdown noted, adjusting time estimates
2019-10-14:18:15:25:WU07:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
2019-10-14:18:15:25:WU07:FS00:Sending unit results: id:07 state:SEND error:NO_ERROR project:14278 run:0 clone:71 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6913f4362c
2019-10-14:18:15:25:WU07:FS00:Uploading 77.56MiB to 128.252.203.10
2019-10-14:18:15:25:WU07:FS00:Connecting to 128.252.203.10:8080
2019-10-14:18:15:27:WU05:FS00:Upload 9.82%
2019-10-14:18:15:30:WU08:FS01:0x21:Completed 610000 out of 1000000 steps (61%)
2019-10-14:18:15:30:WU02:FS01:Upload 13.12%
2019-10-14:18:15:32:WU00:FS01:Upload 29.40%
2019-10-14:18:15:32:WU07:FS00:Upload 0.32%
2019-10-14:18:15:34:WU05:FS00:Upload 9.90%
2019-10-14:18:15:35:WU01:FS00:Upload 26.10%
2019-10-14:18:15:36:WU02:FS01:Upload 13.22%
2019-10-14:18:15:37:WU06:FS01:Upload 19.51%
2019-10-14:18:15:40:WU00:FS01:Upload 29.48%
2019-10-14:18:15:41:WU01:FS00:Upload 26.20%
2019-10-14:18:15:43:WU05:FS00:Upload 10.06%
2019-10-14:18:15:44:WU02:FS01:Upload 13.32%
2019-10-14:18:15:45:WU06:FS01:Upload 19.69%
2019-10-14:18:15:49:WU05:FS00:Upload 10.14%
2019-10-14:18:15:50:WU02:FS01:Upload 13.41%
2019-10-14:18:15:50:WU00:FS01:Upload 29.64%
2019-10-14:18:15:51:WU07:FS00:Upload 0.48%

Posted: **Mon Oct 14, 2019 7:05 pm**

Windows logfile (almost no uploads are going to get through in saturation):

Code: Select all

2019-10-14:15:04:10:WU04:FS01:Upload 0.29%
2019-10-14:15:04:10:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:04:10:WU04:FS01:Trying to send results to collection server
2019-10-14:15:04:10:WU04:FS01:Uploading 64.32MiB to 155.247.166.219
2019-10-14:15:04:10:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-14:15:04:12:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-14:15:04:36:WU05:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
2019-10-14:15:05:25:WU05:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
2019-10-14:15:06:08:WU05:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
2019-10-14:15:06:09:WU06:FS01:Connecting to 65.254.110.245:8080
2019-10-14:15:06:09:WU06:FS01:Assigned to work server 128.252.203.10
2019-10-14:15:06:09:WU06:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 128.252.203.10
2019-10-14:15:06:09:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:06:10:WU06:FS01:Downloading 85.24MiB
2019-10-14:15:06:14:WU05:FS01:0x21:Saving result file logfile_01.txt
2019-10-14:15:06:14:WU05:FS01:0x21:Saving result file checkpointState.xml
2019-10-14:15:06:14:WU05:FS01:0x21:Saving result file checkpt.crc
2019-10-14:15:06:14:WU05:FS01:0x21:Saving result file log.txt
2019-10-14:15:06:14:WU05:FS01:0x21:Saving result file positions.xtc
2019-10-14:15:06:14:WU05:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
2019-10-14:15:06:15:WU05:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
2019-10-14:15:06:15:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:15:06:15:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:15:06:15:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:06:16:WU06:FS01:Download 19.36%
2019-10-14:15:06:22:WU06:FS01:Download 32.33%
2019-10-14:15:06:28:WU06:FS01:Download 42.09%
2019-10-14:15:06:29:WU05:FS01:Upload 0.16%
2019-10-14:15:06:34:WU06:FS01:Download 46.56%
2019-10-14:15:06:39:WU05:FS01:Upload 0.24%
2019-10-14:15:06:40:WU06:FS01:Download 49.35%
2019-10-14:15:06:46:WU06:FS01:Download 50.74%
2019-10-14:15:06:52:WU06:FS01:Download 52.20%
2019-10-14:15:06:58:WU06:FS01:Download 53.89%
2019-10-14:15:07:01:WU05:FS01:Upload 0.32%
2019-10-14:15:07:04:WU06:FS01:Download 55.36%
2019-10-14:15:07:10:WU06:FS01:Download 56.82%
2019-10-14:15:07:12:WU05:FS01:Upload 0.39%
2019-10-14:15:07:16:WU06:FS01:Download 67.38%
2019-10-14:15:07:20:WU05:FS01:Upload 0.55%
2019-10-14:15:07:22:WU06:FS01:Download 80.43%
2019-10-14:15:07:28:WU06:FS01:Download 93.48%
2019-10-14:15:07:28:WU05:FS01:Upload 0.71%
2019-10-14:15:07:31:WU06:FS01:Download complete
2019-10-14:15:07:31:WU06:FS01:Received Unit: id:06 state:DOWNLOAD error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:15:07:31:WU06:FS01:Starting
2019-10-14:15:07:31:WU06:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Jeff\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 06 -suffix 01 -version 705 -lifeline 5536 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
2019-10-14:15:07:31:WU06:FS01:Started FahCore on PID 9948
2019-10-14:15:07:31:WU06:FS01:Core PID:12244
2019-10-14:15:07:31:WU06:FS01:FahCore 0x21 started
2019-10-14:15:07:31:WU06:FS01:0x21:*********************** Log Started 2019-10-14T15:07:31Z ***********************
2019-10-14:15:07:31:WU06:FS01:0x21:Project: 14276 (Run 0, Clone 1, Gen 12)
2019-10-14:15:07:31:WU06:FS01:0x21:Unit: 0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:15:07:31:WU06:FS01:0x21:CPU: 0x00000000000000000000000000000000
2019-10-14:15:07:31:WU06:FS01:0x21:Machine: 1
2019-10-14:15:07:31:WU06:FS01:0x21:Reading tar file core.xml
2019-10-14:15:07:31:WU06:FS01:0x21:Reading tar file integrator.xml
2019-10-14:15:07:31:WU06:FS01:0x21:Reading tar file state.xml
2019-10-14:15:07:31:WU06:FS01:0x21:Reading tar file system.xml
2019-10-14:15:07:31:WU06:FS01:0x21:Digital signatures verified
2019-10-14:15:07:31:WU06:FS01:0x21:Folding@home GPU Core21 Folding@home Core
2019-10-14:15:07:31:WU06:FS01:0x21:Version 0.0.20
2019-10-14:15:07:38:WU05:FS01:Upload 0.95%
2019-10-14:15:07:47:WU05:FS01:Upload 1.03%
2019-10-14:15:07:49:WU06:FS01:0x21:Completed 0 out of 1000000 steps (0%)
2019-10-14:15:07:49:WU06:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2019-10-14:15:08:01:WU05:FS01:Upload 1.18%
2019-10-14:15:08:15:WU05:FS01:Upload 1.26%
2019-10-14:15:08:22:WU05:FS01:Upload 1.34%
2019-10-14:15:08:31:WU05:FS01:Upload 1.50%
2019-10-14:15:08:34:WU06:FS01:0x21:Completed 10000 out of 1000000 steps (1%)
2019-10-14:15:08:41:WU05:FS01:Upload 1.58%
2019-10-14:15:08:48:WU05:FS01:Upload 1.66%
2019-10-14:15:08:54:WU05:FS01:Upload 1.74%
2019-10-14:15:09:00:WU05:FS01:Upload 1.89%
2019-10-14:15:09:14:WU05:FS01:Upload 2.05%
2019-10-14:15:09:18:WU06:FS01:0x21:Completed 20000 out of 1000000 steps (2%)
2019-10-14:15:09:20:WU05:FS01:Upload 2.21%
2019-10-14:15:09:28:WU05:FS01:Upload 2.37%
2019-10-14:15:09:36:WU05:FS01:Upload 2.52%
2019-10-14:15:09:54:WU05:FS01:Upload 2.68%
2019-10-14:15:10:08:WU06:FS01:0x21:Completed 30000 out of 1000000 steps (3%)
2019-10-14:15:10:37:WU05:FS01:Upload 3.00%
2019-10-14:15:10:43:WU05:FS01:Upload 3.08%
2019-10-14:15:10:52:WU06:FS01:0x21:Completed 40000 out of 1000000 steps (4%)
2019-10-14:15:10:54:WU05:FS01:Upload 3.23%
2019-10-14:15:11:00:WU05:FS01:Upload 3.31%
2019-10-14:15:11:15:WU05:FS01:Upload 3.39%
2019-10-14:15:11:31:WU05:FS01:Upload 3.55%
2019-10-14:15:11:38:WU05:FS01:Upload 3.71%
2019-10-14:15:11:42:WU06:FS01:0x21:Completed 50000 out of 1000000 steps (5%)
2019-10-14:15:12:04:WU05:FS01:Upload 3.87%
2019-10-14:15:12:10:WU05:FS01:Upload 4.02%
2019-10-14:15:12:19:WU05:FS01:Upload 4.10%
2019-10-14:15:12:25:WU05:FS01:Upload 4.26%
2019-10-14:15:12:26:WU06:FS01:0x21:Completed 60000 out of 1000000 steps (6%)
2019-10-14:15:12:32:WU05:FS01:Upload 4.50%
2019-10-14:15:12:38:WU05:FS01:Upload 4.65%
2019-10-14:15:12:48:WU05:FS01:Upload 4.73%
2019-10-14:15:13:06:WU05:FS01:Upload 4.89%
2019-10-14:15:13:16:WU06:FS01:0x21:Completed 70000 out of 1000000 steps (7%)
2019-10-14:15:13:19:WU05:FS01:Upload 5.05%
2019-10-14:15:13:28:WU05:FS01:Upload 5.13%
2019-10-14:15:13:36:WU05:FS01:Upload 5.29%
2019-10-14:15:13:42:WU05:FS01:Upload 5.37%
2019-10-14:15:14:00:WU06:FS01:0x21:Completed 80000 out of 1000000 steps (8%)
2019-10-14:15:14:11:WU05:FS01:Upload 5.52%
2019-10-14:15:14:50:WU06:FS01:0x21:Completed 90000 out of 1000000 steps (9%)
2019-10-14:15:15:34:WU06:FS01:0x21:Completed 100000 out of 1000000 steps (10%)
2019-10-14:15:16:24:WU06:FS01:0x21:Completed 110000 out of 1000000 steps (11%)
2019-10-14:15:17:09:WU06:FS01:0x21:Completed 120000 out of 1000000 steps (12%)
2019-10-14:15:17:25:WU05:FS01:Upload 5.60%
2019-10-14:15:17:25:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:17:25:WU05:FS01:Trying to send results to collection server
2019-10-14:15:17:25:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:15:17:25:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:17:26:WARNING:WU05:FS01:WorkServer connection failed on port 8080 trying 80
2019-10-14:15:17:26:WU05:FS01:Connecting to 155.247.166.220:80
2019-10-14:15:17:28:ERROR:WU05:FS01:Exception: Failed to connect to 155.247.166.220:80: No connection could be made because the target machine actively refused it.
2019-10-14:15:17:28:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:15:17:28:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:15:17:28:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:17:36:WU05:FS01:Upload 0.16%
2019-10-14:15:17:42:WU05:FS01:Upload 0.32%
2019-10-14:15:17:48:WU05:FS01:Upload 0.39%
2019-10-14:15:17:55:WU05:FS01:Upload 0.47%
2019-10-14:15:17:58:WU06:FS01:0x21:Completed 130000 out of 1000000 steps (13%)
2019-10-14:15:18:43:WU06:FS01:0x21:Completed 140000 out of 1000000 steps (14%)
2019-10-14:15:19:15:WU05:FS01:Upload 0.63%
2019-10-14:15:19:32:WU06:FS01:0x21:Completed 150000 out of 1000000 steps (15%)
2019-10-14:15:19:59:WU05:FS01:Upload 1.10%
2019-10-14:15:20:17:WU06:FS01:0x21:Completed 160000 out of 1000000 steps (16%)
2019-10-14:15:20:45:WU05:FS01:Upload 1.18%
2019-10-14:15:21:03:WU05:FS01:Upload 1.34%
2019-10-14:15:21:07:WU06:FS01:0x21:Completed 170000 out of 1000000 steps (17%)
2019-10-14:15:21:18:WU05:FS01:Upload 1.50%
2019-10-14:15:21:33:WU05:FS01:Upload 1.74%
2019-10-14:15:21:41:WU05:FS01:Upload 1.89%
2019-10-14:15:21:51:WU06:FS01:0x21:Completed 180000 out of 1000000 steps (18%)
2019-10-14:15:22:41:WU06:FS01:0x21:Completed 190000 out of 1000000 steps (19%)
2019-10-14:15:23:25:WU06:FS01:0x21:Completed 200000 out of 1000000 steps (20%)
2019-10-14:15:23:48:WU05:FS01:Upload 1.97%
2019-10-14:15:23:49:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:23:49:WU05:FS01:Trying to send results to collection server
2019-10-14:15:23:49:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:15:23:49:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:24:00:WU05:FS01:Upload 0.08%
2019-10-14:15:24:14:WU05:FS01:Upload 0.16%
2019-10-14:15:24:14:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:15:24:14:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:15:24:14:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:15:24:14:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:24:15:WU06:FS01:0x21:Completed 210000 out of 1000000 steps (21%)
2019-10-14:15:24:25:WU05:FS01:Upload 0.08%
2019-10-14:15:24:59:WU06:FS01:0x21:Completed 220000 out of 1000000 steps (22%)
2019-10-14:15:25:11:WU05:FS01:Upload 0.16%
2019-10-14:15:25:19:WU05:FS01:Upload 0.32%
2019-10-14:15:25:26:WU05:FS01:Upload 0.47%
2019-10-14:15:25:49:WU06:FS01:0x21:Completed 230000 out of 1000000 steps (23%)
2019-10-14:15:26:34:WU06:FS01:0x21:Completed 240000 out of 1000000 steps (24%)
2019-10-14:15:27:06:WU05:FS01:Upload 0.55%
2019-10-14:15:27:06:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:27:06:WU05:FS01:Trying to send results to collection server
2019-10-14:15:27:06:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:15:27:06:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:27:07:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:15:27:07:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:15:27:07:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:15:27:07:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:27:14:WU05:FS01:Upload 0.16%
2019-10-14:15:27:22:WU05:FS01:Upload 0.24%
2019-10-14:15:27:23:WU06:FS01:0x21:Completed 250000 out of 1000000 steps (25%)
2019-10-14:15:27:37:WU05:FS01:Upload 0.32%
2019-10-14:15:28:06:WU05:FS01:Upload 0.47%
2019-10-14:15:28:08:WU06:FS01:0x21:Completed 260000 out of 1000000 steps (26%)
2019-10-14:15:28:17:WU05:FS01:Upload 0.95%
2019-10-14:15:28:33:WU05:FS01:Upload 1.10%
2019-10-14:15:28:41:WU05:FS01:Upload 1.26%
2019-10-14:15:28:53:WU05:FS01:Upload 1.42%
2019-10-14:15:28:58:WU06:FS01:0x21:Completed 270000 out of 1000000 steps (27%)
2019-10-14:15:29:00:WU05:FS01:Upload 1.66%
2019-10-14:15:29:19:WU05:FS01:Upload 1.74%
2019-10-14:15:29:28:WU05:FS01:Upload 1.89%
2019-10-14:15:29:42:WU06:FS01:0x21:Completed 280000 out of 1000000 steps (28%)
2019-10-14:15:30:10:WU05:FS01:Upload 2.05%
2019-10-14:15:30:32:WU06:FS01:0x21:Completed 290000 out of 1000000 steps (29%)
2019-10-14:15:30:58:WU05:FS01:Upload 2.13%
2019-10-14:15:31:13:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14281 run:0 clone:40 gen:1 core:0x21 unit:0x0000000180fccb0a5d9e115daa9978ef
2019-10-14:15:31:13:WU04:FS01:Uploading 64.32MiB to 128.252.203.10
2019-10-14:15:31:13:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:31:16:WU06:FS01:0x21:Completed 300000 out of 1000000 steps (30%)
2019-10-14:15:31:19:WU04:FS01:Upload 0.10%
2019-10-14:15:32:06:WU06:FS01:0x21:Completed 310000 out of 1000000 steps (31%)
2019-10-14:15:32:36:WU05:FS01:Upload 2.21%
2019-10-14:15:32:36:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:32:36:WU05:FS01:Trying to send results to collection server
2019-10-14:15:32:36:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:15:32:36:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:32:44:WU05:FS01:Upload 0.08%
2019-10-14:15:32:44:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:15:32:44:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:15:32:44:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:15:32:44:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:32:50:WU06:FS01:0x21:Completed 320000 out of 1000000 steps (32%)
2019-10-14:15:33:08:WU04:FS01:Upload 0.19%
2019-10-14:15:33:08:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:33:08:WU04:FS01:Trying to send results to collection server
2019-10-14:15:33:08:WU04:FS01:Uploading 64.32MiB to 155.247.166.219
2019-10-14:15:33:08:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-14:15:33:09:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-14:15:33:40:WU06:FS01:0x21:Completed 330000 out of 1000000 steps (33%)
2019-10-14:15:33:49:WU05:FS01:Upload 0.08%
2019-10-14:15:33:58:WU05:FS01:Upload 0.95%
2019-10-14:15:34:05:WU05:FS01:Upload 1.03%
2019-10-14:15:34:14:WU05:FS01:Upload 1.18%
2019-10-14:15:34:23:WU05:FS01:Upload 1.26%
2019-10-14:15:34:25:WU06:FS01:0x21:Completed 340000 out of 1000000 steps (34%)
2019-10-14:15:35:15:WU06:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
2019-10-14:15:35:59:WU06:FS01:0x21:Completed 360000 out of 1000000 steps (36%)
2019-10-14:15:36:22:WU05:FS01:Upload 1.34%
2019-10-14:15:36:23:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:36:23:WU05:FS01:Trying to send results to collection server
2019-10-14:15:36:23:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:15:36:23:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:36:23:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:15:36:49:WU06:FS01:0x21:Completed 370000 out of 1000000 steps (37%)
2019-10-14:15:36:59:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:15:36:59:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:15:36:59:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:37:34:WU06:FS01:0x21:Completed 380000 out of 1000000 steps (38%)
2019-10-14:15:37:35:WU05:FS01:Upload 0.08%
2019-10-14:15:38:23:WU06:FS01:0x21:Completed 390000 out of 1000000 steps (39%)
2019-10-14:15:39:00:WU05:FS01:Upload 0.16%
2019-10-14:15:39:08:WU06:FS01:0x21:Completed 400000 out of 1000000 steps (40%)
2019-10-14:15:39:14:WU05:FS01:Upload 0.24%
2019-10-14:15:39:57:WU06:FS01:0x21:Completed 410000 out of 1000000 steps (41%)
2019-10-14:15:40:01:WU05:FS01:Upload 0.32%
2019-10-14:15:40:42:WU06:FS01:0x21:Completed 420000 out of 1000000 steps (42%)
2019-10-14:15:41:32:WU06:FS01:0x21:Completed 430000 out of 1000000 steps (43%)
2019-10-14:15:42:16:WU06:FS01:0x21:Completed 440000 out of 1000000 steps (44%)
2019-10-14:15:42:56:WU05:FS01:Upload 0.39%
2019-10-14:15:42:56:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:42:56:WU05:FS01:Trying to send results to collection server
2019-10-14:15:42:56:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:15:42:56:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:42:57:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:15:43:06:WU06:FS01:0x21:Completed 450000 out of 1000000 steps (45%)
2019-10-14:15:43:50:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:15:43:50:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:15:43:50:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:43:50:WU06:FS01:0x21:Completed 460000 out of 1000000 steps (46%)
2019-10-14:15:44:07:WU05:FS01:Upload 0.08%
2019-10-14:15:44:14:WU05:FS01:Upload 0.16%
2019-10-14:15:44:40:WU06:FS01:0x21:Completed 470000 out of 1000000 steps (47%)
2019-10-14:15:44:58:WU05:FS01:Upload 0.24%
2019-10-14:15:45:04:WU05:FS01:Upload 0.32%
2019-10-14:15:45:19:WU05:FS01:Upload 0.39%
2019-10-14:15:45:25:WU06:FS01:0x21:Completed 480000 out of 1000000 steps (48%)
2019-10-14:15:45:25:WU05:FS01:Upload 0.55%
2019-10-14:15:45:31:WU05:FS01:Upload 0.71%
2019-10-14:15:45:42:WU05:FS01:Upload 0.79%
2019-10-14:15:45:50:WU05:FS01:Upload 0.87%
2019-10-14:15:46:05:WU05:FS01:Upload 1.26%
2019-10-14:15:46:14:WU06:FS01:0x21:Completed 490000 out of 1000000 steps (49%)
2019-10-14:15:46:20:WU05:FS01:Upload 1.42%
2019-10-14:15:46:29:WU05:FS01:Upload 1.58%
2019-10-14:15:46:31:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:46:31:WU05:FS01:Trying to send results to collection server
2019-10-14:15:46:31:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:15:46:31:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:46:31:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:15:46:59:WU06:FS01:0x21:Completed 500000 out of 1000000 steps (50%)
2019-10-14:15:47:49:WU06:FS01:0x21:Completed 510000 out of 1000000 steps (51%)
2019-10-14:15:48:33:WU06:FS01:0x21:Completed 520000 out of 1000000 steps (52%)
2019-10-14:15:49:23:WU06:FS01:0x21:Completed 530000 out of 1000000 steps (53%)
2019-10-14:15:50:07:WU06:FS01:0x21:Completed 540000 out of 1000000 steps (54%)
2019-10-14:15:50:57:WU06:FS01:0x21:Completed 550000 out of 1000000 steps (55%)
2019-10-14:15:51:29:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14277 run:0 clone:5 gen:44 core:0x21 unit:0x0000002c80fccb0a5d98ab601193961f
2019-10-14:15:51:29:WU00:FS01:Uploading 71.20MiB to 128.252.203.10
2019-10-14:15:51:29:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:51:36:WU00:FS01:Upload 0.44%
2019-10-14:15:51:42:WU06:FS01:0x21:Completed 560000 out of 1000000 steps (56%)
2019-10-14:15:51:46:WU00:FS01:Upload 0.53%
2019-10-14:15:52:31:WU06:FS01:0x21:Completed 570000 out of 1000000 steps (57%)
2019-10-14:15:52:37:WU00:FS01:Upload 0.61%
2019-10-14:15:52:47:WU00:FS01:Upload 0.70%
2019-10-14:15:52:58:WU00:FS01:Upload 0.79%
2019-10-14:15:53:16:WU06:FS01:0x21:Completed 580000 out of 1000000 steps (58%)
2019-10-14:15:53:43:WU00:FS01:Upload 0.88%
2019-10-14:15:54:01:WU00:FS01:Upload 1.14%
2019-10-14:15:54:06:WU06:FS01:0x21:Completed 590000 out of 1000000 steps (59%)
2019-10-14:15:54:10:WU00:FS01:Upload 1.32%
2019-10-14:15:54:50:WU06:FS01:0x21:Completed 600000 out of 1000000 steps (60%)
2019-10-14:15:54:55:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:15:54:55:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:15:54:55:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:15:55:40:WU06:FS01:0x21:Completed 610000 out of 1000000 steps (61%)
2019-10-14:15:55:45:WU00:FS01:Upload 1.49%
2019-10-14:15:55:45:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:55:45:WU00:FS01:Trying to send results to collection server
2019-10-14:15:55:45:WU00:FS01:Uploading 71.20MiB to 155.247.166.220
2019-10-14:15:55:45:WU00:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:55:51:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-14:15:56:24:WU06:FS01:0x21:Completed 620000 out of 1000000 steps (62%)
2019-10-14:15:57:14:WU06:FS01:0x21:Completed 630000 out of 1000000 steps (63%)
2019-10-14:15:57:29:WU05:FS01:Upload 0.08%
2019-10-14:15:57:29:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:15:57:29:WU05:FS01:Trying to send results to collection server
2019-10-14:15:57:29:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:15:57:29:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:15:57:30:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:15:57:59:WU06:FS01:0x21:Completed 640000 out of 1000000 steps (64%)
2019-10-14:15:58:48:WU06:FS01:0x21:Completed 650000 out of 1000000 steps (65%)
2019-10-14:15:59:33:WU06:FS01:0x21:Completed 660000 out of 1000000 steps (66%)
2019-10-14:16:00:23:WU06:FS01:0x21:Completed 670000 out of 1000000 steps (67%)
2019-10-14:16:01:07:WU06:FS01:0x21:Completed 680000 out of 1000000 steps (68%)
2019-10-14:16:01:57:WU06:FS01:0x21:Completed 690000 out of 1000000 steps (69%)
2019-10-14:16:02:41:WU06:FS01:0x21:Completed 700000 out of 1000000 steps (70%)
2019-10-14:16:03:31:WU06:FS01:0x21:Completed 710000 out of 1000000 steps (71%)
2019-10-14:16:04:16:WU06:FS01:0x21:Completed 720000 out of 1000000 steps (72%)
2019-10-14:16:05:06:WU06:FS01:0x21:Completed 730000 out of 1000000 steps (73%)
2019-10-14:16:05:50:WU06:FS01:0x21:Completed 740000 out of 1000000 steps (74%)
2019-10-14:16:06:40:WU06:FS01:0x21:Completed 750000 out of 1000000 steps (75%)
2019-10-14:16:07:24:WU06:FS01:0x21:Completed 760000 out of 1000000 steps (76%)
2019-10-14:16:08:14:WU06:FS01:0x21:Completed 770000 out of 1000000 steps (77%)
2019-10-14:16:08:59:WU06:FS01:0x21:Completed 780000 out of 1000000 steps (78%)
2019-10-14:16:09:48:WU06:FS01:0x21:Completed 790000 out of 1000000 steps (79%)
2019-10-14:16:10:33:WU06:FS01:0x21:Completed 800000 out of 1000000 steps (80%)
2019-10-14:16:11:23:WU06:FS01:0x21:Completed 810000 out of 1000000 steps (81%)
2019-10-14:16:12:04:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:97 gen:24 core:0x21 unit:0x0000001980fccb0a5d98ab56be241dc4
2019-10-14:16:12:04:WU03:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-14:16:12:04:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:12:07:WU06:FS01:0x21:Completed 820000 out of 1000000 steps (82%)
2019-10-14:16:12:52:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:16:12:52:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:16:12:52:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:12:57:WU06:FS01:0x21:Completed 830000 out of 1000000 steps (83%)
2019-10-14:16:13:00:WU05:FS01:Upload 0.08%
2019-10-14:16:13:07:WU05:FS01:Upload 0.16%
2019-10-14:16:13:42:WU06:FS01:0x21:Completed 840000 out of 1000000 steps (84%)
2019-10-14:16:14:32:WU06:FS01:0x21:Completed 850000 out of 1000000 steps (85%)
2019-10-14:16:14:34:WU03:FS01:Upload 0.10%
2019-10-14:16:14:34:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:14:34:WU03:FS01:Trying to send results to collection server
2019-10-14:16:14:34:WU03:FS01:Uploading 64.60MiB to 155.247.166.220
2019-10-14:16:14:34:WU03:FS01:Connecting to 155.247.166.220:8080
2019-10-14:16:14:35:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-14:16:15:09:WU05:FS01:Upload 0.24%
2019-10-14:16:15:09:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:15:09:WU05:FS01:Trying to send results to collection server
2019-10-14:16:15:09:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:16:15:09:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:16:15:10:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:16:15:16:WU06:FS01:0x21:Completed 860000 out of 1000000 steps (86%)
2019-10-14:16:16:06:WU06:FS01:0x21:Completed 870000 out of 1000000 steps (87%)
2019-10-14:16:16:51:WU06:FS01:0x21:Completed 880000 out of 1000000 steps (88%)
2019-10-14:16:17:40:WU06:FS01:0x21:Completed 890000 out of 1000000 steps (89%)
2019-10-14:16:18:12:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14281 run:0 clone:40 gen:1 core:0x21 unit:0x0000000180fccb0a5d9e115daa9978ef
2019-10-14:16:18:12:WU04:FS01:Uploading 64.32MiB to 128.252.203.10
2019-10-14:16:18:12:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:18:25:WU06:FS01:0x21:Completed 900000 out of 1000000 steps (90%)
2019-10-14:16:18:59:WU04:FS01:Upload 0.19%
2019-10-14:16:19:15:WU06:FS01:0x21:Completed 910000 out of 1000000 steps (91%)
2019-10-14:16:19:52:WU04:FS01:Upload 0.29%
2019-10-14:16:19:59:WU06:FS01:0x21:Completed 920000 out of 1000000 steps (92%)
2019-10-14:16:20:46:WU04:FS01:Upload 0.39%
2019-10-14:16:20:49:WU06:FS01:0x21:Completed 930000 out of 1000000 steps (93%)
2019-10-14:16:21:34:WU06:FS01:0x21:Completed 940000 out of 1000000 steps (94%)
2019-10-14:16:22:23:WU06:FS01:0x21:Completed 950000 out of 1000000 steps (95%)
2019-10-14:16:23:08:WU06:FS01:0x21:Completed 960000 out of 1000000 steps (96%)
2019-10-14:16:23:54:WU04:FS01:Upload 0.49%
2019-10-14:16:23:54:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:23:54:WU04:FS01:Trying to send results to collection server
2019-10-14:16:23:54:WU04:FS01:Uploading 64.32MiB to 155.247.166.219
2019-10-14:16:23:54:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-14:16:23:55:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-14:16:23:58:WU06:FS01:0x21:Completed 970000 out of 1000000 steps (97%)
2019-10-14:16:24:42:WU06:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
2019-10-14:16:25:32:WU06:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
2019-10-14:16:26:17:WU06:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
2019-10-14:16:26:17:WU07:FS01:Connecting to 65.254.110.245:8080
2019-10-14:16:26:17:WU07:FS01:Assigned to work server 128.252.203.10
2019-10-14:16:26:17:WU07:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 128.252.203.10
2019-10-14:16:26:17:WU07:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:26:18:WU07:FS01:Downloading 78.65MiB
2019-10-14:16:26:23:WU06:FS01:0x21:Saving result file logfile_01.txt
2019-10-14:16:26:23:WU06:FS01:0x21:Saving result file checkpointState.xml
2019-10-14:16:26:23:WU06:FS01:0x21:Saving result file checkpt.crc
2019-10-14:16:26:23:WU06:FS01:0x21:Saving result file log.txt
2019-10-14:16:26:23:WU06:FS01:0x21:Saving result file positions.xtc
2019-10-14:16:26:23:WU06:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
2019-10-14:16:26:24:WU07:FS01:Download 21.14%
2019-10-14:16:26:24:WU06:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
2019-10-14:16:26:24:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:16:26:24:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:16:26:24:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:26:30:WU07:FS01:Download 35.28%
2019-10-14:16:26:36:WU07:FS01:Download 49.35%
2019-10-14:16:26:37:WU06:FS01:Upload 0.15%
2019-10-14:16:26:42:WU07:FS01:Download 63.41%
2019-10-14:16:26:44:WU06:FS01:Upload 0.23%
2019-10-14:16:26:48:WU07:FS01:Download 77.56%
2019-10-14:16:26:51:WU06:FS01:Upload 0.31%
2019-10-14:16:26:54:WU07:FS01:Download 91.70%
2019-10-14:16:26:57:WU07:FS01:Download complete
2019-10-14:16:26:57:WU07:FS01:Received Unit: id:07 state:DOWNLOAD error:NO_ERROR project:14275 run:0 clone:96 gen:27 core:0x21 unit:0x0000001b80fccb0a5d98ab5af35ced5a
2019-10-14:16:26:57:WU07:FS01:Starting
2019-10-14:16:26:57:WU07:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Jeff\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 07 -suffix 01 -version 705 -lifeline 5536 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
2019-10-14:16:26:57:WU07:FS01:Started FahCore on PID 13136
2019-10-14:16:26:57:WU07:FS01:Core PID:12152
2019-10-14:16:26:57:WU07:FS01:FahCore 0x21 started
2019-10-14:16:26:58:WU07:FS01:0x21:*********************** Log Started 2019-10-14T16:26:57Z ***********************
2019-10-14:16:26:58:WU07:FS01:0x21:Project: 14275 (Run 0, Clone 96, Gen 27)
2019-10-14:16:26:58:WU07:FS01:0x21:Unit: 0x0000001b80fccb0a5d98ab5af35ced5a
2019-10-14:16:26:58:WU07:FS01:0x21:CPU: 0x00000000000000000000000000000000
2019-10-14:16:26:58:WU07:FS01:0x21:Machine: 1
2019-10-14:16:26:58:WU07:FS01:0x21:Reading tar file core.xml
2019-10-14:16:26:58:WU07:FS01:0x21:Reading tar file integrator.xml
2019-10-14:16:26:58:WU07:FS01:0x21:Reading tar file state.xml
2019-10-14:16:26:58:WU07:FS01:0x21:Reading tar file system.xml
2019-10-14:16:26:58:WU07:FS01:0x21:Digital signatures verified
2019-10-14:16:26:58:WU07:FS01:0x21:Folding@home GPU Core21 Folding@home Core
2019-10-14:16:26:58:WU07:FS01:0x21:Version 0.0.20
2019-10-14:16:27:08:WU06:FS01:Upload 0.46%
2019-10-14:16:27:14:WU07:FS01:0x21:Completed 0 out of 1000000 steps (0%)
2019-10-14:16:27:14:WU07:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2019-10-14:16:27:56:WU07:FS01:0x21:Completed 10000 out of 1000000 steps (1%)
2019-10-14:16:28:38:WU07:FS01:0x21:Completed 20000 out of 1000000 steps (2%)
2019-10-14:16:29:24:WU07:FS01:0x21:Completed 30000 out of 1000000 steps (3%)
2019-10-14:16:29:58:WU06:FS01:Upload 0.54%
2019-10-14:16:29:58:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:29:58:WU06:FS01:Trying to send results to collection server
2019-10-14:16:29:58:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:16:29:58:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:16:29:59:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:16:29:59:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:16:29:59:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:16:29:59:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:30:06:WU07:FS01:0x21:Completed 40000 out of 1000000 steps (4%)
2019-10-14:16:30:09:WU06:FS01:Upload 0.23%
2019-10-14:16:30:37:WU06:FS01:Upload 0.31%
2019-10-14:16:30:46:WU06:FS01:Upload 0.38%
2019-10-14:16:30:52:WU07:FS01:0x21:Completed 50000 out of 1000000 steps (5%)
2019-10-14:16:30:57:WU06:FS01:Upload 0.46%
2019-10-14:16:31:34:WU07:FS01:0x21:Completed 60000 out of 1000000 steps (6%)
2019-10-14:16:32:21:WU07:FS01:0x21:Completed 70000 out of 1000000 steps (7%)
2019-10-14:16:32:21:WU06:FS01:Upload 0.54%
2019-10-14:16:33:03:WU07:FS01:0x21:Completed 80000 out of 1000000 steps (8%)
2019-10-14:16:33:22:WU06:FS01:Upload 0.61%
2019-10-14:16:33:22:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:33:22:WU06:FS01:Trying to send results to collection server
2019-10-14:16:33:22:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:16:33:22:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:16:33:22:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:16:33:22:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:16:33:22:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:16:33:22:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:33:49:WU07:FS01:0x21:Completed 90000 out of 1000000 steps (9%)
2019-10-14:16:34:31:WU07:FS01:0x21:Completed 100000 out of 1000000 steps (10%)
2019-10-14:16:35:16:WU06:FS01:Upload 0.08%
2019-10-14:16:35:16:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:35:16:WU06:FS01:Trying to send results to collection server
2019-10-14:16:35:16:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:16:35:16:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:16:35:17:WU07:FS01:0x21:Completed 110000 out of 1000000 steps (11%)
2019-10-14:16:35:32:WU06:FS01:Upload 0.08%
2019-10-14:16:35:32:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:16:35:32:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:16:35:32:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:16:35:32:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:35:59:WU07:FS01:0x21:Completed 120000 out of 1000000 steps (12%)
2019-10-14:16:36:45:WU07:FS01:0x21:Completed 130000 out of 1000000 steps (13%)
2019-10-14:16:37:27:WU07:FS01:0x21:Completed 140000 out of 1000000 steps (14%)
2019-10-14:16:38:14:WU07:FS01:0x21:Completed 150000 out of 1000000 steps (15%)
2019-10-14:16:38:24:WU06:FS01:Upload 0.08%
2019-10-14:16:38:24:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:38:24:WU06:FS01:Trying to send results to collection server
2019-10-14:16:38:24:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:16:38:24:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:16:38:30:WU06:FS01:Upload 0.08%
2019-10-14:16:38:30:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:16:38:30:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:16:38:30:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:16:38:30:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:38:55:WU07:FS01:0x21:Completed 160000 out of 1000000 steps (16%)
2019-10-14:16:39:42:WU07:FS01:0x21:Completed 170000 out of 1000000 steps (17%)
2019-10-14:16:40:23:WU07:FS01:0x21:Completed 180000 out of 1000000 steps (18%)
2019-10-14:16:40:55:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14278 run:0 clone:48 gen:11 core:0x21 unit:0x0000000b80fccb0a5d98ab68ef27791b
2019-10-14:16:40:55:WU02:FS01:Uploading 77.62MiB to 128.252.203.10
2019-10-14:16:40:55:WU02:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:40:56:WU06:FS01:Upload 0.08%
2019-10-14:16:40:56:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:40:56:WU06:FS01:Trying to send results to collection server
2019-10-14:16:40:56:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:16:40:56:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:16:40:57:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:16:41:10:WU07:FS01:0x21:Completed 190000 out of 1000000 steps (19%)
2019-10-14:16:41:19:WU02:FS01:Upload 0.16%
2019-10-14:16:41:36:WU02:FS01:Upload 0.24%
2019-10-14:16:41:51:WU02:FS01:Upload 0.40%
2019-10-14:16:41:52:WU07:FS01:0x21:Completed 200000 out of 1000000 steps (20%)
2019-10-14:16:41:54:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:16:41:54:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:16:41:54:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:42:14:WU05:FS01:Upload 0.08%
2019-10-14:16:42:38:WU07:FS01:0x21:Completed 210000 out of 1000000 steps (21%)
2019-10-14:16:42:44:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:16:42:44:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:16:42:44:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:42:46:WU02:FS01:Upload 0.48%
2019-10-14:16:42:55:WU06:FS01:Upload 0.08%
2019-10-14:16:43:08:WU06:FS01:Upload 0.15%
2019-10-14:16:43:20:WU07:FS01:0x21:Completed 220000 out of 1000000 steps (22%)
2019-10-14:16:43:37:WU05:FS01:Upload 0.16%
2019-10-14:16:43:52:WU06:FS01:Upload 0.23%
2019-10-14:16:44:07:WU07:FS01:0x21:Completed 230000 out of 1000000 steps (23%)
2019-10-14:16:44:18:WU06:FS01:Upload 0.31%
2019-10-14:16:44:33:WU05:FS01:Upload 0.24%
2019-10-14:16:44:39:WU02:FS01:Upload 0.56%
2019-10-14:16:44:39:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:44:39:WU02:FS01:Trying to send results to collection server
2019-10-14:16:44:39:WU02:FS01:Uploading 77.62MiB to 155.247.166.220
2019-10-14:16:44:39:WU02:FS01:Connecting to 155.247.166.220:8080
2019-10-14:16:44:40:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-14:16:44:48:WU07:FS01:0x21:Completed 240000 out of 1000000 steps (24%)
2019-10-14:16:45:35:WU07:FS01:0x21:Completed 250000 out of 1000000 steps (25%)
2019-10-14:16:45:45:WU06:FS01:Upload 0.38%
2019-10-14:16:45:45:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:45:45:WU06:FS01:Trying to send results to collection server
2019-10-14:16:45:45:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:16:45:45:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:16:45:53:WU06:FS01:Upload 0.08%
2019-10-14:16:45:53:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:16:46:17:WU07:FS01:0x21:Completed 260000 out of 1000000 steps (26%)
2019-10-14:16:47:03:WU07:FS01:0x21:Completed 270000 out of 1000000 steps (27%)
2019-10-14:16:47:45:WU07:FS01:0x21:Completed 280000 out of 1000000 steps (28%)
2019-10-14:16:48:31:WU07:FS01:0x21:Completed 290000 out of 1000000 steps (29%)
2019-10-14:16:49:13:WU07:FS01:0x21:Completed 300000 out of 1000000 steps (30%)
2019-10-14:16:49:35:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:16:49:35:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:16:49:35:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:50:00:WU07:FS01:0x21:Completed 310000 out of 1000000 steps (31%)
2019-10-14:16:50:10:WU06:FS01:Upload 0.15%
2019-10-14:16:50:32:WU05:FS01:Upload 0.87%
2019-10-14:16:50:32:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:50:32:WU05:FS01:Trying to send results to collection server
2019-10-14:16:50:32:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:16:50:32:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:16:50:34:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:16:50:41:WU07:FS01:0x21:Completed 320000 out of 1000000 steps (32%)
2019-10-14:16:50:45:WU06:FS01:Upload 0.31%
2019-10-14:16:51:28:WU07:FS01:0x21:Completed 330000 out of 1000000 steps (33%)
2019-10-14:16:51:33:WU06:FS01:Upload 0.38%
2019-10-14:16:51:44:WU06:FS01:Upload 0.46%
2019-10-14:16:51:52:WU06:FS01:Upload 0.54%
2019-10-14:16:52:00:WU06:FS01:Upload 0.61%
2019-10-14:16:52:10:WU07:FS01:0x21:Completed 340000 out of 1000000 steps (34%)
2019-10-14:16:52:56:WU07:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
2019-10-14:16:53:38:WU07:FS01:0x21:Completed 360000 out of 1000000 steps (36%)
2019-10-14:16:53:43:WU06:FS01:Upload 0.69%
2019-10-14:16:53:43:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:53:43:WU06:FS01:Trying to send results to collection server
2019-10-14:16:53:43:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:16:53:43:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:16:53:44:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:16:54:24:WU07:FS01:0x21:Completed 370000 out of 1000000 steps (37%)
2019-10-14:16:55:06:WU07:FS01:0x21:Completed 380000 out of 1000000 steps (38%)
2019-10-14:16:55:53:WU07:FS01:0x21:Completed 390000 out of 1000000 steps (39%)
2019-10-14:16:56:34:WU07:FS01:0x21:Completed 400000 out of 1000000 steps (40%)
2019-10-14:16:57:21:WU07:FS01:0x21:Completed 410000 out of 1000000 steps (41%)
2019-10-14:16:57:34:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14278 run:0 clone:18 gen:14 core:0x21 unit:0x0000000e80fccb0a5d98ab66d82dd86b
2019-10-14:16:57:34:WU01:FS01:Uploading 77.57MiB to 128.252.203.10
2019-10-14:16:57:34:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-14:16:57:40:WU01:FS01:Upload 0.08%
2019-10-14:16:57:57:WU01:FS01:Upload 0.16%
2019-10-14:16:58:02:WU07:FS01:0x21:Completed 420000 out of 1000000 steps (42%)
2019-10-14:16:58:10:WU01:FS01:Upload 0.24%
2019-10-14:16:58:49:WU07:FS01:0x21:Completed 430000 out of 1000000 steps (43%)
2019-10-14:16:59:31:WU07:FS01:0x21:Completed 440000 out of 1000000 steps (44%)
2019-10-14:16:59:56:WU01:FS01:Upload 0.32%
2019-10-14:16:59:57:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:16:59:57:WU01:FS01:Trying to send results to collection server
2019-10-14:16:59:57:WU01:FS01:Uploading 77.57MiB to 155.247.166.220
2019-10-14:16:59:57:WU01:FS01:Connecting to 155.247.166.220:8080
2019-10-14:16:59:57:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-14:17:00:17:WU07:FS01:0x21:Completed 450000 out of 1000000 steps (45%)
2019-10-14:17:00:41:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:17:00:41:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:17:00:41:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:00:52:WU06:FS01:Upload 0.08%
2019-10-14:17:00:59:WU07:FS01:0x21:Completed 460000 out of 1000000 steps (46%)
2019-10-14:17:01:45:WU07:FS01:0x21:Completed 470000 out of 1000000 steps (47%)
2019-10-14:17:01:46:WU06:FS01:Upload 0.15%
2019-10-14:17:01:55:WU06:FS01:Upload 0.23%
2019-10-14:17:02:18:WU06:FS01:Upload 0.31%
2019-10-14:17:02:27:WU07:FS01:0x21:Completed 480000 out of 1000000 steps (48%)
2019-10-14:17:03:14:WU07:FS01:0x21:Completed 490000 out of 1000000 steps (49%)
2019-10-14:17:03:55:WU07:FS01:0x21:Completed 500000 out of 1000000 steps (50%)
2019-10-14:17:03:56:WU06:FS01:Upload 0.38%
2019-10-14:17:03:56:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:03:56:WU06:FS01:Trying to send results to collection server
2019-10-14:17:03:56:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:17:03:56:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:17:03:56:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:17:04:42:WU07:FS01:0x21:Completed 510000 out of 1000000 steps (51%)
2019-10-14:17:05:23:WU07:FS01:0x21:Completed 520000 out of 1000000 steps (52%)
2019-10-14:17:06:10:WU07:FS01:0x21:Completed 530000 out of 1000000 steps (53%)
2019-10-14:17:06:52:WU07:FS01:0x21:Completed 540000 out of 1000000 steps (54%)
2019-10-14:17:07:38:WU07:FS01:0x21:Completed 550000 out of 1000000 steps (55%)
2019-10-14:17:08:20:WU07:FS01:0x21:Completed 560000 out of 1000000 steps (56%)
2019-10-14:17:09:07:WU07:FS01:0x21:Completed 570000 out of 1000000 steps (57%)
2019-10-14:17:09:48:WU07:FS01:0x21:Completed 580000 out of 1000000 steps (58%)
2019-10-14:17:10:35:WU07:FS01:0x21:Completed 590000 out of 1000000 steps (59%)
2019-10-14:17:11:17:WU07:FS01:0x21:Completed 600000 out of 1000000 steps (60%)
2019-10-14:17:12:03:WU07:FS01:0x21:Completed 610000 out of 1000000 steps (61%)
2019-10-14:17:12:45:WU07:FS01:0x21:Completed 620000 out of 1000000 steps (62%)
2019-10-14:17:13:31:WU07:FS01:0x21:Completed 630000 out of 1000000 steps (63%)
2019-10-14:17:14:13:WU07:FS01:0x21:Completed 640000 out of 1000000 steps (64%)
2019-10-14:17:15:00:WU07:FS01:0x21:Completed 650000 out of 1000000 steps (65%)
2019-10-14:17:15:41:WU07:FS01:0x21:Completed 660000 out of 1000000 steps (66%)
2019-10-14:17:16:28:WU07:FS01:0x21:Completed 670000 out of 1000000 steps (67%)
2019-10-14:17:17:10:WU07:FS01:0x21:Completed 680000 out of 1000000 steps (68%)
2019-10-14:17:17:56:WU07:FS01:0x21:Completed 690000 out of 1000000 steps (69%)
2019-10-14:17:18:38:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:17:18:38:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:17:18:38:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:18:38:WU07:FS01:0x21:Completed 700000 out of 1000000 steps (70%)
2019-10-14:17:19:25:WU07:FS01:0x21:Completed 710000 out of 1000000 steps (71%)
2019-10-14:17:20:06:WU07:FS01:0x21:Completed 720000 out of 1000000 steps (72%)
2019-10-14:17:20:18:WU06:FS01:Upload 0.08%
2019-10-14:17:20:18:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:20:18:WU06:FS01:Trying to send results to collection server
2019-10-14:17:20:18:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:17:20:18:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:17:20:19:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:17:20:53:WU07:FS01:0x21:Completed 730000 out of 1000000 steps (73%)
2019-10-14:17:21:35:WU07:FS01:0x21:Completed 740000 out of 1000000 steps (74%)
2019-10-14:17:22:21:WU07:FS01:0x21:Completed 750000 out of 1000000 steps (75%)
2019-10-14:17:23:03:WU07:FS01:0x21:Completed 760000 out of 1000000 steps (76%)
2019-10-14:17:23:50:WU07:FS01:0x21:Completed 770000 out of 1000000 steps (77%)
2019-10-14:17:24:31:WU07:FS01:0x21:Completed 780000 out of 1000000 steps (78%)
2019-10-14:17:25:18:WU07:FS01:0x21:Completed 790000 out of 1000000 steps (79%)
2019-10-14:17:25:59:WU07:FS01:0x21:Completed 800000 out of 1000000 steps (80%)
2019-10-14:17:26:46:WU07:FS01:0x21:Completed 810000 out of 1000000 steps (81%)
2019-10-14:17:27:28:WU07:FS01:0x21:Completed 820000 out of 1000000 steps (82%)
2019-10-14:17:28:14:WU07:FS01:0x21:Completed 830000 out of 1000000 steps (83%)
2019-10-14:17:28:53:WU05:FS01:Sending unit results: id:05 state:SEND error:NO_ERROR project:14279 run:0 clone:36 gen:12 core:0x21 unit:0x0000000c80fccb0a5d98ab6cc8ff3b85
2019-10-14:17:28:53:WU05:FS01:Uploading 79.22MiB to 128.252.203.10
2019-10-14:17:28:53:WU05:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:28:56:WU07:FS01:0x21:Completed 840000 out of 1000000 steps (84%)
2019-10-14:17:29:02:WU05:FS01:Upload 0.08%
2019-10-14:17:29:43:WU07:FS01:0x21:Completed 850000 out of 1000000 steps (85%)
2019-10-14:17:30:24:WU07:FS01:0x21:Completed 860000 out of 1000000 steps (86%)
2019-10-14:17:30:39:WU05:FS01:Upload 0.16%
2019-10-14:17:30:39:WARNING:WU05:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:30:39:WU05:FS01:Trying to send results to collection server
2019-10-14:17:30:39:WU05:FS01:Uploading 79.22MiB to 155.247.166.220
2019-10-14:17:30:39:WU05:FS01:Connecting to 155.247.166.220:8080
2019-10-14:17:30:45:WU05:FS01:Upload 0.08%
2019-10-14:17:30:45:ERROR:WU05:FS01:Exception: Transfer failed
2019-10-14:17:31:11:WU07:FS01:0x21:Completed 870000 out of 1000000 steps (87%)
2019-10-14:17:31:53:WU07:FS01:0x21:Completed 880000 out of 1000000 steps (88%)
2019-10-14:17:32:40:WU07:FS01:0x21:Completed 890000 out of 1000000 steps (89%)
2019-10-14:17:33:22:WU07:FS01:0x21:Completed 900000 out of 1000000 steps (90%)
2019-10-14:17:34:09:WU07:FS01:0x21:Completed 910000 out of 1000000 steps (91%)
2019-10-14:17:34:13:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14281 run:0 clone:40 gen:1 core:0x21 unit:0x0000000180fccb0a5d9e115daa9978ef
2019-10-14:17:34:13:WU04:FS01:Uploading 64.32MiB to 128.252.203.10
2019-10-14:17:34:13:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:34:51:WU07:FS01:0x21:Completed 920000 out of 1000000 steps (92%)
2019-10-14:17:35:39:WU07:FS01:0x21:Completed 930000 out of 1000000 steps (93%)
2019-10-14:17:36:01:WU04:FS01:Upload 0.10%
2019-10-14:17:36:01:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:36:01:WU04:FS01:Trying to send results to collection server
2019-10-14:17:36:01:WU04:FS01:Uploading 64.32MiB to 155.247.166.219
2019-10-14:17:36:01:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-14:17:36:02:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-14:17:36:21:WU07:FS01:0x21:Completed 940000 out of 1000000 steps (94%)
2019-10-14:17:37:08:WU07:FS01:0x21:Completed 950000 out of 1000000 steps (95%)
2019-10-14:17:37:50:WU07:FS01:0x21:Completed 960000 out of 1000000 steps (96%)
2019-10-14:17:38:37:WU07:FS01:0x21:Completed 970000 out of 1000000 steps (97%)
2019-10-14:17:39:19:WU07:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
2019-10-14:17:40:06:WU07:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
2019-10-14:17:40:48:WU07:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
2019-10-14:17:40:49:WU08:FS01:Connecting to 65.254.110.245:8080
2019-10-14:17:40:49:WU08:FS01:Assigned to work server 128.252.203.10
2019-10-14:17:40:49:WU08:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 128.252.203.10
2019-10-14:17:40:49:WU08:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:40:50:WU08:FS01:Downloading 68.55MiB
2019-10-14:17:40:54:WU07:FS01:0x21:Saving result file logfile_01.txt
2019-10-14:17:40:54:WU07:FS01:0x21:Saving result file checkpointState.xml
2019-10-14:17:40:54:WU07:FS01:0x21:Saving result file checkpt.crc
2019-10-14:17:40:54:WU07:FS01:0x21:Saving result file log.txt
2019-10-14:17:40:54:WU07:FS01:0x21:Saving result file positions.xtc
2019-10-14:17:40:54:WU07:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
2019-10-14:17:40:55:WU07:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
2019-10-14:17:40:55:WU07:FS01:Sending unit results: id:07 state:SEND error:NO_ERROR project:14275 run:0 clone:96 gen:27 core:0x21 unit:0x0000001b80fccb0a5d98ab5af35ced5a
2019-10-14:17:40:55:WU07:FS01:Uploading 74.81MiB to 128.252.203.10
2019-10-14:17:40:55:WU07:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:40:56:WU08:FS01:Download 25.07%
2019-10-14:17:41:02:WU08:FS01:Download 34.01%
2019-10-14:17:41:04:WU07:FS01:Upload 0.17%
2019-10-14:17:41:08:WU08:FS01:Download 41.76%
2019-10-14:17:41:10:WU07:FS01:Upload 0.25%
2019-10-14:17:41:14:WU08:FS01:Download 55.89%
2019-10-14:17:41:17:WU07:FS01:Upload 0.33%
2019-10-14:17:41:20:WU08:FS01:Download 72.03%
2019-10-14:17:41:26:WU08:FS01:Download 88.17%
2019-10-14:17:41:30:WU08:FS01:Download complete
2019-10-14:17:41:30:WU08:FS01:Received Unit: id:08 state:DOWNLOAD error:NO_ERROR project:14274 run:0 clone:93 gen:21 core:0x21 unit:0x0000001580fccb0a5d98ab5597c50a78
2019-10-14:17:41:30:WU08:FS01:Starting
2019-10-14:17:41:30:WU08:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Jeff\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 08 -suffix 01 -version 705 -lifeline 5536 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
2019-10-14:17:41:30:WU08:FS01:Started FahCore on PID 7784
2019-10-14:17:41:30:WU08:FS01:Core PID:1828
2019-10-14:17:41:30:WU08:FS01:FahCore 0x21 started
2019-10-14:17:41:31:WU08:FS01:0x21:*********************** Log Started 2019-10-14T17:41:30Z ***********************
2019-10-14:17:41:31:WU08:FS01:0x21:Project: 14274 (Run 0, Clone 93, Gen 21)
2019-10-14:17:41:31:WU08:FS01:0x21:Unit: 0x0000001580fccb0a5d98ab5597c50a78
2019-10-14:17:41:31:WU08:FS01:0x21:CPU: 0x00000000000000000000000000000000
2019-10-14:17:41:31:WU08:FS01:0x21:Machine: 1
2019-10-14:17:41:31:WU08:FS01:0x21:Reading tar file core.xml
2019-10-14:17:41:31:WU08:FS01:0x21:Reading tar file integrator.xml
2019-10-14:17:41:31:WU08:FS01:0x21:Reading tar file state.xml
2019-10-14:17:41:31:WU08:FS01:0x21:Reading tar file system.xml
2019-10-14:17:41:31:WU08:FS01:0x21:Digital signatures verified
2019-10-14:17:41:31:WU08:FS01:0x21:Folding@home GPU Core21 Folding@home Core
2019-10-14:17:41:31:WU08:FS01:0x21:Version 0.0.20
2019-10-14:17:41:39:WU07:FS01:Upload 0.42%
2019-10-14:17:41:45:WU08:FS01:0x21:Completed 0 out of 1000000 steps (0%)
2019-10-14:17:41:45:WU08:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2019-10-14:17:42:21:WU08:FS01:0x21:Completed 10000 out of 1000000 steps (1%)
2019-10-14:17:42:39:WU07:FS01:Upload 0.50%
2019-10-14:17:42:39:WARNING:WU07:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:42:39:WU07:FS01:Trying to send results to collection server
2019-10-14:17:42:39:WU07:FS01:Uploading 74.81MiB to 155.247.166.220
2019-10-14:17:42:39:WU07:FS01:Connecting to 155.247.166.220:8080
2019-10-14:17:42:40:ERROR:WU07:FS01:Exception: Transfer failed
2019-10-14:17:42:40:WU07:FS01:Sending unit results: id:07 state:SEND error:NO_ERROR project:14275 run:0 clone:96 gen:27 core:0x21 unit:0x0000001b80fccb0a5d98ab5af35ced5a
2019-10-14:17:42:40:WU07:FS01:Uploading 74.81MiB to 128.252.203.10
2019-10-14:17:42:40:WU07:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:42:57:WU08:FS01:0x21:Completed 20000 out of 1000000 steps (2%)
2019-10-14:17:43:38:WU08:FS01:0x21:Completed 30000 out of 1000000 steps (3%)
2019-10-14:17:44:13:WU08:FS01:0x21:Completed 40000 out of 1000000 steps (4%)
2019-10-14:17:44:44:WU07:FS01:Upload 0.08%
2019-10-14:17:44:44:WARNING:WU07:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:44:44:WU07:FS01:Trying to send results to collection server
2019-10-14:17:44:44:WU07:FS01:Uploading 74.81MiB to 155.247.166.220
2019-10-14:17:44:44:WU07:FS01:Connecting to 155.247.166.220:8080
2019-10-14:17:44:45:ERROR:WU07:FS01:Exception: Transfer failed
2019-10-14:17:44:45:WU07:FS01:Sending unit results: id:07 state:SEND error:NO_ERROR project:14275 run:0 clone:96 gen:27 core:0x21 unit:0x0000001b80fccb0a5d98ab5af35ced5a
2019-10-14:17:44:45:WU07:FS01:Uploading 74.81MiB to 128.252.203.10
2019-10-14:17:44:45:WU07:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:44:54:WU08:FS01:0x21:Completed 50000 out of 1000000 steps (5%)
2019-10-14:17:45:09:WU07:FS01:Upload 0.08%
2019-10-14:17:45:29:WU08:FS01:0x21:Completed 60000 out of 1000000 steps (6%)
2019-10-14:17:46:10:WU08:FS01:0x21:Completed 70000 out of 1000000 steps (7%)
2019-10-14:17:46:45:WU08:FS01:0x21:Completed 80000 out of 1000000 steps (8%)
2019-10-14:17:47:26:WU08:FS01:0x21:Completed 90000 out of 1000000 steps (9%)
2019-10-14:17:47:37:WU07:FS01:Upload 0.17%
2019-10-14:17:47:37:WARNING:WU07:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:47:37:WU07:FS01:Trying to send results to collection server
2019-10-14:17:47:37:WU07:FS01:Uploading 74.81MiB to 155.247.166.220
2019-10-14:17:47:37:WU07:FS01:Connecting to 155.247.166.220:8080
2019-10-14:17:47:38:ERROR:WU07:FS01:Exception: Transfer failed
2019-10-14:17:47:38:WU07:FS01:Sending unit results: id:07 state:SEND error:NO_ERROR project:14275 run:0 clone:96 gen:27 core:0x21 unit:0x0000001b80fccb0a5d98ab5af35ced5a
2019-10-14:17:47:38:WU07:FS01:Uploading 74.81MiB to 128.252.203.10
2019-10-14:17:47:38:WU07:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:47:40:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14276 run:0 clone:1 gen:12 core:0x21 unit:0x0000000d80fccb0a5d98ab5bc8be3c80
2019-10-14:17:47:40:WU06:FS01:Uploading 81.46MiB to 128.252.203.10
2019-10-14:17:47:40:WU06:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:47:48:WU07:FS01:Upload 0.08%
2019-10-14:17:48:01:WU08:FS01:0x21:Completed 100000 out of 1000000 steps (10%)
2019-10-14:17:48:18:WU07:FS01:Upload 0.25%
2019-10-14:17:48:42:WU08:FS01:0x21:Completed 110000 out of 1000000 steps (11%)
2019-10-14:17:48:43:WU07:FS01:Upload 0.33%
2019-10-14:17:48:48:WU06:FS01:Upload 0.08%
2019-10-14:17:49:17:WU08:FS01:0x21:Completed 120000 out of 1000000 steps (12%)
2019-10-14:17:49:58:WU08:FS01:0x21:Completed 130000 out of 1000000 steps (13%)
2019-10-14:17:50:30:WU06:FS01:Upload 0.15%
2019-10-14:17:50:30:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:50:30:WU06:FS01:Trying to send results to collection server
2019-10-14:17:50:30:WU06:FS01:Uploading 81.46MiB to 155.247.166.219
2019-10-14:17:50:30:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-14:17:50:34:WU08:FS01:0x21:Completed 140000 out of 1000000 steps (14%)
2019-10-14:17:50:38:WU06:FS01:Upload 0.08%
2019-10-14:17:50:38:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-14:17:50:39:WU07:FS01:Upload 0.50%
2019-10-14:17:50:39:WARNING:WU07:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-14:17:50:39:WU07:FS01:Trying to send results to collection server
2019-10-14:17:50:39:WU07:FS01:Uploading 74.81MiB to 155.247.166.220
2019-10-14:17:50:39:WU07:FS01:Connecting to 155.247.166.220:8080
2019-10-14:17:50:40:ERROR:WU07:FS01:Exception: Transfer failed
2019-10-14:17:50:40:WU07:FS01:Sending unit results: id:07 state:SEND error:NO_ERROR project:14275 run:0 clone:96 gen:27 core:0x21 unit:0x0000001b80fccb0a5d98ab5af35ced5a
2019-10-14:17:50:40:WU07:FS01:Uploading 74.81MiB to 128.252.203.10
2019-10-14:17:50:40:WU07:FS01:Connecting to 128.252.203.10:8080
2019-10-14:17:50:46:WU07:FS01:Upload 0.08%
2019-10-14:17:51:14:WU08:FS01:0x21:Completed 150000 out of 1000000 steps (15%)
2019-10-14:17:51:50:WU08:FS01:0x21:Completed 160000 out of 1000000 steps (16%)
2019-10-14:17:51:55:WU07:FS01:Upload 0.17%

Posted: **Mon Oct 14, 2019 7:11 pm**

It does appear that your system is able to download and process WUs faster than you internet connection can upload them. (Correct me if I'm wrong.) If so, the build-up of a queue is to be expected. Perhaps you can identify the data rates that are actually obtained.

You'll need to find a pattern associated with (A) the FAH server destination (B) perhaps with the time of day (C) anything else that may be applicable.

Do other applications use significant bandwidth during certain times of the day?

'Detected clock skew' means that your FAH client is being suspended. Is your OS configured to hibernate or does it continue to process 24x7? Certainly if an upload starts before the system enters a sleeping state, the WU will not finish uploading unless you allow it to by allowing FAHClient and the processer to remain active. You also may choose to use FINISH or PAUSE judiciously to manually manage what is happening.

Posted: **Tue Oct 15, 2019 3:27 am**

Can the Temple servers be put on a cron job to reboot them every 2 days?

For the first few days after the collection servers have been rebooted, my connection can handle the uploads just fine. I'm noticing after the FAH servers reboot, even with lots of WU uploads even the Windows machine isn't getting failed uploads. When the backlog of WUs occurs, then they are failing to upload often.

It appears that somewhere between 15-25 concurrent uploads is the max for my connection. But, there is currently no way to limit the number of uploads in the FAH Client [or is there???]. I have about 12-15 clients running where each one can have up to 5 uploads, and no good way to limit it, to prevent this mess.

Normally, I have 100+ KB/s overhead on top of the FAH WU uploads. When the servers slow down after 2-4 days of uptime, then that's when the slowdowns and WU upload backlog occurs.
For Internet usage: I know the couple of apps that use the Internet connection, and I have to turn them off once the backlog of WUs start. These apps run just fine normally, until the FAH collection servers slow down.

While I had everything turned off, I did a few connection speed tests. Mid-day (slowest), this was a consistent result:
Google's Speed Test: 20.8 Mbps download, 3.12 Mbps upload, Latency: 29 ms, to a server in: San Francisco Bay Area, CA
https://www.speedtest.net/: 19.20 Mbps Download, 3.26 Mbps Upload, 12ms ping (to a more local server)

For clock skew: the PCs are dedicated FAH machines and run 24x7, until the power goes out. The clock skew message only seems to appear with too many slow uploads failing while folding (just an observation to note).

Folding Forum

Multiple WU's Fail downld/upld to 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*

Re: Multiple WU's Fail to Upload 155.247.166.*