Cant upload the WU in the time allowed.

Nick200 · Post by **Nick200** » Fri Mar 25, 2016 4:38 am

Hi

Yet another weekend with a server refusing to accept completed WUs.

This time it is 140.163.4.241. While the server page indicates it is functioning OK, I cannot complete an upload. It tends to get to between 6% and 40% and then will drop out.

See log file

Code: Select all

*********************** Log Started 2016-03-25T04:15:57Z ***********************
04:15:57:************************* Folding@home Client *************************
04:15:57:      Website: http://folding.stanford.edu/
04:15:57:    Copyright: (c) 2009-2014 Stanford University
04:15:57:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:15:57:         Args: 
04:15:57:       Config: C:/Users/nickm/AppData/Roaming/FAHClient/config.xml
04:15:57:******************************** Build ********************************
04:15:57:      Version: 7.4.4
04:15:57:         Date: Mar 4 2014
04:15:57:         Time: 20:26:54
04:15:57:      SVN Rev: 4130
04:15:57:       Branch: fah/trunk/client
04:15:57:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
04:15:57:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
04:15:57:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
04:15:57:     Platform: win32 XP
04:15:57:         Bits: 32
04:15:57:         Mode: Release
04:15:57:******************************* System ********************************
04:15:57:          CPU: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
04:15:57:       CPU ID: GenuineIntel Family 6 Model 60 Stepping 3
04:15:57:         CPUs: 8
04:15:57:       Memory: 15.85GiB
04:15:57:  Free Memory: 12.19GiB
04:15:57:      Threads: WINDOWS_THREADS
04:15:57:   OS Version: 6.2
04:15:57:  Has Battery: false
04:15:57:   On Battery: false
04:15:57:   UTC Offset: 13
04:15:57:          PID: 7420
04:15:57:          CWD: C:/Users/nickm/AppData/Roaming/FAHClient
04:15:57:           OS: Windows 10 Pro Insider Preview
04:15:57:      OS Arch: AMD64
04:15:57:         GPUs: 2
04:15:57:        GPU 0: NVIDIA:5 GM204 [GeForce GTX 980]
04:15:57:        GPU 1: NVIDIA:5 GM204 [GeForce GTX 980]
04:15:57:         CUDA: 5.2
04:15:57:  CUDA Driver: 8000
04:15:57:Win32 Service: false
04:15:57:***********************************************************************
04:15:57:<config>
04:15:57:  <!-- Slot Control -->
04:15:57:  <power v='full'/>
04:15:57:
04:15:57:  <!-- User Information -->
04:15:57:  <passkey v='********************************'/>
04:15:57:  <team v='142900'/>
04:15:57:  <user v='Montague-Cripps'/>
04:15:57:
04:15:57:  <!-- Folding Slots -->
04:15:57:  <slot id='0' type='CPU'>
04:15:57:    <paused v='true'/>
04:15:57:  </slot>
04:15:57:  <slot id='1' type='GPU'>
04:15:57:    <paused v='true'/>
04:15:57:  </slot>
04:15:57:  <slot id='2' type='GPU'>
04:15:57:    <paused v='true'/>
04:15:57:  </slot>
04:15:57:</config>
04:15:57:Trying to access database...
04:15:57:Successfully acquired database lock
04:15:57:Enabled folding slot 00: PAUSED cpu:6 (by user)
04:15:57:Enabled folding slot 01: PAUSED gpu:0:GM204 [GeForce GTX 980] (by user)
04:15:57:Enabled folding slot 02: PAUSED gpu:1:GM204 [GeForce GTX 980] (by user)
04:15:58:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10487 run:0 clone:14 gen:246 core:0x18 unit:0x00000142538b3dbb54aee6268137e652
04:15:59:WU01:FS01:Uploading 10.24MiB to 140.163.4.235
04:15:59:WU01:FS01:Connecting to 140.163.4.235:8080
04:16:07:WU01:FS01:Upload 1.22%
04:16:21:WU01:FS01:Upload 1.83%
04:16:31:WU01:FS01:Upload 3.05%
04:16:53:WU01:FS01:Upload 3.66%
04:16:53:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
04:16:53:WU01:FS01:Trying to send results to collection server
04:16:54:WU01:FS01:Uploading 10.24MiB to 140.163.4.241
04:16:54:WU01:FS01:Connecting to 140.163.4.241:8080
04:17:03:WU01:FS01:Upload 1.83%
04:17:09:WU01:FS01:Upload 2.44%
04:17:16:WU01:FS01:Upload 3.05%
04:17:26:WU01:FS01:Upload 4.27%
04:17:37:WU01:FS01:Upload 4.88%
04:18:02:WU01:FS01:Upload 6.10%
04:18:02:ERROR:WU01:FS01:Exception: Transfer failed
04:18:02:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10487 run:0 clone:14 gen:246 core:0x18 unit:0x00000142538b3dbb54aee6268137e652
04:18:02:WU01:FS01:Uploading 10.24MiB to 140.163.4.235
04:18:02:WU01:FS01:Connecting to 140.163.4.235:8080
04:19:13:WU01:FS01:Upload 1.22%
04:19:20:WU01:FS01:Upload 1.83%
04:20:07:WU01:FS01:Upload 3.05%
04:20:18:WU01:FS01:Upload 3.66%
04:20:35:WU01:FS01:Upload 4.27%
04:20:43:WU01:FS01:Upload 4.88%
04:20:45:FS00:Unpaused
04:20:45:FS01:Unpaused
04:20:45:FS02:Unpaused
04:20:45:WU00:FS02:Starting
04:20:45:WU00:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 704 -lifeline 7420 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
04:20:45:WU00:FS02:Started FahCore on PID 10524
04:20:46:WU00:FS02:Core PID:11320
04:20:46:WU00:FS02:FahCore 0x21 started
04:20:46:WU02:FS01:Starting
04:20:46:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_18.fah/FahCore_18.exe -dir 02 -suffix 01 -version 704 -lifeline 7420 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
04:20:46:WU02:FS01:Started FahCore on PID 5164
04:20:46:WU02:FS01:Core PID:10300
04:20:46:WU02:FS01:FahCore 0x18 started
04:20:46:WU03:FS00:Starting
04:20:46:WU03:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 03 -suffix 01 -version 704 -lifeline 7420 -checkpoint 15 -np 6
04:20:46:WU03:FS00:Started FahCore on PID 6324
04:20:46:WU03:FS00:Core PID:7792
04:20:46:WU03:FS00:FahCore 0xa4 started
04:20:47:WU03:FS00:0xa4:
04:20:47:WU03:FS00:0xa4:*------------------------------*
04:20:47:WU03:FS00:0xa4:Folding@Home Gromacs GB Core
04:20:47:WU03:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
04:20:47:WU03:FS00:0xa4:
04:20:47:WU03:FS00:0xa4:Preparing to commence simulation
04:20:47:WU03:FS00:0xa4:- Looking at optimizations...
04:20:47:WU03:FS00:0xa4:- Files status OK
04:20:48:WU03:FS00:0xa4:- Expanded 1216091 -> 2904400 (decompressed 238.8 percent)
04:20:48:WU03:FS00:0xa4:Called DecompressByteArray: compressed_data_size=1216091 data_size=2904400, decompressed_data_size=2904400 diff=0
04:20:48:WU03:FS00:0xa4:- Digital signature verified
04:20:48:WU03:FS00:0xa4:
04:20:48:WU03:FS00:0xa4:Project: 11636 (Run 1, Clone 41, Gen 39)
04:20:48:WU03:FS00:0xa4:
04:20:48:WU03:FS00:0xa4:Assembly optimizations on if available.
04:20:48:WU03:FS00:0xa4:Entering M.D.
04:20:48:WU02:FS01:0x18:*********************** Log Started 2016-03-25T04:20:48Z ***********************
04:20:48:WU02:FS01:0x18:Project: 9135 (Run 32, Clone 0, Gen 406)
04:20:48:WU02:FS01:0x18:Unit: 0x000001df0a3b1e615564ed317fdc2bf7
04:20:48:WU02:FS01:0x18:CPU: 0x00000000000000000000000000000000
04:20:48:WU02:FS01:0x18:Machine: 1
04:20:48:WU02:FS01:0x18:Digital signatures verified
04:20:48:WU02:FS01:0x18:Folding@home GPU core18
04:20:48:WU02:FS01:0x18:Version 0.0.4
04:20:48:WU00:FS02:0x21:*********************** Log Started 2016-03-25T04:20:48Z ***********************
04:20:48:WU00:FS02:0x21:Project: 11407 (Run 6, Clone 3, Gen 108)
04:20:48:WU00:FS02:0x21:Unit: 0x000000828ca304f256805dfe5907ace5
04:20:48:WU00:FS02:0x21:CPU: 0x00000000000000000000000000000000
04:20:48:WU00:FS02:0x21:Machine: 2
04:20:48:WU00:FS02:0x21:Digital signatures verified
04:20:48:WU00:FS02:0x21:Folding@home GPU Core21 Folding@home Core
04:20:48:WU00:FS02:0x21:Version 0.0.17
04:20:48:WU02:FS01:0x18:  Found a checkpoint file
04:20:50:WU01:FS01:Upload 5.49%
04:20:50:WU00:FS02:0x21:  Found a checkpoint file
04:20:54:WU03:FS00:0xa4:Using Gromacs checkpoints
04:20:54:WU03:FS00:0xa4:Mapping NT from 6 to 6 
04:20:55:WU03:FS00:0xa4:Resuming from checkpoint
04:20:55:WU03:FS00:0xa4:Verified 03/wudata_01.log
04:20:56:WU03:FS00:0xa4:Verified 03/wudata_01.trr
04:20:59:WU01:FS01:Upload 6.10%
04:21:00:WU03:FS00:0xa4:Verified 03/wudata_01.xtc
04:21:00:WU03:FS00:0xa4:Verified 03/wudata_01.edr
04:21:00:WU03:FS00:0xa4:Completed 576980 out of 1250000 steps  (46%)
04:21:02:Removing old file 'configs/config-20160312-023135.xml'
04:21:02:Saving configuration to config.xml
04:21:02:<config>
04:21:02:  <!-- Slot Control -->
04:21:02:  <power v='full'/>
04:21:02:
04:21:02:  <!-- User Information -->
04:21:02:  <passkey v='********************************'/>
04:21:02:  <team v='142900'/>
04:21:02:  <user v='Montague-Cripps'/>
04:21:02:
04:21:02:  <!-- Folding Slots -->
04:21:02:  <slot id='0' type='CPU'/>
04:21:02:  <slot id='1' type='GPU'/>
04:21:02:  <slot id='2' type='GPU'/>
04:21:02:</config>
04:21:04:WU00:FS02:0x21:Completed 4500000 out of 5000000 steps (90%)
04:21:04:WU00:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
04:21:07:WU01:FS01:Upload 6.71%
04:21:17:WU01:FS01:Upload 7.93%
04:21:21:WU02:FS01:0x18:Completed 700000 out of 2500000 steps (28%)
04:21:21:WU02:FS01:0x18:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
04:21:24:WU01:FS01:Upload 8.54%
04:21:32:WU01:FS01:Upload 9.15%
04:21:42:WU01:FS01:Upload 9.76%
04:21:54:WU01:FS01:Upload 10.98%
04:22:05:WU01:FS01:Upload 12.21%
04:22:13:WU01:FS01:Upload 13.43%
04:22:21:WU01:FS01:Upload 14.04%
04:22:32:WU01:FS01:Upload 14.65%
04:22:40:WU01:FS01:Upload 15.87%
04:22:51:WU01:FS01:Upload 17.09%
04:23:00:WU01:FS01:Upload 19.53%
04:23:05:WU02:FS01:0x18:Completed 725000 out of 2500000 steps (29%)
04:23:09:WU01:FS01:Upload 21.36%
04:23:15:WU01:FS01:Upload 22.58%
04:23:27:WU01:FS01:Upload 23.80%
04:23:36:WU01:FS01:Upload 25.63%
04:23:42:WU01:FS01:Upload 26.85%
04:23:46:WU00:FS02:0x21:Completed 4550000 out of 5000000 steps (91%)
04:23:53:WU01:FS01:Upload 28.07%
04:23:59:WU01:FS01:Upload 28.68%
04:24:06:WU01:FS01:Upload 29.29%
04:24:15:WU01:FS01:Upload 30.51%
04:24:29:WU01:FS01:Upload 31.73%
04:24:39:WU02:FS01:0x18:Completed 750000 out of 2500000 steps (30%)
04:26:08:WU01:FS01:Upload 32.34%
04:26:14:WU02:FS01:0x18:Completed 775000 out of 2500000 steps (31%)
04:26:16:WU01:FS01:Upload 33.56%
04:26:22:WU01:FS01:Upload 34.18%
04:26:26:WU00:FS02:0x21:Completed 4600000 out of 5000000 steps (92%)
04:26:32:WU01:FS01:Upload 34.79%
04:26:40:WU01:FS01:Upload 36.01%
04:26:48:WU01:FS01:Upload 37.23%
04:27:00:WU01:FS01:Upload 37.84%
04:27:09:WU01:FS01:Upload 39.67%
04:27:17:WU01:FS01:Upload 40.28%
04:27:23:WU01:FS01:Upload 40.89%
04:27:43:WU01:FS01:Upload 41.50%
04:27:43:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
04:27:43:WU01:FS01:Trying to send results to collection server
04:27:43:WU01:FS01:Uploading 10.24MiB to 140.163.4.241
04:27:43:WU01:FS01:Connecting to 140.163.4.241:8080
04:27:48:WU03:FS00:0xa4:Completed 587500 out of 1250000 steps  (47%)
04:27:49:WU02:FS01:0x18:Completed 800000 out of 2500000 steps (32%)
04:27:52:WU01:FS01:Upload 1.22%
04:27:59:WU01:FS01:Upload 1.83%
04:29:10:WU00:FS02:0x21:Completed 4650000 out of 5000000 steps (93%)
04:29:17:WU01:FS01:Upload 2.44%
04:29:17:ERROR:WU01:FS01:Exception: Transfer failed
04:29:17:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10487 run:0 clone:14 gen:246 core:0x18 unit:0x00000142538b3dbb54aee6268137e652
04:29:17:WU01:FS01:Uploading 10.24MiB to 140.163.4.235
04:29:17:WU01:FS01:Connecting to 140.163.4.235:8080
04:29:35:WU01:FS01:Upload 1.22%
04:29:37:WU02:FS01:0x18:Completed 825000 out of 2500000 steps (33%)
04:29:46:WU01:FS01:Upload 2.44%
04:29:57:WU01:FS01:Upload 3.66%
04:30:03:WU01:FS01:Upload 4.27%
04:30:11:WU01:FS01:Upload 4.88%
04:30:28:WU01:FS01:Upload 6.71%
04:30:35:WU01:FS01:Upload 7.32%
04:30:44:WU01:FS01:Upload 7.93%
04:31:08:WU01:FS01:Upload 8.54%
04:31:12:WU02:FS01:0x18:Completed 850000 out of 2500000 steps (34%)
04:31:17:WU01:FS01:Upload 9.15%
04:31:26:WU01:FS01:Upload 9.76%
04:31:46:WU01:FS01:Upload 10.37%
04:31:47:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
04:31:47:WU01:FS01:Trying to send results to collection server
04:31:47:WU01:FS01:Uploading 10.24MiB to 140.163.4.241
04:31:47:WU01:FS01:Connecting to 140.163.4.241:8080
04:31:51:WU00:FS02:0x21:Completed 4700000 out of 5000000 steps (94%)
04:31:56:WU01:FS01:Upload 1.22%
04:32:04:WU01:FS01:Upload 1.83%
04:32:12:WU01:FS01:Upload 2.44%
04:32:24:WU01:FS01:Upload 3.05%
04:32:34:WU01:FS01:Upload 3.66%
04:32:45:WU01:FS01:Upload 4.88%
04:32:48:WU02:FS01:0x18:Completed 875000 out of 2500000 steps (35%)
04:32:52:WU01:FS01:Upload 5.49%
04:33:13:WU01:FS01:Upload 6.10%
04:33:19:WU01:FS01:Upload 6.71%
04:33:33:WU01:FS01:Upload 7.93%
04:33:43:WU01:FS01:Upload 8.54%
04:33:49:WU01:FS01:Upload 9.15%
04:33:57:WU01:FS01:Upload 10.37%
04:34:04:WU01:FS01:Upload 10.98%
04:34:20:WU01:FS01:Upload 12.21%
04:34:23:WU02:FS01:0x18:Completed 900000 out of 2500000 steps (36%)
04:34:30:WU01:FS01:Upload 12.82%
04:34:32:WU00:FS02:0x21:Completed 4750000 out of 5000000 steps (95%)
04:34:37:WU01:FS01:Upload 13.43%
04:35:00:WU03:FS00:0xa4:Completed 600000 out of 1250000 steps  (48%)

I have tried to ping it but get repeated time-outs. I have no other internet or broadband issues n case you're wondering

Any suggestions?

Post by **toTOW** » Fri Mar 25, 2016 3:54 pm

Server at MSKCC (140.163.4.*) don't answer to pings (ICMP traffic). Try tracert/traceroute (which will stop responding when entering MSKCC networl) or MTR instead.

Nick200 · Post by **Nick200** » Fri Mar 25, 2016 9:35 pm

So, another folding machine is having upload problems, this time to 128.252.203.2 as far as I can tell

Log file attached below:

Code: Select all

06:28:20:WU01:FS00:Upload 13.00%
06:28:26:WU01:FS00:Upload 13.23%
06:28:36:WU01:FS00:Upload 13.67%
06:28:38:WU03:FS00:0xa4:Completed 3700000 out of 5000000 steps  (74%)
06:28:42:WU01:FS00:Upload 13.90%
06:28:50:WU01:FS00:Upload 14.35%
06:29:10:WU01:FS00:Upload 14.57%
06:29:19:WU01:FS00:Upload 14.80%
06:29:27:WU01:FS00:Upload 15.24%
06:29:40:WU01:FS00:Upload 15.69%
06:29:54:WU01:FS00:Upload 16.14%
06:30:07:WU01:FS00:Upload 16.59%
06:30:26:WU01:FS00:Upload 16.81%
06:30:50:WU01:FS00:Upload 17.04%
06:31:00:WU03:FS00:0xa4:Completed 3750000 out of 5000000 steps  (75%)
06:31:11:WU01:FS00:Upload 17.26%
06:31:20:WU01:FS00:Upload 17.49%
06:31:28:WU01:FS00:Upload 17.71%
06:31:35:WU01:FS00:Upload 17.93%
06:31:43:WU01:FS00:Upload 18.38%
06:31:54:WU01:FS00:Upload 18.83%
06:32:00:WU01:FS00:Upload 19.05%
06:32:08:WU01:FS00:Upload 19.50%
06:32:17:WU01:FS00:Upload 19.73%
06:32:27:WU01:FS00:Upload 20.18%
06:32:33:WU01:FS00:Upload 20.40%
06:33:12:WU01:FS00:Upload 20.62%
06:33:20:WU03:FS00:0xa4:Completed 3800000 out of 5000000 steps  (76%)
06:33:26:WU01:FS00:Upload 21.07%
06:33:32:WU01:FS00:Upload 21.30%
06:33:44:WU01:FS00:Upload 21.74%
06:33:51:WU01:FS00:Upload 22.19%


06:33:57:WU01:FS00:Upload 22.42%
06:34:05:WU01:FS00:Upload 22.64%
06:34:18:WU01:FS00:Upload 23.09%
06:34:25:WU01:FS00:Upload 23.31%
06:34:36:WU01:FS00:Upload 23.76%
06:34:43:WU01:FS00:Upload 24.21%
06:34:58:WU01:FS00:Upload 24.43%
06:35:05:WU01:FS00:Upload 24.66%
06:35:33:WU01:FS00:Upload 25.11%

06:35:42:WU01:FS00:Upload 25.33%
06:35:50:WU01:FS00:Upload 25.78%
06:35:57:WU01:FS00:Upload 26.00%
06:36:03:WU01:FS00:Upload 26.23%
06:36:11:WU01:FS00:Upload 26.45%
06:36:18:WU01:FS00:Upload 26.68%
06:36:34:WU01:FS00:Upload 26.90%
06:36:45:WU01:FS00:Upload 27.35%
06:36:52:WU01:FS00:Upload 28.02%
06:37:02:WU01:FS00:Upload 28.47%
06:37:11:WU01:FS00:Upload 28.92%

06:37:43:WU01:FS00:Upload 29.14%
06:37:51:WU01:FS00:Upload 29.59%
06:37:58:WU01:FS00:Upload 30.04%

06:38:11:WU01:FS00:Upload 30.49%
06:38:18:WU01:FS00:Upload 30.71%
06:38:28:WU01:FS00:Upload 30.94%
06:38:38:WU01:FS00:Upload 31.16%
06:38:48:WU01:FS00:Upload 31.38%
06:38:55:WU01:FS00:Upload 31.61%
06:39:04:WU01:FS00:Upload 32.06%
06:39:10:WU01:FS00:Upload 32.28%
06:39:25:WU01:FS00:Upload 32.50%
06:39:31:WU01:FS00:Upload 32.73%
06:39:40:WU01:FS00:Upload 32.95%
06:40:08:WU01:FS00:Upload 33.18%
06:40:15:WU01:FS00:Upload 33.40%

06:40:23:WU01:FS00:Upload 33.63%
06:40:35:WU01:FS00:Upload 34.07%
06:40:41:WU01:FS00:Upload 34.30%
06:40:49:WU01:FS00:Upload 34.52%
06:41:13:WU01:FS00:Upload 34.97%
06:41:21:WU01:FS00:Upload 35.19%

06:48:11:WU01:FS00:Upload 35.42%
06:48:11:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed

07:02:49:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:11633 run:1 clone:76 gen:19 core:0xa4 unit:0x0000001a4c71bbb056987d04dabf9fd9
07:02:49:WU01:FS00:Uploading 27.88MiB to 128.252.203.2
07:02:49:WU01:FS00:Connecting to 128.252.203.2:8080
07:03:00:WU01:FS00:Upload 0.45%
07:03:06:WU01:FS00:Upload 0.67%
07:03:17:WU01:FS00:Upload 1.12%
07:03:32:WU01:FS00:Upload 1.57%
07:03:39:WU01:FS00:Upload 2.02%
07:03:46:WU01:FS00:Upload 2.47%
07:03:54:WU01:FS00:Upload 2.69%

07:04:01:WU01:FS00:Upload 2.91%
07:04:07:WU01:FS00:Upload 3.14%
07:04:13:WU01:FS00:Upload 3.36%
07:04:24:WU01:FS00:Upload 3.59%
07:04:37:WU01:FS00:Upload 4.04%
07:04:45:WU01:FS00:Upload 4.26%
07:04:52:WU01:FS00:Upload 4.71%
07:05:09:WU01:FS00:Upload 5.16%
07:05:15:WU01:FS00:Upload 5.38%
07:05:29:WU01:FS00:Upload 5.83%
07:05:39:WU01:FS00:Upload 6.28%
07:05:46:WU01:FS00:Upload 6.95%
07:06:02:WU01:FS00:Upload 7.40%
07:06:14:WU01:FS00:Upload 7.85%

07:06:23:WU01:FS00:Upload 8.29%
07:06:31:WU01:FS00:Upload 8.52%
07:06:39:WU01:FS00:Upload 8.74%
07:06:46:WU01:FS00:Upload 9.19%
07:06:53:WU01:FS00:Upload 9.42%
07:07:00:WU01:FS00:Upload 9.86%
07:07:09:WU01:FS00:Upload 10.09%
07:07:17:WU01:FS00:Upload 10.54%
07:07:26:WU01:FS00:Upload 10.98%
07:07:37:WU01:FS00:Upload 11.43%
07:07:49:WU01:FS00:Upload 11.88%
07:07:59:WU01:FS00:Upload 12.33%
07:08:10:WU01:FS00:Upload 12.78%
07:08:18:WU01:FS00:Upload 13.23%
07:08:27:WU01:FS00:Upload 13.67%
07:08:38:WU01:FS00:Upload 14.35%

07:08:46:WU01:FS00:Upload 14.80%

07:08:55:WU01:FS00:Upload 15.24%
07:09:08:WU01:FS00:Upload 15.69%
07:09:16:WU01:FS00:Upload 16.14%
07:09:23:WU01:FS00:Upload 16.36%
07:09:34:WU01:FS00:Upload 16.81%
07:09:41:WU01:FS00:Upload 17.04%
07:09:47:WU01:FS00:Upload 17.26%
07:09:53:WU01:FS00:Upload 17.49%
07:10:02:WU01:FS00:Upload 17.93%
07:10:08:WU01:FS00:Upload 18.16%
07:10:15:WU01:FS00:Upload 18.61%
07:10:24:WU01:FS00:Upload 19.05%
07:10:35:WU01:FS00:Upload 19.50%
07:10:48:WU01:FS00:Upload 19.95%
07:10:55:WU01:FS00:Upload 20.18%

07:11:08:WU01:FS00:Upload 20.62%
07:11:15:WU01:FS00:Upload 21.07%
07:11:21:WU01:FS00:Upload 21.30%
07:11:28:WU01:FS00:Upload 21.74%
07:11:36:WU01:FS00:Upload 22.19%
07:11:44:WU01:FS00:Upload 22.42%
07:11:53:WU01:FS00:Upload 22.64%
07:12:00:WU01:FS00:Upload 22.87%
07:12:07:WU01:FS00:Upload 23.31%
07:12:16:WU01:FS00:Upload 23.76%
07:12:23:WU01:FS00:Upload 24.21%
07:12:31:WU01:FS00:Upload 24.66%
07:13:07:WU01:FS00:Upload 24.88%
07:13:14:WU01:FS00:Upload 25.11%

07:13:27:WU01:FS00:Upload 25.56%
07:13:39:WU01:FS00:Upload 26.00%
07:13:47:WU01:FS00:Upload 26.45%
07:13:53:WU01:FS00:Upload 26.68%
07:14:02:WU01:FS00:Upload 27.12%
07:14:10:WU01:FS00:Upload 27.35%
07:14:17:WU01:FS00:Upload 27.80%
07:14:25:WU01:FS00:Upload 28.25%
07:14:31:WU01:FS00:Upload 28.47%
07:14:42:WU01:FS00:Upload 28.69%
07:15:08:WU01:FS00:Upload 28.92%
07:15:22:WU01:FS00:Upload 29.14%
07:15:29:WU01:FS00:Upload 29.37%

07:15:48:WU01:FS00:Upload 29.59%

07:15:58:WU01:FS00:Upload 30.04%
07:16:05:WU01:FS00:Upload 30.49%
07:16:13:WU01:FS00:Upload 30.94%
07:16:27:WU01:FS00:Upload 31.61%
07:16:40:WU01:FS00:Upload 32.06%
07:16:50:WU01:FS00:Upload 32.28%
07:16:56:WU01:FS00:Upload 32.50%
07:17:06:WU01:FS00:Upload 32.95%
07:17:21:WU01:FS00:Upload 33.18%
07:17:35:WU01:FS00:Upload 33.40%
07:17:53:WU01:FS00:Upload 33.63%

07:18:04:WU01:FS00:Upload 33.85%
07:18:12:WU01:FS00:Upload 34.52%
07:18:21:WU01:FS00:Upload 34.75%
07:18:29:WU01:FS00:Upload 34.97%
07:18:41:WU01:FS00:Upload 35.42%
07:18:55:WU01:FS00:Upload 35.64%
07:19:03:WU01:FS00:Upload 35.87%
07:19:09:WU01:FS00:Upload 36.09%
07:19:21:WU01:FS00:Upload 36.32%

07:19:32:WU01:FS00:Upload 36.54%
07:19:42:WU01:FS00:Upload 36.99%
07:19:51:WU01:FS00:Upload 37.21%
07:19:57:WU01:FS00:Upload 37.44%

07:23:31:WU01:FS00:Upload 37.66%
07:23:31:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed

07:27:37:WU04:FS00:Connecting to 171.67.108.45:8080
07:27:38:WU04:FS00:Assigned to work server 128.252.203.2
07:27:38:WU04:FS00:Requesting new work unit for slot 00: RUNNING cpu:6 from 128.252.203.2
07:27:38:WU04:FS00:Connecting to 128.252.203.2:8080
07:27:39:WU04:FS00:Downloading 1.16MiB
07:27:42:WU04:FS00:Download complete
07:27:42:WU04:FS00:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74

07:29:58:WU03:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
07:30:08:WU03:FS00:0xa4:
07:30:08:WU03:FS00:0xa4:Finished Work Unit:
07:30:08:WU03:FS00:0xa4:- Reading up to 1255848 from "03/wudata_01.trr": Read 1255848
07:30:08:WU03:FS00:0xa4:trr file hash check passed.
07:30:08:WU03:FS00:0xa4:- Reading up to 112656 from "03/wudata_01.xtc": Read 112656
07:30:08:WU03:FS00:0xa4:xtc file hash check passed.
07:30:08:WU03:FS00:0xa4:edr file hash check passed.
07:30:08:WU03:FS00:0xa4:logfile size: 87021
07:30:08:WU03:FS00:0xa4:Leaving Run
07:30:10:WU03:FS00:0xa4:- Writing 1527425 bytes of core data to disk...
07:30:11:WU03:FS00:0xa4:Done: 1526913 -> 1293376 (compressed to 84.7 percent)
07:30:11:WU03:FS00:0xa4:  ... Done.
07:30:11:WU03:FS00:0xa4:- Shutting down core
07:30:11:WU03:FS00:0xa4:
07:30:11:WU03:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
07:30:11:WU03:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
07:30:11:WU03:FS00:Sending unit results: id:03 state:SEND error:NO_ERROR project:6399 run:62 clone:20 gen:91 core:0xa4 unit:0x0000006a0002894b5462cc587635b3d2
07:30:11:WU03:FS00:Uploading 1.23MiB to 155.247.166.219
07:30:11:WU03:FS00:Connecting to 155.247.166.219:8080

13:40:50:WU01:FS00:Uploading 27.88MiB to 128.252.203.2
13:40:50:WU01:FS00:Connecting to 128.252.203.2:8080
13:40:59:WU01:FS00:Upload 0.45%
13:41:07:WU01:FS00:Upload 0.67%
13:41:16:WU01:FS00:Upload 1.12%
13:41:22:WU01:FS00:Upload 1.35%
13:41:30:WU01:FS00:Upload 1.57%
13:41:46:WU01:FS00:Upload 1.79%
13:41:58:WU01:FS00:Upload 2.24%
13:42:11:WU01:FS00:Upload 2.69%
13:42:19:WU01:FS00:Upload 2.91%
13:42:28:WU01:FS00:Upload 3.14%
13:42:40:WU01:FS00:Upload 3.59%
13:42:48:WU01:FS00:Upload 4.04%
13:42:58:WU01:FS00:Upload 4.26%
13:43:04:WU01:FS00:Upload 4.48%
13:43:13:WU01:FS00:Upload 4.93%
13:43:21:WU01:FS00:Upload 5.16%
13:43:27:WU01:FS00:Upload 5.38%
13:43:38:WU01:FS00:Upload 5.60%
13:43:44:WU01:FS00:Upload 5.83%
13:43:55:WU01:FS00:Upload 6.05%
13:44:31:WU01:FS00:Upload 6.50%
13:44:42:WU01:FS00:Upload 6.95%
13:44:57:WU01:FS00:Upload 7.40%
13:45:04:WU01:FS00:Upload 7.62%
13:45:10:WU01:FS00:Upload 7.85%
13:45:24:WU01:FS00:Upload 8.07%
13:46:03:WU01:FS00:Upload 8.29%

13:47:03:WU01:FS00:Upload 8.52%
13:47:03:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed


18:05:59:WU04:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
18:06:00:WU04:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:06:00:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
18:06:00:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
18:06:00:WU04:FS00:Connecting to 128.252.203.2:8080
18:06:00:WU02:FS00:Starting
18:06:00:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 704 -lifeline 7920 -checkpoint 15 -np 6
18:06:00:WU02:FS00:Started FahCore on PID 7712
18:06:00:WU02:FS00:Core PID:4536
18:06:00:WU02:FS00:FahCore 0xa4 started
18:06:01:WU02:FS00:0xa4:
18:06:01:WU02:FS00:0xa4:*------------------------------*
18:06:01:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
18:06:01:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
18:06:01:WU02:FS00:0xa4:
18:06:01:WU02:FS00:0xa4:Preparing to commence simulation
18:06:01:WU02:FS00:0xa4:- Looking at optimizations...
18:06:01:WU02:FS00:0xa4:- Created dyn
18:06:01:WU02:FS00:0xa4:- Files status OK
18:06:01:WU02:FS00:0xa4:- Expanded 1215922 -> 2904180 (decompressed 238.8 percent)
18:06:01:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=1215922 data_size=2904180, decompressed_data_size=2904180 diff=0
18:06:01:WU02:FS00:0xa4:- Digital signature verified
18:06:01:WU02:FS00:0xa4:
18:06:01:WU02:FS00:0xa4:Project: 11632 (Run 1, Clone 52, Gen 28)
18:06:01:WU02:FS00:0xa4:
18:06:01:WU02:FS00:0xa4:Assembly optimizations on if available.
18:06:01:WU02:FS00:0xa4:Entering M.D.
18:06:06:WU04:FS00:Upload 0.45%
18:06:06:WU02:FS00:0xa4:Mapping NT from 6 to 6 

18:07:03:WU04:FS00:Upload 0.67%
18:07:13:WU04:FS00:Upload 1.12%
18:07:19:WU04:FS00:Upload 1.35%
18:07:28:WU04:FS00:Upload 1.57%

18:07:45:WU04:FS00:Upload 2.02%
18:08:02:WU04:FS00:Upload 2.47%
18:09:03:WU04:FS00:Upload 2.69%
18:09:19:WU04:FS00:Upload 2.92%
18:09:27:WU04:FS00:Upload 3.14%
18:09:34:WU04:FS00:Upload 3.37%
18:09:46:WU04:FS00:Upload 3.59%
18:09:52:WU04:FS00:Upload 3.82%
18:10:02:WU04:FS00:Upload 4.27%
18:10:09:WU04:FS00:Upload 4.49%
18:10:15:WU04:FS00:Upload 4.72%
18:10:24:WU04:FS00:Upload 5.17%
18:10:36:WU04:FS00:Upload 5.39%
18:11:37:WU03:FS02:0x18:Completed 1225000 out of 2500000 steps (49%)
18:12:08:WU02:FS00:0xa4:Completed 12500 out of 1250000 steps  (1%)
18:18:10:WU04:FS00:Upload 5.61%
18:18:10:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
18:18:11:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
18:18:11:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
18:18:11:WU04:FS00:Connecting to 128.252.203.2:8080
18:18:16:WU02:FS00:0xa4:Completed 25000 out of 1250000 steps  (2%)
18:18:23:WU04:FS00:Upload 0.45%
18:18:30:WU04:FS00:Upload 0.67%
18:19:10:WU04:FS00:Upload 0.90%
18:19:20:WU04:FS00:Upload 1.35%
18:19:31:WU04:FS00:Upload 1.57%
18:19:48:WU04:FS00:Upload 1.80%
18:19:54:WU04:FS00:Upload 2.02%
18:20:00:WU04:FS00:Upload 2.25%
18:20:13:WU04:FS00:Upload 2.69%
18:20:22:WU04:FS00:Upload 3.14%
18:20:29:WU04:FS00:Upload 3.37%
18:20:41:WU04:FS00:Upload 4.04%
18:20:53:WU04:FS00:Upload 4.27%
18:21:04:WU04:FS00:Upload 4.49%
18:21:14:WU04:FS00:Upload 4.72%
18:21:22:WU04:FS00:Upload 5.17%
18:21:29:WU04:FS00:Upload 5.39%
18:21:40:WU04:FS00:Upload 5.84%

18:21:48:WU04:FS00:Upload 6.06%
18:21:54:WU04:FS00:Upload 6.29%
18:22:00:WU04:FS00:Upload 6.74%
18:22:14:WU04:FS00:Upload 7.19%
18:22:20:WU04:FS00:Upload 7.64%
18:22:30:WU04:FS00:Upload 8.08%
18:22:39:WU04:FS00:Upload 8.31%
18:22:47:WU04:FS00:Upload 8.76%
18:22:56:WU04:FS00:Upload 9.21%
18:23:11:WU04:FS00:Upload 9.43%
18:23:18:WU04:FS00:Upload 9.66%
18:23:27:WU04:FS00:Upload 9.88%
18:23:38:WU04:FS00:Upload 10.33%
18:23:53:WU04:FS00:Upload 10.78%
18:24:00:WU04:FS00:Upload 11.00%
18:24:16:WU04:FS00:Upload 11.23%

18:24:32:WU04:FS00:Upload 11.45%
18:24:47:WU04:FS00:Upload 11.90%

18:24:57:WU04:FS00:Upload 12.35%
18:25:03:WU04:FS00:Upload 12.58%
18:25:47:WU04:FS00:Upload 12.80%
18:25:54:WU04:FS00:Upload 13.03%
18:26:06:WU04:FS00:Upload 13.25%
18:26:20:WU04:FS00:Upload 13.47%
18:26:30:WU04:FS00:Upload 14.15%
18:26:37:WU04:FS00:Upload 14.37%
18:26:49:WU04:FS00:Upload 14.60%
18:27:03:WU04:FS00:Upload 14.82%
18:27:13:WU04:FS00:Upload 15.27%
18:28:12:WU04:FS00:Upload 15.50%
18:28:18:WU04:FS00:Upload 15.72%
18:28:27:WU04:FS00:Upload 15.94%
18:28:38:WU04:FS00:Upload 16.39%
18:28:44:WU04:FS00:Upload 16.62%
18:28:54:WU04:FS00:Upload 17.07%
18:29:06:WU04:FS00:Upload 17.29%
18:29:17:WU04:FS00:Upload 17.52%
18:29:25:WU04:FS00:Upload 17.74%
18:29:34:WU04:FS00:Upload 17.97%
18:29:43:WU04:FS00:Upload 18.42%
18:29:51:WU04:FS00:Upload 18.64%
18:29:58:WU04:FS00:Upload 18.86%

18:31:29:WU04:FS00:Upload 19.09%
18:31:36:WU04:FS00:Upload 19.31%
18:31:43:WU03:FS02:0x18:Completed 1275000 out of 2500000 steps (51%)
18:31:47:WU04:FS00:Upload 19.54%
18:32:01:WU04:FS00:Upload 19.76%
18:32:11:WU04:FS00:Upload 20.21%
18:32:21:WU04:FS00:Upload 20.44%
18:32:32:WU04:FS00:Upload 20.89%
18:32:42:WU04:FS00:Upload 21.33%
18:32:50:WU04:FS00:Upload 21.56%
18:33:00:WU04:FS00:Upload 21.78%
18:33:13:WU04:FS00:Upload 22.01%
18:33:20:WU04:FS00:Upload 22.23%
18:33:32:WU04:FS00:Upload 22.68%
18:33:45:WU04:FS00:Upload 23.13%
18:33:52:WU04:FS00:Upload 23.36%
18:33:59:WU04:FS00:Upload 23.58%
18:34:08:WU04:FS00:Upload 23.80%
18:35:24:WU04:FS00:Upload 24.25%
18:36:00:WU04:FS00:Upload 24.48%
18:36:08:WU04:FS00:Upload 24.70%
18:36:14:WU04:FS00:Upload 24.93%
18:36:21:WU04:FS00:Upload 25.15%

18:36:27:WU04:FS00:Upload 25.38%
18:36:36:WU04:FS00:Upload 25.83%
18:36:45:WU04:FS00:Upload 26.28%
18:37:02:WU04:FS00:Upload 26.50%
18:37:13:WU04:FS00:Upload 26.95%
18:37:20:WU04:FS00:Upload 27.17%
18:37:32:WU04:FS00:Upload 27.62%
18:37:42:WU04:FS00:Upload 28.07%
18:37:50:WU04:FS00:Upload 28.52%
18:37:58:WU04:FS00:Upload 28.97%
18:38:11:WU04:FS00:Upload 29.42%
18:38:22:WU04:FS00:Upload 29.87%
18:38:29:WU04:FS00:Upload 30.32%
18:39:10:WU04:FS00:Upload 30.54%
18:39:29:WU04:FS00:Upload 30.99%
18:39:44:WU04:FS00:Upload 31.22%
18:40:02:WU04:FS00:Upload 31.44%
18:40:10:WU04:FS00:Upload 31.89%
18:40:19:WU04:FS00:Upload 32.34%
18:40:26:WU04:FS00:Upload 32.79%
18:40:39:WU04:FS00:Upload 33.24%
18:40:45:WU04:FS00:Upload 33.46%
18:40:58:WU04:FS00:Upload 33.69%
18:41:13:WU04:FS00:Upload 33.91%
18:41:22:WU04:FS00:Upload 34.36%
18:41:28:WU04:FS00:Upload 34.58%
18:41:37:WU04:FS00:Upload 34.81%
18:41:48:WU04:FS00:Upload 35.03%

18:42:02:WU04:FS00:Upload 35.26%

18:42:14:WU04:FS00:Upload 35.71%
18:42:23:WU04:FS00:Upload 36.16%

18:42:29:WU04:FS00:Upload 36.38%
18:42:54:WU04:FS00:Upload 36.61%
18:43:00:WU04:FS00:Upload 36.83%
18:43:20:WU04:FS00:Upload 37.05%
18:43:30:WU04:FS00:Upload 37.50%
18:43:38:WU04:FS00:Upload 37.73%
18:43:52:WU04:FS00:Upload 37.95%
18:44:02:WU04:FS00:Upload 38.18%
18:44:09:WU04:FS00:Upload 38.63%
18:44:17:WU04:FS00:Upload 38.85%
18:44:25:WU04:FS00:Upload 39.30%
18:44:32:WU04:FS00:Upload 39.52%
18:44:42:WU04:FS00:Upload 39.97%
18:44:51:WU04:FS00:Upload 40.20%
18:44:59:WU04:FS00:Upload 40.42%
18:45:28:WU04:FS00:Upload 40.65%
18:45:34:WU04:FS00:Upload 40.87%
18:45:45:WU04:FS00:Upload 41.32%
18:45:53:WU04:FS00:Upload 41.55%
18:46:02:WU04:FS00:Upload 41.77%
18:46:10:WU04:FS00:Upload 42.00%
18:46:17:WU04:FS00:Upload 42.22%
18:46:23:WU04:FS00:Upload 42.44%
18:46:38:WU04:FS00:Upload 42.67%

18:54:01:WU04:FS00:Upload 42.89%
18:54:01:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
18:54:01:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
18:54:01:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
18:54:01:WU04:FS00:Connecting to 128.252.203.2:8080
18:54:07:WU04:FS00:Upload 0.45%
18:54:20:WU04:FS00:Upload 0.90%
18:54:27:WU02:FS00:0xa4:Completed 100000 out of 1250000 steps  (8%)
18:54:30:WU04:FS00:Upload 1.12%
18:54:42:WU04:FS00:Upload 1.57%
18:54:50:WU04:FS00:Upload 1.80%
18:55:08:WU04:FS00:Upload 2.02%
18:55:19:WU04:FS00:Upload 2.25%
18:55:26:WU04:FS00:Upload 2.69%
18:55:36:WU04:FS00:Upload 2.92%
18:55:51:WU04:FS00:Upload 3.37%
18:55:59:WU04:FS00:Upload 3.82%
18:56:09:WU04:FS00:Upload 4.27%
18:56:16:WU04:FS00:Upload 4.49%
18:56:30:WU04:FS00:Upload 4.72%
18:56:37:WU04:FS00:Upload 4.94%
18:56:45:WU04:FS00:Upload 5.39%
18:56:54:WU04:FS00:Upload 5.61%
18:57:00:WU04:FS00:Upload 6.06%
18:57:08:WU04:FS00:Upload 6.51%
18:57:36:WU04:FS00:Upload 6.74%
18:57:43:WU04:FS00:Upload 6.96%
18:57:50:WU04:FS00:Upload 7.41%
18:58:00:WU04:FS00:Upload 7.64%
18:58:06:WU04:FS00:Upload 7.86%
18:58:12:WU04:FS00:Upload 8.08%
18:58:27:WU04:FS00:Upload 8.76%
18:58:35:WU04:FS00:Upload 8.98%
18:58:51:WU04:FS00:Upload 9.21%
18:59:10:WU04:FS00:Upload 9.88%

19:01:21:WU04:FS00:Upload 10.11%
19:01:21:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
19:01:21:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
19:01:21:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
19:01:21:WU04:FS00:Connecting to 128.252.203.2:8080
19:01:30:WU04:FS00:Upload 0.45%
19:01:36:WU04:FS00:Upload 0.67%
19:01:46:WU04:FS00:Upload 1.12%
19:01:58:WU04:FS00:Upload 1.35%
19:02:04:WU04:FS00:Upload 1.80%
19:02:04:WU03:FS02:0x18:Completed 1350000 out of 2500000 steps (54%)
19:02:11:WU04:FS00:Upload 2.02%
19:02:18:WU04:FS00:Upload 2.25%
19:02:26:WU04:FS00:Upload 2.47%
19:02:50:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:11633 run:1 clone:76 gen:19 core:0xa4 unit:0x0000001a4c71bbb056987d04dabf9fd9
19:02:50:WU01:FS00:Uploading 27.88MiB to 128.252.203.2
19:02:50:WU01:FS00:Connecting to 128.252.203.2:8080
19:02:54:WU04:FS00:Upload 2.69%
19:03:00:WU04:FS00:Upload 2.92%
19:03:09:WU04:FS00:Upload 3.14%
19:03:21:WU04:FS00:Upload 3.59%
19:03:31:WU04:FS00:Upload 3.82%
19:03:37:WU04:FS00:Upload 4.04%
19:03:44:WU04:FS00:Upload 4.27%
19:03:55:WU04:FS00:Upload 4.72%
19:04:02:WU04:FS00:Upload 4.94%
19:04:11:WU04:FS00:Upload 5.17%
19:04:16:WU01:FS00:Upload 0.45%
19:04:16:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
19:04:17:WU04:FS00:Upload 5.39%
19:06:31:WU02:FS00:0xa4:Completed 125000 out of 1250000 steps  (10%)
19:07:09:WU04:FS00:Upload 5.61%
19:08:05:WU04:FS00:Upload 5.84%
19:08:20:WU04:FS00:Upload 6.06%
19:08:30:WU04:FS00:Upload 6.29%
19:08:49:WU04:FS00:Upload 6.74%
19:08:58:WU04:FS00:Upload 7.19%
19:09:06:WU04:FS00:Upload 7.64%
19:09:12:WU04:FS00:Upload 7.86%
19:09:29:WU04:FS00:Upload 8.08%
19:09:41:WU04:FS00:Upload 8.53%
19:09:49:WU04:FS00:Upload 8.76%
19:09:55:WU04:FS00:Upload 8.98%
19:10:12:WU04:FS00:Upload 9.21%
19:10:21:WU04:FS00:Upload 9.43%
19:10:31:WU04:FS00:Upload 9.88%
19:10:39:WU04:FS00:Upload 10.11%
19:10:47:WU04:FS00:Upload 10.55%
19:11:02:WU04:FS00:Upload 10.78%

19:12:24:WU04:FS00:Upload 11.00%
19:12:31:WU04:FS00:Upload 11.23%

19:12:40:WU04:FS00:Upload 11.45%
19:12:46:WU04:FS00:Upload 11.68%
19:12:58:WU04:FS00:Upload 12.13%
19:13:28:WU04:FS00:Upload 12.35%
19:15:36:WU04:FS00:Upload 12.58%
19:15:36:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
19:15:36:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
19:15:36:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
19:15:36:WU04:FS00:Connecting to 128.252.203.2:8080
19:16:00:WU04:FS00:Upload 0.67%
19:16:48:WU00:FS01:0x18:Completed 4350000 out of 5000000 steps (87%)
19:17:14:WU04:FS00:Upload 1.12%
19:17:51:WU04:FS00:Upload 1.57%
19:17:59:WU04:FS00:Upload 1.80%
19:18:12:WU04:FS00:Upload 2.02%
19:18:18:WU04:FS00:Upload 2.25%
19:18:26:WU04:FS00:Upload 2.47%
19:18:33:WU04:FS00:Upload 2.69%

19:18:42:WU04:FS00:Upload 2.92%
19:18:55:WU04:FS00:Upload 3.14%
19:19:08:WU04:FS00:Upload 3.59%
19:19:20:WU04:FS00:Upload 3.82%
19:19:27:WU04:FS00:Upload 4.04%
19:19:35:WU04:FS00:Upload 4.49%
19:19:41:WU04:FS00:Upload 4.72%
19:19:47:WU04:FS00:Upload 4.94%
19:19:55:WU04:FS00:Upload 5.39%
19:20:06:WU04:FS00:Upload 5.61%
19:20:14:WU04:FS00:Upload 5.84%
19:20:20:WU04:FS00:Upload 6.06%
19:20:30:WU04:FS00:Upload 6.51%
19:20:38:WU04:FS00:Upload 6.96%
19:20:49:WU04:FS00:Upload 7.41%
19:20:58:WU04:FS00:Upload 7.86%
19:21:07:WU04:FS00:Upload 8.31%
19:21:16:WU04:FS00:Upload 8.76%
19:21:29:WU04:FS00:Upload 8.98%
19:21:35:WU04:FS00:Upload 9.21%
19:21:43:WU04:FS00:Upload 9.43%
19:21:52:WU04:FS00:Upload 9.88%
19:22:01:WU04:FS00:Upload 10.11%
19:22:12:WU04:FS00:Upload 10.33%

19:22:22:WU04:FS00:Upload 10.55%
19:22:34:WU04:FS00:Upload 11.00%
19:22:41:WU04:FS00:Upload 11.23%
19:22:47:WU04:FS00:Upload 11.45%
19:22:55:WU04:FS00:Upload 11.68%
19:23:01:WU04:FS00:Upload 11.90%
19:23:10:WU04:FS00:Upload 12.35%
19:23:17:WU04:FS00:Upload 12.58%
19:23:24:WU04:FS00:Upload 12.80%
19:23:36:WU04:FS00:Upload 13.25%
19:23:52:WU04:FS00:Upload 13.47%
19:24:05:WU04:FS00:Upload 13.70%
19:24:14:WU04:FS00:Upload 13.92%
19:24:22:WU04:FS00:Upload 14.15%
19:24:29:WU04:FS00:Upload 14.37%

19:26:13:WU04:FS00:Upload 14.60%
19:26:24:WU04:FS00:Upload 15.05%
19:26:40:WU04:FS00:Upload 15.50%
19:26:46:WU04:FS00:Upload 15.94%
19:26:55:WU04:FS00:Upload 16.39%
19:27:03:WU04:FS00:Upload 16.84%
19:27:13:WU04:FS00:Upload 17.29%
19:27:19:WU04:FS00:Upload 17.52%
19:27:32:WU04:FS00:Upload 17.97%
19:27:42:WU04:FS00:Upload 18.42%
19:27:48:WU04:FS00:Upload 18.64%
19:28:02:WU04:FS00:Upload 19.09%
19:28:17:WU04:FS00:Upload 19.31%
19:28:25:WU04:FS00:Upload 19.54%
19:28:35:WU04:FS00:Upload 19.99%
19:28:43:WU04:FS00:Upload 20.21%
19:29:01:WU04:FS00:Upload 20.66%
19:29:10:WU04:FS00:Upload 20.89%
19:29:17:WU04:FS00:Upload 21.11%
19:29:27:WU04:FS00:Upload 21.33%
19:29:33:WU04:FS00:Upload 21.56%
19:29:39:WU04:FS00:Upload 21.78%
19:29:49:WU04:FS00:Upload 22.01%
19:29:58:WU04:FS00:Upload 22.46%
19:30:07:WU04:FS00:Upload 22.91%
19:30:23:WU04:FS00:Upload 23.13%
19:30:31:WU04:FS00:Upload 23.36%
19:30:39:WU04:FS00:Upload 24.03%
19:30:45:WU04:FS00:Upload 24.25%

19:30:55:WU04:FS00:Upload 24.70%
19:31:43:WU04:FS00:Upload 24.93%
19:31:50:WU04:FS00:Upload 25.15%
19:31:59:WU04:FS00:Upload 25.38%
19:32:12:WU04:FS00:Upload 25.60%

19:32:18:WU04:FS00:Upload 25.83%
19:32:31:WU04:FS00:Upload 26.28%

19:35:19:WU04:FS00:Upload 26.50%
19:35:19:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
19:35:19:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
19:35:19:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
19:35:19:WU04:FS00:Connecting to 128.252.203.2:8080
19:35:32:WU04:FS00:Upload 0.67%
19:35:39:WU04:FS00:Upload 0.90%
19:35:45:WU04:FS00:Upload 1.35%
19:35:53:WU04:FS00:Upload 1.80%
19:36:02:WU04:FS00:Upload 2.02%
19:36:17:WU04:FS00:Upload 2.25%
19:36:49:WU02:FS00:0xa4:Completed 187500 out of 1250000 steps  (15%)
19:36:49:WU04:FS00:Upload 2.47%
19:36:59:WU04:FS00:Upload 2.69%
19:37:13:WU04:FS00:Upload 3.14%
19:37:20:WU04:FS00:Upload 3.37%
19:37:27:WU04:FS00:Upload 3.59%
19:37:33:WU04:FS00:Upload 3.82%
19:37:47:WU04:FS00:Upload 4.27%
19:37:54:WU04:FS00:Upload 4.49%
19:38:00:WU04:FS00:Upload 4.72%
19:38:16:WU04:FS00:Upload 5.17%
19:38:32:WU04:FS00:Upload 5.39%
19:38:49:WU04:FS00:Upload 5.84%
19:39:05:WU04:FS00:Upload 6.06%
19:39:28:WU04:FS00:Upload 6.29%
19:39:48:WU04:FS00:Upload 6.51%
19:39:56:WU04:FS00:Upload 6.96%
19:40:11:WU04:FS00:Upload 7.41%
19:40:20:WU04:FS00:Upload 7.64%
19:40:26:WU04:FS00:Upload 7.86%
19:40:37:WU04:FS00:Upload 8.08%
19:40:46:WU04:FS00:Upload 8.53%
19:40:57:WU04:FS00:Upload 8.98%
19:41:06:WU04:FS00:Upload 9.43%

19:47:33:WU04:FS00:Upload 9.66%
19:47:33:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
19:47:34:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
19:47:34:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
19:47:34:WU04:FS00:Connecting to 128.252.203.2:8080
19:47:59:WU04:FS00:Upload 0.45%
19:48:09:WU04:FS00:Upload 0.67%
19:48:15:WU04:FS00:Upload 0.90%
19:48:22:WU04:FS00:Upload 1.12%
19:48:30:WU04:FS00:Upload 1.35%
19:48:41:WU04:FS00:Upload 1.80%
19:48:56:WU02:FS00:0xa4:Completed 212500 out of 1250000 steps  (17%)
19:48:56:WU04:FS00:Upload 2.02%
19:49:04:WU04:FS00:Upload 2.25%
19:49:34:WU04:FS00:Upload 2.47%
19:50:17:WU04:FS00:Upload 2.69%
19:50:27:WU04:FS00:Upload 3.14%
19:50:33:WU04:FS00:Upload 3.37%
19:50:43:WU04:FS00:Upload 3.82%
19:50:53:WU04:FS00:Upload 4.27%
19:51:18:WU04:FS00:Upload 4.72%
19:51:24:WU04:FS00:Upload 4.94%
19:51:29:WU00:FS01:0x18:Completed 4450000 out of 5000000 steps (89%)
19:51:30:WU04:FS00:Upload 5.17%
19:51:41:WU04:FS00:Upload 5.39%
19:52:04:WU04:FS00:Upload 5.61%
19:52:25:WU03:FS02:0x18:Completed 1475000 out of 2500000 steps (59%)
19:52:30:WU04:FS00:Upload 6.06%
19:52:56:WU04:FS00:Upload 6.51%
19:53:05:WU04:FS00:Upload 6.96%
19:53:16:WU04:FS00:Upload 7.41%
19:53:23:WU04:FS00:Upload 7.86%
19:53:29:WU04:FS00:Upload 8.31%
19:53:38:WU04:FS00:Upload 8.76%
19:53:47:WU04:FS00:Upload 9.21%
19:53:59:WU04:FS00:Upload 9.66%
19:54:07:WU04:FS00:Upload 9.88%
19:54:15:WU04:FS00:Upload 10.11%
19:54:26:WU04:FS00:Upload 10.33%
19:54:38:WU04:FS00:Upload 10.78%
19:54:47:WU04:FS00:Upload 11.00%
19:54:53:WU04:FS00:Upload 11.23%
19:54:57:WU02:FS00:0xa4:Completed 225000 out of 1250000 steps  (18%)
19:55:09:WU04:FS00:Upload 11.68%
19:55:17:WU04:FS00:Upload 11.90%
19:55:25:WU04:FS00:Upload 12.13%
19:55:35:WU04:FS00:Upload 12.58%
19:55:42:WU04:FS00:Upload 12.80%
19:55:48:WU04:FS00:Upload 13.03%
19:56:13:WU04:FS00:Upload 13.47%
19:56:19:WU04:FS00:Upload 13.70%
19:56:31:WU04:FS00:Upload 14.15%
19:56:39:WU04:FS00:Upload 14.60%
19:56:46:WU04:FS00:Upload 15.05%
19:57:16:WU04:FS00:Upload 15.27%
19:57:24:WU04:FS00:Upload 15.50%
19:57:34:WU04:FS00:Upload 15.94%
19:57:43:WU04:FS00:Upload 16.39%
19:57:49:WU04:FS00:Upload 16.62%
19:57:57:WU04:FS00:Upload 16.84%
19:58:05:WU04:FS00:Upload 17.29%
19:58:14:WU04:FS00:Upload 17.74%
19:58:25:WU04:FS00:Upload 18.19%
19:58:33:WU04:FS00:Upload 18.64%
19:58:45:WU04:FS00:Upload 19.09%
19:59:07:WU04:FS00:Upload 19.31%
19:59:18:WU04:FS00:Upload 19.99%
19:59:25:WU04:FS00:Upload 20.21%
19:59:39:WU04:FS00:Upload 20.66%
20:00:27:WU04:FS00:Upload 20.89%
20:00:37:WU04:FS00:Upload 21.33%
20:00:47:WU04:FS00:Upload 21.56%
20:00:56:WU04:FS00:Upload 21.78%
20:00:57:WU02:FS00:0xa4:Completed 237500 out of 1250000 steps  (19%)
20:01:06:WU04:FS00:Upload 22.23%
20:01:12:WU04:FS00:Upload 22.68%
20:01:18:WU04:FS00:Upload 23.13%
20:01:28:WU04:FS00:Upload 23.58%
20:01:34:WU04:FS00:Upload 24.03%
20:01:56:WU04:FS00:Upload 24.25%
20:02:03:WU04:FS00:Upload 24.70%
20:02:11:WU04:FS00:Upload 24.93%
20:02:18:WU04:FS00:Upload 25.38%
20:02:29:WU03:FS02:0x18:Completed 1500000 out of 2500000 steps (60%)
20:02:35:WU04:FS00:Upload 25.83%
20:02:42:WU04:FS00:Upload 26.28%
20:02:50:WU04:FS00:Upload 26.50%
20:03:01:WU04:FS00:Upload 26.72%
20:03:12:WU04:FS00:Upload 26.95%
20:03:31:WU04:FS00:Upload 27.40%
20:03:39:WU04:FS00:Upload 27.62%
20:03:46:WU04:FS00:Upload 27.85%
20:03:54:WU04:FS00:Upload 28.07%
20:04:04:WU04:FS00:Upload 28.30%
20:04:13:WU04:FS00:Upload 28.97%
20:04:26:WU04:FS00:Upload 29.64%
20:04:38:WU04:FS00:Upload 30.09%
20:04:44:WU04:FS00:Upload 30.54%
20:04:55:WU04:FS00:Upload 30.99%
20:05:03:WU04:FS00:Upload 31.66%
20:05:13:WU04:FS00:Upload 32.34%
20:05:22:WU04:FS00:Upload 32.56%
20:05:35:WU04:FS00:Upload 33.01%
20:05:42:WU04:FS00:Upload 33.24%
20:05:48:WU04:FS00:Upload 33.69%
20:05:55:WU04:FS00:Upload 33.91%
20:06:11:WU04:FS00:Upload 34.36%
20:06:20:WU04:FS00:Upload 34.58%
20:06:32:WU04:FS00:Upload 35.26%
20:06:42:WU04:FS00:Upload 35.71%
20:06:54:WU04:FS00:Upload 36.38%
20:07:01:WU02:FS00:0xa4:Completed 250000 out of 1250000 steps  (20%)
20:07:01:WU04:FS00:Upload 36.61%
20:07:11:WU04:FS00:Upload 36.83%
20:08:21:WU04:FS00:Upload 37.05%
20:08:30:WU04:FS00:Upload 37.28%
20:08:37:WU04:FS00:Upload 37.73%
20:08:40:WU00:FS01:0x18:Completed 4500000 out of 5000000 steps (90%)
20:08:50:WU04:FS00:Upload 38.40%
20:08:56:WU04:FS00:Upload 38.85%
20:09:25:WU04:FS00:Upload 39.30%
20:09:32:WU04:FS00:Upload 39.52%
20:09:38:WU04:FS00:Upload 39.75%
20:09:55:WU04:FS00:Upload 39.97%
20:10:50:WU04:FS00:Upload 40.20%
20:10:58:WU04:FS00:Upload 40.87%
20:11:04:WU04:FS00:Upload 41.10%
20:11:11:WU04:FS00:Upload 41.55%
20:11:21:WU04:FS00:Upload 42.00%
20:11:30:WU04:FS00:Upload 42.44%
20:11:36:WU04:FS00:Upload 42.67%
20:11:42:WU04:FS00:Upload 42.89%
20:11:50:WU04:FS00:Upload 43.34%
20:11:56:WU04:FS00:Upload 43.57%
20:12:08:WU04:FS00:Upload 44.02%
20:12:15:WU04:FS00:Upload 44.24%
20:12:30:WU04:FS00:Upload 44.47%
20:12:35:WU03:FS02:0x18:Completed 1525000 out of 2500000 steps (61%)
20:12:41:WU04:FS00:Upload 44.69%
20:12:52:WU04:FS00:Upload 45.14%
20:12:59:WU04:FS00:Upload 45.36%
20:13:04:WU02:FS00:0xa4:Completed 262500 out of 1250000 steps  (21%)
20:13:06:WU04:FS00:Upload 45.81%
20:13:15:WU04:FS00:Upload 46.26%
20:13:24:WU04:FS00:Upload 46.49%
20:13:32:WU04:FS00:Upload 46.94%
20:13:41:WU04:FS00:Upload 47.38%
20:13:51:WU04:FS00:Upload 48.06%
20:13:58:WU04:FS00:Upload 48.28%
20:14:05:WU04:FS00:Upload 48.73%
20:14:11:WU04:FS00:Upload 48.96%
20:14:31:WU04:FS00:Upload 49.18%
20:14:37:WU04:FS00:Upload 49.41%
20:14:44:WU04:FS00:Upload 49.63%
20:14:53:WU04:FS00:Upload 49.86%
20:15:04:WU04:FS00:Upload 50.30%
20:17:39:WU04:FS00:Upload 50.53%
20:17:39:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
20:17:39:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
20:17:39:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
20:17:39:WU04:FS00:Connecting to 128.252.203.2:8080
20:17:48:WU04:FS00:Upload 0.45%
20:17:54:WU04:FS00:Upload 0.67%
20:18:01:WU04:FS00:Upload 1.12%
20:18:13:WU04:FS00:Upload 1.57%
20:18:19:WU04:FS00:Upload 1.80%
20:18:28:WU04:FS00:Upload 2.02%
20:18:38:WU04:FS00:Upload 2.25%
20:18:44:WU04:FS00:Upload 2.47%
20:18:53:WU04:FS00:Upload 2.69%
20:19:00:WU04:FS00:Upload 2.92%
20:19:06:WU02:FS00:0xa4:Completed 275000 out of 1250000 steps  (22%)
20:19:11:WU04:FS00:Upload 3.37%
20:19:18:WU04:FS00:Upload 3.59%
20:20:41:WU04:FS00:Upload 3.82%
20:20:41:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed

20:35:36:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
20:35:36:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
20:35:36:WU04:FS00:Connecting to 128.252.203.2:8080
20:35:44:WU04:FS00:Upload 0.45%
20:36:02:WU04:FS00:Upload 0.67%
20:36:16:WU04:FS00:Upload 1.12%
20:36:26:WU04:FS00:Upload 1.35%
20:36:32:WU04:FS00:Upload 1.57%
20:36:40:WU04:FS00:Upload 1.80%
20:36:54:WU04:FS00:Upload 2.25%
20:37:00:WU04:FS00:Upload 2.47%
20:37:11:WU02:FS00:0xa4:Completed 312500 out of 1250000 steps  (25%)
20:37:19:WU04:FS00:Upload 2.92%
20:37:25:WU04:FS00:Upload 3.14%
20:37:35:WU04:FS00:Upload 3.59%
20:37:44:WU04:FS00:Upload 3.82%
20:37:54:WU04:FS00:Upload 4.27%
20:38:00:WU04:FS00:Upload 4.49%
20:38:08:WU04:FS00:Upload 4.72%
20:38:18:WU04:FS00:Upload 5.17%
20:38:25:WU04:FS00:Upload 5.39%
20:38:33:WU04:FS00:Upload 5.61%


20:44:08:WU04:FS00:Upload 5.84%
20:44:08:WARNING:WU04:FS00:Exception: Failed to send results to work server: Transfer failed
20:49:14:WU02:FS00:0xa4:Completed 337500 out of 1250000 steps  (27%)
20:52:52:WU03:FS02:0x18:Completed 1625000 out of 2500000 steps (65%)
20:55:17:WU02:FS00:0xa4:Completed 350000 out of 1250000 steps  (28%)
21:00:48:WU00:FS01:0x18:Completed 4650000 out of 5000000 steps (93%)
21:01:20:WU02:FS00:0xa4:Completed 362500 out of 1250000 steps  (29%)
21:02:56:WU03:FS02:0x18:Completed 1650000 out of 2500000 steps (66%)
21:04:38:WU04:FS00:Sending unit results: id:04 state:SEND error:NO_ERROR project:11633 run:0 clone:56 gen:45 core:0xa4 unit:0x000000314c71bbb056987cf8c1d01d74
21:04:38:WU04:FS00:Uploading 27.83MiB to 128.252.203.2
21:04:38:WU04:FS00:Connecting to 128.252.203.2:8080
21:04:47:WU04:FS00:Upload 0.67%
21:05:03:WU04:FS00:Upload 0.90%
21:05:10:WU04:FS00:Upload 1.12%
21:05:16:WU04:FS00:Upload 1.35%
21:05:24:WU04:FS00:Upload 1.57%
21:05:32:WU04:FS00:Upload 1.80%
21:05:39:WU04:FS00:Upload 2.02%
21:05:46:WU04:FS00:Upload 2.25%
21:06:05:WU04:FS00:Upload 2.47%
21:06:11:WU04:FS00:Upload 2.69%
21:06:20:WU04:FS00:Upload 2.92%
21:06:27:WU04:FS00:Upload 3.14%
21:06:41:WU04:FS00:Upload 3.82%
21:06:47:WU04:FS00:Upload 4.04%
21:06:58:WU04:FS00:Upload 4.27%
21:07:10:WU04:FS00:Upload 4.49%
21:07:16:WU04:FS00:Upload 4.72%
21:07:23:WU04:FS00:Upload 4.94%
21:07:26:WU02:FS00:0xa4:Completed 375000 out of 1250000 steps  (30%)
21:07:31:WU04:FS00:Upload 5.17%
21:07:42:WU04:FS00:Upload 5.39%
21:07:48:WU04:FS00:Upload 5.61%
21:07:59:WU04:FS00:Upload 6.06%
21:08:07:WU04:FS00:Upload 6.29%
21:08:18:WU04:FS00:Upload 6.74%
21:08:26:WU04:FS00:Upload 6.96%
21:08:33:WU04:FS00:Upload 7.41%
21:08:40:WU04:FS00:Upload 7.86%
21:09:01:WU04:FS00:Upload 8.08%
21:09:10:WU04:FS00:Upload 8.53%
21:09:27:WU04:FS00:Upload 8.76%
21:09:37:WU04:FS00:Upload 8.98%
21:09:44:WU04:FS00:Upload 9.21%
21:09:55:WU04:FS00:Upload 9.43%
21:10:02:WU04:FS00:Upload 9.66%
21:10:09:WU04:FS00:Upload 9.88%
21:10:19:WU04:FS00:Upload 10.11%
21:10:26:WU04:FS00:Upload 10.33%
21:10:37:WU04:FS00:Upload 10.55%
21:10:45:WU04:FS00:Upload 10.78%
21:10:57:WU04:FS00:Upload 11.00%
21:11:23:WU04:FS00:Upload 11.23%
21:11:29:WU04:FS00:Upload 11.68%
21:11:36:WU04:FS00:Upload 12.13%
21:11:42:WU04:FS00:Upload 12.58%
21:11:48:WU04:FS00:Upload 12.80%
21:12:01:WU04:FS00:Upload 13.47%
21:12:07:WU04:FS00:Upload 13.70%
21:12:14:WU04:FS00:Upload 13.92%
21:12:25:WU04:FS00:Upload 14.37%
21:12:33:WU04:FS00:Upload 15.05%
21:12:42:WU04:FS00:Upload 16.17%
21:12:54:WU04:FS00:Upload 16.39%
21:12:59:WU03:FS02:0x18:Completed 1675000 out of 2500000 steps (67%)
21:13:02:WU04:FS00:Upload 16.62%
21:13:12:WU04:FS00:Upload 16.84%
21:13:23:WU04:FS00:Upload 17.07%
21:13:31:WU04:FS00:Upload 17.52%
21:13:32:WU02:FS00:0xa4:Completed 387500 out of 1250000 steps  (31%)
21:13:38:WU04:FS00:Upload 17.97%
21:13:53:WU04:FS00:Upload 18.64%

Nick200 · Post by **Nick200** » Fri Mar 25, 2016 10:10 pm

And another machine, this time with three WUs stuck sending, to 140.163.4.241 and 171.65.103.160 and 140.163.4.234

Log file is too massive to send, but shows 131 instances of Failed reports

Post by **bruce** » Fri Mar 25, 2016 11:51 pm

I'm changing the title of this topic from Failed to connect to 140.163.4.241 which is inaccurate. The logs show that the connection is being made, but the uploads are taking too long. This apparently is because the size of the upload package has grown too big for whatever line-speed your ISP has allocated for you.

If you have changed the setting of max-packet-size, set back to the default value. I've contacted the PG, too, because if you're using the default value, they should be limiting the size of the upload packet or allowing additional time. This issue sometimes gets overlooked.

Post by **JohnChodera** » Sat Mar 26, 2016 12:23 am

Thanks everybody for alerting us to issues with the MSK servers (140.163.4.24x). Looking into this now.

Post by **JohnChodera** » Sat Mar 26, 2016 12:27 am

That's odd. This server is set to use a 30 minute timeout, but it looks like the upload timed out after ~13 minutes. That's really strange.

Note that these servers don't answer to pings from the outside world because of firewall restrictions. We're working to try to get these lifted to make debugging easier for everyone.

Post by **JohnChodera** » Sat Mar 26, 2016 12:28 am

I've just doubled the timeout (to 1 hour). Please report any other failed uploads and how long they took to timeout. There may be a bug causing the early timeout.

Nick200 · Post by **Nick200** » Sat Mar 26, 2016 12:56 am

Nothing wrong with my ISP or my upload speeds, Bruce. I get around 40Mbps.

And I have not fiddled with the max-packet-size settings - they are default on all the affected machines.

I recently uploaded 100 Gb to Onedrive and it took 2 hours.

So, a 15Mb WU results file should not take the time it does.

I still have around 8 WUs showing failures to upload across my 4 main FAH machines.

Thanks for altering the timeouts, but that does not seem to have fixed it. Just got this:

Code: Select all

00:53:46:WARNING:WU03:FS00:Exception: Failed to send results to work server: Transfer failed

Post by **bruce** » Sat Mar 26, 2016 4:25 am

Do a traceroute (Linux) or tracert (Windows) to each of those servers and post the results. You can expect to get timeouts after you reach MSK's boundary router (per above) but the intermediate steps may show something useful.
C:\Users\bruce>tracert 140.163.4.241

Tracing route to plfah2-1.mskcc.org [140.163.4.241]
over a maximum of 30 hops:

1 <1 ms 1 ms <1 ms 192.168.1.1
2 10 ms 11 ms 10 ms netblock-208-xxx-xxx-1.dslextreme.com
3 10 ms 10 ms 11 ms lax1_cr1_gig_10_0_120.dslextreme.com [66.218.48.145]
4 12 ms 11 ms 11 ms ge-101-0-0-17.r00.lsanca07.us.bb.gin.ntt.net [129.250.203.93]
5 11 ms 11 ms 12 ms ae-2.r23.lsanca07.us.bb.gin.ntt.net [129.250.3.237]
6 78 ms 78 ms 78 ms ae-6.r22.asbnva02.us.bb.gin.ntt.net [129.250.3.188]
7 93 ms 86 ms 87 ms ae-5.r25.nycmny01.us.bb.gin.ntt.net [129.250.2.148]
8 85 ms 85 ms 85 ms ae-2.r07.nycmny01.us.bb.gin.ntt.net [129.250.3.98]
9 85 ms 86 ms 84 ms xe-0-9-0-28.r07.nycmny01.us.ce.gin.ntt.net [129.250.193.158]
10 86 ms 87 ms 87 ms tengig0-1-0-1.agr03.nwrk01-nj.us.windstream.net[40.128.248.12]
11 99 ms 85 ms 86 ms 67.151.24.149
12 91 ms 84 ms 84 ms 74.8.57.6
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.

Incidentally, 140.163.4.241 is a Collection Server. Information about the Work Server(s) is more important.

You're also asking about 171.65.103.160 and 140.163.4.234 but I don't see them mentioned in the logs you posted.

I see an attempt to upload to some other Work Servers.
140.163.4.235:8080
There may be others.

I need to be able find pairs of messages BOTH containing the same :WU0x: like
HH.MM.SS:WU03:FS00:Uploading x.xxMiB to xxx.xxx.xxx.xxx
...
hh.mm.ss:WARNING:WU03:FS00:Exception: Failed to send results to work server: Transfer failed

so I can subtract HH.MM.SS from hh.mm.ss for that particular work server.

By chance, is part of your bandwidth going to a Peer to Peer (P2P) File Sharing Program or something like that?

Nick200 · Post by **Nick200** » Sat Mar 26, 2016 7:14 am

Tracerts as follows - I am in New Zealand which might explain some of the delays as there are long cable runs under the Pacific that have to be negotiated:

Code: Select all

C:\WINDOWS\system32>tracert 140.163.4.241

Tracing route to plfah2-1.mskcc.org [140.163.4.241]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  192.168.1.1
  2     *        *        *     Request timed out.
  3    18 ms    20 ms    17 ms  lo0.internet.ivpn.pe15.telstraclear.net [218.101.61.30]
  4    21 ms    24 ms    21 ms  203.98.50.2
  5    19 ms    26 ms    22 ms  203.98.50.251
  6    35 ms    43 ms    28 ms  unknown.telstraglobal.net [134.159.174.37]
  7   144 ms   155 ms   151 ms  i-0-0-1-1.1wlt-core01.bx.telstraglobal.net [202.84.223.85]
  8   154 ms   167 ms   155 ms  202.84.136.254
  9   155 ms   158 ms   163 ms  i-0-5-0-4.eqnx03.bi.telstraglobal.net [202.40.149.97]
 10   156 ms   161 ms   160 ms  xo_comm-peer.eqnx03.pr.telstraglobal.net [134.159.62.18]
 11   230 ms   241 ms   235 ms  207.88.13.228.ptr.us.xo.net [207.88.13.228]
 12   242 ms   231 ms   240 ms  207.88.12.148.ptr.us.xo.net [207.88.12.148]
 13   239 ms     *      262 ms  207.88.12.150.ptr.us.xo.net [207.88.12.150]
 14   242 ms   238 ms   237 ms  te-4-1-0.rar3.miami-fl.us.xo.net [207.88.12.161]
 15   235 ms   241 ms   237 ms  207.88.12.144.ptr.us.xo.net [207.88.12.144]
 16   243 ms   233 ms   231 ms  207.88.12.190.ptr.us.xo.net [207.88.12.190]
 17   236 ms   232 ms   249 ms  te0-12-0-0.rar3.sanjose-ca.us.xo.net [207.88.12.189]
 18   236 ms   242 ms   250 ms  207.88.12.164.ptr.us.xo.net [207.88.12.164]
 19   247 ms   237 ms   251 ms  te-4-1-0.rar3.nyc-ny.us.xo.net [207.88.12.169]
 20   236 ms   242 ms   237 ms  216.156.16.131.ptr.us.xo.net [216.156.16.131]
 21   234 ms   236 ms   240 ms  209.220.19.46.ptr.us.xo.net [209.220.19.46]
 22   234 ms   235 ms   233 ms  et11-0-0-0.cr02.nycm01-ny.us.windstream.net [40.129.35.249]
 23   235 ms   233 ms   235 ms  tengig0-1-0-0.agr03.nwrk01-nj.us.windstream.net [40.128.248.10]
 24   247 ms   237 ms   233 ms  67.151.24.149
 25   232 ms   233 ms   234 ms  74.8.57.6
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.

Trace complete.

And another:

Code: Select all

C:\WINDOWS\system32>tracert 128.252.203.2

Tracing route to brodgar.seas.wustl.edu [128.252.203.2]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  192.168.1.1
  2     *        *        *     Request timed out.
  3    15 ms    20 ms    18 ms  lo0.internet.ivpn.pe15.telstraclear.net [218.101.61.30]
  4    19 ms    17 ms    25 ms  203.98.50.2
  5    19 ms    22 ms    24 ms  203.98.50.251
  6    25 ms    30 ms    37 ms  unknown.telstraglobal.net [134.159.174.41]
  7   149 ms   143 ms   146 ms  i-0-5-1-0.1wlt-core01.bx.telstraglobal.net [202.84.142.118]
  8   143 ms   145 ms   147 ms  i-0-0-0-3.eqla01.bi.telstraglobal.net [202.40.149.134]
  9   147 ms   147 ms   146 ms  cogent-peer.eqla01.pr.telstraglobal.net [134.159.63.198]
 10   146 ms   156 ms   147 ms  be2179.ccr22.lax01.atlas.cogentco.com [154.54.41.81]
 11   180 ms   193 ms   178 ms  be2066.ccr22.iah01.atlas.cogentco.com [154.54.7.53]
 12   194 ms   187 ms   185 ms  be2443.ccr22.dfw01.atlas.cogentco.com [154.54.44.230]
 13   202 ms   197 ms   197 ms  be2433.ccr22.mci01.atlas.cogentco.com [154.54.3.214]
 14   208 ms   201 ms   204 ms  be2347.rcr21.stl03.atlas.cogentco.com [154.54.80.174]
 15   204 ms   199 ms   201 ms  xe-0-0-2-commodity-core.nts.wustl.edu [38.104.162.78]
 16   197 ms   200 ms   203 ms  eth7-2-eps-core.nts.wustl.edu [128.252.100.125]
 17   206 ms   207 ms   216 ms  te1-7-1-engineering-core.nts.wustl.edu [128.252.1.50]
 18   211 ms   196 ms   206 ms  brodgar.seas.wustl.edu [128.252.203.2]

Trace complete.
C:\WINDOWS\system32>tracert 128.252.203.2

Tracing route to brodgar.seas.wustl.edu [128.252.203.2]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  192.168.1.1
  2     *        *        *     Request timed out.
  3    15 ms    20 ms    18 ms  lo0.internet.ivpn.pe15.telstraclear.net [218.101.61.30]
  4    19 ms    17 ms    25 ms  203.98.50.2
  5    19 ms    22 ms    24 ms  203.98.50.251
  6    25 ms    30 ms    37 ms  unknown.telstraglobal.net [134.159.174.41]
  7   149 ms   143 ms   146 ms  i-0-5-1-0.1wlt-core01.bx.telstraglobal.net [202.84.142.118]
  8   143 ms   145 ms   147 ms  i-0-0-0-3.eqla01.bi.telstraglobal.net [202.40.149.134]
  9   147 ms   147 ms   146 ms  cogent-peer.eqla01.pr.telstraglobal.net [134.159.63.198]
 10   146 ms   156 ms   147 ms  be2179.ccr22.lax01.atlas.cogentco.com [154.54.41.81]
 11   180 ms   193 ms   178 ms  be2066.ccr22.iah01.atlas.cogentco.com [154.54.7.53]
 12   194 ms   187 ms   185 ms  be2443.ccr22.dfw01.atlas.cogentco.com [154.54.44.230]
 13   202 ms   197 ms   197 ms  be2433.ccr22.mci01.atlas.cogentco.com [154.54.3.214]
 14   208 ms   201 ms   204 ms  be2347.rcr21.stl03.atlas.cogentco.com [154.54.80.174]
 15   204 ms   199 ms   201 ms  xe-0-0-2-commodity-core.nts.wustl.edu [38.104.162.78]
 16   197 ms   200 ms   203 ms  eth7-2-eps-core.nts.wustl.edu [128.252.100.125]
 17   206 ms   207 ms   216 ms  te1-7-1-engineering-core.nts.wustl.edu [128.252.1.50]
 18   211 ms   196 ms   206 ms  brodgar.seas.wustl.edu [128.252.203.2]

Trace complete.

I'll post the full log from one of the machines next.

Nick200 · Post by **Nick200** » Sat Mar 26, 2016 7:27 am

Full log from one machine affected by upload problems, edited to take in down to 60K

Code: Select all

22:57:13:WU05:FS00:0xa4:- Reading up to 811512 from "05/wudata_01.trr": Read 811512
22:57:13:WU05:FS00:0xa4:trr file hash check passed.
22:57:13:WU05:FS00:0xa4:- Reading up to 746352 from "05/wudata_01.xtc": Read 746352
22:57:13:WU05:FS00:0xa4:xtc file hash check passed.
22:57:13:WU05:FS00:0xa4:edr file hash check passed.
22:57:13:WU05:FS00:0xa4:logfile size: 23828
22:57:13:WU05:FS00:0xa4:Leaving Run
22:57:14:WU01:FS01:Upload 9.03%
22:57:17:WU05:FS00:0xa4:- Writing 1584180 bytes of core data to disk...
22:57:18:WU05:FS00:0xa4:Done: 1583668 -> 1538254 (compressed to 97.1 percent)
22:57:18:WU05:FS00:0xa4:  ... Done.
22:57:18:WU05:FS00:0xa4:- Shutting down core
22:57:18:WU05:FS00:0xa4:
22:57:18:WU05:FS00:0xa4:Folding@home Core Shutdown: FINISHED_UNIT
22:57:19:WU05:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:57:19:WU05:FS00:Sending unit results: id:05 state:SEND error:NO_ERROR project:9034 run:92 clone:4 gen:6 core:0xa4 unit:0x00000007ab436c9e5698307745c43415
22:57:19:WU05:FS00:Uploading 1.47MiB to 171.67.108.158
22:57:19:WU05:FS00:Connecting to 171.67.108.158:8080
22:57:19:WU00:FS00:Starting
22:57:19:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 00 -suffix 01 -version 704 -lifeline 9412 -checkpoint 15 -np 6
22:57:19:WU00:FS00:Started FahCore on PID 6152
22:57:19:WU00:FS00:Core PID:6344
22:57:19:WU00:FS00:FahCore 0xa4 started
22:57:19:WU00:FS00:0xa4:
22:57:19:WU00:FS00:0xa4:*------------------------------*
22:57:19:WU00:FS00:0xa4:Folding@Home Gromacs GB Core
22:57:19:WU00:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
22:57:19:WU00:FS00:0xa4:
22:57:19:WU00:FS00:0xa4:Preparing to commence simulation
22:57:19:WU00:FS00:0xa4:- Looking at optimizations...
22:57:19:WU00:FS00:0xa4:- Created dyn
22:57:19:WU00:FS00:0xa4:- Files status OK
22:57:19:WU00:FS00:0xa4:- Expanded 118945 -> 269764 (decompressed 226.7 percent)
22:57:19:WU00:FS00:0xa4:Called DecompressByteArray: compressed_data_size=118945 data_size=269764, decompressed_data_size=269764 diff=0
22:57:19:WU00:FS00:0xa4:- Digital signature verified
22:57:19:WU00:FS00:0xa4:
22:57:19:WU00:FS00:0xa4:Project: 6399 (Run 92, Clone 32, Gen 66)
22:57:19:WU00:FS00:0xa4:
22:57:19:WU00:FS00:0xa4:Assembly optimizations on if available.
22:57:19:WU00:FS00:0xa4:Entering M.D.
22:57:25:WU00:FS00:0xa4:Mapping NT from 6 to 6 
22:57:25:WU00:FS00:0xa4:Completed 0 out of 5000000 steps  (0%)
22:57:29:WU05:FS00:Upload 8.52%
22:57:33:WU01:FS01:Upload 9.60%
22:57:38:WU05:FS00:Upload 12.78%
22:57:45:WU05:FS00:Upload 17.04%
22:57:47:WU03:FS02:Upload 5.41%
22:57:47:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
22:57:47:WU03:FS02:Trying to send results to collection server
22:57:47:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
22:57:47:WU03:FS02:Connecting to 128.252.203.2:8080
22:57:53:WU03:FS02:Upload 0.77%
22:57:59:WU03:FS02:Upload 1.16%
22:58:05:WU03:FS02:Upload 1.93%
22:58:12:WU03:FS02:Upload 2.70%
22:58:25:WU03:FS02:Upload 3.09%
22:58:44:WU03:FS02:Upload 3.48%
22:58:55:WU03:FS02:Upload 4.25%
22:59:04:WU03:FS02:Upload 5.02%
22:59:11:WU03:FS02:Upload 5.41%
22:59:21:WU03:FS02:Upload 6.18%
22:59:31:WU03:FS02:Upload 6.57%
22:59:32:WU05:FS00:Upload 25.55%
22:59:37:WU03:FS02:Upload 6.95%
22:59:46:WU03:FS02:Upload 7.34%
22:59:47:WU00:FS00:0xa4:Completed 50000 out of 5000000 steps  (1%)
22:59:48:WU05:FS00:Upload 29.81%
22:59:58:WU05:FS00:Upload 38.33%
23:00:00:WU03:FS02:Upload 8.11%
23:00:09:WU05:FS00:Upload 46.85%
23:00:09:WU03:FS02:Upload 8.50%
23:00:15:WU05:FS00:Upload 51.11%
23:00:22:WU05:FS00:Upload 55.37%
23:00:25:WU03:FS02:Upload 9.27%
23:00:32:WU01:FS01:Upload 10.16%
23:00:33:ERROR:WU01:FS01:Exception: Transfer failed
23:00:33:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
23:00:33:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
23:00:33:WU01:FS01:Connecting to 140.163.4.235:8080
23:00:35:WU03:FS02:Upload 10.43%
23:00:35:WU05:FS00:Upload 63.88%
23:00:43:WU05:FS00:Upload 72.40%
23:00:46:WU01:FS01:Upload 1.13%
23:00:53:WU01:FS01:Upload 1.69%
23:00:59:WU01:FS01:Upload 2.26%
23:01:11:WU01:FS01:Upload 3.39%
23:01:13:WU03:FS02:Upload 11.20%
23:01:20:WU01:FS01:Upload 3.95%
23:01:26:WU03:FS02:Upload 11.98%
23:01:28:WU01:FS01:Upload 4.52%
23:01:32:WU03:FS02:Upload 12.36%
23:01:35:WU01:FS01:Upload 5.08%

23:01:45:WU01:FS01:Upload 6.21%
23:01:47:WU03:FS02:Upload 13.13%
23:02:07:WU01:FS01:Upload 6.77%
23:02:18:WU01:FS01:Upload 7.34%
23:02:19:WU03:FS02:Upload 13.52%
23:02:34:WU01:FS01:Upload 7.90%

23:02:42:WU01:FS01:Upload 8.47%
23:02:58:WU01:FS01:Upload 9.03%
23:03:04:WU01:FS01:Upload 9.60%
23:03:14:WU01:FS01:Upload 10.16%
23:03:20:WU01:FS01:Upload 10.73%
23:03:30:WU01:FS01:Upload 11.29%
23:03:36:WU01:FS01:Upload 11.85%

23:03:48:WU01:FS01:Upload 13.55%
23:03:59:WU01:FS01:Upload 14.11%
23:04:08:WU01:FS01:Upload 14.68%
23:04:10:WU05:FS00:Upload 76.66%
23:04:15:WU01:FS01:Upload 15.24%
23:04:18:WU05:FS00:Upload 85.18%
23:04:26:WU05:FS00:Upload 93.70%
23:04:28:WU01:FS01:Upload 16.37%
23:04:36:WU01:FS01:Upload 16.94%
23:04:38:WU05:FS00:Upload 97.96%
23:04:47:WU01:FS01:Upload 17.50%
23:04:50:WU05:FS00:Upload complete
23:04:50:WU05:FS00:Server responded WORK_ACK (400)
23:04:50:WU05:FS00:Final credit estimate, 1345.00 points
23:04:50:WU05:FS00:Cleaning up
23:05:13:WU01:FS01:Upload 18.06%
23:05:24:WU01:FS01:Upload 18.63%
23:05:30:WU01:FS01:Upload 19.19%

23:05:42:WU01:FS01:Upload 20.32%
23:05:50:WU01:FS01:Upload 20.89%
23:05:57:WU01:FS01:Upload 21.45%
23:06:07:WU01:FS01:Upload 22.02%
23:06:17:WU01:FS01:Upload 22.58%
23:07:01:WU01:FS01:Upload 23.14%
23:07:08:WU01:FS01:Upload 23.71%
23:07:15:WU01:FS01:Upload 24.27%

23:08:20:WU03:FS02:Upload 13.91%
23:08:20:ERROR:WU03:FS02:Exception: Transfer failed
23:08:20:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
23:08:20:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
23:08:20:WU03:FS02:Connecting to 140.163.4.245:8080
23:08:32:WU03:FS02:Upload 0.77%

23:10:04:WU01:FS01:Upload 24.84%
23:10:12:WU01:FS01:Upload 25.40%
23:10:35:WU01:FS01:Upload 25.97%
23:10:35:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
23:10:35:WU01:FS01:Trying to send results to collection server
23:10:35:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
23:10:35:WU01:FS01:Connecting to 140.163.4.241:8080
23:10:47:WU01:FS01:Upload 1.13%
23:11:08:WU01:FS01:Upload 1.69%
23:11:08:ERROR:WU01:FS01:Exception: Transfer failed
23:11:08:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
23:11:08:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
23:11:08:WU01:FS01:Connecting to 140.163.4.235:8080

23:11:32:WU01:FS01:Upload 1.13%
23:11:41:WU01:FS01:Upload 1.69%
23:11:48:WU01:FS01:Upload 2.26%
23:11:56:WU01:FS01:Upload 2.82%
23:13:59:WU01:FS01:Upload 3.95%
23:13:59:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
23:13:59:WU01:FS01:Trying to send results to collection server
23:13:59:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
23:13:59:WU01:FS01:Connecting to 140.163.4.241:8080
23:14:11:WU01:FS01:Upload 1.69%
23:14:24:WU01:FS01:Upload 2.26%
23:14:55:WU01:FS01:Upload 2.82%
23:15:00:WU03:FS02:Upload 1.16%
23:15:00:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
23:15:00:WU03:FS02:Trying to send results to collection server
23:15:00:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
23:15:00:WU03:FS02:Connecting to 128.252.203.2:8080
23:15:02:WU01:FS01:Upload 3.39%
23:15:08:WU03:FS02:Upload 1.16%
23:15:08:WU01:FS01:Upload 3.95%
23:15:14:WU03:FS02:Upload 1.93%
23:15:16:WU01:FS01:Upload 5.08%
23:15:21:WU03:FS02:Upload 2.70%
23:15:24:WU01:FS01:Upload 5.65%
23:15:28:WU03:FS02:Upload 3.09%
23:15:37:WU03:FS02:Upload 3.86%
23:15:48:WU03:FS02:Upload 4.25%
23:15:52:WU01:FS01:Upload 6.77%
23:15:58:WU01:FS01:Upload 7.34%
23:16:01:WU03:FS02:Upload 4.64%
23:16:04:WU01:FS01:Upload 7.90%
23:16:13:WU01:FS01:Upload 9.03%
23:16:14:WU03:FS02:Upload 5.41%
23:16:20:WU03:FS02:Upload 5.79%
23:16:21:WU01:FS01:Upload 9.60%
23:16:29:WU01:FS01:Upload 10.16%
23:16:31:WU03:FS02:Upload 6.18%
23:16:41:WU03:FS02:Upload 6.57%
23:16:42:WU01:FS01:Upload 10.73%
23:16:48:WU03:FS02:Upload 7.34%
23:16:55:WU03:FS02:Upload 7.73%
23:16:57:WU01:FS01:Upload 11.29%
23:17:03:WU03:FS02:Upload 8.11%
23:17:12:WU03:FS02:Upload 8.50%
23:17:23:WU03:FS02:Upload 8.89%
23:18:06:WU03:FS02:Upload 9.27%
23:18:12:WU01:FS01:Upload 11.85%
23:18:12:WU03:FS02:Upload 9.66%
23:18:21:WU01:FS01:Upload 12.42%
23:18:21:WU03:FS02:Upload 10.43%
23:18:36:WU01:FS01:Upload 12.98%
23:18:45:WU01:FS01:Upload 13.55%
23:18:45:WU03:FS02:Upload 10.82%
23:18:51:WU01:FS01:Upload 14.11%
23:19:08:WU01:FS01:Upload 14.68%
23:19:18:WU01:FS01:Upload 15.24%
23:19:30:WU01:FS01:Upload 16.37%
23:20:05:WU01:FS01:Upload 17.50%
23:20:15:WU03:FS02:Upload 11.20%
23:20:21:WU03:FS02:Upload 11.59%
23:20:29:WU03:FS02:Upload 11.98%
23:20:41:WU03:FS02:Upload 12.75%
23:20:54:WU03:FS02:Upload 13.52%
23:21:01:WU03:FS02:Upload 13.91%
23:21:09:WU01:FS01:Upload 18.06%
23:21:10:WU03:FS02:Upload 14.29%
23:21:21:WU03:FS02:Upload 14.68%
23:21:26:WU01:FS01:Upload 18.63%
23:21:28:WU03:FS02:Upload 15.07%
23:21:35:WU01:FS01:Upload 19.19%
23:21:42:WU01:FS01:Upload 19.76%
23:22:19:WU01:FS01:Upload 20.32%
23:22:22:WU03:FS02:Upload 15.84%
23:22:28:WU03:FS02:Upload 16.23%
23:22:39:WU03:FS02:Upload 16.61%
23:22:45:WU03:FS02:Upload 17.00%
23:22:52:WU03:FS02:Upload 17.77%
23:23:10:WU03:FS02:Upload 18.54%
23:23:22:WU03:FS02:Upload 18.93%
23:23:35:WU03:FS02:Upload 19.32%
23:23:42:WU03:FS02:Upload 19.70%
23:23:42:WU01:FS01:Upload 20.89%
23:23:57:WU03:FS02:Upload 20.09%
23:24:13:WU03:FS02:Upload 20.86%
23:24:22:WU03:FS02:Upload 21.25%
23:24:24:WU01:FS01:Upload 21.45%
23:25:49:WU03:FS02:Upload 21.63%
23:25:49:ERROR:WU03:FS02:Exception: Transfer failed
23:25:49:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
23:25:49:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
23:25:49:WU03:FS02:Connecting to 140.163.4.245:8080
23:25:56:WU03:FS02:Upload 1.16%
23:26:10:WU03:FS02:Upload 1.55%
23:26:21:WU03:FS02:Upload 1.93%
23:26:34:WU03:FS02:Upload 2.70%
23:26:46:WU03:FS02:Upload 3.09%
23:26:52:WU03:FS02:Upload 3.48%
23:27:00:WU03:FS02:Upload 4.25%
23:27:07:WU03:FS02:Upload 4.64%
23:27:25:WU03:FS02:Upload 5.02%
23:27:55:WU03:FS02:Upload 5.41%
23:28:01:WU03:FS02:Upload 5.79%
23:28:11:WU03:FS02:Upload 6.57%
23:28:19:WU03:FS02:Upload 7.34%
23:28:31:WU03:FS02:Upload 7.73%
23:28:38:WU03:FS02:Upload 8.11%
23:28:46:WU03:FS02:Upload 8.50%
23:28:57:WU03:FS02:Upload 9.27%
23:29:06:WU03:FS02:Upload 9.66%
23:29:14:WU03:FS02:Upload 10.04%
23:29:21:WU03:FS02:Upload 10.43%
23:29:25:WU01:FS01:Upload 22.58%
23:29:25:ERROR:WU01:FS01:Exception: Transfer failed
23:29:25:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
23:29:25:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
23:29:25:WU01:FS01:Connecting to 140.163.4.235:8080
23:29:28:WU03:FS02:Upload 10.82%
23:29:39:WU01:FS01:Upload 1.13%
23:29:42:WU03:FS02:Upload 11.20%
23:29:50:WU03:FS02:Upload 11.59%
23:30:05:WU01:FS01:Upload 1.69%
23:30:13:WU01:FS01:Upload 2.26%
23:30:33:WU01:FS01:Upload 2.82%
23:30:33:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
23:30:33:WU01:FS01:Trying to send results to collection server
23:30:33:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
23:30:33:WU01:FS01:Connecting to 140.163.4.241:8080
23:30:41:WU01:FS01:Upload 1.13%
23:32:29:WU03:FS02:Upload 12.36%
23:32:29:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
23:32:29:WU03:FS02:Trying to send results to collection server
23:32:29:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
23:32:29:WU03:FS02:Connecting to 128.252.203.2:8080
23:32:37:WU03:FS02:Upload 0.77%
23:32:44:WU03:FS02:Upload 1.16%
23:32:51:WU03:FS02:Upload 1.55%
23:33:00:WU03:FS02:Upload 1.93%
23:33:06:WU03:FS02:Upload 2.32%
23:33:26:WU01:FS01:Upload 1.69%
23:33:36:WU03:FS02:Upload 2.70%
23:33:40:WU01:FS01:Upload 2.82%
23:33:43:WU03:FS02:Upload 3.09%
23:33:53:WU03:FS02:Upload 3.86%
23:33:54:WU01:FS01:Upload 3.39%
23:34:05:WU03:FS02:Upload 4.25%
23:34:09:WU01:FS01:Upload 3.95%
23:34:13:WU03:FS02:Upload 4.64%
23:34:19:WU01:FS01:Upload 4.52%
23:34:22:WU03:FS02:Upload 5.41%
23:34:32:WU01:FS01:Upload 5.08%
23:34:39:WU01:FS01:Upload 5.65%
23:34:42:WU03:FS02:Upload 5.79%
23:34:46:WU01:FS01:Upload 6.21%
23:34:53:WU01:FS01:Upload 6.77%
23:34:55:WU03:FS02:Upload 6.18%
23:35:01:WU01:FS01:Upload 7.34%
23:35:06:WU03:FS02:Upload 6.57%
23:35:08:WU01:FS01:Upload 7.90%
23:35:13:WU03:FS02:Upload 6.95%
23:35:20:WU01:FS01:Upload 8.47%
23:35:22:WU03:FS02:Upload 7.34%
23:35:28:WU01:FS01:Upload 9.60%
23:35:31:WU03:FS02:Upload 8.11%
23:35:38:WU01:FS01:Upload 10.73%
23:35:45:WU01:FS01:Upload 11.29%
23:35:51:WU01:FS01:Upload 11.85%
23:35:54:WU03:FS02:Upload 8.50%
23:36:01:WU03:FS02:Upload 8.89%
23:36:03:WU01:FS01:Upload 13.55%
23:36:11:WU03:FS02:Upload 9.27%
23:36:19:WU03:FS02:Upload 10.04%
23:36:27:WU03:FS02:Upload 10.82%
23:36:35:WU01:FS01:Upload 14.11%
23:36:37:WU03:FS02:Upload 11.59%
23:36:43:WU03:FS02:Upload 11.98%
23:36:50:WU03:FS02:Upload 12.36%
23:36:55:WU01:FS01:Upload 15.24%
23:37:01:WU03:FS02:Upload 13.13%
23:37:15:WU03:FS02:Upload 13.52%
23:37:19:WU01:FS01:Upload 16.37%
23:37:19:ERROR:WU01:FS01:Exception: Transfer failed
23:37:19:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
23:37:19:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
23:37:19:WU01:FS01:Connecting to 140.163.4.235:8080
23:37:22:WU03:FS02:Upload 13.91%
23:37:27:WU01:FS01:Upload 1.69%
23:37:32:WU03:FS02:Upload 14.68%
23:37:39:WU03:FS02:Upload 15.07%
23:37:45:WU03:FS02:Upload 15.45%
23:37:55:WU03:FS02:Upload 16.23%
23:38:04:WU03:FS02:Upload 16.61%
23:38:21:WU01:FS01:Upload 2.26%
23:38:27:WU01:FS01:Upload 2.82%
23:38:29:WU03:FS02:Upload 17.00%
23:38:35:WU03:FS02:Upload 17.38%
23:38:37:WU01:FS01:Upload 3.95%
23:38:48:WU03:FS02:Upload 18.16%
23:38:54:WU03:FS02:Upload 18.54%
23:39:02:WU03:FS02:Upload 19.32%
23:39:09:WU03:FS02:Upload 19.70%
23:39:20:WU03:FS02:Upload 20.09%
23:39:36:WU01:FS01:Upload 5.08%
23:39:36:WU03:FS02:Upload 20.47%
23:40:47:WU03:FS02:Upload 20.86%
23:40:48:ERROR:WU03:FS02:Exception: Transfer failed
23:40:48:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
23:40:48:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
23:40:48:WU03:FS02:Connecting to 140.163.4.245:8080
23:41:01:WU03:FS02:Upload 0.77%
23:41:13:WU03:FS02:Upload 1.55%
23:41:22:WU03:FS02:Upload 1.93%
23:41:29:WU03:FS02:Upload 2.32%
23:41:36:WU01:FS01:Upload 5.65%
23:41:36:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
23:41:36:WU01:FS01:Trying to send results to collection server
23:41:36:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
23:41:36:WU01:FS01:Connecting to 140.163.4.241:8080
23:41:48:WU01:FS01:Upload 1.13%
23:41:49:WU03:FS02:Upload 3.09%
23:42:08:WU03:FS02:Upload 3.48%
23:42:13:WU01:FS01:Upload 1.69%
23:42:13:ERROR:WU01:FS01:Exception: Transfer failed
23:42:29:WU03:FS02:Upload 3.86%
23:42:29:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
23:42:29:WU03:FS02:Trying to send results to collection server
23:42:29:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
23:42:29:WU03:FS02:Connecting to 128.252.203.2:8080
23:42:35:WU03:FS02:Upload 0.77%
23:42:52:WU03:FS02:Upload 1.16%
23:43:13:WU03:FS02:Upload 1.55%
23:43:20:WU03:FS02:Upload 1.93%
23:43:26:WU03:FS02:Upload 2.32%
23:43:34:WU03:FS02:Upload 3.09%
23:43:42:WU03:FS02:Upload 3.86%
23:43:53:WU03:FS02:Upload 4.64%
23:44:06:WU03:FS02:Upload 5.41%
23:44:14:WU03:FS02:Upload 5.79%
23:44:23:WU03:FS02:Upload 6.18%
23:44:29:WU03:FS02:Upload 6.57%
23:44:42:WU03:FS02:Upload 7.73%
23:44:48:WU03:FS02:Upload 8.50%
23:44:55:WU03:FS02:Upload 8.89%
23:45:01:WU03:FS02:Upload 9.27%
23:45:09:WU03:FS02:Upload 10.04%
23:45:20:WU03:FS02:Upload 10.82%
23:45:26:WU03:FS02:Upload 11.20%
23:45:36:WU03:FS02:Upload 11.98%
23:45:44:WU03:FS02:Upload 12.36%
23:46:03:WU03:FS02:Upload 13.13%
23:46:34:WU03:FS02:Upload 13.52%
23:46:50:WU03:FS02:Upload 14.29%
23:46:57:WU03:FS02:Upload 14.68%
23:47:03:WU03:FS02:Upload 15.07%
23:47:13:WU03:FS02:Upload 15.45%
23:47:21:WU03:FS02:Upload 15.84%
23:47:34:WU03:FS02:Upload 16.23%
23:48:25:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
23:48:25:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
23:48:25:WU01:FS01:Connecting to 140.163.4.235:8080
23:49:24:WU01:FS01:Upload 1.13%
23:49:39:WU01:FS01:Upload 2.26%
23:49:42:WU03:FS02:Upload 16.61%
23:49:42:ERROR:WU03:FS02:Exception: Transfer failed
23:49:43:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
23:49:43:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
23:49:43:WU03:FS02:Connecting to 140.163.4.245:8080
23:49:46:WU01:FS01:Upload 2.82%
23:49:52:WU01:FS01:Upload 3.39%
23:49:55:WU03:FS02:Upload 0.77%
23:50:03:WU03:FS02:Upload 1.16%
23:50:09:WU03:FS02:Upload 1.55%
23:50:14:WU01:FS01:Upload 3.95%
23:50:18:WU03:FS02:Upload 1.93%
23:50:20:WU01:FS01:Upload 4.52%
23:50:25:WU03:FS02:Upload 2.32%
23:50:30:WU01:FS01:Upload 5.08%
23:50:33:WU03:FS02:Upload 2.70%
23:50:38:WU01:FS01:Upload 5.65%
23:50:44:WU03:FS02:Upload 3.09%
23:50:56:WU03:FS02:Upload 3.86%
23:51:03:WU03:FS02:Upload 4.64%
23:51:09:WU03:FS02:Upload 5.02%
23:51:11:WU01:FS01:Upload 6.21%
23:51:11:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
23:51:11:WU01:FS01:Trying to send results to collection server
23:51:11:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
23:51:11:WU01:FS01:Connecting to 140.163.4.241:8080
23:51:18:WU03:FS02:Upload 5.79%
23:51:20:WU01:FS01:Upload 1.13%
23:51:30:WU03:FS02:Upload 6.57%
23:51:34:WU01:FS01:Upload 2.26%
23:51:45:WU01:FS01:Upload 2.82%
23:52:12:WU01:FS01:Upload 3.39%
23:52:12:ERROR:WU01:FS01:Exception: Transfer failed
23:52:14:WU03:FS02:Upload 6.95%
23:52:20:WU03:FS02:Upload 7.34%
23:52:28:WU03:FS02:Upload 7.73%
23:52:36:WU03:FS02:Upload 8.50%
23:52:59:WU03:FS02:Upload 9.27%
23:53:05:WU03:FS02:Upload 9.66%
23:53:11:WU03:FS02:Upload 10.04%
23:53:22:WU03:FS02:Upload 10.43%
23:53:30:WU03:FS02:Upload 10.82%
23:53:38:WU03:FS02:Upload 11.59%
23:53:45:WU03:FS02:Upload 12.36%
23:53:53:WU03:FS02:Upload 12.75%
23:54:13:WU03:FS02:Upload 13.13%
23:55:06:WU03:FS02:Upload 13.52%
23:55:32:WU03:FS02:Upload 13.91%
23:55:41:WU03:FS02:Upload 14.29%
23:56:00:WU03:FS02:Upload 14.68%
23:56:23:WU03:FS02:Upload 15.07%
23:56:35:WU03:FS02:Upload 15.45%
23:56:42:WU03:FS02:Upload 15.84%
23:56:54:WU03:FS02:Upload 16.61%
23:57:03:WU03:FS02:Upload 17.00%
23:57:15:WU03:FS02:Upload 17.38%
23:57:22:WU03:FS02:Upload 17.77%
23:57:29:WU03:FS02:Upload 18.16%
23:57:37:WU03:FS02:Upload 18.54%
23:57:45:WU03:FS02:Upload 18.93%
23:58:03:WU03:FS02:Upload 19.32%
23:58:18:WU03:FS02:Upload 20.09%
23:58:26:WU03:FS02:Upload 20.47%
23:58:33:WU03:FS02:Upload 20.86%
23:58:39:WU03:FS02:Upload 21.25%
23:58:53:WU03:FS02:Upload 22.02%
23:58:59:WU03:FS02:Upload 22.41%
23:59:06:WU03:FS02:Upload 22.79%
23:59:22:WU03:FS02:Upload 23.18%
00:00:13:WU03:FS02:Upload 23.57%
00:00:29:WU03:FS02:Upload 23.95%
00:00:39:WU03:FS02:Upload 24.34%
00:00:50:WU03:FS02:Upload 24.72%
00:01:01:WU03:FS02:Upload 25.11%
00:01:09:WU03:FS02:Upload 25.50%
00:01:18:WU03:FS02:Upload 25.88%
00:01:32:WU03:FS02:Upload 26.27%
00:01:55:WU03:FS02:Upload 26.66%
00:02:19:WU03:FS02:Upload 27.04%
00:02:27:WU03:FS02:Upload 27.43%
00:03:45:WU03:FS02:Upload 27.81%
00:04:05:WU03:FS02:Upload 28.20%
00:04:13:WU03:FS02:Upload 28.59%
00:04:21:WU03:FS02:Upload 28.97%
00:04:34:WU03:FS02:Upload 29.36%
00:05:04:WU03:FS02:Upload 29.75%
00:05:14:WU03:FS02:Upload 30.13%
00:05:24:WU03:FS02:Upload 30.52%
00:06:15:WU03:FS02:Upload 30.91%
00:06:21:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
00:06:21:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
00:06:21:WU01:FS01:Connecting to 140.163.4.235:8080
00:06:26:WU00:FS00:0xa4:Completed 1750000 out of 5000000 steps  (35%)
00:06:34:WU01:FS01:Upload 1.13%
00:07:05:WU01:FS01:Upload 1.69%
00:07:38:WU03:FS02:Upload 31.29%
00:07:38:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
00:07:38:WU03:FS02:Trying to send results to collection server
00:07:38:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
00:07:38:WU03:FS02:Connecting to 128.252.203.2:8080
00:07:51:WU03:FS02:Upload 0.77%
00:08:05:WU03:FS02:Upload 1.16%
00:08:17:WU03:FS02:Upload 1.55%
00:08:25:WU03:FS02:Upload 2.32%
00:08:43:WU01:FS01:Upload 2.26%
00:08:43:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
00:08:43:WU01:FS01:Trying to send results to collection server
00:08:43:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
00:08:43:WU01:FS01:Connecting to 140.163.4.241:8080
00:08:57:WU01:FS01:Upload 1.13%
00:09:09:WU01:FS01:Upload 2.26%
00:11:01:WU03:FS02:Upload 2.70%
00:11:01:ERROR:WU03:FS02:Exception: Transfer failed
00:11:01:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
00:11:01:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
00:11:01:WU03:FS02:Connecting to 140.163.4.245:8080
00:11:52:WU01:FS01:Upload 2.82%
00:12:02:WU01:FS01:Upload 3.39%
00:12:12:WU01:FS01:Upload 3.95%
00:12:22:WU01:FS01:Upload 4.52%
00:12:34:WU01:FS01:Upload 5.08%
00:12:41:WU03:FS02:Upload 0.77%
00:12:41:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
00:12:41:WU03:FS02:Trying to send results to collection server
00:12:41:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
00:12:41:WU03:FS02:Connecting to 128.252.203.2:8080
00:12:42:WU01:FS01:Upload 5.65%
00:12:49:WU01:FS01:Upload 6.21%
00:12:51:WU03:FS02:Upload 0.77%
00:13:00:WU03:FS02:Upload 1.16%
00:13:06:WU03:FS02:Upload 1.55%
00:13:15:WU03:FS02:Upload 1.93%
00:13:32:WU03:FS02:Upload 2.32%
00:13:43:WU03:FS02:Upload 3.09%
00:13:52:WU03:FS02:Upload 3.48%
00:13:59:WU03:FS02:Upload 3.86%
00:14:35:WU01:FS01:Upload 6.77%
00:14:44:WU01:FS01:Upload 7.34%
00:14:51:WU01:FS01:Upload 7.90%
00:14:59:WU01:FS01:Upload 8.47%
00:15:05:WU01:FS01:Upload 9.03%
00:15:13:WU01:FS01:Upload 9.60%
00:15:25:WU03:FS02:Upload 4.25%
00:15:27:WU01:FS01:Upload 10.16%
00:15:37:WU01:FS01:Upload 11.29%
00:15:45:WU01:FS01:Upload 11.85%
00:15:56:WU01:FS01:Upload 12.42%
00:16:03:WU01:FS01:Upload 13.55%
00:16:11:WU01:FS01:Upload 14.11%
00:16:23:WU01:FS01:Upload 15.24%
00:16:30:WU01:FS01:Upload 15.81%
00:16:43:WU01:FS01:Upload 16.37%
00:16:48:WU03:FS02:Upload 4.64%
00:16:48:ERROR:WU03:FS02:Exception: Transfer failed
00:16:49:WU01:FS01:Upload 16.94%
00:17:02:WU01:FS01:Upload 17.50%
00:17:19:WU01:FS01:Upload 18.63%
00:17:31:WU01:FS01:Upload 19.19%
00:18:52:WU01:FS01:Upload 19.76%
00:19:06:WU01:FS01:Upload 20.32%
00:19:13:WU01:FS01:Upload 20.89%
00:19:47:WU01:FS01:Upload 22.02%
00:20:21:WU01:FS01:Upload 22.58%
00:20:28:WU01:FS01:Upload 23.14%
00:20:35:WU01:FS01:Upload 23.71%
00:20:41:WU01:FS01:Upload 24.27%
00:20:52:WU01:FS01:Upload 24.84%
00:21:02:WU01:FS01:Upload 25.97%
00:21:10:WU01:FS01:Upload 26.53%
00:21:16:WU01:FS01:Upload 27.10%
00:21:27:WU01:FS01:Upload 27.66%
00:21:54:WU01:FS01:Upload 28.23%
00:21:55:WU00:FS00:0xa4:Completed 2100000 out of 5000000 steps  (42%)
00:22:01:WU01:FS01:Upload 28.79%
00:22:07:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
00:22:07:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
00:22:07:WU03:FS02:Connecting to 140.163.4.245:8080
00:22:08:WU01:FS01:Upload 29.35%
00:22:19:WU03:FS02:Upload 0.77%
00:22:21:WU01:FS01:Upload 29.92%
00:22:26:WU03:FS02:Upload 1.16%
00:22:42:WU03:FS02:Upload 1.93%
00:22:44:WU01:FS01:Upload 30.48%
00:22:44:ERROR:WU01:FS01:Exception: Transfer failed
00:22:56:WU03:FS02:Upload 2.70%
00:23:08:WU03:FS02:Upload 3.09%
00:23:23:WU03:FS02:Upload 3.86%
00:23:33:WU03:FS02:Upload 4.64%
00:23:41:WU03:FS02:Upload 5.02%
00:23:49:WU03:FS02:Upload 5.41%
00:23:57:WU03:FS02:Upload 5.79%
00:24:14:WU03:FS02:Upload 6.57%
00:24:28:WU03:FS02:Upload 6.95%
00:24:34:WU03:FS02:Upload 7.34%
00:24:41:WU03:FS02:Upload 7.73%
00:24:48:WU03:FS02:Upload 8.11%
00:25:07:WU03:FS02:Upload 8.89%
00:25:19:WU03:FS02:Upload 9.27%
00:25:30:WU03:FS02:Upload 10.04%
00:25:36:WU03:FS02:Upload 10.43%
00:25:49:WU03:FS02:Upload 10.82%
00:25:59:WU03:FS02:Upload 11.20%
00:26:01:WU00:FS00:0xa4:Completed 2200000 out of 5000000 steps  (44%)
00:26:12:WU03:FS02:Upload 11.98%
00:26:20:WU03:FS02:Upload 12.36%
00:26:37:WU03:FS02:Upload 12.75%
00:26:59:WU03:FS02:Upload 13.13%
00:28:12:WU00:FS00:0xa4:Completed 2250000 out of 5000000 steps  (45%)
00:28:27:WU03:FS02:Upload 13.52%
00:28:27:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
00:28:27:WU03:FS02:Trying to send results to collection server
00:28:27:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
00:28:27:WU03:FS02:Connecting to 128.252.203.2:8080
00:28:40:WU03:FS02:Upload 0.77%
00:28:46:WU03:FS02:Upload 1.16%
00:28:56:WU03:FS02:Upload 1.55%
00:29:30:WU03:FS02:Upload 1.93%
00:29:40:WU03:FS02:Upload 2.32%
00:29:51:WU03:FS02:Upload 2.70%
00:30:01:WU03:FS02:Upload 3.09%
00:30:12:WU03:FS02:Upload 3.48%
00:31:14:WU03:FS02:Upload 3.86%
00:31:24:WU03:FS02:Upload 4.25%
00:31:42:WU03:FS02:Upload 4.64%
00:31:53:WU03:FS02:Upload 5.02%
00:31:59:WU03:FS02:Upload 5.41%
00:35:24:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
00:35:24:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
00:35:24:WU01:FS01:Connecting to 140.163.4.235:8080
00:35:34:WU01:FS01:Upload 1.13%
00:35:41:WU01:FS01:Upload 1.69%
00:36:16:WU01:FS01:Upload 2.26%
00:36:33:WU01:FS01:Upload 3.39%
00:36:46:WU01:FS01:Upload 3.95%
00:36:53:WU01:FS01:Upload 4.52%
00:37:09:WU01:FS01:Upload 5.08%
00:37:15:WU01:FS01:Upload 5.65%
00:37:21:WU01:FS01:Upload 6.21%
00:37:33:WU01:FS01:Upload 6.77%
00:37:47:WU01:FS01:Upload 7.34%
00:37:59:WU01:FS01:Upload 8.47%
00:38:14:WU01:FS01:Upload 9.03%
00:38:28:WU01:FS01:Upload 9.60%
00:38:33:WU03:FS02:Upload 6.18%
00:38:33:ERROR:WU03:FS02:Exception: Transfer failed
00:38:47:WU01:FS01:Upload 10.16%
00:38:55:WU01:FS01:Upload 10.73%
00:39:06:WU01:FS01:Upload 11.85%
00:39:14:WU01:FS01:Upload 12.42%
00:39:23:WU01:FS01:Upload 12.98%
00:39:32:WU01:FS01:Upload 13.55%
00:39:47:WU01:FS01:Upload 14.68%
00:39:57:WU01:FS01:Upload 15.24%
00:40:04:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
00:40:04:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
00:40:04:WU03:FS02:Connecting to 140.163.4.245:8080
00:40:04:WU01:FS01:Upload 15.81%
00:40:15:WU03:FS02:Upload 1.16%
00:40:17:WU01:FS01:Upload 16.37%
00:40:25:WU01:FS01:Upload 16.94%
00:40:33:WU01:FS01:Upload 17.50%
00:40:41:WU03:FS02:Upload 1.55%
00:40:44:WU01:FS01:Upload 18.63%
00:40:55:WU01:FS01:Upload 19.19%
00:41:02:WU01:FS01:Upload 19.76%
00:41:08:WU01:FS01:Upload 20.32%
00:41:21:WU01:FS01:Upload 20.89%
00:41:30:WU01:FS01:Upload 21.45%
00:42:04:WU01:FS01:Upload 22.02%
00:43:16:WU01:FS01:Upload 22.58%
00:43:25:WU01:FS01:Upload 23.14%
00:43:41:WU01:FS01:Upload 23.71%
00:43:48:WU01:FS01:Upload 24.27%
00:44:06:WU01:FS01:Upload 24.84%
00:44:19:WU01:FS01:Upload 25.40%
00:44:26:WU01:FS01:Upload 25.97%
00:44:34:WU01:FS01:Upload 27.10%
00:44:37:WU03:FS02:Upload 1.93%
00:44:37:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
00:44:37:WU03:FS02:Trying to send results to collection server
00:44:38:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
00:44:38:WU03:FS02:Connecting to 128.252.203.2:8080
00:44:40:WU01:FS01:Upload 27.66%
00:44:45:WU03:FS02:Upload 0.77%
00:44:47:WU01:FS01:Upload 28.23%
00:44:51:WU03:FS02:Upload 1.16%
00:44:56:WU01:FS01:Upload 28.79%
00:44:59:WU03:FS02:Upload 1.55%
00:45:07:WU01:FS01:Upload 29.92%
00:45:17:WU01:FS01:Upload 30.48%
00:45:20:WU03:FS02:Upload 2.32%
00:45:26:WU01:FS01:Upload 31.05%
00:45:26:WU03:FS02:Upload 2.70%
00:45:38:WU03:FS02:Upload 3.09%
00:45:45:WU03:FS02:Upload 3.86%
00:45:46:WU01:FS01:Upload 31.61%
00:45:46:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
00:45:46:WU01:FS01:Trying to send results to collection server
00:45:46:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
00:45:46:WU01:FS01:Connecting to 140.163.4.241:8080
00:45:54:WU01:FS01:Upload 1.13%
00:45:56:WU03:FS02:Upload 4.64%
00:46:06:WU01:FS01:Upload 1.69%
00:46:09:WU03:FS02:Upload 5.02%
00:46:14:WU01:FS01:Upload 2.26%
00:46:19:WU03:FS02:Upload 5.79%
00:46:25:WU01:FS01:Upload 3.39%
00:46:28:WU03:FS02:Upload 6.18%
00:46:39:WU03:FS02:Upload 6.57%
00:46:48:WU01:FS01:Upload 3.95%
00:46:48:ERROR:WU01:FS01:Exception: Transfer failed
00:46:49:WU02:FS02:0x21:Completed 520000 out of 2000000 steps (26%)
00:47:46:WU00:FS00:0xa4:Completed 2650000 out of 5000000 steps  (53%)
00:48:21:WU03:FS02:Upload 6.95%
00:48:21:ERROR:WU03:FS02:Exception: Transfer failed
01:09:06:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
01:09:06:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
01:09:06:WU03:FS02:Connecting to 140.163.4.245:8080
01:09:19:WU03:FS02:Upload 0.77%
01:09:25:WU03:FS02:Upload 1.16%
01:09:32:WU03:FS02:Upload 1.93%
01:09:42:WU03:FS02:Upload 2.70%
01:09:48:WU03:FS02:Upload 3.09%
01:10:08:WU03:FS02:Upload 3.48%
01:10:14:WU03:FS02:Upload 3.86%
01:10:24:WU03:FS02:Upload 4.25%
01:11:51:WU03:FS02:Upload 4.64%
01:12:02:WU03:FS02:Upload 5.02%
01:12:09:WU03:FS02:Upload 5.41%
01:12:15:WU03:FS02:Upload 5.79%
01:12:24:WU03:FS02:Upload 6.57%
01:12:31:WU03:FS02:Upload 6.95%
01:12:57:WU03:FS02:Upload 7.73%
01:13:06:WU03:FS02:Upload 8.11%
01:13:14:WU03:FS02:Upload 8.50%
01:13:29:WU03:FS02:Upload 8.89%
01:13:37:WU03:FS02:Upload 9.27%
01:13:43:WU03:FS02:Upload 9.66%
01:13:56:WU03:FS02:Upload 10.43%
01:14:03:WU03:FS02:Upload 11.20%
01:14:13:WU03:FS02:Upload 11.98%
01:14:19:WU03:FS02:Upload 12.36%
01:15:56:WU03:FS02:Upload 12.75%
01:15:56:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
01:15:56:WU03:FS02:Trying to send results to collection server
01:15:56:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
01:15:56:WU03:FS02:Connecting to 128.252.203.2:8080
01:16:05:WU03:FS02:Upload 0.77%
01:16:24:WU03:FS02:Upload 1.16%
01:19:23:WU03:FS02:Upload 1.55%
01:19:23:ERROR:WU03:FS02:Exception: Transfer failed
01:22:22:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
01:22:22:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
01:22:22:WU01:FS01:Connecting to 140.163.4.235:8080
01:22:28:WU01:FS01:Upload 1.13%
01:22:42:WU01:FS01:Upload 2.26%
01:22:49:WU01:FS01:Upload 2.82%
01:23:00:WU01:FS01:Upload 3.95%
01:23:07:WU01:FS01:Upload 4.52%
01:23:21:WU01:FS01:Upload 5.65%
01:23:28:WU01:FS01:Upload 6.21%
01:23:39:WU01:FS01:Upload 7.34%
01:23:46:WU01:FS01:Upload 7.90%
01:24:15:WU01:FS01:Upload 8.47%
01:24:26:WU01:FS01:Upload 9.60%
01:24:38:WU01:FS01:Upload 10.16%
01:26:10:WU01:FS01:Upload 10.73%
01:26:16:WU01:FS01:Upload 11.29%
01:26:24:WU01:FS01:Upload 11.85%
01:26:34:WU01:FS01:Upload 12.98%
01:26:51:WU01:FS01:Upload 13.55%
01:26:59:WU01:FS01:Upload 14.11%
01:30:24:WU01:FS01:Upload 14.68%
01:30:24:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
01:30:24:WU01:FS01:Trying to send results to collection server
01:30:24:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
01:30:24:WU01:FS01:Connecting to 140.163.4.241:8080
01:30:36:WU01:FS01:Upload 1.13%
01:31:35:WU01:FS01:Upload 1.69%
01:33:46:WU02:FS02:0x21:Completed 580000 out of 2000000 steps (29%)
01:33:59:WU00:FS00:0xa4:Completed 3600000 out of 5000000 steps  (72%)
01:35:08:WU01:FS01:Upload 2.26%
01:35:18:WU01:FS01:Upload 3.39%
01:35:28:WU01:FS01:Upload 3.95%
01:35:49:WU01:FS01:Upload 4.52%
01:36:03:WU01:FS01:Upload 5.08%
01:36:10:WU01:FS01:Upload 5.65%
01:36:39:WU00:FS00:0xa4:Completed 3650000 out of 5000000 steps  (73%)
01:38:08:WU04:FS01:0x21:Completed 2350000 out of 5000000 steps (47%)
01:38:25:WU01:FS01:Upload 6.77%
01:38:33:WU01:FS01:Upload 7.34%
01:38:42:WU01:FS01:Upload 8.47%
01:38:51:WU01:FS01:Upload 9.60%
01:39:00:WU01:FS01:Upload 10.16%
01:39:04:WU00:FS00:0xa4:Completed 3700000 out of 5000000 steps  (74%)
01:39:09:WU01:FS01:Upload 10.73%
01:39:16:WU01:FS01:Upload 11.29%
01:39:26:WU01:FS01:Upload 11.85%
01:39:33:WU01:FS01:Upload 12.42%
01:39:40:WU01:FS01:Upload 13.55%
01:39:46:WU01:FS01:Upload 14.11%
01:39:56:WU01:FS01:Upload 14.68%
01:40:05:WU01:FS01:Upload 15.81%
01:40:13:WU01:FS01:Upload 16.37%
01:40:19:WU01:FS01:Upload 16.94%
01:40:29:WU01:FS01:Upload 18.06%
01:40:39:WU01:FS01:Upload 18.63%
01:40:47:WU01:FS01:Upload 19.19%
01:41:03:WU01:FS01:Upload 19.76%
01:41:10:WU01:FS01:Upload 20.32%
01:41:20:WU01:FS01:Upload 20.89%
01:41:21:WU00:FS00:0xa4:Completed 3750000 out of 5000000 steps  (75%)
01:41:31:WU01:FS01:Upload 21.45%
01:41:40:WU01:FS01:Upload 22.02%
01:41:53:WU01:FS01:Upload 22.58%
01:42:02:WU01:FS01:Upload 23.71%
01:42:12:WU01:FS01:Upload 24.84%
01:42:35:WU01:FS01:Upload 25.40%
01:42:35:ERROR:WU01:FS01:Exception: Transfer failed
01:43:56:WU00:FS00:0xa4:Completed 3800000 out of 5000000 steps  (76%)
01:46:31:WU00:FS00:0xa4:Completed 3850000 out of 5000000 steps  (77%)
01:47:24:WU04:FS01:0x21:Completed 2400000 out of 5000000 steps (48%)
01:48:55:WU00:FS00:0xa4:Completed 3900000 out of 5000000 steps  (78%)
01:49:07:WU02:FS02:0x21:Completed 600000 out of 2000000 steps (30%)
01:51:22:WU00:FS00:0xa4:Completed 3950000 out of 5000000 steps  (79%)
01:53:37:WU00:FS00:0xa4:Completed 4000000 out of 5000000 steps  (80%)
01:55:36:WU00:FS00:0xa4:Completed 4050000 out of 5000000 steps  (81%)
01:56:05:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
01:56:05:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
01:56:05:WU03:FS02:Connecting to 140.163.4.245:8080
01:56:28:WU04:FS01:0x21:Completed 2450000 out of 5000000 steps (49%)
01:57:54:WU00:FS00:0xa4:Completed 4100000 out of 5000000 steps  (82%)
01:58:22:WU03:FS02:Upload 0.77%
01:58:22:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
01:58:22:WU03:FS02:Trying to send results to collection server
01:58:22:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
01:58:22:WU03:FS02:Connecting to 128.252.203.2:8080
01:58:36:WU03:FS02:Upload 0.77%
01:58:43:WU03:FS02:Upload 1.55%
01:58:50:WU03:FS02:Upload 1.93%
01:59:07:WU03:FS02:Upload 2.32%
01:59:14:WU03:FS02:Upload 2.70%
01:59:22:WU03:FS02:Upload 3.09%
01:59:28:WU03:FS02:Upload 3.86%
01:59:34:WU03:FS02:Upload 4.25%
01:59:48:WU03:FS02:Upload 5.02%
01:59:54:WU03:FS02:Upload 5.41%
02:00:06:WU03:FS02:Upload 6.18%
02:00:09:WU00:FS00:0xa4:Completed 4150000 out of 5000000 steps  (83%)
02:00:16:WU03:FS02:Upload 6.57%
02:00:28:WU03:FS02:Upload 7.34%
02:00:36:WU03:FS02:Upload 8.11%
02:00:46:WU03:FS02:Upload 8.50%
02:01:13:WU03:FS02:Upload 8.89%
02:01:55:WU03:FS02:Upload 9.27%
02:02:01:WU03:FS02:Upload 9.66%
02:02:07:WU03:FS02:Upload 10.04%
02:02:09:WU00:FS00:0xa4:Completed 4200000 out of 5000000 steps  (84%)
02:02:16:WU03:FS02:Upload 10.43%
02:02:25:WU03:FS02:Upload 11.59%
02:02:38:WU03:FS02:Upload 12.36%
02:02:47:WU03:FS02:Upload 12.75%
02:02:54:WU03:FS02:Upload 13.13%
02:03:09:WU03:FS02:Upload 13.91%
02:03:17:WU03:FS02:Upload 14.29%
02:03:40:WU03:FS02:Upload 14.68%
02:04:02:WU03:FS02:Upload 15.07%
02:04:08:WU00:FS00:0xa4:Completed 4250000 out of 5000000 steps  (85%)
02:04:12:WU03:FS02:Upload 15.45%
02:04:20:WU03:FS02:Upload 15.84%
02:04:27:WU03:FS02:Upload 16.23%
02:04:36:WU03:FS02:Upload 16.61%
02:04:45:WU03:FS02:Upload 17.00%
02:04:55:WU02:FS02:0x21:Completed 620000 out of 2000000 steps (31%)
02:04:55:WU03:FS02:Upload 17.38%
02:05:09:WU03:FS02:Upload 18.16%
02:05:23:WU03:FS02:Upload 18.54%
02:05:27:WU04:FS01:0x21:Completed 2500000 out of 5000000 steps (50%)
02:05:30:WU03:FS02:Upload 18.93%
02:05:40:WU03:FS02:Upload 19.70%
02:05:52:WU03:FS02:Upload 20.09%
02:06:00:WU03:FS02:Upload 20.47%
02:06:04:WU00:FS00:0xa4:Completed 4300000 out of 5000000 steps  (86%)
02:06:08:WU03:FS02:Upload 20.86%
02:08:03:WU00:FS00:0xa4:Completed 4350000 out of 5000000 steps  (87%)
02:10:00:WU00:FS00:0xa4:Completed 4400000 out of 5000000 steps  (88%)
02:11:58:WU00:FS00:0xa4:Completed 4450000 out of 5000000 steps  (89%)
02:13:00:WU03:FS02:Upload 21.25%
02:13:00:ERROR:WU03:FS02:Exception: Transfer failed
02:13:55:WU00:FS00:0xa4:Completed 4500000 out of 5000000 steps  (90%)
02:14:33:WU04:FS01:0x21:Completed 2550000 out of 5000000 steps (51%)
02:15:51:WU00:FS00:0xa4:Completed 4550000 out of 5000000 steps  (91%)
02:17:48:WU00:FS00:0xa4:Completed 4600000 out of 5000000 steps  (92%)
02:19:46:WU00:FS00:0xa4:Completed 4650000 out of 5000000 steps  (93%)
02:20:31:WU02:FS02:0x21:Completed 640000 out of 2000000 steps (32%)
02:21:42:WU00:FS00:0xa4:Completed 4700000 out of 5000000 steps  (94%)
02:23:32:WU04:FS01:0x21:Completed 2600000 out of 5000000 steps (52%)
02:23:40:WU00:FS00:0xa4:Completed 4750000 out of 5000000 steps  (95%)
02:25:37:WU00:FS00:0xa4:Completed 4800000 out of 5000000 steps  (96%)
02:27:35:WU00:FS00:0xa4:Completed 4850000 out of 5000000 steps  (97%)
02:29:33:WU00:FS00:0xa4:Completed 4900000 out of 5000000 steps  (98%)
02:31:29:WU00:FS00:0xa4:Completed 4950000 out of 5000000 steps  (99%)
02:31:30:WU05:FS00:Connecting to 171.67.108.45:8080
02:31:31:WU05:FS00:Assigned to work server 128.252.203.2
02:31:31:WU05:FS00:Requesting new work unit for slot 00: RUNNING cpu:6 from 128.252.203.2
02:31:31:WU05:FS00:Connecting to 128.252.203.2:8080
02:31:33:WU05:FS00:Downloading 1.16MiB
02:31:36:WU05:FS00:Download complete
02:31:36:WU05:FS00:Received Unit: id:05 state:DOWNLOAD error:NO_ERROR project:11628 run:0 clone:41 gen:26 core:0xa4 unit:0x000000204c71bbb056971e6e1d72c27b
02:32:33:WU04:FS01:0x21:Completed 2650000 out of 5000000 steps (53%)
02:33:26:WU00:FS00:0xa4:Completed 5000000 out of 5000000 steps  (100%)
02:33:26:WU00:FS00:0xa4:DynamicWrapper: Finished Work Unit: sleep=10000
02:33:35:WU00:FS00:0xa4:
02:33:36:WU00:FS00:0xa4:Finished Work Unit:
02:33:36:WU00:FS00:0xa4:- Reading up to 1255848 from "00/wudata_01.trr": Read 1255848
02:33:36:WU00:FS00:0xa4:trr file hash check passed.
02:33:36:WU00:FS00:0xa4:- Reading up to 113904 from "00/wudata_01.xtc": Read 113904
02:33:36:WU00:FS00:0xa4:xtc file hash check passed.
02:33:36:WU00:FS00:0xa4:edr file hash check passed.
02:33:36:WU00:FS00:0xa4:logfile size: 86963
02:33:36:WU00:FS00:0xa4:Leaving Run
02:33:36:WU00:FS00:0xa4:- Writing 1528615 bytes of core data to disk...
02:33:36:WU00:FS00:0xa4:Done: 1528103 -> 1295967 (compressed to 84.8 percent)
02:33:36:WU00:FS00:0xa4:  ... Done.
02:33:36:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:33:36:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:6399 run:92 clone:32 gen:66 core:0xa4 unit:0x000000530002894b5462cc99f8b9c51f
02:33:36:WU00:FS00:Uploading 1.24MiB to 155.247.166.219
02:33:36:WU00:FS00:Connecting to 155.247.166.219:8080
02:33:36:WU05:FS00:Starting
02:33:36:WU05:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 05 -suffix 01 -version 704 -lifeline 9412 -checkpoint 15 -np 6
02:33:37:WU05:FS00:Started FahCore on PID 1768
02:33:37:WU05:FS00:Core PID:8560
02:33:37:WU05:FS00:FahCore 0xa4 started
02:33:37:WU05:FS00:0xa4:
02:33:37:WU05:FS00:0xa4:*------------------------------*
02:33:37:WU05:FS00:0xa4:Folding@Home Gromacs GB Core
02:33:37:WU05:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
02:33:37:WU05:FS00:0xa4:
02:33:37:WU05:FS00:0xa4:Preparing to commence simulation
02:33:37:WU05:FS00:0xa4:- Looking at optimizations...
02:33:37:WU05:FS00:0xa4:- Created dyn
02:33:37:WU05:FS00:0xa4:- Files status OK
02:33:37:WU05:FS00:0xa4:- Expanded 1215697 -> 2902760 (decompressed 238.7 percent)
02:33:37:WU05:FS00:0xa4:Called DecompressByteArray: compressed_data_size=1215697 data_size=2902760, decompressed_data_size=2902760 diff=0
02:33:37:WU05:FS00:0xa4:- Digital signature verified
02:33:37:WU05:FS00:0xa4:
02:33:37:WU05:FS00:0xa4:Project: 11628 (Run 0, Clone 41, Gen 26)
02:33:37:WU05:FS00:0xa4:
02:33:37:WU05:FS00:0xa4:Assembly optimizations on if available.
02:33:37:WU05:FS00:0xa4:Entering M.D.
02:33:42:WU00:FS00:Upload 20.22%
02:33:43:WU05:FS00:0xa4:Mapping NT from 6 to 6 
02:33:43:WU05:FS00:0xa4:Completed 0 out of 1250000 steps  (0%)
02:34:05:WU00:FS00:Upload 25.27%
02:34:12:WU00:FS00:Upload 30.33%
02:34:20:WU00:FS00:Upload 40.44%
02:34:31:WU00:FS00:Upload 45.49%
02:34:37:WU00:FS00:Upload 50.55%
02:34:48:WU00:FS00:Upload 55.60%
02:35:10:WU00:FS00:Upload 65.71%
02:35:24:WU00:FS00:Upload 75.82%
02:35:33:WU00:FS00:Upload 80.88%
02:36:03:WU00:FS00:Upload 85.93%
02:36:05:WU02:FS02:0x21:Completed 660000 out of 2000000 steps (33%)
02:36:11:WU00:FS00:Upload 90.99%
02:36:17:WU00:FS00:Upload 96.04%
02:36:43:WU00:FS00:Upload complete
02:36:43:WU00:FS00:Server responded WORK_ACK (400)
02:36:43:WU00:FS00:Final credit estimate, 2048.00 points
02:36:43:WU00:FS00:Cleaning up
02:38:23:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
02:38:23:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
02:38:23:WU01:FS01:Connecting to 140.163.4.235:8080
02:38:37:WU01:FS01:Upload 1.69%
02:38:56:WU01:FS01:Upload 3.39%
02:39:02:WU05:FS00:0xa4:Completed 12500 out of 1250000 steps  (1%)
02:39:05:WU01:FS01:Upload 3.95%
02:39:25:WU01:FS01:Upload 4.52%
02:39:25:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
02:39:25:WU01:FS01:Trying to send results to collection server
02:39:25:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
02:39:25:WU01:FS01:Connecting to 140.163.4.241:8080
02:39:40:WU01:FS01:Upload 1.13%
02:40:12:WU01:FS01:Upload 2.26%
02:40:22:WU01:FS01:Upload 2.82%
02:40:29:WU01:FS01:Upload 3.39%
02:40:44:WU01:FS01:Upload 3.95%
02:41:34:WU04:FS01:0x21:Completed 2700000 out of 5000000 steps (54%)
02:41:49:WU01:FS01:Upload 4.52%
02:42:53:WU01:FS01:Upload 5.08%
02:44:34:WU05:FS00:0xa4:Completed 25000 out of 1250000 steps  (2%)
02:44:54:WU01:FS01:Upload 5.65%
02:44:54:ERROR:WU01:FS01:Exception: Transfer failed
02:50:33:WU04:FS01:0x21:Completed 2750000 out of 5000000 steps (55%)
02:50:45:WU05:FS00:0xa4:Completed 37500 out of 1250000 steps  (3%)
02:51:21:WU02:FS02:0x21:Completed 680000 out of 2000000 steps (34%)
02:56:40:WU05:FS00:0xa4:Completed 50000 out of 1250000 steps  (4%)
02:59:35:WU04:FS01:0x21:Completed 2800000 out of 5000000 steps (56%)
03:02:35:WU05:FS00:0xa4:Completed 62500 out of 1250000 steps  (5%)
03:06:55:WU02:FS02:0x21:Completed 700000 out of 2000000 steps (35%)
03:08:07:WU05:FS00:0xa4:Completed 75000 out of 1250000 steps  (6%)
03:08:32:WU04:FS01:0x21:Completed 2850000 out of 5000000 steps (57%)
03:12:06:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
03:12:06:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
03:12:06:WU03:FS02:Connecting to 140.163.4.245:8080
03:12:14:WU03:FS02:Upload 0.77%
03:12:28:WU03:FS02:Upload 1.16%
03:12:37:WU03:FS02:Upload 1.93%
03:12:43:WU03:FS02:Upload 2.32%
03:12:51:WU03:FS02:Upload 2.70%
03:12:57:WU03:FS02:Upload 3.09%
03:13:05:WU03:FS02:Upload 3.48%
03:13:33:WU03:FS02:Upload 4.25%
03:13:39:WU05:FS00:0xa4:Completed 87500 out of 1250000 steps  (7%)
03:14:03:WU03:FS02:Upload 4.64%
03:14:10:WU03:FS02:Upload 5.02%
03:14:19:WU03:FS02:Upload 5.41%
03:17:32:WU04:FS01:0x21:Completed 2900000 out of 5000000 steps (58%)
03:17:47:WU03:FS02:Upload 5.79%
03:17:56:WU03:FS02:Upload 6.57%
03:18:11:WU03:FS02:Upload 7.34%
03:18:20:WU03:FS02:Upload 8.11%
03:18:27:WU03:FS02:Upload 8.50%
03:18:37:WU03:FS02:Upload 8.89%
03:18:44:WU03:FS02:Upload 9.27%
03:18:50:WU03:FS02:Upload 9.66%
03:18:56:WU03:FS02:Upload 10.04%
03:19:10:WU03:FS02:Upload 10.43%
03:19:19:WU03:FS02:Upload 10.82%
03:19:26:WU03:FS02:Upload 11.59%
03:19:27:WU05:FS00:0xa4:Completed 100000 out of 1250000 steps  (8%)
03:19:35:WU03:FS02:Upload 11.98%
03:19:43:WU03:FS02:Upload 12.36%
03:19:50:WU03:FS02:Upload 12.75%
03:20:07:WU03:FS02:Upload 13.13%
03:20:18:WU03:FS02:Upload 13.52%
03:20:25:WU03:FS02:Upload 13.91%
03:20:34:WU03:FS02:Upload 14.29%
03:20:44:WU03:FS02:Upload 14.68%
03:20:57:WU03:FS02:Upload 15.45%
03:21:08:WU03:FS02:Upload 15.84%
03:21:21:WU03:FS02:Upload 16.23%
03:21:30:WU03:FS02:Upload 16.61%
03:21:42:WU03:FS02:Upload 17.38%
03:21:52:WU03:FS02:Upload 17.77%
03:22:00:WU03:FS02:Upload 18.16%
03:22:06:WU02:FS02:0x21:Completed 720000 out of 2000000 steps (36%)
03:22:15:WU03:FS02:Upload 18.54%
03:22:45:WU03:FS02:Upload 19.32%
03:22:51:WU03:FS02:Upload 19.70%
03:23:09:WU03:FS02:Upload 20.47%
03:23:26:WU03:FS02:Upload 20.86%
03:23:32:WU03:FS02:Upload 21.25%
03:24:05:WU03:FS02:Upload 21.63%
03:24:18:WU03:FS02:Upload 22.02%
03:24:45:WU03:FS02:Upload 22.41%
03:24:54:WU03:FS02:Upload 22.79%
03:25:01:WU03:FS02:Upload 23.18%
03:25:07:WU03:FS02:Upload 23.57%
03:25:38:WU03:FS02:Upload 24.34%
03:25:44:WU03:FS02:Upload 24.72%
03:25:52:WU03:FS02:Upload 25.11%
03:26:01:WU03:FS02:Upload 25.50%
03:26:41:WU03:FS02:Upload 25.88%
03:26:47:WU03:FS02:Upload 26.27%
03:26:55:WU03:FS02:Upload 26.66%
03:27:01:WU03:FS02:Upload 27.04%
03:27:10:WU03:FS02:Upload 27.43%
03:27:27:WU03:FS02:Upload 27.81%
03:27:41:WU03:FS02:Upload 28.59%
03:27:49:WU03:FS02:Upload 28.97%
03:28:01:WU03:FS02:Upload 29.36%
03:28:26:WU03:FS02:Upload 29.75%
03:28:35:WU03:FS02:Upload 30.13%
03:28:59:WU03:FS02:Upload 30.52%
03:29:21:WU03:FS02:Upload 30.91%
03:29:40:WU03:FS02:Upload 31.29%
03:31:03:WU03:FS02:Upload 31.68%

03:31:54:WU03:FS02:Upload 32.06%
03:32:02:WU03:FS02:Upload 32.45%
03:37:34:WU03:FS02:Upload 32.84%
03:37:34:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
03:37:34:WU03:FS02:Trying to send results to collection server
03:37:34:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
03:37:34:WU03:FS02:Connecting to 128.252.203.2:8080
03:37:42:WU03:FS02:Upload 0.77%
03:37:51:WU03:FS02:Upload 1.16%
03:37:59:WU03:FS02:Upload 1.55%
03:38:07:WU03:FS02:Upload 1.93%
03:38:16:WU03:FS02:Upload 2.70%
03:38:22:WU03:FS02:Upload 3.48%
03:38:29:WU03:FS02:Upload 3.86%
03:38:42:WU03:FS02:Upload 4.25%
03:38:49:WU03:FS02:Upload 4.64%
03:38:59:WU03:FS02:Upload 5.02%
03:46:35:WU03:FS02:Upload 5.41%
03:46:35:ERROR:WU03:FS02:Exception: Transfer failed
******************************* Date: 2016-03-26 *******************************
04:41:23:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
04:41:23:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
04:41:23:WU01:FS01:Connecting to 140.163.4.235:8080
04:41:35:WU01:FS01:Upload 1.13%
04:42:05:WU01:FS01:Upload 1.69%
04:42:06:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
04:42:06:WU01:FS01:Trying to send results to collection server
04:42:06:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
04:42:06:WU01:FS01:Connecting to 140.163.4.241:8080
04:42:14:WU01:FS01:Upload 1.13%
04:42:20:WU01:FS01:Upload 1.69%
04:42:29:WU01:FS01:Upload 2.26%
04:42:38:WU01:FS01:Upload 2.82%
04:44:12:WU01:FS01:Upload 3.39%
04:44:12:ERROR:WU01:FS01:Exception: Transfer failed

05:15:05:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
05:15:05:WU03:FS02:Uploading 16.18MiB to 140.163.4.245
05:15:05:WU03:FS02:Connecting to 140.163.4.245:8080
05:15:16:WU03:FS02:Upload 0.77%
05:15:31:WU03:FS02:Upload 1.16%
05:15:41:WU03:FS02:Upload 1.93%
05:15:51:WU03:FS02:Upload 3.09%
05:16:00:WU03:FS02:Upload 3.86%
05:16:08:WU03:FS02:Upload 4.25%
05:16:28:WU03:FS02:Upload 5.02%
05:16:36:WU03:FS02:Upload 5.41%
05:16:56:WU03:FS02:Upload 6.18%
05:17:06:WU03:FS02:Upload 6.95%
05:17:15:WU03:FS02:Upload 7.34%
05:20:11:WU03:FS02:Upload 7.73%
05:20:11:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
05:20:11:WU03:FS02:Trying to send results to collection server
05:20:11:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
05:20:11:WU03:FS02:Connecting to 128.252.203.2:8080
05:20:18:WU03:FS02:Upload 0.77%
05:20:24:WU03:FS02:Upload 1.16%
05:20:31:WU03:FS02:Upload 1.55%
05:20:37:WU03:FS02:Upload 1.93%
05:20:53:WU03:FS02:Upload 2.70%
05:21:02:WU03:FS02:Upload 3.48%
05:21:34:WU03:FS02:Upload 3.86%
05:21:55:WU03:FS02:Upload 4.25%
05:22:33:WU03:FS02:Upload 4.64%
05:22:39:WU03:FS02:Upload 5.02%
05:22:46:WU03:FS02:Upload 5.41%

05:22:57:WU03:FS02:Upload 5.79%
05:23:04:WU03:FS02:Upload 6.18%
05:23:17:WU03:FS02:Upload 6.95%
05:23:24:WU03:FS02:Upload 7.34%
05:23:36:WU03:FS02:Upload 8.89%
05:23:46:WU03:FS02:Upload 9.66%
05:23:52:WU03:FS02:Upload 10.04%
05:24:00:WU03:FS02:Upload 10.82%
05:24:06:WU03:FS02:Upload 11.20%

05:24:23:WU03:FS02:Upload 11.98%
05:24:31:WU03:FS02:Upload 12.75%
05:24:45:WU03:FS02:Upload 13.52%
05:24:53:WU03:FS02:Upload 13.91%
05:25:00:WU03:FS02:Upload 14.29%
05:25:08:WU03:FS02:Upload 14.68%
05:25:18:WU03:FS02:Upload 15.45%
05:25:24:WU03:FS02:Upload 15.84%
05:25:30:WU03:FS02:Upload 16.23%
05:25:46:WU03:FS02:Upload 17.00%
05:25:56:WU03:FS02:Upload 17.77%
05:26:04:WU03:FS02:Upload 18.16%
05:26:17:WU03:FS02:Upload 18.54%
05:26:27:WU03:FS02:Upload 18.93%
05:26:34:WU03:FS02:Upload 19.70%
05:26:40:WU03:FS02:Upload 20.09%
05:26:50:WU03:FS02:Upload 20.47%
05:26:56:WU03:FS02:Upload 20.86%
05:27:03:WU03:FS02:Upload 21.63%
05:27:10:WU03:FS02:Upload 22.02%
05:27:18:WU03:FS02:Upload 22.79%
05:27:27:WU03:FS02:Upload 23.57%
05:27:38:WU03:FS02:Upload 24.34%
05:27:47:WU03:FS02:Upload 24.72%
05:27:56:WU03:FS02:Upload 25.50%
05:28:03:WU03:FS02:Upload 25.88%
05:28:10:WU03:FS02:Upload 26.27%
05:28:21:WU03:FS02:Upload 27.04%
05:28:40:WU03:FS02:Upload 27.81%
05:28:49:WU03:FS02:Upload 28.20%
05:29:23:WU03:FS02:Upload 28.59%
05:30:20:WU03:FS02:Upload 28.97%

05:30:29:WU03:FS02:Upload 29.36%
05:30:36:WU03:FS02:Upload 29.75%
05:30:49:WU03:FS02:Upload 30.13%
05:31:17:WU03:FS02:Upload 30.52%
05:31:32:WU03:FS02:Upload 30.91%

05:32:28:WU03:FS02:Upload 31.29%
05:32:37:WU03:FS02:Upload 32.45%
05:32:48:WU03:FS02:Upload 32.84%
05:33:15:WU03:FS02:Upload 33.22%
05:33:22:WU03:FS02:Upload 33.61%
05:33:30:WU03:FS02:Upload 34.00%
05:33:37:WU03:FS02:Upload 35.54%
05:33:46:WU03:FS02:Upload 36.31%
05:33:56:WU03:FS02:Upload 37.47%
05:34:05:WU03:FS02:Upload 39.02%
05:34:19:WU03:FS02:Upload 39.79%
05:34:29:WU03:FS02:Upload 40.56%
05:34:39:WU03:FS02:Upload 40.95%
05:34:46:WU03:FS02:Upload 41.34%
05:34:57:WU03:FS02:Upload 42.11%
05:35:04:WU03:FS02:Upload 42.88%
05:35:35:WU03:FS02:Upload 43.27%
05:35:35:ERROR:WU03:FS02:Exception: Transfer failed

06:57:59:WU05:FS00:0xa4:
06:58:35:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:10493 run:1 clone:21 gen:68 core:0x21 unit:0x000000548ca304f555d61665b263c85b
06:58:35:WU03:FS02:Uploading 16.18MiB to 140.163.4.245

06:58:36:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
06:58:36:WU01:FS01:Uploading 11.07MiB to 140.163.4.235

06:58:36:WU02:FS02:Starting
06:58:36:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 704 -lifeline 9412 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
06:58:36:WU02:FS02:Started FahCore on PID 10720
06:58:36:WU02:FS02:Core PID:4340
06:58:36:WU02:FS02:FahCore 0x21 started
06:58:36:WU04:FS01:Starting
06:58:36:WU04:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 04 -suffix 01 -version 704 -lifeline 9412 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
06:58:36:WU04:FS01:Started FahCore on PID 11792
06:58:36:WU04:FS01:Core PID:8668
06:58:36:WU04:FS01:FahCore 0x21 started
06:58:36:WU05:FS00:Starting
06:58:36:WU05:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/nickm/AppData/Roaming/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 05 -suffix 01 -version 704 -lifeline 9412 -checkpoint 15 -np 6
06:58:36:WU05:FS00:Started FahCore on PID 10192
06:58:36:WU05:FS00:Core PID:11028
06:58:36:WU05:FS00:FahCore 0xa4 started
06:58:36:WU02:FS02:0x21:*********************** Log Started 2016-03-26T06:58:36Z ***********************
06:58:36:WU02:FS02:0x21:Project: 13104 (Run 4, Clone 21, Gen 7)
06:58:36:WU02:FS02:0x21:Unit: 0x00000008ab436c9056b4f8084c1fc5fe
06:58:36:WU02:FS02:0x21:CPU: 0x00000000000000000000000000000000
06:58:36:WU02:FS02:0x21:Machine: 2
06:58:36:WU02:FS02:0x21:Digital signatures verified
06:58:36:WU02:FS02:0x21:Folding@home GPU Core21 Folding@home Core
06:58:36:WU02:FS02:0x21:Version 0.0.17
06:58:37:WU04:FS01:0x21:*********************** Log Started 2016-03-26T06:58:36Z ***********************
06:58:37:WU04:FS01:0x21:Project: 11406 (Run 4, Clone 16, Gen 61)
06:58:37:WU04:FS01:0x21:Unit: 0x0000004d8ca304f25686b1d2242e2186
06:58:37:WU04:FS01:0x21:CPU: 0x00000000000000000000000000000000
06:58:37:WU04:FS01:0x21:Machine: 1
06:58:37:WU04:FS01:0x21:Digital signatures verified
06:58:37:WU04:FS01:0x21:Folding@home GPU Core21 Folding@home Core
06:58:37:WU04:FS01:0x21:Version 0.0.17
06:58:37:WU05:FS00:0xa4:
06:58:37:WU05:FS00:0xa4:*------------------------------*
06:58:37:WU05:FS00:0xa4:Folding@Home Gromacs GB Core
06:58:37:WU05:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
06:58:37:WU05:FS00:0xa4:
06:58:37:WU05:FS00:0xa4:Preparing to commence simulation
06:58:37:WU05:FS00:0xa4:- Looking at optimizations...
06:58:37:WU05:FS00:0xa4:- Files status OK
06:58:37:WU05:FS00:0xa4:- Expanded 1215697 -> 2902760 (decompressed 238.7 percent)
06:58:37:WU05:FS00:0xa4:Called DecompressByteArray: compressed_data_size=1215697 data_size=2902760, decompressed_data_size=2902760 diff=0
06:58:37:WU05:FS00:0xa4:- Digital signature verified
06:58:37:WU05:FS00:0xa4:
06:58:37:WU05:FS00:0xa4:Project: 11628 (Run 0, Clone 41, Gen 26)

06:59:26:WU01:FS01:Upload 2.82%
06:59:33:WU01:FS01:Upload 3.39%
06:59:58:WU01:FS01:Upload 3.95%
07:00:45:WU01:FS01:Upload 4.52%
07:00:54:WU01:FS01:Upload 5.08%
07:01:06:WU01:FS01:Upload 5.65%
07:01:14:WU01:FS01:Upload 6.21%
07:01:24:WU01:FS01:Upload 6.77%
07:01:54:WU01:FS01:Upload 7.34%
07:02:03:WU01:FS01:Upload 7.90%
07:02:10:WU01:FS01:Upload 8.47%
07:02:16:WU01:FS01:Upload 9.60%

07:02:22:WU01:FS01:Upload 10.16%
07:02:46:WU01:FS01:Upload 11.29%
07:02:46:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
07:02:46:WU01:FS01:Trying to send results to collection server
07:02:46:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
07:02:46:WU01:FS01:Connecting to 140.163.4.241:8080
07:02:49:WU03:FS02:Upload 0.77%
07:02:49:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
07:02:49:WU03:FS02:Trying to send results to collection server
07:02:49:WU03:FS02:Uploading 16.18MiB to 128.252.203.2
07:02:49:WU03:FS02:Connecting to 128.252.203.2:8080
07:02:58:WU03:FS02:Upload 0.77%
07:03:06:WU03:FS02:Upload 1.16%
07:03:07:WU01:FS01:Upload 1.13%

07:03:12:WU03:FS02:Upload 1.55%
07:03:21:WU03:FS02:Upload 1.93%
07:03:29:WU03:FS02:Upload 2.32%
07:03:38:WU03:FS02:Upload 2.70%
07:03:46:WU03:FS02:Upload 3.48%
07:04:01:WU03:FS02:Upload 3.86%
07:04:10:WU03:FS02:Upload 4.25%
07:04:19:WU01:FS01:Upload 1.69%
07:04:22:WU03:FS02:Upload 5.02%
07:04:31:WU03:FS02:Upload 5.41%
07:04:37:WU01:FS01:Upload 2.26%
07:04:37:WU03:FS02:Upload 5.79%
07:04:44:WU01:FS01:Upload 3.39%
07:04:49:WU03:FS02:Upload 6.57%
07:04:56:WU01:FS01:Upload 4.52%
07:05:03:WU01:FS01:Upload 5.08%
07:05:05:WU03:FS02:Upload 6.95%
07:05:14:WU01:FS01:Upload 6.21%
07:05:15:WU03:FS02:Upload 7.34%
07:05:23:WU03:FS02:Upload 7.73%
07:05:29:WU03:FS02:Upload 8.11%
07:05:29:WU01:FS01:Upload 7.34%
07:05:39:WU01:FS01:Upload 7.90%
07:05:45:WU03:FS02:Upload 8.89%
07:05:47:WU01:FS01:Upload 8.47%
07:05:52:WU03:FS02:Upload 9.27%
07:05:55:WU01:FS01:Upload 9.60%
07:06:01:WU01:FS01:Upload 10.16%
07:06:03:WU03:FS02:Upload 10.04%
07:06:08:WU01:FS01:Upload 10.73%
07:06:17:WU01:FS01:Upload 11.29%
07:06:23:WU01:FS01:Upload 11.85%
07:06:34:WU03:FS02:Upload 10.82%
07:06:42:WU03:FS02:Upload 11.20%
07:06:49:WU03:FS02:Upload 11.59%
07:07:10:WU03:FS02:Upload 11.98%
07:07:17:WU03:FS02:Upload 12.36%
07:07:24:WU03:FS02:Upload 13.13%
07:07:32:WU03:FS02:Upload 13.52%
07:07:38:WU03:FS02:Upload 13.91%
07:07:47:WU03:FS02:Upload 14.68%
07:07:56:WU03:FS02:Upload 15.45%
07:08:01:WU01:FS01:Upload 12.42%
07:08:03:WU03:FS02:Upload 15.84%
07:08:10:WU01:FS01:Upload 12.98%
07:08:17:WU03:FS02:Upload 16.23%
07:08:17:WU01:FS01:Upload 13.55%
07:08:24:WU03:FS02:Upload 16.61%
07:08:28:WU01:FS01:Upload 14.11%
07:08:30:WU03:FS02:Upload 17.00%
07:08:45:WU01:FS01:Upload 14.68%
07:08:45:WU03:FS02:Upload 17.38%
07:08:54:WU03:FS02:Upload 17.77%
07:09:05:WU03:FS02:Upload 18.16%
07:09:47:WU03:FS02:Upload 19.70%
07:09:56:WU03:FS02:Upload 20.09%
07:10:06:WU01:FS01:Upload 15.24%
07:10:07:WU03:FS02:Upload 20.47%
07:10:14:WU01:FS01:Upload 15.81%
07:10:21:WU01:FS01:Upload 16.37%
07:10:30:WU01:FS01:Upload 16.94%
07:10:31:WU03:FS02:Upload 21.25%
07:10:41:WU01:FS01:Upload 17.50%
07:10:42:WU03:FS02:Upload 21.63%
07:10:50:WU03:FS02:Upload 22.41%
07:10:51:WU01:FS01:Upload 18.06%
07:11:00:WU01:FS01:Upload 18.63%
07:11:01:WU03:FS02:Upload 22.79%
07:11:07:WU01:FS01:Upload 19.19%
07:11:16:WU01:FS01:Upload 19.76%
07:11:29:WU01:FS01:Upload 20.32%
07:11:38:WU01:FS01:Upload 20.89%
07:11:39:WU03:FS02:Upload 23.57%
07:11:46:WU01:FS01:Upload 21.45%
07:11:47:WU03:FS02:Upload 24.34%
07:11:52:WU01:FS01:Upload 22.02%
07:11:54:WU03:FS02:Upload 24.72%
07:11:59:WU01:FS01:Upload 22.58%
07:12:02:WU03:FS02:Upload 25.11%
07:12:09:WU03:FS02:Upload 25.50%
07:12:13:WU01:FS01:Upload 23.14%
07:12:19:WU01:FS01:Upload 23.71%
07:12:22:WU03:FS02:Upload 26.27%
07:13:00:WU01:FS01:Upload 24.27%
07:13:08:WU01:FS01:Upload 24.84%
07:13:23:WU01:FS01:Upload 25.97%
07:13:43:WU01:FS01:Upload 26.53%
07:13:43:ERROR:WU01:FS01:Exception: Transfer failed
07:13:44:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:10471 run:0 clone:7 gen:310 core:0x18 unit:0x00000190538b3dbb53bea78bf5e04740
07:13:44:WU01:FS01:Uploading 11.07MiB to 140.163.4.235
07:13:44:WU01:FS01:Connecting to 140.163.4.235:8080
07:13:58:WU01:FS01:Upload 1.13%
07:14:23:WU01:FS01:Upload 1.69%
07:14:23:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
07:14:23:WU01:FS01:Trying to send results to collection server
07:14:23:WU01:FS01:Uploading 11.07MiB to 140.163.4.241
07:14:23:WU01:FS01:Connecting to 140.163.4.241:8080
07:14:34:WU01:FS01:Upload 1.13%

Foxbat · Post by **Foxbat** » Sat Mar 26, 2016 4:47 pm

Looks like you have two Work Units (WU01 and WU03) in the queue trying to transmit. In your log, the other WUs your system has processed and sent up results in a timely fashion. It probably doesn't help that these two are trying to be sent at the same time.

Does your ISP impose and sort of upload throttling to prevent Torrenting or Home Servers? With these GPU slot WU uploads being in the 18 MB range, I wouldn't think that would be big enough to attract attention. And like you said, transferring to OneDrive showed 50 GB/hr, but that could be due to OneDrive having your content server hosted in New Zealand or Australia, while you're trying to upload to the USA.

If the WUs are late, I'd hate to suggest throwing away processed results, but if it were happening to me, I'd Finish Folding on any current projects, wait until they got uploaded, stop the Folding Client, and delete the contents of your Work directory and the Queue file (using accepted methods; I'm on Ubuntu so I'm not familiar with the 64-bit Windows methods). Then I'd fire up Folding again. But that's me, and that's a bit drastic, and there's no guarantee it wouldn't just happen again.

Post by **bruce** » Sat Mar 26, 2016 6:58 pm

Yes, the two WUs bot transmit at the same time, so your bandwidth is effectively half what it would be otherwise.

Foxbat has the right idea, but let me suggest some minor changes:
First, if the WUs are late, the client should delete them automatically. Most likely it will delete only one and when only one is left, my guess is that it will upload successful.

I would not delete delete the work directory, only half of it. WU01 is in a subdirectory "02" and WU03 is in a subdirectory "03" so you can select which one you want to delete and leave the other to upload. Personally, I wish the client contained a way to manually suspend the uploading of a second WU for cases like this, but they're not working on any client enhancements, so it's not going to happen in V7.

Is the IP error rate especially high when you're using as much bandwidth as you can? Is there anything you can do about that?

Nick200 · Post by **Nick200** » Sat Mar 26, 2016 7:11 pm

Ok, getting silly now. Loads of uploads failing as in screenshot, from just one of four machines
http://imageshack.com/a/img922/5452/Xw3gAO.png

Folding Forum

Cant upload the WU in the time allowed.

Cant upload the WU in the time allowed.

Re: Failed to connect to 140.163.4.241

Re: Failed to connect to 140.163.4.241

Re: Failed to connect to 140.163.4.241

Re: Failed to connect to 140.163.4.241

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.

Re: Cant upload the WU in the time allowed.