Page 6 of 12
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sat Oct 05, 2019 3:24 am
by Celso Azevedo
No issues uploading so far, but sometimes downloads from 155.247.166.xxx get stuck without me noticing (and
this is the result). As others mentioned here, it can only be fixed by rebooting the machine.
Is there any way to avoid these servers? Changing the "cause preference" setting doesn't seem to help in this case.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sat Oct 05, 2019 8:07 pm
by JiiPee
bruce wrote:Your really posting in the wrong topic. The problems that have been a challenge for most other donors have been with the two servers 155.247.166.* and your log doesn't mention either them.
The easiest way to sort out the clients from each slot is to use the filter in FAHControl's log viewer. Create a view for each slot and/or each WU. (There may be a few lines missing because the filter doesn't recognize them.)
Downloading all three WUs concurrently is going to be slow but that's probably due to congestion on your own ISP connection. When I find myself needing to download all 3 WUs, I pick the one that I want to start first and do just that one, especially if some other tasks are using the internet connection. That way one or two GPUs can be computing while the last one or two WUs are downloading.
The setting "next-unit-percentage" was specifically designed for slow connections back in the day that we all had a POTS modem.
Can some give a good old kick for servers. Uploading or downloading for GPU works is not working.
Code: Select all
******************************* Date: 2019-10-04 *******************************
01:28:19:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14228 run:279 clone:1 gen:23 core:0x21 unit:0x0000002380fccb0a5d716f4bb9cb92eb
01:28:19:WU00:FS01:Uploading 130.82MiB to 128.252.203.10
01:28:19:WU00:FS01:Connecting to 128.252.203.10:8080
01:28:48:WU00:FS01:Upload 0.10%
01:28:48:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
01:28:48:WU00:FS01:Trying to send results to collection server
01:28:48:WU00:FS01:Uploading 130.82MiB to 155.247.166.220
01:28:48:WU00:FS01:Connecting to 155.247.166.220:8080
01:28:49:ERROR:WU00:FS01:Exception: Transfer failed
******************************* Date: 2019-10-05 *******************************
07:28:19:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14228 run:279 clone:1 gen:23 core:0x21 unit:0x0000002380fccb0a5d716f4bb9cb92eb
07:28:19:WU00:FS01:Uploading 130.82MiB to 128.252.203.10
07:28:19:WU00:FS01:Connecting to 128.252.203.10:8080
07:28:43:WU00:FS01:Upload 0.10%
07:28:43:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
07:28:43:WU00:FS01:Trying to send results to collection server
07:28:43:WU00:FS01:Uploading 130.82MiB to 155.247.166.220
07:28:43:WU00:FS01:Connecting to 155.247.166.220:8080
07:28:44:ERROR:WU00:FS01:Exception: Transfer failed
******************************* Date: 2019-10-05 *******************************
13:28:19:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14228 run:279 clone:1 gen:23 core:0x21 unit:0x0000002380fccb0a5d716f4bb9cb92eb
13:28:19:WU00:FS01:Uploading 130.82MiB to 128.252.203.10
13:28:19:WU00:FS01:Connecting to 128.252.203.10:8080
13:28:42:WU00:FS01:Upload 0.10%
13:28:42:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:28:42:WU00:FS01:Trying to send results to collection server
13:28:42:WU00:FS01:Uploading 130.82MiB to 155.247.166.220
13:28:42:WU00:FS01:Connecting to 155.247.166.220:8080
13:28:43:ERROR:WU00:FS01:Exception: Transfer failed
******************************* Date: 2019-10-05 *******************************
19:28:19:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14228 run:279 clone:1 gen:23 core:0x21 unit:0x0000002380fccb0a5d716f4bb9cb92eb
19:28:19:WU00:FS01:Uploading 130.82MiB to 128.252.203.10
19:28:19:WU00:FS01:Connecting to 128.252.203.10:8080
19:28:45:WU00:FS01:Upload 0.10%
19:28:45:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
19:28:45:WU00:FS01:Trying to send results to collection server
19:28:45:WU00:FS01:Uploading 130.82MiB to 155.247.166.220
19:28:45:WU00:FS01:Connecting to 155.247.166.220:8080
19:28:46:ERROR:WU00:FS01:Exception: Transfer failed
It's been like this already days. Sometimes you MIGHT be lucky and get something downloaded, but I haven't been able to download anything for GPU many days. CPU folding is working fine.
https://www.dropbox.com/s/ir4grck7fhg23 ... wnload.PNG
https://www.dropbox.com/s/0vqhn5g7y6xopb1/gpusend.PNG
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sun Oct 06, 2019 12:47 am
by JimF
I just started up again a couple of days ago on a GTX 1070 (Ubuntu 18.04), but have not had a problem from that server yet.
I could post the log if anyone is interested, but it is long and uninteresting insofar as I can see.
So maybe there is a routing problem somewhere? I have seen it happen.
(I am not far from Temple U. if it makes a difference).
Code: Select all
C:\Windows\system32>tracert 155.247.166.220
Tracing route to vav4.ocis.temple.edu [155.247.166.220]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.0.1
2 19 ms 26 ms 13 ms 10.50.48.1
3 15 ms 19 ms 19 ms bdle8-sub211.aggr1.phdl.pa.rcn.net [207.172.196.239]
4 18 ms 23 ms 25 ms hge0-0-0-4.core2.phdl.pa.rcn.net [207.172.19.220]
5 19 ms 18 ms 20 ms 207.172.19.100
6 21 ms 27 ms 20 ms hge0-4-0-1.border1.eqnx.va.rcn.net [207.172.19.24]
7 20 ms 25 ms 25 ms eeq-exchange.tr01-asbnva01.transitrail.net [206.126.236.45]
8 * * * Request timed out.
9 21 ms 27 ms 23 ms 216.27.100.14
10 23 ms 23 ms 19 ms 216.27.100.13
11 30 ms 21 ms 29 ms remote.templeconwell.magpi.net [216.27.98.238]
12 * * * Request timed out.
13 20 ms 28 ms 23 ms vav4.ocis.temple.edu [155.247.166.220]
Trace complete.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sun Oct 06, 2019 12:50 am
by Sn0wy23
Several days I've had to kill processes a few times a day and bring back online to get the frozen slots to work. I bought a new card the other month and thought it was just my config, but reading through this and checking the info, I to am having issues with this 155.247.166. with both download and uploading. I had a project take 42 minutes to upload (Longest one I have seen in logs), used to be so much quicker. Downloads frozen on both machines required same thing, full exit of client, kill process and restart the client. I'd rather not have to reboot my machines if possible.
Hopefully things will improve and the servers can take the strain.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sun Oct 06, 2019 12:09 pm
by JiiPee
I did manually delete that WU what didn't upload and now folding has been working fine at least one night.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sun Oct 06, 2019 10:55 pm
by bruce
The network admins at Temple.edu have been working on this problem but apparently it's an intermittent problem which makes it difficult for them to isolate (and correct). About the time they do something to get the backlog of uploads accepted by either the WS or the CS, it seems like the problem has been fixed so they re-enable the "normal" level of WU assignments. This takes maybe a day before the backlog builds up again and complaints start again.
If my assumptions are correct, it's a difficult job of finding a reasonable level of server assignment rates that the servers can sustain at a stable rate of upload/download performance. Though II don't know, I suspect that the servers are not dedicated to FAH but are shared by other campus computer functions, forcing the balancing act to become an inter-departmental issue. Moreover, if FAH needs to invest in more computer resources, it probably becomes a campus-wide budgeting issue .... Good luck on finding a quick solution.
Most of what I'm saying is based on guesswork, to if I'm wrong, I apologize.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Mon Oct 07, 2019 6:46 am
by FaaR
bruce wrote:Your really posting in the wrong topic. The problems that have been a challenge for most other donors have been with the two servers 155.247.166.* and your log doesn't mention either them.
I am? Well I originally made my own thread, and it got merged with this one so I kept posting here.
If I'm posting in the wrong thread I feel I can hardly be the one to blame for it!
Anyhow since the thread is (mostly) about downloading WUs I thought you'd appreciate the feedback, especially as I've been having mostly slow downloads of late.
Downloading all three WUs concurrently is going to be slow but that's probably due to congestion on your own ISP connection.
Not likely. Swedish ISPs don't cap or throttle their broadband users willy nilly, and I have 500Mbit/s connection; it eats multiple 70MB downloads for breakfast honestly.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Mon Oct 07, 2019 8:41 pm
by JiiPee
bruce wrote:The network admins at Temple.edu have been working on this problem but apparently it's an intermittent problem which makes it difficult for them to isolate (and correct). About the time they do something to get the backlog of uploads accepted by either the WS or the CS, it seems like the problem has been fixed so they re-enable the "normal" level of WU assignments. This takes maybe a day before the backlog builds up again and complaints start again.
If my assumptions are correct, it's a difficult job of finding a reasonable level of server assignment rates that the servers can sustain at a stable rate of upload/download performance. Though II don't know, I suspect that the servers are not dedicated to FAH but are shared by other campus computer functions, forcing the balancing act to become an inter-departmental issue. Moreover, if FAH needs to invest in more computer resources, it probably becomes a campus-wide budgeting issue .... Good luck on finding a quick solution.
Most of what I'm saying is based on guesswork, to if I'm wrong, I apologize.
I have that deleted WU still archived with 7zip on my desktop, would it be any use to send it somewhere for inspect why it didn't like to get through, because after I deleted it from fah working folder, all has been running fine.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Tue Oct 08, 2019 12:46 am
by bruce
Delete them. Preserving WUs locally has never been useful. By the time anything can be done with them manually, they will have expired and will have been reissued to someone else.
They, and many other WU, had troubles getting through due to some kind of resource limitation at temple.edu. Most were retried several times and some did get through.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Tue Oct 08, 2019 6:50 pm
by HaloJones
Oh, browser caches. My clients are largely Linux-based.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Fri Oct 11, 2019 6:44 pm
by dfgirl12
It looks like it's time for another reboot on both servers: 155.247.166.*. Those 2 servers seem to make it to 3-4 days of uptime before the slow/failing uploads begin.
I went this past week without any noticeable WU upload backups, until today.
The daily reboots on 128.252.203.10 seem to help. It looks like 128.252.203.10 was just rebooted.
Code: Select all
2019-10-11:17:29:07:WU01:FS01:Upload 45.73%
2019-10-11:17:29:11:WU02:FS01:Upload 43.65%
2019-10-11:17:29:12:WU00:FS01:Upload 45.66%
2019-10-11:17:29:12:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:17:29:12:WU02:FS01:Trying to send results to collection server
2019-10-11:17:29:12:WU02:FS01:Uploading 81.47MiB to 155.247.166.220
2019-10-11:17:29:12:WU02:FS01:Connecting to 155.247.166.220:8080
2019-10-11:17:29:13:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-11:17:29:13:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14276 run:0 clone:30 gen:7 core:0x21 unit:0x0000000780fccb0a5d98ab5ca79c42f4
2019-10-11:17:29:13:WU02:FS01:Uploading 81.47MiB to 128.252.203.10
2019-10-11:17:29:13:WU02:FS01:Connecting to 128.252.203.10:8080
2019-10-11:17:29:14:WU01:FS01:Upload 45.91%
2019-10-11:17:29:19:WU00:FS01:Upload 45.95%
2019-10-11:17:29:20:WU01:FS01:Upload 45.99%
2019-10-11:17:29:21:WU02:FS01:Upload 0.23%
2019-10-11:17:29:25:WU03:FS01:0x21:Completed 520000 out of 1000000 steps (52%)
2019-10-11:17:29:27:WU00:FS01:Upload 46.15%
2019-10-11:17:29:27:WU01:FS01:Upload 46.34%
2019-10-11:17:29:33:WU01:FS01:Upload 46.70%
2019-10-11:17:29:37:WU00:FS01:Upload 46.44%
2019-10-11:17:29:38:WU02:FS01:Upload 0.38%
2019-10-11:17:29:46:WU00:FS01:Upload 46.63%
2019-10-11:17:29:54:WU00:FS01:Upload 46.92%
2019-10-11:17:29:57:WU01:FS01:Upload 46.96%
2019-10-11:17:30:01:WU00:FS01:Upload 47.11%
2019-10-11:17:30:05:WU03:FS01:0x21:Completed 530000 out of 1000000 steps (53%)
2019-10-11:17:30:08:WU00:FS01:Upload 47.40%
2019-10-11:17:30:14:WU01:FS01:Upload 47.13%
2019-10-11:17:30:15:WU00:FS01:Upload 47.60%
2019-10-11:17:30:20:WU01:FS01:Upload 47.31%
2019-10-11:17:30:21:WU00:FS01:Upload 47.89%
2019-10-11:17:30:26:WU01:FS01:Upload 47.57%
2019-10-11:17:30:34:WU01:FS01:Upload 47.84%
2019-10-11:17:30:40:WU02:FS01:Upload 0.46%
2019-10-11:17:30:40:WU03:FS01:0x21:Completed 540000 out of 1000000 steps (54%)
2019-10-11:17:30:46:WU02:FS01:Upload 0.77%
2019-10-11:17:30:52:WU02:FS01:Upload 1.00%
2019-10-11:17:30:54:WU00:FS01:Upload 48.08%
2019-10-11:17:30:57:WU01:FS01:Upload 47.92%
2019-10-11:17:30:59:WU02:FS01:Upload 1.30%
2019-10-11:17:31:00:WU00:FS01:Upload 48.37%
2019-10-11:17:31:05:WU02:FS01:Upload 1.46%
2019-10-11:17:31:05:WU01:FS01:Upload 48.01%
2019-10-11:17:31:07:WU00:FS01:Upload 48.56%
2019-10-11:17:31:12:WU02:FS01:Upload 1.76%
2019-10-11:17:31:13:WU00:FS01:Upload 48.76%
2019-10-11:17:31:13:WU01:FS01:Upload 48.19%
2019-10-11:17:31:19:WU00:FS01:Upload 48.95%
2019-10-11:17:31:20:WU03:FS01:0x21:Completed 550000 out of 1000000 steps (55%)
2019-10-11:17:31:21:WU01:FS01:Upload 48.45%
2019-10-11:17:31:25:WU00:FS01:Upload 49.24%
2019-10-11:17:31:27:WU01:FS01:Upload 48.63%
2019-10-11:17:31:33:WU01:FS01:Upload 48.89%
2019-10-11:17:31:37:WU00:FS01:Upload 49.63%
2019-10-11:17:31:42:WU01:FS01:Upload 49.15%
2019-10-11:17:31:48:WU00:FS01:Upload 49.82%
2019-10-11:17:31:50:WU01:FS01:Upload 49.33%
2019-10-11:17:31:56:WU03:FS01:0x21:Completed 560000 out of 1000000 steps (56%)
2019-10-11:17:31:57:WU00:FS01:Upload 50.02%
2019-10-11:17:31:58:WU01:FS01:Upload 49.59%
2019-10-11:17:32:04:WU00:FS01:Upload 50.21%
2019-10-11:17:32:06:WU01:FS01:Upload 49.86%
2019-10-11:17:32:12:WU00:FS01:Upload 50.50%
2019-10-11:17:32:13:WU01:FS01:Upload 50.12%
2019-10-11:17:32:21:WU01:FS01:Upload 50.47%
2019-10-11:17:32:22:WU00:FS01:Upload 50.79%
2019-10-11:17:32:27:WU01:FS01:Upload 50.65%
2019-10-11:17:32:29:WU00:FS01:Upload 51.08%
2019-10-11:17:32:33:WU01:FS01:Upload 50.91%
2019-10-11:17:32:35:WU02:FS01:Upload 1.92%
2019-10-11:17:32:35:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:17:32:35:WU02:FS01:Trying to send results to collection server
2019-10-11:17:32:35:WU02:FS01:Uploading 81.47MiB to 155.247.166.220
2019-10-11:17:32:35:WU02:FS01:Connecting to 155.247.166.220:8080
2019-10-11:17:32:36:WU03:FS01:0x21:Completed 570000 out of 1000000 steps (57%)
2019-10-11:17:32:36:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-11:17:32:36:WU00:FS01:Upload 51.37%
2019-10-11:17:32:39:WU01:FS01:Upload 51.26%
2019-10-11:17:32:45:WU01:FS01:Upload 51.52%
2019-10-11:17:32:51:WU01:FS01:Upload 51.70%
2019-10-11:17:32:57:WU00:FS01:Upload 51.47%
2019-10-11:17:33:03:WU00:FS01:Upload 51.85%
2019-10-11:17:33:09:WU00:FS01:Upload 52.14%
2019-10-11:17:33:11:WU03:FS01:0x21:Completed 580000 out of 1000000 steps (58%)
2019-10-11:17:33:22:WU00:FS01:Upload 52.34%
2019-10-11:17:33:28:WU00:FS01:Upload 52.53%
2019-10-11:17:33:34:WU00:FS01:Upload 52.82%
2019-10-11:17:33:41:WU00:FS01:Upload 53.01%
2019-10-11:17:33:42:WU01:FS01:Upload 51.79%
2019-10-11:17:33:47:WU00:FS01:Upload 53.21%
2019-10-11:17:33:50:WU01:FS01:Upload 51.96%
2019-10-11:17:33:51:WU03:FS01:0x21:Completed 590000 out of 1000000 steps (59%)
2019-10-11:17:33:57:WU01:FS01:Upload 52.23%
2019-10-11:17:33:58:WU00:FS01:Upload 53.40%
2019-10-11:17:34:03:WU01:FS01:Upload 52.49%
2019-10-11:17:34:09:WU01:FS01:Upload 52.84%
2019-10-11:17:34:15:WU01:FS01:Upload 53.89%
2019-10-11:17:34:21:WU01:FS01:Upload 54.07%
2019-10-11:17:34:27:WU03:FS01:0x21:Completed 600000 out of 1000000 steps (60%)
2019-10-11:17:34:28:WU01:FS01:Upload 54.33%
2019-10-11:17:34:35:WU01:FS01:Upload 54.60%
2019-10-11:17:34:41:WU01:FS01:Upload 54.77%
2019-10-11:17:34:49:WU01:FS01:Upload 55.12%
2019-10-11:17:34:56:WU01:FS01:Upload 55.39%
2019-10-11:17:34:58:WU00:FS01:Upload 53.50%
2019-10-11:17:34:58:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:17:34:58:WU00:FS01:Trying to send results to collection server
2019-10-11:17:34:58:WU00:FS01:Uploading 64.60MiB to 155.247.166.219
2019-10-11:17:34:58:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-11:17:34:58:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-11:17:34:58:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14274 run:0 clone:0 gen:39 core:0x21 unit:0x0000002880fccb0a5d98ab50a43fcdbe
2019-10-11:17:34:58:WU00:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-11:17:34:58:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-11:17:35:02:WU01:FS01:Upload 55.56%
2019-10-11:17:35:06:WU00:FS01:Upload 0.29%
2019-10-11:17:35:07:WU03:FS01:0x21:Completed 610000 out of 1000000 steps (61%)
2019-10-11:17:35:09:WU01:FS01:Upload 55.74%
2019-10-11:17:35:12:WU00:FS01:Upload 0.39%
2019-10-11:17:35:18:WU01:FS01:Upload 55.91%
2019-10-11:17:35:18:WU00:FS01:Upload 0.68%
2019-10-11:17:35:24:WU00:FS01:Upload 0.87%
2019-10-11:17:35:32:WU00:FS01:Upload 1.06%
2019-10-11:17:35:38:WU00:FS01:Upload 1.26%
2019-10-11:17:35:42:WU03:FS01:0x21:Completed 620000 out of 1000000 steps (62%)
2019-10-11:17:35:43:WU01:FS01:Upload 56.00%
2019-10-11:17:35:48:WU00:FS01:Upload 1.35%
2019-10-11:17:35:52:WU01:FS01:Upload 56.26%
2019-10-11:17:35:54:WU00:FS01:Upload 1.64%
2019-10-11:17:36:00:WU01:FS01:Upload 56.53%
2019-10-11:17:36:02:WU00:FS01:Upload 2.03%
2019-10-11:17:36:07:WU01:FS01:Upload 56.70%
2019-10-11:17:36:14:WU00:FS01:Upload 2.32%
2019-10-11:17:36:15:WU01:FS01:Upload 56.88%
2019-10-11:17:36:21:WU01:FS01:Upload 57.05%
2019-10-11:17:36:22:WU03:FS01:0x21:Completed 630000 out of 1000000 steps (63%)
2019-10-11:17:36:29:WU01:FS01:Upload 57.23%
2019-10-11:17:36:36:WU01:FS01:Upload 57.58%
2019-10-11:17:36:38:WU00:FS01:Upload 2.42%
2019-10-11:17:36:42:WU01:FS01:Upload 57.84%
2019-10-11:17:36:46:WU00:FS01:Upload 2.61%
2019-10-11:17:36:55:WU01:FS01:Upload 58.11%
2019-10-11:17:36:58:WU03:FS01:0x21:Completed 640000 out of 1000000 steps (64%)
2019-10-11:17:37:00:WU00:FS01:Upload 2.90%
2019-10-11:17:37:01:WU01:FS01:Upload 58.46%
2019-10-11:17:37:08:WU01:FS01:Upload 58.63%
2019-10-11:17:37:09:WU00:FS01:Upload 3.39%
2019-10-11:17:37:15:WU01:FS01:Upload 58.90%
2019-10-11:17:37:16:WU00:FS01:Upload 3.68%
2019-10-11:17:37:21:WU01:FS01:Upload 59.07%
2019-10-11:17:37:22:WU00:FS01:Upload 3.87%
2019-10-11:17:37:27:WU01:FS01:Upload 59.25%
2019-10-11:17:37:29:WU00:FS01:Upload 4.26%
2019-10-11:17:37:33:WU01:FS01:Upload 59.42%
2019-10-11:17:37:36:WU00:FS01:Upload 4.55%
2019-10-11:17:37:38:WU03:FS01:0x21:Completed 650000 out of 1000000 steps (65%)
2019-10-11:17:37:43:WU00:FS01:Upload 4.84%
2019-10-11:17:37:46:WU01:FS01:Upload 59.60%
2019-10-11:17:37:53:WU01:FS01:Upload 59.95%
2019-10-11:17:37:56:WU00:FS01:Upload 5.22%
2019-10-11:17:37:59:WU01:FS01:Upload 60.21%
2019-10-11:17:38:02:WU00:FS01:Upload 5.51%
2019-10-11:17:38:05:WU01:FS01:Upload 60.56%
2019-10-11:17:38:08:WU00:FS01:Upload 5.80%
2019-10-11:17:38:12:WU01:FS01:Upload 60.74%
2019-10-11:17:38:14:WU03:FS01:0x21:Completed 660000 out of 1000000 steps (66%)
2019-10-11:17:38:15:WU00:FS01:Upload 6.09%
2019-10-11:17:38:18:WU01:FS01:Upload 60.91%
2019-10-11:17:38:24:WU00:FS01:Upload 6.19%
2019-10-11:17:38:29:WU01:FS01:Upload 61.18%
2019-10-11:17:38:30:WU00:FS01:Upload 6.48%
2019-10-11:17:38:36:WU01:FS01:Upload 61.44%
2019-10-11:17:38:39:WU00:FS01:Upload 6.77%
2019-10-11:17:38:42:WU01:FS01:Upload 61.70%
2019-10-11:17:38:47:WU00:FS01:Upload 6.87%
2019-10-11:17:38:53:WU03:FS01:0x21:Completed 670000 out of 1000000 steps (67%)
2019-10-11:17:39:17:WU00:FS01:Upload 6.97%
2019-10-11:17:39:29:WU03:FS01:0x21:Completed 680000 out of 1000000 steps (68%)
2019-10-11:17:39:33:WU01:FS01:Upload 62.06%
2019-10-11:17:39:39:WU01:FS01:Upload 62.23%
2019-10-11:17:39:45:WU01:FS01:Upload 62.41%
2019-10-11:17:39:53:WU01:FS01:Upload 62.85%
2019-10-11:17:39:59:WU01:FS01:Upload 63.02%
2019-10-11:17:40:08:WU01:FS01:Upload 63.46%
2019-10-11:17:40:09:WU03:FS01:0x21:Completed 690000 out of 1000000 steps (69%)
2019-10-11:17:40:16:WU01:FS01:Upload 63.64%
2019-10-11:17:40:24:WU01:FS01:Upload 63.90%
2019-10-11:17:40:30:WU01:FS01:Upload 64.07%
2019-10-11:17:40:38:WU01:FS01:Upload 64.34%
2019-10-11:17:40:44:WU00:FS01:Upload 7.06%
2019-10-11:17:40:44:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:17:40:44:WU00:FS01:Trying to send results to collection server
2019-10-11:17:40:44:WU00:FS01:Uploading 64.60MiB to 155.247.166.219
2019-10-11:17:40:44:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-11:17:40:45:WU03:FS01:0x21:Completed 700000 out of 1000000 steps (70%)
2019-10-11:17:40:45:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-11:17:40:55:WU01:FS01:Upload 64.69%
2019-10-11:17:41:05:WU01:FS01:Upload 64.86%
2019-10-11:17:41:12:WU01:FS01:Upload 65.04%
2019-10-11:17:41:19:WU01:FS01:Upload 65.13%
2019-10-11:17:41:25:WU03:FS01:0x21:Completed 710000 out of 1000000 steps (71%)
2019-10-11:17:41:25:WU01:FS01:Upload 65.39%
2019-10-11:17:41:32:WU01:FS01:Upload 65.74%
2019-10-11:17:41:38:WU01:FS01:Upload 65.92%
2019-10-11:17:41:44:WU01:FS01:Upload 66.18%
2019-10-11:17:41:56:WU01:FS01:Upload 66.36%
2019-10-11:17:42:00:WU03:FS01:0x21:Completed 720000 out of 1000000 steps (72%)
2019-10-11:17:42:02:WU01:FS01:Upload 66.53%
2019-10-11:17:42:09:WU01:FS01:Upload 66.71%
2019-10-11:17:42:18:WU01:FS01:Upload 66.97%
2019-10-11:17:42:24:WU01:FS01:Upload 67.15%
2019-10-11:17:42:30:WU01:FS01:Upload 67.32%
2019-10-11:17:42:40:WU03:FS01:0x21:Completed 730000 out of 1000000 steps (73%)
2019-10-11:17:42:41:WU01:FS01:Upload 67.67%
2019-10-11:17:42:48:WU01:FS01:Upload 67.94%
2019-10-11:17:42:57:WU01:FS01:Upload 68.29%
2019-10-11:17:43:03:WU01:FS01:Upload 68.81%
2019-10-11:17:43:09:WU01:FS01:Upload 69.08%
2019-10-11:17:43:16:WU03:FS01:0x21:Completed 740000 out of 1000000 steps (74%)
2019-10-11:17:43:16:WU01:FS01:Upload 69.34%
2019-10-11:17:43:22:WU01:FS01:Upload 69.52%
2019-10-11:17:43:28:WU01:FS01:Upload 69.69%
2019-10-11:17:43:34:WU01:FS01:Upload 69.96%
2019-10-11:17:43:42:WU01:FS01:Upload 70.22%
2019-10-11:17:43:51:WU01:FS01:Upload 70.39%
2019-10-11:17:43:56:WU03:FS01:0x21:Completed 750000 out of 1000000 steps (75%)
2019-10-11:17:43:57:WU01:FS01:Upload 70.57%
2019-10-11:17:44:03:WU01:FS01:Upload 70.83%
2019-10-11:17:44:10:WU01:FS01:Upload 71.01%
2019-10-11:17:44:17:WU01:FS01:Upload 71.36%
2019-10-11:17:44:24:WU01:FS01:Upload 71.54%
2019-10-11:17:44:30:WU01:FS01:Upload 71.80%
2019-10-11:17:44:31:WU03:FS01:0x21:Completed 760000 out of 1000000 steps (76%)
2019-10-11:17:44:36:WU01:FS01:Upload 72.06%
2019-10-11:17:44:44:WU01:FS01:Upload 72.15%
2019-10-11:17:44:51:WU01:FS01:Upload 72.41%
2019-10-11:17:44:58:WU01:FS01:Upload 72.59%
2019-10-11:17:45:05:WU01:FS01:Upload 72.76%
2019-10-11:17:45:11:WU01:FS01:Upload 72.85%
2019-10-11:17:45:11:WU03:FS01:0x21:Completed 770000 out of 1000000 steps (77%)
2019-10-11:17:45:39:WU01:FS01:Upload 73.03%
2019-10-11:17:45:47:WU01:FS01:Upload 73.29%
2019-10-11:17:45:47:WU03:FS01:0x21:Completed 780000 out of 1000000 steps (78%)
2019-10-11:17:45:54:WU01:FS01:Upload 73.47%
2019-10-11:17:46:00:WU01:FS01:Upload 73.73%
2019-10-11:17:46:06:WU01:FS01:Upload 74.08%
2019-10-11:17:46:13:WU01:FS01:Upload 74.43%
2019-10-11:17:46:21:WU01:FS01:Upload 74.78%
2019-10-11:17:46:27:WU03:FS01:0x21:Completed 790000 out of 1000000 steps (79%)
2019-10-11:17:46:27:WU01:FS01:Upload 75.13%
2019-10-11:17:46:33:WU01:FS01:Upload 75.31%
2019-10-11:17:46:39:WU01:FS01:Upload 75.49%
2019-10-11:17:46:47:WU01:FS01:Upload 75.57%
2019-10-11:17:46:55:WU01:FS01:Upload 75.84%
2019-10-11:17:47:02:WU03:FS01:0x21:Completed 800000 out of 1000000 steps (80%)
2019-10-11:17:47:36:WU01:FS01:Upload 76.19%
2019-10-11:17:47:42:WU03:FS01:0x21:Completed 810000 out of 1000000 steps (81%)
2019-10-11:17:47:45:WU01:FS01:Upload 76.28%
2019-10-11:17:47:53:WU01:FS01:Upload 76.71%
2019-10-11:17:48:01:WU01:FS01:Upload 77.07%
2019-10-11:17:48:07:WU01:FS01:Upload 77.33%
2019-10-11:17:48:13:WU01:FS01:Upload 77.59%
2019-10-11:17:48:18:WU03:FS01:0x21:Completed 820000 out of 1000000 steps (82%)
2019-10-11:17:48:19:WU01:FS01:Upload 77.77%
2019-10-11:17:48:26:WU01:FS01:Upload 78.03%
2019-10-11:17:48:34:WU01:FS01:Upload 78.38%
2019-10-11:17:48:40:WU01:FS01:Upload 78.91%
2019-10-11:17:48:50:WU01:FS01:Upload 79.00%
2019-10-11:17:48:56:WU01:FS01:Upload 79.26%
2019-10-11:17:48:58:WU03:FS01:0x21:Completed 830000 out of 1000000 steps (83%)
2019-10-11:17:49:06:WU01:FS01:Upload 79.35%
2019-10-11:17:49:16:WU01:FS01:Upload 79.61%
2019-10-11:17:49:23:WU01:FS01:Upload 80.05%
2019-10-11:17:49:33:WU03:FS01:0x21:Completed 840000 out of 1000000 steps (84%)
2019-10-11:17:49:38:WU01:FS01:Upload 80.22%
2019-10-11:17:49:44:WU01:FS01:Upload 80.58%
2019-10-11:17:49:50:WU01:FS01:Upload 80.75%
2019-10-11:17:49:56:WU01:FS01:Upload 80.93%
2019-10-11:17:50:05:WU01:FS01:Upload 81.28%
2019-10-11:17:50:11:WU01:FS01:Upload 81.63%
2019-10-11:17:50:13:WU03:FS01:0x21:Completed 850000 out of 1000000 steps (85%)
2019-10-11:17:50:17:WU01:FS01:Upload 81.80%
2019-10-11:17:50:25:WU01:FS01:Upload 82.07%
2019-10-11:17:50:31:WU01:FS01:Upload 82.24%
2019-10-11:17:50:40:WU01:FS01:Upload 82.59%
2019-10-11:17:50:47:WU01:FS01:Upload 82.86%
2019-10-11:17:50:49:WU03:FS01:0x21:Completed 860000 out of 1000000 steps (86%)
2019-10-11:17:50:53:WU01:FS01:Upload 83.03%
2019-10-11:17:50:59:WU01:FS01:Upload 83.21%
2019-10-11:17:51:05:WU01:FS01:Upload 83.47%
2019-10-11:17:51:29:WU03:FS01:0x21:Completed 870000 out of 1000000 steps (87%)
2019-10-11:17:51:44:WU01:FS01:Upload 83.56%
2019-10-11:17:51:51:WU01:FS01:Upload 83.74%
2019-10-11:17:51:58:WU01:FS01:Upload 84.35%
2019-10-11:17:52:05:WU03:FS01:0x21:Completed 880000 out of 1000000 steps (88%)
2019-10-11:17:52:13:WU01:FS01:Upload 84.61%
2019-10-11:17:52:19:WU01:FS01:Upload 84.70%
2019-10-11:17:52:25:WU01:FS01:Upload 85.05%
2019-10-11:17:52:33:WU01:FS01:Upload 85.40%
2019-10-11:17:52:42:WU01:FS01:Upload 85.58%
2019-10-11:17:52:44:WU03:FS01:0x21:Completed 890000 out of 1000000 steps (89%)
2019-10-11:17:52:50:WU01:FS01:Upload 85.93%
2019-10-11:17:52:57:WU01:FS01:Upload 86.11%
2019-10-11:17:53:03:WU01:FS01:Upload 86.28%
2019-10-11:17:53:20:WU03:FS01:0x21:Completed 900000 out of 1000000 steps (90%)
2019-10-11:17:53:23:WU01:FS01:Upload 86.54%
2019-10-11:17:53:46:WU01:FS01:Upload 86.63%
2019-10-11:17:54:00:WU03:FS01:0x21:Completed 910000 out of 1000000 steps (91%)
2019-10-11:17:54:36:WU03:FS01:0x21:Completed 920000 out of 1000000 steps (92%)
2019-10-11:17:54:39:WU01:FS01:Upload 86.72%
2019-10-11:17:54:47:WU01:FS01:Upload 86.90%
2019-10-11:17:55:02:WU01:FS01:Upload 87.25%
2019-10-11:17:55:08:WU01:FS01:Upload 87.33%
2019-10-11:17:55:14:WU01:FS01:Upload 87.60%
2019-10-11:17:55:16:WU03:FS01:0x21:Completed 930000 out of 1000000 steps (93%)
2019-10-11:17:55:25:WU01:FS01:Upload 87.77%
2019-10-11:17:55:32:WU01:FS01:Upload 87.95%
2019-10-11:17:55:38:WU01:FS01:Upload 88.30%
2019-10-11:17:55:46:WU01:FS01:Upload 88.48%
2019-10-11:17:55:51:WU03:FS01:0x21:Completed 940000 out of 1000000 steps (94%)
2019-10-11:17:55:52:WU01:FS01:Upload 88.65%
2019-10-11:17:55:59:WU01:FS01:Upload 88.91%
2019-10-11:17:56:01:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:17:56:01:WU01:FS01:Trying to send results to collection server
2019-10-11:17:56:01:WU01:FS01:Uploading 71.21MiB to 155.247.166.219
2019-10-11:17:56:01:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-11:17:56:01:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-11:17:56:02:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14277 run:0 clone:14 gen:9 core:0x21 unit:0x0000000980fccb0a5d98ab61dba7b8d2
2019-10-11:17:56:02:WU01:FS01:Uploading 71.21MiB to 128.252.203.10
2019-10-11:17:56:02:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-11:17:56:17:WU01:FS01:Upload 0.18%
2019-10-11:17:56:23:WU01:FS01:Upload 0.44%
2019-10-11:17:56:30:WU01:FS01:Upload 0.53%
2019-10-11:17:56:31:WU03:FS01:0x21:Completed 950000 out of 1000000 steps (95%)
2019-10-11:17:56:43:WU01:FS01:Upload 0.79%
2019-10-11:17:57:07:WU03:FS01:0x21:Completed 960000 out of 1000000 steps (96%)
2019-10-11:17:57:47:WU03:FS01:0x21:Completed 970000 out of 1000000 steps (97%)
2019-10-11:17:58:23:WU03:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
2019-10-11:17:58:29:WU01:FS01:Upload 0.88%
2019-10-11:17:58:29:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:17:58:29:WU01:FS01:Trying to send results to collection server
2019-10-11:17:58:29:WU01:FS01:Uploading 71.21MiB to 155.247.166.219
2019-10-11:17:58:29:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-11:17:58:30:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-11:17:58:30:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14277 run:0 clone:14 gen:9 core:0x21 unit:0x0000000980fccb0a5d98ab61dba7b8d2
2019-10-11:17:58:30:WU01:FS01:Uploading 71.21MiB to 128.252.203.10
2019-10-11:17:58:30:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-11:17:59:02:WU03:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
2019-10-11:17:59:22:WU01:FS01:Upload 0.09%
2019-10-11:17:59:29:WU01:FS01:Upload 0.18%
2019-10-11:17:59:35:WU01:FS01:Upload 0.35%
2019-10-11:17:59:38:WU03:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
2019-10-11:17:59:39:WU04:FS01:Connecting to 65.254.110.245:8080
2019-10-11:17:59:39:WU04:FS01:Assigned to work server 128.252.203.10
2019-10-11:17:59:39:WU04:FS01:Requesting new work unit for slot 01: RUNNING gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 128.252.203.10
2019-10-11:17:59:39:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-11:17:59:40:WU04:FS01:Downloading 78.65MiB
2019-10-11:17:59:43:WU03:FS01:0x21:Saving result file logfile_01.txt
2019-10-11:17:59:43:WU03:FS01:0x21:Saving result file checkpointState.xml
2019-10-11:17:59:43:WU03:FS01:0x21:Saving result file checkpt.crc
2019-10-11:17:59:43:WU03:FS01:0x21:Saving result file log.txt
2019-10-11:17:59:43:WU03:FS01:0x21:Saving result file positions.xtc
2019-10-11:17:59:43:WU03:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
2019-10-11:17:59:44:WU03:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
2019-10-11:17:59:44:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:88 gen:2 core:0x21 unit:0x0000000380fccb0a5d98ab550aad0d99
2019-10-11:17:59:44:WU03:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-11:17:59:44:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-11:17:59:44:WU01:FS01:Upload 0.53%
2019-10-11:17:59:46:WU04:FS01:Download 21.77%
2019-10-11:17:59:52:WU04:FS01:Download 33.22%
2019-10-11:17:59:53:WU01:FS01:Upload 0.79%
2019-10-11:17:59:54:WU03:FS01:Upload 0.19%
2019-10-11:17:59:58:WU04:FS01:Download 46.80%
2019-10-11:17:59:59:WU01:FS01:Upload 0.97%
2019-10-11:18:00:01:WU03:FS01:Upload 0.39%
2019-10-11:18:00:04:WU04:FS01:Download 59.92%
2019-10-11:18:00:05:WU01:FS01:Upload 1.23%
2019-10-11:18:00:10:WU04:FS01:Download 74.06%
2019-10-11:18:00:10:WU03:FS01:Upload 0.58%
2019-10-11:18:00:11:WU01:FS01:Upload 1.40%
2019-10-11:18:00:16:WU04:FS01:Download 85.74%
2019-10-11:18:00:16:WU03:FS01:Upload 0.77%
2019-10-11:18:00:22:WU04:FS01:Download 90.51%
2019-10-11:18:00:23:WU03:FS01:Upload 0.87%
2019-10-11:18:00:24:WU01:FS01:Upload 1.58%
2019-10-11:18:00:28:WU04:FS01:Download 93.61%
2019-10-11:18:00:34:WU04:FS01:Download 95.20%
2019-10-11:18:00:40:WU04:FS01:Download 96.71%
2019-10-11:18:00:41:WU01:FS01:Upload 1.67%
2019-10-11:18:00:45:WU04:FS01:Download complete
2019-10-11:18:00:45:WU04:FS01:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:14275 run:0 clone:72 gen:10 core:0x21 unit:0x0000000a80fccb0a5d98ab5926b03bf4
2019-10-11:18:00:45:WU04:FS01:Starting
2019-10-11:18:00:45:WU04:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Jeff\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 04 -suffix 01 -version 705 -lifeline 59208 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
2019-10-11:18:00:45:WU04:FS01:Started FahCore on PID 89948
2019-10-11:18:00:45:WU04:FS01:Core PID:91380
2019-10-11:18:00:45:WU04:FS01:FahCore 0x21 started
2019-10-11:18:00:45:WU03:FS01:Upload 1.06%
2019-10-11:18:00:45:WU04:FS01:0x21:*********************** Log Started 2019-10-11T18:00:45Z ***********************
2019-10-11:18:00:45:WU04:FS01:0x21:Project: 14275 (Run 0, Clone 72, Gen 10)
2019-10-11:18:00:45:WU04:FS01:0x21:Unit: 0x0000000a80fccb0a5d98ab5926b03bf4
2019-10-11:18:00:45:WU04:FS01:0x21:CPU: 0x00000000000000000000000000000000
2019-10-11:18:00:45:WU04:FS01:0x21:Machine: 1
2019-10-11:18:00:45:WU04:FS01:0x21:Reading tar file core.xml
2019-10-11:18:00:45:WU04:FS01:0x21:Reading tar file integrator.xml
2019-10-11:18:00:45:WU04:FS01:0x21:Reading tar file state.xml
2019-10-11:18:00:45:WU04:FS01:0x21:Reading tar file system.xml
2019-10-11:18:00:46:WU04:FS01:0x21:Digital signatures verified
2019-10-11:18:00:46:WU04:FS01:0x21:Folding@home GPU Core21 Folding@home Core
2019-10-11:18:00:46:WU04:FS01:0x21:Version 0.0.20
2019-10-11:18:00:47:WU01:FS01:Upload 2.02%
2019-10-11:18:01:01:WU04:FS01:0x21:Completed 0 out of 1000000 steps (0%)
2019-10-11:18:01:01:WU04:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2019-10-11:18:01:04:WU03:FS01:Upload 1.16%
2019-10-11:18:01:13:WU03:FS01:Upload 1.26%
2019-10-11:18:01:20:WU03:FS01:Upload 1.64%
2019-10-11:18:01:28:WU03:FS01:Upload 1.84%
2019-10-11:18:01:39:WU01:FS01:Upload 2.19%
2019-10-11:18:01:43:WU04:FS01:0x21:Completed 10000 out of 1000000 steps (1%)
2019-10-11:18:01:45:WU03:FS01:Upload 2.03%
2019-10-11:18:01:52:WU01:FS01:Upload 2.28%
2019-10-11:18:01:56:WU03:FS01:Upload 2.13%
2019-10-11:18:02:02:WU01:FS01:Upload 2.37%
2019-10-11:18:02:05:WU03:FS01:Upload 2.23%
2019-10-11:18:02:09:WU01:FS01:Upload 2.46%
2019-10-11:18:02:16:WU03:FS01:Upload 2.32%
2019-10-11:18:02:25:WU04:FS01:0x21:Completed 20000 out of 1000000 steps (2%)
2019-10-11:18:02:25:WU03:FS01:Upload 2.52%
2019-10-11:18:02:36:WU01:FS01:Upload 2.55%
2019-10-11:18:02:42:WU01:FS01:Upload 2.72%
2019-10-11:18:02:50:WU01:FS01:Upload 2.90%
2019-10-11:18:03:11:WU04:FS01:0x21:Completed 30000 out of 1000000 steps (3%)
2019-10-11:18:03:52:WU04:FS01:0x21:Completed 40000 out of 1000000 steps (4%)
2019-10-11:18:04:00:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14274 run:0 clone:0 gen:39 core:0x21 unit:0x0000002880fccb0a5d98ab50a43fcdbe
2019-10-11:18:04:00:WU00:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-11:18:04:00:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:04:04:WU01:FS01:Upload 3.07%
2019-10-11:18:04:07:WU00:FS01:Upload 0.10%
2019-10-11:18:04:15:WU01:FS01:Upload 3.34%
2019-10-11:18:04:15:WU00:FS01:Upload 0.29%
2019-10-11:18:04:21:WU00:FS01:Upload 0.48%
2019-10-11:18:04:27:WU00:FS01:Upload 0.58%
2019-10-11:18:04:28:WU01:FS01:Upload 3.51%
2019-10-11:18:04:34:WU01:FS01:Upload 3.69%
2019-10-11:18:04:35:WU00:FS01:Upload 0.77%
2019-10-11:18:04:38:WU04:FS01:0x21:Completed 50000 out of 1000000 steps (5%)
2019-10-11:18:04:41:WU00:FS01:Upload 0.97%
2019-10-11:18:04:44:WU01:FS01:Upload 3.95%
2019-10-11:18:04:51:WU00:FS01:Upload 1.06%
2019-10-11:18:04:51:WU01:FS01:Upload 4.13%
2019-10-11:18:04:57:WU01:FS01:Upload 4.30%
2019-10-11:18:04:57:WU00:FS01:Upload 1.26%
2019-10-11:18:04:59:WU03:FS01:Upload 2.71%
2019-10-11:18:04:59:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:04:59:WU03:FS01:Trying to send results to collection server
2019-10-11:18:04:59:WU03:FS01:Uploading 64.60MiB to 155.247.166.220
2019-10-11:18:04:59:WU03:FS01:Connecting to 155.247.166.220:8080
2019-10-11:18:04:59:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-11:18:05:00:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:88 gen:2 core:0x21 unit:0x0000000380fccb0a5d98ab550aad0d99
2019-10-11:18:05:00:WU03:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-11:18:05:00:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:05:06:WU00:FS01:Upload 1.45%
2019-10-11:18:05:07:WU01:FS01:Upload 4.48%
2019-10-11:18:05:11:WU03:FS01:Upload 0.19%
2019-10-11:18:05:14:WU01:FS01:Upload 4.74%
2019-10-11:18:05:19:WU00:FS01:Upload 1.64%
2019-10-11:18:05:20:WU04:FS01:0x21:Completed 60000 out of 1000000 steps (6%)
2019-10-11:18:05:22:WU03:FS01:Upload 0.39%
2019-10-11:18:05:25:WU00:FS01:Upload 1.84%
2019-10-11:18:05:25:WU01:FS01:Upload 4.92%
2019-10-11:18:05:31:WU00:FS01:Upload 1.93%
2019-10-11:18:05:31:WU03:FS01:Upload 0.58%
2019-10-11:18:05:37:WU00:FS01:Upload 2.03%
2019-10-11:18:05:45:WU00:FS01:Upload 2.23%
2019-10-11:18:05:51:WU01:FS01:Upload 5.09%
2019-10-11:18:05:54:WU00:FS01:Upload 2.52%
2019-10-11:18:05:57:WU01:FS01:Upload 5.18%
2019-10-11:18:06:01:WU00:FS01:Upload 2.81%
2019-10-11:18:06:04:WU01:FS01:Upload 5.44%
2019-10-11:18:06:06:WU04:FS01:0x21:Completed 70000 out of 1000000 steps (7%)
2019-10-11:18:06:10:WU00:FS01:Upload 3.10%
2019-10-11:18:06:13:WU01:FS01:Upload 5.79%
2019-10-11:18:06:21:WU03:FS01:Upload 0.68%
2019-10-11:18:06:27:WU00:FS01:Upload 3.19%
2019-10-11:18:06:28:WU03:FS01:Upload 0.87%
2019-10-11:18:06:36:WU03:FS01:Upload 1.06%
2019-10-11:18:06:43:WU03:FS01:Upload 1.26%
2019-10-11:18:06:48:WU04:FS01:0x21:Completed 80000 out of 1000000 steps (8%)
2019-10-11:18:07:05:WU00:FS01:Upload 3.29%
2019-10-11:18:07:05:WU01:FS01:Upload 5.88%
2019-10-11:18:07:12:WU00:FS01:Upload 3.48%
2019-10-11:18:07:13:WU01:FS01:Upload 6.14%
2019-10-11:18:07:19:WU01:FS01:Upload 6.23%
2019-10-11:18:07:22:WU00:FS01:Upload 3.68%
2019-10-11:18:07:29:WU00:FS01:Upload 3.77%
2019-10-11:18:07:29:WU01:FS01:Upload 6.32%
2019-10-11:18:07:34:WU04:FS01:0x21:Completed 90000 out of 1000000 steps (9%)
2019-10-11:18:07:36:WU01:FS01:Upload 6.50%
2019-10-11:18:07:45:WU01:FS01:Upload 6.67%
2019-10-11:18:07:55:WU01:FS01:Upload 6.85%
2019-10-11:18:08:02:WU01:FS01:Upload 7.11%
2019-10-11:18:08:05:WU00:FS01:Upload 4.06%
2019-10-11:18:08:10:WU01:FS01:Upload 7.29%
2019-10-11:18:08:15:WU00:FS01:Upload 4.35%
2019-10-11:18:08:16:WU04:FS01:0x21:Completed 100000 out of 1000000 steps (10%)
2019-10-11:18:08:16:WU01:FS01:Upload 7.37%
2019-10-11:18:08:22:WU01:FS01:Upload 7.55%
2019-10-11:18:08:30:WU00:FS01:Upload 4.55%
2019-10-11:18:08:32:WU01:FS01:Upload 7.81%
2019-10-11:18:08:37:WU00:FS01:Upload 4.74%
2019-10-11:18:08:39:WU01:FS01:Upload 7.99%
2019-10-11:18:08:43:WU00:FS01:Upload 4.84%
2019-10-11:18:08:46:WU01:FS01:Upload 8.08%
2019-10-11:18:08:52:WU01:FS01:Upload 8.25%
2019-10-11:18:09:02:WU04:FS01:0x21:Completed 110000 out of 1000000 steps (11%)
2019-10-11:18:09:02:WU01:FS01:Upload 8.51%
2019-10-11:18:09:08:WU01:FS01:Upload 9.04%
2019-10-11:18:09:16:WU01:FS01:Upload 9.30%
2019-10-11:18:09:22:WU01:FS01:Upload 9.39%
2019-10-11:18:09:31:WU01:FS01:Upload 9.66%
2019-10-11:18:09:36:WU03:FS01:Upload 1.35%
2019-10-11:18:09:36:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:09:36:WU03:FS01:Trying to send results to collection server
2019-10-11:18:09:36:WU03:FS01:Uploading 64.60MiB to 155.247.166.220
2019-10-11:18:09:36:WU03:FS01:Connecting to 155.247.166.220:8080
2019-10-11:18:09:37:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-11:18:09:37:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:88 gen:2 core:0x21 unit:0x0000000380fccb0a5d98ab550aad0d99
2019-10-11:18:09:37:WU03:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-11:18:09:37:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:09:39:WU01:FS01:Upload 9.74%
2019-10-11:18:09:44:WU04:FS01:0x21:Completed 120000 out of 1000000 steps (12%)
2019-10-11:18:09:47:WU01:FS01:Upload 10.01%
2019-10-11:18:09:47:WU03:FS01:Upload 0.10%
2019-10-11:18:09:53:WU01:FS01:Upload 10.09%
2019-10-11:18:10:00:WU01:FS01:Upload 10.36%
2019-10-11:18:10:09:WU03:FS01:Upload 0.29%
2019-10-11:18:10:14:WU00:FS01:Upload 5.03%
2019-10-11:18:10:14:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:10:14:WU00:FS01:Trying to send results to collection server
2019-10-11:18:10:14:WU00:FS01:Uploading 64.60MiB to 155.247.166.219
2019-10-11:18:10:14:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-11:18:10:15:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-11:18:10:28:WU03:FS01:Upload 0.48%
2019-10-11:18:10:30:WU04:FS01:0x21:Completed 130000 out of 1000000 steps (13%)
2019-10-11:18:10:35:WU03:FS01:Upload 0.68%
2019-10-11:18:10:42:WU03:FS01:Upload 0.87%
2019-10-11:18:10:43:WU01:FS01:Upload 10.53%
2019-10-11:18:10:49:WU01:FS01:Upload 10.80%
2019-10-11:18:10:55:WU01:FS01:Upload 10.97%
2019-10-11:18:10:56:WU03:FS01:Upload 1.06%
2019-10-11:18:11:02:WU03:FS01:Upload 1.26%
2019-10-11:18:11:02:WU01:FS01:Upload 11.15%
2019-10-11:18:11:10:WU03:FS01:Upload 1.45%
2019-10-11:18:11:11:WU01:FS01:Upload 11.41%
2019-10-11:18:11:11:WU04:FS01:0x21:Completed 140000 out of 1000000 steps (14%)
2019-10-11:18:11:16:WU03:FS01:Upload 1.64%
2019-10-11:18:11:18:WU01:FS01:Upload 11.59%
2019-10-11:18:11:24:WU03:FS01:Upload 1.84%
2019-10-11:18:11:34:WU01:FS01:Upload 11.94%
2019-10-11:18:11:41:WU01:FS01:Upload 12.11%
2019-10-11:18:11:49:WU01:FS01:Upload 12.38%
2019-10-11:18:11:56:WU01:FS01:Upload 12.64%
2019-10-11:18:11:58:WU04:FS01:0x21:Completed 150000 out of 1000000 steps (15%)
2019-10-11:18:12:01:WU03:FS01:Upload 1.93%
2019-10-11:18:12:04:WU01:FS01:Upload 12.73%
2019-10-11:18:12:12:WU01:FS01:Upload 12.90%
2019-10-11:18:12:22:WU01:FS01:Upload 13.17%
2019-10-11:18:12:22:WU03:FS01:Upload 2.03%
2019-10-11:18:12:31:WU03:FS01:Upload 2.32%
2019-10-11:18:12:39:WU03:FS01:Upload 2.61%
2019-10-11:18:12:39:WU04:FS01:0x21:Completed 160000 out of 1000000 steps (16%)
2019-10-11:18:12:47:WU03:FS01:Upload 3.00%
2019-10-11:18:12:56:WU03:FS01:Upload 3.29%
2019-10-11:18:13:09:WU03:FS01:Upload 3.58%
2019-10-11:18:13:15:WU03:FS01:Upload 3.77%
2019-10-11:18:13:21:WU03:FS01:Upload 3.87%
2019-10-11:18:13:25:WU04:FS01:0x21:Completed 170000 out of 1000000 steps (17%)
2019-10-11:18:13:27:WU03:FS01:Upload 4.06%
2019-10-11:18:13:32:WU01:FS01:Upload 13.34%
2019-10-11:18:13:39:WU01:FS01:Upload 13.52%
2019-10-11:18:13:47:WU01:FS01:Upload 13.78%
2019-10-11:18:13:53:WU01:FS01:Upload 13.87%
2019-10-11:18:14:00:WU01:FS01:Upload 14.13%
2019-10-11:18:14:07:WU04:FS01:0x21:Completed 180000 out of 1000000 steps (18%)
2019-10-11:18:14:07:WU01:FS01:Upload 14.31%
2019-10-11:18:14:16:WU01:FS01:Upload 14.39%
2019-10-11:18:14:24:WU01:FS01:Upload 14.57%
2019-10-11:18:14:30:WU01:FS01:Upload 14.75%
2019-10-11:18:14:37:WU01:FS01:Upload 14.92%
2019-10-11:18:14:42:WU03:FS01:Upload 4.16%
2019-10-11:18:14:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:14:42:WU03:FS01:Trying to send results to collection server
2019-10-11:18:14:42:WU03:FS01:Uploading 64.60MiB to 155.247.166.220
2019-10-11:18:14:42:WU03:FS01:Connecting to 155.247.166.220:8080
2019-10-11:18:14:43:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-11:18:14:43:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:88 gen:2 core:0x21 unit:0x0000000380fccb0a5d98ab550aad0d99
2019-10-11:18:14:43:WU03:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-11:18:14:43:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:14:44:WU01:FS01:Upload 15.18%
2019-10-11:18:14:52:WU03:FS01:Upload 0.19%
2019-10-11:18:14:53:WU04:FS01:0x21:Completed 190000 out of 1000000 steps (19%)
2019-10-11:18:14:59:WU01:FS01:Upload 15.27%
2019-10-11:18:14:59:WU03:FS01:Upload 0.39%
2019-10-11:18:15:05:WU01:FS01:Upload 15.45%
2019-10-11:18:15:12:WU01:FS01:Upload 15.80%
2019-10-11:18:15:20:WU03:FS01:Upload 0.58%
2019-10-11:18:15:21:WU01:FS01:Upload 15.97%
2019-10-11:18:15:26:WU03:FS01:Upload 0.77%
2019-10-11:18:15:27:WU01:FS01:Upload 16.15%
2019-10-11:18:15:35:WU04:FS01:0x21:Completed 200000 out of 1000000 steps (20%)
2019-10-11:18:15:48:WU01:FS01:Upload 16.33%
2019-10-11:18:15:48:WU03:FS01:Upload 0.87%
2019-10-11:18:16:06:WU03:FS01:Upload 1.06%
2019-10-11:18:16:12:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14276 run:0 clone:30 gen:7 core:0x21 unit:0x0000000780fccb0a5d98ab5ca79c42f4
2019-10-11:18:16:12:WU02:FS01:Uploading 81.47MiB to 128.252.203.10
2019-10-11:18:16:12:WU02:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:16:13:WU03:FS01:Upload 1.35%
2019-10-11:18:16:18:WU02:FS01:Upload 0.15%
2019-10-11:18:16:20:WU01:FS01:Upload 16.41%
2019-10-11:18:16:21:WU04:FS01:0x21:Completed 210000 out of 1000000 steps (21%)
2019-10-11:18:16:22:WU03:FS01:Upload 1.64%
2019-10-11:18:16:25:WU02:FS01:Upload 0.23%
2019-10-11:18:16:26:WU01:FS01:Upload 16.50%
2019-10-11:18:16:28:WU03:FS01:Upload 1.84%
2019-10-11:18:16:33:WU02:FS01:Upload 0.38%
2019-10-11:18:16:33:WU01:FS01:Upload 16.68%
2019-10-11:18:16:35:WU03:FS01:Upload 2.13%
2019-10-11:18:16:41:WU01:FS01:Upload 16.85%
2019-10-11:18:16:41:WU03:FS01:Upload 2.23%
2019-10-11:18:16:47:WU03:FS01:Upload 2.52%
2019-10-11:18:16:48:WU02:FS01:Upload 0.54%
2019-10-11:18:16:53:WU01:FS01:Upload 16.94%
2019-10-11:18:16:53:WU03:FS01:Upload 2.61%
2019-10-11:18:16:56:WU02:FS01:Upload 0.69%
2019-10-11:18:17:01:WU01:FS01:Upload 17.12%
2019-10-11:18:17:01:WU03:FS01:Upload 2.81%
2019-10-11:18:17:03:WU04:FS01:0x21:Completed 220000 out of 1000000 steps (22%)
2019-10-11:18:17:07:WU03:FS01:Upload 3.00%
2019-10-11:18:17:16:WU03:FS01:Upload 3.29%
2019-10-11:18:17:26:WU01:FS01:Upload 17.29%
2019-10-11:18:17:34:WU01:FS01:Upload 17.55%
2019-10-11:18:17:49:WU01:FS01:Upload 17.82%
2019-10-11:18:17:49:WU04:FS01:0x21:Completed 230000 out of 1000000 steps (23%)
2019-10-11:18:17:56:WU01:FS01:Upload 18.17%
2019-10-11:18:18:06:WU01:FS01:Upload 18.26%
2019-10-11:18:18:12:WU03:FS01:Upload 3.58%
2019-10-11:18:18:16:WU01:FS01:Upload 18.52%
2019-10-11:18:18:20:WU03:FS01:Upload 3.68%
2019-10-11:18:18:25:WU01:FS01:Upload 18.70%
2019-10-11:18:18:29:WU02:FS01:Upload 0.77%
2019-10-11:18:18:29:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:18:29:WU02:FS01:Trying to send results to collection server
2019-10-11:18:18:29:WU02:FS01:Uploading 81.47MiB to 155.247.166.220
2019-10-11:18:18:29:WU02:FS01:Connecting to 155.247.166.220:8080
2019-10-11:18:18:29:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-11:18:18:31:WU04:FS01:0x21:Completed 240000 out of 1000000 steps (24%)
2019-10-11:18:18:35:WU01:FS01:Upload 19.13%
2019-10-11:18:18:40:WU03:FS01:Upload 3.77%
2019-10-11:18:18:41:WU01:FS01:Upload 19.31%
2019-10-11:18:18:48:WU01:FS01:Upload 19.49%
2019-10-11:18:18:55:WU03:FS01:Upload 4.06%
2019-10-11:18:18:56:WU01:FS01:Upload 19.66%
2019-10-11:18:19:02:WU01:FS01:Upload 19.84%
2019-10-11:18:19:02:WU03:FS01:Upload 4.16%
2019-10-11:18:19:08:WU01:FS01:Upload 19.92%
2019-10-11:18:19:13:WU03:FS01:Upload 4.45%
2019-10-11:18:19:18:WU04:FS01:0x21:Completed 250000 out of 1000000 steps (25%)
2019-10-11:18:19:20:WU01:FS01:Upload 20.10%
2019-10-11:18:19:28:WU01:FS01:Upload 20.28%
2019-10-11:18:19:34:WU01:FS01:Upload 20.45%
2019-10-11:18:19:40:WU01:FS01:Upload 20.71%
2019-10-11:18:19:47:WU01:FS01:Upload 20.98%
2019-10-11:18:19:53:WU01:FS01:Upload 21.15%
2019-10-11:18:19:59:WU01:FS01:Upload 21.33%
2019-10-11:18:20:00:WU04:FS01:0x21:Completed 260000 out of 1000000 steps (26%)
2019-10-11:18:20:02:WU03:FS01:Upload 4.55%
2019-10-11:18:20:06:WU01:FS01:Upload 21.59%
2019-10-11:18:20:08:WU03:FS01:Upload 4.64%
2019-10-11:18:20:14:WU03:FS01:Upload 4.84%
2019-10-11:18:20:18:WU01:FS01:Upload 21.77%
2019-10-11:18:20:26:WU01:FS01:Upload 21.94%
2019-10-11:18:20:44:WU01:FS01:Upload 22.12%
2019-10-11:18:20:46:WU03:FS01:Upload 5.03%
2019-10-11:18:20:46:WU04:FS01:0x21:Completed 270000 out of 1000000 steps (27%)
2019-10-11:18:20:52:WU01:FS01:Upload 22.47%
2019-10-11:18:20:53:WU03:FS01:Upload 5.13%
2019-10-11:18:21:00:WU03:FS01:Upload 5.42%
2019-10-11:18:21:02:WU01:FS01:Upload 22.73%
2019-10-11:18:21:06:WU03:FS01:Upload 5.61%
2019-10-11:18:21:08:WU01:FS01:Upload 22.91%
2019-10-11:18:21:14:WU03:FS01:Upload 5.90%
2019-10-11:18:21:15:WU01:FS01:Upload 23.00%
2019-10-11:18:21:20:WU03:FS01:Upload 6.10%
2019-10-11:18:21:26:WU03:FS01:Upload 6.29%
2019-10-11:18:21:28:WU04:FS01:0x21:Completed 280000 out of 1000000 steps (28%)
2019-10-11:18:21:31:WU01:FS01:Upload 23.17%
2019-10-11:18:21:32:WU03:FS01:Upload 6.48%
2019-10-11:18:21:37:WU01:FS01:Upload 23.44%
2019-10-11:18:21:44:WU01:FS01:Upload 23.70%
2019-10-11:18:21:55:WU01:FS01:Upload 23.87%
2019-10-11:18:22:13:WU03:FS01:Upload 6.58%
2019-10-11:18:22:14:WU04:FS01:0x21:Completed 290000 out of 1000000 steps (29%)
2019-10-11:18:22:31:WU01:FS01:Upload 23.96%
2019-10-11:18:22:49:WU03:FS01:Upload 6.68%
2019-10-11:18:22:56:WU04:FS01:0x21:Completed 300000 out of 1000000 steps (30%)
2019-10-11:18:23:05:WU03:FS01:Upload 6.87%
2019-10-11:18:23:12:WU03:FS01:Upload 6.97%
2019-10-11:18:23:22:WU03:FS01:Upload 7.06%
2019-10-11:18:23:22:WU01:FS01:Upload 24.05%
2019-10-11:18:23:28:WU03:FS01:Upload 7.35%
2019-10-11:18:23:29:WU01:FS01:Upload 24.14%
2019-10-11:18:23:37:WU03:FS01:Upload 7.45%
2019-10-11:18:23:43:WU04:FS01:0x21:Completed 310000 out of 1000000 steps (31%)
2019-10-11:18:23:52:WU03:FS01:Upload 7.64%
2019-10-11:18:23:58:WU03:FS01:Upload 7.84%
2019-10-11:18:24:20:WU01:FS01:Upload 24.23%
2019-10-11:18:24:25:WU04:FS01:0x21:Completed 320000 out of 1000000 steps (32%)
2019-10-11:18:24:26:WU01:FS01:Upload 24.31%
2019-10-11:18:24:40:WU01:FS01:Upload 24.49%
2019-10-11:18:24:48:WU01:FS01:Upload 24.66%
2019-10-11:18:25:03:WU01:FS01:Upload 24.93%
2019-10-11:18:25:11:WU04:FS01:0x21:Completed 330000 out of 1000000 steps (33%)
2019-10-11:18:25:18:WU03:FS01:Upload 8.32%
2019-10-11:18:25:18:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:25:18:WU03:FS01:Trying to send results to collection server
2019-10-11:18:25:18:WU03:FS01:Uploading 64.60MiB to 155.247.166.220
2019-10-11:18:25:18:WU03:FS01:Connecting to 155.247.166.220:8080
2019-10-11:18:25:19:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-11:18:25:19:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:88 gen:2 core:0x21 unit:0x0000000380fccb0a5d98ab550aad0d99
2019-10-11:18:25:19:WU03:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-11:18:25:19:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:25:26:WU01:FS01:Upload 25.10%
2019-10-11:18:25:39:WU01:FS01:Upload 25.37%
2019-10-11:18:25:45:WU03:FS01:Upload 0.19%
2019-10-11:18:25:47:WU01:FS01:Upload 25.54%
2019-10-11:18:25:54:WU04:FS01:0x21:Completed 340000 out of 1000000 steps (34%)
2019-10-11:18:25:58:WU01:FS01:Upload 25.81%
2019-10-11:18:26:04:WU01:FS01:Upload 25.98%
2019-10-11:18:26:14:WU01:FS01:Upload 26.07%
2019-10-11:18:26:23:WU01:FS01:Upload 26.24%
2019-10-11:18:26:34:WU01:FS01:Upload 26.33%
2019-10-11:18:26:40:WU04:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
2019-10-11:18:27:20:WU03:FS01:Upload 0.39%
2019-10-11:18:27:22:WU04:FS01:0x21:Completed 360000 out of 1000000 steps (36%)
2019-10-11:18:27:38:WU03:FS01:Upload 0.58%
2019-10-11:18:28:09:WU04:FS01:0x21:Completed 370000 out of 1000000 steps (37%)
2019-10-11:18:28:15:WU03:FS01:Upload 0.68%
2019-10-11:18:28:22:WU03:FS01:Upload 0.87%
2019-10-11:18:28:29:WU03:FS01:Upload 1.06%
2019-10-11:18:28:51:WU04:FS01:0x21:Completed 380000 out of 1000000 steps (38%)
2019-10-11:18:28:54:WU03:FS01:Upload 1.16%
2019-10-11:18:29:04:WU03:FS01:Upload 1.35%
2019-10-11:18:29:14:WU03:FS01:Upload 1.55%
2019-10-11:18:29:32:WU03:FS01:Upload 1.74%
2019-10-11:18:29:38:WU04:FS01:0x21:Completed 390000 out of 1000000 steps (39%)
2019-10-11:18:30:00:WU01:FS01:Upload 26.42%
2019-10-11:18:30:00:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:30:00:WU01:FS01:Trying to send results to collection server
2019-10-11:18:30:00:WU01:FS01:Uploading 71.21MiB to 155.247.166.219
2019-10-11:18:30:00:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-11:18:30:00:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-11:18:30:01:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14277 run:0 clone:14 gen:9 core:0x21 unit:0x0000000980fccb0a5d98ab61dba7b8d2
2019-10-11:18:30:01:WU01:FS01:Uploading 71.21MiB to 128.252.203.10
2019-10-11:18:30:01:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:30:20:WU04:FS01:0x21:Completed 400000 out of 1000000 steps (40%)
2019-10-11:18:30:32:WU03:FS01:Upload 1.84%
2019-10-11:18:30:32:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:30:32:WU03:FS01:Trying to send results to collection server
2019-10-11:18:30:32:WU03:FS01:Uploading 64.60MiB to 155.247.166.220
2019-10-11:18:30:32:WU03:FS01:Connecting to 155.247.166.220:8080
2019-10-11:18:30:32:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-11:18:30:33:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14274 run:0 clone:88 gen:2 core:0x21 unit:0x0000000380fccb0a5d98ab550aad0d99
2019-10-11:18:30:33:WU03:FS01:Uploading 64.60MiB to 128.252.203.10
2019-10-11:18:30:33:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:30:39:WU03:FS01:Upload 0.19%
2019-10-11:18:30:45:WU03:FS01:Upload 0.29%
2019-10-11:18:30:59:WU03:FS01:Upload 0.48%
2019-10-11:18:31:06:WU04:FS01:0x21:Completed 410000 out of 1000000 steps (41%)
2019-10-11:18:31:07:WU03:FS01:Upload 0.68%
2019-10-11:18:31:16:WU03:FS01:Upload 0.87%
2019-10-11:18:31:28:WU03:FS01:Upload 1.16%
2019-10-11:18:31:48:WU04:FS01:0x21:Completed 420000 out of 1000000 steps (42%)
2019-10-11:18:31:51:WU03:FS01:Upload 1.26%
2019-10-11:18:31:54:WU01:FS01:Upload 0.18%
2019-10-11:18:31:54:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-11:18:31:54:WU01:FS01:Trying to send results to collection server
2019-10-11:18:31:54:WU01:FS01:Uploading 71.21MiB to 155.247.166.219
2019-10-11:18:31:54:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-11:18:32:02:WU01:FS01:Upload 0.09%
2019-10-11:18:32:02:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-11:18:32:34:WU03:FS01:Upload 1.35%
2019-10-11:18:32:35:WU04:FS01:0x21:Completed 430000 out of 1000000 steps (43%)
2019-10-11:18:32:38:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14277 run:0 clone:14 gen:9 core:0x21 unit:0x0000000980fccb0a5d98ab61dba7b8d2
2019-10-11:18:32:38:WU01:FS01:Uploading 71.21MiB to 128.252.203.10
2019-10-11:18:32:38:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-11:18:32:42:WU03:FS01:Upload 1.45%
2019-10-11:18:32:49:WU03:FS01:Upload 1.64%
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Fri Oct 11, 2019 9:44 pm
by rewron
OP here. Just an update on the original WU's failing to upload, that started this thread.
All 3 WU's have now expired. All continuously attempted to upload until expiration. They were not slow to upload, but were "failed transfers."
The 3rd WU expired only today, and was promptly replaced by WU 14189 (4, 162 6) that also will not upload. As Bruce pointed out earlier with the original WU's, this WU also has an unusually large content/load of 112.08MiB (at least large for me.)
Other WU's on this machine using this server upload normally.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sat Oct 12, 2019 2:39 pm
by dfgirl12
Time to disconnect more PCs.
There's not much point in doing this, if you can't return the results (All from the last 24 hours):
<<Image: Had 16 WUs waiting to send for a single folding slot>>
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sat Oct 12, 2019 4:07 pm
by toTOW
Projects in 142xx series are not served by 155.247.166.* servers, it's a different issue ...
I had two machines (CPU slots) with stuck downloads from 155.247.166.219 this week, but it seems to have cleared itself after a few days, my clients are now directed to other servers.
Re: Multiple WU's Fail to Upload 155.247.166.*
Posted: Sat Oct 12, 2019 4:14 pm
by bruce
Can you open 128.252.203.10 (which is NOT the subject of this topic) and/or 155.247.166.219 in your browser?
Show us FAH's log showing the recent upload attempts. Then restart the client and show us how the new log tries to start uploading and what messages follow that. Post that information in a new topic and I'll move your post to maintain the new topic's continuity.
