155.247.166.* has been failing to complete workload downloads for high-end GPUs since 1900 EST last night and the problem continues to this minute. There is a one in eight chance you will complete a download based on N=46 reboots. Yes, I stayed up most of the night trying to keep my servers running. And failing miserably.
If this were a one-time problem, I'd shrug it off as I have been in the IT industry for fifty years. But these specific Temple servers have been having the exact problems reported here since September. The IT root cause appears to be that projects have been allowed to become too large for the FAH upload/download infrastructure. That was an avoidable problem. Now, let's get this fixed once and for all for fast GPUs. Otherwise, start assigning fast GPUs to other work units.
Or F@H could implement "pay to play" where you get charged a fee to guarantee work-unit delivery. Hey, it's a charity, right?
Fix the damn problem pronto!
Multiple WU's Fail downld/upld to 155.247.166.*
Moderators: Site Moderators, FAHC Science Team
-
Catalina588
- Posts: 41
- Joined: Thu Oct 09, 2008 8:59 pm
Re: Multiple WU's Fail downld/upld to 155.247.166.*
I am down too, just when I thought it was fixed.
I would think that they would just block the offending uploads, which are CPU as I understand it.
They are losing all the GPU uploads too.
I would think that they would just block the offending uploads, which are CPU as I understand it.
They are losing all the GPU uploads too.
Re: Multiple WU's Fail downld/upld to 155.247.166.*
The error Exception: Failed to find any IP addresses for assignment servers reported above is not related to the discussion of recent traffic overloads on specific FAH servers. That's a problem with your DNS.
Yes, the upload/download speeds have deteriorated, but I'm not having any troubles with WU time-outs. The improvements from the bug fix in FAHCore_a7 has certainly helped but apparently not enough.
As I said earlier, new servers are being planned but it takes time (and red-tape) to get money allocated and to provision the new hardware. A permanent solution IS BEING pursued.
Short term solutions are also being worked. e.g.- Attempts to adjust the traffic to/from that server but such adjustments don't take effect immediately (delayed by the delays from the time of WU assignment to the time the results are returned) so such adjustments are very difficult to match with the average bandwidth need vs. the length of communications queue length.
Yes, the upload/download speeds have deteriorated, but I'm not having any troubles with WU time-outs. The improvements from the bug fix in FAHCore_a7 has certainly helped but apparently not enough.
As I said earlier, new servers are being planned but it takes time (and red-tape) to get money allocated and to provision the new hardware. A permanent solution IS BEING pursued.
Short term solutions are also being worked. e.g.- Attempts to adjust the traffic to/from that server but such adjustments don't take effect immediately (delayed by the delays from the time of WU assignment to the time the results are returned) so such adjustments are very difficult to match with the average bandwidth need vs. the length of communications queue length.
Posting FAH's log:
How to provide enough info to get helpful support.
How to provide enough info to get helpful support.
-
rwh202
- Posts: 410
- Joined: Mon Nov 15, 2010 8:51 pm
- Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces - Location: South Coast, UK
Re: Multiple WU's Fail downld/upld to 155.247.166.*
That's just the start of a log after a restart to try and get an assignment - it's the normal sequence of events on Linux where the client starts up before networking gets fully initialised. I believe it was posted to just illustrate rebooting working in getting work after a stall.bruce wrote:The error Exception: Failed to find any IP addresses for assignment servers reported above is not related to the discussion of recent traffic overloads on specific FAH servers. That's a problem with your DNS.
Re: Multiple WU's Fail downld/upld to 155.247.166.*
After restarting your OS, the connections that you have opened on server XXX will have been closed and you'll be starting with a new connection. I expect that to work whether or not the server, itself, has been restarted.
Posting FAH's log:
How to provide enough info to get helpful support.
How to provide enough info to get helpful support.
-
vvoelz
- Pande Group Member
- Posts: 552
- Joined: Sun Dec 02, 2007 8:07 pm
- Location: Temple University, Philadelphia PA
Re: Multiple WU's Fail downld/upld to 155.247.166.*
Sorry to hear about the download/upload delays. We thought we had fixed this issue, but it looks like it's back. For the time being I have: (1) changed the assignment weights of vav3 (155.247.166.219) and vav4 (155.247.166.220) back down to 30%, and (2) rebooted vav3 and vav4. I also will confer with Temple's IT Services to see if anything changed that would affect network speeds.
Besides the client fix, we previously turned down the assign weights down to 30%, which seemed to help. At about 3 PM (EST) on Friday Nov 15 I turned the assignment weights up back up to 100%, so I wonder if that may be at least partially to blame. To those that have posted recently, what is the *earliest* time you noticed the problem reappearing? Mon Nov 18?
Keep in mind that we have two new servers that are scheduled to ship from the vendor *next week*. I hope to get these up and running ASAP, so please bear with us!
Besides the client fix, we previously turned down the assign weights down to 30%, which seemed to help. At about 3 PM (EST) on Friday Nov 15 I turned the assignment weights up back up to 100%, so I wonder if that may be at least partially to blame. To those that have posted recently, what is the *earliest* time you noticed the problem reappearing? Mon Nov 18?
Keep in mind that we have two new servers that are scheduled to ship from the vendor *next week*. I hope to get these up and running ASAP, so please bear with us!
Re: Multiple WU's Fail downld/upld to 155.247.166.*
vvoelz, most of the projects seem to hang after a WU takes about 1 minute or more to download.
smaller wus (for CPU) don't have to download that long, and the connection with the server can be closed, freeing resources to other queued clients.
If you keep loading large WUs on that server, each client will act like a DDOS attack, and your server will stall.
smaller wus (for CPU) don't have to download that long, and the connection with the server can be closed, freeing resources to other queued clients.
If you keep loading large WUs on that server, each client will act like a DDOS attack, and your server will stall.
Re: Multiple WU's Fail downld/upld to 155.247.166.*
That is almost a trick question, but of the three machines that I know went down, the one where the problem first appeared (i.e., it uploaded and then stopped), the last upload completed on November 19:vvoelz wrote:To those that have posted recently, what is the *earliest* time you noticed the problem reappearing? Mon Nov 18?
Since that is UTC, that converts to 22:44:27, or 10:44 PM on Monday, November 18 EST.03:44:27:WU00:FS01:Cleaning up
The other two machines returned one result each after that time, but then got no more work.
Still another machine continued for a day, and then stopped getting work.
Maybe that will help.
Re: Multiple WU's Fail downld/upld to 155.247.166.*
@vvoelz- I think there's a very easy workaround for this problem (even with 100% assign weights), which is to restart the FAH WS software (or reboot the server if it's a dedicated FAH server) every ~2 days. Setting up a restart/reboot script to run on a schedule should be easy: like a cron job or something similar, so a human doesn't need to be involved.
From my observations: the problem typically happens during business hours 10am-5pm PST (more internet traffic) once the FAH WS software uptime goes over 3-5 days (more accumulated broken connections not being: reset, or reset quickly). The problem doesn't happen when the FAH WS uptime is less than 2-3 days. And, it is less likely to occur on the weekend, or at night (if it does, it usually started earlier during business hours). When the large GPU WUs (100MB) were running (not right now) with a GPU client folding time of ~1 hour, then this issue can happen in 2-3 days of FAH WS uptime on the week days.
The problem seems to be from the FAH WS (and FAH Client) software not having a good failed transfer ('no more data sent or received' timeout) and gracefully recovering from broken file transfers. Hopefully this helps
I reported this most recent download issues at about 7:30pm PST on Monday (1-2 hours after it started), and it probably was slowly increasing in severity before that on Monday (5 days of WS uptime).
From my observations: the problem typically happens during business hours 10am-5pm PST (more internet traffic) once the FAH WS software uptime goes over 3-5 days (more accumulated broken connections not being: reset, or reset quickly). The problem doesn't happen when the FAH WS uptime is less than 2-3 days. And, it is less likely to occur on the weekend, or at night (if it does, it usually started earlier during business hours). When the large GPU WUs (100MB) were running (not right now) with a GPU client folding time of ~1 hour, then this issue can happen in 2-3 days of FAH WS uptime on the week days.
The problem seems to be from the FAH WS (and FAH Client) software not having a good failed transfer ('no more data sent or received' timeout) and gracefully recovering from broken file transfers. Hopefully this helps
I reported this most recent download issues at about 7:30pm PST on Monday (1-2 hours after it started), and it probably was slowly increasing in severity before that on Monday (5 days of WS uptime).
-
parkut
- Posts: 363
- Joined: Tue Feb 12, 2008 7:33 am
- Hardware configuration: Running exclusively Linux headless blades. All are dedicated crunching machines.
- Location: SE Michigan, USA
Re: Multiple WU's Fail downld/upld to 155.247.166.*
The earliest download stall I saw occured @:
20:47:44:WU02:FS02:Download 48.94%
***** Date: 2019-11-18 ************
20:47:44:WU02:FS02:Download 48.94%
***** Date: 2019-11-18 ************
-
vvoelz
- Pande Group Member
- Posts: 552
- Joined: Sun Dec 02, 2007 8:07 pm
- Location: Temple University, Philadelphia PA
Re: Multiple WU's Fail downld/upld to 155.247.166.*
Thanks for the feedback everybody. As suggested, we are restarting the servers more regularly.
In addition, we have made some changes to the network connection settings on 155.247.166.219 (vav3) and 155.247.166.220 (vav4), namely: (1) decreasing the number of allowed simultaneous connections per port, (2) decreasing the maximum time to wait for a read or write from the client (default was 30 min! now set to 5 min), and (3) decreasing the maximum time a connection to the WS is allowed to live. We think this might improve things in the case that too many simultaneous connections are allowed, preventing any of them from completing.
As we bring our assignment weight back up to normal, please let us know if you continue to see problems. --VInce
In addition, we have made some changes to the network connection settings on 155.247.166.219 (vav3) and 155.247.166.220 (vav4), namely: (1) decreasing the number of allowed simultaneous connections per port, (2) decreasing the maximum time to wait for a read or write from the client (default was 30 min! now set to 5 min), and (3) decreasing the maximum time a connection to the WS is allowed to live. We think this might improve things in the case that too many simultaneous connections are allowed, preventing any of them from completing.
As we bring our assignment weight back up to normal, please let us know if you continue to see problems. --VInce
-
toTOW
- Site Moderator
- Posts: 6359
- Joined: Sun Dec 02, 2007 10:38 am
- Location: Bordeaux, France
- Contact:
Re: Multiple WU's Fail downld/upld to 155.247.166.*
Last time I had issue from your servers was on Monday, November 18th at 10pm (UTC) ... strangely, only one machine which is on a different network was affected out of four.
Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
Re: Multiple WU's Fail downld/upld to 155.247.166.*
VVoelz, is it possible to implement some sort of queuing approach, rather than giving everyone access at once to down/upload?
As long as you have x-amount of connections going at a same time, and an y-amount of queues, it should offer a much more streamlined experience, as to trying to serve everyone at the same time.
As long as you have x-amount of connections going at a same time, and an y-amount of queues, it should offer a much more streamlined experience, as to trying to serve everyone at the same time.
Re: Multiple WU's Fail downld/upld to 155.247.166.*
I have 2 results that have failed to upload (14 attempts each). The computer is a 3900X running linux mint 19.2 with an RTX 2070 and RTX 2070S and the Nvidia driver version is 435.21. I tried restarting the FAHClient and rebooting the computer but that didn't help.
Here's one of the logs:
Here's one of the logs:
Code: Select all
******************************* Date: 2019-11-29 *******************************
20:13:41:WU03:FS01:Connecting to 65.254.110.245:8080
20:13:41:WU03:FS01:Assigned to work server 128.252.203.10
20:13:41:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:TU106 [GeForce RTX 2070 Rev. A] M 7465 from 128.252.203.10
20:13:41:WU03:FS01:Connecting to 128.252.203.10:8080
20:13:41:WU03:FS01:Downloading 73.30MiB
20:13:47:WU03:FS01:Download 80.15%
20:13:48:WU03:FS01:Download complete
20:13:48:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
20:13:48:WU03:FS01:Starting
20:13:48:WU03:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_21.fah/FahCore_21 -dir 03 -suffix 01 -version 705 -lifeline 5410 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
20:13:48:WU03:FS01:Started FahCore on PID 9419
20:13:48:WU03:FS01:Core PID:9423
20:13:48:WU03:FS01:FahCore 0x21 started
20:13:48:WU03:FS01:0x21:*********************** Log Started 2019-11-29T20:13:48Z ***********************
20:13:48:WU03:FS01:0x21:Project: 14264 (Run 0, Clone 67, Gen 54)
20:13:48:WU03:FS01:0x21:Unit: 0x0000003d80fccb0a5daa0009ba2a3e33
20:13:48:WU03:FS01:0x21:CPU: 0x00000000000000000000000000000000
20:13:48:WU03:FS01:0x21:Machine: 1
20:13:48:WU03:FS01:0x21:Reading tar file core.xml
20:13:48:WU03:FS01:0x21:Reading tar file integrator.xml
20:13:48:WU03:FS01:0x21:Reading tar file state.xml
20:13:48:WU03:FS01:0x21:Reading tar file system.xml
20:13:48:WU03:FS01:0x21:Digital signatures verified
20:13:48:WU03:FS01:0x21:Folding@home GPU Core21 Folding@home Core
20:13:48:WU03:FS01:0x21:Version 0.0.20
20:14:06:WU03:FS01:0x21:Completed 0 out of 1000000 steps (0%)
20:14:06:WU03:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
20:15:08:WU03:FS01:0x21:Completed 10000 out of 1000000 steps (1%)
20:16:10:WU03:FS01:0x21:Completed 20000 out of 1000000 steps (2%)
20:17:18:WU03:FS01:0x21:Completed 30000 out of 1000000 steps (3%)
20:18:20:WU03:FS01:0x21:Completed 40000 out of 1000000 steps (4%)
20:19:27:WU03:FS01:0x21:Completed 50000 out of 1000000 steps (5%)
20:20:29:WU03:FS01:0x21:Completed 60000 out of 1000000 steps (6%)
20:21:36:WU03:FS01:0x21:Completed 70000 out of 1000000 steps (7%)
20:22:38:WU03:FS01:0x21:Completed 80000 out of 1000000 steps (8%)
20:23:45:WU03:FS01:0x21:Completed 90000 out of 1000000 steps (9%)
20:24:46:WU03:FS01:0x21:Completed 100000 out of 1000000 steps (10%)
20:25:54:WU03:FS01:0x21:Completed 110000 out of 1000000 steps (11%)
20:26:55:WU03:FS01:0x21:Completed 120000 out of 1000000 steps (12%)
20:28:02:WU03:FS01:0x21:Completed 130000 out of 1000000 steps (13%)
20:29:04:WU03:FS01:0x21:Completed 140000 out of 1000000 steps (14%)
20:30:11:WU03:FS01:0x21:Completed 150000 out of 1000000 steps (15%)
20:31:13:WU03:FS01:0x21:Completed 160000 out of 1000000 steps (16%)
20:32:20:WU03:FS01:0x21:Completed 170000 out of 1000000 steps (17%)
20:33:22:WU03:FS01:0x21:Completed 180000 out of 1000000 steps (18%)
20:34:29:WU03:FS01:0x21:Completed 190000 out of 1000000 steps (19%)
20:35:31:WU03:FS01:0x21:Completed 200000 out of 1000000 steps (20%)
20:36:37:WU03:FS01:0x21:Completed 210000 out of 1000000 steps (21%)
20:37:39:WU03:FS01:0x21:Completed 220000 out of 1000000 steps (22%)
20:38:45:WU03:FS01:0x21:Completed 230000 out of 1000000 steps (23%)
20:39:47:WU03:FS01:0x21:Completed 240000 out of 1000000 steps (24%)
20:40:54:WU03:FS01:0x21:Completed 250000 out of 1000000 steps (25%)
20:41:56:WU03:FS01:0x21:Completed 260000 out of 1000000 steps (26%)
20:43:02:WU03:FS01:0x21:Completed 270000 out of 1000000 steps (27%)
20:44:04:WU03:FS01:0x21:Completed 280000 out of 1000000 steps (28%)
20:45:11:WU03:FS01:0x21:Completed 290000 out of 1000000 steps (29%)
20:46:12:WU03:FS01:0x21:Completed 300000 out of 1000000 steps (30%)
20:47:20:WU03:FS01:0x21:Completed 310000 out of 1000000 steps (31%)
20:48:22:WU03:FS01:0x21:Completed 320000 out of 1000000 steps (32%)
20:49:29:WU03:FS01:0x21:Completed 330000 out of 1000000 steps (33%)
20:50:30:WU03:FS01:0x21:Completed 340000 out of 1000000 steps (34%)
20:51:37:WU03:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
20:52:39:WU03:FS01:0x21:Completed 360000 out of 1000000 steps (36%)
20:53:46:WU03:FS01:0x21:Completed 370000 out of 1000000 steps (37%)
20:54:47:WU03:FS01:0x21:Completed 380000 out of 1000000 steps (38%)
20:55:54:WU03:FS01:0x21:Completed 390000 out of 1000000 steps (39%)
20:56:56:WU03:FS01:0x21:Completed 400000 out of 1000000 steps (40%)
20:58:03:WU03:FS01:0x21:Completed 410000 out of 1000000 steps (41%)
20:59:05:WU03:FS01:0x21:Completed 420000 out of 1000000 steps (42%)
21:00:11:WU03:FS01:0x21:Completed 430000 out of 1000000 steps (43%)
21:01:13:WU03:FS01:0x21:Completed 440000 out of 1000000 steps (44%)
21:02:20:WU03:FS01:0x21:Completed 450000 out of 1000000 steps (45%)
21:03:22:WU03:FS01:0x21:Completed 460000 out of 1000000 steps (46%)
21:04:29:WU03:FS01:0x21:Completed 470000 out of 1000000 steps (47%)
21:05:31:WU03:FS01:0x21:Completed 480000 out of 1000000 steps (48%)
21:06:37:WU03:FS01:0x21:Completed 490000 out of 1000000 steps (49%)
21:07:39:WU03:FS01:0x21:Completed 500000 out of 1000000 steps (50%)
21:08:46:WU03:FS01:0x21:Completed 510000 out of 1000000 steps (51%)
21:09:47:WU03:FS01:0x21:Completed 520000 out of 1000000 steps (52%)
21:10:55:WU03:FS01:0x21:Completed 530000 out of 1000000 steps (53%)
21:11:56:WU03:FS01:0x21:Completed 540000 out of 1000000 steps (54%)
21:13:03:WU03:FS01:0x21:Completed 550000 out of 1000000 steps (55%)
21:14:05:WU03:FS01:0x21:Completed 560000 out of 1000000 steps (56%)
21:15:12:WU03:FS01:0x21:Completed 570000 out of 1000000 steps (57%)
21:16:14:WU03:FS01:0x21:Completed 580000 out of 1000000 steps (58%)
21:17:20:WU03:FS01:0x21:Completed 590000 out of 1000000 steps (59%)
21:18:22:WU03:FS01:0x21:Completed 600000 out of 1000000 steps (60%)
21:19:28:WU03:FS01:0x21:Completed 610000 out of 1000000 steps (61%)
21:20:30:WU03:FS01:0x21:Completed 620000 out of 1000000 steps (62%)
21:21:37:WU03:FS01:0x21:Completed 630000 out of 1000000 steps (63%)
21:22:39:WU03:FS01:0x21:Completed 640000 out of 1000000 steps (64%)
21:23:46:WU03:FS01:0x21:Completed 650000 out of 1000000 steps (65%)
21:24:47:WU03:FS01:0x21:Completed 660000 out of 1000000 steps (66%)
21:25:55:WU03:FS01:0x21:Completed 670000 out of 1000000 steps (67%)
21:26:56:WU03:FS01:0x21:Completed 680000 out of 1000000 steps (68%)
21:28:04:WU03:FS01:0x21:Completed 690000 out of 1000000 steps (69%)
21:29:06:WU03:FS01:0x21:Completed 700000 out of 1000000 steps (70%)
21:30:13:WU03:FS01:0x21:Completed 710000 out of 1000000 steps (71%)
21:31:14:WU03:FS01:0x21:Completed 720000 out of 1000000 steps (72%)
21:32:22:WU03:FS01:0x21:Completed 730000 out of 1000000 steps (73%)
21:33:23:WU03:FS01:0x21:Completed 740000 out of 1000000 steps (74%)
21:34:30:WU03:FS01:0x21:Completed 750000 out of 1000000 steps (75%)
21:35:32:WU03:FS01:0x21:Completed 760000 out of 1000000 steps (76%)
21:36:39:WU03:FS01:0x21:Completed 770000 out of 1000000 steps (77%)
21:37:41:WU03:FS01:0x21:Completed 780000 out of 1000000 steps (78%)
21:38:48:WU03:FS01:0x21:Completed 790000 out of 1000000 steps (79%)
21:39:50:WU03:FS01:0x21:Completed 800000 out of 1000000 steps (80%)
21:40:56:WU03:FS01:0x21:Completed 810000 out of 1000000 steps (81%)
21:41:57:WU03:FS01:0x21:Completed 820000 out of 1000000 steps (82%)
21:43:04:WU03:FS01:0x21:Completed 830000 out of 1000000 steps (83%)
21:44:05:WU03:FS01:0x21:Completed 840000 out of 1000000 steps (84%)
21:45:11:WU03:FS01:0x21:Completed 850000 out of 1000000 steps (85%)
21:46:12:WU03:FS01:0x21:Completed 860000 out of 1000000 steps (86%)
21:47:19:WU03:FS01:0x21:Completed 870000 out of 1000000 steps (87%)
21:48:21:WU03:FS01:0x21:Completed 880000 out of 1000000 steps (88%)
21:49:28:WU03:FS01:0x21:Completed 890000 out of 1000000 steps (89%)
21:50:30:WU03:FS01:0x21:Completed 900000 out of 1000000 steps (90%)
21:51:36:WU03:FS01:0x21:Completed 910000 out of 1000000 steps (91%)
21:52:38:WU03:FS01:0x21:Completed 920000 out of 1000000 steps (92%)
21:53:45:WU03:FS01:0x21:Completed 930000 out of 1000000 steps (93%)
21:54:47:WU03:FS01:0x21:Completed 940000 out of 1000000 steps (94%)
21:55:54:WU03:FS01:0x21:Completed 950000 out of 1000000 steps (95%)
21:56:55:WU03:FS01:0x21:Completed 960000 out of 1000000 steps (96%)
21:58:02:WU03:FS01:0x21:Completed 970000 out of 1000000 steps (97%)
21:59:04:WU03:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
22:00:11:WU03:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
22:01:12:WU03:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
22:01:18:WU03:FS01:0x21:Saving result file logfile_01.txt
22:01:18:WU03:FS01:0x21:Saving result file checkpointState.xml
22:01:18:WU03:FS01:0x21:Saving result file checkpt.crc
22:01:18:WU03:FS01:0x21:Saving result file log.txt
22:01:19:WU03:FS01:0x21:Saving result file positions.xtc
22:01:19:WU03:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
22:01:19:WU03:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:01:19:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
22:01:19:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
22:01:19:WU03:FS01:Connecting to 128.252.203.10:8080
22:17:55:WU03:FS01:Upload 91.60%
22:17:55:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
22:17:55:WU03:FS01:Trying to send results to collection server
22:17:55:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
22:17:55:WU03:FS01:Connecting to 155.247.166.219:8080
22:17:55:ERROR:WU03:FS01:Exception: Transfer failed
22:17:55:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
22:17:55:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
22:17:55:WU03:FS01:Connecting to 128.252.203.10:8080
22:18:01:WU03:FS01:Upload 61.99%
22:34:10:WU03:FS01:Upload 87.92%
22:34:10:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
22:34:10:WU03:FS01:Trying to send results to collection server
22:34:10:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
22:34:10:WU03:FS01:Connecting to 155.247.166.219:8080
22:34:10:ERROR:WU03:FS01:Exception: Transfer failed
22:34:10:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
22:34:10:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
22:34:10:WU03:FS01:Connecting to 128.252.203.10:8080
22:50:55:WU03:FS01:Upload 90.25%
22:50:55:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
22:50:55:WU03:FS01:Trying to send results to collection server
22:50:55:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
22:50:55:WU03:FS01:Connecting to 155.247.166.219:8080
22:50:55:ERROR:WU03:FS01:Exception: Transfer failed
22:50:55:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
22:50:55:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
22:50:55:WU03:FS01:Connecting to 128.252.203.10:8080
22:51:01:WU03:FS01:Upload 85.95%
23:07:41:WU03:FS01:Upload 90.07%
23:07:41:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
23:07:41:WU03:FS01:Trying to send results to collection server
23:07:41:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
23:07:41:WU03:FS01:Connecting to 155.247.166.219:8080
23:07:41:ERROR:WU03:FS01:Exception: Transfer failed
23:07:41:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
23:07:41:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
23:07:41:WU03:FS01:Connecting to 128.252.203.10:8080
23:24:10:WU03:FS01:Upload 91.60%
23:24:10:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
23:24:10:WU03:FS01:Trying to send results to collection server
23:24:10:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
23:24:10:WU03:FS01:Connecting to 155.247.166.219:8080
23:24:10:ERROR:WU03:FS01:Exception: Transfer failed
23:24:11:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
23:24:11:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
23:24:11:WU03:FS01:Connecting to 128.252.203.10:8080
23:24:17:WU03:FS01:Upload 29.52%
23:40:35:WU03:FS01:Upload 89.27%
23:40:35:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
23:40:35:WU03:FS01:Trying to send results to collection server
23:40:35:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
23:40:35:WU03:FS01:Connecting to 155.247.166.219:8080
23:40:35:ERROR:WU03:FS01:Exception: Transfer failed
23:40:36:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
23:40:36:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
23:40:36:WU03:FS01:Connecting to 128.252.203.10:8080
23:57:21:WU03:FS01:Upload 91.24%
23:57:21:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
23:57:21:WU03:FS01:Trying to send results to collection server
23:57:21:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
23:57:21:WU03:FS01:Connecting to 155.247.166.219:8080
23:57:21:ERROR:WU03:FS01:Exception: Transfer failed
23:57:21:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
23:57:21:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
23:57:21:WU03:FS01:Connecting to 128.252.203.10:8080
******************************* Date: 2019-11-29 *******************************
20:13:41:WU03:FS01:Connecting to 65.254.110.245:8080
20:13:41:WU03:FS01:Assigned to work server 128.252.203.10
20:13:41:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:TU106 [GeForce RTX 2070 Rev. A] M 7465 from 128.252.203.10
20:13:41:WU03:FS01:Connecting to 128.252.203.10:8080
20:13:41:WU03:FS01:Downloading 73.30MiB
20:13:47:WU03:FS01:Download 80.15%
20:13:48:WU03:FS01:Download complete
20:13:48:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
20:13:48:WU03:FS01:Starting
20:13:48:WU03:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/v7/lin/64bit/Core_21.fah/FahCore_21 -dir 03 -suffix 01 -version 705 -lifeline 5410 -checkpoint 30 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
20:13:48:WU03:FS01:Started FahCore on PID 9419
20:13:48:WU03:FS01:Core PID:9423
20:13:48:WU03:FS01:FahCore 0x21 started
20:13:48:WU03:FS01:0x21:*********************** Log Started 2019-11-29T20:13:48Z ***********************
20:13:48:WU03:FS01:0x21:Project: 14264 (Run 0, Clone 67, Gen 54)
20:13:48:WU03:FS01:0x21:Unit: 0x0000003d80fccb0a5daa0009ba2a3e33
20:13:48:WU03:FS01:0x21:CPU: 0x00000000000000000000000000000000
20:13:48:WU03:FS01:0x21:Machine: 1
20:13:48:WU03:FS01:0x21:Reading tar file core.xml
20:13:48:WU03:FS01:0x21:Reading tar file integrator.xml
20:13:48:WU03:FS01:0x21:Reading tar file state.xml
20:13:48:WU03:FS01:0x21:Reading tar file system.xml
20:13:48:WU03:FS01:0x21:Digital signatures verified
20:13:48:WU03:FS01:0x21:Folding@home GPU Core21 Folding@home Core
20:13:48:WU03:FS01:0x21:Version 0.0.20
20:14:06:WU03:FS01:0x21:Completed 0 out of 1000000 steps (0%)
20:14:06:WU03:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
20:15:08:WU03:FS01:0x21:Completed 10000 out of 1000000 steps (1%)
20:16:10:WU03:FS01:0x21:Completed 20000 out of 1000000 steps (2%)
20:17:18:WU03:FS01:0x21:Completed 30000 out of 1000000 steps (3%)
20:18:20:WU03:FS01:0x21:Completed 40000 out of 1000000 steps (4%)
20:19:27:WU03:FS01:0x21:Completed 50000 out of 1000000 steps (5%)
20:20:29:WU03:FS01:0x21:Completed 60000 out of 1000000 steps (6%)
20:21:36:WU03:FS01:0x21:Completed 70000 out of 1000000 steps (7%)
20:22:38:WU03:FS01:0x21:Completed 80000 out of 1000000 steps (8%)
20:23:45:WU03:FS01:0x21:Completed 90000 out of 1000000 steps (9%)
20:24:46:WU03:FS01:0x21:Completed 100000 out of 1000000 steps (10%)
20:25:54:WU03:FS01:0x21:Completed 110000 out of 1000000 steps (11%)
20:26:55:WU03:FS01:0x21:Completed 120000 out of 1000000 steps (12%)
20:28:02:WU03:FS01:0x21:Completed 130000 out of 1000000 steps (13%)
20:29:04:WU03:FS01:0x21:Completed 140000 out of 1000000 steps (14%)
20:30:11:WU03:FS01:0x21:Completed 150000 out of 1000000 steps (15%)
20:31:13:WU03:FS01:0x21:Completed 160000 out of 1000000 steps (16%)
20:32:20:WU03:FS01:0x21:Completed 170000 out of 1000000 steps (17%)
20:33:22:WU03:FS01:0x21:Completed 180000 out of 1000000 steps (18%)
20:34:29:WU03:FS01:0x21:Completed 190000 out of 1000000 steps (19%)
20:35:31:WU03:FS01:0x21:Completed 200000 out of 1000000 steps (20%)
20:36:37:WU03:FS01:0x21:Completed 210000 out of 1000000 steps (21%)
20:37:39:WU03:FS01:0x21:Completed 220000 out of 1000000 steps (22%)
20:38:45:WU03:FS01:0x21:Completed 230000 out of 1000000 steps (23%)
20:39:47:WU03:FS01:0x21:Completed 240000 out of 1000000 steps (24%)
20:40:54:WU03:FS01:0x21:Completed 250000 out of 1000000 steps (25%)
20:41:56:WU03:FS01:0x21:Completed 260000 out of 1000000 steps (26%)
20:43:02:WU03:FS01:0x21:Completed 270000 out of 1000000 steps (27%)
20:44:04:WU03:FS01:0x21:Completed 280000 out of 1000000 steps (28%)
20:45:11:WU03:FS01:0x21:Completed 290000 out of 1000000 steps (29%)
20:46:12:WU03:FS01:0x21:Completed 300000 out of 1000000 steps (30%)
20:47:20:WU03:FS01:0x21:Completed 310000 out of 1000000 steps (31%)
20:48:22:WU03:FS01:0x21:Completed 320000 out of 1000000 steps (32%)
20:49:29:WU03:FS01:0x21:Completed 330000 out of 1000000 steps (33%)
20:50:30:WU03:FS01:0x21:Completed 340000 out of 1000000 steps (34%)
20:51:37:WU03:FS01:0x21:Completed 350000 out of 1000000 steps (35%)
20:52:39:WU03:FS01:0x21:Completed 360000 out of 1000000 steps (36%)
20:53:46:WU03:FS01:0x21:Completed 370000 out of 1000000 steps (37%)
20:54:47:WU03:FS01:0x21:Completed 380000 out of 1000000 steps (38%)
20:55:54:WU03:FS01:0x21:Completed 390000 out of 1000000 steps (39%)
20:56:56:WU03:FS01:0x21:Completed 400000 out of 1000000 steps (40%)
20:58:03:WU03:FS01:0x21:Completed 410000 out of 1000000 steps (41%)
20:59:05:WU03:FS01:0x21:Completed 420000 out of 1000000 steps (42%)
21:00:11:WU03:FS01:0x21:Completed 430000 out of 1000000 steps (43%)
21:01:13:WU03:FS01:0x21:Completed 440000 out of 1000000 steps (44%)
21:02:20:WU03:FS01:0x21:Completed 450000 out of 1000000 steps (45%)
21:03:22:WU03:FS01:0x21:Completed 460000 out of 1000000 steps (46%)
21:04:29:WU03:FS01:0x21:Completed 470000 out of 1000000 steps (47%)
21:05:31:WU03:FS01:0x21:Completed 480000 out of 1000000 steps (48%)
21:06:37:WU03:FS01:0x21:Completed 490000 out of 1000000 steps (49%)
21:07:39:WU03:FS01:0x21:Completed 500000 out of 1000000 steps (50%)
21:08:46:WU03:FS01:0x21:Completed 510000 out of 1000000 steps (51%)
21:09:47:WU03:FS01:0x21:Completed 520000 out of 1000000 steps (52%)
21:10:55:WU03:FS01:0x21:Completed 530000 out of 1000000 steps (53%)
21:11:56:WU03:FS01:0x21:Completed 540000 out of 1000000 steps (54%)
21:13:03:WU03:FS01:0x21:Completed 550000 out of 1000000 steps (55%)
21:14:05:WU03:FS01:0x21:Completed 560000 out of 1000000 steps (56%)
21:15:12:WU03:FS01:0x21:Completed 570000 out of 1000000 steps (57%)
21:16:14:WU03:FS01:0x21:Completed 580000 out of 1000000 steps (58%)
21:17:20:WU03:FS01:0x21:Completed 590000 out of 1000000 steps (59%)
21:18:22:WU03:FS01:0x21:Completed 600000 out of 1000000 steps (60%)
21:19:28:WU03:FS01:0x21:Completed 610000 out of 1000000 steps (61%)
21:20:30:WU03:FS01:0x21:Completed 620000 out of 1000000 steps (62%)
21:21:37:WU03:FS01:0x21:Completed 630000 out of 1000000 steps (63%)
21:22:39:WU03:FS01:0x21:Completed 640000 out of 1000000 steps (64%)
21:23:46:WU03:FS01:0x21:Completed 650000 out of 1000000 steps (65%)
21:24:47:WU03:FS01:0x21:Completed 660000 out of 1000000 steps (66%)
21:25:55:WU03:FS01:0x21:Completed 670000 out of 1000000 steps (67%)
21:26:56:WU03:FS01:0x21:Completed 680000 out of 1000000 steps (68%)
21:28:04:WU03:FS01:0x21:Completed 690000 out of 1000000 steps (69%)
21:29:06:WU03:FS01:0x21:Completed 700000 out of 1000000 steps (70%)
21:30:13:WU03:FS01:0x21:Completed 710000 out of 1000000 steps (71%)
21:31:14:WU03:FS01:0x21:Completed 720000 out of 1000000 steps (72%)
21:32:22:WU03:FS01:0x21:Completed 730000 out of 1000000 steps (73%)
21:33:23:WU03:FS01:0x21:Completed 740000 out of 1000000 steps (74%)
21:34:30:WU03:FS01:0x21:Completed 750000 out of 1000000 steps (75%)
21:35:32:WU03:FS01:0x21:Completed 760000 out of 1000000 steps (76%)
21:36:39:WU03:FS01:0x21:Completed 770000 out of 1000000 steps (77%)
21:37:41:WU03:FS01:0x21:Completed 780000 out of 1000000 steps (78%)
21:38:48:WU03:FS01:0x21:Completed 790000 out of 1000000 steps (79%)
21:39:50:WU03:FS01:0x21:Completed 800000 out of 1000000 steps (80%)
21:40:56:WU03:FS01:0x21:Completed 810000 out of 1000000 steps (81%)
21:41:57:WU03:FS01:0x21:Completed 820000 out of 1000000 steps (82%)
21:43:04:WU03:FS01:0x21:Completed 830000 out of 1000000 steps (83%)
21:44:05:WU03:FS01:0x21:Completed 840000 out of 1000000 steps (84%)
21:45:11:WU03:FS01:0x21:Completed 850000 out of 1000000 steps (85%)
21:46:12:WU03:FS01:0x21:Completed 860000 out of 1000000 steps (86%)
21:47:19:WU03:FS01:0x21:Completed 870000 out of 1000000 steps (87%)
21:48:21:WU03:FS01:0x21:Completed 880000 out of 1000000 steps (88%)
21:49:28:WU03:FS01:0x21:Completed 890000 out of 1000000 steps (89%)
21:50:30:WU03:FS01:0x21:Completed 900000 out of 1000000 steps (90%)
21:51:36:WU03:FS01:0x21:Completed 910000 out of 1000000 steps (91%)
21:52:38:WU03:FS01:0x21:Completed 920000 out of 1000000 steps (92%)
21:53:45:WU03:FS01:0x21:Completed 930000 out of 1000000 steps (93%)
21:54:47:WU03:FS01:0x21:Completed 940000 out of 1000000 steps (94%)
21:55:54:WU03:FS01:0x21:Completed 950000 out of 1000000 steps (95%)
21:56:55:WU03:FS01:0x21:Completed 960000 out of 1000000 steps (96%)
21:58:02:WU03:FS01:0x21:Completed 970000 out of 1000000 steps (97%)
21:59:04:WU03:FS01:0x21:Completed 980000 out of 1000000 steps (98%)
22:00:11:WU03:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
22:01:12:WU03:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
22:01:18:WU03:FS01:0x21:Saving result file logfile_01.txt
22:01:18:WU03:FS01:0x21:Saving result file checkpointState.xml
22:01:18:WU03:FS01:0x21:Saving result file checkpt.crc
22:01:18:WU03:FS01:0x21:Saving result file log.txt
22:01:19:WU03:FS01:0x21:Saving result file positions.xtc
22:01:19:WU03:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
22:01:19:WU03:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:01:19:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
22:01:19:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
22:01:19:WU03:FS01:Connecting to 128.252.203.10:8080
22:17:55:WU03:FS01:Upload 91.60%
22:17:55:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
22:17:55:WU03:FS01:Trying to send results to collection server
22:17:55:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
22:17:55:WU03:FS01:Connecting to 155.247.166.219:8080
22:17:55:ERROR:WU03:FS01:Exception: Transfer failed
22:17:55:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
22:17:55:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
22:17:55:WU03:FS01:Connecting to 128.252.203.10:8080
22:18:01:WU03:FS01:Upload 61.99%
22:34:10:WU03:FS01:Upload 87.92%
22:34:10:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
22:34:10:WU03:FS01:Trying to send results to collection server
22:34:10:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
22:34:10:WU03:FS01:Connecting to 155.247.166.219:8080
22:34:10:ERROR:WU03:FS01:Exception: Transfer failed
22:34:10:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
22:34:10:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
22:34:10:WU03:FS01:Connecting to 128.252.203.10:8080
22:50:55:WU03:FS01:Upload 90.25%
22:50:55:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
22:50:55:WU03:FS01:Trying to send results to collection server
22:50:55:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
22:50:55:WU03:FS01:Connecting to 155.247.166.219:8080
22:50:55:ERROR:WU03:FS01:Exception: Transfer failed
22:50:55:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
22:50:55:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
22:50:55:WU03:FS01:Connecting to 128.252.203.10:8080
22:51:01:WU03:FS01:Upload 85.95%
23:07:41:WU03:FS01:Upload 90.07%
23:07:41:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
23:07:41:WU03:FS01:Trying to send results to collection server
23:07:41:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
23:07:41:WU03:FS01:Connecting to 155.247.166.219:8080
23:07:41:ERROR:WU03:FS01:Exception: Transfer failed
23:07:41:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
23:07:41:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
23:07:41:WU03:FS01:Connecting to 128.252.203.10:8080
23:24:10:WU03:FS01:Upload 91.60%
23:24:10:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
23:24:10:WU03:FS01:Trying to send results to collection server
23:24:10:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
23:24:10:WU03:FS01:Connecting to 155.247.166.219:8080
23:24:10:ERROR:WU03:FS01:Exception: Transfer failed
23:24:11:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
23:24:11:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
23:24:11:WU03:FS01:Connecting to 128.252.203.10:8080
23:24:17:WU03:FS01:Upload 29.52%
23:40:35:WU03:FS01:Upload 89.27%
23:40:35:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
23:40:35:WU03:FS01:Trying to send results to collection server
23:40:35:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
23:40:35:WU03:FS01:Connecting to 155.247.166.219:8080
23:40:35:ERROR:WU03:FS01:Exception: Transfer failed
23:40:36:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
23:40:36:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
23:40:36:WU03:FS01:Connecting to 128.252.203.10:8080
23:57:21:WU03:FS01:Upload 91.24%
23:57:21:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
23:57:21:WU03:FS01:Trying to send results to collection server
23:57:21:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
23:57:21:WU03:FS01:Connecting to 155.247.166.219:8080
23:57:21:ERROR:WU03:FS01:Exception: Transfer failed
23:57:21:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
23:57:21:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
23:57:21:WU03:FS01:Connecting to 128.252.203.10:8080
00:14:06:WU03:FS01:Upload 91.06%
00:14:06:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
*********************** Log Started 2019-11-30T00:18:23Z ***********************
00:18:23:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
00:18:23:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
00:18:23:WU03:FS01:Connecting to 128.252.203.10:8080
00:18:29:WU03:FS01:Upload 33.82%
00:18:35:WU03:FS01:Upload 84.33%
*********************** Log Started 2019-11-30T00:18:23Z ***********************
00:18:23:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
00:18:23:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
00:18:23:WU03:FS01:Connecting to 128.252.203.10:8080
00:18:29:WU03:FS01:Upload 33.82%
00:18:35:WU03:FS01:Upload 84.33%
00:35:10:WU03:FS01:Upload 89.36%
00:35:10:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
00:35:10:WU03:FS01:Trying to send results to collection server
00:35:10:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
00:35:10:WU03:FS01:Connecting to 155.247.166.219:8080
00:35:10:ERROR:WU03:FS01:Exception: Transfer failed
00:35:10:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
00:35:10:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
00:35:10:WU03:FS01:Connecting to 128.252.203.10:8080
00:35:16:WU03:FS01:Upload 89.09%
00:51:49:WU03:FS01:Upload 91.15%
00:51:49:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
00:51:49:WU03:FS01:Trying to send results to collection server
00:51:49:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
00:51:49:WU03:FS01:Connecting to 155.247.166.219:8080
00:51:49:ERROR:WU03:FS01:Exception: Transfer failed
00:51:50:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
00:51:50:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
00:51:50:WU03:FS01:Connecting to 128.252.203.10:8080
01:08:35:WU03:FS01:Upload 91.24%
01:08:35:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
01:08:35:WU03:FS01:Trying to send results to collection server
01:08:35:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
01:08:35:WU03:FS01:Connecting to 155.247.166.219:8080
01:08:35:ERROR:WU03:FS01:Exception: Transfer failed
01:08:35:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
01:08:35:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
01:08:35:WU03:FS01:Connecting to 128.252.203.10:8080
01:25:20:WU03:FS01:Upload 90.43%
01:25:20:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
01:25:20:WU03:FS01:Trying to send results to collection server
01:25:20:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
01:25:20:WU03:FS01:Connecting to 155.247.166.219:8080
01:25:20:ERROR:WU03:FS01:Exception: Transfer failed
01:25:21:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
01:25:21:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
01:25:21:WU03:FS01:Connecting to 128.252.203.10:8080
01:25:27:WU03:FS01:Upload 76.62%
01:41:35:WU03:FS01:Upload 88.01%
01:41:35:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
01:41:35:WU03:FS01:Trying to send results to collection server
01:41:35:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
01:41:35:WU03:FS01:Connecting to 155.247.166.219:8080
01:41:35:ERROR:WU03:FS01:Exception: Transfer failed
01:41:36:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
01:41:36:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
01:41:36:WU03:FS01:Connecting to 128.252.203.10:8080
01:58:21:WU03:FS01:Upload 90.07%
01:58:21:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
01:58:21:WU03:FS01:Trying to send results to collection server
01:58:21:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
01:58:21:WU03:FS01:Connecting to 155.247.166.219:8080
01:58:21:ERROR:WU03:FS01:Exception: Transfer failed
01:58:21:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
01:58:21:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
01:58:21:WU03:FS01:Connecting to 128.252.203.10:8080
01:58:27:WU03:FS01:Upload 87.47%
02:14:58:WU03:FS01:Upload 91.42%
02:14:58:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
02:14:58:WU03:FS01:Trying to send results to collection server
02:14:58:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
02:14:58:WU03:FS01:Connecting to 155.247.166.219:8080
02:14:58:ERROR:WU03:FS01:Exception: Transfer failed
02:14:58:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
02:14:58:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
02:14:58:WU03:FS01:Connecting to 128.252.203.10:8080
02:31:44:WU03:FS01:Upload 91.15%
02:31:44:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
02:31:44:WU03:FS01:Trying to send results to collection server
02:31:44:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
02:31:44:WU03:FS01:Connecting to 155.247.166.219:8080
02:31:44:ERROR:WU03:FS01:Exception: Transfer failed
02:32:55:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
02:32:55:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
02:32:55:WU03:FS01:Connecting to 128.252.203.10:8080
02:49:24:WU03:FS01:Upload 91.42%
02:49:25:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
02:49:25:WU03:FS01:Trying to send results to collection server
02:49:25:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
02:49:25:WU03:FS01:Connecting to 155.247.166.219:8080
02:49:25:ERROR:WU03:FS01:Exception: Transfer failed
03:01:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
03:01:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
03:01:57:WU03:FS01:Connecting to 128.252.203.10:8080
03:18:42:WU03:FS01:Upload 91.60%
03:18:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
03:18:42:WU03:FS01:Trying to send results to collection server
03:18:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
03:18:42:WU03:FS01:Connecting to 155.247.166.219:8080
03:18:42:ERROR:WU03:FS01:Exception: Transfer failed
03:48:56:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
03:48:56:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
03:48:56:WU03:FS01:Connecting to 128.252.203.10:8080
04:05:42:WU03:FS01:Upload 91.51%
04:05:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
04:05:42:WU03:FS01:Trying to send results to collection server
04:05:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
04:05:42:WU03:FS01:Connecting to 155.247.166.219:8080
04:05:42:ERROR:WU03:FS01:Exception: Transfer failed
05:04:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
05:04:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
05:04:57:WU03:FS01:Connecting to 128.252.203.10:8080
05:05:03:WU03:FS01:Upload 88.82%
05:21:43:WU03:FS01:Upload 90.79%
05:21:43:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
05:21:43:WU03:FS01:Trying to send results to collection server
05:21:43:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
05:21:43:WU03:FS01:Connecting to 155.247.166.219:8080
05:21:43:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2019-11-30 *******************************
07:07:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
07:07:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
07:07:57:WU03:FS01:Connecting to 128.252.203.10:8080
07:24:42:WU03:FS01:Upload 91.51%
07:24:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
07:24:42:WU03:FS01:Trying to send results to collection server
07:24:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
07:24:42:WU03:FS01:Connecting to 155.247.166.219:8080
07:24:42:ERROR:WU03:FS01:Exception: Transfer failed
02:31:44:WU03:FS01:Upload 91.15%
02:31:44:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
02:31:44:WU03:FS01:Trying to send results to collection server
02:31:44:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
02:31:44:WU03:FS01:Connecting to 155.247.166.219:8080
02:31:44:ERROR:WU03:FS01:Exception: Transfer failed
02:32:55:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
02:32:55:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
02:32:55:WU03:FS01:Connecting to 128.252.203.10:8080
02:49:24:WU03:FS01:Upload 91.42%
02:49:25:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
02:49:25:WU03:FS01:Trying to send results to collection server
02:49:25:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
02:49:25:WU03:FS01:Connecting to 155.247.166.219:8080
02:49:25:ERROR:WU03:FS01:Exception: Transfer failed
03:01:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
03:01:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
03:01:57:WU03:FS01:Connecting to 128.252.203.10:8080
03:18:42:WU03:FS01:Upload 91.60%
03:18:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
03:18:42:WU03:FS01:Trying to send results to collection server
03:18:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
03:18:42:WU03:FS01:Connecting to 155.247.166.219:8080
03:18:42:ERROR:WU03:FS01:Exception: Transfer failed
03:48:56:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
03:48:56:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
03:48:56:WU03:FS01:Connecting to 128.252.203.10:8080
04:05:42:WU03:FS01:Upload 91.51%
04:05:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
04:05:42:WU03:FS01:Trying to send results to collection server
04:05:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
04:05:42:WU03:FS01:Connecting to 155.247.166.219:8080
04:05:42:ERROR:WU03:FS01:Exception: Transfer failed
05:04:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
05:04:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
05:04:57:WU03:FS01:Connecting to 128.252.203.10:8080
05:05:03:WU03:FS01:Upload 88.82%
05:21:43:WU03:FS01:Upload 90.79%
05:21:43:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
05:21:43:WU03:FS01:Trying to send results to collection server
05:21:43:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
05:21:43:WU03:FS01:Connecting to 155.247.166.219:8080
05:21:43:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2019-11-30 *******************************
07:07:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
07:07:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
07:07:57:WU03:FS01:Connecting to 128.252.203.10:8080
07:24:42:WU03:FS01:Upload 91.51%
07:24:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
07:24:42:WU03:FS01:Trying to send results to collection server
07:24:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
07:24:42:WU03:FS01:Connecting to 155.247.166.219:8080
07:24:42:ERROR:WU03:FS01:Exception: Transfer failed
02:49:24:WU03:FS01:Upload 91.42%
02:49:25:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
02:49:25:WU03:FS01:Trying to send results to collection server
02:49:25:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
02:49:25:WU03:FS01:Connecting to 155.247.166.219:8080
02:49:25:ERROR:WU03:FS01:Exception: Transfer failed
03:01:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
03:01:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
03:01:57:WU03:FS01:Connecting to 128.252.203.10:8080
03:18:42:WU03:FS01:Upload 91.60%
03:18:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
03:18:42:WU03:FS01:Trying to send results to collection server
03:18:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
03:18:42:WU03:FS01:Connecting to 155.247.166.219:8080
03:18:42:ERROR:WU03:FS01:Exception: Transfer failed
03:48:56:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
03:48:56:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
03:48:56:WU03:FS01:Connecting to 128.252.203.10:8080
04:05:42:WU03:FS01:Upload 91.51%
04:05:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
04:05:42:WU03:FS01:Trying to send results to collection server
04:05:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
04:05:42:WU03:FS01:Connecting to 155.247.166.219:8080
04:05:42:ERROR:WU03:FS01:Exception: Transfer failed
05:04:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
05:04:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
05:04:57:WU03:FS01:Connecting to 128.252.203.10:8080
05:05:03:WU03:FS01:Upload 88.82%
05:21:43:WU03:FS01:Upload 90.79%
05:21:43:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
05:21:43:WU03:FS01:Trying to send results to collection server
05:21:43:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
05:21:43:WU03:FS01:Connecting to 155.247.166.219:8080
05:21:43:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2019-11-30 *******************************
07:07:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
07:07:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
07:07:57:WU03:FS01:Connecting to 128.252.203.10:8080
07:24:42:WU03:FS01:Upload 91.51%
07:24:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
07:24:42:WU03:FS01:Trying to send results to collection server
07:24:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
07:24:42:WU03:FS01:Connecting to 155.247.166.219:8080
07:24:42:ERROR:WU03:FS01:Exception: Transfer failed
02:49:24:WU03:FS01:Upload 91.42%
02:49:25:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
02:49:25:WU03:FS01:Trying to send results to collection server
02:49:25:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
02:49:25:WU03:FS01:Connecting to 155.247.166.219:8080
02:49:25:ERROR:WU03:FS01:Exception: Transfer failed
03:01:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
03:01:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
03:01:57:WU03:FS01:Connecting to 128.252.203.10:8080
03:18:42:WU03:FS01:Upload 91.60%
03:18:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
03:18:42:WU03:FS01:Trying to send results to collection server
03:18:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
03:18:42:WU03:FS01:Connecting to 155.247.166.219:8080
03:18:42:ERROR:WU03:FS01:Exception: Transfer failed
03:48:56:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
03:48:56:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
03:48:56:WU03:FS01:Connecting to 128.252.203.10:8080
04:05:42:WU03:FS01:Upload 91.51%
04:05:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
04:05:42:WU03:FS01:Trying to send results to collection server
04:05:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
04:05:42:WU03:FS01:Connecting to 155.247.166.219:8080
04:05:42:ERROR:WU03:FS01:Exception: Transfer failed
05:04:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
05:04:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
05:04:57:WU03:FS01:Connecting to 128.252.203.10:8080
05:05:03:WU03:FS01:Upload 88.82%
05:21:43:WU03:FS01:Upload 90.79%
05:21:43:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
05:21:43:WU03:FS01:Trying to send results to collection server
05:21:43:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
05:21:43:WU03:FS01:Connecting to 155.247.166.219:8080
05:21:43:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2019-11-30 *******************************
07:07:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
07:07:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
07:07:57:WU03:FS01:Connecting to 128.252.203.10:8080
07:24:42:WU03:FS01:Upload 91.51%
07:24:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
07:24:42:WU03:FS01:Trying to send results to collection server
07:24:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
07:24:42:WU03:FS01:Connecting to 155.247.166.219:8080
07:24:42:ERROR:WU03:FS01:Exception: Transfer failed
10:26:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
10:26:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
10:26:57:WU03:FS01:Connecting to 128.252.203.10:8080
10:43:27:WU03:FS01:Upload 90.07%
10:43:27:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
10:43:27:WU03:FS01:Trying to send results to collection server
10:43:27:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
10:43:27:WU03:FS01:Connecting to 155.247.166.219:8080
10:43:27:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2019-11-30 *******************************
******************************* Date: 2019-11-30 *******************************
07:07:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
07:07:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
07:07:57:WU03:FS01:Connecting to 128.252.203.10:8080
07:24:42:WU03:FS01:Upload 91.51%
07:24:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
07:24:42:WU03:FS01:Trying to send results to collection server
07:24:42:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
07:24:42:WU03:FS01:Connecting to 155.247.166.219:8080
07:24:42:ERROR:WU03:FS01:Exception: Transfer failed
10:26:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
10:26:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
10:26:57:WU03:FS01:Connecting to 128.252.203.10:8080
10:43:27:WU03:FS01:Upload 90.07%
10:43:27:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
10:43:27:WU03:FS01:Trying to send results to collection server
10:43:27:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
10:43:27:WU03:FS01:Connecting to 155.247.166.219:8080
10:43:27:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2019-11-30 *******************************
10:26:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
10:26:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
10:26:57:WU03:FS01:Connecting to 128.252.203.10:8080
10:43:27:WU03:FS01:Upload 90.07%
10:43:27:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
10:43:27:WU03:FS01:Trying to send results to collection server
10:43:27:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
10:43:27:WU03:FS01:Connecting to 155.247.166.219:8080
10:43:27:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2019-11-30 *******************************
10:26:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
10:26:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
10:26:57:WU03:FS01:Connecting to 128.252.203.10:8080
10:43:27:WU03:FS01:Upload 90.07%
10:43:27:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
10:43:27:WU03:FS01:Trying to send results to collection server
10:43:27:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
10:43:27:WU03:FS01:Connecting to 155.247.166.219:8080
10:43:27:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2019-11-30 *******************************
15:48:57:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14264 run:0 clone:67 gen:54 core:0x21 unit:0x0000003d80fccb0a5daa0009ba2a3e33
15:48:57:WU03:FS01:Uploading 69.67MiB to 128.252.203.10
15:48:57:WU03:FS01:Connecting to 128.252.203.10:8080
16:05:26:WU03:FS01:Upload 91.78%
16:05:26:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
16:05:26:WU03:FS01:Trying to send results to collection server
16:05:26:WU03:FS01:Uploading 69.67MiB to 155.247.166.219
16:05:26:WU03:FS01:Connecting to 155.247.166.219:8080
16:05:26:ERROR:WU03:FS01:Exception: Transfer failedRe: Multiple WU's Fail downld/upld to 155.247.166.*
If a server has been down, clients which have tried to connect will undergo an exponential decay in their retry frequency. Thus those who have been trying for a while will not immediately attempt to upload or download. The server will still experience a burst of attempts but the backlog should gradually decay as long as the server can eventually handle the steady-state backlog.MeeLee wrote:VVoelz, is it possible to implement some sort of queuing approach, rather than giving everyone access at once to down/upload?
As long as you have x-amount of connections going at a same time, and an y-amount of queues, it should offer a much more streamlined experience, as to trying to serve everyone at the same time.
If the steady-state backlog exceeds the capacity that the server can handle, nothing can be done except to buy more servers -- and they're already on order. The server can only block the those excess connections and that won't resolve anything. It's the client that initiates the connections and it doesn't have enough information to decide if you should be queued, and if so, for how long. Clients cannot manage a queue that it cannot perceive.
There will still be excess clients creating a DoS attack on the server. Some will get throughk, but many will be blocked and gripes will continue.
Posting FAH's log:
How to provide enough info to get helpful support.
How to provide enough info to get helpful support.