Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

jrsynow · Post by **jrsynow** » Thu Dec 22, 2022 4:14 am

Hi,

I noticed that a handful of my FAH clients were struggling to upload completed work units to the Temple University servers about a month ago, and still struggle to this day. It's only when my workstations upload to these servers that I notices the slowdowns. I've noticed the slowdowns at two locations with three different ISPs. The slowdowns seem to be greatly amplified by the slower upload rates of an ISP. There will be slivers of times where a lot of data gets pushed through, then the upload will slow to a trickle and even sometimes fail and restart. Below is a snippet of one of my log files where it takes 45 minutes and three attempts to upload 31Mbytes of data (~20 minutes if you don't count the additional attempts) with a 2Mbit upload connection (should take less than 3 minutes). I've run speed tests in parallel/afterwards just to make sure that its not a connection problem on my end, and they've all come back close to the advertised connection speed. The server status page shows that the servers have been up for two months straight, maybe the need a reboot? Please let me know if there's anything I can do to help debug this issue.

Code: Select all

02:55:23:WU00:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
02:55:24:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:55:24:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18449 run:1 clone:29 gen:528 core:0x22 unit:0x0000001d000002100000481100000001
02:55:24:WU00:FS00:Uploading 31.14MiB to 129.32.209.202
02:55:24:WU00:FS00:Connecting to 129.32.209.202:8080
02:55:30:WU00:FS00:Upload 4.21%
02:55:36:WU00:FS00:Upload 9.03%
02:55:42:WU00:FS00:Upload 14.45%
02:55:55:WU00:FS00:Upload 14.85%
02:56:12:WU01:FS02:0x22:Completed 362500 out of 1250000 steps (29%)
02:56:53:WU00:FS00:Upload 15.05%
02:57:50:WU00:FS00:Upload 16.46%
02:57:57:WU01:FS02:0x22:Completed 375000 out of 1250000 steps (30%)
02:57:59:WU01:FS02:0x22:Checkpoint completed at step 375000
02:58:14:WU00:FS00:Upload 16.66%
02:58:20:WU00:FS00:Upload 20.47%
02:58:26:WU00:FS00:Upload 23.68%
02:59:23:WU00:FS00:Upload 25.09%
02:59:43:WU01:FS02:0x22:Completed 387500 out of 1250000 steps (31%)
03:00:49:WU00:FS00:Upload 25.29%
03:00:55:WU00:FS00:Upload 29.50%
03:01:01:WU00:FS00:Upload 32.71%
03:01:25:WU01:FS02:0x22:Completed 400000 out of 1250000 steps (32%)
03:01:28:WU01:FS02:0x22:Checkpoint completed at step 400000
03:02:01:WU00:FS00:Upload 34.32%
03:02:35:WU00:FS00:Upload 34.52%
03:02:41:WU00:FS00:Upload 38.73%
03:02:48:WU00:FS00:Upload 42.34%
03:03:11:WU01:FS02:0x22:Completed 412500 out of 1250000 steps (33%)
03:03:13:WU00:FS00:Upload 43.95%
03:04:25:WU00:FS00:Upload 44.15%
03:04:58:WU01:FS02:0x22:Completed 425000 out of 1250000 steps (34%)
03:05:00:WU01:FS02:0x22:Checkpoint completed at step 425000
03:05:10:WU00:FS00:Upload 45.55%
03:05:27:WU00:FS00:Upload 45.76%
03:06:15:WU00:FS00:Upload 46.96%
03:06:28:WU00:FS00:Upload 47.16%
03:06:46:WU01:FS02:0x22:Completed 437500 out of 1250000 steps (35%)
03:07:14:WU00:FS00:Upload 48.36%
03:07:20:WU00:FS00:Upload 49.37%
03:07:26:WU00:FS00:Upload 53.58%
03:07:33:WU00:FS00:Upload 55.59%
03:07:42:WU00:FS00:Upload 55.99%
03:07:48:WU00:FS00:Upload 60.00%
03:07:54:WU00:FS00:Upload 64.42%
03:08:00:WU00:FS00:Upload 65.82%
03:08:06:WU00:FS00:Upload 70.24%
03:08:12:WU00:FS00:Upload 74.65%
03:08:18:WU00:FS00:Upload 77.46%
03:08:31:WU01:FS02:0x22:Completed 450000 out of 1250000 steps (36%)
03:08:33:WU01:FS02:0x22:Checkpoint completed at step 450000
03:09:33:WU00:FS00:Upload 79.07%
03:09:39:WU00:FS00:Upload 80.07%
03:09:52:WU00:FS00:Upload 80.27%
03:10:22:WU01:FS02:0x22:Completed 462500 out of 1250000 steps (37%)
03:10:26:WU00:FS00:Upload 80.47%
03:10:51:WU00:FS00:Upload 81.68%
03:11:19:WU00:FS00:Upload 84.29%
03:11:25:WU00:FS00:Upload 88.70%
03:11:32:WU00:FS00:Upload 92.92%
03:12:09:WU01:FS02:0x22:Completed 475000 out of 1250000 steps (38%)
03:12:11:WU01:FS02:0x22:Checkpoint completed at step 475000
03:12:57:WU00:FS00:Upload 94.72%
03:13:36:WU00:FS00:Upload 94.92%
03:13:57:WU01:FS02:0x22:Completed 487500 out of 1250000 steps (39%)
03:15:28:WU00:FS00:Upload 96.33%
03:15:28:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
03:15:28:WU00:FS00:Trying to send results to collection server
03:15:28:WU00:FS00:Uploading 31.14MiB to 129.32.209.205
03:15:28:WU00:FS00:Connecting to 129.32.209.205:8080
03:15:34:WU00:FS00:Upload 4.41%
03:15:40:WU00:FS00:Upload 9.03%
03:15:43:WU01:FS02:0x22:Completed 500000 out of 1250000 steps (40%)
03:15:46:WU01:FS02:0x22:Checkpoint completed at step 500000
03:15:47:WU00:FS00:Upload 14.05%
03:15:53:WU00:FS00:Upload 18.46%
03:15:59:WU00:FS00:Upload 22.68%
03:16:06:WU00:FS00:Upload 26.69%
03:16:14:WU00:FS00:Upload 28.10%
03:17:20:WU00:FS00:Upload 28.30%
03:17:29:WU00:FS00:Upload 28.50%
03:17:30:WU01:FS02:0x22:Completed 512500 out of 1250000 steps (41%)
03:17:35:WU00:FS00:Upload 32.51%
03:17:41:WU00:FS00:Upload 35.92%
03:18:36:WU00:FS00:Upload 36.93%
03:18:42:WU00:FS00:Upload 37.93%
03:19:08:WU00:FS00:Upload 38.13%
03:19:14:WU00:FS00:Upload 39.33%
03:19:17:WU01:FS02:0x22:Completed 525000 out of 1250000 steps (42%)
03:19:19:WU01:FS02:0x22:Checkpoint completed at step 525000
03:19:32:WU00:FS00:Upload 40.74%
03:19:38:WU00:FS00:Upload 44.75%
03:19:45:WU00:FS00:Upload 48.97%
03:20:24:WU00:FS00:Upload 50.57%
03:21:09:WU01:FS02:0x22:Completed 537500 out of 1250000 steps (43%)
03:21:29:WU00:FS00:Upload 50.77%
03:22:04:WU00:FS00:Upload 52.18%
03:22:41:WU00:FS00:Upload 52.38%
03:22:56:WU00:FS00:Upload 53.58%
03:22:59:WU01:FS02:0x22:Completed 550000 out of 1250000 steps (44%)
03:23:01:WU01:FS02:0x22:Checkpoint completed at step 550000
03:23:45:WU00:FS00:Upload 53.78%
03:24:46:WU00:FS00:Upload 55.19%
03:24:50:WU01:FS02:0x22:Completed 562500 out of 1250000 steps (45%)
03:25:51:WU00:FS00:Upload 56.59%
03:25:52:ERROR:WU00:FS00:Exception: Transfer failed
03:25:52:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:18449 run:1 clone:29 gen:528 core:0x22 unit:0x0000001d000002100000481100000001
03:25:52:WU00:FS00:Uploading 31.14MiB to 129.32.209.202
03:25:52:WU00:FS00:Connecting to 129.32.209.202:8080
03:25:58:WU00:FS00:Upload 5.02%
03:26:04:WU00:FS00:Upload 10.03%
03:26:10:WU00:FS00:Upload 13.24%
03:26:37:WU01:FS02:0x22:Completed 575000 out of 1250000 steps (46%)
03:26:39:WU01:FS02:0x22:Checkpoint completed at step 575000
03:26:40:WU00:FS00:Upload 13.85%
03:27:09:WU00:FS00:Upload 14.05%
03:27:15:WU00:FS00:Upload 18.06%
03:27:21:WU00:FS00:Upload 21.47%
03:27:55:WU00:FS00:Upload 22.88%
03:28:26:WU01:FS02:0x22:Completed 587500 out of 1250000 steps (47%)
03:29:04:WU00:FS00:Upload 23.08%
03:29:10:WU00:FS00:Upload 27.09%
03:29:17:WU00:FS00:Upload 30.90%
03:29:41:WU00:FS00:Upload 32.11%
03:30:13:WU01:FS02:0x22:Completed 600000 out of 1250000 steps (48%)
03:30:15:WU01:FS02:0x22:Checkpoint completed at step 600000
03:31:03:WU00:FS00:Upload 32.31%
03:31:09:WU00:FS00:Upload 33.11%
03:31:15:WU00:FS00:Upload 37.33%
03:31:22:WU00:FS00:Upload 39.53%
03:31:28:WU00:FS00:Upload 43.55%
03:31:34:WU00:FS00:Upload 46.56%
03:32:00:WU01:FS02:0x22:Completed 612500 out of 1250000 steps (49%)
03:32:19:WU00:FS00:Upload 47.96%
03:33:18:WU00:FS00:Upload 48.16%
03:33:24:WU00:FS00:Upload 52.18%
03:33:30:WU00:FS00:Upload 55.59%
03:33:45:WU01:FS02:0x22:Completed 625000 out of 1250000 steps (50%)
03:33:47:WU01:FS02:0x22:Checkpoint completed at step 625000
03:34:27:WU00:FS00:Upload 56.99%
03:34:49:WU00:FS00:Upload 57.19%
03:34:55:WU00:FS00:Upload 61.61%
03:35:01:WU00:FS00:Upload 64.82%
03:35:12:WU00:FS00:Upload 65.82%
03:35:34:WU01:FS02:0x22:Completed 637500 out of 1250000 steps (51%)
03:36:14:WU00:FS00:Upload 66.02%
03:36:20:WU00:FS00:Upload 67.03%
03:36:26:WU00:FS00:Upload 70.84%
03:36:32:WU00:FS00:Upload 73.85%
03:37:18:WU01:FS02:0x22:Completed 650000 out of 1250000 steps (52%)
03:37:21:WU01:FS02:0x22:Checkpoint completed at step 650000
03:37:33:WU00:FS00:Upload 74.25%
03:38:07:WU00:FS00:Upload 74.45%
03:38:14:WU00:FS00:Upload 75.26%
03:38:20:WU00:FS00:Upload 76.86%
03:39:03:WU00:FS00:Upload 77.46%
03:39:07:WU01:FS02:0x22:Completed 662500 out of 1250000 steps (53%)
03:39:09:WU00:FS00:Upload 80.27%
03:39:16:WU00:FS00:Upload 81.88%
03:39:22:WU00:FS00:Upload 84.89%
03:39:28:WU00:FS00:Upload 87.50%
03:39:34:WU00:FS00:Upload 90.91%
03:39:40:WU00:FS00:Upload 93.32%
03:40:12:WU00:FS00:Upload 94.12%
03:40:18:WU00:FS00:Upload 97.93%
03:40:24:WU00:FS00:Upload complete
03:40:24:WU00:FS00:Server responded WORK_ACK (400)
03:40:24:WU00:FS00:Final credit estimate, 290100.00 points
03:40:24:WU00:FS00:Cleaning up

Post by **Joe_H** » Thu Dec 22, 2022 3:58 pm

I have messaged the person in charge of the servers there, hopefully they can take a look into this.

jrsynow · Post by **jrsynow** » Fri Dec 30, 2022 6:45 pm

It looks like this may be resolved now. The past two jobs that have uploaded results for a Temple Univ. project have completed in 3-4 minutes. Thanks Joe!

kmoyer · Post by **kmoyer** » Fri Jan 06, 2023 4:22 pm

Good Moring

Unfortunately this issue is not yet resolved for clients on slower ISP connections. I have a system that has been attempting to upload 5 work units since prior to 2023-01-05T04:24:22Z. I have stopped folding on the system altogether for now until such time that the Work Units can upload. The behavior is as described by jrsynow, the Work Unit will begin uploading fine, and then the upload slows to a crawl before stopping and failing to upload. I have inserted logs below separating them by SLOT for readability. I have had to substantially trim them due to the 60k character limit. I have retained the last 13 upload attempts for each Work Unit.

NOTE:
The client did successfully upload Work Units to other servers, just not the 129.32.209.20x servers. The client is connected to an ISP connection of ~8 Mbit/sec DOWN, ~0.8 Mbit/sec UP. I have another system at a different location with an ISP connection ~50x faster. It is uploading Work Units to the 129.32.209.20X servers without issue.

I have had this system folding on the same ISP connection since 2011. I have had isolated upload issues before though never for so many Work Units at the same time and for such an extended period of time.

Thanks and Regards,
Kyle

Code: Select all

SLOT #00

*********************** Log Started 2023-01-05T04:24:22Z ***********************
...
09:32:33:WU00:FS01:Trying to send results to collection server
09:32:33:WU00:FS01:Uploading 29.40MiB to 129.32.209.204
09:32:33:WU00:FS01:Connecting to 129.32.209.204:8080
09:32:49:WU00:FS01:Upload 0.43%
09:32:56:WU00:FS01:Upload 2.13%
09:33:04:WU00:FS01:Upload 2.98%
09:33:10:WU00:FS01:Upload 4.89%
09:33:16:WU00:FS01:Upload 6.80%
09:33:22:WU00:FS01:Upload 9.14%
09:33:34:WU00:FS01:Upload 11.05%
09:33:43:WU00:FS01:Upload 11.27%
09:33:53:WU00:FS01:Upload 11.69%
09:33:59:WU00:FS01:Upload 13.60%
09:34:05:WU00:FS01:Upload 15.30%
09:34:11:WU00:FS01:Upload 15.94%
09:34:18:WU00:FS01:Upload 16.15%
09:34:32:WU00:FS01:Upload 16.58%
09:34:48:WU00:FS01:Upload 16.79%
09:34:55:WU00:FS01:Upload 17.01%
09:35:01:WU00:FS01:Upload 17.22%
09:35:07:WU00:FS01:Upload 18.92%
09:35:13:WU00:FS01:Upload 20.62%
09:35:19:WU00:FS01:Upload 22.53%
09:35:25:WU00:FS01:Upload 24.23%
09:35:36:WU00:FS01:Upload 24.87%
09:35:42:WU00:FS01:Upload 26.57%
09:35:51:WU00:FS01:Upload 28.06%
09:38:29:WU00:FS01:Upload 29.76%
09:39:38:WU00:FS01:Upload 29.97%
09:41:20:WU00:FS01:Upload 31.25%
09:42:28:WU00:FS01:Upload 31.46%
09:42:34:WU00:FS01:Upload 33.37%
09:42:40:WU00:FS01:Upload 35.29%
09:42:47:WU00:FS01:Upload 36.14%
09:42:54:WU00:FS01:Upload 36.35%
09:43:00:WU00:FS01:Upload 38.05%
09:43:08:WU00:FS01:Upload 39.32%
09:44:42:WU00:FS01:Upload 41.02%
09:44:42:ERROR:WU00:FS01:Exception: Transfer failed
10:22:10:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18448 run:0 clone:319 gen:10 core:0x22 unit:0x0000013f0000000a0000481000000000
10:22:10:WU00:FS01:Uploading 29.40MiB to 129.32.209.202
10:22:10:WU00:FS01:Connecting to 129.32.209.202:8080
10:22:16:WU00:FS01:Upload 1.91%
10:22:22:WU00:FS01:Upload 4.46%
10:22:28:WU00:FS01:Upload 8.72%
10:22:38:WU00:FS01:Upload 8.93%
10:22:44:WU00:FS01:Upload 10.84%
10:22:52:WU00:FS01:Upload 11.90%
10:22:58:WU00:FS01:Upload 13.82%
10:23:07:WU00:FS01:Upload 15.09%
10:24:24:WU00:FS01:Upload 16.58%
******************************* Date: 2023-01-06 *******************************
10:27:20:WU00:FS01:Upload 16.79%
10:27:53:WU00:FS01:Upload 17.22%
10:27:59:WU00:FS01:Upload 18.92%
10:28:07:WU00:FS01:Upload 20.19%
10:28:53:WU00:FS01:Upload 21.47%
10:29:40:WU00:FS01:Upload 21.68%
10:29:46:WU00:FS01:Upload 23.59%
10:29:52:WU00:FS01:Upload 25.51%
10:30:00:WU00:FS01:Upload 26.57%
10:30:06:WU00:FS01:Upload 28.06%
10:32:09:WU00:FS01:Upload 28.70%
10:32:09:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
10:32:09:WU00:FS01:Trying to send results to collection server
10:32:09:WU00:FS01:Uploading 29.40MiB to 129.32.209.204
10:32:09:WU00:FS01:Connecting to 129.32.209.204:8080
10:32:19:WU00:FS01:Upload 0.43%
10:32:25:WU00:FS01:Upload 2.34%
10:32:31:WU00:FS01:Upload 4.46%
10:32:37:WU00:FS01:Upload 8.93%
10:32:50:WU00:FS01:Upload 9.14%
10:32:56:WU00:FS01:Upload 9.35%
10:33:14:WU00:FS01:Upload 9.57%
10:33:20:WU00:FS01:Upload 10.84%
10:33:26:WU00:FS01:Upload 12.75%
10:33:38:WU00:FS01:Upload 13.18%
10:33:44:WU00:FS01:Upload 15.09%
10:33:53:WU00:FS01:Upload 16.58%
10:34:38:WU00:FS01:Upload 18.07%
10:37:57:WU00:FS01:Upload 18.28%
10:39:24:WU00:FS01:Upload 18.49%
10:39:30:WU00:FS01:Upload 20.62%
10:39:36:WU00:FS01:Upload 22.32%
10:39:48:WU00:FS01:Upload 22.96%
10:39:54:WU00:FS01:Upload 24.87%
10:40:03:WU00:FS01:Upload 26.36%
10:43:03:WU00:FS01:Upload 28.06%
10:43:34:WU00:FS01:Upload 28.27%
10:43:34:ERROR:WU00:FS01:Exception: Transfer failed
11:22:10:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18448 run:0 clone:319 gen:10 core:0x22 unit:0x0000013f0000000a0000481000000000
11:22:10:WU00:FS01:Uploading 29.40MiB to 129.32.209.202
11:22:10:WU00:FS01:Connecting to 129.32.209.202:8080
11:22:16:WU00:FS01:Upload 1.91%
11:22:22:WU00:FS01:Upload 4.46%
11:22:28:WU00:FS01:Upload 8.72%
11:22:38:WU00:FS01:Upload 9.14%
11:22:44:WU00:FS01:Upload 10.84%
11:22:52:WU00:FS01:Upload 11.90%
11:22:58:WU00:FS01:Upload 13.82%
11:23:06:WU00:FS01:Upload 14.88%
11:23:27:WU00:FS01:Upload 16.37%
11:26:19:WU00:FS01:Upload 16.58%
11:26:58:WU00:FS01:Upload 17.86%
11:28:33:WU00:FS01:Upload 18.07%
11:28:33:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:28:33:WU00:FS01:Trying to send results to collection server
11:28:33:WU00:FS01:Uploading 29.40MiB to 129.32.209.204
11:28:33:WU00:FS01:Connecting to 129.32.209.204:8080
11:28:54:WU00:FS01:Upload 0.43%
11:29:00:WU00:FS01:Upload 3.61%
11:29:13:WU00:FS01:Upload 6.38%
11:29:19:WU00:FS01:Upload 8.29%
11:29:27:WU00:FS01:Upload 9.14%
11:29:33:WU00:FS01:Upload 11.27%
11:29:41:WU00:FS01:Upload 12.33%
11:30:16:WU00:FS01:Upload 13.82%
11:30:16:ERROR:WU00:FS01:Exception: Transfer failed
12:22:10:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18448 run:0 clone:319 gen:10 core:0x22 unit:0x0000013f0000000a0000481000000000
12:22:10:WU00:FS01:Uploading 29.40MiB to 129.32.209.202
12:22:10:WU00:FS01:Connecting to 129.32.209.202:8080
12:22:16:WU00:FS01:Upload 1.91%
12:22:22:WU00:FS01:Upload 4.46%
12:22:28:WU00:FS01:Upload 8.72%
12:22:39:WU00:FS01:Upload 9.14%
12:22:45:WU00:FS01:Upload 11.05%
12:22:53:WU00:FS01:Upload 11.90%
12:22:59:WU00:FS01:Upload 13.82%
12:23:07:WU00:FS01:Upload 15.09%
12:23:52:WU00:FS01:Upload 16.58%
12:26:02:WU00:FS01:Upload 16.79%
12:27:23:WU00:FS01:Upload 18.07%
12:27:23:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
12:27:23:WU00:FS01:Trying to send results to collection server
12:27:23:WU00:FS01:Uploading 29.40MiB to 129.32.209.204
12:27:23:WU00:FS01:Connecting to 129.32.209.204:8080
12:27:30:WU00:FS01:Upload 0.21%
12:27:36:WU00:FS01:Upload 1.91%
12:27:43:WU00:FS01:Upload 2.98%
12:27:49:WU00:FS01:Upload 4.04%
12:27:57:WU00:FS01:Upload 4.68%
12:30:10:WU00:FS01:Upload 4.89%
12:32:36:WU00:FS01:Upload 5.10%
12:32:37:ERROR:WU00:FS01:Exception: Transfer failed
13:22:10:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18448 run:0 clone:319 gen:10 core:0x22 unit:0x0000013f0000000a0000481000000000
13:22:10:WU00:FS01:Uploading 29.40MiB to 129.32.209.202
13:22:10:WU00:FS01:Connecting to 129.32.209.202:8080
13:22:16:WU00:FS01:Upload 1.91%
13:22:22:WU00:FS01:Upload 3.83%
13:22:28:WU00:FS01:Upload 5.74%
13:22:34:WU00:FS01:Upload 8.93%
13:22:47:WU00:FS01:Upload 11.48%
13:22:53:WU00:FS01:Upload 13.39%
13:23:02:WU00:FS01:Upload 14.45%
13:23:08:WU00:FS01:Upload 16.37%
13:23:17:WU00:FS01:Upload 17.64%
13:25:52:WU00:FS01:Upload 19.13%
13:27:15:WU00:FS01:Upload 19.34%
13:27:15:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:27:15:WU00:FS01:Trying to send results to collection server
13:27:15:WU00:FS01:Uploading 29.40MiB to 129.32.209.204
13:27:15:WU00:FS01:Connecting to 129.32.209.204:8080
13:27:21:WU00:FS01:Upload 1.49%
13:27:27:WU00:FS01:Upload 1.91%
13:27:33:WU00:FS01:Upload 2.98%
13:27:39:WU00:FS01:Upload 7.23%
13:28:10:WU00:FS01:Upload 7.44%
13:28:16:WU00:FS01:Upload 9.35%
13:28:29:WU00:FS01:Upload 10.42%
13:28:35:WU00:FS01:Upload 12.12%
13:28:44:WU00:FS01:Upload 13.60%
13:30:03:WU00:FS01:Upload 15.09%
13:33:19:WU00:FS01:Upload 15.30%
13:33:25:WU00:FS01:Upload 16.37%
13:36:43:WU00:FS01:Upload 16.79%
13:36:49:WU00:FS01:Upload 18.92%
13:36:55:WU00:FS01:Upload 20.83%
13:37:01:WU00:FS01:Upload 21.26%
13:37:08:WU00:FS01:Upload 21.47%
13:37:14:WU00:FS01:Upload 23.38%
13:37:21:WU00:FS01:Upload 24.66%
13:37:27:WU00:FS01:Upload 26.57%
13:37:33:WU00:FS01:Upload 28.27%
13:37:39:WU00:FS01:Upload 28.91%
13:38:21:WU00:FS01:Upload 30.18%
13:38:21:ERROR:WU00:FS01:Exception: Transfer failed
14:22:10:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18448 run:0 clone:319 gen:10 core:0x22 unit:0x0000013f0000000a0000481000000000
14:22:10:WU00:FS01:Uploading 29.40MiB to 129.32.209.202
14:22:10:WU00:FS01:Connecting to 129.32.209.202:8080
14:22:37:WU00:FS01:Upload 0.64%
14:22:51:WU00:FS01:Upload 5.53%
14:22:57:WU00:FS01:Upload 7.23%
14:23:03:WU00:FS01:Upload 8.08%
14:23:09:WU00:FS01:Upload 9.78%
14:23:15:WU00:FS01:Upload 10.63%
14:23:22:WU00:FS01:Upload 10.84%
14:23:28:WU00:FS01:Upload 12.54%
14:23:38:WU00:FS01:Upload 14.24%
14:23:46:WU00:FS01:Upload 14.67%
14:23:52:WU00:FS01:Upload 16.37%
14:23:58:WU00:FS01:Upload 17.64%
14:24:21:WU00:FS01:Upload 18.49%
14:24:27:WU00:FS01:Upload 20.62%
14:24:33:WU00:FS01:Upload 22.32%
14:24:46:WU00:FS01:Upload 23.59%
14:24:52:WU00:FS01:Upload 25.08%
14:25:04:WU00:FS01:Upload 25.72%
14:25:19:WU00:FS01:Upload 26.78%
14:28:26:WU00:FS01:Upload 27.00%
14:31:11:WU00:FS01:Upload 27.21%
14:31:17:WU00:FS01:Upload 29.33%
14:31:26:WU00:FS01:Upload 30.40%
14:31:32:WU00:FS01:Upload 32.10%
14:31:43:WU00:FS01:Upload 33.80%
14:32:18:WU00:FS01:Upload 35.50%
14:32:18:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
14:32:18:WU00:FS01:Trying to send results to collection server
14:32:18:WU00:FS01:Uploading 29.40MiB to 129.32.209.204
14:32:18:WU00:FS01:Connecting to 129.32.209.204:8080
14:32:25:WU00:FS01:Upload 0.64%
14:32:31:WU00:FS01:Upload 1.91%
14:33:19:WU00:FS01:Upload 2.98%
14:33:49:WU00:FS01:Upload 3.19%
14:33:50:ERROR:WU00:FS01:Exception: Transfer failed
15:22:11:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18448 run:0 clone:319 gen:10 core:0x22 unit:0x0000013f0000000a0000481000000000
15:22:11:WU00:FS01:Uploading 29.40MiB to 129.32.209.202
15:22:11:WU00:FS01:Connecting to 129.32.209.202:8080
15:22:39:WU00:FS01:Upload 0.64%
15:22:45:WU00:FS01:Upload 5.53%
15:22:57:WU00:FS01:Upload 5.74%
15:23:03:WU00:FS01:Upload 7.65%
15:23:12:WU00:FS01:Upload 12.12%
15:23:18:WU00:FS01:Upload 13.82%
15:23:27:WU00:FS01:Upload 14.88%
15:23:33:WU00:FS01:Upload 16.79%
15:23:46:WU00:FS01:Upload 17.64%
15:23:52:WU00:FS01:Upload 19.56%
15:23:58:WU00:FS01:Upload 21.26%
15:24:11:WU00:FS01:Upload 21.68%
15:24:17:WU00:FS01:Upload 23.59%
15:24:25:WU00:FS01:Upload 24.66%
15:25:50:WU00:FS01:Upload 26.15%
15:29:47:WU00:FS01:Upload 26.36%
15:31:25:WU00:FS01:Upload 26.57%
15:31:31:WU00:FS01:Upload 28.48%
15:31:37:WU00:FS01:Upload 30.40%
15:31:43:WU00:FS01:Upload 30.82%
15:32:05:WU00:FS01:Upload 32.52%
15:32:14:WU00:FS01:Upload 33.16%
15:32:59:WU00:FS01:Upload 33.37%
15:32:59:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
15:32:59:WU00:FS01:Trying to send results to collection server
15:32:59:WU00:FS01:Uploading 29.40MiB to 129.32.209.204
15:32:59:WU00:FS01:Connecting to 129.32.209.204:8080
15:33:06:WU00:FS01:Upload 0.21%
15:33:16:WU00:FS01:Upload 0.43%
15:33:22:WU00:FS01:Upload 1.70%
15:33:29:WU00:FS01:Upload 3.40%
15:33:38:WU00:FS01:Upload 6.80%
15:35:04:WU00:FS01:Upload 8.50%
15:35:04:ERROR:WU00:FS01:Exception: Transfer failed

SLOT #01

*********************** Log Started 2023-01-05T04:24:22Z ***********************
...
09:31:14:WU01:FS01:Uploading 33.03MiB to 129.32.209.202
09:31:14:WU01:FS01:Connecting to 129.32.209.202:8080
09:31:20:WU01:FS01:Upload 1.51%
09:31:26:WU01:FS01:Upload 2.65%
09:32:17:WU01:FS01:Upload 3.03%
09:32:17:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
09:32:17:WU01:FS01:Trying to send results to collection server
09:32:17:WU01:FS01:Uploading 33.03MiB to 129.32.209.200
09:32:17:WU01:FS01:Connecting to 129.32.209.200:8080
09:32:23:WU01:FS01:Upload 1.51%
09:32:29:WU01:FS01:Upload 3.03%
09:32:39:WU01:FS01:Upload 4.54%
09:34:19:WU01:FS01:Upload 5.87%
09:34:19:ERROR:WU01:FS01:Exception: Transfer failed
******************************* Date: 2023-01-06 *******************************
10:31:14:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:4 clone:369 gen:91 core:0x22 unit:0x000001710000005b0000481100000004
10:31:14:WU01:FS01:Uploading 33.03MiB to 129.32.209.202
10:31:14:WU01:FS01:Connecting to 129.32.209.202:8080
10:31:20:WU01:FS01:Upload 2.08%
10:31:26:WU01:FS01:Upload 3.97%
10:31:32:WU01:FS01:Upload 7.76%
10:31:42:WU01:FS01:Upload 7.95%
10:31:48:WU01:FS01:Upload 9.46%
10:31:57:WU01:FS01:Upload 10.41%
10:32:03:WU01:FS01:Upload 12.30%
10:32:12:WU01:FS01:Upload 13.44%
10:34:49:WU01:FS01:Upload 14.76%
10:36:26:WU01:FS01:Upload 14.95%
10:36:32:WU01:FS01:Upload 16.84%
10:36:38:WU01:FS01:Upload 18.36%
10:36:51:WU01:FS01:Upload 18.92%
10:36:57:WU01:FS01:Upload 20.44%
10:37:06:WU01:FS01:Upload 21.76%
10:39:18:WU01:FS01:Upload 23.09%
10:42:44:WU01:FS01:Upload 23.28%
10:43:09:WU01:FS01:Upload 23.47%
10:43:15:WU01:FS01:Upload 25.17%
10:43:21:WU01:FS01:Upload 26.87%
10:43:34:WU01:FS01:Upload 27.25%
10:43:42:WU01:FS01:Upload 27.44%
10:43:53:WU01:FS01:Upload 27.63%
10:43:59:WU01:FS01:Upload 29.33%
10:44:09:WU01:FS01:Upload 30.66%
10:44:29:WU01:FS01:Upload 30.85%
10:44:35:WU01:FS01:Upload 32.55%
10:44:43:WU01:FS01:Upload 33.69%
10:47:51:WU01:FS01:Upload 35.20%
10:51:49:WU01:FS01:Upload 35.39%
10:52:19:WU01:FS01:Upload 35.58%
10:52:19:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
10:52:19:WU01:FS01:Trying to send results to collection server
10:52:19:WU01:FS01:Uploading 33.03MiB to 129.32.209.200
10:52:19:WU01:FS01:Connecting to 129.32.209.200:8080
10:52:25:WU01:FS01:Upload 1.70%
10:52:31:WU01:FS01:Upload 3.78%
10:52:37:WU01:FS01:Upload 7.76%
10:52:51:WU01:FS01:Upload 7.95%
10:52:58:WU01:FS01:Upload 8.52%
10:53:18:WU01:FS01:Upload 8.89%
10:56:13:WU01:FS01:Upload 10.41%
10:56:13:ERROR:WU01:FS01:Exception: Transfer failed
11:31:14:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:4 clone:369 gen:91 core:0x22 unit:0x000001710000005b0000481100000004
11:31:14:WU01:FS01:Uploading 33.03MiB to 129.32.209.202
11:31:14:WU01:FS01:Connecting to 129.32.209.202:8080
11:31:20:WU01:FS01:Upload 2.08%
11:31:26:WU01:FS01:Upload 3.97%
11:31:41:WU01:FS01:Upload 7.95%
11:31:47:WU01:FS01:Upload 9.46%
11:31:56:WU01:FS01:Upload 10.60%
11:32:02:WU01:FS01:Upload 12.30%
11:32:11:WU01:FS01:Upload 13.44%
11:32:33:WU01:FS01:Upload 14.76%
11:35:45:WU01:FS01:Upload 14.95%
11:36:09:WU01:FS01:Upload 15.14%
11:36:15:WU01:FS01:Upload 17.03%
11:36:21:WU01:FS01:Upload 18.55%
11:36:27:WU01:FS01:Upload 19.11%
11:36:33:WU01:FS01:Upload 20.63%
11:38:27:WU01:FS01:Upload 21.01%
11:41:24:WU01:FS01:Upload 21.20%
11:44:30:WU01:FS01:Upload 21.39%
11:44:41:WU01:FS01:Upload 21.57%
11:44:47:WU01:FS01:Upload 22.90%
11:46:25:WU01:FS01:Upload 23.09%
11:49:10:WU01:FS01:Upload 23.28%
11:51:15:WU01:FS01:Upload 23.47%
11:51:21:WU01:FS01:Upload 23.85%
11:52:16:WU01:FS01:Upload 24.98%
11:56:01:WU01:FS01:Upload 25.17%
11:56:31:WU01:FS01:Upload 25.36%
11:56:31:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
11:56:31:WU01:FS01:Trying to send results to collection server
11:56:31:WU01:FS01:Uploading 33.03MiB to 129.32.209.200
11:56:31:WU01:FS01:Connecting to 129.32.209.200:8080
11:56:37:WU01:FS01:Upload 0.57%
11:56:50:WU01:FS01:Upload 1.51%
11:56:56:WU01:FS01:Upload 3.22%
11:57:02:WU01:FS01:Upload 5.87%
11:57:10:WU01:FS01:Upload 8.33%
11:57:25:WU01:FS01:Upload 8.52%
11:57:35:WU01:FS01:Upload 8.71%
11:57:41:WU01:FS01:Upload 10.41%
11:58:06:WU01:FS01:Upload 11.73%
11:58:12:WU01:FS01:Upload 11.92%
11:58:18:WU01:FS01:Upload 13.63%
11:58:37:WU01:FS01:Upload 14.95%
11:58:57:WU01:FS01:Upload 15.14%
11:59:03:WU01:FS01:Upload 15.90%
11:59:42:WU01:FS01:Upload 16.28%
11:59:48:WU01:FS01:Upload 17.03%
12:00:41:WU01:FS01:Upload 17.22%
12:00:47:WU01:FS01:Upload 18.74%
12:00:57:WU01:FS01:Upload 19.87%
12:02:48:WU01:FS01:Upload 21.39%
12:02:48:ERROR:WU01:FS01:Exception: Transfer failed
12:31:14:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:4 clone:369 gen:91 core:0x22 unit:0x000001710000005b0000481100000004
12:31:14:WU01:FS01:Uploading 33.03MiB to 129.32.209.202
12:31:14:WU01:FS01:Connecting to 129.32.209.202:8080
12:31:20:WU01:FS01:Upload 1.89%
12:31:27:WU01:FS01:Upload 3.60%
12:31:33:WU01:FS01:Upload 5.49%
12:31:39:WU01:FS01:Upload 7.95%
12:31:53:WU01:FS01:Upload 10.60%
12:31:59:WU01:FS01:Upload 12.30%
12:32:07:WU01:FS01:Upload 13.06%
12:32:13:WU01:FS01:Upload 14.76%
12:32:21:WU01:FS01:Upload 15.90%
12:34:31:WU01:FS01:Upload 17.22%
12:35:41:WU01:FS01:Upload 17.41%
12:37:46:WU01:FS01:Upload 18.55%
12:38:41:WU01:FS01:Upload 19.68%
12:38:47:WU01:FS01:Upload 21.39%
12:38:53:WU01:FS01:Upload 23.09%
12:39:00:WU01:FS01:Upload 23.85%
12:39:06:WU01:FS01:Upload 25.55%
12:40:26:WU01:FS01:Upload 25.74%
12:43:48:WU01:FS01:Upload 25.93%
12:46:59:WU01:FS01:Upload 26.12%
12:49:52:WU01:FS01:Upload 26.31%
12:49:58:WU01:FS01:Upload 27.63%
12:50:17:WU01:FS01:Upload 28.01%
12:53:10:WU01:FS01:Upload 28.20%
12:55:00:WU01:FS01:Upload 28.39%
12:55:01:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
12:55:01:WU01:FS01:Trying to send results to collection server
12:55:01:WU01:FS01:Uploading 33.03MiB to 129.32.209.200
12:55:01:WU01:FS01:Connecting to 129.32.209.200:8080
12:55:13:WU01:FS01:Upload 0.38%
12:55:19:WU01:FS01:Upload 2.27%
12:55:25:WU01:FS01:Upload 4.73%
12:55:54:WU01:FS01:Upload 7.00%
12:56:20:WU01:FS01:Upload 7.19%
12:56:26:WU01:FS01:Upload 8.14%
12:56:35:WU01:FS01:Upload 8.71%
12:56:41:WU01:FS01:Upload 10.22%
12:56:51:WU01:FS01:Upload 11.73%
12:57:42:WU01:FS01:Upload 13.25%
13:01:02:WU01:FS01:Upload 13.44%
13:02:18:WU01:FS01:Upload 13.63%
13:02:53:WU01:FS01:Upload 14.76%
13:02:53:ERROR:WU01:FS01:Exception: Transfer failed
13:31:14:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:4 clone:369 gen:91 core:0x22 unit:0x000001710000005b0000481100000004
13:31:14:WU01:FS01:Uploading 33.03MiB to 129.32.209.202
13:31:14:WU01:FS01:Connecting to 129.32.209.202:8080
13:31:20:WU01:FS01:Upload 1.89%
13:31:26:WU01:FS01:Upload 3.97%
13:31:42:WU01:FS01:Upload 7.95%
13:31:48:WU01:FS01:Upload 9.65%
13:31:57:WU01:FS01:Upload 10.60%
13:32:03:WU01:FS01:Upload 11.54%
13:32:26:WU01:FS01:Upload 12.49%
13:32:45:WU01:FS01:Upload 12.68%
13:32:53:WU01:FS01:Upload 14.00%
13:33:29:WU01:FS01:Upload 14.19%
13:33:35:WU01:FS01:Upload 15.33%
13:33:42:WU01:FS01:Upload 15.90%
13:34:13:WU01:FS01:Upload 16.46%
13:34:19:WU01:FS01:Upload 17.60%
13:35:23:WU01:FS01:Upload 17.79%
13:35:47:WU01:FS01:Upload 18.92%
13:36:00:WU01:FS01:Upload 19.11%
13:36:12:WU01:FS01:Upload 20.25%
13:39:10:WU01:FS01:Upload 21.20%
13:42:35:WU01:FS01:Upload 21.39%
13:43:45:WU01:FS01:Upload 21.57%
13:44:37:WU01:FS01:Upload 22.90%
13:44:37:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
13:44:37:WU01:FS01:Trying to send results to collection server
13:44:37:WU01:FS01:Uploading 33.03MiB to 129.32.209.200
13:44:37:WU01:FS01:Connecting to 129.32.209.200:8080
13:44:44:WU01:FS01:Upload 0.38%
13:44:50:WU01:FS01:Upload 0.95%
13:44:56:WU01:FS01:Upload 3.03%
13:45:02:WU01:FS01:Upload 6.81%
13:45:12:WU01:FS01:Upload 7.00%
13:45:18:WU01:FS01:Upload 8.71%
13:45:26:WU01:FS01:Upload 9.46%
13:45:32:WU01:FS01:Upload 11.17%
13:45:42:WU01:FS01:Upload 12.68%
13:49:00:WU01:FS01:Upload 14.19%
13:50:33:WU01:FS01:Upload 14.38%
13:50:39:WU01:FS01:Upload 16.09%
13:50:45:WU01:FS01:Upload 17.60%
13:50:52:WU01:FS01:Upload 18.36%
13:50:58:WU01:FS01:Upload 18.55%
13:51:04:WU01:FS01:Upload 20.25%
13:51:12:WU01:FS01:Upload 21.57%
13:51:18:WU01:FS01:Upload 23.47%
13:51:27:WU01:FS01:Upload 24.79%
13:52:13:WU01:FS01:Upload 25.93%
13:54:04:WU01:FS01:Upload 26.12%
13:54:48:WU01:FS01:Upload 27.06%
13:54:49:ERROR:WU01:FS01:Exception: Transfer failed
14:31:14:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:4 clone:369 gen:91 core:0x22 unit:0x000001710000005b0000481100000004
14:31:14:WU01:FS01:Uploading 33.03MiB to 129.32.209.202
14:31:14:WU01:FS01:Connecting to 129.32.209.202:8080
14:31:31:WU01:FS01:Upload 0.38%
14:31:38:WU01:FS01:Upload 0.57%
14:31:44:WU01:FS01:Upload 0.76%
14:31:50:WU01:FS01:Upload 0.95%
14:32:02:WU01:FS01:Upload 3.03%
14:32:08:WU01:FS01:Upload 4.73%
14:32:17:WU01:FS01:Upload 5.87%
14:32:41:WU01:FS01:Upload 6.25%
14:32:47:WU01:FS01:Upload 7.76%
14:32:53:WU01:FS01:Upload 9.27%
14:32:59:WU01:FS01:Upload 10.79%
14:33:05:WU01:FS01:Upload 14.00%
14:33:15:WU01:FS01:Upload 14.19%
14:33:21:WU01:FS01:Upload 15.71%
14:33:29:WU01:FS01:Upload 16.84%
14:35:59:WU01:FS01:Upload 18.36%
14:37:46:WU01:FS01:Upload 18.55%
14:39:38:WU01:FS01:Upload 19.68%
14:41:06:WU01:FS01:Upload 19.87%
14:43:40:WU01:FS01:Upload 21.01%
14:43:40:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
14:43:40:WU01:FS01:Trying to send results to collection server
14:43:40:WU01:FS01:Uploading 33.03MiB to 129.32.209.200
14:43:40:WU01:FS01:Connecting to 129.32.209.200:8080
14:43:46:WU01:FS01:Upload 0.76%
14:43:52:WU01:FS01:Upload 2.27%
14:43:58:WU01:FS01:Upload 4.92%
14:44:13:WU01:FS01:Upload 7.57%
14:44:24:WU01:FS01:Upload 7.76%
14:44:30:WU01:FS01:Upload 9.27%
14:44:56:WU01:FS01:Upload 10.60%
14:45:02:WU01:FS01:Upload 12.11%
14:45:11:WU01:FS01:Upload 13.25%
14:47:53:WU01:FS01:Upload 14.57%
14:49:17:WU01:FS01:Upload 14.76%
14:52:20:WU01:FS01:Upload 15.90%
14:52:32:WU01:FS01:Upload 16.09%
14:54:09:WU01:FS01:Upload 17.03%
14:54:39:WU01:FS01:Upload 17.22%
14:54:39:ERROR:WU01:FS01:Exception: Transfer failed
15:31:14:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:4 clone:369 gen:91 core:0x22 unit:0x000001710000005b0000481100000004
15:31:14:WU01:FS01:Uploading 33.03MiB to 129.32.209.202
15:31:14:WU01:FS01:Connecting to 129.32.209.202:8080
15:31:45:WU01:FS01:Upload 0.57%
15:31:46:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
15:31:46:WU01:FS01:Trying to send results to collection server
15:31:46:WU01:FS01:Uploading 33.03MiB to 129.32.209.200
15:31:46:WU01:FS01:Connecting to 129.32.209.200:8080
15:31:53:WU01:FS01:Upload 0.38%
15:31:59:WU01:FS01:Upload 2.27%
15:32:05:WU01:FS01:Upload 4.16%
15:32:19:WU01:FS01:Upload 7.57%
15:32:25:WU01:FS01:Upload 9.08%
15:32:34:WU01:FS01:Upload 10.22%
15:33:09:WU01:FS01:Upload 11.73%
15:33:09:ERROR:WU01:FS01:Exception: Transfer failed

SLOT #02

*********************** Log Started 2023-01-05T04:24:22Z ***********************
...
09:53:38:WU02:FS01:Uploading 21.05MiB to 129.32.209.202
09:53:38:WU02:FS01:Connecting to 129.32.209.202:8080
09:53:44:WU02:FS01:Upload 2.97%
09:53:50:WU02:FS01:Upload 6.24%
09:54:06:WU02:FS01:Upload 12.47%
09:54:12:WU02:FS01:Upload 14.85%
09:54:21:WU02:FS01:Upload 16.33%
09:54:27:WU02:FS01:Upload 19.01%
09:54:35:WU02:FS01:Upload 20.79%
09:55:06:WU02:FS01:Upload 22.87%
09:55:36:WU02:FS01:Upload 23.16%
09:55:36:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
09:55:36:WU02:FS01:Trying to send results to collection server
09:55:36:WU02:FS01:Uploading 21.05MiB to 129.32.209.207
09:55:36:WU02:FS01:Connecting to 129.32.209.207:8080
09:55:50:WU02:FS01:Upload 0.59%
09:55:56:WU02:FS01:Upload 3.27%
09:56:02:WU02:FS01:Upload 6.53%
09:56:13:WU02:FS01:Upload 12.77%
09:56:19:WU02:FS01:Upload 13.07%
09:56:25:WU02:FS01:Upload 15.74%
09:56:34:WU02:FS01:Upload 16.93%
09:58:22:WU02:FS01:Upload 18.71%
10:01:30:WU02:FS01:Upload 19.01%
10:04:38:WU02:FS01:Upload 19.30%
10:05:36:WU02:FS01:Upload 19.60%
10:07:31:WU02:FS01:Upload 21.68%
10:10:35:WU02:FS01:Upload 21.97%
10:13:45:WU02:FS01:Upload 22.27%
10:16:57:WU02:FS01:Upload 22.57%
10:17:32:WU02:FS01:Upload 22.87%
10:17:38:WU02:FS01:Upload 23.16%
10:19:49:WU02:FS01:Upload 24.94%
10:23:07:WU02:FS01:Upload 25.24%
10:26:24:WU02:FS01:Upload 25.54%
******************************* Date: 2023-01-06 *******************************
10:29:51:WU02:FS01:Upload 25.84%
10:32:33:WU02:FS01:Upload 26.13%
10:32:33:ERROR:WU02:FS01:Exception: Transfer failed
10:53:38:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18447 run:4 clone:103 gen:525 core:0x22 unit:0x000000670000020d0000480f00000004
10:53:38:WU02:FS01:Uploading 21.05MiB to 129.32.209.202
10:53:38:WU02:FS01:Connecting to 129.32.209.202:8080
10:53:46:WU02:FS01:Upload 0.59%
10:53:55:WU02:FS01:Upload 2.38%
10:54:29:WU02:FS01:Upload 2.97%
10:54:29:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
10:54:29:WU02:FS01:Trying to send results to collection server
10:54:29:WU02:FS01:Uploading 21.05MiB to 129.32.209.207
10:54:29:WU02:FS01:Connecting to 129.32.209.207:8080
10:54:45:WU02:FS01:Upload 0.89%
10:54:52:WU02:FS01:Upload 1.48%
10:54:58:WU02:FS01:Upload 5.64%
10:55:11:WU02:FS01:Upload 9.50%
10:55:18:WU02:FS01:Upload 10.99%
10:55:24:WU02:FS01:Upload 13.66%
10:55:30:WU02:FS01:Upload 16.04%
10:55:36:WU02:FS01:Upload 16.93%
10:55:42:WU02:FS01:Upload 19.30%
10:58:40:WU02:FS01:Upload 19.60%
11:02:10:WU02:FS01:Upload 19.90%
11:02:40:WU02:FS01:Upload 20.19%
11:02:40:ERROR:WU02:FS01:Exception: Transfer failed
11:53:38:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18447 run:4 clone:103 gen:525 core:0x22 unit:0x000000670000020d0000480f00000004
11:53:38:WU02:FS01:Uploading 21.05MiB to 129.32.209.202
11:53:38:WU02:FS01:Connecting to 129.32.209.202:8080
11:53:49:WU02:FS01:Upload 0.59%
11:53:57:WU02:FS01:Upload 1.19%
11:55:00:WU02:FS01:Upload 2.38%
11:55:06:WU02:FS01:Upload 5.05%
11:55:12:WU02:FS01:Upload 8.61%
11:55:23:WU02:FS01:Upload 11.88%
11:55:29:WU02:FS01:Upload 14.55%
11:55:37:WU02:FS01:Upload 16.04%
11:55:43:WU02:FS01:Upload 18.41%
11:55:53:WU02:FS01:Upload 21.08%
11:58:41:WU02:FS01:Upload 23.16%
11:58:41:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
11:58:41:WU02:FS01:Trying to send results to collection server
11:58:41:WU02:FS01:Uploading 21.05MiB to 129.32.209.207
11:58:41:WU02:FS01:Connecting to 129.32.209.207:8080
11:58:50:WU02:FS01:Upload 0.59%
11:58:56:WU02:FS01:Upload 2.97%
11:59:03:WU02:FS01:Upload 5.35%
11:59:14:WU02:FS01:Upload 6.83%
11:59:20:WU02:FS01:Upload 9.50%
11:59:26:WU02:FS01:Upload 10.69%
12:00:13:WU02:FS01:Upload 11.58%
12:00:13:ERROR:WU02:FS01:Exception: Transfer failed
12:53:38:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18447 run:4 clone:103 gen:525 core:0x22 unit:0x000000670000020d0000480f00000004
12:53:38:WU02:FS01:Uploading 21.05MiB to 129.32.209.202
12:53:38:WU02:FS01:Connecting to 129.32.209.202:8080
12:53:44:WU02:FS01:Upload 2.67%
12:53:50:WU02:FS01:Upload 6.24%
12:54:00:WU02:FS01:Upload 12.18%
12:54:06:WU02:FS01:Upload 12.47%
12:54:12:WU02:FS01:Upload 13.96%
12:54:20:WU02:FS01:Upload 14.25%
12:54:26:WU02:FS01:Upload 16.93%
12:54:32:WU02:FS01:Upload 19.30%
12:54:38:WU02:FS01:Upload 20.19%
12:55:28:WU02:FS01:Upload 22.57%
12:55:28:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
12:55:28:WU02:FS01:Trying to send results to collection server
12:55:28:WU02:FS01:Uploading 21.05MiB to 129.32.209.207
12:55:28:WU02:FS01:Connecting to 129.32.209.207:8080
12:55:34:WU02:FS01:Upload 0.59%
12:55:41:WU02:FS01:Upload 1.78%
12:55:48:WU02:FS01:Upload 2.97%
12:56:16:WU02:FS01:Upload 3.56%
12:56:22:WU02:FS01:Upload 5.94%
12:56:28:WU02:FS01:Upload 8.02%
12:56:44:WU02:FS01:Upload 8.31%
12:57:00:WU02:FS01:Upload 8.61%
12:57:06:WU02:FS01:Upload 11.28%
12:57:13:WU02:FS01:Upload 12.47%
12:57:19:WU02:FS01:Upload 15.14%
12:57:25:WU02:FS01:Upload 17.82%
12:57:33:WU02:FS01:Upload 21.68%
12:57:46:WU02:FS01:Upload 21.97%
12:57:52:WU02:FS01:Upload 24.35%
12:58:01:WU02:FS01:Upload 26.43%
12:59:02:WU02:FS01:Upload 28.51%
13:01:53:WU02:FS01:Upload 28.80%
13:01:59:WU02:FS01:Upload 31.77%
13:02:05:WU02:FS01:Upload 34.45%
13:02:12:WU02:FS01:Upload 35.34%
13:02:21:WU02:FS01:Upload 35.63%
13:02:27:WU02:FS01:Upload 36.53%
13:02:33:WU02:FS01:Upload 39.20%
13:02:47:WU02:FS01:Upload 40.98%
13:02:53:WU02:FS01:Upload 43.65%
13:02:59:WU02:FS01:Upload 46.33%
13:03:05:WU02:FS01:Upload 47.22%
13:06:14:WU02:FS01:Upload 49.29%
13:09:00:WU02:FS01:Upload 49.59%
13:12:23:WU02:FS01:Upload 51.97%
13:13:27:WU02:FS01:Upload 52.26%
13:15:33:WU02:FS01:Upload 54.34%
13:17:15:WU02:FS01:Upload 54.64%
13:17:21:WU02:FS01:Upload 57.31%
13:17:27:WU02:FS01:Upload 59.99%
13:17:41:WU02:FS01:Upload 60.58%
13:17:47:WU02:FS01:Upload 62.95%
13:17:55:WU02:FS01:Upload 64.74%
13:20:41:WU02:FS01:Upload 67.11%
13:24:24:WU02:FS01:Upload 67.41%
13:26:11:WU02:FS01:Upload 67.71%
13:27:44:WU02:FS01:Upload 69.78%
13:27:44:ERROR:WU02:FS01:Exception: Transfer failed
13:53:38:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18447 run:4 clone:103 gen:525 core:0x22 unit:0x000000670000020d0000480f00000004
13:53:38:WU02:FS01:Uploading 21.05MiB to 129.32.209.202
13:53:38:WU02:FS01:Connecting to 129.32.209.202:8080
13:53:44:WU02:FS01:Upload 2.67%
13:53:50:WU02:FS01:Upload 6.24%
13:53:56:WU02:FS01:Upload 12.18%
13:54:11:WU02:FS01:Upload 12.47%
13:54:17:WU02:FS01:Upload 15.14%
13:54:25:WU02:FS01:Upload 16.63%
13:54:31:WU02:FS01:Upload 19.30%
13:54:40:WU02:FS01:Upload 21.08%
13:56:25:WU02:FS01:Upload 23.16%
13:57:56:WU02:FS01:Upload 23.46%
14:00:24:WU02:FS01:Upload 25.24%
14:00:46:WU02:FS01:Upload 25.54%
14:01:17:WU02:FS01:Upload 27.02%
14:03:04:WU02:FS01:Upload 27.32%
14:04:51:WU02:FS01:Upload 29.10%
14:05:01:WU02:FS01:Upload 30.88%
14:05:38:WU02:FS01:Upload 31.18%
14:05:44:WU02:FS01:Upload 33.85%
14:05:50:WU02:FS01:Upload 36.23%
14:05:57:WU02:FS01:Upload 37.42%
14:06:03:WU02:FS01:Upload 40.09%
14:06:45:WU02:FS01:Upload 40.39%
14:07:16:WU02:FS01:Upload 40.68%
14:07:16:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
14:07:16:WU02:FS01:Trying to send results to collection server
14:07:16:WU02:FS01:Uploading 21.05MiB to 129.32.209.207
14:07:16:WU02:FS01:Connecting to 129.32.209.207:8080
14:07:22:WU02:FS01:Upload 3.27%
14:07:28:WU02:FS01:Upload 6.24%
14:07:44:WU02:FS01:Upload 12.47%
14:07:50:WU02:FS01:Upload 14.85%
14:07:59:WU02:FS01:Upload 16.63%
14:08:05:WU02:FS01:Upload 19.30%
14:08:13:WU02:FS01:Upload 21.08%
14:11:26:WU02:FS01:Upload 23.46%
14:11:49:WU02:FS01:Upload 23.76%
14:12:21:WU02:FS01:Upload 25.24%
14:14:59:WU02:FS01:Upload 25.54%
14:16:17:WU02:FS01:Upload 27.32%
14:18:10:WU02:FS01:Upload 27.62%
14:18:16:WU02:FS01:Upload 30.59%
14:18:22:WU02:FS01:Upload 32.96%
14:18:28:WU02:FS01:Upload 33.56%
14:18:35:WU02:FS01:Upload 33.85%
14:18:41:WU02:FS01:Upload 36.23%
14:18:55:WU02:FS01:Upload 38.01%
14:19:01:WU02:FS01:Upload 40.39%
14:19:10:WU02:FS01:Upload 42.46%
14:23:08:WU02:FS01:Upload 44.84%
14:24:09:WU02:FS01:Upload 45.14%
14:24:09:ERROR:WU02:FS01:Exception: Transfer failed
14:53:38:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18447 run:4 clone:103 gen:525 core:0x22 unit:0x000000670000020d0000480f00000004
14:53:38:WU02:FS01:Uploading 21.05MiB to 129.32.209.202
14:53:38:WU02:FS01:Connecting to 129.32.209.202:8080
14:54:10:WU02:FS01:Upload 0.89%
14:54:16:WU02:FS01:Upload 3.86%
14:54:22:WU02:FS01:Upload 7.72%
14:54:44:WU02:FS01:Upload 10.99%
14:54:50:WU02:FS01:Upload 13.07%
14:54:58:WU02:FS01:Upload 14.85%
14:55:40:WU02:FS01:Upload 16.93%
14:58:21:WU02:FS01:Upload 17.22%
14:58:54:WU02:FS01:Upload 19.01%
15:00:26:WU02:FS01:Upload 19.30%
15:00:32:WU02:FS01:Upload 20.49%
15:01:28:WU02:FS01:Upload 21.08%
15:01:36:WU02:FS01:Upload 23.16%
15:02:53:WU02:FS01:Upload 24.94%
15:02:59:WU02:FS01:Upload 27.62%
15:03:05:WU02:FS01:Upload 30.29%
15:03:20:WU02:FS01:Upload 31.77%
15:03:32:WU02:FS01:Upload 33.85%
15:03:38:WU02:FS01:Upload 36.82%
15:03:44:WU02:FS01:Upload 39.20%
15:03:56:WU02:FS01:Upload 40.39%
15:04:02:WU02:FS01:Upload 42.76%
15:04:11:WU02:FS01:Upload 44.84%
15:04:38:WU02:FS01:Upload 46.62%
15:07:03:WU02:FS01:Upload 46.92%
15:09:27:WU02:FS01:Upload 47.22%
15:11:52:WU02:FS01:Upload 49.29%
15:12:46:WU02:FS01:Upload 49.59%
15:14:25:WU02:FS01:Upload 51.37%
15:14:52:WU02:FS01:Upload 51.67%
15:16:23:WU02:FS01:Upload 53.45%
15:16:29:WU02:FS01:Upload 56.42%
15:16:35:WU02:FS01:Upload 59.09%
15:16:48:WU02:FS01:Upload 59.99%
15:16:54:WU02:FS01:Upload 62.36%
15:17:03:WU02:FS01:Upload 64.44%
15:17:09:WU02:FS01:Upload 65.03%
15:17:15:WU02:FS01:Upload 67.41%
15:17:21:WU02:FS01:Upload 70.08%
15:17:27:WU02:FS01:Upload 70.97%
15:17:38:WU02:FS01:Upload 73.35%
15:20:33:WU02:FS01:Upload 73.65%
15:24:09:WU02:FS01:Upload 73.94%
15:24:27:WU02:FS01:Upload 74.24%
15:24:33:WU02:FS01:Upload 74.54%
15:26:40:WU02:FS01:Upload 76.32%
15:30:17:WU02:FS01:Upload 76.61%
15:30:23:WU02:FS01:Upload 78.69%
15:33:12:WU02:FS01:Upload 78.99%
15:33:12:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
15:33:12:WU02:FS01:Trying to send results to collection server
15:33:12:WU02:FS01:Uploading 21.05MiB to 129.32.209.207
15:33:12:WU02:FS01:Connecting to 129.32.209.207:8080
15:33:19:WU02:FS01:Upload 0.59%
15:33:25:WU02:FS01:Upload 1.78%
15:33:32:WU02:FS01:Upload 3.56%
15:35:06:WU02:FS01:Upload 5.05%
15:35:06:ERROR:WU02:FS01:Exception: Transfer failed
15:53:38:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18447 run:4 clone:103 gen:525 core:0x22 unit:0x000000670000020d0000480f00000004
15:53:38:WU02:FS01:Uploading 21.05MiB to 129.32.209.202
15:53:38:WU02:FS01:Connecting to 129.32.209.202:8080
15:53:44:WU02:FS01:Upload 2.67%
15:53:50:WU02:FS01:Upload 5.94%
15:54:00:WU02:FS01:Upload 12.18%
15:54:06:WU02:FS01:Upload 12.77%
15:54:12:WU02:FS01:Upload 15.44%
15:54:23:WU02:FS01:Upload 16.04%
15:55:29:WU02:FS01:Upload 17.82%
15:57:54:WU02:FS01:Upload 18.11%
16:00:28:WU02:FS01:Upload 18.41%
16:00:46:WU02:FS01:Upload 18.71%
16:00:52:WU02:FS01:Upload 21.38%
16:01:00:WU02:FS01:Upload 23.16%
16:01:55:WU02:FS01:Upload 25.24%

SLOT #03

*********************** Log Started 2023-01-05T04:24:22Z ***********************
...
09:31:36:WU03:FS01:Trying to send results to collection server
09:31:36:WU03:FS01:Uploading 29.48MiB to 129.32.209.204
09:31:36:WU03:FS01:Connecting to 129.32.209.204:8080
09:31:42:WU03:FS01:Upload 1.70%
09:31:48:WU03:FS01:Upload 4.24%
09:31:54:WU03:FS01:Upload 8.69%
09:32:05:WU03:FS01:Upload 8.90%
09:32:11:WU03:FS01:Upload 10.60%
09:32:25:WU03:FS01:Upload 11.87%
09:32:34:WU03:FS01:Upload 12.93%
09:32:46:WU03:FS01:Upload 13.14%
09:32:52:WU03:FS01:Upload 13.57%
09:32:59:WU03:FS01:Upload 14.42%
09:33:19:WU03:FS01:Upload 14.63%
09:33:30:WU03:FS01:Upload 14.84%
09:33:36:WU03:FS01:Upload 16.75%
09:33:42:WU03:FS01:Upload 18.44%
09:34:11:WU03:FS01:Upload 18.87%
09:34:17:WU03:FS01:Upload 19.50%
09:34:23:WU03:FS01:Upload 21.41%
09:34:30:WU03:FS01:Upload 21.62%
09:34:36:WU03:FS01:Upload 23.11%
09:34:47:WU03:FS01:Upload 24.38%
09:34:53:WU03:FS01:Upload 26.07%
09:35:00:WU03:FS01:Upload 26.71%
09:35:27:WU03:FS01:Upload 27.13%
09:35:33:WU03:FS01:Upload 27.56%
09:37:10:WU03:FS01:Upload 27.77%
09:40:20:WU03:FS01:Upload 27.98%
09:43:06:WU03:FS01:Upload 28.19%
09:43:06:ERROR:WU03:FS01:Exception: Transfer failed
******************************* Date: 2023-01-06 *******************************
10:26:48:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:18448 run:6 clone:271 gen:145 core:0x22 unit:0x0000010f000000910000481000000006
10:26:48:WU03:FS01:Uploading 29.48MiB to 129.32.209.202
10:26:48:WU03:FS01:Connecting to 129.32.209.202:8080
10:26:54:WU03:FS01:Upload 1.91%
10:27:00:WU03:FS01:Upload 4.24%
10:27:06:WU03:FS01:Upload 8.69%
10:27:16:WU03:FS01:Upload 8.90%
10:27:22:WU03:FS01:Upload 10.60%
10:27:31:WU03:FS01:Upload 11.66%
10:27:37:WU03:FS01:Upload 13.78%
10:27:45:WU03:FS01:Upload 14.84%
10:30:43:WU03:FS01:Upload 16.54%
10:32:37:WU03:FS01:Upload 16.75%
10:32:37:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
10:32:37:WU03:FS01:Trying to send results to collection server
10:32:37:WU03:FS01:Uploading 29.48MiB to 129.32.209.204
10:32:37:WU03:FS01:Connecting to 129.32.209.204:8080
10:32:43:WU03:FS01:Upload 0.42%
10:32:49:WU03:FS01:Upload 2.33%
10:32:55:WU03:FS01:Upload 4.45%
10:33:05:WU03:FS01:Upload 8.69%
10:33:11:WU03:FS01:Upload 10.39%
10:34:06:WU03:FS01:Upload 10.60%
10:37:06:WU03:FS01:Upload 10.81%
10:40:14:WU03:FS01:Upload 11.02%
10:40:14:ERROR:WU03:FS01:Exception: Transfer failed
11:26:48:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:18448 run:6 clone:271 gen:145 core:0x22 unit:0x0000010f000000910000481000000006
11:26:49:WU03:FS01:Uploading 29.48MiB to 129.32.209.202
11:26:49:WU03:FS01:Connecting to 129.32.209.202:8080
11:26:55:WU03:FS01:Upload 2.33%
11:27:01:WU03:FS01:Upload 4.03%
11:27:07:WU03:FS01:Upload 6.15%
11:27:13:WU03:FS01:Upload 8.69%
11:27:28:WU03:FS01:Upload 11.66%
11:27:34:WU03:FS01:Upload 13.36%
11:27:44:WU03:FS01:Upload 14.42%
11:27:50:WU03:FS01:Upload 16.32%
11:27:58:WU03:FS01:Upload 17.38%
11:28:05:WU03:FS01:Upload 17.81%
11:28:11:WU03:FS01:Upload 19.72%
11:28:17:WU03:FS01:Upload 21.62%
11:28:31:WU03:FS01:Upload 22.47%
11:28:37:WU03:FS01:Upload 24.17%
11:28:46:WU03:FS01:Upload 25.44%
11:29:28:WU03:FS01:Upload 27.13%
11:29:28:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
11:29:28:WU03:FS01:Trying to send results to collection server
11:29:28:WU03:FS01:Uploading 29.48MiB to 129.32.209.204
11:29:28:WU03:FS01:Connecting to 129.32.209.204:8080
11:29:47:WU03:FS01:Upload 0.42%
11:29:53:WU03:FS01:Upload 2.12%
11:29:59:WU03:FS01:Upload 5.09%
11:30:11:WU03:FS01:Upload 7.63%
11:30:17:WU03:FS01:Upload 9.54%
11:30:25:WU03:FS01:Upload 10.60%
11:30:31:WU03:FS01:Upload 12.30%
11:30:41:WU03:FS01:Upload 13.99%
11:32:34:WU03:FS01:Upload 15.48%
11:32:34:ERROR:WU03:FS01:Exception: Transfer failed
12:26:49:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:18448 run:6 clone:271 gen:145 core:0x22 unit:0x0000010f000000910000481000000006
12:26:49:WU03:FS01:Uploading 29.48MiB to 129.32.209.202
12:26:49:WU03:FS01:Connecting to 129.32.209.202:8080
12:26:55:WU03:FS01:Upload 2.33%
12:27:01:WU03:FS01:Upload 4.45%
12:27:11:WU03:FS01:Upload 8.90%
12:27:17:WU03:FS01:Upload 9.12%
12:27:23:WU03:FS01:Upload 11.02%
12:27:32:WU03:FS01:Upload 11.87%
12:27:38:WU03:FS01:Upload 12.72%
12:27:49:WU03:FS01:Upload 13.78%
12:29:02:WU03:FS01:Upload 14.84%
12:31:32:WU03:FS01:Upload 15.05%
12:32:22:WU03:FS01:Upload 15.26%
12:32:22:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
12:32:22:WU03:FS01:Trying to send results to collection server
12:32:22:WU03:FS01:Uploading 29.48MiB to 129.32.209.204
12:32:22:WU03:FS01:Connecting to 129.32.209.204:8080
12:32:29:WU03:FS01:Upload 0.42%
12:32:35:WU03:FS01:Upload 2.54%
12:32:41:WU03:FS01:Upload 4.88%
12:32:51:WU03:FS01:Upload 9.12%
12:32:57:WU03:FS01:Upload 9.33%
12:33:03:WU03:FS01:Upload 11.24%
12:33:11:WU03:FS01:Upload 12.08%
12:33:17:WU03:FS01:Upload 13.99%
12:33:25:WU03:FS01:Upload 15.26%
12:34:23:WU03:FS01:Upload 16.75%
12:37:11:WU03:FS01:Upload 16.96%
12:38:09:WU03:FS01:Upload 18.23%
12:39:57:WU03:FS01:Upload 18.44%
12:41:42:WU03:FS01:Upload 19.72%
12:41:48:WU03:FS01:Upload 21.62%
12:41:54:WU03:FS01:Upload 23.53%
12:42:09:WU03:FS01:Upload 24.38%
12:42:15:WU03:FS01:Upload 26.07%
12:42:23:WU03:FS01:Upload 27.35%
12:43:14:WU03:FS01:Upload 29.04%
12:44:38:WU03:FS01:Upload 29.25%
12:44:38:ERROR:WU03:FS01:Exception: Transfer failed
13:26:49:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:18448 run:6 clone:271 gen:145 core:0x22 unit:0x0000010f000000910000481000000006
13:26:49:WU03:FS01:Uploading 29.48MiB to 129.32.209.202
13:26:49:WU03:FS01:Connecting to 129.32.209.202:8080
13:26:55:WU03:FS01:Upload 2.33%
13:27:01:WU03:FS01:Upload 4.45%
13:27:11:WU03:FS01:Upload 8.90%
13:27:19:WU03:FS01:Upload 9.12%
13:27:25:WU03:FS01:Upload 10.39%
13:27:45:WU03:FS01:Upload 11.45%
13:27:51:WU03:FS01:Upload 13.14%
13:28:01:WU03:FS01:Upload 15.05%
13:29:51:WU03:FS01:Upload 16.75%
13:31:52:WU03:FS01:Upload 16.96%
13:31:52:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
13:31:52:WU03:FS01:Trying to send results to collection server
13:31:52:WU03:FS01:Uploading 29.48MiB to 129.32.209.204
13:31:52:WU03:FS01:Connecting to 129.32.209.204:8080
13:31:58:WU03:FS01:Upload 0.64%
13:32:11:WU03:FS01:Upload 1.70%
13:32:17:WU03:FS01:Upload 3.60%
13:32:23:WU03:FS01:Upload 6.57%
13:32:44:WU03:FS01:Upload 9.54%
13:32:51:WU03:FS01:Upload 10.18%
13:33:01:WU03:FS01:Upload 10.81%
13:33:07:WU03:FS01:Upload 12.72%
13:33:13:WU03:FS01:Upload 14.63%
13:33:29:WU03:FS01:Upload 15.05%
13:33:37:WU03:FS01:Upload 15.90%
13:33:45:WU03:FS01:Upload 16.11%
13:33:51:WU03:FS01:Upload 18.02%
13:33:57:WU03:FS01:Upload 19.72%
13:34:03:WU03:FS01:Upload 21.62%
13:34:22:WU03:FS01:Upload 23.32%
13:34:28:WU03:FS01:Upload 25.01%
13:34:37:WU03:FS01:Upload 26.50%
13:35:14:WU03:FS01:Upload 27.77%
13:38:13:WU03:FS01:Upload 27.98%
13:40:38:WU03:FS01:Upload 28.19%
13:40:51:WU03:FS01:Upload 28.41%
13:42:18:WU03:FS01:Upload 29.68%
13:44:24:WU03:FS01:Upload 29.89%
13:44:30:WU03:FS01:Upload 30.31%
13:45:37:WU03:FS01:Upload 31.80%
13:45:37:ERROR:WU03:FS01:Exception: Transfer failed
14:26:49:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:18448 run:6 clone:271 gen:145 core:0x22 unit:0x0000010f000000910000481000000006
14:26:49:WU03:FS01:Uploading 29.48MiB to 129.32.209.202
14:26:49:WU03:FS01:Connecting to 129.32.209.202:8080
14:27:18:WU03:FS01:Upload 0.64%
14:27:24:WU03:FS01:Upload 5.72%
14:27:35:WU03:FS01:Upload 5.94%
14:27:41:WU03:FS01:Upload 7.84%
14:27:59:WU03:FS01:Upload 8.48%
14:28:05:WU03:FS01:Upload 10.18%
14:28:17:WU03:FS01:Upload 11.24%
14:28:23:WU03:FS01:Upload 12.93%
14:28:29:WU03:FS01:Upload 14.63%
14:28:41:WU03:FS01:Upload 15.05%
14:28:47:WU03:FS01:Upload 16.75%
14:28:56:WU03:FS01:Upload 18.23%
14:32:17:WU03:FS01:Upload 19.72%
14:32:17:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
14:32:17:WU03:FS01:Trying to send results to collection server
14:32:17:WU03:FS01:Uploading 29.48MiB to 129.32.209.204
14:32:17:WU03:FS01:Connecting to 129.32.209.204:8080
14:32:23:WU03:FS01:Upload 1.91%
14:32:29:WU03:FS01:Upload 3.60%
14:34:05:WU03:FS01:Upload 7.21%
14:34:05:ERROR:WU03:FS01:Exception: Transfer failed
15:26:49:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:18448 run:6 clone:271 gen:145 core:0x22 unit:0x0000010f000000910000481000000006
15:26:49:WU03:FS01:Uploading 29.48MiB to 129.32.209.202
15:26:49:WU03:FS01:Connecting to 129.32.209.202:8080
15:27:13:WU03:FS01:Upload 0.64%
15:27:30:WU03:FS01:Upload 5.72%
15:27:36:WU03:FS01:Upload 7.42%
15:27:43:WU03:FS01:Upload 8.06%
15:27:49:WU03:FS01:Upload 9.75%
15:28:01:WU03:FS01:Upload 10.81%
15:28:07:WU03:FS01:Upload 12.51%
15:28:14:WU03:FS01:Upload 13.57%
15:28:20:WU03:FS01:Upload 15.48%
15:28:26:WU03:FS01:Upload 17.17%
15:28:41:WU03:FS01:Upload 18.66%
15:28:47:WU03:FS01:Upload 20.56%
15:28:55:WU03:FS01:Upload 21.20%
15:29:01:WU03:FS01:Upload 23.11%
15:29:09:WU03:FS01:Upload 24.17%
15:30:12:WU03:FS01:Upload 25.44%
15:32:48:WU03:FS01:Upload 25.65%
15:32:54:WU03:FS01:Upload 27.77%
15:33:00:WU03:FS01:Upload 29.68%
15:33:06:WU03:FS01:Upload 30.10%
15:33:12:WU03:FS01:Upload 32.01%
15:33:54:WU03:FS01:Upload 32.22%
15:33:54:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
15:33:54:WU03:FS01:Trying to send results to collection server
15:33:54:WU03:FS01:Uploading 29.48MiB to 129.32.209.204
15:33:54:WU03:FS01:Connecting to 129.32.209.204:8080
15:34:00:WU03:FS01:Upload 2.12%
15:34:06:WU03:FS01:Upload 4.45%
15:34:16:WU03:FS01:Upload 8.69%
15:34:22:WU03:FS01:Upload 8.90%
15:34:28:WU03:FS01:Upload 10.81%
15:34:35:WU03:FS01:Upload 11.45%
15:34:41:WU03:FS01:Upload 13.36%
15:34:50:WU03:FS01:Upload 14.84%
15:37:17:WU03:FS01:Upload 16.54%
15:38:20:WU03:FS01:Upload 16.75%
15:38:26:WU03:FS01:Upload 18.66%
15:38:32:WU03:FS01:Upload 20.35%
15:38:38:WU03:FS01:Upload 20.99%
15:38:45:WU03:FS01:Upload 21.20%
15:38:51:WU03:FS01:Upload 22.89%
15:38:59:WU03:FS01:Upload 24.38%
15:39:05:WU03:FS01:Upload 26.29%
15:39:13:WU03:FS01:Upload 27.56%
15:41:37:WU03:FS01:Upload 29.04%
15:42:30:WU03:FS01:Upload 30.31%
15:42:37:WU03:FS01:Upload 31.59%
15:43:04:WU03:FS01:Upload 32.65%
15:43:10:WU03:FS01:Upload 34.77%
15:43:16:WU03:FS01:Upload 36.46%
15:43:39:WU03:FS01:Upload 38.16%
15:43:53:WU03:FS01:Upload 38.58%
15:43:59:WU03:FS01:Upload 40.28%
15:44:29:WU03:FS01:Upload 41.76%
15:44:35:WU03:FS01:Upload 43.67%
15:45:03:WU03:FS01:Upload 45.15%
15:45:09:WU03:FS01:Upload 47.06%
15:45:17:WU03:FS01:Upload 48.12%
15:45:35:WU03:FS01:Upload 48.33%
15:45:55:WU03:FS01:Upload 48.55%
15:46:01:WU03:FS01:Upload 50.24%
15:46:18:WU03:FS01:Upload 51.30%
15:46:33:WU03:FS01:Upload 51.51%
15:46:39:WU03:FS01:Upload 53.21%
15:46:47:WU03:FS01:Upload 54.48%
15:47:13:WU03:FS01:Upload 54.69%
15:47:19:WU03:FS01:Upload 56.60%
15:47:49:WU03:FS01:Upload 57.66%
15:47:55:WU03:FS01:Upload 59.57%
15:48:06:WU03:FS01:Upload 59.99%
15:48:12:WU03:FS01:Upload 61.69%
15:48:21:WU03:FS01:Upload 63.17%
15:50:24:WU03:FS01:Upload 64.87%
15:50:24:ERROR:WU03:FS01:Exception: Transfer failed

SLOT #04

*********************** Log Started 2023-01-05T04:24:22Z ***********************
...
09:54:36:WU04:FS01:Trying to send results to collection server
09:54:36:WU04:FS01:Uploading 31.96MiB to 129.32.209.203
09:54:36:WU04:FS01:Connecting to 129.32.209.203:8080
09:54:43:WU04:FS01:Upload 0.39%
09:54:49:WU04:FS01:Upload 2.15%
09:54:55:WU04:FS01:Upload 4.11%
09:55:01:WU04:FS01:Upload 8.21%
09:55:11:WU04:FS01:Upload 8.41%
09:55:17:WU04:FS01:Upload 10.17%
09:55:23:WU04:FS01:Upload 10.75%
09:55:29:WU04:FS01:Upload 12.51%
09:55:35:WU04:FS01:Upload 14.08%
09:55:41:WU04:FS01:Upload 14.67%
09:57:13:WU04:FS01:Upload 16.03%
09:59:44:WU04:FS01:Upload 16.23%
10:02:34:WU04:FS01:Upload 16.43%
10:03:21:WU04:FS01:Upload 16.62%
10:03:41:WU04:FS01:Upload 17.99%
10:06:12:WU04:FS01:Upload 18.19%
10:08:29:WU04:FS01:Upload 18.38%
10:10:14:WU04:FS01:Upload 19.95%
10:12:16:WU04:FS01:Upload 20.14%
10:12:22:WU04:FS01:Upload 21.51%
10:13:57:WU04:FS01:Upload 21.71%
10:15:03:WU04:FS01:Upload 21.90%
10:15:09:WU04:FS01:Upload 23.66%
10:15:15:WU04:FS01:Upload 25.23%
10:15:21:WU04:FS01:Upload 25.81%
10:17:27:WU04:FS01:Upload 27.38%
10:21:19:WU04:FS01:Upload 27.57%
10:23:52:WU04:FS01:Upload 27.77%
10:23:58:WU04:FS01:Upload 29.53%
10:24:04:WU04:FS01:Upload 31.09%
10:24:10:WU04:FS01:Upload 31.68%
******************************* Date: 2023-01-06 *******************************
10:26:50:WU04:FS01:Upload 33.05%
10:28:00:WU04:FS01:Upload 33.24%
10:28:00:ERROR:WU04:FS01:Exception: Transfer failed
10:43:21:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:18449 run:2 clone:448 gen:37 core:0x22 unit:0x000001c0000000250000481100000002
10:43:21:WU04:FS01:Uploading 31.96MiB to 129.32.209.202
10:43:21:WU04:FS01:Connecting to 129.32.209.202:8080
10:43:27:WU04:FS01:Upload 0.20%
10:43:33:WU04:FS01:Upload 1.96%
10:43:39:WU04:FS01:Upload 3.91%
10:43:45:WU04:FS01:Upload 7.82%
10:44:02:WU04:FS01:Upload 8.02%
10:44:12:WU04:FS01:Upload 8.21%
10:44:18:WU04:FS01:Upload 9.97%
10:44:51:WU04:FS01:Upload 11.54%
10:44:57:WU04:FS01:Upload 13.10%
10:45:03:WU04:FS01:Upload 14.67%
10:45:13:WU04:FS01:Upload 14.86%
10:45:19:WU04:FS01:Upload 16.43%
10:45:29:WU04:FS01:Upload 17.99%
10:47:44:WU04:FS01:Upload 19.55%
10:51:36:WU04:FS01:Upload 19.75%
10:51:50:WU04:FS01:Upload 19.95%
10:51:56:WU04:FS01:Upload 21.71%
10:52:02:WU04:FS01:Upload 23.47%
10:52:08:WU04:FS01:Upload 23.86%
10:52:43:WU04:FS01:Upload 25.62%
10:52:44:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
10:52:44:WU04:FS01:Trying to send results to collection server
10:52:44:WU04:FS01:Uploading 31.96MiB to 129.32.209.203
10:52:44:WU04:FS01:Connecting to 129.32.209.203:8080
10:52:50:WU04:FS01:Upload 2.15%
10:52:56:WU04:FS01:Upload 3.72%
10:53:10:WU04:FS01:Upload 5.28%
10:53:21:WU04:FS01:Upload 6.65%
10:53:27:WU04:FS01:Upload 7.43%
10:53:33:WU04:FS01:Upload 9.19%
10:53:39:WU04:FS01:Upload 12.51%
10:54:08:WU04:FS01:Upload 12.91%
10:54:14:WU04:FS01:Upload 14.47%
10:54:30:WU04:FS01:Upload 15.84%
10:54:36:WU04:FS01:Upload 17.40%
10:54:44:WU04:FS01:Upload 18.38%
10:57:40:WU04:FS01:Upload 19.95%
10:58:33:WU04:FS01:Upload 20.14%
11:00:35:WU04:FS01:Upload 21.31%
11:00:45:WU04:FS01:Upload 21.51%
11:02:11:WU04:FS01:Upload 22.68%
11:02:17:WU04:FS01:Upload 24.44%
11:02:23:WU04:FS01:Upload 26.20%
11:02:38:WU04:FS01:Upload 26.99%
11:02:44:WU04:FS01:Upload 28.55%
11:02:52:WU04:FS01:Upload 29.72%
11:04:24:WU04:FS01:Upload 31.29%
11:07:14:WU04:FS01:Upload 31.48%
11:09:38:WU04:FS01:Upload 31.68%
11:11:31:WU04:FS01:Upload 33.24%
11:14:25:WU04:FS01:Upload 33.44%
11:14:31:WU04:FS01:Upload 34.81%
11:16:29:WU04:FS01:Upload 35.00%
11:18:11:WU04:FS01:Upload 35.20%
11:18:22:WU04:FS01:Upload 36.57%
11:21:01:WU04:FS01:Upload 36.76%
11:21:07:WU04:FS01:Upload 38.13%
11:23:05:WU04:FS01:Upload 38.33%
11:23:06:ERROR:WU04:FS01:Exception: Transfer failed
11:43:22:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:18449 run:2 clone:448 gen:37 core:0x22 unit:0x000001c0000000250000481100000002
11:43:22:WU04:FS01:Uploading 31.96MiB to 129.32.209.202
11:43:22:WU04:FS01:Connecting to 129.32.209.202:8080
11:43:28:WU04:FS01:Upload 2.15%
11:43:34:WU04:FS01:Upload 4.30%
11:43:49:WU04:FS01:Upload 8.21%
11:43:55:WU04:FS01:Upload 9.78%
11:44:04:WU04:FS01:Upload 10.95%
11:44:10:WU04:FS01:Upload 12.71%
11:44:19:WU04:FS01:Upload 13.88%
11:46:51:WU04:FS01:Upload 15.45%
11:47:55:WU04:FS01:Upload 15.64%
11:49:27:WU04:FS01:Upload 16.82%
11:50:27:WU04:FS01:Upload 17.01%
11:50:33:WU04:FS01:Upload 18.77%
11:50:39:WU04:FS01:Upload 20.53%
11:50:46:WU04:FS01:Upload 21.31%
11:50:54:WU04:FS01:Upload 21.51%
11:51:00:WU04:FS01:Upload 23.27%
11:51:07:WU04:FS01:Upload 24.25%
11:51:16:WU04:FS01:Upload 25.62%
11:53:09:WU04:FS01:Upload 26.59%
11:53:34:WU04:FS01:Upload 26.79%
11:53:40:WU04:FS01:Upload 27.96%
11:53:46:WU04:FS01:Upload 29.72%
11:53:52:WU04:FS01:Upload 31.09%
11:54:09:WU04:FS01:Upload 32.46%
11:54:15:WU04:FS01:Upload 34.02%
11:54:25:WU04:FS01:Upload 35.59%
11:55:42:WU04:FS01:Upload 37.15%
11:55:42:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
11:55:42:WU04:FS01:Trying to send results to collection server
11:55:42:WU04:FS01:Uploading 31.96MiB to 129.32.209.203
11:55:42:WU04:FS01:Connecting to 129.32.209.203:8080
11:56:01:WU04:FS01:Upload 0.39%
11:56:07:WU04:FS01:Upload 3.32%
11:56:21:WU04:FS01:Upload 6.06%
11:56:27:WU04:FS01:Upload 7.82%
11:56:35:WU04:FS01:Upload 8.60%
11:56:42:WU04:FS01:Upload 9.78%
11:57:12:WU04:FS01:Upload 10.75%
11:57:18:WU04:FS01:Upload 12.51%
11:57:24:WU04:FS01:Upload 14.08%
11:57:40:WU04:FS01:Upload 14.86%
11:57:53:WU04:FS01:Upload 15.06%
11:57:59:WU04:FS01:Upload 16.82%
11:58:33:WU04:FS01:Upload 18.38%
11:58:39:WU04:FS01:Upload 19.95%
11:58:49:WU04:FS01:Upload 21.51%
11:59:04:WU04:FS01:Upload 21.71%
11:59:28:WU04:FS01:Upload 21.90%
11:59:34:WU04:FS01:Upload 23.07%
11:59:42:WU04:FS01:Upload 23.86%
11:59:48:WU04:FS01:Upload 24.64%
11:59:54:WU04:FS01:Upload 25.03%
12:00:00:WU04:FS01:Upload 26.59%
12:00:08:WU04:FS01:Upload 27.57%
12:00:15:WU04:FS01:Upload 28.16%
12:00:21:WU04:FS01:Upload 29.92%
12:00:27:WU04:FS01:Upload 31.48%
12:00:43:WU04:FS01:Upload 32.26%
12:00:50:WU04:FS01:Upload 32.66%
12:01:18:WU04:FS01:Upload 32.85%
12:01:31:WU04:FS01:Upload 34.22%
12:01:37:WU04:FS01:Upload 36.18%
12:01:43:WU04:FS01:Upload 37.74%
12:01:55:WU04:FS01:Upload 38.72%
12:02:01:WU04:FS01:Upload 40.28%
12:02:10:WU04:FS01:Upload 41.46%
12:02:16:WU04:FS01:Upload 43.22%
12:02:25:WU04:FS01:Upload 44.58%
12:03:57:WU04:FS01:Upload 45.95%
12:05:55:WU04:FS01:Upload 46.15%
12:06:23:WU04:FS01:Upload 47.32%
12:08:04:WU04:FS01:Upload 47.52%
12:08:39:WU04:FS01:Upload 48.69%
12:08:39:ERROR:WU04:FS01:Exception: Transfer failed
12:43:22:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:18449 run:2 clone:448 gen:37 core:0x22 unit:0x000001c0000000250000481100000002
12:43:22:WU04:FS01:Uploading 31.96MiB to 129.32.209.202
12:43:22:WU04:FS01:Connecting to 129.32.209.202:8080
12:43:28:WU04:FS01:Upload 2.15%
12:43:34:WU04:FS01:Upload 4.11%
12:43:49:WU04:FS01:Upload 8.21%
12:43:55:WU04:FS01:Upload 9.78%
12:44:03:WU04:FS01:Upload 10.75%
12:44:09:WU04:FS01:Upload 12.51%
12:44:18:WU04:FS01:Upload 13.69%
12:45:40:WU04:FS01:Upload 15.06%
12:48:52:WU04:FS01:Upload 15.25%
12:50:17:WU04:FS01:Upload 16.43%
12:52:10:WU04:FS01:Upload 16.62%
12:54:49:WU04:FS01:Upload 17.79%
12:54:49:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
12:54:49:WU04:FS01:Trying to send results to collection server
12:54:49:WU04:FS01:Uploading 31.96MiB to 129.32.209.203
12:54:49:WU04:FS01:Connecting to 129.32.209.203:8080
12:54:55:WU04:FS01:Upload 1.76%
12:55:01:WU04:FS01:Upload 3.91%
12:55:07:WU04:FS01:Upload 8.02%
12:55:20:WU04:FS01:Upload 8.21%
12:55:33:WU04:FS01:Upload 8.41%
12:55:39:WU04:FS01:Upload 9.39%
12:55:49:WU04:FS01:Upload 10.36%
12:55:55:WU04:FS01:Upload 11.93%
12:56:02:WU04:FS01:Upload 12.91%
12:56:20:WU04:FS01:Upload 14.08%
12:56:50:WU04:FS01:Upload 14.27%
12:56:50:ERROR:WU04:FS01:Exception: Transfer failed
13:43:22:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:18449 run:2 clone:448 gen:37 core:0x22 unit:0x000001c0000000250000481100000002
13:43:22:WU04:FS01:Uploading 31.96MiB to 129.32.209.202
13:43:22:WU04:FS01:Connecting to 129.32.209.202:8080
13:43:28:WU04:FS01:Upload 1.96%
13:43:34:WU04:FS01:Upload 4.11%
13:43:53:WU04:FS01:Upload 8.21%
13:43:59:WU04:FS01:Upload 9.97%
13:44:07:WU04:FS01:Upload 10.75%
13:44:13:WU04:FS01:Upload 12.51%
13:44:21:WU04:FS01:Upload 13.69%
13:45:05:WU04:FS01:Upload 15.06%
13:46:04:WU04:FS01:Upload 15.25%
13:46:04:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
13:46:04:WU04:FS01:Trying to send results to collection server
13:46:04:WU04:FS01:Uploading 31.96MiB to 129.32.209.203
13:46:04:WU04:FS01:Connecting to 129.32.209.203:8080
13:46:10:WU04:FS01:Upload 1.76%
13:46:16:WU04:FS01:Upload 4.11%
13:46:24:WU04:FS01:Upload 7.63%
13:46:30:WU04:FS01:Upload 9.00%
13:46:36:WU04:FS01:Upload 10.95%
13:46:43:WU04:FS01:Upload 11.73%
13:47:22:WU04:FS01:Upload 13.30%
13:50:46:WU04:FS01:Upload 13.49%
13:50:46:ERROR:WU04:FS01:Exception: Transfer failed
14:43:22:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:18449 run:2 clone:448 gen:37 core:0x22 unit:0x000001c0000000250000481100000002
14:43:22:WU04:FS01:Uploading 31.96MiB to 129.32.209.202
14:43:22:WU04:FS01:Connecting to 129.32.209.202:8080
14:43:28:WU04:FS01:Upload 1.76%
14:43:34:WU04:FS01:Upload 4.11%
14:43:44:WU04:FS01:Upload 8.02%
14:43:58:WU04:FS01:Upload 8.21%
14:44:07:WU04:FS01:Upload 8.41%
14:44:13:WU04:FS01:Upload 9.97%
14:44:40:WU04:FS01:Upload 11.73%
14:44:46:WU04:FS01:Upload 13.30%
14:45:14:WU04:FS01:Upload 14.86%
14:45:20:WU04:FS01:Upload 15.25%
14:45:26:WU04:FS01:Upload 16.82%
14:45:34:WU04:FS01:Upload 17.79%
14:45:56:WU04:FS01:Upload 19.16%
14:49:05:WU04:FS01:Upload 19.36%
14:49:23:WU04:FS01:Upload 19.55%
14:49:51:WU04:FS01:Upload 20.53%
14:51:49:WU04:FS01:Upload 20.73%
14:53:14:WU04:FS01:Upload 21.90%
14:53:30:WU04:FS01:Upload 23.07%
14:53:48:WU04:FS01:Upload 23.27%
14:53:54:WU04:FS01:Upload 25.03%
14:54:00:WU04:FS01:Upload 26.59%
14:54:06:WU04:FS01:Upload 27.18%
14:54:21:WU04:FS01:Upload 27.38%
14:54:34:WU04:FS01:Upload 27.57%
14:54:40:WU04:FS01:Upload 27.77%
14:55:04:WU04:FS01:Upload 27.96%
14:55:10:WU04:FS01:Upload 29.14%
14:55:18:WU04:FS01:Upload 30.70%
14:55:24:WU04:FS01:Upload 32.07%
14:55:30:WU04:FS01:Upload 33.83%
14:55:36:WU04:FS01:Upload 35.39%
14:55:42:WU04:FS01:Upload 37.15%
14:55:54:WU04:FS01:Upload 37.54%
14:56:00:WU04:FS01:Upload 39.11%
14:56:08:WU04:FS01:Upload 40.28%
14:58:26:WU04:FS01:Upload 41.85%
14:59:45:WU04:FS01:Upload 42.04%
14:59:51:WU04:FS01:Upload 43.80%
14:59:57:WU04:FS01:Upload 45.56%
15:00:10:WU04:FS01:Upload 46.15%
15:00:16:WU04:FS01:Upload 47.71%
15:00:26:WU04:FS01:Upload 49.28%
15:00:35:WU04:FS01:Upload 49.47%
15:01:47:WU04:FS01:Upload 50.65%
15:01:47:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
15:01:47:WU04:FS01:Trying to send results to collection server
15:01:47:WU04:FS01:Uploading 31.96MiB to 129.32.209.203
15:01:47:WU04:FS01:Connecting to 129.32.209.203:8080
15:01:53:WU04:FS01:Upload 1.76%
15:01:59:WU04:FS01:Upload 3.91%
15:02:14:WU04:FS01:Upload 8.02%
15:02:20:WU04:FS01:Upload 9.58%
15:02:28:WU04:FS01:Upload 10.56%
15:02:34:WU04:FS01:Upload 12.12%
15:02:43:WU04:FS01:Upload 13.49%
15:03:29:WU04:FS01:Upload 14.86%
15:03:30:ERROR:WU04:FS01:Exception: Transfer failed
15:43:22:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:18449 run:2 clone:448 gen:37 core:0x22 unit:0x000001c0000000250000481100000002
15:43:22:WU04:FS01:Uploading 31.96MiB to 129.32.209.202
15:43:22:WU04:FS01:Connecting to 129.32.209.202:8080
15:43:28:WU04:FS01:Upload 0.78%
15:43:34:WU04:FS01:Upload 2.54%
15:43:40:WU04:FS01:Upload 5.08%
15:44:10:WU04:FS01:Upload 7.63%
15:44:16:WU04:FS01:Upload 9.19%
15:44:36:WU04:FS01:Upload 10.75%
15:44:48:WU04:FS01:Upload 10.95%
15:44:54:WU04:FS01:Upload 12.71%
15:45:22:WU04:FS01:Upload 13.88%
15:45:28:WU04:FS01:Upload 15.45%
15:45:39:WU04:FS01:Upload 16.03%
15:45:45:WU04:FS01:Upload 17.60%
15:46:01:WU04:FS01:Upload 18.77%
15:46:11:WU04:FS01:Upload 18.97%
15:46:17:WU04:FS01:Upload 20.73%
15:46:23:WU04:FS01:Upload 22.29%
15:46:53:WU04:FS01:Upload 22.68%
15:46:59:WU04:FS01:Upload 24.25%
15:47:07:WU04:FS01:Upload 25.23%
15:47:27:WU04:FS01:Upload 25.42%
15:47:33:WU04:FS01:Upload 27.18%
15:47:41:WU04:FS01:Upload 28.16%
15:49:07:WU04:FS01:Upload 29.72%
15:49:07:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
15:49:07:WU04:FS01:Trying to send results to collection server
15:49:07:WU04:FS01:Uploading 31.96MiB to 129.32.209.203
15:49:07:WU04:FS01:Connecting to 129.32.209.203:8080
15:49:13:WU04:FS01:Upload 1.76%
15:49:19:WU04:FS01:Upload 3.91%
15:49:34:WU04:FS01:Upload 8.02%
15:49:40:WU04:FS01:Upload 9.39%
15:49:49:WU04:FS01:Upload 10.56%
15:49:55:WU04:FS01:Upload 12.12%
15:50:03:WU04:FS01:Upload 13.30%
15:51:11:WU04:FS01:Upload 14.67%
15:53:17:WU04:FS01:Upload 14.86%
15:54:22:WU04:FS01:Upload 16.03%
15:54:22:ERROR:WU04:FS01:Exception: Transfer failed

kmoyer · Post by **kmoyer** » Sat Jan 07, 2023 9:50 pm

All my Work Units got dumped for passing the deadline and none of them where ever successfully uploaded. I'm folding again and will see if I continue to get units from those servers that fail to upload.

kmoyer · Post by **kmoyer** » Sun Jan 08, 2023 9:01 pm

I now have 3 more Work Units failing to upload to the servers as of the time of this post.

Post by **Joe_H** » Sun Jan 08, 2023 9:39 pm

The persons responsible for the servers were notified some time ago. They should be looking into the problem. However so far there are only a few reports of problems, they may be having trouble identifying a cause. I do know my systems have uploaded about a dozen WUs over the last 48 hours or so to Temple and no delays are shown in my logs.

kmoyer · Post by **kmoyer** » Mon Jan 09, 2023 12:40 am

Yes, my system connected to a very good internet connection has uploaded multiple in the last 96 hours without issue. The problem is with systems with a slow UPLOAD connection. My problem system takes 4 minutes or more to upload a given work unit where my good system takes seconds.

Throttling upload throughput to 0.8 Mbit/sec or less would allow for easy reproduction of the issue.

Post by **Joe_H** » Mon Jan 09, 2023 12:45 am

I specifically am talking about having no problems with upload speeds to the same servers you are talking about. So the problem is with specific connections to those servers, not connections to them in general.

kmoyer · Post by **kmoyer** » Tue Jan 10, 2023 1:54 am

Yes, correct.

jrsynow · Post by **jrsynow** » Mon Jan 30, 2023 4:12 am

Glad I'm not the only one having these issues, I spoke too soon with my earlier message (and got lucky with a lot of 18601 work units the last month). If it helps, the ISPs that have struggled uploading to these servers are AT&T Uverse (18MBit dn/2MBit up) and AT&T mobile (~40MBit dn / 5~10MBit up). Probably 75% of the time the mobile connection uploads fine. My 3rd ISP Wide Open West (500MBit dn / 50MBit up) never has uploading issues.

Logs from tonight's upload attempt:

Code: Select all

02:28:10:WU01:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
02:28:11:WU01:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
02:28:11:WU01:FS02:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:0 clone:219 gen:200 core:0x22 unit:0x000000db000000c80000481100000000
02:28:11:WU01:FS02:Uploading 31.02MiB to 129.32.209.202
02:28:11:WU01:FS02:Connecting to 129.32.209.202:8080
02:28:17:WU01:FS02:Upload 0.81%
02:28:23:WU01:FS02:Upload 3.02%
02:28:47:WU01:FS02:Upload 5.04%
02:28:53:WU01:FS02:Upload 9.27%
02:28:59:WU01:FS02:Upload 13.50%
02:30:06:WU01:FS02:Upload 17.33%
02:30:06:WARNING:WU01:FS02:Exception: Failed to send results to work server: Transfer failed
02:30:06:WU01:FS02:Trying to send results to collection server
02:30:06:WU01:FS02:Uploading 31.02MiB to 129.32.209.206
02:30:06:WU01:FS02:Connecting to 129.32.209.206:8080
02:30:12:WU01:FS02:Upload 4.43%
02:30:18:WU01:FS02:Upload 10.07%
02:30:27:WU01:FS02:Upload 13.90%
02:30:55:WU01:FS02:Upload 15.31%
02:32:07:WU01:FS02:Upload 15.51%
02:33:27:WU01:FS02:Upload 15.72%
02:34:47:WU01:FS02:Upload 15.92%
02:34:56:WU01:FS02:Upload 16.12%
02:35:02:WU01:FS02:Upload 18.34%
02:35:08:WU01:FS02:Upload 22.57%
02:35:17:WU01:FS02:Upload 24.38%
02:36:42:WU01:FS02:Upload 24.58%
02:37:20:WU01:FS02:Upload 25.79%
02:37:54:WU01:FS02:Upload 25.99%
02:38:00:WU01:FS02:Upload 30.22%
02:38:07:WU01:FS02:Upload 34.05%
02:39:51:WU01:FS02:Upload 35.66%
02:40:55:WU01:FS02:Upload 35.86%
02:42:19:WU01:FS02:Upload 37.27%
02:42:46:WU01:FS02:Upload 37.48%
02:43:32:WU01:FS02:Upload 38.69%
02:44:27:WU01:FS02:Upload 38.89%
02:44:33:WU01:FS02:Upload 43.32%
02:44:39:WU01:FS02:Upload 46.34%
02:45:54:WU01:FS02:Upload 47.95%
02:46:44:WU01:FS02:Upload 48.16%
02:46:50:WU01:FS02:Upload 52.59%
02:46:56:WU01:FS02:Upload 56.01%
02:48:09:WU01:FS02:Upload 57.62%
02:48:34:WU01:FS02:Upload 57.83%
02:49:36:WU01:FS02:Upload 59.24%
02:50:37:WU01:FS02:Upload 60.65%
02:50:37:ERROR:WU01:FS02:Exception: Transfer failed
02:50:37:WU01:FS02:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:0 clone:219 gen:200 core:0x22 unit:0x000000db000000c80000481100000000
02:50:37:WU01:FS02:Uploading 31.02MiB to 129.32.209.202
02:50:37:WU01:FS02:Connecting to 129.32.209.202:8080
02:50:43:WU01:FS02:Upload 2.82%
02:50:51:WU01:FS02:Upload 5.24%
02:51:22:WU01:FS02:Upload 6.45%
02:52:52:WU01:FS02:Upload 6.65%
02:53:49:WU01:FS02:Upload 6.85%
02:54:42:WU01:FS02:Upload 7.05%
02:55:07:WU01:FS02:Upload 7.25%
02:55:50:WU01:FS02:Upload 8.46%
02:56:47:WU01:FS02:Upload 8.66%
02:57:44:WU01:FS02:Upload 8.87%
02:58:27:WU01:FS02:Upload 9.07%
02:58:33:WU01:FS02:Upload 13.30%
02:58:42:WU01:FS02:Upload 16.92%
02:58:48:WU01:FS02:Upload 21.16%
02:58:54:WU01:FS02:Upload 24.78%
03:00:05:WU01:FS02:Upload 26.39%
03:00:39:WU01:FS02:Upload 26.60%
03:01:07:WU01:FS02:Upload 27.81%
03:01:55:WU01:FS02:Upload 28.01%
03:02:01:WU01:FS02:Upload 32.64%
03:02:07:WU01:FS02:Upload 36.07%
03:02:39:WU01:FS02:Upload 37.68%
03:03:17:WU01:FS02:Upload 37.88%
03:03:34:WU01:FS02:Upload 39.09%
03:04:14:WU01:FS02:Upload 39.29%
03:04:30:WU01:FS02:Upload 40.50%
03:05:06:WU01:FS02:Upload 40.70%
03:05:20:WU01:FS02:Upload 41.91%
03:05:49:WU01:FS02:Upload 42.11%
03:06:09:WU01:FS02:Upload 43.32%
03:06:21:WU01:FS02:Upload 43.52%
03:06:27:WU01:FS02:Upload 47.75%
03:06:34:WU01:FS02:Upload 51.58%
03:07:46:WU01:FS02:Upload 53.19%
03:08:11:WU01:FS02:Upload 53.39%
03:08:17:WU01:FS02:Upload 57.22%
03:08:24:WU01:FS02:Upload 61.45%
03:08:54:WU01:FS02:Upload 63.07%
03:09:53:WU01:FS02:Upload 63.27%
03:09:53:WARNING:WU01:FS02:Exception: Failed to send results to work server: Transfer failed
03:09:53:WU01:FS02:Trying to send results to collection server
03:09:53:WU01:FS02:Uploading 31.02MiB to 129.32.209.206
03:09:53:WU01:FS02:Connecting to 129.32.209.206:8080
03:09:59:WU01:FS02:Upload 4.84%
03:10:05:WU01:FS02:Upload 10.07%
03:10:13:WU01:FS02:Upload 14.31%
03:10:19:WU01:FS02:Upload 18.54%
03:10:25:WU01:FS02:Upload 21.76%
03:11:18:WU01:FS02:Upload 22.97%
03:11:42:WU01:FS02:Upload 23.17%
03:11:48:WU01:FS02:Upload 27.40%
03:11:55:WU01:FS02:Upload 31.23%
03:12:14:WU01:FS02:Upload 32.84%
03:13:27:WU01:FS02:Upload 33.04%
03:13:42:WU01:FS02:Upload 33.25%
03:14:09:WU01:FS02:Upload 34.45%
03:14:53:WU01:FS02:Upload 34.66%
03:14:59:WU01:FS02:Upload 38.69%
03:15:05:WU01:FS02:Upload 42.11%
03:15:38:WU01:FS02:Upload 43.52%
03:17:00:WU01:FS02:Upload 43.72%
03:17:06:WU01:FS02:Upload 48.16%
03:17:12:WU01:FS02:Upload 52.39%
03:17:58:WU01:FS02:Upload 55.21%
03:18:58:WU01:FS02:Upload 55.41%
03:20:27:WU01:FS02:Upload 56.82%
03:20:43:WU01:FS02:Upload 58.23%
03:21:00:WU01:FS02:Upload 58.43%
03:21:21:WU01:FS02:Upload 59.64%
03:21:39:WU01:FS02:Upload 61.05%
03:22:23:WU01:FS02:Upload 63.87%
03:22:23:ERROR:WU01:FS02:Exception: Transfer failed <<< Rebooted the router here to see if it would help, it didn't
03:22:23:WU01:FS02:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:0 clone:219 gen:200 core:0x22 unit:0x000000db000000c80000481100000000
03:22:23:WU01:FS02:Uploading 31.02MiB to 129.32.209.202
03:22:23:WU01:FS02:Connecting to 129.32.209.202:8080
03:22:23:WARNING:WU01:FS02:WorkServer connection failed on port 8080 trying 80
03:22:23:WU01:FS02:Connecting to 129.32.209.202:80
03:22:23:WARNING:WU01:FS02:Exception: Failed to send results to work server: Failed to connect to 129.32.209.202:80: A socket operation was attempted to an unreachable host.
.
.
.
03:26:37:WU01:FS02:Trying to send results to collection server
03:26:38:WU01:FS02:Uploading 31.02MiB to 129.32.209.206
03:26:38:WU01:FS02:Connecting to 129.32.209.206:8080
03:26:38:WARNING:WU01:FS02:WorkServer connection failed on port 8080 trying 80
03:26:38:WU01:FS02:Connecting to 129.32.209.206:80
03:26:38:ERROR:WU01:FS02:Exception: Failed to connect to 129.32.209.206:80: A socket operation was attempted to an unreachable host.
03:30:52:WU01:FS02:Sending unit results: id:01 state:SEND error:NO_ERROR project:18449 run:0 clone:219 gen:200 core:0x22 unit:0x000000db000000c80000481100000000
03:30:52:WU01:FS02:Uploading 31.02MiB to 129.32.209.202
03:30:52:WU01:FS02:Connecting to 129.32.209.202:8080
03:30:58:WU01:FS02:Upload 4.63%
03:31:04:WU01:FS02:Upload 10.28%
03:31:10:WU01:FS02:Upload 14.91%
03:32:17:WU01:FS02:Upload 15.72%
03:33:26:WU01:FS02:Upload 15.92%
03:33:35:WU01:FS02:Upload 16.12%
03:33:41:WU01:FS02:Upload 20.15%
03:33:47:WU01:FS02:Upload 23.17%
03:34:22:WU01:FS02:Upload 23.78%
03:34:53:WU01:FS02:Upload 23.98%
03:35:28:WU01:FS02:Upload 25.19%
03:35:34:WU01:FS02:Upload 27.81%
03:35:40:WU01:FS02:Upload 31.83%
03:35:46:WU01:FS02:Upload 34.86%
03:36:26:WU01:FS02:Upload 35.86%
03:37:19:WU01:FS02:Upload 36.07%
03:38:20:WU01:FS02:Upload 37.48%
03:38:36:WU01:FS02:Upload 37.68%
03:38:59:WU01:FS02:Upload 38.89%
03:39:33:WU01:FS02:Upload 39.09%
03:40:06:WU01:FS02:Upload 40.30%
03:40:43:WU01:FS02:Upload 40.50%
03:40:49:WU01:FS02:Upload 44.93%
03:40:55:WU01:FS02:Upload 47.95%
03:41:01:WU01:FS02:Upload 49.97%
03:41:07:WU01:FS02:Upload 54.00%
03:41:13:WU01:FS02:Upload 55.01%
03:41:19:WU01:FS02:Upload 59.24%
03:41:25:WU01:FS02:Upload 62.26%
03:42:24:WU01:FS02:Upload 63.47%
03:43:55:WU01:FS02:Upload 63.67%
03:44:06:WU01:FS02:Upload 64.88%
03:45:53:WU01:FS02:Upload 65.08%
03:46:05:WU01:FS02:Upload 65.28%
03:47:20:WU01:FS02:Upload 66.49%
03:47:39:WU01:FS02:Upload 66.69%
03:47:45:WU01:FS02:Upload 70.32%
03:47:51:WU01:FS02:Upload 73.54%
03:49:29:WU01:FS02:Upload 74.95%
03:49:35:WU01:FS02:Upload 75.15%
03:50:25:WU01:FS02:Upload 76.16%
03:50:54:WU01:FS02:Upload 76.36%
03:51:00:WU01:FS02:Upload 80.39%
03:51:06:WU01:FS02:Upload 83.82%
03:52:11:WU01:FS02:Upload 85.23%
03:53:14:WU01:FS02:Upload 85.43%
03:53:42:WU01:FS02:Upload 86.64%
03:54:34:WU01:FS02:Upload 86.84%
03:54:40:WU01:FS02:Upload 91.07%
03:54:46:WU01:FS02:Upload 94.30%
03:55:15:WU01:FS02:Upload 95.71%
03:55:53:WU01:FS02:Upload 95.91% < so close T_T
03:55:53:WARNING:WU01:FS02:Exception: Failed to send results to work server: Transfer failed
03:55:53:WU01:FS02:Trying to send results to collection server
03:55:53:WU01:FS02:Uploading 31.02MiB to 129.32.209.206
03:55:53:WU01:FS02:Connecting to 129.32.209.206:8080
03:55:59:WU01:FS02:Upload 4.43%
03:56:05:WU01:FS02:Upload 8.66%
03:56:11:WU01:FS02:Upload 12.49%
03:56:17:WU01:FS02:Upload 13.30%
03:57:35:WU01:FS02:Upload 13.50%
03:57:50:WU01:FS02:Upload 13.70%
03:57:56:WU01:FS02:Upload 17.73%
03:58:02:WU01:FS02:Upload 20.75%
03:58:08:WU01:FS02:Upload 24.58%
03:58:14:WU01:FS02:Upload 27.81%
03:59:38:WU01:FS02:Upload 29.22%
03:59:50:WU01:FS02:Upload 29.42%
03:59:56:WU01:FS02:Upload 33.85%
04:00:05:WU01:FS02:Upload 37.27%
04:00:28:WU01:FS02:Upload 38.69%
04:01:24:WU01:FS02:Upload 38.89%
04:02:46:WU01:FS02:Upload 40.30%
04:04:02:WU01:FS02:Upload 41.71%
04:04:08:WU01:FS02:Upload 41.91%

Pop Piasa · Post by **Pop Piasa** » Wed Mar 22, 2023 6:40 pm

Is there any attention being given to the issue of T.U. collection servers dropping slow uploads?
Perhaps adjusting a time-out parameter on the server would help, but it seems to keep slowing down until it just quits.

18:05:57:WU00:FS01:Trying to send results to collection server
18:05:57:WU00:FS01:Uploading 64.73MiB to 129.32.209.207
18:05:57:WU00:FS01:Connecting to 129.32.209.207:8080
18:06:18:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
18:06:18:WU00:FS01:Connecting to 129.32.209.207:80
18:06:18:WU00:FS01:Upload 0.10%
18:06:24:WU00:FS01:Upload 2.12%
18:06:30:WU00:FS01:Upload 4.92%
18:06:38:WU00:FS01:Upload 7.72%
18:08:01:WU00:FS01:Upload 8.88%
18:08:03:WU03:Upload 24.45%
18:08:52:WU02:FS02:0x22:Completed 1950000 out of 5000000 steps (39%)
18:09:42:WU03:Upload 24.56%
18:09:55:WU00:FS01:Upload 8.98%
18:11:41:WU03:Upload 25.80%
18:11:50:WU00:FS01:Upload 9.08%
18:13:43:WU03:Upload 25.91%
18:13:45:WU00:FS01:Upload 9.17%
18:13:59:WU00:FS01:Upload 9.27%
18:14:05:WU00:FS01:Upload 11.30%
18:14:11:WU00:FS01:Upload 13.42%
18:14:17:WU00:FS01:Upload 14.48%
18:15:34:WU02:FS02:0x22:Completed 2000000 out of 5000000 steps (40%)
18:15:34:WU02:FS02:0x22:Checkpoint completed at step 2000000
18:15:58:WU03:Upload 26.03%
18:15:58:WARNING:WU03:Exception: Failed to send results to work server: Transfer failed

Post by **Joe_H** » Thu Mar 23, 2023 8:25 pm

Pop Piasa wrote: ↑Wed Mar 22, 2023 6:40 pm Is there any attention being given to the issue of T.U. collection servers dropping slow uploads?

Multiple reports have been given to the researchers at TU, they have looked into them and can not find a cause. Complicating things is that it does not affect all connections, for example I have never seen this problem on my systems connecting to TU servers.

Pop Piasa · Post by **Pop Piasa** » Thu Mar 23, 2023 10:15 pm

Thanks, I first saw this in Dec '22 and switched to doing BOINC for a while, hoping the bug would disappear over time. I recently discovered that I can upload project 18717 so I can just do those for now.
I wish I could abort the transfers that are now trying in vain to upload to Temple, and also servers 34.72.228.44 and 66.170.111.50. (All are on port 80 as 8080 errors out. The latter 2 servers are specified as 'assign' if that is of significance here.)
I wish I didn't have to wait for them to expire to send them back to the server to be reassigned. Now that failed tasks receive no credit, it would be nice to be able to cancel a problem upload.

Post by **Joe_H** » Thu Mar 23, 2023 11:10 pm

Status "Assign" just means that WS is both sending out and receiving WUs. So not of significance here.

You mention that Project 18717 is uploading for you, that goes to server 34.72.228.44 which you mention having problems with later in your post. So which projects going to that server are you having problems with?

One thing to check if you see WUs reaching full upload, but sticking around is whether they uploaded and the ACK packet from the server was dropped before getting back to your client. You can check a WU status here - https://apps.foldingathome.org/wu. There are some ISPs that set their routers to drop ACK packets as not being crucial to deliver as part of their traffic management under higher loads.

Folding Forum

Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)

Re: Struggles Uploading Completed Jobs to Temple University Servers (129.32.209.20x)