Multiple WU's Fail downld/upld to 155.247.166.*

Moderators: Site Moderators, FAHC Science Team

Post Reply
kmoyer
Posts: 8
Joined: Mon Oct 03, 2016 7:58 pm

Re: New Work Units Fail to Complete Download

Post by kmoyer »

I've been experiencing the issue of work units failing to download throughout 2019. Typically experienced this 1-2 times per month. Note that a computer reboot has always resolved the issue for me.
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: New Work Units Fail to Complete Download

Post by bruce »

From what I've been able to determine, two Work Servers had problems for about a day: 155.247.166.219 and 155.247.166.220. WUs were downloaded and uploaded at increasingly slow rates until they started failing midway though the transfer. I don't understand what happened to those incomplete transfers but they seem to have left traces of the corruptions on our local systems.

Two things had to happen. First, the overload of the servers at 155.247.166.* was corrected, allowing for new appropriately good connections. Second. whatever was left over from the incomplete transfers somehow had also corrupted our local ISP/Router networking, and restarting them was required.
Top
dfgirl12
Posts: 38
Joined: Fri Aug 21, 2009 8:34 am

Re: Multiple WU's Fail to Upload

Post by dfgirl12 »

The network slowness seems to only be during day-time hours in the US [Please tell most everyone to stop using the Internet so FAH uploads will work better... :lol: ].

Linux seem to keep pushing data through at a snails pace until the upload finishes without failing.
The poor Windows machine is backing up with WUs failing to upload to most any server with this much congestion:

Code: Select all

2019-10-01:00:13:49:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14229 run:500 clone:0 gen:46 core:0x21 unit:0x0000004880fccb0a5d6553da6655cf13
2019-10-01:00:13:49:WU03:FS01:Uploading 51.38MiB to 128.252.203.10
2019-10-01:00:13:49:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:13:55:WU03:FS01:Upload 0.12%
2019-10-01:00:14:16:WU03:FS01:Upload 0.36%
2019-10-01:00:15:00:WU03:FS01:Upload 0.49%
2019-10-01:00:15:29:WU03:FS01:Upload 0.61%
2019-10-01:00:15:40:WU03:FS01:Upload 0.73%
2019-10-01:00:15:52:WU03:FS01:Upload 1.09%
2019-10-01:00:15:58:WU03:FS01:Upload 1.34%
2019-10-01:00:16:20:WU03:FS01:Upload 1.58%
2019-10-01:00:16:29:WU03:FS01:Upload 1.70%
2019-10-01:00:16:37:WU03:FS01:Upload 1.95%
2019-10-01:00:16:45:WU03:FS01:Upload 2.07%
2019-10-01:00:16:51:WU03:FS01:Upload 2.43%
2019-10-01:00:17:04:WU03:FS01:Upload 2.55%
2019-10-01:00:17:09:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:66 clone:0 gen:33 core:0x21 unit:0x0000002980fccb0a5d6ed20de7fad410
2019-10-01:00:17:09:WU01:FS01:Uploading 65.87MiB to 128.252.203.10
2019-10-01:00:17:09:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:17:11:WU03:FS01:Upload 2.68%
2019-10-01:00:17:16:WU01:FS01:Upload 0.09%
2019-10-01:00:17:17:WU03:FS01:Upload 2.92%
2019-10-01:00:17:22:WU01:FS01:Upload 0.28%
2019-10-01:00:17:25:WU03:FS01:Upload 3.28%
2019-10-01:00:17:37:WU03:FS01:Upload 3.41%
2019-10-01:00:17:44:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14227 run:258 clone:1 gen:12 core:0x21 unit:0x0000001780fccb0a5d6545e0721b8725
2019-10-01:00:17:44:WU04:FS01:Uploading 93.71MiB to 128.252.203.10
2019-10-01:00:17:44:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:17:55:WU03:FS01:Upload 3.53%
2019-10-01:00:17:58:WU01:FS01:Upload 0.38%
2019-10-01:00:18:18:WU01:FS01:Upload 0.47%
2019-10-01:00:18:26:WU01:FS01:Upload 0.66%
2019-10-01:00:18:50:WU01:FS01:Upload 0.76%
2019-10-01:00:18:59:WU01:FS01:Upload 1.04%
2019-10-01:00:19:00:WU04:FS01:Upload 0.07%
2019-10-01:00:19:00:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:19:00:WU04:FS01:Trying to send results to collection server
2019-10-01:00:19:00:WU04:FS01:Uploading 93.71MiB to 155.247.166.220
2019-10-01:00:19:00:WU04:FS01:Connecting to 155.247.166.220:8080
2019-10-01:00:19:03:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-01:00:19:14:WU01:FS01:Upload 1.14%
2019-10-01:00:19:42:WU03:FS01:Upload 3.77%
2019-10-01:00:19:42:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:19:42:WU03:FS01:Trying to send results to collection server
2019-10-01:00:19:42:WU03:FS01:Uploading 51.38MiB to 155.247.166.219
2019-10-01:00:19:42:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:19:43:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-01:00:21:10:WU01:FS01:Upload 1.23%
2019-10-01:00:21:10:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:21:10:WU01:FS01:Trying to send results to collection server
2019-10-01:00:21:10:WU01:FS01:Uploading 65.87MiB to 155.247.166.219
2019-10-01:00:21:10:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:21:11:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-01:00:38:22:FS01:Unpaused
2019-10-01:00:38:22:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14233 run:5 clone:43 gen:94 core:0x21 unit:0x0000007780fccb0a5d65490f2bd0992e
2019-10-01:00:38:22:WU00:FS01:Uploading 69.10MiB to 128.252.203.10
2019-10-01:00:38:22:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:38:23:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:66 clone:0 gen:33 core:0x21 unit:0x0000002980fccb0a5d6ed20de7fad410
2019-10-01:00:38:23:WU01:FS01:Uploading 65.87MiB to 128.252.203.10
2019-10-01:00:38:23:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:38:23:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14230 run:526 clone:0 gen:28 core:0x21 unit:0x0000002d80fccb0a5d654c5fb2a6a95c
2019-10-01:00:38:23:WU02:FS01:Uploading 51.59MiB to 128.252.203.10
2019-10-01:00:38:23:WU02:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:38:23:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14229 run:500 clone:0 gen:46 core:0x21 unit:0x0000004880fccb0a5d6553da6655cf13
2019-10-01:00:38:23:WU03:FS01:Uploading 51.38MiB to 128.252.203.10
2019-10-01:00:38:23:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:38:23:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14227 run:258 clone:1 gen:12 core:0x21 unit:0x0000001780fccb0a5d6545e0721b8725
2019-10-01:00:38:23:WU04:FS01:Uploading 93.71MiB to 128.252.203.10
2019-10-01:00:38:23:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:38:24:WU05:FS01:Starting
2019-10-01:00:38:24:WU05:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Jeff\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 05 -suffix 01 -version 705 -lifeline 32904 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
2019-10-01:00:38:24:WU05:FS01:Started FahCore on PID 54192
2019-10-01:00:38:24:WU05:FS01:Core PID:52416
2019-10-01:00:38:24:WU05:FS01:FahCore 0x21 started
2019-10-01:00:38:24:WU05:FS01:0x21:*********************** Log Started 2019-10-01T00:38:24Z ***********************
2019-10-01:00:38:24:WU05:FS01:0x21:Project: 14191 (Run 32, Clone 3, Gen 57)
2019-10-01:00:38:24:WU05:FS01:0x21:Unit: 0x0000004a0002894c5d5d7438dbfe671e
2019-10-01:00:38:24:WU05:FS01:0x21:CPU: 0x00000000000000000000000000000000
2019-10-01:00:38:24:WU05:FS01:0x21:Machine: 1
2019-10-01:00:38:24:WU05:FS01:0x21:Digital signatures verified
2019-10-01:00:38:24:WU05:FS01:0x21:Folding@home GPU Core21 Folding@home Core
2019-10-01:00:38:24:WU05:FS01:0x21:Version 0.0.20
2019-10-01:00:38:24:WU05:FS01:0x21:  Found a checkpoint file
2019-10-01:00:38:25:WU05:FS01:0x21:Completed 13250000 out of 25000000 steps (53%)
2019-10-01:00:38:25:WU05:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
2019-10-01:00:38:31:FS01:Paused
2019-10-01:00:38:31:FS01:Shutting core down
2019-10-01:00:38:31:WU05:FS01:0x21:WARNING:Console control signal 1 on PID 52416
2019-10-01:00:38:31:WU05:FS01:0x21:Exiting, please wait. . .
2019-10-01:00:38:31:WU05:FS01:0x21:Folding@home Core Shutdown: INTERRUPTED
2019-10-01:00:38:31:WU05:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
2019-10-01:00:38:47:WU00:FS01:Upload 0.09%
2019-10-01:00:38:53:WU00:FS01:Upload 0.18%
2019-10-01:00:39:00:WU02:FS01:Upload 0.12%
2019-10-01:00:39:03:WU00:FS01:Upload 0.27%
2019-10-01:00:39:12:WU04:FS01:Upload 0.07%
2019-10-01:00:39:13:WU00:FS01:Upload 0.45%
2019-10-01:00:39:19:WU00:FS01:Upload 0.63%
2019-10-01:00:39:25:WU00:FS01:Upload 0.81%
2019-10-01:00:39:32:WU04:FS01:Upload 0.13%
2019-10-01:00:39:33:WU00:FS01:Upload 0.99%
2019-10-01:00:39:35:WU02:FS01:Upload 0.24%
2019-10-01:00:39:41:WU00:FS01:Upload 1.18%
2019-10-01:00:39:42:WU02:FS01:Upload 0.36%
2019-10-01:00:39:44:WU04:FS01:Upload 0.20%
2019-10-01:00:39:51:WU01:FS01:Upload 0.09%
2019-10-01:00:39:51:WU03:FS01:Upload 0.12%
2019-10-01:00:39:51:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:39:51:WU03:FS01:Trying to send results to collection server
2019-10-01:00:39:51:WU03:FS01:Uploading 51.38MiB to 155.247.166.219
2019-10-01:00:39:51:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:39:52:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-01:00:39:53:WU02:FS01:Upload 0.48%
2019-10-01:00:39:57:WU04:FS01:Upload 0.27%
2019-10-01:00:40:00:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14229 run:500 clone:0 gen:46 core:0x21 unit:0x0000004880fccb0a5d6553da6655cf13
2019-10-01:00:40:00:WU03:FS01:Uploading 51.38MiB to 128.252.203.10
2019-10-01:00:40:00:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:40:03:WU02:FS01:Upload 0.73%
2019-10-01:00:40:11:WU01:FS01:Upload 0.28%
2019-10-01:00:40:12:WU03:FS01:Upload 0.12%
2019-10-01:00:40:15:WU04:FS01:Upload 0.33%
2019-10-01:00:40:18:WU02:FS01:Upload 0.97%
2019-10-01:00:40:21:WU04:FS01:Upload 0.40%
2019-10-01:00:40:22:WU01:FS01:Upload 0.47%
2019-10-01:00:40:24:WU02:FS01:Upload 1.09%
2019-10-01:00:40:25:WU03:FS01:Upload 0.36%
2019-10-01:00:40:30:WU04:FS01:Upload 0.53%
2019-10-01:00:40:33:WU03:FS01:Upload 0.61%
2019-10-01:00:40:34:WU01:FS01:Upload 0.76%
2019-10-01:00:40:39:WU04:FS01:Upload 0.73%
2019-10-01:00:40:40:WU02:FS01:Upload 1.33%
2019-10-01:00:40:46:WU04:FS01:Upload 0.93%
2019-10-01:00:40:50:WU01:FS01:Upload 1.04%
2019-10-01:00:40:53:WU04:FS01:Upload 1.07%
2019-10-01:00:40:59:WU01:FS01:Upload 1.23%
2019-10-01:00:41:00:WU03:FS01:Upload 0.85%
2019-10-01:00:41:02:WU04:FS01:Upload 1.27%
2019-10-01:00:41:03:WU00:FS01:Upload 1.36%
2019-10-01:00:41:03:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:41:03:WU00:FS01:Trying to send results to collection server
2019-10-01:00:41:03:WU00:FS01:Uploading 69.10MiB to 155.247.166.219
2019-10-01:00:41:03:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:41:04:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-01:00:41:04:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14233 run:5 clone:43 gen:94 core:0x21 unit:0x0000007780fccb0a5d65490f2bd0992e
2019-10-01:00:41:04:WU00:FS01:Uploading 69.10MiB to 128.252.203.10
2019-10-01:00:41:04:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:41:07:WU02:FS01:Upload 1.45%
2019-10-01:00:41:07:WU03:FS01:Upload 1.09%
2019-10-01:00:41:09:WU04:FS01:Upload 1.47%
2019-10-01:00:41:11:WU00:FS01:Upload 0.27%
2019-10-01:00:41:13:WU03:FS01:Upload 1.34%
2019-10-01:00:41:16:WU04:FS01:Upload 1.60%
2019-10-01:00:41:16:WU02:FS01:Upload 1.70%
2019-10-01:00:41:18:WU00:FS01:Upload 0.54%
2019-10-01:00:41:21:WU01:FS01:Upload 1.33%
2019-10-01:00:41:21:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:41:21:WU01:FS01:Trying to send results to collection server
2019-10-01:00:41:21:WU01:FS01:Uploading 65.87MiB to 155.247.166.219
2019-10-01:00:41:21:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:41:22:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-01:00:41:22:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:66 clone:0 gen:33 core:0x21 unit:0x0000002980fccb0a5d6ed20de7fad410
2019-10-01:00:41:22:WU01:FS01:Uploading 65.87MiB to 128.252.203.10
2019-10-01:00:41:22:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:41:23:WU03:FS01:Upload 1.58%
2019-10-01:00:41:24:WU02:FS01:Upload 1.82%
2019-10-01:00:41:24:WU04:FS01:Upload 1.87%
2019-10-01:00:41:31:WU03:FS01:Upload 1.95%
2019-10-01:00:41:33:WU01:FS01:Upload 0.19%
2019-10-01:00:41:36:WU04:FS01:Upload 1.93%
2019-10-01:00:41:38:WU03:FS01:Upload 2.19%
2019-10-01:00:41:38:WU00:FS01:Upload 0.72%
2019-10-01:00:41:44:WU04:FS01:Upload 2.07%
2019-10-01:00:41:44:WU00:FS01:Upload 0.81%
2019-10-01:00:41:47:WU03:FS01:Upload 2.31%
2019-10-01:00:41:49:WU02:FS01:Upload 1.94%
2019-10-01:00:41:50:WU04:FS01:Upload 2.27%
2019-10-01:00:41:51:WU00:FS01:Upload 0.99%
2019-10-01:00:41:52:WU01:FS01:Upload 0.28%
2019-10-01:00:41:54:WU03:FS01:Upload 2.43%
2019-10-01:00:41:57:WU02:FS01:Upload 2.18%
2019-10-01:00:42:00:WU00:FS01:Upload 1.27%
2019-10-01:00:42:03:WU04:FS01:Upload 2.40%
2019-10-01:00:42:06:WU00:FS01:Upload 1.36%
2019-10-01:00:42:09:WU01:FS01:Upload 0.47%
2019-10-01:00:42:09:WU02:FS01:Upload 2.54%
2019-10-01:00:42:13:WU00:FS01:Upload 1.54%
2019-10-01:00:42:16:WU04:FS01:Upload 2.47%
2019-10-01:00:42:17:WU01:FS01:Upload 0.66%
2019-10-01:00:42:20:WU03:FS01:Upload 2.55%
2019-10-01:00:42:21:WU00:FS01:Upload 1.81%
2019-10-01:00:42:26:WU03:FS01:Upload 2.68%
2019-10-01:00:42:27:WU04:FS01:Upload 2.60%
2019-10-01:00:42:33:WU01:FS01:Upload 0.76%
2019-10-01:00:42:33:WU00:FS01:Upload 1.90%
2019-10-01:00:42:33:WU04:FS01:Upload 2.73%
2019-10-01:00:42:36:WU03:FS01:Upload 3.04%
2019-10-01:00:42:40:WU04:FS01:Upload 2.93%
2019-10-01:00:42:41:WU01:FS01:Upload 0.95%
2019-10-01:00:42:41:WU00:FS01:Upload 2.17%
2019-10-01:00:42:42:WU03:FS01:Upload 3.28%
2019-10-01:00:42:47:WU01:FS01:Upload 1.14%
2019-10-01:00:42:47:WU04:FS01:Upload 3.07%
2019-10-01:00:42:52:WU03:FS01:Upload 3.65%
2019-10-01:00:42:53:WU01:FS01:Upload 1.23%
2019-10-01:00:42:54:WU04:FS01:Upload 3.33%
2019-10-01:00:43:00:WU00:FS01:Upload 2.26%
2019-10-01:00:43:01:WU03:FS01:Upload 3.89%
2019-10-01:00:43:04:WU04:FS01:Upload 3.53%
2019-10-01:00:43:06:WU01:FS01:Upload 1.33%
2019-10-01:00:43:07:WU00:FS01:Upload 2.35%
2019-10-01:00:43:11:WU04:FS01:Upload 3.67%
2019-10-01:00:43:12:WU01:FS01:Upload 1.52%
2019-10-01:00:43:16:WU03:FS01:Upload 4.26%
2019-10-01:00:43:18:WU01:FS01:Upload 1.61%
2019-10-01:00:43:18:WU00:FS01:Upload 2.44%
2019-10-01:00:43:22:WU04:FS01:Upload 3.87%
2019-10-01:00:43:23:WU03:FS01:Upload 4.62%
2019-10-01:00:43:24:WU01:FS01:Upload 1.80%
2019-10-01:00:43:26:WU00:FS01:Upload 2.71%
2019-10-01:00:43:29:WU02:FS01:Upload 2.79%
2019-10-01:00:43:30:WU03:FS01:Upload 4.87%
2019-10-01:00:43:30:WU04:FS01:Upload 4.07%
2019-10-01:00:43:31:WU01:FS01:Upload 1.99%
2019-10-01:00:43:36:WU00:FS01:Upload 2.80%
2019-10-01:00:43:37:WU02:FS01:Upload 2.91%
2019-10-01:00:43:37:WU04:FS01:Upload 4.27%
2019-10-01:00:43:38:WU01:FS01:Upload 2.09%
2019-10-01:00:43:42:WU00:FS01:Upload 2.89%
2019-10-01:00:43:43:WU03:FS01:Upload 5.11%
2019-10-01:00:43:44:WU01:FS01:Upload 2.28%
2019-10-01:00:43:44:WU02:FS01:Upload 3.27%
2019-10-01:00:43:49:WU00:FS01:Upload 3.17%
2019-10-01:00:43:50:WU01:FS01:Upload 2.47%
2019-10-01:00:43:51:WU03:FS01:Upload 5.47%
2019-10-01:00:43:52:WU02:FS01:Upload 3.39%
2019-10-01:00:43:57:WU00:FS01:Upload 3.35%
2019-10-01:00:44:00:WU02:FS01:Upload 3.63%
2019-10-01:00:44:02:WU03:FS01:Upload 5.60%
2019-10-01:00:44:04:WU00:FS01:Upload 3.71%
2019-10-01:00:44:08:WU02:FS01:Upload 4.00%
2019-10-01:00:44:08:WU01:FS01:Upload 2.85%
2019-10-01:00:44:09:WU03:FS01:Upload 5.84%
2019-10-01:00:44:14:WU04:FS01:Upload 4.40%
2019-10-01:00:44:15:WU01:FS01:Upload 2.94%
2019-10-01:00:44:16:WU02:FS01:Upload 4.12%
2019-10-01:00:44:18:WU03:FS01:Upload 6.08%
2019-10-01:00:44:21:WU04:FS01:Upload 4.60%
2019-10-01:00:44:22:WU02:FS01:Upload 4.36%
2019-10-01:00:44:22:WU01:FS01:Upload 3.04%
2019-10-01:00:44:24:WU03:FS01:Upload 6.33%
2019-10-01:00:44:30:WU02:FS01:Upload 4.60%
2019-10-01:00:44:32:WU03:FS01:Upload 6.57%
2019-10-01:00:44:38:WU03:FS01:Upload 6.69%
2019-10-01:00:44:38:WU02:FS01:Upload 4.97%
2019-10-01:00:44:44:WU02:FS01:Upload 5.21%
2019-10-01:00:44:44:WU03:FS01:Upload 6.81%
2019-10-01:00:44:50:WU03:FS01:Upload 7.06%
2019-10-01:00:44:51:WU02:FS01:Upload 5.45%
2019-10-01:00:44:57:WU03:FS01:Upload 7.30%
2019-10-01:00:44:58:WU02:FS01:Upload 5.82%
2019-10-01:00:45:03:WU03:FS01:Upload 7.54%
2019-10-01:00:45:04:WU02:FS01:Upload 5.94%
2019-10-01:00:45:09:WU03:FS01:Upload 7.91%
2019-10-01:00:45:11:WU01:FS01:Upload 3.13%
2019-10-01:00:45:16:WU03:FS01:Upload 8.39%
2019-10-01:00:45:20:WU02:FS01:Upload 6.06%
2019-10-01:00:45:20:WU01:FS01:Upload 3.51%
2019-10-01:00:45:26:WU01:FS01:Upload 3.61%
2019-10-01:00:45:27:WU03:FS01:Upload 8.51%
2019-10-01:00:45:32:WU01:FS01:Upload 3.80%
2019-10-01:00:45:35:WU03:FS01:Upload 8.76%
2019-10-01:00:45:37:WU02:FS01:Upload 6.30%
2019-10-01:00:45:38:WU01:FS01:Upload 4.08%
2019-10-01:00:45:44:WU01:FS01:Upload 4.36%
2019-10-01:00:45:48:WU02:FS01:Upload 6.78%
2019-10-01:00:45:52:WU01:FS01:Upload 4.65%
2019-10-01:00:45:52:WU00:FS01:Upload 3.80%
2019-10-01:00:45:52:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:45:52:WU00:FS01:Trying to send results to collection server
2019-10-01:00:45:52:WU00:FS01:Uploading 69.10MiB to 155.247.166.219
2019-10-01:00:45:52:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:45:52:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-01:00:45:52:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14233 run:5 clone:43 gen:94 core:0x21 unit:0x0000007780fccb0a5d65490f2bd0992e
2019-10-01:00:45:52:WU00:FS01:Uploading 69.10MiB to 128.252.203.10
2019-10-01:00:45:52:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:45:58:WU02:FS01:Upload 7.03%
2019-10-01:00:45:58:WU01:FS01:Upload 4.84%
2019-10-01:00:46:00:WU00:FS01:Upload 0.09%
2019-10-01:00:46:04:WU02:FS01:Upload 7.63%
2019-10-01:00:46:06:WU00:FS01:Upload 0.27%
2019-10-01:00:46:14:WU02:FS01:Upload 8.00%
2019-10-01:00:46:15:WU00:FS01:Upload 0.36%
2019-10-01:00:46:15:WU04:FS01:Upload 4.74%
2019-10-01:00:46:15:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:46:15:WU04:FS01:Trying to send results to collection server
2019-10-01:00:46:15:WU04:FS01:Uploading 93.71MiB to 155.247.166.220
2019-10-01:00:46:15:WU04:FS01:Connecting to 155.247.166.220:8080
2019-10-01:00:46:17:WU01:FS01:Upload 5.03%
2019-10-01:00:46:23:WU04:FS01:Upload 0.07%
2019-10-01:00:46:23:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-01:00:46:23:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14227 run:258 clone:1 gen:12 core:0x21 unit:0x0000001780fccb0a5d6545e0721b8725
2019-10-01:00:46:23:WU04:FS01:Uploading 93.71MiB to 128.252.203.10
2019-10-01:00:46:23:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:46:45:WU01:FS01:Upload 5.12%
2019-10-01:00:46:52:WU01:FS01:Upload 5.50%
2019-10-01:00:46:52:WU03:FS01:Upload 8.88%
2019-10-01:00:46:52:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:46:52:WU03:FS01:Trying to send results to collection server
2019-10-01:00:46:52:WU03:FS01:Uploading 51.38MiB to 155.247.166.219
2019-10-01:00:46:52:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:46:52:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-01:00:46:53:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14229 run:500 clone:0 gen:46 core:0x21 unit:0x0000004880fccb0a5d6553da6655cf13
2019-10-01:00:46:53:WU03:FS01:Uploading 51.38MiB to 128.252.203.10
2019-10-01:00:46:53:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:46:56:WU02:FS01:Upload 8.12%
2019-10-01:00:46:59:WU03:FS01:Upload 0.24%
2019-10-01:00:46:59:WU01:FS01:Upload 5.79%
2019-10-01:00:47:06:WU02:FS01:Upload 8.24%
2019-10-01:00:47:07:WU01:FS01:Upload 6.07%
2019-10-01:00:47:14:WU02:FS01:Upload 8.60%
2019-10-01:00:47:17:WU01:FS01:Upload 6.36%
2019-10-01:00:47:19:WU04:FS01:Upload 0.07%
2019-10-01:00:47:21:WU03:FS01:Upload 0.36%
2019-10-01:00:47:21:WU02:FS01:Upload 8.97%
2019-10-01:00:47:25:WU01:FS01:Upload 6.64%
2019-10-01:00:47:28:WU02:FS01:Upload 9.33%
2019-10-01:00:47:31:WU01:FS01:Upload 6.83%
2019-10-01:00:47:35:WU02:FS01:Upload 9.57%
2019-10-01:00:47:39:WU01:FS01:Upload 7.02%
2019-10-01:00:47:39:WU04:FS01:Upload 0.13%
2019-10-01:00:47:42:WU02:FS01:Upload 10.06%
2019-10-01:00:47:47:WU01:FS01:Upload 7.21%
2019-10-01:00:47:48:WU02:FS01:Upload 10.66%
2019-10-01:00:47:49:WU04:FS01:Upload 0.20%
2019-10-01:00:47:53:WU01:FS01:Upload 7.40%
2019-10-01:00:47:55:WU04:FS01:Upload 0.33%
2019-10-01:00:47:59:WU02:FS01:Upload 11.02%
2019-10-01:00:48:00:WU01:FS01:Upload 7.59%
2019-10-01:00:48:08:WU01:FS01:Upload 7.78%
2019-10-01:00:48:08:WU02:FS01:Upload 11.39%
2019-10-01:00:48:14:WU02:FS01:Upload 11.75%
2019-10-01:00:48:14:WU01:FS01:Upload 7.97%
2019-10-01:00:48:21:WU01:FS01:Upload 8.35%
2019-10-01:00:48:21:WU02:FS01:Upload 11.87%
2019-10-01:00:48:28:WU02:FS01:Upload 12.24%
2019-10-01:00:48:35:WU02:FS01:Upload 12.48%
2019-10-01:00:48:38:WU00:FS01:Upload 0.45%
2019-10-01:00:48:39:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:48:39:WU00:FS01:Trying to send results to collection server
2019-10-01:00:48:39:WU00:FS01:Uploading 69.10MiB to 155.247.166.219
2019-10-01:00:48:39:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:48:39:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-01:00:48:39:WU01:FS01:Upload 8.44%
2019-10-01:00:48:46:WU01:FS01:Upload 8.73%
2019-10-01:00:48:47:WU03:FS01:Upload 0.61%
2019-10-01:00:48:54:WU03:FS01:Upload 0.97%
2019-10-01:00:48:57:WU01:FS01:Upload 9.01%
2019-10-01:00:49:04:WU03:FS01:Upload 1.34%
2019-10-01:00:49:11:WU03:FS01:Upload 1.58%
2019-10-01:00:49:21:WU03:FS01:Upload 1.70%
2019-10-01:00:49:29:WU03:FS01:Upload 1.82%
2019-10-01:00:49:32:WU01:FS01:Upload 9.30%
2019-10-01:00:49:32:WU04:FS01:Upload 0.40%
2019-10-01:00:49:32:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:49:32:WU04:FS01:Trying to send results to collection server
2019-10-01:00:49:32:WU04:FS01:Uploading 93.71MiB to 155.247.166.220
2019-10-01:00:49:32:WU04:FS01:Connecting to 155.247.166.220:8080
2019-10-01:00:49:44:WU04:FS01:Upload 0.07%
2019-10-01:00:49:45:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-01:00:49:45:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14227 run:258 clone:1 gen:12 core:0x21 unit:0x0000001780fccb0a5d6545e0721b8725
2019-10-01:00:49:45:WU04:FS01:Uploading 93.71MiB to 128.252.203.10
2019-10-01:00:49:45:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:49:47:WU03:FS01:Upload 1.95%
2019-10-01:00:49:55:WU03:FS01:Upload 2.19%
2019-10-01:00:50:06:WU03:FS01:Upload 2.31%
2019-10-01:00:50:07:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14233 run:5 clone:43 gen:94 core:0x21 unit:0x0000007780fccb0a5d65490f2bd0992e
2019-10-01:00:50:07:WU00:FS01:Uploading 69.10MiB to 128.252.203.10
2019-10-01:00:50:07:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:50:10:WU01:FS01:Upload 9.39%
2019-10-01:00:50:10:WU02:FS01:Upload 12.84%
2019-10-01:00:50:10:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:50:10:WU02:FS01:Trying to send results to collection server
2019-10-01:00:50:10:WU02:FS01:Uploading 51.59MiB to 155.247.166.219
2019-10-01:00:50:10:WU02:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:50:10:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-01:00:50:11:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14230 run:526 clone:0 gen:28 core:0x21 unit:0x0000002d80fccb0a5d654c5fb2a6a95c
2019-10-01:00:50:11:WU02:FS01:Uploading 51.59MiB to 128.252.203.10
2019-10-01:00:50:11:WU02:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:50:14:WU04:FS01:Upload 0.07%
2019-10-01:00:50:14:WU00:FS01:Upload 0.18%
2019-10-01:00:50:15:WU03:FS01:Upload 2.55%
2019-10-01:00:50:22:WU02:FS01:Upload 0.24%
2019-10-01:00:50:24:WU00:FS01:Upload 0.36%
2019-10-01:00:50:27:WU01:FS01:Upload 9.58%
2019-10-01:00:50:30:WU00:FS01:Upload 0.54%
2019-10-01:00:50:31:WU02:FS01:Upload 0.36%
2019-10-01:00:50:34:WU01:FS01:Upload 9.87%
2019-10-01:00:50:36:WU03:FS01:Upload 2.68%
2019-10-01:00:50:37:WU02:FS01:Upload 0.61%
2019-10-01:00:50:39:WU04:FS01:Upload 0.13%
2019-10-01:00:50:40:WU01:FS01:Upload 10.06%
2019-10-01:00:50:44:WU03:FS01:Upload 2.92%
2019-10-01:00:50:44:WU00:FS01:Upload 0.81%
2019-10-01:00:50:48:WU01:FS01:Upload 10.34%
2019-10-01:00:50:50:WU00:FS01:Upload 1.09%
2019-10-01:00:50:51:WU03:FS01:Upload 3.16%
2019-10-01:00:50:54:WU01:FS01:Upload 10.63%
2019-10-01:00:50:58:WU00:FS01:Upload 1.36%
2019-10-01:00:50:59:WU03:FS01:Upload 3.41%
2019-10-01:00:50:59:WU02:FS01:Upload 0.85%
2019-10-01:00:51:05:WU02:FS01:Upload 1.09%
2019-10-01:00:51:09:WU03:FS01:Upload 3.77%
2019-10-01:00:51:11:WU02:FS01:Upload 1.33%
2019-10-01:00:51:11:WU00:FS01:Upload 1.45%
2019-10-01:00:51:13:WU04:FS01:Upload 0.20%
2019-10-01:00:51:16:WU03:FS01:Upload 4.14%
2019-10-01:00:51:18:WU00:FS01:Upload 1.63%
2019-10-01:00:51:21:WU04:FS01:Upload 0.27%
2019-10-01:00:51:23:WU03:FS01:Upload 4.26%
2019-10-01:00:51:25:WU00:FS01:Upload 1.81%
2019-10-01:00:51:29:WU02:FS01:Upload 1.57%
2019-10-01:00:51:31:WU00:FS01:Upload 1.99%
2019-10-01:00:51:34:WU03:FS01:Upload 4.50%
2019-10-01:00:51:36:WU02:FS01:Upload 1.94%
2019-10-01:00:51:37:WU00:FS01:Upload 2.26%
2019-10-01:00:51:41:WU03:FS01:Upload 4.87%
2019-10-01:00:51:43:WU00:FS01:Upload 2.62%
2019-10-01:00:51:44:WU04:FS01:Upload 0.40%
2019-10-01:00:51:44:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:51:44:WU04:FS01:Trying to send results to collection server
2019-10-01:00:51:44:WU04:FS01:Uploading 93.71MiB to 155.247.166.220
2019-10-01:00:51:44:WU04:FS01:Connecting to 155.247.166.220:8080
2019-10-01:00:51:45:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-01:00:51:49:WU02:FS01:Upload 2.30%
2019-10-01:00:51:50:WU03:FS01:Upload 5.35%
2019-10-01:00:51:54:WU00:FS01:Upload 2.71%
2019-10-01:00:51:55:WU02:FS01:Upload 2.67%
2019-10-01:00:52:00:WU03:FS01:Upload 5.60%
2019-10-01:00:52:04:WU02:FS01:Upload 3.03%
2019-10-01:00:52:09:WU03:FS01:Upload 5.84%
2019-10-01:00:52:13:WU02:FS01:Upload 3.39%
2019-10-01:00:52:16:WU00:FS01:Upload 2.80%
2019-10-01:00:52:32:WU01:FS01:Upload 10.72%
2019-10-01:00:52:33:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:52:33:WU01:FS01:Trying to send results to collection server
2019-10-01:00:52:33:WU01:FS01:Uploading 65.87MiB to 155.247.166.219
2019-10-01:00:52:33:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:52:41:WU01:FS01:Upload 0.09%
2019-10-01:00:52:41:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-01:00:52:41:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:66 clone:0 gen:33 core:0x21 unit:0x0000002980fccb0a5d6ed20de7fad410
2019-10-01:00:52:41:WU01:FS01:Uploading 65.87MiB to 128.252.203.10
2019-10-01:00:52:41:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:52:51:WU00:FS01:Upload 2.89%
2019-10-01:00:52:52:WU02:FS01:Upload 3.51%
2019-10-01:00:53:10:WU00:FS01:Upload 2.98%
2019-10-01:00:53:14:WU01:FS01:Upload 0.09%
2019-10-01:00:53:17:WU00:FS01:Upload 3.08%
2019-10-01:00:53:31:WU01:FS01:Upload 0.19%
2019-10-01:00:53:41:WU03:FS01:Upload 6.08%
2019-10-01:00:53:41:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:53:41:WU03:FS01:Trying to send results to collection server
2019-10-01:00:53:41:WU03:FS01:Uploading 51.38MiB to 155.247.166.219
2019-10-01:00:53:41:WU03:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:53:41:WU01:FS01:Upload 0.38%
2019-10-01:00:53:42:ERROR:WU03:FS01:Exception: Transfer failed
2019-10-01:00:53:42:WU00:FS01:Upload 3.17%
2019-10-01:00:53:42:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:14229 run:500 clone:0 gen:46 core:0x21 unit:0x0000004880fccb0a5d6553da6655cf13
2019-10-01:00:53:42:WU03:FS01:Uploading 51.38MiB to 128.252.203.10
2019-10-01:00:53:42:WU03:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:53:48:WU03:FS01:Upload 0.24%
2019-10-01:00:53:49:WU00:FS01:Upload 3.26%
2019-10-01:00:53:56:WU00:FS01:Upload 3.44%
2019-10-01:00:53:57:WU03:FS01:Upload 0.61%
2019-10-01:00:53:59:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14227 run:258 clone:1 gen:12 core:0x21 unit:0x0000001780fccb0a5d6545e0721b8725
2019-10-01:00:53:59:WU04:FS01:Uploading 93.71MiB to 128.252.203.10
2019-10-01:00:53:59:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:54:02:WU00:FS01:Upload 3.53%
2019-10-01:00:54:02:WU02:FS01:Upload 3.76%
2019-10-01:00:54:04:WU03:FS01:Upload 1.34%
2019-10-01:00:54:18:WU01:FS01:Upload 0.47%
2019-10-01:00:54:26:WU03:FS01:Upload 1.58%
2019-10-01:00:54:38:WU04:FS01:Upload 0.33%
2019-10-01:00:54:42:WU03:FS01:Upload 1.70%
2019-10-01:00:54:42:WU02:FS01:Upload 4.12%
2019-10-01:00:54:44:WU00:FS01:Upload 4.07%
2019-10-01:00:54:48:WU03:FS01:Upload 1.82%
2019-10-01:00:54:48:WU02:FS01:Upload 4.24%
2019-10-01:00:54:54:WU02:FS01:Upload 4.48%
2019-10-01:00:54:55:WU01:FS01:Upload 0.57%
2019-10-01:00:54:57:WU04:FS01:Upload 0.40%
2019-10-01:00:55:04:WU00:FS01:Upload 4.16%
2019-10-01:00:55:10:WU02:FS01:Upload 4.85%
2019-10-01:00:55:10:WU01:FS01:Upload 0.66%
2019-10-01:00:55:12:WU00:FS01:Upload 4.34%
2019-10-01:00:55:16:WU04:FS01:Upload 0.47%
2019-10-01:00:55:16:WU02:FS01:Upload 5.09%
2019-10-01:00:55:21:WU00:FS01:Upload 4.52%
2019-10-01:00:55:24:WU03:FS01:Upload 1.95%
2019-10-01:00:55:25:WU04:FS01:Upload 0.60%
2019-10-01:00:55:25:WU02:FS01:Upload 5.33%
2019-10-01:00:55:30:WU03:FS01:Upload 2.19%
2019-10-01:00:55:30:WU00:FS01:Upload 4.61%
2019-10-01:00:55:31:WU02:FS01:Upload 5.57%
2019-10-01:00:55:37:WU02:FS01:Upload 5.82%
2019-10-01:00:55:37:WU03:FS01:Upload 2.31%
2019-10-01:00:55:38:WU00:FS01:Upload 4.79%
2019-10-01:00:55:43:WU01:FS01:Upload 0.76%
2019-10-01:00:55:43:WU02:FS01:Upload 6.18%
2019-10-01:00:55:46:WU03:FS01:Upload 2.55%
2019-10-01:00:55:50:WU00:FS01:Upload 4.88%
2019-10-01:00:55:50:WU02:FS01:Upload 6.42%
2019-10-01:00:55:53:WU03:FS01:Upload 2.68%
2019-10-01:00:55:53:WU01:FS01:Upload 1.04%
2019-10-01:00:56:02:WU03:FS01:Upload 3.04%
2019-10-01:00:56:03:WU01:FS01:Upload 1.33%
2019-10-01:00:56:04:WU00:FS01:Upload 5.16%
2019-10-01:00:56:08:WU03:FS01:Upload 3.28%
2019-10-01:00:56:14:WU03:FS01:Upload 3.41%
2019-10-01:00:56:16:WU01:FS01:Upload 1.52%
2019-10-01:00:56:17:WU00:FS01:Upload 5.34%
2019-10-01:00:56:20:WU02:FS01:Upload 6.54%
2019-10-01:00:56:21:WU03:FS01:Upload 3.53%
2019-10-01:00:56:23:WU00:FS01:Upload 5.52%
2019-10-01:00:56:27:WU03:FS01:Upload 3.77%
2019-10-01:00:56:32:WU00:FS01:Upload 5.70%
2019-10-01:00:56:34:WU03:FS01:Upload 4.01%
2019-10-01:00:56:36:WU02:FS01:Upload 6.66%
2019-10-01:00:56:39:WU00:FS01:Upload 6.06%
2019-10-01:00:56:44:WU02:FS01:Upload 6.91%
2019-10-01:00:56:44:WU03:FS01:Upload 4.26%
2019-10-01:00:56:45:WU00:FS01:Upload 6.15%
2019-10-01:00:56:50:WU02:FS01:Upload 7.15%
2019-10-01:00:56:53:WU03:FS01:Upload 4.50%
2019-10-01:00:56:59:WU04:FS01:Upload 0.73%
2019-10-01:00:56:59:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:56:59:WU04:FS01:Trying to send results to collection server
2019-10-01:00:56:59:WU04:FS01:Uploading 93.71MiB to 155.247.166.220
2019-10-01:00:56:59:WU04:FS01:Connecting to 155.247.166.220:8080
2019-10-01:00:56:59:WU00:FS01:Upload 6.33%
2019-10-01:00:57:00:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-01:00:57:01:WU02:FS01:Upload 7.51%
2019-10-01:00:57:03:WU03:FS01:Upload 4.74%
2019-10-01:00:57:07:WU00:FS01:Upload 6.51%
2019-10-01:00:57:07:WU02:FS01:Upload 7.75%
2019-10-01:00:57:10:WU03:FS01:Upload 4.99%
2019-10-01:00:57:13:WU02:FS01:Upload 8.00%
2019-10-01:00:57:17:WU00:FS01:Upload 6.78%
2019-10-01:00:57:18:WU03:FS01:Upload 5.23%
2019-10-01:00:57:19:WU02:FS01:Upload 8.12%
2019-10-01:00:57:23:WU00:FS01:Upload 6.96%
2019-10-01:00:57:24:WU03:FS01:Upload 5.47%
2019-10-01:00:57:25:WU02:FS01:Upload 8.48%
2019-10-01:00:57:33:WU03:FS01:Upload 5.60%
2019-10-01:00:57:33:WU02:FS01:Upload 8.72%
2019-10-01:00:57:38:WU00:FS01:Upload 7.06%
2019-10-01:00:57:40:WU03:FS01:Upload 5.72%
2019-10-01:00:57:42:WU02:FS01:Upload 8.97%
2019-10-01:00:57:45:WU00:FS01:Upload 7.24%
2019-10-01:00:57:48:WU02:FS01:Upload 9.33%
2019-10-01:00:57:52:WU00:FS01:Upload 7.42%
2019-10-01:00:57:53:WU03:FS01:Upload 5.96%
2019-10-01:00:57:55:WU02:FS01:Upload 9.69%
2019-10-01:00:58:00:WU03:FS01:Upload 6.20%
2019-10-01:00:58:00:WU00:FS01:Upload 7.60%
2019-10-01:00:58:01:WU02:FS01:Upload 10.06%
2019-10-01:00:58:07:WU03:FS01:Upload 6.45%
2019-10-01:00:58:07:WU02:FS01:Upload 10.30%
2019-10-01:00:58:09:WU01:FS01:Upload 1.71%
2019-10-01:00:58:09:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:00:58:09:WU01:FS01:Trying to send results to collection server
2019-10-01:00:58:09:WU01:FS01:Uploading 65.87MiB to 155.247.166.219
2019-10-01:00:58:09:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-01:00:58:09:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-01:00:58:09:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:66 clone:0 gen:33 core:0x21 unit:0x0000002980fccb0a5d6ed20de7fad410
2019-10-01:00:58:09:WU01:FS01:Uploading 65.87MiB to 128.252.203.10
2019-10-01:00:58:09:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-01:00:58:13:WU03:FS01:Upload 6.69%
2019-10-01:00:58:15:WU02:FS01:Upload 10.78%
2019-10-01:00:58:16:WU01:FS01:Upload 0.28%
2019-10-01:00:58:35:WU01:FS01:Upload 0.38%
2019-10-01:00:58:48:WU02:FS01:Upload 11.15%
2019-10-01:00:58:49:WU00:FS01:Upload 7.87%
2019-10-01:00:58:55:WU02:FS01:Upload 11.27%
2019-10-01:00:58:59:WU00:FS01:Upload 7.96%
2019-10-01:00:59:02:WU03:FS01:Upload 6.93%
2019-10-01:00:59:04:WU02:FS01:Upload 11.39%
2019-10-01:00:59:06:WU00:FS01:Upload 8.05%
2019-10-01:00:59:10:WU03:FS01:Upload 7.06%
2019-10-01:00:59:10:WU02:FS01:Upload 11.63%
2019-10-01:00:59:13:WU00:FS01:Upload 8.14%
2019-10-01:00:59:16:WU02:FS01:Upload 11.99%
2019-10-01:00:59:20:WU00:FS01:Upload 8.32%
2019-10-01:00:59:23:WU02:FS01:Upload 12.36%
2019-10-01:00:59:25:WU01:FS01:Upload 0.47%
2019-10-01:00:59:29:WU03:FS01:Upload 7.18%
2019-10-01:00:59:32:WU00:FS01:Upload 8.50%
2019-10-01:00:59:32:WU02:FS01:Upload 12.60%
2019-10-01:00:59:44:WU01:FS01:Upload 0.57%
2019-10-01:01:00:00:WU00:FS01:Upload 8.59%
2019-10-01:01:00:05:WU02:FS01:Upload 12.72%
2019-10-01:01:00:11:WU01:FS01:Upload 0.66%
2019-10-01:01:00:12:WU02:FS01:Upload 12.96%
2019-10-01:01:00:12:WU03:FS01:Upload 7.30%
2019-10-01:01:00:19:WU02:FS01:Upload 13.08%
2019-10-01:01:00:26:WU00:FS01:Upload 8.68%
2019-10-01:01:00:27:WU01:FS01:Upload 0.76%
2019-10-01:01:00:29:WU02:FS01:Upload 13.33%
2019-10-01:01:00:35:WU02:FS01:Upload 13.57%
2019-10-01:01:00:36:WU00:FS01:Upload 8.86%
2019-10-01:01:00:42:WU02:FS01:Upload 13.81%
2019-10-01:01:00:43:WU01:FS01:Upload 0.85%
2019-10-01:01:00:46:WU03:FS01:Upload 7.42%
2019-10-01:01:00:50:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14227 run:258 clone:1 gen:12 core:0x21 unit:0x0000001780fccb0a5d6545e0721b8725
2019-10-01:01:00:50:WU04:FS01:Uploading 93.71MiB to 128.252.203.10
2019-10-01:01:00:50:WU04:FS01:Connecting to 128.252.203.10:8080
2019-10-01:01:00:54:WU00:FS01:Upload 9.05%
2019-10-01:01:00:57:WU01:FS01:Upload 1.14%
2019-10-01:01:00:59:WU03:FS01:Upload 7.66%
2019-10-01:01:01:00:WU00:FS01:Upload 9.23%
2019-10-01:01:01:03:WU01:FS01:Upload 1.42%
2019-10-01:01:01:04:WU04:FS01:Upload 0.07%
2019-10-01:01:01:06:WU00:FS01:Upload 9.41%
2019-10-01:01:01:07:WU03:FS01:Upload 8.03%
2019-10-01:01:01:09:WU01:FS01:Upload 1.61%
2019-10-01:01:01:13:WU00:FS01:Upload 9.59%
2019-10-01:01:01:18:WU03:FS01:Upload 8.15%
2019-10-01:01:01:19:WU00:FS01:Upload 9.77%
2019-10-01:01:01:25:WU00:FS01:Upload 9.86%
2019-10-01:01:01:25:WU03:FS01:Upload 8.39%
2019-10-01:01:01:31:WU03:FS01:Upload 8.64%
2019-10-01:01:01:33:WU02:FS01:Upload 13.93%
2019-10-01:01:01:34:WU00:FS01:Upload 10.04%
2019-10-01:01:01:37:WU03:FS01:Upload 9.00%
2019-10-01:01:01:40:WU00:FS01:Upload 10.22%
2019-10-01:01:01:44:WU04:FS01:Upload 0.13%
2019-10-01:01:01:51:WU03:FS01:Upload 9.24%
2019-10-01:01:01:54:WU00:FS01:Upload 10.49%
2019-10-01:01:01:55:WU04:FS01:Upload 0.27%
2019-10-01:01:01:59:WU02:FS01:Upload 14.05%
2019-10-01:01:02:00:WU03:FS01:Upload 9.49%
2019-10-01:01:02:00:WU00:FS01:Upload 10.58%
2019-10-01:01:02:07:WU00:FS01:Upload 10.67%
2019-10-01:01:02:09:WU03:FS01:Upload 9.61%
2019-10-01:01:02:13:WU00:FS01:Upload 11.04%
2019-10-01:01:02:16:WU03:FS01:Upload 9.85%
2019-10-01:01:02:25:WU00:FS01:Upload 11.22%
2019-10-01:01:02:27:WU03:FS01:Upload 9.97%
2019-10-01:01:02:32:WU00:FS01:Upload 11.40%
2019-10-01:01:02:34:WU01:FS01:Upload 1.71%
2019-10-01:01:02:34:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:01:02:34:WU01:FS01:Trying to send results to collection server
2019-10-01:01:02:34:WU01:FS01:Uploading 65.87MiB to 155.247.166.219
2019-10-01:01:02:34:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-01:01:02:34:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-01:01:02:39:WU00:FS01:Upload 11.58%
2019-10-01:01:02:41:WU04:FS01:Upload 0.40%
2019-10-01:01:02:50:WU04:FS01:Upload 0.53%
2019-10-01:01:04:06:WU04:FS01:Upload 0.80%
2019-10-01:01:04:06:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-01:01:04:06:WU04:FS01:Trying to send results to collection server
2019-10-01:01:04:06:WU04:FS01:Uploading 93.71MiB to 155.247.166.220
2019-10-01:01:04:06:WU04:FS01:Connecting to 155.247.166.220:8080
2019-10-01:01:04:07:ERROR:WU04:FS01:Exception: Transfer failed
Top
kwerboom
Posts: 58
Joined: Mon Dec 12, 2011 7:31 pm

Re: New Work Units Fail to Complete Download

Post by kwerboom »

Oh thank God, I'm not the only one with this problem. I've been hit by this problem on and off all last month. I just thought it was my crappy rural DSL connection. Then these past few hours, every WU upload has resulted in a stalled download and a restart to my computer (which uses Windows 10 1903). Pausing and Unpausing does nothing to convince FaH Client to start looking again. Does anyone know what's going on?
Image
Top
gordonbb
Posts: 511
Joined: Mon May 21, 2018 4:12 pm
Hardware configuration: Ubuntu 22.04.2 LTS; NVidia 525.60.11; 2 x 4070ti; 4070; 4060ti; 3x 3080; 3070ti; 3070
Location: Great White North

Re: Multiple WU's Fail to Upload

Post by gordonbb »

gordonbb wrote:Stalled uploads from 155.247.166.220 still occurring but much less frequent (I've only seen one today across 6 slots) so the adjustment of the assignment weight has helped.
With our folding month competition starting I have 12 GPU slots going and I’ve had to reboot 5 times in the last hour across 4 systems all failing to download from 155.247.166.220.

Only one successfully started a download but stalled part way through. The rest said downloading a 6MB file then no progress indicated and they just sat there
Image
Top
HaloJones
Posts: 906
Joined: Thu Jul 24, 2008 10:16 am

Re: New Work Units Fail to Complete Download

Post by HaloJones »

@Bruce, across all OS, downloads from 155.247.166.219 and 155.247.166.220 have been starting but just stopping. Only a reboot seems to fix it. Haven't had a problem for a few days so maybe it's fixed now.
single 1070

Image
Top
HaloJones
Posts: 906
Joined: Thu Jul 24, 2008 10:16 am

Re: New Work Units Fail to Complete Download

Post by HaloJones »

Spoke to soon!

Code: Select all

11:51:45:WU01:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
11:51:46:WU00:FS01:Connecting to 65.254.110.245:8080
11:51:46:WU00:FS01:Assigned to work server 155.247.166.220
11:51:46:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:GP104 [GeForce GTX 1070] 6463 from 155.247.166.220
11:51:46:WU00:FS01:Connecting to 155.247.166.220:8080
11:51:49:WU00:FS01:Downloading 3.17MiB
Upload worked fine but no further communication from 155.247.166.220 after six minutes. This is on 100mb/s fibre connection.

post reboot

Code: Select all

11:58:46:WU00:FS01:Connecting to 65.254.110.245:8080
11:58:48:WU00:FS01:Assigned to work server 128.252.203.10
11:58:48:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:GP104 [GeForce GTX 1070] 6463 from 128.252.203.10
11:58:48:WU00:FS01:Connecting to 128.252.203.10:8080
11:58:49:WU00:FS01:Downloading 6.55MiB
11:58:50:WU00:FS01:Download complete
11:58:51:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR
received 6MB wu in 1 second from a different server.
single 1070

Image
Top
parkut
Posts: 363
Joined: Tue Feb 12, 2008 7:33 am
Hardware configuration: Running exclusively Linux headless blades. All are dedicated crunching machines.
Location: SE Michigan, USA

Re: New Work Units Fail to Complete Download

Post by parkut »

Been seeing issues where downloads stall, and never complete. The log snippet is typical. A reboot usually fixes the problem
Model Name: GTX 960
Driver Version: 430.40
Gpu temp: 38C
Client Version: 7.5.1
Core Version: 0.0.20

Code: Select all

21:04:44:WU00:FS02:Download 7.21%
21:04:34:WU00:FS02:Download 4.12%
21:04:22:WU00:FS02:Download 3.09%
21:04:16:WU00:FS02:Download 2.06%
21:04:09:WU00:FS02:Downloading 6.07MiB
21:04:09:WU00:FS02:Connecting to 155.247.166.220:8080
21:04:09:WU00:FS02:Requesting new work unit for slot 02: RUNNING gpu:0:GM206 [GeForce GTX 960] 2308 from 155.247.166.220
21:04:09:WU00:FS02:Assigned to work server 155.247.166.220
21:04:08:WU00:FS02:Connecting to 65.254.110.245:8080
Top
bollix47
Posts: 2959
Joined: Sun Dec 02, 2007 5:04 am
Location: Canada

Re: Multiple WU's Fail to Upload

Post by bollix47 »

@Gordonbb

When you say you rebooted 5 times was that just the computer involved or did you reboot your router and/or your router/modem too?

I haven't had any problems from that or any other server after turning off my router for 30 seconds.
Top
gordonbb
Posts: 511
Joined: Mon May 21, 2018 4:12 pm
Hardware configuration: Ubuntu 22.04.2 LTS; NVidia 525.60.11; 2 x 4070ti; 4070; 4060ti; 3x 3080; 3070ti; 3070
Location: Great White North

Re: Multiple WU's Fail to Upload

Post by gordonbb »

@bollix47 it was across 4 systems. I rebooted the systems but not the Asus Router running Tomato or the vDSL modem which is in bridge mode.

I can ping 155.247.166.220 and DNS resolution works OK and it certainly isn't an ARP issue :-) and a traceroute works OK.

Code: Select all

 traceroute to 155.247.166.220 (155.247.166.220), 30 hops max, 60 byte packets
 1  ******* removed *******  0.348 ms  0.452 ms  0.551 ms
 2  lo0-0-lns01-tor2.teksavvy.com (206.248.155.132)  11.073 ms  11.231 ms  11.326 ms
 3  ae0-2150-bdr01-tor.teksavvy.com (69.196.136.172)  11.704 ms  11.750 ms  11.828 ms
 4  toro-b1-link.telia.net (62.115.171.248)  12.247 ms  11.957 ms  12.417 ms
 5  cogent-ic-340548-toro-b1.c.telia.net (62.115.168.49)  12.532 ms  12.579 ms  12.786 ms
 6  be2994.ccr22.cle04.atlas.cogentco.com (154.54.31.233)  19.758 ms  18.420 ms  18.457 ms
 7  be2890.ccr42.jfk02.atlas.cogentco.com (154.54.82.246)  26.436 ms  26.168 ms  26.155 ms
 8  be2333.rcr22.phl01.atlas.cogentco.com (154.54.5.2)  28.193 ms  29.892 ms  29.648 ms
 9  be3164.rcr21.phl03.atlas.cogentco.com (154.54.25.142)  28.650 ms  29.080 ms  28.714 ms
10  38.140.148.74 (38.140.148.74)  28.344 ms  28.704 ms  28.781 ms
11  * * *
12  vav4.ocis.temple.edu (155.247.166.220)  29.321 ms  28.769 ms  29.098 ms
I had rebooted my router the other day for another reason but I'll try rebooting both it and the modem again.

Three more after I woke this morning:

Code: Select all

doc2:
11:35:52:WU01:FS00:0x21:Completed 4950000 out of 5000000 steps (99%)
11:35:53:WU00:FS00:Connecting to 65.254.110.245:8080
11:35:53:WU00:FS00:Assigned to work server 155.247.166.220
11:35:53:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:GP106 [GeForce GTX 1060 6GB] 4372 from 155.247.166.220
11:35:53:WU00:FS00:Connecting to 155.247.166.220:8080
11:35:53:WU00:FS00:Downloading 3.41MiB
11:37:37:WU01:FS00:0x21:Completed 5000000 out of 5000000 steps (100%)
11:37:37:WU01:FS00:0x21:Saving result file logfile_01.txt
11:37:37:WU01:FS00:0x21:Saving result file checkpointState.xml
11:37:37:WU01:FS00:0x21:Saving result file checkpt.crc
11:37:37:WU01:FS00:0x21:Saving result file log.txt
11:37:37:WU01:FS00:0x21:Saving result file positions.xtc
11:37:37:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
11:37:38:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:37:38:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:13816 run:0 clone:1599 gen:365 core:0x21 unit:0x0000020280fccb045b3a656bb6214630
11:37:38:WU01:FS00:Uploading 6.52MiB to 128.252.203.4
11:37:38:WU01:FS00:Connecting to 128.252.203.4:8080
11:37:43:WU01:FS00:Upload complete
11:37:43:WU01:FS00:Server responded WORK_ACK (400)
11:37:43:WU01:FS00:Final credit estimate, 53847.00 points
11:37:43:WU01:FS00:Cleaning up

... fold7 ...

10:36:00:WU00:FS01:0x21:Completed 495000 out of 500000 steps (99%)
10:36:01:WU02:FS01:Connecting to 65.254.110.245:8080
10:36:01:WU02:FS01:Assigned to work server 155.247.166.220
10:36:01:WU02:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
10:36:01:WU02:FS01:Connecting to 155.247.166.220:8080
10:36:01:WU02:FS01:Downloading 6.06MiB
10:36:28:WU00:FS01:0x21:Completed 500000 out of 500000 steps (100%)
10:36:32:WU00:FS01:0x21:Saving result file logfile_01.txt
10:36:32:WU00:FS01:0x21:Saving result file checkpointState.xml
10:36:32:WU00:FS01:0x21:Saving result file checkpt.crc
10:36:32:WU00:FS01:0x21:Saving result file log.txt
10:36:32:WU00:FS01:0x21:Saving result file positions.xtc
10:36:32:WU00:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
10:36:33:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
10:36:33:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14230 run:19 clone:0 gen:38 core:0x21 unit:0x0000003080fccb0a5d65517b00c3f965
10:36:33:WU00:FS01:Uploading 51.59MiB to 128.252.203.10
10:36:33:WU00:FS01:Connecting to 128.252.203.10:8080
10:36:39:WU00:FS01:Upload 16.11%
10:36:45:WU00:FS01:Upload 31.26%
10:36:51:WU00:FS01:Upload 46.16%
10:36:57:WU00:FS01:Upload 61.18%
10:37:03:WU00:FS01:Upload 75.48%
10:37:09:WU00:FS01:Upload 90.13%
10:37:15:WU00:FS01:Upload complete
10:37:15:WU00:FS01:Server responded WORK_ACK (400)
10:37:15:WU00:FS01:Final credit estimate, 51223.00 points
10:37:15:WU00:FS01:Cleaning up
******************************* Date: 2019-10-01 *******************************

... kvm1 ...

10:56:32:WU01:FS00:0x21:Completed 1980000 out of 2000000 steps (99%)
10:56:33:WU00:FS00:Connecting to 65.254.110.245:8080
10:56:33:WU00:FS00:Assigned to work server 155.247.166.220
10:56:33:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TU106 [GeForce RTX 2070 Rev. A] M 7465 from 155.247.166.220
10:56:33:WU00:FS00:Connecting to 155.247.166.220:8080
10:56:33:WU00:FS00:Downloading 2.50MiB
10:58:17:WU01:FS00:0x21:Completed 2000000 out of 2000000 steps (100%)
10:58:21:WU01:FS00:0x21:Saving result file logfile_01.txt
10:58:21:WU01:FS00:0x21:Saving result file checkpointState.xml
10:58:21:WU01:FS00:0x21:Saving result file checkpt.crc
10:58:21:WU01:FS00:0x21:Saving result file log.txt
10:58:21:WU01:FS00:0x21:Saving result file positions.xtc
10:58:21:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
10:58:21:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
10:58:21:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14224 run:8 clone:9 gen:55 core:0x21 unit:0x0000005380fccb0a5d6545d963396546
10:58:21:WU01:FS00:Uploading 89.45MiB to 128.252.203.10
10:58:21:WU01:FS00:Connecting to 128.252.203.10:8080
10:58:27:WU01:FS00:Upload 7.20%
10:58:33:WU01:FS00:Upload 16.00%
10:58:39:WU01:FS00:Upload 24.18%
10:58:45:WU01:FS00:Upload 32.77%
10:58:51:WU01:FS00:Upload 41.16%
10:58:57:WU01:FS00:Upload 49.61%
10:59:03:WU01:FS00:Upload 57.93%
10:59:09:WU01:FS00:Upload 66.24%
10:59:15:WU01:FS00:Upload 74.77%
10:59:21:WU01:FS00:Upload 83.22%
10:59:27:WU01:FS00:Upload 91.40%
10:59:33:WU01:FS00:Upload 99.99%
10:59:34:WU01:FS00:Upload complete
10:59:34:WU01:FS00:Server responded WORK_ACK (400)
10:59:34:WU01:FS00:Final credit estimate, 202944.00 points
10:59:34:WU01:FS00:Cleaning up
I rebooted my vDSL modem and then rebooted the three systems that had stuck slots. Unfortunately none of them connected to 155.247.166.220 so all came back.

I'll continue to monitor the rigs and if I have another failure I will then try rebooting the Asus Router but senior management (AKA THE wife) is home sick and is bing-watching NetFlix and will not be amused at another dint in the SLA :-)
Last edited by gordonbb on Tue Oct 01, 2019 4:20 pm, edited 1 time in total.
Image
Top
HaloJones
Posts: 906
Joined: Thu Jul 24, 2008 10:16 am

Re: New Work Units Fail to Complete Download

Post by HaloJones »

And again. Can these not be removed from the allocation servers?
single 1070

Image
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by bruce »

This slowness seems to have been identified as congestion that's associated with a specific campus. The problems being reported WERE associated with the work/collection servers 155.247.166.* at temple.edu and they've pretty much been fixed. If not, keep talking about them.

Perhaps there's a new problem being associated with servers 128.252.203.* at wustl.edu. That would be treated as an independent problem and they need to be categorized as such.

Whenever the problem can be associated with a single campus, FAH can refer the problem to the network admins of the appropriate campus.
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: New Work Units Fail to Complete Download

Post by bruce »

HaloJones wrote:And again. Can these not be removed from the allocation servers?
Not likely. The projects done by temple U will be associated with servers at *.temple.edu, There is currently no plan to put them on a server at (say) wustl.edu.

Each project has a global priority. One of the (temporary) fixes can be a priority reduction of projects at temple.edu which will divert more assignments to (say) wustl.edu.

Unfortunately such tuning adjustments take time to respond. Suppose I turn down the priority of specific projects and those projects involve 24 hrs to process. I won't see a reduction in uploads until the next day. Even then, not all WUs that have been assigned will be returned in exactly 24 hrs, so I'd probably wait to confirm "it worked"

Assuming that works and the congestion has been alleviated, I still need my projects to be processed so I'm going to gradually increase those priorities so it may take a week or two to balance the global network. Then it may take a month or two to identify exactly which piece of equipment needs more capacity -- and several months to allocate funds to buy a new router(??) if that's the problem.

I'm sorry to say that it also took a number of days to spot the fact that most of the complaints were associated with servers 155.247.166.*
Top
gordonbb
Posts: 511
Joined: Mon May 21, 2018 4:12 pm
Hardware configuration: Ubuntu 22.04.2 LTS; NVidia 525.60.11; 2 x 4070ti; 4070; 4060ti; 3x 3080; 3070ti; 3070
Location: Great White North

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by gordonbb »

had another failure. It started downloading then stalled at 29%

Code: Select all

17:44:16:WU02:FS00:0x21:Completed 495000 out of 500000 steps (99%)
17:44:16:WU00:FS00:Connecting to 65.254.110.245:8080
17:44:16:WU00:FS00:Assigned to work server 128.252.203.4
17:44:16:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TU106 [GeForce RTX 2070 Rev. A] M 7465 from 128.252.203.4
17:44:16:WU00:FS00:Connecting to 128.252.203.4:8080
17:44:16:ERROR:WU00:FS00:Exception: Server did not assign work unit
17:44:16:WU00:FS00:Connecting to 65.254.110.245:8080
17:44:16:WU00:FS00:Assigned to work server 155.247.166.220
17:44:16:WU00:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TU106 [GeForce RTX 2070 Rev. A] M 7465 from 155.247.166.220
17:44:16:WU00:FS00:Connecting to 155.247.166.220:8080
17:44:17:WU00:FS00:Downloading 2.61MiB
17:44:23:WU00:FS00:Download 14.38%
17:44:32:WU00:FS00:Download 19.18%
17:44:38:WU00:FS00:Download 21.57%
17:44:43:WU02:FS00:0x21:Completed 500000 out of 500000 steps (100%)
17:44:49:WU02:FS00:0x21:Saving result file logfile_01.txt
17:44:49:WU02:FS00:0x21:Saving result file checkpointState.xml
17:44:50:WU02:FS00:0x21:Saving result file checkpt.crc
17:44:50:WU02:FS00:0x21:Saving result file log.txt
17:44:50:WU02:FS00:0x21:Saving result file positions.xtc
17:44:50:WU02:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
17:44:50:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
17:44:50:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14230 run:184 clone:1 gen:32 core:0x21 unit:0x0000002f80fccb0a5d654fcadda8deb5
17:44:50:WU02:FS00:Uploading 51.62MiB to 128.252.203.10
17:44:50:WU02:FS00:Connecting to 128.252.203.10:8080
17:44:56:WU02:FS00:Upload 13.20%
17:45:02:WU02:FS00:Upload 27.97%
17:45:06:WU00:FS00:Download 23.97%
17:45:08:WU02:FS00:Upload 42.50%
17:45:14:WU02:FS00:Upload 56.79%
17:45:20:WU02:FS00:Upload 70.35%
17:45:26:WU02:FS00:Upload 81.85%
17:45:32:WU02:FS00:Upload 92.87%
17:45:37:WU02:FS00:Upload complete
17:45:37:WU02:FS00:Server responded WORK_ACK (400)
17:45:37:WU02:FS00:Final credit estimate, 51300.00 points
17:45:37:WU02:FS00:Cleaning up
Rebooting router ...
Image
Top
gordonbb
Posts: 511
Joined: Mon May 21, 2018 4:12 pm
Hardware configuration: Ubuntu 22.04.2 LTS; NVidia 525.60.11; 2 x 4070ti; 4070; 4060ti; 3x 3080; 3070ti; 3070
Location: Great White North

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by gordonbb »

Code: Select all

root@fold6:~# traceroute -I 155.247.166.220
traceroute to 155.247.166.220 (155.247.166.220), 30 hops max, 60 byte packets
 1  **** removed *****  0.346 ms  0.406 ms  0.505 ms
 2  lo0-0-lns03-tor.teksavvy.com (206.248.155.139)  11.240 ms  11.322 ms  11.413 ms
 3  ae1-2140-bdr01-tor2.teksavvy.com (69.196.136.138)  11.653 ms  11.861 ms  11.907 ms
 4  toro-b1-link.telia.net (62.115.61.240)  12.250 ms  12.340 ms  12.386 ms
 5  cogent-ic-340548-toro-b1.c.telia.net (62.115.168.49)  12.513 ms  12.559 ms  12.606 ms
 6  be2994.ccr22.cle04.atlas.cogentco.com (154.54.31.233)  20.260 ms  18.406 ms  18.712 ms
 7  be2890.ccr42.jfk02.atlas.cogentco.com (154.54.82.246)  26.045 ms  26.085 ms  25.712 ms
 8  be2333.rcr22.phl01.atlas.cogentco.com (154.54.5.2)  28.684 ms  28.796 ms  28.797 ms
 9  be3164.rcr21.phl03.atlas.cogentco.com (154.54.25.142)  29.139 ms  29.410 ms  29.476 ms
10  38.140.148.74 (38.140.148.74)  29.254 ms  29.302 ms  29.348 ms
11  * * *
12  vav4.ocis.temple.edu (155.247.166.220)  30.377 ms  29.152 ms  29.293 ms

Code: Select all

PING 155.247.166.220 (155.247.166.220) 56(84) bytes of data.
64 bytes from 155.247.166.220: icmp_seq=1 ttl=52 time=29.1 ms
64 bytes from 155.247.166.220: icmp_seq=2 ttl=52 time=28.8 ms
64 bytes from 155.247.166.220: icmp_seq=3 ttl=52 time=29.1 ms
64 bytes from 155.247.166.220: icmp_seq=4 ttl=52 time=29.1 ms
64 bytes from 155.247.166.220: icmp_seq=5 ttl=52 time=29.2 ms
64 bytes from 155.247.166.220: icmp_seq=6 ttl=52 time=28.9 ms
64 bytes from 155.247.166.220: icmp_seq=7 ttl=52 time=28.8 ms
64 bytes from 155.247.166.220: icmp_seq=8 ttl=52 time=29.2 ms
64 bytes from 155.247.166.220: icmp_seq=9 ttl=52 time=29.3 ms
64 bytes from 155.247.166.220: icmp_seq=10 ttl=52 time=28.9 ms
Does temple have a iPerf server? If so, I can run some tests from work on Inet2 and home on the Internet
Image
Top
Post Reply

Return to “Issues with a specific server”