Multiple WU's Fail downld/upld to 155.247.166.*

Moderators: Site Moderators, FAHC Science Team

Post Reply
gordonbb
Posts: 511
Joined: Mon May 21, 2018 4:12 pm
Hardware configuration: Ubuntu 22.04.2 LTS; NVidia 525.60.11; 2 x 4070ti; 4070; 4060ti; 3x 3080; 3070ti; 3070
Location: Great White North

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by gordonbb »

Nope,

Rebooting the router hasn't fixed it:

Code: Select all

18:54:08:WU00:FS00:0x21:Completed 495000 out of 500000 steps (99%)
18:54:08:WU01:FS00:Connecting to 65.254.110.245:8080
18:54:08:WU01:FS00:Assigned to work server 155.247.166.220
18:54:08:WU01:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TU106 [GeForce RTX 2070 Rev. A] M 7465 from 155.247.166.220
18:54:08:WU01:FS00:Connecting to 155.247.166.220:8080
18:54:08:WU01:FS00:Downloading 2.34MiB
18:54:35:WU00:FS00:0x21:Completed 500000 out of 500000 steps (100%)
18:54:42:WU00:FS00:0x21:Saving result file logfile_01.txt
18:54:42:WU00:FS00:0x21:Saving result file checkpointState.xml
18:54:42:WU00:FS00:0x21:Saving result file checkpt.crc
18:54:42:WU00:FS00:0x21:Saving result file log.txt
18:54:42:WU00:FS00:0x21:Saving result file positions.xtc
18:54:42:WU00:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
18:54:42:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:54:42:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:14230 run:174 clone:0 gen:41 core:0x21 unit:0x0000004580fccb0a5d654fe4ff013a47
18:54:42:WU00:FS00:Uploading 52.00MiB to 128.252.203.10
18:54:42:WU00:FS00:Connecting to 128.252.203.10:8080
18:54:48:WU00:FS00:Upload 1.68%
18:54:54:WU00:FS00:Upload 4.69%
18:55:00:WU00:FS00:Upload 9.50%
18:55:06:WU00:FS00:Upload 14.54%
18:55:12:WU00:FS00:Upload 20.07%
18:55:18:WU00:FS00:Upload 31.97%
18:55:24:WU00:FS00:Upload 47.11%
18:55:30:WU00:FS00:Upload 62.14%
18:55:36:WU00:FS00:Upload 75.96%
18:55:42:WU00:FS00:Upload 90.62%
18:55:47:WU00:FS00:Upload complete
18:55:47:WU00:FS00:Server responded WORK_ACK (400)
18:55:47:WU00:FS00:Final credit estimate, 50919.00 points
18:55:47:WU00:FS00:Cleaning up
It seems this system is cursed to just get WUs from temple :-(

and another system - same thing:

Code: Select all

18:44:32:WU01:FS00:0x21:Completed 990000 out of 1000000 steps (99%)
18:44:33:WU02:FS00:Connecting to 65.254.110.245:8080
18:44:33:WU02:FS00:Assigned to work server 155.247.166.220
18:44:33:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TU116 [GeForce GTX 1660 Ti] from 155.247.166.220
18:44:33:WU02:FS00:Connecting to 155.247.166.220:8080
18:44:33:WU02:FS00:Downloading 2.34MiB
18:44:39:WU02:FS00:Download 10.68%
18:46:21:WU01:FS00:0x21:Completed 1000000 out of 1000000 steps (100%)
18:46:48:WU01:FS00:0x21:Saving result file logfile_01.txt
18:46:48:WU01:FS00:0x21:Saving result file checkpointState.xml
18:46:48:WU01:FS00:0x21:Saving result file checkpt.crc
18:46:48:WU01:FS00:0x21:Saving result file log.txt
18:46:48:WU01:FS00:0x21:Saving result file positions.xtc
18:46:48:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
18:46:49:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:46:49:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:620 clone:0 gen:41 core:0x21 unit:0x0000003780fccb0a5d6ed21430cc8371
18:46:49:WU01:FS00:Uploading 65.87MiB to 128.252.203.10
18:46:49:WU01:FS00:Connecting to 128.252.203.10:8080
18:46:55:WU01:FS00:Upload 3.42%
18:47:02:WU01:FS00:Upload 7.59%
18:47:08:WU01:FS00:Upload 13.19%
18:47:14:WU01:FS00:Upload 18.50%
18:47:20:WU01:FS00:Upload 23.72%
18:47:26:WU01:FS00:Upload 28.75%
18:47:32:WU01:FS00:Upload 34.35%
18:47:38:WU01:FS00:Upload 40.04%
18:47:44:WU01:FS00:Upload 44.88%
18:47:50:WU01:FS00:Upload 50.58%
18:47:56:WU01:FS00:Upload 56.46%
18:48:02:WU01:FS00:Upload 61.58%
18:48:08:WU01:FS00:Upload 67.66%
18:48:14:WU01:FS00:Upload 74.77%
18:48:20:WU01:FS00:Upload 81.32%
18:48:26:WU01:FS00:Upload 86.92%
18:48:32:WU01:FS00:Upload 93.75%
18:48:38:WU01:FS00:Upload 99.35%
18:48:40:WU01:FS00:Upload complete
18:48:40:WU01:FS00:Server responded WORK_ACK (400)
18:48:40:WU01:FS00:Final credit estimate, 71427.00 points
18:48:40:WU01:FS00:Cleaning up
******************************* Date: 2019-10-01 *******************************
Image
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by bruce »

gordonbb wrote:18:55:47:WU00:FS00:Cleaning up[/code]It seems this system is cursed to just get WUs from temple :-(
According to https://apps.foldingathome.org/serverstats, some servers have WUs for FAHCore_21 and some for Gro_A7. Apparently you only want WUs for GPUs, so the only choices that exist are wustl, temple, and mskcc. Looking at psummary, you'll see that there are a few projects that might be assigning on wustl or mskcc but we really don't have a method to figure out which of those projects can be assigned to you're particular species of GPU or whether any of those projects have WUs to be assigned.

Obviously sazerac.seas.wustl.edu doesn't count. It reports that it has zero WUs. (and I recently saw the AS send somebody there and the message was something like No WU was assigned. I don't understand that interchange)
orkney has 4201
plfah2-1 has 7889
fnts2 doesn't seem to be assigning.

Core21 jobs are assigning at a rate of 1,381/hr of which most seem to be coming from orkney with a rather small fraction from temple.

All these numbers are shapshots of some rather dynamic values.


You didn't report what kind of GPU you have that is looking for an assignment.
Top
gordonbb
Posts: 511
Joined: Mon May 21, 2018 4:12 pm
Hardware configuration: Ubuntu 22.04.2 LTS; NVidia 525.60.11; 2 x 4070ti; 4070; 4060ti; 3x 3080; 3070ti; 3070
Location: Great White North

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by gordonbb »

Code: Select all

03:00:01:WU01:FS00:0x21:Completed 495000 out of 500000 steps (99%)
03:00:02:WU02:FS00:Connecting to 65.254.110.245:8080
03:00:02:WU02:FS00:Assigned to work server 155.247.166.220
03:00:02:WU02:FS00:Requesting new work unit for slot 00: RUNNING gpu:0:TU106 [GeForce RTX 2060 Super] from 155.247.166.220
03:00:02:WU02:FS00:Connecting to 155.247.166.220:8080
03:00:02:WU02:FS00:Downloading 6.06MiB
03:00:08:WU02:FS00:Download 14.43%
03:00:14:WU02:FS00:Download 22.67%
03:00:20:WU02:FS00:Download 27.83%
03:00:31:WU01:FS00:0x21:Completed 500000 out of 500000 steps (100%)
03:00:32:WU02:FS00:Download 29.89%
03:00:37:WU01:FS00:0x21:Saving result file logfile_01.txt
03:00:37:WU01:FS00:0x21:Saving result file checkpointState.xml
03:00:37:WU01:FS00:0x21:Saving result file checkpt.crc
03:00:37:WU01:FS00:0x21:Saving result file log.txt
03:00:37:WU01:FS00:0x21:Saving result file positions.xtc
03:00:37:WU01:FS00:0x21:Folding@home Core Shutdown: FINISHED_UNIT
03:00:38:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
03:00:38:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14229 run:1083 clone:1 gen:47 core:0x21 unit:0x0000003b80fccb0a5d716f871dbe513b
03:00:38:WU01:FS00:Uploading 51.47MiB to 128.252.203.10
03:00:38:WU01:FS00:Connecting to 128.252.203.10:8080
03:00:41:WU02:FS00:Download 31.95%
03:00:44:WU01:FS00:Upload 3.04%
03:00:50:WU01:FS00:Upload 6.19%
03:00:50:WU02:FS00:Download 34.01%
03:00:56:WU01:FS00:Upload 11.17%
03:01:01:WU02:FS00:Download 36.07%
03:01:02:WU01:FS00:Upload 15.06%
03:01:08:WU01:FS00:Upload 18.70%
03:01:08:WU02:FS00:Download 38.13%
03:01:14:WU01:FS00:Upload 23.80%
03:01:20:WU01:FS00:Upload 31.45%
03:01:23:WU02:FS00:Download 39.16%
03:01:26:WU01:FS00:Upload 38.13%
03:01:32:WU01:FS00:Upload 45.78%
03:01:38:WU01:FS00:Upload 59.74%
03:01:43:WU02:FS00:Download 40.19%
03:01:44:WU01:FS00:Upload 75.41%
03:01:50:WU01:FS00:Upload 91.07%
03:01:55:WU01:FS00:Upload complete
03:01:55:WU01:FS00:Server responded WORK_ACK (400)
03:01:55:WU01:FS00:Final credit estimate, 48337.00 points
03:01:55:WU01:FS00:Cleaning up
03:02:10:WU02:FS00:Download 41.22%
03:02:45:WU02:FS00:Download 43.29%
This is getting really tiresome. This makes about the 12th or 13th slot stuck today.

These are NVidia GPUs:

2 x GTX 1060 6GB; 1 GTX 1660Ti; 1070Ti; 1 RTX 2060; 1 RTX 2060 Super. 1 RTX 2070; 5 RTX 2070 Supers

I'm trying to be positive here and provide any information I can that can help the folks at Temple fix the issue, whatever it may be. We've just started a month-long Folding Competition with over 1700 users (about 980 currently active) and are set to hit around 350,000,000 points today even with many of the participants experiencing their systems locking up when a download stalls.

I do realize that no one ever says "Hey, good job" when things work as we expect and it is a testament to the F@H team that things do work so well as it is a complex system with many components. The timing is just unfortunate in my case as ordinarily I might just pull some GPUs off F@H and have them do some BOINC or take them off-line and clean and re-paste them while the issues get resolved.
Image
Top
gordonbb
Posts: 511
Joined: Mon May 21, 2018 4:12 pm
Hardware configuration: Ubuntu 22.04.2 LTS; NVidia 525.60.11; 2 x 4070ti; 4070; 4060ti; 3x 3080; 3070ti; 3070
Location: Great White North

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by gordonbb »

Caught one WU from 155.247.166.220 that successfully downloaded. There was a transfer error, possibly downloading the WU, but it recovered gracefully

Code: Select all

12:52:22:WU02:FS01:0x21:Completed 990000 out of 1000000 steps (99%)
12:52:23:WU03:FS01:Connecting to 65.254.110.245:8080
12:52:23:WU03:FS01:Assigned to work server 155.247.166.220
12:52:23:WU03:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
12:52:23:WU03:FS01:Connecting to 155.247.166.220:8080
12:52:23:WU03:FS01:Downloading 6.06MiB
12:52:29:WU03:FS01:Download 6.18%
12:52:36:WU03:FS01:Download 12.37%
12:52:54:WU03:FS01:Download 15.46%
12:53:00:WU03:FS01:Download 17.52%
12:53:17:WU02:FS01:0x21:Completed 1000000 out of 1000000 steps (100%)
12:53:23:WU02:FS01:0x21:Saving result file logfile_01.txt
12:53:23:WU02:FS01:0x21:Saving result file checkpointState.xml
12:53:23:WU02:FS01:0x21:Saving result file checkpt.crc
12:53:23:WU02:FS01:0x21:Saving result file log.txt
12:53:23:WU02:FS01:0x21:Saving result file positions.xtc
12:53:23:WU02:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
12:53:24:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:53:24:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14249 run:1135 clone:1 gen:19 core:0x21 unit:0x0000001980fccb0a5d716fa1db100ac9
12:53:24:WU02:FS01:Uploading 66.04MiB to 128.252.203.10
12:53:24:WU02:FS01:Connecting to 128.252.203.10:8080
12:53:30:WU02:FS01:Upload 12.30%
12:53:36:WU02:FS01:Upload 23.75%
12:53:42:WU02:FS01:Upload 35.11%
12:53:48:WU02:FS01:Upload 45.33%
12:53:54:WU02:FS01:Upload 56.78%
12:54:00:WU02:FS01:Upload 67.19%
12:54:06:WU02:FS01:Upload 78.64%
12:54:10:WU03:FS01:Download 21.64%
12:54:11:ERROR:WU03:FS01:Exception: Transfer failed
12:54:11:WU03:FS01:Connecting to 65.254.110.245:8080
12:54:12:WU03:FS01:Assigned to work server 155.247.166.220
12:54:12:WU03:FS01:Requesting new work unit for slot 01: READY gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
12:54:12:WU03:FS01:Connecting to 155.247.166.220:8080
12:54:12:WU02:FS01:Upload 90.00%
12:54:12:WU03:FS01:Downloading 3.35MiB
12:54:19:WU03:FS01:Download 7.45%
12:54:19:WU02:FS01:Upload complete
12:54:19:WU02:FS01:Server responded WORK_ACK (400)
12:54:19:WU02:FS01:Final credit estimate, 103074.00 points
12:54:19:WU02:FS01:Cleaning up
12:54:26:WU03:FS01:Download 22.36%
12:54:33:WU03:FS01:Download 33.54%
12:55:07:WU03:FS01:Download 37.27%
12:55:14:WU03:FS01:Download 46.59%
12:55:24:WU03:FS01:Download 52.18%
12:55:30:WU03:FS01:Download 55.90%
12:55:36:WU03:FS01:Download 81.99%
12:55:38:WU03:FS01:Download complete
12:55:38:WU03:FS01:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:14186 run:3 clone:92 gen:103 core:0x21 unit:0x000000b40002894c5d38973e8dca7656
12:55:38:WU03:FS01:Starting
12:55:38:WU03:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 03 -suffix 01 -version 705 -lifeline 1762 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu 1
12:55:38:WU03:FS01:Started FahCore on PID 18128
12:55:38:WU03:FS01:Core PID:18132
12:55:38:WU03:FS01:FahCore 0x21 started
12:55:39:WU03:FS01:0x21:*********************** Log Started 2019-10-02T12:55:38Z ***********************
12:55:39:WU03:FS01:0x21:Project: 14186 (Run 3, Clone 92, Gen 103)
12:55:39:WU03:FS01:0x21:Unit: 0x000000b40002894c5d38973e8dca7656
12:55:39:WU03:FS01:0x21:CPU: 0x00000000000000000000000000000000
12:55:39:WU03:FS01:0x21:Machine: 1
12:55:39:WU03:FS01:0x21:Reading tar file core.xml
12:55:39:WU03:FS01:0x21:Reading tar file integrator.xml
12:55:39:WU03:FS01:0x21:Reading tar file state.xml
12:55:39:WU03:FS01:0x21:Reading tar file system.xml
12:55:39:WU03:FS01:0x21:Digital signatures verified
12:55:39:WU03:FS01:0x21:Folding@home GPU Core21 Folding@home Core
12:55:39:WU03:FS01:0x21:Version 0.0.20
12:55:41:WU03:FS01:0x21:Completed 0 out of 12500000 steps (0%)
12:55:41:WU03:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
12:56:23:WU03:FS01:0x21:Completed 125000 out of 12500000 steps (1%)
No slots showed up stalled at "Downloading" this morning so hopefully a good sign.
Image
Top
rickoic
Posts: 320
Joined: Sat May 23, 2009 4:49 pm
Hardware configuration: eVga x299 DARK 2070 Super, eVGA 2080, eVga 1070, eVga 2080 Super
MSI x399 eVga 2080, eVga 1070, eVga 1070, GT970
Location: Mississippi near Memphis, Tn

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by rickoic »

Had to reboot all 3 of my computers this morning. See this was first reported on the 24th of Sep. Looking at my past progress I think it's been going on since at least the 19th of Sep. I have seen no progress reports about anyone seeing anything wrong just observations and obviously no results. Not sure how determination made to be assigned to these servers, but couldn't someone decrease the probability of being assigned to them?
I'm folding because Dec 2005 I had radical prostate surgery.
Lost brother to spinal cancer, brother-in-law to prostate cancer.
Several 1st cousins lost and a few who have survived.
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by bruce »

The next time somebody decides they need to reboot the computers or the network, try clearing the cache first and see if that helps.
Top
HaloJones
Posts: 906
Joined: Thu Jul 24, 2008 10:16 am

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by HaloJones »

@Bruce, which cache please and how to clear it?
single 1070

Image
Top
dfgirl12
Posts: 38
Joined: Fri Aug 21, 2009 8:34 am

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by dfgirl12 »

Tuesday's slowness wasn't that bad, but everything got backed-up 6-10+ WUs deep per folding slot waiting to upload on Wednesday. With 50+ concurrent uploads not much is getting through until I disconnect 50% of them. It will take 2-3 days to upload the last 12 hours worth of finished WUs at the current upload rate. :roll:

The worst one is the Windows machine:

Code: Select all

2019-10-03:07:34:40:Enabled folding slot 01: PAUSED gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 (by user)
2019-10-03:07:34:40:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14196 run:5 clone:6 gen:2 core:0x21 unit:0x000000020002894b5d9246573bee776e
2019-10-03:07:34:40:WU06:FS01:Uploading 19.40MiB to 155.247.166.219
2019-10-03:07:34:40:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:840 clone:1 gen:24 core:0x21 unit:0x0000002180fccb0a5d716f9c57a670b1
2019-10-03:07:34:40:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:34:40:WU01:FS01:Uploading 65.77MiB to 128.252.203.10
2019-10-03:07:34:40:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:34:40:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14196 run:1 clone:7 gen:0 core:0x21 unit:0x000000000002894b5d9246403f27a828
2019-10-03:07:34:40:WU04:FS01:Uploading 19.38MiB to 155.247.166.219
2019-10-03:07:34:40:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:34:40:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14196 run:14 clone:13 gen:0 core:0x21 unit:0x000000000002894b5d92468acdbde5bf
2019-10-03:07:34:40:WU02:FS01:Uploading 19.39MiB to 155.247.166.219
2019-10-03:07:34:40:WU02:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:34:40:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14250 run:266 clone:0 gen:53 core:0x21 unit:0x0000004680fccb0a5d6ed218434b43f3
2019-10-03:07:34:40:WU00:FS01:Uploading 66.76MiB to 128.252.203.10
2019-10-03:07:34:40:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:34:43:8:127.0.0.1:New Web connection
2019-10-03:07:34:50:WU06:FS01:Upload 0.32%
2019-10-03:07:34:59:WU00:FS01:Upload 0.09%
2019-10-03:07:35:07:WU00:FS01:Upload 0.19%
2019-10-03:07:35:17:WU01:FS01:Upload 0.10%
2019-10-03:07:35:27:WU01:FS01:Upload 0.19%
2019-10-03:07:35:42:WU06:FS01:Upload 0.64%
2019-10-03:07:35:45:WU01:FS01:Upload 0.29%
2019-10-03:07:35:50:WU06:FS01:Upload 0.97%
2019-10-03:07:36:06:WU02:FS01:Upload 0.32%
2019-10-03:07:36:06:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:36:06:WU02:FS01:Trying to send results to collection server
2019-10-03:07:36:06:WU02:FS01:Uploading 19.39MiB to 128.252.203.4
2019-10-03:07:36:06:WU02:FS01:Connecting to 128.252.203.4:8080
2019-10-03:07:36:39:WU04:FS01:Upload 0.65%
2019-10-03:07:36:39:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:36:39:WU04:FS01:Trying to send results to collection server
2019-10-03:07:36:39:WU04:FS01:Uploading 19.38MiB to 128.252.203.4
2019-10-03:07:36:39:WU04:FS01:Connecting to 128.252.203.4:8080
2019-10-03:07:36:40:WU00:FS01:Upload 0.28%
2019-10-03:07:36:40:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:36:40:WU00:FS01:Trying to send results to collection server
2019-10-03:07:36:40:WU00:FS01:Uploading 66.76MiB to 155.247.166.219
2019-10-03:07:36:40:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:36:49:WU00:FS01:Upload 0.09%
2019-10-03:07:36:49:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-03:07:36:49:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14250 run:266 clone:0 gen:53 core:0x21 unit:0x0000004680fccb0a5d6ed218434b43f3
2019-10-03:07:36:49:WU00:FS01:Uploading 66.76MiB to 128.252.203.10
2019-10-03:07:36:49:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:37:55:WU01:FS01:Upload 0.38%
2019-10-03:07:37:55:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:37:55:WU01:FS01:Trying to send results to collection server
2019-10-03:07:37:55:WU01:FS01:Uploading 65.77MiB to 155.247.166.219
2019-10-03:07:37:55:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:37:57:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-03:07:37:58:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:840 clone:1 gen:24 core:0x21 unit:0x0000002180fccb0a5d716f9c57a670b1
2019-10-03:07:37:58:WU01:FS01:Uploading 65.77MiB to 128.252.203.10
2019-10-03:07:37:58:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:38:11:WU01:FS01:Upload 0.10%
2019-10-03:07:38:21:WU01:FS01:Upload 0.19%
2019-10-03:07:38:23:WU02:FS01:Upload 0.32%
2019-10-03:07:38:23:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-03:07:38:23:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14196 run:14 clone:13 gen:0 core:0x21 unit:0x000000000002894b5d92468acdbde5bf
2019-10-03:07:38:23:WU02:FS01:Uploading 19.39MiB to 155.247.166.219
2019-10-03:07:38:23:WU02:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:38:30:WU02:FS01:Upload 0.32%
2019-10-03:07:38:31:WU01:FS01:Upload 0.29%
2019-10-03:07:38:39:WU04:FS01:Upload 0.32%
2019-10-03:07:38:39:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-03:07:38:40:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14196 run:1 clone:7 gen:0 core:0x21 unit:0x000000000002894b5d9246403f27a828
2019-10-03:07:38:40:WU04:FS01:Uploading 19.38MiB to 155.247.166.219
2019-10-03:07:38:40:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:38:43:WU06:FS01:Upload 1.29%
2019-10-03:07:38:43:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:38:43:WU06:FS01:Trying to send results to collection server
2019-10-03:07:38:43:WU06:FS01:Uploading 19.40MiB to 155.247.166.220
2019-10-03:07:38:43:WU06:FS01:Connecting to 155.247.166.220:8080
2019-10-03:07:38:48:WU01:FS01:Upload 0.38%
2019-10-03:07:38:58:WU01:FS01:Upload 0.48%
2019-10-03:07:39:03:WU04:FS01:Upload 0.32%
2019-10-03:07:39:07:WU01:FS01:Upload 0.67%
2019-10-03:07:39:28:WU00:FS01:Upload 0.09%
2019-10-03:07:39:28:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:39:28:WU00:FS01:Trying to send results to collection server
2019-10-03:07:39:28:WU00:FS01:Uploading 66.76MiB to 155.247.166.219
2019-10-03:07:39:28:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:39:28:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-03:07:39:28:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14250 run:266 clone:0 gen:53 core:0x21 unit:0x0000004680fccb0a5d6ed218434b43f3
2019-10-03:07:39:28:WU00:FS01:Uploading 66.76MiB to 128.252.203.10
2019-10-03:07:39:28:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:39:51:WU00:FS01:Upload 0.09%
2019-10-03:07:39:53:WU02:FS01:Upload 0.64%
2019-10-03:07:39:53:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:39:53:WU02:FS01:Trying to send results to collection server
2019-10-03:07:39:53:WU02:FS01:Uploading 19.39MiB to 128.252.203.4
2019-10-03:07:39:53:WU02:FS01:Connecting to 128.252.203.4:8080
2019-10-03:07:40:02:WU00:FS01:Upload 0.19%
2019-10-03:07:40:08:WU00:FS01:Upload 0.28%
2019-10-03:07:40:13:WU06:FS01:Upload 0.32%
2019-10-03:07:40:26:WU06:FS01:Upload 0.97%
2019-10-03:07:40:27:WU00:FS01:Upload 0.37%
2019-10-03:07:40:32:WU06:FS01:Upload 1.29%
2019-10-03:07:40:37:WU00:FS01:Upload 0.47%
2019-10-03:07:40:40:WU06:FS01:Upload 1.93%
2019-10-03:07:40:59:WU01:FS01:Upload 0.76%
2019-10-03:07:40:59:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:40:59:WU01:FS01:Trying to send results to collection server
2019-10-03:07:40:59:WU01:FS01:Uploading 65.77MiB to 155.247.166.219
2019-10-03:07:40:59:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:41:00:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-03:07:41:00:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:840 clone:1 gen:24 core:0x21 unit:0x0000002180fccb0a5d716f9c57a670b1
2019-10-03:07:41:00:WU01:FS01:Uploading 65.77MiB to 128.252.203.10
2019-10-03:07:41:00:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:41:02:WU06:FS01:Upload 2.26%
2019-10-03:07:41:13:WU01:FS01:Upload 0.10%
2019-10-03:07:41:19:WU01:FS01:Upload 0.19%
2019-10-03:07:41:42:WU02:FS01:Upload 0.32%
2019-10-03:07:41:42:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-03:07:41:42:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14196 run:14 clone:13 gen:0 core:0x21 unit:0x000000000002894b5d92468acdbde5bf
2019-10-03:07:41:42:WU02:FS01:Uploading 19.39MiB to 155.247.166.219
2019-10-03:07:41:42:WU02:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:41:51:WU02:FS01:Upload 0.32%
2019-10-03:07:42:02:WU01:FS01:Upload 0.38%
2019-10-03:07:42:29:WU00:FS01:Upload 0.56%
2019-10-03:07:42:29:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:42:29:WU00:FS01:Trying to send results to collection server
2019-10-03:07:42:29:WU00:FS01:Uploading 66.76MiB to 155.247.166.219
2019-10-03:07:42:29:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:42:29:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-03:07:42:29:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14250 run:266 clone:0 gen:53 core:0x21 unit:0x0000004680fccb0a5d6ed218434b43f3
2019-10-03:07:42:29:WU00:FS01:Uploading 66.76MiB to 128.252.203.10
2019-10-03:07:42:29:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:42:39:WU00:FS01:Upload 0.19%
2019-10-03:07:42:40:WU01:FS01:Upload 0.48%
2019-10-03:07:42:41:WU06:FS01:Upload 2.58%
2019-10-03:07:42:46:WU00:FS01:Upload 0.28%
2019-10-03:07:42:55:WU00:FS01:Upload 0.37%
2019-10-03:07:43:01:WU01:FS01:Upload 0.57%
2019-10-03:07:43:13:WU02:FS01:Upload 0.64%
2019-10-03:07:43:27:WU02:FS01:Upload 0.97%
2019-10-03:07:43:33:WU00:FS01:Upload 0.47%
2019-10-03:07:43:50:WU00:FS01:Upload 0.56%
2019-10-03:07:43:58:WU00:FS01:Upload 0.75%
2019-10-03:07:43:58:WU01:FS01:Upload 0.67%
2019-10-03:07:43:58:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:43:58:WU01:FS01:Trying to send results to collection server
2019-10-03:07:43:58:WU01:FS01:Uploading 65.77MiB to 155.247.166.219
2019-10-03:07:43:58:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:43:59:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-03:07:43:59:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:840 clone:1 gen:24 core:0x21 unit:0x0000002180fccb0a5d716f9c57a670b1
2019-10-03:07:43:59:WU01:FS01:Uploading 65.77MiB to 128.252.203.10
2019-10-03:07:43:59:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:44:15:WU01:FS01:Upload 0.10%
2019-10-03:07:44:22:WU00:FS01:Upload 0.94%
2019-10-03:07:44:25:WU01:FS01:Upload 0.29%
2019-10-03:07:44:28:WU00:FS01:Upload 1.03%
2019-10-03:07:44:41:WU06:FS01:Upload 2.90%
2019-10-03:07:44:41:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-03:07:44:41:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14196 run:5 clone:6 gen:2 core:0x21 unit:0x000000020002894b5d9246573bee776e
2019-10-03:07:44:41:WU06:FS01:Uploading 19.40MiB to 155.247.166.219
2019-10-03:07:44:41:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:44:48:WU01:FS01:Upload 0.38%
2019-10-03:07:44:57:WU00:FS01:Upload 1.12%
2019-10-03:07:44:58:WU01:FS01:Upload 0.67%
2019-10-03:07:45:00:WU02:FS01:Upload 1.29%
2019-10-03:07:45:10:WU01:FS01:Upload 0.76%
2019-10-03:07:45:10:WU04:FS01:Upload 0.65%
2019-10-03:07:45:10:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:45:10:WU04:FS01:Trying to send results to collection server
2019-10-03:07:45:10:WU04:FS01:Uploading 19.38MiB to 128.252.203.4
2019-10-03:07:45:10:WU04:FS01:Connecting to 128.252.203.4:8080
2019-10-03:07:45:12:WU02:FS01:Upload 1.61%
2019-10-03:07:45:20:WU02:FS01:Upload 1.93%
2019-10-03:07:45:30:WU01:FS01:Upload 0.86%
2019-10-03:07:45:38:WU01:FS01:Upload 0.95%
2019-10-03:07:45:45:WU02:FS01:Upload 2.26%
2019-10-03:07:45:53:WU02:FS01:Upload 2.58%
2019-10-03:07:46:10:WU00:FS01:Upload 1.22%
2019-10-03:07:46:10:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:46:10:WU00:FS01:Trying to send results to collection server
2019-10-03:07:46:10:WU00:FS01:Uploading 66.76MiB to 155.247.166.219
2019-10-03:07:46:10:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:46:11:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-03:07:46:11:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14250 run:266 clone:0 gen:53 core:0x21 unit:0x0000004680fccb0a5d6ed218434b43f3
2019-10-03:07:46:11:WU00:FS01:Uploading 66.76MiB to 128.252.203.10
2019-10-03:07:46:11:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:46:20:WU02:FS01:Upload 2.90%
2019-10-03:07:46:26:WU02:FS01:Upload 3.22%
2019-10-03:07:46:29:WU00:FS01:Upload 0.19%
2019-10-03:07:46:31:WU06:FS01:Upload 0.32%
2019-10-03:07:46:31:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:46:31:WU06:FS01:Trying to send results to collection server
2019-10-03:07:46:31:WU06:FS01:Uploading 19.40MiB to 155.247.166.220
2019-10-03:07:46:31:WU06:FS01:Connecting to 155.247.166.220:8080
2019-10-03:07:46:35:WU01:FS01:Upload 1.05%
2019-10-03:07:46:38:WU06:FS01:Upload 0.32%
2019-10-03:07:46:41:WU01:FS01:Upload 1.14%
2019-10-03:07:46:57:WU00:FS01:Upload 0.28%
2019-10-03:07:47:05:WU00:FS01:Upload 0.47%
2019-10-03:07:47:15:WU06:FS01:Upload 0.64%
2019-10-03:07:47:22:WU01:FS01:Upload 1.24%
2019-10-03:07:47:29:WU02:FS01:Upload 3.55%
2019-10-03:07:47:32:WU01:FS01:Upload 1.33%
2019-10-03:07:47:40:WU01:FS01:Upload 1.43%
2019-10-03:07:47:40:WU04:FS01:Upload 0.32%
2019-10-03:07:47:40:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-03:07:47:40:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14196 run:1 clone:7 gen:0 core:0x21 unit:0x000000000002894b5d9246403f27a828
2019-10-03:07:47:40:WU04:FS01:Uploading 19.38MiB to 155.247.166.219
2019-10-03:07:47:40:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:47:42:WU00:FS01:Upload 0.56%
2019-10-03:07:47:51:WU01:FS01:Upload 1.62%
2019-10-03:07:47:59:WU01:FS01:Upload 1.71%
2019-10-03:07:48:02:WU02:FS01:Upload 3.87%
2019-10-03:07:48:04:WU04:FS01:Upload 0.32%
2019-10-03:07:48:07:WU01:FS01:Upload 1.90%
2019-10-03:07:48:13:WU04:FS01:Upload 0.65%
2019-10-03:07:48:23:WU04:FS01:Upload 0.97%
2019-10-03:07:48:25:WU01:FS01:Upload 2.00%
2019-10-03:07:48:37:WU01:FS01:Upload 2.19%
2019-10-03:07:48:37:WU06:FS01:Upload 0.97%
2019-10-03:07:48:37:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-03:07:48:37:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14196 run:5 clone:6 gen:2 core:0x21 unit:0x000000020002894b5d9246573bee776e
2019-10-03:07:48:37:WU06:FS01:Uploading 19.40MiB to 155.247.166.219
2019-10-03:07:48:37:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:48:43:WU06:FS01:Upload 0.32%
2019-10-03:07:48:45:WU01:FS01:Upload 2.28%
2019-10-03:07:49:04:WU06:FS01:Upload 0.64%
2019-10-03:07:49:12:WU06:FS01:Upload 0.97%
2019-10-03:07:49:15:WU02:FS01:Upload 4.19%
2019-10-03:07:49:15:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:49:15:WU02:FS01:Trying to send results to collection server
2019-10-03:07:49:15:WU02:FS01:Uploading 19.39MiB to 128.252.203.4
2019-10-03:07:49:15:WU02:FS01:Connecting to 128.252.203.4:8080
2019-10-03:07:49:22:WU02:FS01:Upload 0.32%
2019-10-03:07:49:29:WU02:FS01:Upload 0.64%
2019-10-03:07:49:38:WU02:FS01:Upload 1.29%
2019-10-03:07:49:45:WU02:FS01:Upload 1.61%
2019-10-03:07:49:52:WU02:FS01:Upload 2.26%
2019-10-03:07:49:54:WU04:FS01:Upload 1.29%
2019-10-03:07:49:54:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:49:54:WU04:FS01:Trying to send results to collection server
2019-10-03:07:49:54:WU04:FS01:Uploading 19.38MiB to 128.252.203.4
2019-10-03:07:49:54:WU04:FS01:Connecting to 128.252.203.4:8080
2019-10-03:07:50:02:WU00:FS01:Upload 0.66%
2019-10-03:07:50:02:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:50:02:WU00:FS01:Trying to send results to collection server
2019-10-03:07:50:02:WU00:FS01:Uploading 66.76MiB to 155.247.166.219
2019-10-03:07:50:02:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:50:02:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-03:07:50:26:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14250 run:266 clone:0 gen:53 core:0x21 unit:0x0000004680fccb0a5d6ed218434b43f3
2019-10-03:07:50:26:WU00:FS01:Uploading 66.76MiB to 128.252.203.10
2019-10-03:07:50:26:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:50:37:WU00:FS01:Upload 0.19%
2019-10-03:07:50:55:WU00:FS01:Upload 0.37%
2019-10-03:07:51:12:WU04:FS01:Upload 0.32%
2019-10-03:07:51:12:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-03:07:51:12:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14196 run:1 clone:7 gen:0 core:0x21 unit:0x000000000002894b5d9246403f27a828
2019-10-03:07:51:12:WU04:FS01:Uploading 19.38MiB to 155.247.166.219
2019-10-03:07:51:12:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:51:19:WU04:FS01:Upload 0.32%
2019-10-03:07:51:35:WU00:FS01:Upload 0.47%
2019-10-03:07:51:40:WU02:FS01:Upload 2.58%
2019-10-03:07:51:40:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-03:07:51:40:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14196 run:14 clone:13 gen:0 core:0x21 unit:0x000000000002894b5d92468acdbde5bf
2019-10-03:07:51:40:WU02:FS01:Uploading 19.39MiB to 155.247.166.219
2019-10-03:07:51:40:WU02:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:51:48:WU04:FS01:Upload 0.65%
2019-10-03:07:51:54:WU02:FS01:Upload 0.32%
2019-10-03:07:51:57:WU04:FS01:Upload 0.97%
2019-10-03:07:52:17:WU02:FS01:Upload 0.64%
2019-10-03:07:52:24:WU02:FS01:Upload 0.97%
2019-10-03:07:52:29:WU01:FS01:Upload 2.38%
2019-10-03:07:52:29:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:52:29:WU01:FS01:Trying to send results to collection server
2019-10-03:07:52:29:WU01:FS01:Uploading 65.77MiB to 155.247.166.219
2019-10-03:07:52:29:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:52:29:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-03:07:52:29:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:840 clone:1 gen:24 core:0x21 unit:0x0000002180fccb0a5d716f9c57a670b1
2019-10-03:07:52:29:WU01:FS01:Uploading 65.77MiB to 128.252.203.10
2019-10-03:07:52:29:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:52:31:WU04:FS01:Upload 1.61%
2019-10-03:07:52:32:WU00:FS01:Upload 0.56%
2019-10-03:07:52:32:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:52:32:WU00:FS01:Trying to send results to collection server
2019-10-03:07:52:32:WU00:FS01:Uploading 66.76MiB to 155.247.166.219
2019-10-03:07:52:32:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:52:33:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-03:07:52:35:WU02:FS01:Upload 1.29%
2019-10-03:07:52:37:WU01:FS01:Upload 0.10%
2019-10-03:07:52:43:WU01:FS01:Upload 0.19%
2019-10-03:07:52:49:WU01:FS01:Upload 0.29%
2019-10-03:07:52:51:WU04:FS01:Upload 1.94%
2019-10-03:07:52:58:WU06:FS01:Upload 1.61%
2019-10-03:07:52:58:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:52:58:WU06:FS01:Trying to send results to collection server
2019-10-03:07:52:58:WU06:FS01:Uploading 19.40MiB to 155.247.166.220
2019-10-03:07:52:58:WU06:FS01:Connecting to 155.247.166.220:8080
2019-10-03:07:52:58:WU02:FS01:Upload 1.61%
2019-10-03:07:52:59:WU04:FS01:Upload 2.26%
2019-10-03:07:53:05:WU01:FS01:Upload 0.48%
2019-10-03:07:53:15:WU04:FS01:Upload 2.58%
2019-10-03:07:53:20:WU01:FS01:Upload 0.67%
2019-10-03:07:53:27:WU02:FS01:Upload 1.93%
2019-10-03:07:53:33:WU02:FS01:Upload 2.26%
2019-10-03:07:54:07:WU04:FS01:Upload 2.90%
2019-10-03:07:54:38:WU04:FS01:Upload 3.23%
2019-10-03:07:54:57:WU01:FS01:Upload 0.76%
2019-10-03:07:54:57:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:54:57:WU01:FS01:Trying to send results to collection server
2019-10-03:07:54:57:WU01:FS01:Uploading 65.77MiB to 155.247.166.219
2019-10-03:07:54:57:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:54:58:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-03:07:55:23:WU04:FS01:Upload 3.55%
2019-10-03:07:55:37:WU06:FS01:Upload 0.32%
2019-10-03:07:55:37:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-03:07:55:37:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14196 run:5 clone:6 gen:2 core:0x21 unit:0x000000020002894b5d9246573bee776e
2019-10-03:07:55:37:WU06:FS01:Uploading 19.40MiB to 155.247.166.219
2019-10-03:07:55:37:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:55:50:WU04:FS01:Upload 3.87%
2019-10-03:07:55:50:WU06:FS01:Upload 0.32%
2019-10-03:07:56:00:WU06:FS01:Upload 0.64%
2019-10-03:07:56:03:WU02:FS01:Upload 2.58%
2019-10-03:07:56:03:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:56:03:WU02:FS01:Trying to send results to collection server
2019-10-03:07:56:03:WU02:FS01:Uploading 19.39MiB to 128.252.203.4
2019-10-03:07:56:03:WU02:FS01:Connecting to 128.252.203.4:8080
2019-10-03:07:56:04:WU04:FS01:Upload 4.19%
2019-10-03:07:56:08:WU06:FS01:Upload 0.97%
2019-10-03:07:56:10:WU04:FS01:Upload 4.52%
2019-10-03:07:56:31:WU04:FS01:Upload 4.84%
2019-10-03:07:56:33:WU02:FS01:Upload 0.32%
2019-10-03:07:56:36:WU06:FS01:Upload 1.29%
2019-10-03:07:56:40:WU04:FS01:Upload 5.16%
2019-10-03:07:56:44:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:840 clone:1 gen:24 core:0x21 unit:0x0000002180fccb0a5d716f9c57a670b1
2019-10-03:07:56:44:WU01:FS01:Uploading 65.77MiB to 128.252.203.10
2019-10-03:07:56:44:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:56:45:WU02:FS01:Upload 0.64%
2019-10-03:07:56:53:WU06:FS01:Upload 1.61%
2019-10-03:07:57:01:WU01:FS01:Upload 0.10%
2019-10-03:07:57:17:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14250 run:266 clone:0 gen:53 core:0x21 unit:0x0000004680fccb0a5d6ed218434b43f3
2019-10-03:07:57:17:WU00:FS01:Uploading 66.76MiB to 128.252.203.10
2019-10-03:07:57:17:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-03:07:57:17:WU04:FS01:Upload 5.81%
2019-10-03:07:57:20:WU01:FS01:Upload 0.29%
2019-10-03:07:57:29:WU04:FS01:Upload 6.13%
2019-10-03:07:57:30:WU01:FS01:Upload 0.38%
2019-10-03:07:57:38:WU01:FS01:Upload 0.48%
2019-10-03:07:58:30:WU04:FS01:Upload 6.45%
2019-10-03:07:58:50:WU00:FS01:Upload 0.09%
2019-10-03:07:58:50:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:58:50:WU00:FS01:Trying to send results to collection server
2019-10-03:07:58:50:WU00:FS01:Uploading 66.76MiB to 155.247.166.219
2019-10-03:07:58:50:WU00:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:58:51:ERROR:WU00:FS01:Exception: Transfer failed
2019-10-03:07:58:55:WU06:FS01:Upload 1.93%
2019-10-03:07:58:55:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:58:55:WU06:FS01:Trying to send results to collection server
2019-10-03:07:58:55:WU06:FS01:Uploading 19.40MiB to 155.247.166.220
2019-10-03:07:58:55:WU06:FS01:Connecting to 155.247.166.220:8080
2019-10-03:07:59:05:WU06:FS01:Upload 0.32%
2019-10-03:07:59:12:WU06:FS01:Upload 0.64%
2019-10-03:07:59:19:WU06:FS01:Upload 0.97%
2019-10-03:07:59:27:WU06:FS01:Upload 1.29%
2019-10-03:07:59:29:WU01:FS01:Upload 0.57%
2019-10-03:07:59:29:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:07:59:29:WU01:FS01:Trying to send results to collection server
2019-10-03:07:59:29:WU01:FS01:Uploading 65.77MiB to 155.247.166.219
2019-10-03:07:59:29:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:59:30:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-03:07:59:37:WU02:FS01:Upload 0.97%
2019-10-03:07:59:37:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-03:07:59:37:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14196 run:14 clone:13 gen:0 core:0x21 unit:0x000000000002894b5d92468acdbde5bf
2019-10-03:07:59:37:WU02:FS01:Uploading 19.39MiB to 155.247.166.219
2019-10-03:07:59:37:WU02:FS01:Connecting to 155.247.166.219:8080
2019-10-03:07:59:47:WU06:FS01:Upload 1.61%
2019-10-03:07:59:52:WU02:FS01:Upload 0.32%
2019-10-03:08:00:02:WU02:FS01:Upload 0.64%
2019-10-03:08:00:17:WU02:FS01:Upload 1.29%
2019-10-03:08:00:19:WU06:FS01:Upload 1.93%
2019-10-03:08:01:16:WU06:FS01:Upload 2.26%
2019-10-03:08:01:36:WU06:FS01:Upload 2.58%
2019-10-03:08:01:42:WU04:FS01:Upload 6.77%
2019-10-03:08:01:42:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:08:01:42:WU04:FS01:Trying to send results to collection server
2019-10-03:08:01:42:WU04:FS01:Uploading 19.38MiB to 128.252.203.4
2019-10-03:08:01:42:WU04:FS01:Connecting to 128.252.203.4:8080
2019-10-03:08:01:52:WU04:FS01:Upload 0.32%
2019-10-03:08:02:05:WU04:FS01:Upload 0.65%
2019-10-03:08:02:11:WU02:FS01:Upload 1.61%
2019-10-03:08:02:11:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:08:02:11:WU02:FS01:Trying to send results to collection server
2019-10-03:08:02:11:WU02:FS01:Uploading 19.39MiB to 128.252.203.4
2019-10-03:08:02:11:WU02:FS01:Connecting to 128.252.203.4:8080
2019-10-03:08:02:17:WU02:FS01:Upload 0.64%
2019-10-03:08:02:25:WU02:FS01:Upload 1.29%
2019-10-03:08:02:51:WU02:FS01:Upload 1.93%
2019-10-03:08:03:03:WU02:FS01:Upload 2.26%
2019-10-03:08:03:10:WU02:FS01:Upload 2.90%
2019-10-03:08:03:18:WU02:FS01:Upload 3.55%
2019-10-03:08:03:34:WU06:FS01:Upload 2.90%
2019-10-03:08:03:34:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-03:08:03:34:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14196 run:5 clone:6 gen:2 core:0x21 unit:0x000000020002894b5d9246573bee776e
2019-10-03:08:03:34:WU06:FS01:Uploading 19.40MiB to 155.247.166.219
2019-10-03:08:03:34:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-03:08:03:35:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:14249 run:840 clone:1 gen:24 core:0x21 unit:0x0000002180fccb0a5d716f9c57a670b1
2019-10-03:08:03:35:WU01:FS01:Uploading 65.77MiB to 128.252.203.10
2019-10-03:08:03:35:WU01:FS01:Connecting to 128.252.203.10:8080
2019-10-03:08:03:41:WU01:FS01:Upload 0.10%
2019-10-03:08:03:49:WU06:FS01:Upload 0.32%
2019-10-03:08:04:03:WU01:FS01:Upload 0.19%
2019-10-03:08:04:09:WU01:FS01:Upload 0.29%
2019-10-03:08:04:15:WU04:FS01:Upload 0.97%
2019-10-03:08:04:15:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-03:08:04:15:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14196 run:1 clone:7 gen:0 core:0x21 unit:0x000000000002894b5d9246403f27a828
2019-10-03:08:04:15:WU04:FS01:Uploading 19.38MiB to 155.247.166.219
2019-10-03:08:04:15:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-03:08:04:15:WU01:FS01:Upload 0.38%
2019-10-03:08:04:30:WU01:FS01:Upload 0.48%
2019-10-03:08:04:39:WU01:FS01:Upload 0.57%
2019-10-03:08:04:45:WU01:FS01:Upload 0.67%
2019-10-03:08:04:53:WU04:FS01:Upload 0.65%
2019-10-03:08:05:36:WU06:FS01:Upload 0.64%
2019-10-03:08:05:36:WARNING:WU06:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:08:05:36:WU06:FS01:Trying to send results to collection server
2019-10-03:08:05:36:WU06:FS01:Uploading 19.40MiB to 155.247.166.220
2019-10-03:08:05:36:WU06:FS01:Connecting to 155.247.166.220:8080
2019-10-03:08:05:40:WU02:FS01:Upload 3.87%
2019-10-03:08:05:40:ERROR:WU02:FS01:Exception: Transfer failed
2019-10-03:08:05:40:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:14196 run:14 clone:13 gen:0 core:0x21 unit:0x000000000002894b5d92468acdbde5bf
2019-10-03:08:05:40:WU02:FS01:Uploading 19.39MiB to 155.247.166.219
2019-10-03:08:05:40:WU02:FS01:Connecting to 155.247.166.219:8080
2019-10-03:08:06:05:WU06:FS01:Upload 0.32%
2019-10-03:08:06:12:WU02:FS01:Upload 0.32%
2019-10-03:08:06:47:WU04:FS01:Upload 0.97%
2019-10-03:08:06:47:WU02:FS01:Upload 0.64%
2019-10-03:08:06:47:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:08:06:47:WU04:FS01:Trying to send results to collection server
2019-10-03:08:06:47:WU04:FS01:Uploading 19.38MiB to 128.252.203.4
2019-10-03:08:06:47:WU04:FS01:Connecting to 128.252.203.4:8080
2019-10-03:08:06:55:WU04:FS01:Upload 0.65%
2019-10-03:08:07:05:WU04:FS01:Upload 1.29%
2019-10-03:08:07:23:WU02:FS01:Upload 0.97%
2019-10-03:08:07:25:WU04:FS01:Upload 1.94%
2019-10-03:08:07:51:WU06:FS01:Upload 0.64%
2019-10-03:08:07:51:ERROR:WU06:FS01:Exception: Transfer failed
2019-10-03:08:07:51:WU06:FS01:Sending unit results: id:06 state:SEND error:NO_ERROR project:14196 run:5 clone:6 gen:2 core:0x21 unit:0x000000020002894b5d9246573bee776e
2019-10-03:08:07:51:WU06:FS01:Uploading 19.40MiB to 155.247.166.219
2019-10-03:08:07:51:WU06:FS01:Connecting to 155.247.166.219:8080
2019-10-03:08:07:57:WU01:FS01:Upload 0.76%
2019-10-03:08:07:57:WU06:FS01:Upload 0.32%
2019-10-03:08:07:57:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:08:07:57:WU01:FS01:Trying to send results to collection server
2019-10-03:08:07:57:WU01:FS01:Uploading 65.77MiB to 155.247.166.219
2019-10-03:08:07:57:WU01:FS01:Connecting to 155.247.166.219:8080
2019-10-03:08:07:58:ERROR:WU01:FS01:Exception: Transfer failed
2019-10-03:08:08:09:WU06:FS01:Upload 0.64%
2019-10-03:08:08:22:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14250 run:266 clone:0 gen:53 core:0x21 unit:0x0000004680fccb0a5d6ed218434b43f3
2019-10-03:08:08:22:WU00:FS01:Uploading 66.76MiB to 128.252.203.10
2019-10-03:08:08:22:WU00:FS01:Connecting to 128.252.203.10:8080
2019-10-03:08:08:51:WU04:FS01:Upload 2.26%
2019-10-03:08:08:51:ERROR:WU04:FS01:Exception: Transfer failed
2019-10-03:08:08:51:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:14196 run:1 clone:7 gen:0 core:0x21 unit:0x000000000002894b5d9246403f27a828
2019-10-03:08:08:51:WU04:FS01:Uploading 19.38MiB to 155.247.166.219
2019-10-03:08:08:51:WU04:FS01:Connecting to 155.247.166.219:8080
2019-10-03:08:09:01:WU06:FS01:Upload 0.97%
2019-10-03:08:09:03:WU02:FS01:Upload 1.29%
2019-10-03:08:09:03:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
2019-10-03:08:09:03:WU02:FS01:Trying to send results to collection server
2019-10-03:08:09:03:WU02:FS01:Uploading 19.39MiB to 128.252.203.4
2019-10-03:08:09:03:WU02:FS01:Connecting to 128.252.203.4:8080
2019-10-03:08:09:07:WU06:FS01:Upload 1.29%
Can the collection servers be rebooted daily (or 3 times a week) until more servers are brought online that can handle the larger WU file size congestion?
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by bruce »

From your report, it looks like only three Collection Servers are being used.

155.247.166.219
155.247.166.220
128.252.203.4
(I don't really consider the first two because they're also the Work Servers at Temple)

Is anybody seeing slow uploads to other CSs?
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by bruce »

HaloJones wrote:@Bruce, which cache please and how to clear it?
See things like https://clear-my-cache.com/en/windows.html
Top
dfgirl12
Posts: 38
Joined: Fri Aug 21, 2009 8:34 am

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by dfgirl12 »

Roughly 1-2 hours after the (2) servers got rebooted (from the uptime here: https://apps.foldingathome.org/serverstats), my backlog of WUs finished uploading.
Thanks! The server reboots do make a difference, but are obviously not ideal.
Top
FaaR
Posts: 66
Joined: Tue Aug 19, 2008 1:32 am

Slow WU downloads...

Post by FaaR »

Still getting really slow downloads, sadly. Took a pause for some days while tweaking my CPU to get boost clocks working properly on it again and when I come back to folding, downloading 2x 70+ MB GPU WUs took nearly 20 minutes. It's like being back in the good old POTS modem days again... ;)

Code: Select all

*********************** Log Started 2019-10-04T19:55:43Z ***********************
19:55:43:************************* Folding@home Client *************************
19:55:43:      Website: http://folding.stanford.edu/
19:55:43:    Copyright: (c) 2009-2014 Stanford University
19:55:43:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:55:43:         Args: 
19:55:43:       Config: C:/Users/faarg/AppData/Roaming/FAHClient/config.xml
19:55:43:******************************** Build ********************************
19:55:43:      Version: 7.4.4
19:55:43:         Date: Mar 4 2014
19:55:43:         Time: 20:26:54
19:55:43:      SVN Rev: 4130
19:55:43:       Branch: fah/trunk/client
19:55:43:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
19:55:43:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
19:55:43:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
19:55:43:     Platform: win32 XP
19:55:43:         Bits: 32
19:55:43:         Mode: Release
19:55:43:******************************* System ********************************
19:55:43:          CPU: Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz
19:55:43:       CPU ID: GenuineIntel Family 6 Model 85 Stepping 4
19:55:43:         CPUs: 20
19:55:43:       Memory: 63.68GiB
19:55:43:  Free Memory: 55.81GiB
19:55:43:      Threads: WINDOWS_THREADS
19:55:43:   OS Version: 6.2
19:55:43:  Has Battery: true
19:55:43:   On Battery: false
19:55:43:   UTC Offset: 2
19:55:43:          PID: 3984
19:55:43:          CWD: C:/Users/faarg/AppData/Roaming/FAHClient
19:55:43:           OS: Windows 10 Enterprise
19:55:43:      OS Arch: AMD64
19:55:43:         GPUs: 2
19:55:43:        GPU 0: ATI:5 [Radeon Rx vega]
19:55:43:        GPU 1: ATI:5 [Radeon Rx vega]
19:55:43:         CUDA: Not detected
19:55:43:Win32 Service: false
19:55:43:***********************************************************************
19:55:43:<config>
19:55:43:  <!-- Slot Control -->
19:55:43:  <power v='MEDIUM'/>
19:55:43:
19:55:43:  <!-- User Information -->
19:55:43:  <passkey v='********************************'/>
19:55:43:  <team v='37451'/>
19:55:43:  <user v='Kapten_Stofil'/>
19:55:43:
19:55:43:  <!-- Folding Slots -->
19:55:43:  <slot id='0' type='CPU'/>
19:55:43:  <slot id='1' type='GPU'>
19:55:43:    <paused v='true'/>
19:55:43:  </slot>
19:55:43:  <slot id='2' type='GPU'>
19:55:43:    <paused v='true'/>
19:55:43:  </slot>
19:55:43:</config>
19:55:44:Trying to access database...
19:55:44:Successfully acquired database lock
19:55:44:Enabled folding slot 00: READY cpu:17
19:55:44:Enabled folding slot 01: PAUSED gpu:0:[Radeon Rx vega] (by user)
19:55:44:Enabled folding slot 02: PAUSED gpu:1:[Radeon Rx vega] (by user)
19:55:44:WU00:FS00:Connecting to 65.254.110.245:8080
19:55:45:WU00:FS00:Assigned to work server 128.252.203.9
19:55:45:WU00:FS00:Requesting new work unit for slot 00: READY cpu:17 from 128.252.203.9
19:55:45:WU00:FS00:Connecting to 128.252.203.9:8080
19:55:47:WU00:FS00:Downloading 8.14MiB
19:55:54:WU00:FS00:Download 11.52%
19:56:00:WU00:FS00:Download 16.12%
19:56:01:FS01:Unpaused
19:56:01:FS02:Unpaused
19:56:02:WU01:FS01:Connecting to 65.254.110.245:80
19:56:02:WU02:FS02:Connecting to 65.254.110.245:80
19:56:02:WU01:FS01:Assigned to work server 128.252.203.10
19:56:02:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:[Radeon Rx vega] from 128.252.203.10
19:56:02:WU02:FS02:Assigned to work server 128.252.203.10
19:56:02:WU01:FS01:Connecting to 128.252.203.10:8080
19:56:02:WU02:FS02:Requesting new work unit for slot 02: READY gpu:1:[Radeon Rx vega] from 128.252.203.10
19:56:02:WU02:FS02:Connecting to 128.252.203.10:8080
19:56:04:WU01:FS01:Downloading 72.63MiB
19:56:04:WU02:FS02:Downloading 65.72MiB
19:56:06:WU00:FS00:Download 19.96%
19:56:10:WU01:FS01:Download 0.95%
19:56:11:WU02:FS02:Download 0.67%
19:56:12:WU00:FS00:Download 24.57%
19:56:16:WU01:FS01:Download 1.63%
19:56:17:WU02:FS02:Download 1.33%
19:56:18:WU00:FS00:Download 28.41%
19:56:22:WU01:FS01:Download 2.32%
19:56:23:WU02:FS02:Download 2.19%
19:56:24:WU00:FS00:Download 34.55%
19:56:29:WU01:FS01:Download 3.01%
19:56:29:WU02:FS02:Download 2.76%
19:56:30:WU00:FS00:Download 37.62%
19:56:35:WU01:FS01:Download 3.53%
19:56:36:WU00:FS00:Download 42.23%
19:56:36:WU02:FS02:Download 3.33%
19:56:41:WU01:FS01:Download 3.96%
19:56:42:WU02:FS02:Download 3.71%
19:56:42:WU00:FS00:Download 46.84%
19:56:45:Removing old file 'configs/config-20190822-081556.xml'
19:56:45:Saving configuration to config.xml
19:56:45:<config>
19:56:45:  <!-- Slot Control -->
19:56:45:  <power v='MEDIUM'/>
19:56:45:
19:56:45:  <!-- User Information -->
19:56:45:  <passkey v='********************************'/>
19:56:45:  <team v='37451'/>
19:56:45:  <user v='Kapten_Stofil'/>
19:56:45:
19:56:45:  <!-- Folding Slots -->
19:56:45:  <slot id='0' type='CPU'/>
19:56:45:  <slot id='1' type='GPU'/>
19:56:45:  <slot id='2' type='GPU'/>
19:56:45:</config>
19:56:47:WU01:FS01:Download 4.47%
19:56:48:WU00:FS00:Download 52.98%
19:56:48:WU02:FS02:Download 4.47%
19:56:53:WU01:FS01:Download 5.16%
19:56:54:WU02:FS02:Download 4.95%
19:56:54:WU00:FS00:Download 59.12%
19:56:59:WU01:FS01:Download 5.51%
19:57:00:WU00:FS00:Download 62.96%
19:57:00:WU02:FS02:Download 5.61%
19:57:05:WU01:FS01:Download 5.85%
19:57:06:WU02:FS02:Download 6.09%
19:57:06:WU00:FS00:Download 67.57%
19:57:11:WU01:FS01:Download 6.37%
19:57:12:WU02:FS02:Download 6.66%
19:57:12:WU00:FS00:Download 71.41%
19:57:17:WU01:FS01:Download 6.80%
19:57:18:WU02:FS02:Download 7.42%
19:57:19:WU00:FS00:Download 76.01%
19:57:23:WU01:FS01:Download 7.31%
19:57:24:WU02:FS02:Download 8.08%
19:57:26:WU00:FS00:Download 81.39%
19:57:29:WU01:FS01:Download 7.83%
19:57:30:WU02:FS02:Download 8.84%
19:57:32:WU00:FS00:Download 87.53%
19:57:35:WU01:FS01:Download 8.26%
19:57:37:WU02:FS02:Download 9.61%
19:57:38:WU00:FS00:Download 92.90%
19:57:41:WU01:FS01:Download 8.61%
19:57:43:WU02:FS02:Download 10.18%
19:57:44:WU00:FS00:Download 96.74%
19:57:47:WU01:FS01:Download 9.04%
19:57:48:WU00:FS00:Download complete
19:57:48:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13829 run:460 clone:0 gen:1 core:0xa7 unit:0x0000000180fccb095d66d3d587400d64
19:57:48:WU00:FS00:Starting
19:57:48:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/faarg/AppData/Roaming/FAHClient/cores/cores.foldingathome.org/Win32/x86/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 704 -lifeline 3984 -checkpoint 15 -np 17
19:57:48:WU00:FS00:Started FahCore on PID 4076
19:57:49:WU00:FS00:Core PID:4252
19:57:49:WU00:FS00:FahCore 0xa7 started
19:57:49:WU02:FS02:Download 10.84%
19:57:50:WU00:FS00:0xa7:*********************** Log Started 2019-10-04T19:57:49Z ***********************
19:57:50:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
19:57:50:WU00:FS00:0xa7:       Type: 0xa7
19:57:50:WU00:FS00:0xa7:       Core: Gromacs
19:57:50:WU00:FS00:0xa7:    Website: https://foldingathome.org/
19:57:50:WU00:FS00:0xa7:  Copyright: (c) 2009-2018 foldingathome.org
19:57:50:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:57:50:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 704 -lifeline 4076 -checkpoint 15 -np
19:57:50:WU00:FS00:0xa7:             17
19:57:50:WU00:FS00:0xa7:     Config: <none>
19:57:50:WU00:FS00:0xa7:************************************ Build *************************************
19:57:50:WU00:FS00:0xa7:    Version: 0.0.17
19:57:50:WU00:FS00:0xa7:       Date: Apr 25 2018
19:57:50:WU00:FS00:0xa7:       Time: 11:02:26
19:57:50:WU00:FS00:0xa7: Repository: Git
19:57:50:WU00:FS00:0xa7:   Revision: fd11abfb405c921e66db1226933e9dd2d18d2acc
19:57:50:WU00:FS00:0xa7:     Branch: master
19:57:50:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
19:57:50:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
19:57:50:WU00:FS00:0xa7:   Platform: win32 10
19:57:50:WU00:FS00:0xa7:       Bits: 32
19:57:50:WU00:FS00:0xa7:       Mode: Release
19:57:50:WU00:FS00:0xa7:       SIMD: sse2
19:57:50:WU00:FS00:0xa7:************************************ System ************************************
19:57:50:WU00:FS00:0xa7:        CPU: Unknown
19:57:50:WU00:FS00:0xa7:     CPU ID: 
19:57:50:WU00:FS00:0xa7:       CPUs: 20
19:57:50:WU00:FS00:0xa7:     Memory: 63.68GiB
19:57:50:WU00:FS00:0xa7:Free Memory: 55.75GiB
19:57:50:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
19:57:50:WU00:FS00:0xa7: OS Version: 6.2
19:57:50:WU00:FS00:0xa7:Has Battery: true
19:57:50:WU00:FS00:0xa7: On Battery: false
19:57:50:WU00:FS00:0xa7: UTC Offset: 2
19:57:50:WU00:FS00:0xa7:        PID: 4252
19:57:50:WU00:FS00:0xa7:        CWD: C:\Users\faarg\AppData\Roaming\FAHClient\work
19:57:50:WU00:FS00:0xa7:         OS: Windows 10 Enterprise
19:57:50:WU00:FS00:0xa7:    OS Arch: AMD64
19:57:50:WU00:FS00:0xa7:********************************************************************************
19:57:50:WU00:FS00:0xa7:Project: 13829 (Run 460, Clone 0, Gen 1)
19:57:50:WU00:FS00:0xa7:Unit: 0x0000000180fccb095d66d3d587400d64
19:57:50:WU00:FS00:0xa7:Reading tar file core.xml
19:57:50:WU00:FS00:0xa7:Reading tar file frame1.tpr
19:57:50:WU00:FS00:0xa7:Digital signatures verified
19:57:50:WU00:FS00:0xa7:Reducing thread count from 17 to 16 to avoid domain decomposition by a prime number > 3
19:57:50:WU00:FS00:0xa7:Calling: mdrun -s frame1.tpr -o frame1.trr -x frame1.xtc -cpt 15 -nt 16
19:57:50:WU00:FS00:0xa7:Steps: first=125000 total=125000
19:57:54:WU01:FS01:Download 9.64%
19:57:55:WU02:FS02:Download 11.51%
19:57:56:WU00:FS00:0xa7:Completed 1 out of 125000 steps (0%)
19:58:00:WU01:FS01:Download 10.24%
19:58:01:WU02:FS02:Download 12.08%
19:58:06:WU01:FS01:Download 10.67%
19:58:07:WU02:FS02:Download 12.93%
19:58:12:WU01:FS01:Download 11.44%
19:58:13:WU02:FS02:Download 13.60%
19:58:18:WU01:FS01:Download 12.13%
19:58:19:WU02:FS02:Download 14.74%
19:58:24:WU01:FS01:Download 12.65%
19:58:25:WU02:FS02:Download 15.50%
19:58:30:WU01:FS01:Download 12.99%
19:58:31:WU02:FS02:Download 15.79%
19:58:36:WU01:FS01:Download 13.42%
19:58:37:WU02:FS02:Download 16.45%
19:58:42:WU01:FS01:Download 13.85%
19:58:43:WU02:FS02:Download 16.93%
19:58:48:WU01:FS01:Download 14.28%
19:58:48:WU00:FS00:0xa7:Completed 1250 out of 125000 steps (1%)
19:58:49:WU02:FS02:Download 17.69%
19:58:54:WU01:FS01:Download 14.71%
19:58:55:WU02:FS02:Download 18.26%
19:59:00:WU01:FS01:Download 15.15%
19:59:01:WU02:FS02:Download 18.64%
19:59:07:WU01:FS01:Download 15.49%
19:59:07:WU02:FS02:Download 19.21%
19:59:13:WU02:FS02:Download 19.69%
19:59:13:WU01:FS01:Download 16.09%
19:59:19:WU02:FS02:Download 20.26%
19:59:20:WU01:FS01:Download 16.52%
19:59:25:WU02:FS02:Download 21.11%
19:59:26:WU01:FS01:Download 16.95%
19:59:31:WU02:FS02:Download 21.59%
19:59:33:WU01:FS01:Download 17.47%
19:59:37:WU02:FS02:Download 22.16%
19:59:40:WU01:FS01:Download 17.98%
19:59:41:WU00:FS00:0xa7:Completed 2500 out of 125000 steps (2%)
19:59:43:WU02:FS02:Download 23.02%
19:59:46:WU01:FS01:Download 18.67%
19:59:49:WU02:FS02:Download 23.59%
19:59:53:WU01:FS01:Download 19.28%
19:59:55:WU02:FS02:Download 23.97%
19:59:59:WU01:FS01:Download 19.88%
20:00:01:WU02:FS02:Download 24.35%
20:00:05:WU01:FS01:Download 20.57%
20:00:07:WU02:FS02:Download 24.92%
20:00:11:WU01:FS01:Download 21.00%
20:00:14:WU02:FS02:Download 25.58%
20:00:17:WU01:FS01:Download 21.77%
20:00:20:WU02:FS02:Download 26.34%
20:00:23:WU01:FS01:Download 22.55%
20:00:26:WU02:FS02:Download 26.91%
20:00:29:WU01:FS01:Download 23.23%
20:00:32:WU00:FS00:0xa7:Completed 3750 out of 125000 steps (3%)
20:00:32:WU02:FS02:Download 27.58%
20:00:35:WU01:FS01:Download 23.66%
20:00:38:WU02:FS02:Download 28.34%
20:00:41:WU01:FS01:Download 24.18%
20:00:45:WU02:FS02:Download 29.20%
20:00:47:WU01:FS01:Download 24.70%
20:00:51:WU02:FS02:Download 29.58%
20:00:54:WU01:FS01:Download 25.30%
20:00:57:WU02:FS02:Download 30.24%
20:01:00:WU01:FS01:Download 25.90%
20:01:03:WU02:FS02:Download 31.00%
20:01:06:WU01:FS01:Download 26.50%
20:01:09:WU02:FS02:Download 31.77%
20:01:12:WU01:FS01:Download 26.93%
20:01:16:WU02:FS02:Download 32.24%
20:01:18:WU01:FS01:Download 27.36%
20:01:22:WU02:FS02:Download 32.81%
20:01:23:WU00:FS00:0xa7:Completed 5000 out of 125000 steps (4%)
20:01:24:WU01:FS01:Download 27.79%
20:01:29:WU02:FS02:Download 33.48%
20:01:30:WU01:FS01:Download 28.31%
20:01:35:WU02:FS02:Download 34.05%
20:01:36:WU01:FS01:Download 28.91%
20:01:42:WU02:FS02:Download 34.43%
20:01:42:WU01:FS01:Download 29.77%
20:01:48:WU01:FS01:Download 30.20%
20:01:49:WU02:FS02:Download 34.90%
20:01:54:WU01:FS01:Download 30.72%
20:01:55:WU02:FS02:Download 35.28%
20:02:00:WU01:FS01:Download 31.41%
20:02:01:WU02:FS02:Download 36.14%
20:02:06:WU01:FS01:Download 31.93%
20:02:07:WU02:FS02:Download 36.81%
20:02:12:WU01:FS01:Download 32.53%
20:02:13:WU02:FS02:Download 37.38%
20:02:14:WU00:FS00:0xa7:Completed 6250 out of 125000 steps (5%)
20:02:18:WU01:FS01:Download 33.04%
20:02:19:WU02:FS02:Download 37.95%
20:02:24:WU01:FS01:Download 33.56%
20:02:25:WU02:FS02:Download 38.61%
20:02:30:WU01:FS01:Download 34.42%
20:02:31:WU02:FS02:Download 39.28%
20:02:37:WU01:FS01:Download 35.02%
20:02:38:WU02:FS02:Download 39.66%
20:02:43:WU01:FS01:Download 35.80%
20:02:44:WU02:FS02:Download 40.42%
20:02:49:WU01:FS01:Download 36.57%
20:02:50:WU02:FS02:Download 41.18%
20:02:55:WU01:FS01:Download 37.35%
20:02:56:WU02:FS02:Download 42.13%
20:03:02:WU02:FS02:Download 42.89%
20:03:02:WU01:FS01:Download 37.95%
20:03:06:WU00:FS00:0xa7:Completed 7500 out of 125000 steps (6%)
20:03:08:WU02:FS02:Download 43.37%
20:03:08:WU01:FS01:Download 38.64%
20:03:14:WU01:FS01:Download 39.33%
20:03:15:WU02:FS02:Download 43.94%
20:03:20:WU01:FS01:Download 40.10%
20:03:21:WU02:FS02:Download 44.51%
20:03:26:WU01:FS01:Download 40.62%
20:03:28:WU02:FS02:Download 44.98%
20:03:32:WU01:FS01:Download 41.22%
20:03:35:WU02:FS02:Download 45.46%
20:03:38:WU01:FS01:Download 42.17%
20:03:41:WU02:FS02:Download 46.22%
20:03:45:WU01:FS01:Download 42.77%
20:03:47:WU02:FS02:Download 47.08%
20:03:51:WU01:FS01:Download 43.28%
20:03:53:WU02:FS02:Download 47.65%
20:03:57:WU00:FS00:0xa7:Completed 8750 out of 125000 steps (7%)
20:03:57:WU01:FS01:Download 43.71%
20:04:00:WU02:FS02:Download 48.22%
20:04:03:WU01:FS01:Download 44.23%
20:04:06:WU02:FS02:Download 48.88%
20:04:09:WU01:FS01:Download 45.18%
20:04:12:WU02:FS02:Download 49.55%
20:04:15:WU01:FS01:Download 45.95%
20:04:18:WU02:FS02:Download 50.22%
20:04:21:WU01:FS01:Download 46.47%
20:04:24:WU02:FS02:Download 50.88%
20:04:27:WU01:FS01:Download 47.16%
20:04:30:WU02:FS02:Download 51.74%
20:04:33:WU01:FS01:Download 47.67%
20:04:36:WU02:FS02:Download 52.31%
20:04:39:WU01:FS01:Download 48.45%
20:04:42:WU02:FS02:Download 52.88%
20:04:45:WU01:FS01:Download 48.96%
20:04:48:WU00:FS00:0xa7:Completed 10000 out of 125000 steps (8%)
20:04:48:WU02:FS02:Download 53.45%
20:04:51:WU01:FS01:Download 49.48%
20:04:54:WU02:FS02:Download 53.92%
20:04:57:WU01:FS01:Download 50.00%
20:05:00:WU02:FS02:Download 54.50%
20:05:03:WU01:FS01:Download 50.68%
20:05:06:WU02:FS02:Download 55.16%
20:05:09:WU01:FS01:Download 51.20%
20:05:12:WU02:FS02:Download 55.83%
20:05:15:WU01:FS01:Download 52.06%
20:05:18:WU02:FS02:Download 56.21%
20:05:21:WU01:FS01:Download 52.49%
20:05:24:WU02:FS02:Download 56.97%
20:05:27:WU01:FS01:Download 53.18%
20:05:30:WU02:FS02:Download 57.44%
20:05:33:WU01:FS01:Download 53.87%
20:05:36:WU02:FS02:Download 58.01%
20:05:39:WU01:FS01:Download 54.38%
20:05:39:WU00:FS00:0xa7:Completed 11250 out of 125000 steps (9%)
20:05:42:WU02:FS02:Download 58.97%
20:05:45:WU01:FS01:Download 54.90%
20:05:48:WU02:FS02:Download 59.54%
20:05:51:WU01:FS01:Download 55.50%
20:05:54:WU02:FS02:Download 60.01%
20:05:57:WU01:FS01:Download 56.02%
20:06:00:WU02:FS02:Download 60.68%
20:06:03:WU01:FS01:Download 56.71%
20:06:06:WU02:FS02:Download 61.34%
20:06:09:WU01:FS01:Download 57.31%
20:06:12:WU02:FS02:Download 61.91%
20:06:15:WU01:FS01:Download 57.65%
20:06:19:WU02:FS02:Download 62.58%
20:06:21:WU01:FS01:Download 58.00%
20:06:25:WU02:FS02:Download 63.24%
20:06:28:WU01:FS01:Download 58.60%
20:06:30:WU00:FS00:0xa7:Completed 12500 out of 125000 steps (10%)
20:06:31:WU02:FS02:Download 63.72%
20:06:34:WU01:FS01:Download 58.95%
20:06:37:WU02:FS02:Download 64.29%
20:06:40:WU01:FS01:Download 59.38%
20:06:43:WU02:FS02:Download 64.96%
20:06:46:WU01:FS01:Download 59.81%
20:06:49:WU02:FS02:Download 65.91%
20:06:53:WU01:FS01:Download 60.49%
20:06:56:WU02:FS02:Download 66.67%
20:06:59:WU01:FS01:Download 61.01%
20:07:02:WU02:FS02:Download 67.14%
20:07:05:WU01:FS01:Download 61.44%
20:07:08:WU02:FS02:Download 67.71%
20:07:12:WU01:FS01:Download 61.87%
20:07:14:WU02:FS02:Download 68.29%
20:07:18:WU01:FS01:Download 62.30%
20:07:20:WU02:FS02:Download 68.95%
20:07:21:WU00:FS00:0xa7:Completed 13750 out of 125000 steps (11%)
20:07:24:WU01:FS01:Download 62.82%
20:07:26:WU02:FS02:Download 69.52%
20:07:30:WU01:FS01:Download 63.59%
20:07:32:WU02:FS02:Download 69.90%
20:07:36:WU01:FS01:Download 64.19%
20:07:38:WU02:FS02:Download 70.38%
20:07:42:WU01:FS01:Download 64.80%
20:07:44:WU02:FS02:Download 70.85%
20:07:48:WU01:FS01:Download 65.40%
20:07:50:WU02:FS02:Download 71.52%
20:07:54:WU01:FS01:Download 66.09%
20:07:56:WU02:FS02:Download 72.18%
20:08:00:WU01:FS01:Download 66.69%
20:08:02:WU02:FS02:Download 72.57%
20:08:06:WU01:FS01:Download 67.21%
20:08:08:WU02:FS02:Download 73.04%
20:08:12:WU01:FS01:Download 67.81%
20:08:12:WU00:FS00:0xa7:Completed 15000 out of 125000 steps (12%)
20:08:14:WU02:FS02:Download 73.61%
20:08:18:WU01:FS01:Download 68.41%
20:08:20:WU02:FS02:Download 74.09%
20:08:24:WU01:FS01:Download 68.93%
20:08:26:WU02:FS02:Download 74.56%
20:08:31:WU01:FS01:Download 69.53%
20:08:32:WU02:FS02:Download 75.23%
20:08:37:WU01:FS01:Download 70.05%
20:08:38:WU02:FS02:Download 75.89%
20:08:43:WU01:FS01:Download 70.48%
20:08:44:WU02:FS02:Download 76.56%
20:08:50:WU01:FS01:Download 71.34%
20:08:51:WU02:FS02:Download 77.32%
20:08:56:WU01:FS01:Download 71.77%
20:08:57:WU02:FS02:Download 78.18%
20:09:02:WU01:FS01:Download 72.63%
20:09:03:WU00:FS00:0xa7:Completed 16250 out of 125000 steps (13%)
20:09:04:WU02:FS02:Download 78.75%
20:09:08:WU01:FS01:Download 73.40%
20:09:11:WU02:FS02:Download 79.41%
20:09:14:WU01:FS01:Download 73.92%
20:09:17:WU02:FS02:Download 79.89%
20:09:21:WU01:FS01:Download 74.44%
20:09:23:WU02:FS02:Download 80.36%
20:09:27:WU01:FS01:Download 75.04%
20:09:29:WU02:FS02:Download 80.93%
20:09:33:WU01:FS01:Download 75.90%
20:09:35:WU02:FS02:Download 81.41%
20:09:39:WU01:FS01:Download 76.50%
20:09:41:WU02:FS02:Download 82.08%
20:09:45:WU01:FS01:Download 77.10%
20:09:47:WU02:FS02:Download 82.36%
20:09:51:WU01:FS01:Download 77.53%
20:09:54:WU02:FS02:Download 82.93%
20:09:55:WU00:FS00:0xa7:Completed 17500 out of 125000 steps (14%)
20:09:57:WU01:FS01:Download 77.88%
20:10:00:WU02:FS02:Download 83.50%
20:10:03:WU01:FS01:Download 78.31%
20:10:06:WU02:FS02:Download 84.17%
20:10:09:WU01:FS01:Download 78.82%
20:10:12:WU02:FS02:Download 84.74%
20:10:15:WU01:FS01:Download 79.17%
20:10:18:WU02:FS02:Download 85.50%
20:10:22:WU01:FS01:Download 79.60%
20:10:24:WU02:FS02:Download 86.07%
20:10:28:WU01:FS01:Download 80.20%
20:10:30:WU02:FS02:Download 86.64%
20:10:34:WU01:FS01:Download 80.72%
20:10:36:WU02:FS02:Download 87.12%
20:10:40:WU01:FS01:Download 81.49%
20:10:43:WU02:FS02:Download 87.78%
20:10:46:WU00:FS00:0xa7:Completed 18750 out of 125000 steps (15%)
20:10:46:WU01:FS01:Download 81.92%
20:10:49:WU02:FS02:Download 88.45%
20:10:52:WU01:FS01:Download 82.44%
20:10:55:WU02:FS02:Download 89.02%
20:10:58:WU01:FS01:Download 82.95%
20:11:02:WU02:FS02:Download 89.78%
20:11:04:WU01:FS01:Download 83.56%
20:11:08:WU02:FS02:Download 90.35%
20:11:10:WU01:FS01:Download 84.07%
20:11:14:WU02:FS02:Download 91.30%
20:11:16:WU01:FS01:Download 84.59%
20:11:20:WU02:FS02:Download 91.97%
20:11:22:WU01:FS01:Download 85.11%
20:11:26:WU02:FS02:Download 92.73%
20:11:28:WU01:FS01:Download 85.54%
20:11:32:WU02:FS02:Download 93.49%
20:11:35:WU01:FS01:Download 85.88%
20:11:37:WU00:FS00:0xa7:Completed 20000 out of 125000 steps (16%)
20:11:38:WU02:FS02:Download 94.25%
20:11:41:WU01:FS01:Download 86.40%
20:11:44:WU02:FS02:Download 94.72%
20:11:47:WU01:FS01:Download 87.08%
20:11:50:WU02:FS02:Download 95.49%
20:11:53:WU01:FS01:Download 87.52%
20:11:56:WU02:FS02:Download 96.25%
20:11:59:WU01:FS01:Download 88.03%
20:12:02:WU02:FS02:Download 97.10%
20:12:05:WU01:FS01:Download 88.38%
20:12:08:WU02:FS02:Download 97.67%
20:12:11:WU01:FS01:Download 88.81%
20:12:14:WU02:FS02:Download 98.24%
20:12:18:WU01:FS01:Download 89.32%
20:12:20:WU02:FS02:Download 99.10%
20:12:24:WU01:FS01:Download 89.67%
20:12:26:WU02:FS02:Download 99.57%
20:12:28:WU00:FS00:0xa7:Completed 21250 out of 125000 steps (17%)
20:12:30:WU01:FS01:Download 90.27%
20:12:32:WU02:FS02:Download 99.96%
20:12:33:WU02:FS02:Download complete
20:12:33:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:14224 run:7 clone:1 gen:30 core:0x21 unit:0x0000002e80fccb0a5d6545d978b2b37f
20:12:33:WU02:FS02:Starting
20:12:33:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/faarg/AppData/Roaming/FAHClient/cores/cores.foldingathome.org/Win32/AMD64/ATI/R600/Core_21.fah/FahCore_21.exe -dir 02 -suffix 01 -version 704 -lifeline 3984 -checkpoint 15 -gpu 1 -gpu-vendor ati
20:12:33:WU02:FS02:Started FahCore on PID 22036
20:12:33:WU02:FS02:Core PID:5864
20:12:33:WU02:FS02:FahCore 0x21 started
20:12:34:WU02:FS02:0x21:*********************** Log Started 2019-10-04T20:12:33Z ***********************
20:12:34:WU02:FS02:0x21:Project: 14224 (Run 7, Clone 1, Gen 30)
20:12:34:WU02:FS02:0x21:Unit: 0x0000002e80fccb0a5d6545d978b2b37f
20:12:34:WU02:FS02:0x21:CPU: 0x00000000000000000000000000000000
20:12:34:WU02:FS02:0x21:Machine: 2
20:12:34:WU02:FS02:0x21:Reading tar file core.xml
20:12:34:WU02:FS02:0x21:Reading tar file integrator.xml
20:12:34:WU02:FS02:0x21:Reading tar file state.xml
20:12:34:WU02:FS02:0x21:Reading tar file system.xml
20:12:35:WU02:FS02:0x21:Digital signatures verified
20:12:35:WU02:FS02:0x21:Folding@home GPU Core21 Folding@home Core
20:12:35:WU02:FS02:0x21:Version 0.0.20
20:12:36:WU01:FS01:Download 91.04%
20:12:42:WU01:FS01:Download 91.56%
20:12:49:WU01:FS01:Download 91.90%
20:12:55:WU01:FS01:Download 92.42%
20:12:59:WU02:FS02:0x21:Completed 0 out of 2000000 steps (0%)
20:12:59:WU02:FS02:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
20:13:01:WU01:FS01:Download 93.02%
20:13:07:WU01:FS01:Download 93.54%
20:13:14:WU01:FS01:Download 93.97%
20:13:20:WU01:FS01:Download 94.49%
20:13:23:WU00:FS00:0xa7:Completed 22500 out of 125000 steps (18%)
20:13:26:WU01:FS01:Download 94.92%
20:13:32:WU01:FS01:Download 95.26%
20:13:38:WU01:FS01:Download 95.86%
20:13:44:WU01:FS01:Download 96.64%
20:13:50:WU01:FS01:Download 96.98%
20:13:56:WU01:FS01:Download 97.58%
20:14:02:WU01:FS01:Download 98.10%
20:14:08:WU01:FS01:Download 98.62%
20:14:14:WU01:FS01:Download 98.96%
20:14:14:WU00:FS00:0xa7:Completed 23750 out of 125000 steps (19%)
20:14:20:WU01:FS01:Download 99.56%
20:14:25:WU01:FS01:Download complete
20:14:25:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14233 run:3 clone:42 gen:28 core:0x21 unit:0x0000001c80fccb0a5d654915a85c885c
20:14:25:WU01:FS01:Starting
20:14:25:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/faarg/AppData/Roaming/FAHClient/cores/cores.foldingathome.org/Win32/AMD64/ATI/R600/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 3984 -checkpoint 15 -gpu 0 -gpu-vendor ati
20:14:25:WU01:FS01:Started FahCore on PID 13956
20:14:25:WU01:FS01:Core PID:13380
20:14:25:WU01:FS01:FahCore 0x21 started
20:14:26:WU01:FS01:0x21:*********************** Log Started 2019-10-04T20:14:25Z ***********************
20:14:26:WU01:FS01:0x21:Project: 14233 (Run 3, Clone 42, Gen 28)
20:14:26:WU01:FS01:0x21:Unit: 0x0000001c80fccb0a5d654915a85c885c
20:14:26:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
20:14:26:WU01:FS01:0x21:Machine: 1
20:14:26:WU01:FS01:0x21:Reading tar file core.xml
20:14:26:WU01:FS01:0x21:Reading tar file integrator.xml
20:14:26:WU01:FS01:0x21:Reading tar file state.xml
20:14:26:WU01:FS01:0x21:Reading tar file system.xml
20:14:27:WU01:FS01:0x21:Digital signatures verified
20:14:27:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
20:14:27:WU01:FS01:0x21:Version 0.0.20
20:14:51:WU01:FS01:0x21:Completed 0 out of 1000000 steps (0%)
20:14:51:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
20:15:00:WU02:FS02:0x21:Completed 20000 out of 2000000 steps (1%)
20:15:06:WU00:FS00:0xa7:Completed 25000 out of 125000 steps (20%)
20:15:57:WU00:FS00:0xa7:Completed 26250 out of 125000 steps (21%)
20:16:01:WU01:FS01:0x21:Completed 10000 out of 1000000 steps (1%)
20:16:49:WU00:FS00:0xa7:Completed 27500 out of 125000 steps (22%)
I didn't sort out the CPU client from the log or separate the two GPU clients - sorry. I'm kinda crap at this stuff. Anyway, hope this helps, guys. You're doing good work keeping this project running, thank you. :)
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by bruce »

Your really posting in the wrong topic. The problems that have been a challenge for most other donors have been with the two servers 155.247.166.* and your log doesn't mention either them.

The easiest way to sort out the clients from each slot is to use the filter in FAHControl's log viewer. Create a view for each slot and/or each WU. (There may be a few lines missing because the filter doesn't recognize them.)

Downloading all three WUs concurrently is going to be slow but that's probably due to congestion on your own ISP connection. When I find myself needing to download all 3 WUs, I pick the one that I want to start first and do just that one, especially if some other tasks are using the internet connection. That way one or two GPUs can be computing while the last one or two WUs are downloading.

The setting "next-unit-percentage" was specifically designed for slow connections back in the day that we all had a POTS modem.
Top
gordonbb
Posts: 511
Joined: Mon May 21, 2018 4:12 pm
Hardware configuration: Ubuntu 22.04.2 LTS; NVidia 525.60.11; 2 x 4070ti; 4070; 4060ti; 3x 3080; 3070ti; 3070
Location: Great White North

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by gordonbb »

Brutal, Brutal Brutal since ~ 18:00 EDT. Had to reboot systems multiple times. With 17 slots running It's been pretty much constant reboots. Downloads get to 20-30% then just freeze. Rebooting sometimes works the first time but on several systems I've had to reboot 3-4 times to get a new WU successfully downloaded from 155.247.166.220.

Here's the current one I'm fighting with:

Code: Select all

01:15:36:WU00:FS01:0x21:Completed 1960000 out of 2000000 steps (98%)
01:17:31:WU00:FS01:0x21:Completed 1980000 out of 2000000 steps (99%)
01:17:32:WU02:FS01:Connecting to 65.254.110.245:8080
01:17:32:WU02:FS01:Assigned to work server 155.247.166.220
01:17:32:WU02:FS01:Requesting new work unit for slot 01: RUNNING gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
01:17:32:WU02:FS01:Connecting to 155.247.166.220:8080
01:17:32:WU02:FS01:Downloading 15.87MiB
01:17:38:WU02:FS01:Download 3.15%
01:17:44:WU02:FS01:Download 6.30%
01:17:56:WU02:FS01:Download 10.63%
01:18:28:WU02:FS01:Download 11.82%
01:19:25:WU00:FS01:0x21:Completed 2000000 out of 2000000 steps (100%)
01:19:31:WU00:FS01:0x21:Saving result file logfile_01.txt
01:19:31:WU00:FS01:0x21:Saving result file checkpointState.xml
01:19:31:WU00:FS01:0x21:Saving result file checkpt.crc
01:19:31:WU00:FS01:0x21:Saving result file log.txt
01:19:31:WU00:FS01:0x21:Saving result file positions.xtc
01:19:32:WU00:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
01:19:32:WU00:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
01:19:32:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:14228 run:533 clone:0 gen:36 core:0x21 unit:0x0000003580fccb0a5d6553d8edd55a20
01:19:32:WU00:FS01:Uploading 94.21MiB to 128.252.203.10
01:19:32:WU00:FS01:Connecting to 128.252.203.10:8080
01:19:38:WU00:FS01:Upload 6.44%
01:19:44:WU00:FS01:Upload 15.32%
01:19:50:WU00:FS01:Upload 23.02%
01:19:56:WU00:FS01:Upload 31.45%
01:20:02:WU00:FS01:Upload 38.81%
01:20:08:WU00:FS01:Upload 46.51%
01:20:14:WU00:FS01:Upload 54.60%
01:20:20:WU00:FS01:Upload 62.43%
01:20:26:WU00:FS01:Upload 70.19%
01:20:32:WU00:FS01:Upload 78.22%
01:20:38:WU00:FS01:Upload 86.04%
01:20:44:WU00:FS01:Upload 93.87%
01:20:50:WU00:FS01:Upload complete
01:20:50:WU00:FS01:Server responded WORK_ACK (400)
01:20:50:WU00:FS01:Final credit estimate, 205470.00 points
01:20:50:WU00:FS01:Cleaning up
*********************** Log Started 2019-10-05T01:27:56Z ***********************
01:28:00:WU00:FS01:Connecting to 65.254.110.245:8080
01:28:01:WU00:FS01:Assigned to work server 155.247.166.220
01:28:01:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
01:28:01:WU00:FS01:Connecting to 155.247.166.220:8080
01:28:02:WU00:FS01:Downloading 6.07MiB
*********************** Log Started 2019-10-05T01:32:22Z ***********************
01:32:25:WU00:FS01:Connecting to 65.254.110.245:8080
01:32:27:WU00:FS01:Assigned to work server 155.247.166.220
01:32:27:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
01:32:27:WU00:FS01:Connecting to 155.247.166.220:8080
01:32:27:WU00:FS01:Downloading 6.06MiB
*********************** Log Started 2019-10-05T01:32:22Z ***********************
01:32:25:WU00:FS01:Connecting to 65.254.110.245:8080
01:32:27:WU00:FS01:Assigned to work server 155.247.166.220
01:32:27:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
01:32:27:WU00:FS01:Connecting to 155.247.166.220:8080
01:32:27:WU00:FS01:Downloading 6.06MiB
*********************** Log Started 2019-10-05T01:52:47Z ***********************
01:52:49:WU00:FS01:Connecting to 65.254.110.245:8080
01:52:50:WU00:FS01:Assigned to work server 155.247.166.220
01:52:50:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
01:52:50:WU00:FS01:Connecting to 155.247.166.220:8080
01:52:51:WU00:FS01:Downloading 3.17MiB
01:53:06:WU00:FS01:Download 1.97%
01:53:23:WU00:FS01:Download 5.92%
01:53:29:WU00:FS01:Download 9.86%
01:53:36:WU00:FS01:Download 11.83%
01:53:51:WU00:FS01:Download 17.75%
01:54:05:WU00:FS01:Download 21.69%
*********************** Log Started 2019-10-05T01:52:47Z ***********************
01:52:49:WU00:FS01:Connecting to 65.254.110.245:8080
01:52:50:WU00:FS01:Assigned to work server 155.247.166.220
01:52:50:WU00:FS01:Requesting new work unit for slot 01: READY gpu:1:TU104 [GeForce RTX 2070 Super] 8218 from 155.247.166.220
01:52:50:WU00:FS01:Connecting to 155.247.166.220:8080
01:52:51:WU00:FS01:Downloading 3.17MiB
01:53:06:WU00:FS01:Download 1.97%
01:53:23:WU00:FS01:Download 5.92%
01:53:29:WU00:FS01:Download 9.86%
01:53:36:WU00:FS01:Download 11.83%
01:53:51:WU00:FS01:Download 17.75%
01:54:05:WU00:FS01:Download 21.69%
01:54:31:WU00:FS01:Download 23.66%
01:56:29:WU00:FS01:Download 25.64%
and it appears to have stalled again so another reboot. At least these things have SSDs so there's less mechanical devices being stressed with the constant power cycling.

And no, it's not just me. Many of the members of my Team are reporting the same issues.
Image
Top
MeeLee
Posts: 1339
Joined: Tue Feb 19, 2019 10:16 pm

Re: Multiple WU's Fail to Upload 155.247.166.*

Post by MeeLee »

Is this still going on?
Sucks big time, as I've spent considerable amount of time setting and tuning my servers, only for them to shut down the very first day of my vacation!
It'll be 2 weeks of lost time, before I can reset them. All of them..
Top
Post Reply

Return to “Issues with a specific server”