I believe I'm having the same issue as well. I have two completed WU's that are unable to upload. The upload is going to 155.247.164.214 for each of these. I've also tried restarting but that hasn't helped:
17:20:36:WU00:FS00:0xa7:*********************** Log Started 2020-03-16T17:20:36Z ***********************
17:20:36:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
17:20:36:WU00:FS00:0xa7: Type: 0xa7
17:20:36:14:127.0.0.1:New Web connection
17:20:36:WU00:FS00:0xa7: Core: Gromacs
17:20:37:WU00:FS00:0xa7: Args: -dir 00 -suffix 01 -version 705 -lifeline 16432 -checkpoint 15 -np
17:20:37:WU00:FS00:0xa7: 7
17:20:38:WU00:FS00:0xa7:************************************ CBang *************************************
17:20:38:WU00:FS00:0xa7: Date: Oct 26 2019
17:20:38:WU00:FS00:0xa7: Time: 01:38:25
17:20:38:WU00:FS00:0xa7: Revision: c46a1a011a24143739ac7218c5a435f66777f62f
17:20:38:WU00:FS00:0xa7: Branch: master
17:20:39:WU00:FS00:0xa7: Compiler: Visual C++ 2008
17:20:39:WU00:FS00:0xa7: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:20:39:WU00:FS00:0xa7: Platform: win32 10
17:20:39:WU00:FS00:0xa7: Bits: 64
17:20:39:WU00:FS00:0xa7: Mode: Release
17:20:39:WU00:FS00:0xa7:************************************ System ************************************
17:20:39:WU00:FS00:0xa7: CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
17:20:39:WU00:FS00:0xa7: CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
17:20:39:WU00:FS00:0xa7: CPUs: 8
17:20:39:WU00:FS00:0xa7: Memory: 63.94GiB
17:20:39:WU00:FS00:0xa7:Free Memory: 57.53GiB
17:20:39:WU00:FS00:0xa7: Threads: WINDOWS_THREADS
17:20:39:WU00:FS00:0xa7: OS Version: 6.2
17:20:39:WU00:FS00:0xa7:Has Battery: false
17:20:39:WU00:FS00:0xa7: On Battery: false
17:20:39:WU00:FS00:0xa7: UTC Offset: -7
17:20:39:WU00:FS00:0xa7: PID: 10620
17:20:39:WU00:FS00:0xa7: CWD: C:\Users\someo\AppData\Roaming\FAHClient\work
17:20:39:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
17:20:39:WU00:FS00:0xa7: Version: 0.0.18
17:20:39:WU00:FS00:0xa7: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:20:39:WU00:FS00:0xa7: Copyright: 2019 foldingathome.org
17:20:39:WU00:FS00:0xa7: Homepage: https://foldingathome.org/
17:20:39:WU00:FS00:0xa7: Date: Oct 26 2019
17:20:39:WU00:FS00:0xa7: Time: 01:52:30
17:20:39:WU00:FS00:0xa7: Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
17:20:39:WU00:FS00:0xa7: Branch: master
17:20:39:WU00:FS00:0xa7: Compiler: Visual C++ 2008
17:20:39:WU00:FS00:0xa7: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:20:39:WU00:FS00:0xa7: Platform: win32 10
17:20:39:WU00:FS00:0xa7: Bits: 64
17:20:39:WU00:FS00:0xa7: Mode: Release
17:20:39:WU00:FS00:0xa7:************************************ Build *************************************
17:20:39:WU00:FS00:0xa7: SIMD: avx_256
17:20:39:WU00:FS00:0xa7:********************************************************************************
17:20:39:WU00:FS00:0xa7:Project: 14328 (Run 7, Clone 5406, Gen 12)
17:20:39:WU00:FS00:0xa7:Unit: 0x0000000c9bf7a4d65e6d0cd840ca2d2e
17:20:39:WU00:FS00:0xa7:Reading tar file core.xml
17:20:39:WU00:FS00:0xa7:Reading tar file frame12.tpr
17:20:40:WU02:FS01:Upload 0.23%
17:20:40:ERROR:WU02:FS01:Exception: Transfer failed
17:20:43:WU00:FS00:0xa7:Digital signatures verified
17:20:43:WU00:FS00:0xa7:Reducing thread count from 7 to 6 to avoid domain decomposition by a prime number > 3
17:20:43:WU00:FS00:0xa7:Calling: mdrun -s frame12.tpr -o frame12.trr -cpt 15 -nt 6
17:20:45:WU00:FS00:0xa7:Steps: first=3000000 total=250000
17:20:49:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
17:21:14:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11758 run:0 clone:1593 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7713e9c30c8f
17:21:14:WU01:FS01:Uploading 55.24MiB to 155.247.164.213
17:21:14:WU01:FS01:Connecting to 155.247.164.213:8080
17:21:14:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
17:21:14:WU01:FS01:Trying to send results to collection server
17:21:14:WU01:FS01:Uploading 55.24MiB to 155.247.164.214
17:21:14:WU01:FS01:Connecting to 155.247.164.214:8080
17:21:15:ERROR:WU01:FS01:Exception: Transfer failed
17:21:17:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:2028 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7714600f16eb
17:21:17:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
17:21:17:WU02:FS01:Connecting to 155.247.164.213:8080
17:21:17:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
17:21:17:WU03:FS01:Connecting to 65.254.110.245:8080
17:21:17:WU02:FS01:Trying to send results to collection server
17:21:17:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
17:21:17:WU02:FS01:Connecting to 155.247.164.214:8080
17:21:17:WU03:FS01:Assigned to work server 140.163.4.241
17:21:17:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] 8873 from 140.163.4.241
17:21:17:WU03:FS01:Connecting to 140.163.4.241:8080
17:21:17:ERROR:WU02:FS01:Exception: Transfer failed
17:21:38:WARNING:WU03:FS01:WorkServer connection failed on port 8080 trying 80
17:21:38:WU03:FS01:Connecting to 140.163.4.241:80
17:21:49:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
17:22:00:ERROR:WU03:FS01:Exception: Failed to connect to 140.163.4.241:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
17:22:38:WU00:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
17:22:51:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11758 run:0 clone:1593 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7713e9c30c8f
17:22:51:WU01:FS01:Uploading 55.24MiB to 155.247.164.213
17:22:51:WU01:FS01:Connecting to 155.247.164.213:8080
17:22:52:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
17:22:52:WU01:FS01:Trying to send results to collection server
17:22:52:WU01:FS01:Uploading 55.24MiB to 155.247.164.214
17:22:52:WU01:FS01:Connecting to 155.247.164.214:8080
17:22:52:ERROR:WU01:FS01:Exception: Transfer failed
17:22:54:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:2028 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7714600f16eb
17:22:54:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
17:22:54:WU02:FS01:Connecting to 155.247.164.213:8080
17:22:54:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
17:22:54:WU02:FS01:Trying to send results to collection server
17:22:54:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
17:22:54:WU02:FS01:Connecting to 155.247.164.214:8080
17:22:54:WU03:FS01:Connecting to 65.254.110.245:8080
17:22:54:ERROR:WU02:FS01:Exception: Transfer failed
17:22:54:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
17:22:54:WU03:FS01:Connecting to 18.218.241.186:80
17:22:55:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
17:22:55:ERROR:WU03:FS01:Exception: Could not get an assignment
17:23:28:WU00:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
17:24:16:WU00:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
17:25:09:WU00:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
17:25:28:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11758 run:0 clone:1593 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7713e9c30c8f
17:25:28:WU01:FS01:Uploading 55.24MiB to 155.247.164.213
17:25:28:WU01:FS01:Connecting to 155.247.164.213:8080
17:25:29:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
17:25:29:WU01:FS01:Trying to send results to collection server
17:25:29:WU01:FS01:Uploading 55.24MiB to 155.247.164.214
17:25:29:WU01:FS01:Connecting to 155.247.164.214:8080
17:25:29:ERROR:WU01:FS01:Exception: Transfer failed
17:25:31:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:2028 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7714600f16eb
17:25:31:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
17:25:31:WU02:FS01:Connecting to 155.247.164.213:8080
17:25:31:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
17:25:31:WU02:FS01:Trying to send results to collection server
17:25:31:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
17:25:31:WU02:FS01:Connecting to 155.247.164.214:8080
17:25:31:WU03:FS01:Connecting to 65.254.110.245:8080
17:25:31:ERROR:WU02:FS01:Exception: Transfer failed
17:25:32:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
17:25:32:WU03:FS01:Connecting to 18.218.241.186:80
17:25:32:WU03:FS01:Assigned to work server 140.163.4.231
17:25:32:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] 8873 from 140.163.4.231
17:25:32:WU03:FS01:Connecting to 140.163.4.231:8080
17:25:53:WARNING:WU03:FS01:WorkServer connection failed on port 8080 trying 80
17:25:53:WU03:FS01:Connecting to 140.163.4.231:80
17:25:55:WU00:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
17:26:14:ERROR:WU03:FS01:Exception: Failed to connect to 140.163.4.231:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
17:26:42:WU00:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
17:27:29:WU00:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
17:28:15:WU00:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
17:28:28:FS01:Paused
17:28:31:FS01:Unpaused
17:28:31:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11758 run:0 clone:1593 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7713e9c30c8f
17:28:31:WU01:FS01:Uploading 55.24MiB to 155.247.164.213
17:28:31:WU01:FS01:Connecting to 155.247.164.213:8080
17:28:31:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:2028 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7714600f16eb
17:28:31:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
17:28:31:WU02:FS01:Connecting to 155.247.164.213:8080
17:28:31:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
17:28:31:WU01:FS01:Trying to send results to collection server
17:28:31:WU01:FS01:Uploading 55.24MiB to 155.247.164.214
17:28:31:WU01:FS01:Connecting to 155.247.164.214:8080
17:28:31:WU03:FS01:Connecting to 65.254.110.245:8080
17:28:31:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
17:28:31:WU02:FS01:Trying to send results to collection server
17:28:31:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
17:28:31:WU02:FS01:Connecting to 155.247.164.214:8080
17:28:32:ERROR:WU01:FS01:Exception: Transfer failed
17:28:32:WU03:FS01:Assigned to work server 155.247.164.213
17:28:32:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] 8873 from 155.247.164.213
17:28:32:WU03:FS01:Connecting to 155.247.164.213:8080
17:28:32:ERROR:WU02:FS01:Exception: Transfer failed
17:28:32:ERROR:WU03:FS01:Exception: 10001: Server responded: HTTP_SERVICE_UNAVAILABLE
17:29:08:WU00:FS00:0xa7:Completed 25000 out of 250000 steps (10%)
17:30:00:WU00:FS00:0xa7:Completed 27500 out of 250000 steps (11%)
17:30:08:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11758 run:0 clone:1593 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7713e9c30c8f
17:30:08:WU01:FS01:Uploading 55.24MiB to 155.247.164.213
17:30:08:WU01:FS01:Connecting to 155.247.164.213:8080
17:30:08:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:2028 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7714600f16eb
17:30:08:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
17:30:08:WU02:FS01:Connecting to 155.247.164.213:8080
17:30:08:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
17:30:08:WU01:FS01:Trying to send results to collection server
17:30:08:WU01:FS01:Uploading 55.24MiB to 155.247.164.214
17:30:08:WU01:FS01:Connecting to 155.247.164.214:8080
17:30:09:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
17:30:09:WU02:FS01:Trying to send results to collection server
17:30:09:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
17:30:09:WU02:FS01:Connecting to 155.247.164.214:8080
17:30:09:ERROR:WU01:FS01:Exception: Transfer failed
17:30:09:WU03:FS01:Connecting to 65.254.110.245:8080
17:30:09:ERROR:WU02:FS01:Exception: Transfer failed
17:30:09:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
17:30:09:WU03:FS01:Connecting to 18.218.241.186:80
17:30:09:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
17:30:09:ERROR:WU03:FS01:Exception: Could not get an assignment
17:30:50:WU00:FS00:0xa7:Completed 30000 out of 250000 steps (12%)
17:31:44:WU00:FS00:0xa7:Completed 32500 out of 250000 steps (13%)
17:32:29:WU00:FS00:0xa7:Completed 35000 out of 250000 steps (14%)
17:32:45:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11758 run:0 clone:1593 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7713e9c30c8f
17:32:45:WU01:FS01:Uploading 55.24MiB to 155.247.164.213
17:32:45:WU01:FS01:Connecting to 155.247.164.213:8080
17:32:45:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
17:32:45:WU01:FS01:Trying to send results to collection server
17:32:46:WU01:FS01:Uploading 55.24MiB to 155.247.164.214
17:32:46:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:2028 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7714600f16eb
17:32:46:WU01:FS01:Connecting to 155.247.164.214:8080
17:32:46:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
17:32:46:WU02:FS01:Connecting to 155.247.164.213:8080
17:32:46:WU03:FS01:Connecting to 65.254.110.245:8080
17:32:46:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
17:32:46:WU02:FS01:Trying to send results to collection server
17:32:46:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
17:32:46:WU02:FS01:Connecting to 155.247.164.214:8080
17:32:46:WU03:FS01:Assigned to work server 140.163.4.241
17:32:46:WU03:FS01:Requesting new work unit for slot 01: READY gpu:0:GP104 [GeForce GTX 1080] 8873 from 140.163.4.241
17:32:46:WU03:FS01:Connecting to 140.163.4.241:8080
17:32:46:ERROR:WU01:FS01:Exception: Transfer failed
17:32:46:ERROR:WU02:FS01:Exception: Transfer failed
17:33:07:WARNING:WU03:FS01:WorkServer connection failed on port 8080 trying 80
17:33:07:WU03:FS01:Connecting to 140.163.4.241:80
17:33:17:WU00:FS00:0xa7:Completed 37500 out of 250000 steps (15%)
17:33:28:ERROR:WU03:FS01:Exception: Failed to connect to 140.163.4.241:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
17:34:12:WU00:FS00:0xa7:Completed 40000 out of 250000 steps (16%)
17:35:12:WU00:FS00:0xa7:Completed 42500 out of 250000 steps (17%)
17:36:04:WU00:FS00:0xa7:Completed 45000 out of 250000 steps (18%)
17:36:52:WU00:FS00:0xa7:Completed 47500 out of 250000 steps (19%)
17:37:00:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11758 run:0 clone:1593 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7713e9c30c8f
17:37:00:WU01:FS01:Uploading 55.24MiB to 155.247.164.213
17:37:00:WU01:FS01:Connecting to 155.247.164.213:8080
17:37:00:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:2028 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7714600f16eb
17:37:00:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
17:37:00:WU02:FS01:Connecting to 155.247.164.213:8080
17:37:00:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
17:37:00:WU01:FS01:Trying to send results to collection server
17:37:00:WU01:FS01:Uploading 55.24MiB to 155.247.164.214
17:37:00:WU01:FS01:Connecting to 155.247.164.214:8080
17:37:00:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
17:37:00:WU02:FS01:Trying to send results to collection server
17:37:00:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
17:37:00:WU02:FS01:Connecting to 155.247.164.214:8080
17:37:00:ERROR:WU01:FS01:Exception: Transfer failed
17:37:00:WU03:FS01:Connecting to 65.254.110.245:8080
17:37:00:ERROR:WU02:FS01:Exception: Transfer failed
WU's are coming in fine, I just can't seem to do anything with them once they're complete
suchamoneypit wrote:My clients can't connect to .214, so it is down. Are we able to switch servers so we can fold ? Or is waiting the only option.
I would be interested in this as well. Instead of trying again and again to connect to a certain server, every try coming with longer intervals, is it possible to leave that server be for a bit (let's say after 3 or 4 failed attempts) and find another one?
05:05:41:WU03:FS02:Sending unit results: id:03 state:SEND error:NO_ERROR project:11758 run:0 clone:1594 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d7713abba7d70
05:05:41:WU03:FS02:Uploading 55.24MiB to 155.247.164.213
05:05:41:WU03:FS02:Connecting to 155.247.164.213:8080
05:05:41:WARNING:WU03:FS02:Exception: Failed to send results to work server: Transfer failed
05:05:41:WU03:FS02:Trying to send results to collection server
05:05:41:WU03:FS02:Uploading 55.24MiB to 155.247.164.214
05:05:41:WU03:FS02:Connecting to 155.247.164.214:8080
05:05:43:ERROR:WU03:FS02:Exception: Transfer failed
07:51:30:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11758 run:0 clone:1738 gen:0 core:0x22 unit:0x000000019bf7a4d55e6d771350a88925
07:51:30:WU01:FS01:Uploading 55.25MiB to 155.247.164.213
07:51:30:WU01:FS01:Connecting to 155.247.164.213:8080
07:51:31:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
07:51:31:WU01:FS01:Trying to send results to collection server
07:51:31:WU01:FS01:Uploading 55.25MiB to 155.247.164.214
07:51:31:WU01:FS01:Connecting to 155.247.164.214:8080
07:51:33:ERROR:WU01:FS01:Exception: Transfer failed
It's been stuck for hours now.
Ryzen 9 3900x - CM MA410M TUF - Asus Rog Crosshair VIII Hero (Wi-Fi) - Corsair Vengeance RGB Pro 2x16GB 3200MHz - 2x GTX 980 - Asus ROG Thor 850W - Cougar Panzer Evo - Corsair MP600 1TB / Samsung 860 EVO 1TB / Seagate Barracuda 4TB - Windows 10 Pro
Yep, I have now not been able to upload this specific WU for over 48hrs. And to be clear, the machines have completed multiple other WUs meanwhile, including receiving and uploading to this specific server (213), for e.g:
10:14:16:WU02:FS01:Connecting to 65.254.110.245:8080
10:14:16:WU02:FS01:Assigned to work server 155.247.164.213
10:14:16:WU02:FS01:Requesting new work unit for slot 01: READY gpu:0:TU106M [GeForce RTX 2060 Mobile] from 155.247.164.213
10:14:16:WU02:FS01:Connecting to 155.247.164.213:8080
10:14:27:WU02:FS01:Downloading 11.98MiB
10:14:33:WU02:FS01:Download 84.00%
10:14:34:WU02:FS01:Download complete
10:14:34:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:11753 run:0 clone:3574 gen:1 core:0x22 unit:0x000000029bf7a4d55e6d76caa76041b9
10:14:34:WU02:FS01:Starting
10:14:34:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\alxbelu\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 705 -lifeline 11980 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
10:14:34:WU02:FS01:Started FahCore on PID 18220
10:14:34:WU02:FS01:Core PID:9752
10:14:34:WU02:FS01:FahCore 0x22 started
10:14:35:WU02:FS01:0x22:*********************** Log Started 2020-03-16T10:14:34Z ***********************
10:14:35:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
10:14:35:WU02:FS01:0x22: Type: 0x22
10:14:35:WU02:FS01:0x22: Core: Core22
10:14:35:WU02:FS01:0x22: Website: https://foldingathome.org/
10:14:35:WU02:FS01:0x22: Copyright: (c) 2009-2018 foldingathome.org
10:14:35:WU02:FS01:0x22: Author: John Chodera <john.chodera@choderalab.org> and Rafal Wiewiora
10:14:35:WU02:FS01:0x22: <rafal.wiewiora@choderalab.org>
10:14:35:WU02:FS01:0x22: Args: -dir 02 -suffix 01 -version 705 -lifeline 18220 -checkpoint 15
10:14:35:WU02:FS01:0x22: -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
10:14:35:WU02:FS01:0x22: 0 -gpu 0
10:14:35:WU02:FS01:0x22: Config: <none>
10:14:35:WU02:FS01:0x22:************************************ Build *************************************
10:14:35:WU02:FS01:0x22: Version: 0.0.2
10:14:35:WU02:FS01:0x22: Date: Dec 6 2019
10:14:35:WU02:FS01:0x22: Time: 21:30:31
10:14:35:WU02:FS01:0x22: Repository: Git
10:14:35:WU02:FS01:0x22: Revision: abeb39247cc72df5af0f63723edafadb23d5dfbe
10:14:35:WU02:FS01:0x22: Branch: HEAD
10:14:35:WU02:FS01:0x22: Compiler: Visual C++ 2008
10:14:35:WU02:FS01:0x22: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
10:14:35:WU02:FS01:0x22: Platform: win32 10
10:14:35:WU02:FS01:0x22: Bits: 64
10:14:35:WU02:FS01:0x22: Mode: Release
10:14:35:WU02:FS01:0x22:************************************ System ************************************
10:14:35:WU02:FS01:0x22: CPU: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
10:14:35:WU02:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 158 Stepping 10
10:14:35:WU02:FS01:0x22: CPUs: 12
10:14:35:WU02:FS01:0x22: Memory: 15.85GiB
10:14:35:WU02:FS01:0x22:Free Memory: 8.79GiB
10:14:35:WU02:FS01:0x22: Threads: WINDOWS_THREADS
10:14:35:WU02:FS01:0x22: OS Version: 6.2
10:14:35:WU02:FS01:0x22:Has Battery: true
10:14:35:WU02:FS01:0x22: On Battery: false
10:14:35:WU02:FS01:0x22: UTC Offset: 1
10:14:35:WU02:FS01:0x22: PID: 9752
10:14:35:WU02:FS01:0x22: CWD: C:\Users\alxbelu\AppData\Roaming\FAHClient\work
10:14:35:WU02:FS01:0x22: OS: Windows 10 Home
10:14:35:WU02:FS01:0x22: OS Arch: AMD64
10:14:35:WU02:FS01:0x22:********************************************************************************
10:14:35:WU02:FS01:0x22:Project: 11753 (Run 0, Clone 3574, Gen 1)
10:14:35:WU02:FS01:0x22:Unit: 0x000000029bf7a4d55e6d76caa76041b9
10:14:35:WU02:FS01:0x22:Reading tar file core.xml
10:14:35:WU02:FS01:0x22:Reading tar file integrator.xml
10:14:35:WU02:FS01:0x22:Reading tar file state.xml
10:14:38:WU02:FS01:0x22:Reading tar file system.xml
10:14:39:WU02:FS01:0x22:Digital signatures verified
10:14:39:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
10:14:39:WU02:FS01:0x22:Version 0.0.2
10:15:08:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:1756 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771303ec7ef7
10:15:08:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
10:15:08:WU00:FS01:Connecting to 155.247.164.213:8080
10:15:08:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
10:15:08:WU00:FS01:Trying to send results to collection server
10:15:08:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
10:15:08:WU00:FS01:Connecting to 155.247.164.214:8080
10:15:09:ERROR:WU00:FS01:Exception: Transfer failed
10:15:09:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
10:15:09:WU02:FS01:0x22:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
10:17:00:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
10:17:45:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:1756 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771303ec7ef7
10:17:45:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
10:17:45:WU00:FS01:Connecting to 155.247.164.213:8080
10:17:47:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
10:17:47:WU00:FS01:Trying to send results to collection server
10:17:47:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
10:17:47:WU00:FS01:Connecting to 155.247.164.214:8080
10:17:47:ERROR:WU00:FS01:Exception: Transfer failed
10:18:51:WU02:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
10:20:41:WU02:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
10:21:59:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:1756 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771303ec7ef7
10:21:59:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
10:21:59:WU00:FS01:Connecting to 155.247.164.213:8080
10:22:01:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
10:22:01:WU00:FS01:Trying to send results to collection server
10:22:01:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
10:22:01:WU00:FS01:Connecting to 155.247.164.214:8080
10:22:01:ERROR:WU00:FS01:Exception: Transfer failed
10:22:32:WU02:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
10:24:23:WU02:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
10:26:25:WU02:FS01:0x22:Completed 60000 out of 1000000 steps (6%)
10:28:16:WU02:FS01:0x22:Completed 70000 out of 1000000 steps (7%)
10:28:51:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:1756 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771303ec7ef7
10:28:51:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
10:28:51:WU00:FS01:Connecting to 155.247.164.213:8080
10:28:51:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
10:28:51:WU00:FS01:Trying to send results to collection server
10:28:51:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
10:28:51:WU00:FS01:Connecting to 155.247.164.214:8080
10:28:52:ERROR:WU00:FS01:Exception: Transfer failed
10:30:06:WU02:FS01:0x22:Completed 80000 out of 1000000 steps (8%)
10:31:57:WU02:FS01:0x22:Completed 90000 out of 1000000 steps (9%)
10:33:47:WU02:FS01:0x22:Completed 100000 out of 1000000 steps (10%)
10:35:49:WU02:FS01:0x22:Completed 110000 out of 1000000 steps (11%)
10:37:40:WU02:FS01:0x22:Completed 120000 out of 1000000 steps (12%)
10:39:30:WU02:FS01:0x22:Completed 130000 out of 1000000 steps (13%)
10:39:56:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:1756 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771303ec7ef7
10:39:56:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
10:39:56:WU00:FS01:Connecting to 155.247.164.213:8080
10:39:57:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
10:39:57:WU00:FS01:Trying to send results to collection server
10:39:57:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
10:39:57:WU00:FS01:Connecting to 155.247.164.214:8080
10:39:57:ERROR:WU00:FS01:Exception: Transfer failed
10:41:21:WU02:FS01:0x22:Completed 140000 out of 1000000 steps (14%)
10:43:12:WU02:FS01:0x22:Completed 150000 out of 1000000 steps (15%)
10:45:13:WU02:FS01:0x22:Completed 160000 out of 1000000 steps (16%)
10:47:03:WU02:FS01:0x22:Completed 170000 out of 1000000 steps (17%)
10:48:54:WU02:FS01:0x22:Completed 180000 out of 1000000 steps (18%)
10:50:45:WU02:FS01:0x22:Completed 190000 out of 1000000 steps (19%)
10:52:35:WU02:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
10:54:37:WU02:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
10:56:27:WU02:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
10:57:53:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:1756 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771303ec7ef7
10:57:53:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
10:57:53:WU00:FS01:Connecting to 155.247.164.213:8080
10:57:54:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
10:57:54:WU00:FS01:Trying to send results to collection server
10:57:54:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
10:57:54:WU00:FS01:Connecting to 155.247.164.214:8080
10:57:54:ERROR:WU00:FS01:Exception: Transfer failed
10:58:18:WU02:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
11:00:08:WU02:FS01:0x22:Completed 240000 out of 1000000 steps (24%)
11:01:59:WU02:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
11:04:01:WU02:FS01:0x22:Completed 260000 out of 1000000 steps (26%)
11:05:51:WU02:FS01:0x22:Completed 270000 out of 1000000 steps (27%)
11:07:42:WU02:FS01:0x22:Completed 280000 out of 1000000 steps (28%)
11:09:33:WU02:FS01:0x22:Completed 290000 out of 1000000 steps (29%)
11:11:24:WU02:FS01:0x22:Completed 300000 out of 1000000 steps (30%)
11:13:25:WU02:FS01:0x22:Completed 310000 out of 1000000 steps (31%)
11:15:16:WU02:FS01:0x22:Completed 320000 out of 1000000 steps (32%)
11:17:07:WU02:FS01:0x22:Completed 330000 out of 1000000 steps (33%)
11:18:57:WU02:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
11:20:48:WU02:FS01:0x22:Completed 350000 out of 1000000 steps (35%)
11:22:50:WU02:FS01:0x22:Completed 360000 out of 1000000 steps (36%)
11:24:41:WU02:FS01:0x22:Completed 370000 out of 1000000 steps (37%)
11:26:31:WU02:FS01:0x22:Completed 380000 out of 1000000 steps (38%)
11:26:55:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:1756 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771303ec7ef7
11:26:55:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
11:26:55:WU00:FS01:Connecting to 155.247.164.213:8080
11:26:56:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:26:56:WU00:FS01:Trying to send results to collection server
11:26:56:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
11:26:56:WU00:FS01:Connecting to 155.247.164.214:8080
11:26:57:ERROR:WU00:FS01:Exception: Transfer failed
11:28:19:WU02:FS01:0x22:Completed 390000 out of 1000000 steps (39%)
11:30:08:WU02:FS01:0x22:Completed 400000 out of 1000000 steps (40%)
11:32:00:WU02:FS01:0x22:Completed 410000 out of 1000000 steps (41%)
11:33:48:WU02:FS01:0x22:Completed 420000 out of 1000000 steps (42%)
11:35:37:WU02:FS01:0x22:Completed 430000 out of 1000000 steps (43%)
11:37:25:WU02:FS01:0x22:Completed 440000 out of 1000000 steps (44%)
11:39:13:WU02:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
11:41:06:WU02:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
11:42:54:WU02:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
11:44:42:WU02:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
11:46:31:WU02:FS01:0x22:Completed 490000 out of 1000000 steps (49%)
11:48:21:WU02:FS01:0x22:Completed 500000 out of 1000000 steps (50%)
11:50:17:WU02:FS01:0x22:Completed 510000 out of 1000000 steps (51%)
11:52:10:WU02:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
11:54:02:WU02:FS01:0x22:Completed 530000 out of 1000000 steps (53%)
11:55:54:WU02:FS01:0x22:Completed 540000 out of 1000000 steps (54%)
11:57:46:WU02:FS01:0x22:Completed 550000 out of 1000000 steps (55%)
11:59:43:WU02:FS01:0x22:Completed 560000 out of 1000000 steps (56%)
12:01:35:WU02:FS01:0x22:Completed 570000 out of 1000000 steps (57%)
12:03:27:WU02:FS01:0x22:Completed 580000 out of 1000000 steps (58%)
12:05:20:WU02:FS01:0x22:Completed 590000 out of 1000000 steps (59%)
12:07:12:WU02:FS01:0x22:Completed 600000 out of 1000000 steps (60%)
12:09:08:WU02:FS01:0x22:Completed 610000 out of 1000000 steps (61%)
12:11:01:WU02:FS01:0x22:Completed 620000 out of 1000000 steps (62%)
12:12:53:WU02:FS01:0x22:Completed 630000 out of 1000000 steps (63%)
12:13:54:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:1756 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771303ec7ef7
12:13:54:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
12:13:54:WU00:FS01:Connecting to 155.247.164.213:8080
12:13:55:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
12:13:55:WU00:FS01:Trying to send results to collection server
12:13:55:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
12:13:55:WU00:FS01:Connecting to 155.247.164.214:8080
12:13:55:ERROR:WU00:FS01:Exception: Transfer failed
12:14:46:WU02:FS01:0x22:Completed 640000 out of 1000000 steps (64%)
12:16:40:WU02:FS01:0x22:Completed 650000 out of 1000000 steps (65%)
12:18:43:WU02:FS01:0x22:Completed 660000 out of 1000000 steps (66%)
12:20:37:WU02:FS01:0x22:Completed 670000 out of 1000000 steps (67%)
12:22:31:WU02:FS01:0x22:Completed 680000 out of 1000000 steps (68%)
12:24:25:WU02:FS01:0x22:Completed 690000 out of 1000000 steps (69%)
12:26:19:WU02:FS01:0x22:Completed 700000 out of 1000000 steps (70%)
12:28:22:WU02:FS01:0x22:Completed 710000 out of 1000000 steps (71%)
12:30:16:WU02:FS01:0x22:Completed 720000 out of 1000000 steps (72%)
12:32:10:WU02:FS01:0x22:Completed 730000 out of 1000000 steps (73%)
12:34:04:WU02:FS01:0x22:Completed 740000 out of 1000000 steps (74%)
12:35:58:WU02:FS01:0x22:Completed 750000 out of 1000000 steps (75%)
12:38:01:WU02:FS01:0x22:Completed 760000 out of 1000000 steps (76%)
12:39:55:WU02:FS01:0x22:Completed 770000 out of 1000000 steps (77%)
12:41:49:WU02:FS01:0x22:Completed 780000 out of 1000000 steps (78%)
12:43:43:WU02:FS01:0x22:Completed 790000 out of 1000000 steps (79%)
12:45:37:WU02:FS01:0x22:Completed 800000 out of 1000000 steps (80%)
12:47:41:WU02:FS01:0x22:Completed 810000 out of 1000000 steps (81%)
12:49:35:WU02:FS01:0x22:Completed 820000 out of 1000000 steps (82%)
12:51:29:WU02:FS01:0x22:Completed 830000 out of 1000000 steps (83%)
12:53:22:WU02:FS01:0x22:Completed 840000 out of 1000000 steps (84%)
12:55:16:WU02:FS01:0x22:Completed 850000 out of 1000000 steps (85%)
12:57:20:WU02:FS01:0x22:Completed 860000 out of 1000000 steps (86%)
12:59:14:WU02:FS01:0x22:Completed 870000 out of 1000000 steps (87%)
13:01:08:WU02:FS01:0x22:Completed 880000 out of 1000000 steps (88%)
13:03:02:WU02:FS01:0x22:Completed 890000 out of 1000000 steps (89%)
13:04:55:WU02:FS01:0x22:Completed 900000 out of 1000000 steps (90%)
13:06:59:WU02:FS01:0x22:Completed 910000 out of 1000000 steps (91%)
13:08:53:WU02:FS01:0x22:Completed 920000 out of 1000000 steps (92%)
13:10:46:WU02:FS01:0x22:Completed 930000 out of 1000000 steps (93%)
13:12:40:WU02:FS01:0x22:Completed 940000 out of 1000000 steps (94%)
13:14:34:WU02:FS01:0x22:Completed 950000 out of 1000000 steps (95%)
13:16:37:WU02:FS01:0x22:Completed 960000 out of 1000000 steps (96%)
13:18:31:WU02:FS01:0x22:Completed 970000 out of 1000000 steps (97%)
13:20:25:WU02:FS01:0x22:Completed 980000 out of 1000000 steps (98%)
13:22:19:WU02:FS01:0x22:Completed 990000 out of 1000000 steps (99%)
13:22:19:WU01:FS01:Connecting to 65.254.110.245:8080
13:22:19:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:22:19:WU01:FS01:Connecting to 18.218.241.186:80
13:22:20:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:22:20:ERROR:WU01:FS01:Exception: Could not get an assignment
13:22:20:WU01:FS01:Connecting to 65.254.110.245:8080
13:22:21:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:22:21:WU01:FS01:Connecting to 18.218.241.186:80
13:22:21:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:22:21:ERROR:WU01:FS01:Exception: Could not get an assignment
13:23:20:WU01:FS01:Connecting to 65.254.110.245:8080
13:23:21:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:23:21:WU01:FS01:Connecting to 18.218.241.186:80
13:23:21:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:23:21:ERROR:WU01:FS01:Exception: Could not get an assignment
13:24:12:WU02:FS01:0x22:Completed 1000000 out of 1000000 steps (100%)
13:24:22:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
13:24:22:WU02:FS01:0x22:Saving result file checkpointState.xml
13:24:30:WU02:FS01:0x22:Saving result file checkpt.crc
13:24:30:WU02:FS01:0x22:Saving result file positions.xtc
13:24:34:WU02:FS01:0x22:Saving result file science.log
13:24:34:WU02:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
13:24:35:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
13:24:35:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11753 run:0 clone:3574 gen:1 core:0x22 unit:0x000000029bf7a4d55e6d76caa76041b9
13:24:35:WU02:FS01:Uploading 21.92MiB to 155.247.164.213
13:24:35:WU02:FS01:Connecting to 155.247.164.213:8080
13:24:41:WU02:FS01:Upload complete
13:24:41:WU02:FS01:Server responded WORK_ACK (400)
13:24:41:WU02:FS01:Final credit estimate, 105000.00 points
13:24:41:WU02:FS01:Cleaning up
Note how between receiving the WU (11753) from 213 and successfully uploading the result, a previous WU (11758) fails to upload to the very same server (213) 8 times.
It seems unlikely that this is a simple overload issue, rather I suspect the servers were rebooted/reconfigured after sending out a previous WU, and now no longer accepts any submissions from those.
edit: it seems many of us are also specifically having issues with WU 11758 and 213/214?
alxbelu wrote:Yep, I have now not been able to upload this specific WU for over 48hrs. And to be clear, the machines have completed multiple other WUs meanwhile, including receiving and uploading to this specific server (213), for e.g:
edit: it seems many of us are also specifically having issues with WU 11758 and 213/214?
I have the same issue with 11758 and .214.
Ryzen 9 3900x - CM MA410M TUF - Asus Rog Crosshair VIII Hero (Wi-Fi) - Corsair Vengeance RGB Pro 2x16GB 3200MHz - 2x GTX 980 - Asus ROG Thor 850W - Cougar Panzer Evo - Corsair MP600 1TB / Samsung 860 EVO 1TB / Seagate Barracuda 4TB - Windows 10 Pro
I have the same problem, two results not uploading to .213 / .214 for over 24 hours. And the credit going down as the delay increases is extremely annoying. Not sure why that "feature" was implemented, but I believe it should be disabled, or at the very least made less aggressive.
11:02:13:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
11:02:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
11:02:13:WU00:FS01:Connecting to 155.247.164.213:8080
11:02:14:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:02:14:WU00:FS01:Trying to send results to collection server
11:02:14:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
11:02:14:WU00:FS01:Connecting to 155.247.164.214:8080
11:02:15:ERROR:WU00:FS01:Exception: Transfer failed
12:37:28:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:3900 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771ad0cf1564
12:37:28:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
12:37:28:WU02:FS01:Connecting to 155.247.164.213:8080
12:37:29:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
12:37:29:WU02:FS01:Trying to send results to collection server
12:37:29:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
12:37:29:WU02:FS01:Connecting to 155.247.164.214:8080
12:37:30:ERROR:WU02:FS01:Exception: Transfer failed
Also, both yesterday and today I wasn't able to get new work units for extended periods of time. For example today I didn't receive any new units in the last 6 hours, after finishing my previous unit. My guess is that the two units that can't be sent are blocking receiving units as well.
I've restarted the software, which seemed to be stuck doing nothing, we'll see how it goes. But initial signs are not good:
13:38:19:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
13:38:19:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:3900 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771ad0cf1564
13:38:19:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
13:38:19:WU00:FS01:Connecting to 155.247.164.213:8080
13:38:19:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
13:38:19:WU02:FS01:Connecting to 155.247.164.213:8080
13:38:19:WU01:FS00:Connecting to 65.254.110.245:8080
13:38:19:WU03:FS01:Connecting to 65.254.110.245:8080
13:38:19:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
13:38:19:WU02:FS01:Trying to send results to collection server
13:38:19:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
13:38:19:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:38:19:WU02:FS01:Connecting to 155.247.164.214:8080
13:38:19:WU00:FS01:Trying to send results to collection server
13:38:20:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
13:38:20:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:38:20:WU00:FS01:Connecting to 155.247.164.214:8080
13:38:20:WU01:FS00:Connecting to 18.218.241.186:80
13:38:20:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:38:20:WU03:FS01:Connecting to 18.218.241.186:80
13:38:20:ERROR:WU02:FS01:Exception: Transfer failed
13:38:20:ERROR:WU00:FS01:Exception: Transfer failed
13:38:20:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:3900 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771ad0cf1564
13:38:20:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
13:38:20:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:38:20:WU02:FS01:Connecting to 155.247.164.213:8080
13:38:20:ERROR:WU01:FS00:Exception: Could not get an assignment
13:38:20:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:38:20:ERROR:WU03:FS01:Exception: Could not get an assignment
13:38:20:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
13:38:20:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
13:38:20:WU01:FS00:Connecting to 65.254.110.245:8080
13:38:20:WU00:FS01:Connecting to 155.247.164.213:8080
13:38:21:WU03:FS01:Connecting to 65.254.110.245:8080
13:38:21:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
13:38:21:WU02:FS01:Trying to send results to collection server
13:38:21:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
13:38:21:WU02:FS01:Connecting to 155.247.164.214:8080
13:38:21:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:38:21:WU00:FS01:Trying to send results to collection server
13:38:21:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
13:38:21:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:38:21:WU00:FS01:Connecting to 155.247.164.214:8080
13:38:21:WU01:FS00:Connecting to 18.218.241.186:80
13:38:21:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:38:21:WU03:FS01:Connecting to 18.218.241.186:80
13:38:21:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:38:21:ERROR:WU01:FS00:Exception: Could not get an assignment
13:38:21:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:38:21:ERROR:WU03:FS01:Exception: Could not get an assignment
13:38:22:ERROR:WU00:FS01:Exception: Transfer failed
13:38:22:13:127.0.0.1:New Web connection
13:38:23:ERROR:WU02:FS01:Exception: Transfer failed
13:39:20:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
13:39:20:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
13:39:20:WU00:FS01:Connecting to 155.247.164.213:8080
13:39:21:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:3900 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771ad0cf1564
13:39:21:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
13:39:21:WU02:FS01:Connecting to 155.247.164.213:8080
13:39:21:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:39:21:WU00:FS01:Trying to send results to collection server
13:39:21:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
13:39:21:WU00:FS01:Connecting to 155.247.164.214:8080
13:39:21:WU01:FS00:Connecting to 65.254.110.245:8080
13:39:21:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
13:39:21:WU02:FS01:Trying to send results to collection server
13:39:21:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
13:39:21:WU02:FS01:Connecting to 155.247.164.214:8080
13:39:21:ERROR:WU00:FS01:Exception: Transfer failed
13:39:21:WU03:FS01:Connecting to 65.254.110.245:8080
13:39:21:ERROR:WU02:FS01:Exception: Transfer failed
13:39:21:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:39:21:WU01:FS00:Connecting to 18.218.241.186:80
13:39:22:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:39:22:WU03:FS01:Connecting to 18.218.241.186:80
13:39:22:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:39:22:ERROR:WU01:FS00:Exception: Could not get an assignment
13:39:22:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:39:22:ERROR:WU03:FS01:Exception: Could not get an assignment
vnicolici wrote:
Also, both yesterday and today I wasn't able to get new work units for extended periods of time. For example today I didn't receive any new units in the last 6 hours, after finishing my previous unit. My guess is that the two units that can't be sent are blocking receiving units as well.
I've restarted the software, which seemed to be stuck doing nothing, we'll see how it goes. But initial signs are not good:
13:38:19:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
13:38:19:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:3900 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771ad0cf1564
13:38:19:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
13:38:19:WU00:FS01:Connecting to 155.247.164.213:8080
13:38:19:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
13:38:19:WU02:FS01:Connecting to 155.247.164.213:8080
13:38:19:WU01:FS00:Connecting to 65.254.110.245:8080
13:38:19:WU03:FS01:Connecting to 65.254.110.245:8080
13:38:19:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
13:38:19:WU02:FS01:Trying to send results to collection server
13:38:19:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
13:38:19:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:38:19:WU02:FS01:Connecting to 155.247.164.214:8080
13:38:19:WU00:FS01:Trying to send results to collection server
13:38:20:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
13:38:20:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:38:20:WU00:FS01:Connecting to 155.247.164.214:8080
13:38:20:WU01:FS00:Connecting to 18.218.241.186:80
13:38:20:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:38:20:WU03:FS01:Connecting to 18.218.241.186:80
13:38:20:ERROR:WU02:FS01:Exception: Transfer failed
13:38:20:ERROR:WU00:FS01:Exception: Transfer failed
13:38:20:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:3900 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771ad0cf1564
13:38:20:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
13:38:20:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:38:20:WU02:FS01:Connecting to 155.247.164.213:8080
13:38:20:ERROR:WU01:FS00:Exception: Could not get an assignment
13:38:20:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:38:20:ERROR:WU03:FS01:Exception: Could not get an assignment
13:38:20:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
13:38:20:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
13:38:20:WU01:FS00:Connecting to 65.254.110.245:8080
13:38:20:WU00:FS01:Connecting to 155.247.164.213:8080
13:38:21:WU03:FS01:Connecting to 65.254.110.245:8080
13:38:21:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
13:38:21:WU02:FS01:Trying to send results to collection server
13:38:21:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
13:38:21:WU02:FS01:Connecting to 155.247.164.214:8080
13:38:21:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:38:21:WU00:FS01:Trying to send results to collection server
13:38:21:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
13:38:21:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:38:21:WU00:FS01:Connecting to 155.247.164.214:8080
13:38:21:WU01:FS00:Connecting to 18.218.241.186:80
13:38:21:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:38:21:WU03:FS01:Connecting to 18.218.241.186:80
13:38:21:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:38:21:ERROR:WU01:FS00:Exception: Could not get an assignment
13:38:21:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:38:21:ERROR:WU03:FS01:Exception: Could not get an assignment
13:38:22:ERROR:WU00:FS01:Exception: Transfer failed
13:38:22:13:127.0.0.1:New Web connection
13:38:23:ERROR:WU02:FS01:Exception: Transfer failed
13:39:20:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
13:39:20:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
13:39:20:WU00:FS01:Connecting to 155.247.164.213:8080
13:39:21:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11758 run:0 clone:3900 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d771ad0cf1564
13:39:21:WU02:FS01:Uploading 55.24MiB to 155.247.164.213
13:39:21:WU02:FS01:Connecting to 155.247.164.213:8080
13:39:21:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
13:39:21:WU00:FS01:Trying to send results to collection server
13:39:21:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
13:39:21:WU00:FS01:Connecting to 155.247.164.214:8080
13:39:21:WU01:FS00:Connecting to 65.254.110.245:8080
13:39:21:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
13:39:21:WU02:FS01:Trying to send results to collection server
13:39:21:WU02:FS01:Uploading 55.24MiB to 155.247.164.214
13:39:21:WU02:FS01:Connecting to 155.247.164.214:8080
13:39:21:ERROR:WU00:FS01:Exception: Transfer failed
13:39:21:WU03:FS01:Connecting to 65.254.110.245:8080
13:39:21:ERROR:WU02:FS01:Exception: Transfer failed
13:39:21:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:39:21:WU01:FS00:Connecting to 18.218.241.186:80
13:39:22:WARNING:WU03:FS01:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
13:39:22:WU03:FS01:Connecting to 18.218.241.186:80
13:39:22:WARNING:WU01:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:39:22:ERROR:WU01:FS00:Exception: Could not get an assignment
13:39:22:WARNING:WU03:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:39:22:ERROR:WU03:FS01:Exception: Could not get an assignment
As your client is actually trying to fetch new WUs, on separate streams (i.e. WU01 and WU03), I don't see any indication that the completed WUs (WU00 & WU02, both project 11758) are blocking anything - this part of the problem is that the servers/researchers are overloaded (can't dish out enough new WUs). However not being able to submit 11758 to 213/214 does seem to be something else and separate from the general overload issue.
alxbelu wrote:
As your client is actually trying to fetch new WUs, on separate streams (i.e. WU01 and WU03), I don't see any indication that the completed WUs (WU00 & WU02, both project 11758) are blocking anything - this part of the problem is that the servers/researchers are overloaded (can't dish out enough new WUs). However not being able to submit 11758 to 213/214 does seem to be something else and separate from the general overload issue.
Indeed, after restarting the software that particular issue went away. Now it's attempting to send and get units. Without success so far. Before the restart it didn't even do that, there were no attempts after finishing the previous unit. To restart it I had to end the previous instance from the Windows Task Manager, it refused to close gracefully. So clearly something was stuck there.
There's a retry timer for each WUXX which increases with every failed attempt (easily increasing beyond multiple hours), pausing and unpausing the associated folding slots usually resets them to 0 (but restarting has the same effect).
Yes, I'm aware of the exponential backoff feature, but it didn't seem to be the case here, since it stopped trying to get units completely, 0 attempts after finishing the previous unit, without me pausing anything (I was sleeping at the time).
16:29:27:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:11753 run:0 clone:3381 gen:0 core:0x22 unit:0x000000009bf7a4d55e6d76cab65037c1
16:29:28:WU01:FS01:Uploading 49.92MiB to 155.247.164.213
16:29:28:WU01:FS01:Connecting to 155.247.164.213:8080
16:29:34:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
16:29:34:WU01:FS01:Trying to send results to collection server
16:29:34:WU01:FS01:Uploading 49.92MiB to 155.247.164.214
16:29:34:WU01:FS01:Connecting to 155.247.164.214:8080
16:29:35:ERROR:WU01:FS01:Exception: Transfer failed
The above is repeated a few dozen times so I won't bore everyone.
The interesting bit: while this has been pending upload forever, many WU's have since been completed and uploaded to that very .214.
But I ain't got a clue as to what or what's behind this.
This does appear to be a server overload issue. There has been a very sharp (and welcome!) increase in F@H activity, and the team are working to put more servers and work online. The system is fairly good at retrying until transfers succeed, though in the present circumstances some failures are possible.
There is nothing that can be done to transfer uploads to a different server, you should see in the logs that most projects will try two different addresses for uploads in any case.
davidcoton wrote:This does appear to be a server overload issue. There has been a very sharp (and welcome!) increase in F@H activity, and the team are working to put more servers and work online. The system is fairly good at retrying until transfers succeed, though in the present circumstances some failures are possible.
There is nothing that can be done to transfer uploads to a different server, you should see in the logs that most projects will try two different addresses for uploads in any case.
Honestly, I do not think this is an overload issue; we are multiple people having specific issues with mainly WU 11758 submitting to 213/214 that were distributed during the weekend, and we are multiple people that have successfully received and submitted WUs from/to these very servers since. The servers in question were offline for large parts of at least Sunday, and I very much suspect that this is a serverside config issue (i.e. it might have crashed and not saved which clients received which WUs).