Failed to send results to work server: Transfer failed

[Quad2000]

Hello together, I have the Problem, that sometimes the Client 7.6.9 doesn´t upload the finished file. See log file. Is this a Problem at my Client or PC or only the current overload of the folding Servers?
Thank you for your help! I wish you a nice and healthy Weekend!

14:15:48:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14415 run:0 clone:177 gen:43 core:0x22 unit:0x000000380d5262775e839e5d6083156b
14:15:48:WU00:FS01:Uploading 201.55MiB to 13.82.98.119
14:15:48:WU00:FS01:Connecting to 13.82.98.119:8080
14:16:03:WU00:FS01:Upload 0.03%
14:16:38:WU00:FS01:Upload 0.09%
14:16:38:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
14:16:38:WU00:FS01:Trying to send results to collection server
14:16:38:WU00:FS01:Uploading 201.55MiB to 52.224.109.74
14:16:38:WU00:FS01:Connecting to 52.224.109.74:8080
14:16:39:ERROR:WU00:FS01:Exception: Transfer failed
14:32:11:WU02:FS01:Connecting to 65.254.110.245:80
14:32:12:WARNING:WU02:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration

[Joe_H]

Usually it is just the overload on the server network. If the WU doesn't upload within a few hours, then let us know and we can contact the persons managing the servers.

[Quad2000]

Hello Joe_H, thank you very much for your help. The computer was since yesterday online (only one Client Restart this morning) but no upload possible. Can you please arrange the contact? Thank you very much!

Code: Select all

*********************** Log Started 2020-04-25T04:56:02Z ***********************
04:56:02:****************************** FAHClient ******************************
04:56:02:        Version: 7.6.9
04:56:02:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:56:02:      Copyright: 2020 foldingathome.org
04:56:02:       Homepage: https://foldingathome.org/
04:56:02:           Date: Apr 17 2020
04:56:02:           Time: 11:13:06
04:56:02:       Revision: 398c2b17fa535e0cc6c9d10856b2154c32771646
04:56:02:         Branch: master
04:56:02:       Compiler: Visual C++ 2008
04:56:02:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
04:56:02:       Platform: win32 10
04:56:02:           Bits: 32
04:56:02:           Mode: Release
04:56:02:           Args: --open-web-control
04:56:02:         Config: C:\Users\HJG-RX480\AppData\Roaming\FAHClient\config.xml
04:56:02:******************************** CBang ********************************
04:56:02:           Date: Apr 17 2020
04:56:02:           Time: 11:10:09
04:56:02:       Revision: 2fb0be7809c5e45287a122ca5fbc15b5ae859a3b
04:56:02:         Branch: master
04:56:02:       Compiler: Visual C++ 2008
04:56:02:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
04:56:02:       Platform: win32 10
04:56:02:           Bits: 32
04:56:02:           Mode: Release
04:56:02:******************************* System ********************************
04:56:02:            CPU: Intel(R) Celeron(R) CPU G3900 @ 2.80GHz
04:56:02:         CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
04:56:02:           CPUs: 2
04:56:02:         Memory: 15.96GiB
04:56:02:    Free Memory: 12.88GiB
04:56:02:        Threads: WINDOWS_THREADS
04:56:02:     OS Version: 6.2
04:56:02:    Has Battery: false
04:56:02:     On Battery: false
04:56:02:     UTC Offset: 2
04:56:02:            PID: 5436
04:56:02:            CWD: C:\Users\HJG-RX480\AppData\Roaming\FAHClient
04:56:02:             OS: Windows 10 Enterprise
04:56:02:        OS Arch: AMD64
04:56:02:           GPUs: 1
04:56:02:          GPU 0: Bus:1 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
04:56:02:                 470/480/570/580/590]
04:56:02:           CUDA: Not detected: Failed to open dynamic library 'nvcuda.dll': Das
04:56:02:                 angegebene Modul wurde nicht gefunden.
04:56:02:
04:56:02:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:3004.8
04:56:02:  Win32 Service: false
04:56:02:******************************* libFAH ********************************
04:56:02:           Date: Apr 15 2020
04:56:02:           Time: 14:53:14
04:56:02:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
04:56:02:         Branch: master
04:56:02:       Compiler: Visual C++ 2008
04:56:02:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
04:56:02:       Platform: win32 10
04:56:02:           Bits: 32
04:56:02:           Mode: Release
04:56:02:***********************************************************************
04:56:02:<config>
04:56:02:  <!-- Folding Core -->
04:56:02:  <checkpoint v='30'/>
04:56:02:
04:56:02:  <!-- Folding Slot Configuration -->
04:56:02:  <cause v='COVID_19'/>
04:56:02:
04:56:02:  <!-- Network -->
04:56:02:  <proxy v=':8080'/>
04:56:02:
04:56:02:  <!-- Remote Command Server -->
04:56:02:  <password v='*****'/>
04:56:02:
04:56:02:  <!-- Slot Control -->
04:56:02:  <pause-on-battery v='false'/>
04:56:02:  <power v='FULL'/>
04:56:02:
04:56:02:  <!-- User Information -->
04:56:02:  <passkey v='*****'/>
04:56:02:  <team v='250626'/>
04:56:02:  <user v='HJG-RX480'/>
04:56:02:
04:56:02:  <!-- Folding Slots -->
04:56:02:  <slot id='1' type='GPU'/>
04:56:02:</config>
04:56:02:Trying to access database...
04:56:02:Successfully acquired database lock
04:56:02:Enabled folding slot 01: READY gpu:0:Ellesmere XT [Radeon RX 470/480/570/580/590]
04:56:02:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14415 run:0 clone:177 gen:43 core:0x22 unit:0x000000380d5262775e839e5d6083156b
04:56:02:WU00:FS01:Uploading 201.55MiB to 13.82.98.119
04:56:02:WU00:FS01:Connecting to 13.82.98.119:8080
04:56:02:WU01:FS01:Connecting to 65.254.110.245:80
04:56:03:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
04:56:03:WU01:FS01:Connecting to 18.218.241.186:80
04:56:04:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
04:56:04:WU01:FS01:Connecting to 65.254.110.245:80
04:56:04:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
04:56:04:WU01:FS01:Connecting to 18.218.241.186:80
04:56:04:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
04:56:04:ERROR:WU01:FS01:Exception: Could not get an assignment
04:56:04:WU01:FS01:Connecting to 65.254.110.245:80
04:56:05:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
04:56:05:WU01:FS01:Connecting to 18.218.241.186:80
04:56:05:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
04:56:05:WU01:FS01:Connecting to 65.254.110.245:80
04:56:06:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
04:56:06:WU01:FS01:Connecting to 18.218.241.186:80
04:56:06:WU01:FS01:Assigned to work server 128.252.203.10
04:56:06:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:Ellesmere XT [Radeon RX 470/480/570/580/590] from 128.252.203.10
04:56:06:WU01:FS01:Connecting to 128.252.203.10:8080
04:56:06:19:127.0.0.1:New Web session
04:56:27:WARNING:WU01:FS01:WorkServer connection failed on port 8080 trying 80
04:56:27:WU01:FS01:Connecting to 128.252.203.10:80
04:56:34:WU00:FS01:Upload 0.09%
04:56:34:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
04:56:34:WU00:FS01:Trying to send results to collection server
04:56:34:WU00:FS01:Uploading 201.55MiB to 52.224.109.74
04:56:34:WU00:FS01:Connecting to 52.224.109.74:8080
04:56:35:ERROR:WU00:FS01:Exception: Transfer failed
04:56:35:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14415 run:0 clone:177 gen:43 core:0x22 unit:0x000000380d5262775e839e5d6083156b
04:56:35:WU00:FS01:Uploading 201.55MiB to 13.82.98.119
04:56:35:WU00:FS01:Connecting to 13.82.98.119:8080
04:56:48:ERROR:WU01:FS01:Exception: Failed to connect to 128.252.203.10:80: Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat.
04:57:05:WU01:FS01:Connecting to 65.254.110.245:80
04:57:05:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
04:57:05:WU01:FS01:Connecting to 18.218.241.186:80
04:57:06:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
04:57:06:WU01:FS01:Connecting to 65.254.110.245:80
04:57:06:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
04:57:06:WU01:FS01:Connecting to 18.218.241.186:80
04:57:07:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
04:57:07:ERROR:WU01:FS01:Exception: Could not get an assignment
04:57:10:WU00:FS01:Upload 0.09%
04:57:10:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
04:57:10:WU00:FS01:Trying to send results to collection server
04:57:10:WU00:FS01:Uploading 201.55MiB to 52.224.109.74
04:57:10:WU00:FS01:Connecting to 52.224.109.74:8080
04:57:10:ERROR:WU00:FS01:Exception: Transfer failed
04:57:35:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14415 run:0 clone:177 gen:43 core:0x22 unit:0x000000380d5262775e839e5d6083156b
04:57:35:WU00:FS01:Uploading 201.55MiB to 13.82.98.119
04:57:35:WU00:FS01:Connecting to 13.82.98.119:8080
04:58:16:WU00:FS01:Upload 0.09%
04:58:16:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
04:58:16:WU00:FS01:Trying to send results to collection server
04:58:16:WU00:FS01:Uploading 201.55MiB to 52.224.109.74
04:58:16:WU00:FS01:Connecting to 52.224.109.74:8080
04:58:16:ERROR:WU00:FS01:Exception: Transfer failed
04:58:42:WU01:FS01:Connecting to 65.254.110.245:80
04:58:42:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
04:58:42:WU01:FS01:Connecting to 18.218.241.186:80
04:58:43:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
04:58:43:WU01:FS01:Connecting to 65.254.110.245:80
04:58:43:WARNING:WU01:FS01:Failed to get assignment from '65.254.110.245:80': No WUs available for this configuration
04:58:43:WU01:FS01:Connecting to 18.218.241.186:80
04:58:44:WARNING:WU01:FS01:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
04:58:44:ERROR:WU01:FS01:Exception: Could not get an assignment
04:59:13:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14415 run:0 clone:177 gen:43 core:0x22 unit:0x000000380d5262775e839e5d6083156b
04:59:13:WU00:FS01:Uploading 201.55MiB to 13.82.98.119
04:59:13:WU00:FS01:Connecting to 13.82.98.119:8080
04:59:34:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
04:59:34:WU00:FS01:Connecting to 13.82.98.119:80
04:59:34:WU00:FS01:Upload 0.03%
05:00:00:WU00:FS01:Upload 0.09%
05:00:00:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
05:00:00:WU00:FS01:Trying to send results to collection server
05:00:00:WU00:FS01:Uploading 201.55MiB to 52.224.109.74
05:00:00:WU00:FS01:Connecting to 52.224.109.74:8080
05:00:00:ERROR:WU00:FS01:Exception: Transfer failed

Mod Edit: Added Code Tags - PantherX

[uyaem]

I can also see that in this case, I see "error:FAULTY" which means that the work unit is broken.
Did you have a computer crash, or did a hard reset while FAH was running?

@more knowledgeable people
Will/can the server reject an upload if the WU is erroneous?
Then again, the upload was super slow in the first place (see timestamps), so it is probably server/network overload.

[PantherX]

The "SEND error:FAULTY" means that the WU didn't successfully finish. The reason is unknown as it could be hardware related or it is indeed a bad WU. Hence the server sends out another copy to double check if it is a bad WU or not. If a certain number of WUs have returned as bad, then the Server automatically prevents it from being re-assigned. That limit used to be 3 but am not sure if that changed or not.

The Server will perform a validation check once it receives the WU. If it passes, you get credits and if it fails, it will print the message "Server didn't like result..." in your log file.

[Quad2000]

@uyaem: If I remember correct I had a Restart but no hard crash.

@PantherX: The message "Server didn´t like result" I didn´t get in my log. Can I delete the WU? This file hase 200MB so I should find it in Explorer? Does this solve the Problem?

[Neil-B]

@Quad2000 .. There should be no need to delete WU as client will keep retrying to send (which hopefully it will) … until such time as expiration date passes when it will then delete the WU itself (but hopefully won't come to that) .. Your FAH Client should continue to fold other WUs whilst trying to send the WU in the background (at intervals) - it is not unheard of to have 2 or more WUs waiting for upload.

Delays uploading to servers do happen - usually rarely but with the recent surge it happens more often as things change quicker - the team work as quickly as possible to rectify these issues when they get reports but it can sometimes take a fair bit of time (think days not minutes/hours) especially if a weekend (or holiday) is involved.

[intrepidpursuit]

I am repeatedly getting transfer failed as well on two WUs with the same number and my GPU keeps trying to process a 3rd. It has been more than 24 hours and the uploads keep failing. My other GPU has uploaded to the same server with no problems, but this WU keeps showing up and using 17 hours worth of power and never uploading. Here is a snippet of code. I've rebooted several times trying to solve this problem, but not while the WUs were processing.

Code: Select all

05:21:47:WU03:FS01:Sending unit results: id:03 state:SEND error:NO_ERROR project:16435 run:345 clone:0 gen:1 core:0x22 unit:0x0000000203854c135e9a4efbb242a572
05:21:47:WU03:FS01:Uploading 140.30MiB to 3.133.76.19
05:21:47:WU03:FS01:Connecting to 3.133.76.19:8080
05:21:47:WARNING:WU03:FS01:Exception: Failed to send results to work server: Transfer failed
05:21:47:WU03:FS01:Trying to send results to collection server
05:21:47:WU03:FS01:Uploading 140.30MiB to 3.21.157.11
05:21:47:WU03:FS01:Connecting to 3.21.157.11:8080
05:21:47:ERROR:WU03:FS01:Exception: Transfer failed
05:21:47:WU04:FS01:Sending unit results: id:04 state:SEND error:NO_ERROR project:16435 run:44 clone:0 gen:2 core:0x22 unit:0x0000000403854c135e9a4ef79ece60d9
05:21:47:WU04:FS01:Uploading 133.48MiB to 3.133.76.19
05:21:47:WU04:FS01:Connecting to 3.133.76.19:8080
05:21:47:WARNING:WU04:FS01:Exception: Failed to send results to work server: Transfer failed
05:21:47:WU04:FS01:Trying to send results to collection server
05:21:47:WU04:FS01:Uploading 133.48MiB to 3.21.157.11
05:21:47:WU04:FS01:Connecting to 3.21.157.11:8080
05:21:48:ERROR:WU04:FS01:Exception: Transfer failed

[bruce]

The FAH networking people are looking into problems with 3.21.157.11

[Mark12547]

The owner of the server at 3.133.76.19 is being contacted about the problem, according to this thread: viewtopic.php?f=18&t=34744&start=0

At this time, your best bet is to let the FaH client continue to run and the uploads may succeed when the servers are less busy.

[Neil-B]

Is it the same WU that keeps turning up (same project, run, clone, gen code) or different WUs from the same project? … If different WUs from the same Project then previous guidance applies - just let the client handle the retries and hopefully they will get uploaded … If the same WU (PRCG same each time) then there may be more of an issue.

[hschm92]

Hi, I also have problems with uploading files from 14415 and 14416, which both belong to the same project on myosin.

19:31:58:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:14416 run:0 clone:452 gen:36 core:0x22 unit:0x000000290d5262775e84c8a77f078fa2
19:31:58:WU02:FS01:Uploading 253.20MiB to 13.82.98.119
19:31:58:WU02:FS01:Connecting to 13.82.98.119:8080
19:31:59:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:14415 run:0 clone:527 gen:24 core:0x22 unit:0x000000200d5262775e839e5cf4cae045
19:31:59:WU00:FS01:Uploading 275.38MiB to 13.82.98.119
19:31:59:WU00:FS01:Connecting to 13.82.98.119:8080
19:32:05:WU00:FS01:Upload 0.07%
19:32:05:WU02:FS01:Upload 0.07%
19:32:05:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
19:32:05:WU00:FS01:Trying to send results to collection server
19:32:05:WU00:FS01:Uploading 275.38MiB to 52.224.109.74
19:32:05:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
19:32:05:WU00:FS01:Connecting to 52.224.109.74:8080
19:32:05:WU02:FS01:Trying to send results to collection server
19:32:05:WU02:FS01:Uploading 253.20MiB to 52.224.109.74
19:32:05:WU02:FS01:Connecting to 52.224.109.74:8080
19:32:06:ERROR:WU02:FS01:Exception: Transfer failed
19:32:06:ERROR:WU00:FS01:Exception: Transfer failed

[intrepidpursuit]

Is it the same WU that keeps turning up (same project, run, clone, gen code) or different WUs from the same project? … If different WUs from the same Project then previous guidance applies - just let the client handle the retries and hopefully they will get uploaded … If the same WU (PRCG same each time) then there may be more of an issue.

The project is the same but the run, clone, and gen code are different. I'll just let them sit and queue until further notice. Not sure what else I could do anyway.

[bruce]

Right. There's nothing else that you can do. FAH's networking and support folks have been aware of the problems for some time now. I think the cloud servers are not as reliable as they should be, but I don't know enough about the details to make any aspersions.

[Hamrat]

Right. There's nothing else that you can do. FAH's networking and support folks have been aware of the problems for some time now. I think the cloud servers are not as reliable as they should be, but I don't know enough about the details to make any aspersions.

I'm also having this problem. Is there anything that we can do to sort this out? I see that there are no collection servers available to the IP at the moment. Should we just delete the jobs and let the system request new ones or let the work units time out?