Page 6 of 12
Re: Send Errors - 155.247.164.213 & .214
Posted: Sat Mar 21, 2020 5:15 pm
by tsunami2311
I was going crazy other night wonder why the WU isnt being sent Then I cam here and read same thing..
Code: Select all
*********************** Log Started 2020-03-21T17:00:12Z ***********************
17:00:12:************************* Folding@home Client *************************
17:00:12: Website: https://foldingathome.org/
17:00:12: Copyright: (c) 2009-2018 foldingathome.org
17:00:12: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:00:12: Args: --open-web-control
17:00:12: Config: C:\Users\****\AppData\Roaming\FAHClient\config.xml
17:00:12:******************************** Build ********************************
17:00:12: Version: 7.5.1
17:00:12: Date: May 11 2018
17:00:12: Time: 13:06:32
17:00:12: Repository: Git
17:00:12: Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
17:00:12: Branch: master
17:00:12: Compiler: Visual C++ 2008
17:00:12: Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
17:00:12: Platform: win32 10
17:00:12: Bits: 32
17:00:12: Mode: Release
17:00:12:******************************* System ********************************
17:00:12: CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
17:00:12: CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
17:00:12: CPUs: 8
17:00:12: Memory: 15.95GiB
17:00:12: Free Memory: 13.19GiB
17:00:12: Threads: WINDOWS_THREADS
17:00:12: OS Version: 6.2
17:00:12: Has Battery: false
17:00:12: On Battery: false
17:00:12: UTC Offset: -4
17:00:12: PID: 1016
17:00:12: CWD: C:\Users\*****\AppData\Roaming\FAHClient
17:00:12: OS: Windows 10 Enterprise
17:00:12: OS Arch: AMD64
17:00:12: GPUs: 1
17:00:12: GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070 Ti] 8186
17:00:12: CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:10.2
17:00:12:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:442.59
17:00:12: Win32 Service: false
17:00:12:***********************************************************************
17:00:12:<config>
17:00:12: <!-- Folding Core -->
17:00:12: <gpu-usage v='50'/>
17:00:12:
17:00:12: <!-- Network -->
17:00:12: <proxy v=':8080'/>
17:00:12:
17:00:12: <!-- User Information -->
17:00:12: <team v='69411'/>
17:00:12: <user v='tsunami231'/>
17:00:12:
17:00:12: <!-- Work Unit Control -->
17:00:12: <max-units v='3'/>
17:00:12: <next-unit-percentage v='98'/>
17:00:12:
17:00:12: <!-- Folding Slots -->
17:00:12: <slot id='0' type='CPU'>
17:00:12: <cpus v='3'/>
17:00:12: <paused v='true'/>
17:00:12: </slot>
17:00:12: <slot id='1' type='GPU'>
17:00:12: <paused v='true'/>
17:00:12: </slot>
17:00:12:</config>
17:00:12:Trying to access database...
17:00:12:Successfully acquired database lock
17:00:12:Enabled folding slot 00: PAUSED cpu:3 (by user)
17:00:12:Enabled folding slot 01: PAUSED gpu:0:GP104 [GeForce GTX 1070 Ti] 8186 (by user)
17:00:12:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:00:12:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:00:12:WU00:FS01:Connecting to 155.247.164.213:8080
17:00:13:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:00:13:WU00:FS01:Trying to send results to collection server
17:00:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:00:13:WU00:FS01:Connecting to 155.247.164.214:8080
17:00:13:ERROR:WU00:FS01:Exception: Transfer failed
17:00:13:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:00:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:00:13:WU00:FS01:Connecting to 155.247.164.213:8080
17:00:13:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:00:13:WU00:FS01:Trying to send results to collection server
17:00:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:00:13:WU00:FS01:Connecting to 155.247.164.214:8080
17:00:13:ERROR:WU00:FS01:Exception: Transfer failed
17:00:14:13:127.0.0.1:New Web connection
17:01:13:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:01:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:01:13:WU00:FS01:Connecting to 155.247.164.213:8080
17:01:13:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:01:13:WU00:FS01:Trying to send results to collection server
17:01:13:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:01:13:WU00:FS01:Connecting to 155.247.164.214:8080
17:01:13:ERROR:WU00:FS01:Exception: Transfer failed
17:02:50:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:02:50:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:02:50:WU00:FS01:Connecting to 155.247.164.213:8080
17:02:50:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:02:50:WU00:FS01:Trying to send results to collection server
17:02:50:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:02:50:WU00:FS01:Connecting to 155.247.164.214:8080
17:02:51:ERROR:WU00:FS01:Exception: Transfer failed
17:05:28:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:05:28:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:05:28:WU00:FS01:Connecting to 155.247.164.213:8080
17:05:28:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:05:28:WU00:FS01:Trying to send results to collection server
17:05:28:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:05:28:WU00:FS01:Connecting to 155.247.164.214:8080
17:05:28:ERROR:WU00:FS01:Exception: Transfer failed
17:09:42:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:2325 gen:0 core:0x22 unit:0x000000069bf7a4d55e6d771534b82be1
17:09:42:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
17:09:42:WU00:FS01:Connecting to 155.247.164.213:8080
17:09:44:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:09:44:WU00:FS01:Trying to send results to collection server
17:09:44:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
17:09:44:WU00:FS01:Connecting to 155.247.164.214:8080
17:09:48:ERROR:WU00:FS01:Exception: Transfer failed
I been turn off the restarting the client and once the "retry attempts start to take 10+ minutes, I dont mind doing WU on my GPU or CPU but sit there for hours doing the work with both cpu and gpu pushing 70c just to not be able to send the results then it was special if it times out. Either Turn the server off till it fixed there is not point assigning work if it cant be collected. just be comes waste on time and engery on both parties
Re: Send Errors - 155.247.164.213 & .214
Posted: Sat Mar 21, 2020 6:57 pm
by vnicolici
vangli wrote:Found just the same as Whittle . Transmission ends with HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE analyzing with wireshark. One possibility seems to be that the collector isn't able to receive a chunk of 55 Mbyte . Not very uncommon in a web server setup. Could it be that a reconfiguration to allow chunks of 64 Mbyte or something like that will help? Wireshark analyze show that a connection is established, but then disconnected a part out in transmission
Issue persists for me as well. I'm unable to upload two units finished Monday and Tuesday, and Wireshark shows the same error:
Code: Select all
HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE\r\n
<html><head><title>413 HTTP_REQUEST_ENTITY_TOO_LARGE</title></head><body><h1>413 HTTP_REQUEST_ENTITY_TOO_LARGE</h1></body></html>
Corresponding log entry:
Code: Select all
18:34:52:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11758 run:0 clone:5213 gen:0 core:0x22 unit:0x000000009bf7a4d55e6e8abff76953ed
18:34:52:WU00:FS01:Uploading 55.24MiB to 155.247.164.213
18:34:52:WU00:FS01:Connecting to 155.247.164.213:8080
18:34:53:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
18:34:53:WU00:FS01:Trying to send results to collection server
18:34:53:WU00:FS01:Uploading 55.24MiB to 155.247.164.214
18:34:53:WU00:FS01:Connecting to 155.247.164.214:8080
18:34:55:ERROR:WU00:FS01:Exception: Transfer failed
A list of the most recent failure replies from .213 and .214 when trying to upload, as seen with Wireshark:
Code: Select all
No. Timestamp Source Destination Length Info Src port Dest port
93 2020-03-21 18:30:38.018503 155.247.164.213 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54675
155 2020-03-21 18:30:38.282700 155.247.164.214 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54676
185 2020-03-21 18:30:38.292086 155.247.164.213 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54677
216 2020-03-21 18:30:38.908294 155.247.164.214 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54680
653 2020-03-21 18:32:15.336840 155.247.164.213 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54729
742 2020-03-21 18:32:15.627243 155.247.164.213 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54731
874 2020-03-21 18:32:16.442598 155.247.164.214 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54732
876 2020-03-21 18:32:16.526702 155.247.164.214 192.168.32.135 316 HTTP/1.1 503 HTTP_SERVICE_UNAVAILABLE (text/html) 8080 54733
3488 2020-03-21 18:34:52.556879 155.247.164.213 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54842
3522 2020-03-21 18:34:52.809037 155.247.164.214 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54843
3538 2020-03-21 18:34:52.839153 155.247.164.213 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54844
3632 2020-03-21 18:34:55.006346 155.247.164.214 192.168.32.135 60 HTTP/1.0 413 HTTP_REQUEST_ENTITY_TOO_LARGE (text/html) 8080 54845
When I asked in a separate thread for the server to send the clients more details about the problem, I got a reply that it's not a good idea, because some people would attempt to cheat if the server would give more error details to the clients.
While that kind of makes sense, although I'm not a big fan of security through obscurity, the problem is the client is not even displaying all the information the servers actually send to it. That doesn't make sense, since any potential cheater could easily find that information using Wireshark.
If the information returned by the server can't be made more verbose for security reasons, at the very least the client should be modified to show all the information the server actually sends it in reply, there is no good reason to hide it in my opinion.
I mean it's almost a week of wasting people's time troubleshooting this because somebody didn't add a 2-3 lines of code needed to log all the information sent by the server. That's not cool.
If anyone needs the full Wireshark capture of the communication to troubleshoot this further I can send it to him / her, unfortunately it seems this forum doesn't seem to support attaching files to a comment, but I can send a link to the capture file to anyone that needs it.
Re: Send Errors - 155.247.164.213 & .214
Posted: Sat Mar 21, 2020 7:17 pm
by ap1978
I've got 3 completed WUs waiting.
Re: Send Errors - 155.247.164.213 & .214
Posted: Sat Mar 21, 2020 7:22 pm
by misha-mzs
I've got 5 already
Re: Send Errors - 155.247.164.213 & .214
Posted: Sat Mar 21, 2020 8:50 pm
by vangli
According to the server status page, the software has just been upgraded from version 9.5.2 to 9.5.6. for both 213 and 214. However same sad result for mine part. Will start a new wireshark session to see if it any changes in communication.
Edit: Same result looking in with wireshark, however some data seems to be transfered before disconnecting with same message, 413 HTTP_REQUEST_ENTITY_TOO_LARGE
Re: Send Errors - 155.247.164.213 & .214
Posted: Sat Mar 21, 2020 10:41 pm
by tsunami2311
I now have 3 to send that wont send, The server should stop Assigning work if CS cant accept it for these job, it just waste if they wind up timing out for both sides
Re: Send Errors - 155.247.164.213 & .214
Posted: Sat Mar 21, 2020 11:15 pm
by vnicolici
Now we know that, unless the servers lie "to prevent cheating", the problem is most likely the result size, since they return HTTP_REQUEST_ENTITY_TOO_LARGE when attempting to upload the result.
I looked through my logs from this week to try to get more insight about the problem. This is what I found so far:
- out of the 58 units I processed so far 56 results have uploaded successfully;
- the 2 WU results that fail uploading are both 55.24 MiB in size and are both from project 11758 and the attempts are to servers 155.247.164.213 & .214;
- I didn't process any other WUs from project 11758;
- the largest WU results accepted by either .213 or .214 has been a unit of 21.92 MiB from project 11753, which was accepted after 3 failed attempts.
- I found 2 other WU results that are also both 55.24 MiB in size, from a different project, 11764, but those were successfully uploaded to 128.252.203.10;
- the remaining WU results from the logs have been smaller, the largest ones being 5 WU results of 49.92 MiB accepted by 128.252.203.10 and 128.252.203.2
So this evidence suggests the issue is related to the servers .213 / .214 or project 11758 not being configured properly to accept large WU results, and that the configured result limit is somewhere between 21.92 MiB and 55.24MiB.
I hope this helps the people in charge of the server/project fix the issue.
Re: Send Errors - 155.247.164.213 & .214
Posted: Sat Mar 21, 2020 11:55 pm
by Empie
Getting "Failed to send results to work server: Transfer failed" on both of these too.
Re: Send Errors - 155.247.164.213 & .214
Posted: Sun Mar 22, 2020 12:10 am
by L.shawcross
Still having problems sending results to 155.247.164.213
And 155.247.164.214
I have two WU 11758 ready to send but fails
If I ping either server I receive a response within 40ms or so
Im not interested in points I just want the data to get through and do the job I have my PC running flat out for.
Any update on this? anything I can do from my end?
Re: Send Errors - 155.247.164.213 & .214
Posted: Sun Mar 22, 2020 1:29 am
by _r2w_ben
vnicolici wrote:- I found 2 other WU results that are also both 55.24 MiB in size, from a different project, 11764, but those were successfully uploaded to 128.252.203.10;
- the remaining WU results from the logs have been smaller, the largest ones being 5 WU results of 49.92 MiB accepted by 128.252.203.10 and 128.252.203.2
I check my logs and have uploaded large results to other servers. Whatever request size related setting is different on those needs to be applied to .213/.214.
Code: Select all
112.77MiB to 128.252.203.10
86.85MiB to 128.252.203.4
86.96MiB to 140.163.4.241
Re: Send Errors - 155.247.164.213 & .214
Posted: Sun Mar 22, 2020 8:38 am
by vangli
After a week, without any substantial information or solution from the administrators of those two servers, either directly or through moderators of this forum, the only conclusion is to close down those two servers until the problem is fixed or they are able to stop sending out new WUs which cannot be collected. I am almost there to stop supporting F&H and return to BOINC projects. This neglicting of supporters are not good.
Re: Send Errors - 155.247.164.213 & .214
Posted: Sun Mar 22, 2020 10:16 am
by davidcoton
@vangli I understand your frustration. The issues are known to the team, and are regularly passed up to them. We could post platitudes in reply, but I prefer to say nothing until there is some news coming back. Please understand that the team is very small, AIUI only one full-time tech person for IT, the rest is done by the researchers who also have to take care of the science side and the vast step-change in the size of the operation -- not to mention that COVID-19 projects are entirely new work.
Re: Send Errors - 155.247.164.213 & .214
Posted: Sun Mar 22, 2020 10:51 am
by vangli
davidcoton wrote:@vangli I understand your frustration. The issues are known to the team, and are regularly passed up to them. We could post platitudes in reply, but I prefer to say nothing until there is some news coming back. Please understand that the team is very small, AIUI only one full-time tech person for IT, the rest is done by the researchers who also have to take care of the science side and the vast step-change in the size of the operation -- not to mention that COVID-19 projects are entirely new work.
This is such simple information I ask for. Nothing more. I quiet well understand the situation with this information. It tells me and many other that they are seriously working on it with small resources, and aware of the situation. Thanks
Re: Send Errors - 155.247.164.213 & .214
Posted: Sun Mar 22, 2020 6:46 pm
by vangli
Just out of pure curiosity, when I read what shown when i hoover over the CS tag in server status, the green Yes:
On server vav16.ocis.temple.edu (155.247.164.214):
Accepted: 155.247.166.219 155.247.166.220 128.252.203.2 128.252.203.9 40.114.52.201 13.90.152.57 37.187.12.48
Failed: 13.59.169.226 155.247.164.213 155.247.164.214 140.163.4.231 140.163.4.232 140.163.4.241 128.174.73.74 128.252.203.10 128.252.203.4 13.82.98.119
and on server vav15.ocis.temple.edu (155.247.164.213):
Accepted: 155.247.164.214
Failed:
The WU's in question has 155.247.164.213 as work server and 155.247.164.214 as collection server. No other possibilities.
Am I right in that a collection server accept WU's originated from those Accepted addresses only? If failed or not mentioned, they will not? If this is true, a WU from 155.247.164.213 which only mention a single collection server, namely 155.247.164.214, cannot be uploaded. The 213 work server block it self for upload, and 214 does not accept anything from itself. Am I right?
Re: Send Errors - 155.247.164.213 & .214
Posted: Mon Mar 23, 2020 12:05 am
by davidcoton
vangli wrote:
Am I right?
Not entirely. Work servers accept their own work (usually) without using the CS mechanism. It is only when that fails that the client calls the CS that was set when the WU was downloaded.
Every WS should specify a CS (AFAIK it can only specify one), but this is not always done. The CS may be another WS or a separate server.