Unable to upload to servers 140.163.4.200 & 140.163.4.210

[Teia]

Hi,

I have 3 WUs failling to send result to work servers. I've got a "Exception: 10002: Received short response, expected 512 bytes, got 0" exception.

I edit my post, the only remaining issues are with Work Server 140.163.4.200 and Collection Server 140.163.4.210.

Code: Select all

08:09:48:WU02:FS01:Upload 96.25%
08:10:19:WARNING:WU02:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
08:10:19:WU02:FS01:Trying to send results to collection server
08:10:19:WU02:FS01:Uploading 18.96MiB to 140.163.4.210
08:10:19:WU02:FS01:Connecting to 140.163.4.210:8080
08:10:51:ERROR:WU02:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
08:26:50:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:17316 run:0 clone:2720 gen:148 core:0x22 unit:0x00000aa000000094000043a400000000
08:26:50:WU02:FS01:Uploading 18.96MiB to 140.163.4.200
08:26:50:WU02:FS01:Connecting to 140.163.4.200:8080
08:27:14:WU02:FS01:Upload 1.65%
08:27:20:WU02:FS01:Upload 8.90%
08:27:26:WU02:FS01:Upload 15.49%
08:27:32:WU02:FS01:Upload 25.38%
08:27:38:WU02:FS01:Upload 36.59%
08:27:44:WU02:FS01:Upload 50.76%
08:27:50:WU02:FS01:Upload 72.19%
08:27:56:WU02:FS01:Upload 94.28%
08:28:28:WARNING:WU02:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
08:28:28:WU02:FS01:Trying to send results to collection server
08:28:28:WU02:FS01:Uploading 18.96MiB to 140.163.4.210
08:28:28:WU02:FS01:Connecting to 140.163.4.210:8080
08:28:34:WU02:FS01:Upload 67.57%
08:28:40:WU02:FS01:Upload 88.34%
08:29:13:ERROR:WU02:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0

Teia

[gacuxz]

Got stuck with 128.252.203.10 too. Same error message.

[Teia]

The problem was with my AV (Kaspersky Security Cloud).

[Neil-B]

Thanks for confirming this .. useful information since it confirms av issue not server issue .. allows us to share experiences with others who may believe this type of error is server issue and can't be av

[gacuxz]

Sorry, for me it was a Kaspersky AV problem indeed:
viewtopic.php?f=18&t=36810&p=349272#p349272

[Jayfreck]

I think there is still something wrong with these servers. I've had a bunch of work units stuck on send for over 24 hours now. This is what keeps happening:

09:13:59:WU00:FS01:Trying to send results to collection server
09:13:59:WU00:FS01:Uploading 8.43MiB to 140.163.4.200
09:13:59:WU00:FS01:Connecting to 140.163.4.200:8080
09:14:29:WU00:FS01:Upload 2.22%
09:14:29:ERROR:WU00:FS01:Exception: Transfer failed

tried with AV disabled, made no difference

[Neil-B]

What firewall/av .. at least two have been identified an not necessarily being fully disabled when people think they are .. I'm may also be isp/route issues .. and I am sure someone will have flagged to the server admins to check that side as well

[Neil-B]

One check some people use is alternative connection such as using a mobile hotspot .. in some cases this has bypassed issue and helps track down where it is occuring

[ajm]

I checked the logs of two machines (4 GPUs) going backward in time. There was some trouble with 140.163.4.200 around 07:30 System Time ("now" being 14:15). I found one such error during an upload, but 18.188.125.154 immediately took over as CS:

Code: Select all

07:24:36:WU01:FS02:Uploading 24.07MiB to 140.163.4.200
07:24:36:WU01:FS02:Connecting to 140.163.4.200:8080
07:25:07:WU01:FS02:Upload 0.78%
07:25:07:WARNING:WU01:FS02:Exception: Failed to send results to work server: Transfer failed
07:25:07:WU01:FS02:Trying to send results to collection server
07:25:07:WU01:FS02:Uploading 24.07MiB to 18.188.125.154
07:25:07:WU01:FS02:Connecting to 18.188.125.154:8080
07:25:13:WU01:FS02:Upload 11.95%
07:25:19:WU01:FS02:Upload 24.67%
07:25:25:WU01:FS02:Upload 37.66%
07:25:31:WU01:FS02:Upload 50.64%
07:25:37:WU01:FS02:Upload 63.62%
07:25:43:WU01:FS02:Upload 76.35%
07:25:49:WU01:FS02:Upload 89.33%
07:25:54:WU01:FS02:Upload complete
07:25:54:WU01:FS02:Server responded WORK_ACK (400)
07:25:54:WU01:FS02:Final credit estimate, 200601.00 points
07:25:54:WU01:FS02:Cleaning up

Maybe this CS connection was setup in the meantime? This is with McAfee.

[gacuxz]

Got another one stuck WU with Kaspersky Endpoint Antivirus:

Code: Select all

07:04:54:WU02:FS00:0xa7:Completed 123750 out of 125000 steps (99%)
07:07:20:WU02:FS00:0xa7:Completed 125000 out of 125000 steps (100%)
07:07:22:WU02:FS00:0xa7:Saving result file ..\logfile_01.txt
07:07:22:WU02:FS00:0xa7:Saving result file frame226.trr
07:07:22:WU02:FS00:0xa7:Saving result file frame226.xtc
07:07:22:WU02:FS00:0xa7:Saving result file md.log
07:07:22:WU02:FS00:0xa7:Saving result file science.log
07:07:22:WU02:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
07:07:22:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
07:07:22:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:07:22:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:07:22:WU02:FS00:Connecting to 66.170.111.50:8080
07:07:53:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:07:53:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:07:53:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:07:53:WU02:FS00:Connecting to 66.170.111.50:8080
07:08:23:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:08:53:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:08:53:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:08:53:WU02:FS00:Connecting to 66.170.111.50:8080
07:09:23:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:10:30:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:10:30:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:10:30:WU02:FS00:Connecting to 66.170.111.50:8080
07:11:00:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:13:07:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:13:07:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:13:07:WU02:FS00:Connecting to 66.170.111.50:8080
07:13:38:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:17:22:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:17:22:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:17:22:WU02:FS00:Connecting to 66.170.111.50:8080
07:17:52:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:24:13:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:24:13:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:24:13:WU02:FS00:Connecting to 66.170.111.50:8080
07:24:43:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:35:19:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:35:19:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:35:19:WU02:FS00:Connecting to 66.170.111.50:8080
07:35:49:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0

There is nothing in Kaspersky logs related to the work server F@H tries to connect to or F@H application itself. Meantime as F@H can't connect to work server looks like it can't get an address of the collection server and at that time it's noted as 0.0.0.0. But as soon as I disable Kaspersky Endpoint AV WU is almost instantly uploaded:

Code: Select all

07:42:57:WU00:FS00:0xa7:Completed 3125 out of 62500 steps (5%)
07:49:33:WU00:FS00:0xa7:Completed 3750 out of 62500 steps (6%)
07:53:15:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:53:15:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:53:15:WU02:FS00:Connecting to 66.170.111.50:8080
07:53:21:WU02:FS00:Upload 12.90%
07:53:27:WU02:FS00:Upload 29.88%
07:53:33:WU02:FS00:Upload 46.86%
07:53:39:WU02:FS00:Upload 63.16%
07:53:45:WU02:FS00:Upload 80.14%
07:53:51:WU02:FS00:Upload 96.44%
07:53:52:WU02:FS00:Upload complete
07:53:52:WU02:FS00:Server responded WORK_ACK (400)
07:53:52:WU02:FS00:Final credit estimate, 2897.00 points
07:53:52:WU02:FS00:Cleaning up

[Neil-B]

There 0.0.0.0 means no CS is set for the project which isn't unusual .. unfortunately I usd a different av/firewall so can't help with the lack of log messages but the behaviour does indicate it may be the cause of the blockages .. are the any specific inspection settings iirc another folder found by turning this off it resolved their issue

[bruce]

This sort of issue can also be complicated by the fact that Collection Servers are assigned at the time a WU is downloaded. If the Work Server happens to go off-line you can end up with multiple WUs from a single project which are trying to go back to different Collection Servers. I have never seen a specific PRCG change from one CS to another CS but I have seen PRCG1 and PRCG2 trying to upload to different CSs. This is very rare, though.

[Joe_H]

Also not a user of Kaspersky AV, but in general this is what is required for the F@h software to communicate with the servers:

FAHClient needs the same permissions to both upload and download data over ports 80 and 8080 as a browser like Firefox, Chrome, Edge or others. A number of AV or firewall products only allow this for "known" browsers, FAHClient is often not considered "known" and an exception needs to be added.

The connection type is HTTP over a TCP connection.

The data has to pass unmodified.]

A final issue people can run into is routers, both local and in between the client machine and the F@h servers, is dropping of HTTP ACK packets. Some routers can have traffic priority settings where they drop the ACK packets as unimportant during high loads. The thought appears to be its a browser, and if a load does not complete, a person will just hit "Refresh". This most often shows up on uploads where the WU is sent, but no ACK is received back. The client will retry sending until it either gets an ACK or the WU reaches the final deadline.

[ipkh]

My system can't connect to .200 at the moment.
All connection attempts fail.

[ajm]

My system can't connect to .200 at the moment.
All connection attempts fail.

Same here. Last successful connection at 06:52:02 (now is 09:27:09)