Unable to upload to servers 140.163.4.200 & 140.163.4.210

Moderators: Site Moderators, FAHC Science Team

Teia
Posts: 2
Joined: Wed Feb 17, 2021 4:29 pm

Unable to upload to servers 140.163.4.200 & 140.163.4.210

Post by Teia »

Hi,

I have 3 WUs failling to send result to work servers. I've got a "Exception: 10002: Received short response, expected 512 bytes, got 0" exception.

I edit my post, the only remaining issues are with Work Server 140.163.4.200 and Collection Server 140.163.4.210.

Code: Select all

08:09:48:WU02:FS01:Upload 96.25%
08:10:19:WARNING:WU02:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
08:10:19:WU02:FS01:Trying to send results to collection server
08:10:19:WU02:FS01:Uploading 18.96MiB to 140.163.4.210
08:10:19:WU02:FS01:Connecting to 140.163.4.210:8080
08:10:51:ERROR:WU02:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
08:26:50:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:17316 run:0 clone:2720 gen:148 core:0x22 unit:0x00000aa000000094000043a400000000
08:26:50:WU02:FS01:Uploading 18.96MiB to 140.163.4.200
08:26:50:WU02:FS01:Connecting to 140.163.4.200:8080
08:27:14:WU02:FS01:Upload 1.65%
08:27:20:WU02:FS01:Upload 8.90%
08:27:26:WU02:FS01:Upload 15.49%
08:27:32:WU02:FS01:Upload 25.38%
08:27:38:WU02:FS01:Upload 36.59%
08:27:44:WU02:FS01:Upload 50.76%
08:27:50:WU02:FS01:Upload 72.19%
08:27:56:WU02:FS01:Upload 94.28%
08:28:28:WARNING:WU02:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
08:28:28:WU02:FS01:Trying to send results to collection server
08:28:28:WU02:FS01:Uploading 18.96MiB to 140.163.4.210
08:28:28:WU02:FS01:Connecting to 140.163.4.210:8080
08:28:34:WU02:FS01:Upload 67.57%
08:28:40:WU02:FS01:Upload 88.34%
08:29:13:ERROR:WU02:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
Teia
Last edited by Teia on Thu Feb 18, 2021 8:55 am, edited 1 time in total.
gacuxz
Posts: 10
Joined: Thu May 21, 2020 8:19 am

Re: Unable to upload to servers 206.223.170.146 & 128.252.20

Post by gacuxz »

Got stuck with 128.252.203.10 too. Same error message.
Teia
Posts: 2
Joined: Wed Feb 17, 2021 4:29 pm

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by Teia »

The problem was with my AV (Kaspersky Security Cloud).
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by Neil-B »

Thanks for confirming this .. useful information since it confirms av issue not server issue :) .. allows us to share experiences with others who may believe this type of error is server issue and can't be av :)
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
gacuxz
Posts: 10
Joined: Thu May 21, 2020 8:19 am

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by gacuxz »

Sorry, for me it was a Kaspersky AV problem indeed:
viewtopic.php?f=18&t=36810&p=349272#p349272
Jayfreck
Posts: 8
Joined: Tue Mar 20, 2018 9:16 am

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by Jayfreck »

I think there is still something wrong with these servers. I've had a bunch of work units stuck on send for over 24 hours now. This is what keeps happening:

09:13:59:WU00:FS01:Trying to send results to collection server
09:13:59:WU00:FS01:Uploading 8.43MiB to 140.163.4.200
09:13:59:WU00:FS01:Connecting to 140.163.4.200:8080
09:14:29:WU00:FS01:Upload 2.22%
09:14:29:ERROR:WU00:FS01:Exception: Transfer failed

tried with AV disabled, made no difference
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by Neil-B »

What firewall/av .. at least two have been identified an not necessarily being fully disabled when people think they are .. I'm may also be isp/route issues .. and I am sure someone will have flagged to the server admins to check that side as well
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by Neil-B »

One check some people use is alternative connection such as using a mobile hotspot .. in some cases this has bypassed issue and helps track down where it is occuring
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
ajm
Posts: 750
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by ajm »

I checked the logs of two machines (4 GPUs) going backward in time. There was some trouble with 140.163.4.200 around 07:30 System Time ("now" being 14:15). I found one such error during an upload, but 18.188.125.154 immediately took over as CS:

Code: Select all

07:24:36:WU01:FS02:Uploading 24.07MiB to 140.163.4.200
07:24:36:WU01:FS02:Connecting to 140.163.4.200:8080
07:25:07:WU01:FS02:Upload 0.78%
07:25:07:WARNING:WU01:FS02:Exception: Failed to send results to work server: Transfer failed
07:25:07:WU01:FS02:Trying to send results to collection server
07:25:07:WU01:FS02:Uploading 24.07MiB to 18.188.125.154
07:25:07:WU01:FS02:Connecting to 18.188.125.154:8080
07:25:13:WU01:FS02:Upload 11.95%
07:25:19:WU01:FS02:Upload 24.67%
07:25:25:WU01:FS02:Upload 37.66%
07:25:31:WU01:FS02:Upload 50.64%
07:25:37:WU01:FS02:Upload 63.62%
07:25:43:WU01:FS02:Upload 76.35%
07:25:49:WU01:FS02:Upload 89.33%
07:25:54:WU01:FS02:Upload complete
07:25:54:WU01:FS02:Server responded WORK_ACK (400)
07:25:54:WU01:FS02:Final credit estimate, 200601.00 points
07:25:54:WU01:FS02:Cleaning up
Maybe this CS connection was setup in the meantime? This is with McAfee.
gacuxz
Posts: 10
Joined: Thu May 21, 2020 8:19 am

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by gacuxz »

Got another one stuck WU with Kaspersky Endpoint Antivirus:

Code: Select all

07:04:54:WU02:FS00:0xa7:Completed 123750 out of 125000 steps (99%)
07:07:20:WU02:FS00:0xa7:Completed 125000 out of 125000 steps (100%)
07:07:22:WU02:FS00:0xa7:Saving result file ..\logfile_01.txt
07:07:22:WU02:FS00:0xa7:Saving result file frame226.trr
07:07:22:WU02:FS00:0xa7:Saving result file frame226.xtc
07:07:22:WU02:FS00:0xa7:Saving result file md.log
07:07:22:WU02:FS00:0xa7:Saving result file science.log
07:07:22:WU02:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
07:07:22:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
07:07:22:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:07:22:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:07:22:WU02:FS00:Connecting to 66.170.111.50:8080
07:07:53:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:07:53:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:07:53:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:07:53:WU02:FS00:Connecting to 66.170.111.50:8080
07:08:23:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:08:53:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:08:53:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:08:53:WU02:FS00:Connecting to 66.170.111.50:8080
07:09:23:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:10:30:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:10:30:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:10:30:WU02:FS00:Connecting to 66.170.111.50:8080
07:11:00:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:13:07:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:13:07:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:13:07:WU02:FS00:Connecting to 66.170.111.50:8080
07:13:38:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:17:22:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:17:22:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:17:22:WU02:FS00:Connecting to 66.170.111.50:8080
07:17:52:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:24:13:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:24:13:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:24:13:WU02:FS00:Connecting to 66.170.111.50:8080
07:24:43:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
07:35:19:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:35:19:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:35:19:WU02:FS00:Connecting to 66.170.111.50:8080
07:35:49:WARNING:WU02:FS00:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
There is nothing in Kaspersky logs related to the work server F@H tries to connect to or F@H application itself. Meantime as F@H can't connect to work server looks like it can't get an address of the collection server and at that time it's noted as 0.0.0.0. But as soon as I disable Kaspersky Endpoint AV WU is almost instantly uploaded:

Code: Select all

07:42:57:WU00:FS00:0xa7:Completed 3125 out of 62500 steps (5%)
07:49:33:WU00:FS00:0xa7:Completed 3750 out of 62500 steps (6%)
07:53:15:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:17409 run:0 clone:866 gen:226 core:0xa7 unit:0x00000362000000e20000440100000000
07:53:15:WU02:FS00:Uploading 9.20MiB to 66.170.111.50
07:53:15:WU02:FS00:Connecting to 66.170.111.50:8080
07:53:21:WU02:FS00:Upload 12.90%
07:53:27:WU02:FS00:Upload 29.88%
07:53:33:WU02:FS00:Upload 46.86%
07:53:39:WU02:FS00:Upload 63.16%
07:53:45:WU02:FS00:Upload 80.14%
07:53:51:WU02:FS00:Upload 96.44%
07:53:52:WU02:FS00:Upload complete
07:53:52:WU02:FS00:Server responded WORK_ACK (400)
07:53:52:WU02:FS00:Final credit estimate, 2897.00 points
07:53:52:WU02:FS00:Cleaning up
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by Neil-B »

There 0.0.0.0 means no CS is set for the project which isn't unusual .. unfortunately I usd a different av/firewall so can't help with the lack of log messages but the behaviour does indicate it may be the cause of the blockages .. are the any specific inspection settings iirc another folder found by turning this off it resolved their issue
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by bruce »

This sort of issue can also be complicated by the fact that Collection Servers are assigned at the time a WU is downloaded. If the Work Server happens to go off-line you can end up with multiple WUs from a single project which are trying to go back to different Collection Servers. I have never seen a specific PRCG change from one CS to another CS but I have seen PRCG1 and PRCG2 trying to upload to different CSs. This is very rare, though.
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by Joe_H »

Also not a user of Kaspersky AV, but in general this is what is required for the F@h software to communicate with the servers:

FAHClient needs the same permissions to both upload and download data over ports 80 and 8080 as a browser like Firefox, Chrome, Edge or others. A number of AV or firewall products only allow this for "known" browsers, FAHClient is often not considered "known" and an exception needs to be added.

The connection type is HTTP over a TCP connection.

The data has to pass unmodified.]

A final issue people can run into is routers, both local and in between the client machine and the F@h servers, is dropping of HTTP ACK packets. Some routers can have traffic priority settings where they drop the ACK packets as unimportant during high loads. The thought appears to be its a browser, and if a load does not complete, a person will just hit "Refresh". This most often shows up on uploads where the WU is sent, but no ACK is received back. The client will retry sending until it either gets an ACK or the WU reaches the final deadline.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
ipkh
Posts: 173
Joined: Thu Jul 16, 2015 2:03 pm

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by ipkh »

My system can't connect to .200 at the moment.
All connection attempts fail.
ajm
Posts: 750
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: Unable to upload to servers 140.163.4.200 & 140.163.4.21

Post by ajm »

ipkh wrote:My system can't connect to .200 at the moment.
All connection attempts fail.
Same here. Last successful connection at 06:52:02 (now is 09:27:09)
Post Reply