Upload failures to mostly *.wustl.edu endpoints

[Billy.Oneal]

I have 2 machines running FAH and am constantly seeing output in logs like:

Code: Select all

16:36:41:WU19:FS01:Sending unit results: id:19 state:SEND error:NO_ERROR project:17257 run:774 clone:5 gen:3 core:0x22 unit:0x00000005000000030000436900000306
16:36:41:WU19:FS01:Uploading 51.39MiB to 128.252.203.10
16:36:41:WU19:FS01:Connecting to 128.252.203.10:8080
16:37:11:WARNING:WU19:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0

I don't think it's an issue going on with my computers in particular, because I see things working for some servers but not for others. Looking at the last couple days of logs, I have 17 WORK_ACKs from the following servers:

Code: Select all

[*] 207.53.233.146:8080 (fah.redesignscience.com) x6
[*] 34.72.228.44:8080 (stxfahwork01.silicontx.com) x3
[*] 140.163.4.210:8080 (pllwskifah2.mskcc.org) x2
[*] 128.252.203.13:8080 (highland3.engr.wustl.edu)
[*] 128.252.203.9:8080 (islay.seas.wustl.edu)
[*] 128.174.73.74:8080 x2 (ds01.scs.illinois.edu)
[*] 129.32.209.202:8080 (vav19.fah.temple.edu)
[*] 178.174.196.138:8080 (fah-ws1.bahnhof.net)

I have over 2000 failures to upload (and 20 pending completed WUs on this machine); here is a sampling of recent upload failures:

Code: Select all

[*] 128.252.203.1:8080
[*] 128.252.203.9:8080  (But wait, I also have WORK_ACKs from here?)
[*] 128.252.203.10:8080
[*] 128.252.203.11:8080
[*] 128.252.203.13:8080
[*] 128.252.203.14:8080
[*] 206.223.170.146:8080

The stuck WUs are for projects 17257, 17258, and 18201.

Anything I can do to help diagnose? Thanks folks! (Hope your stats server issue gets fixed)

I've put all the logs in

Code: Select all

https://gist.github.com/BillyONeal/c97f452b99475f16a089a9206dc32140

[gunnarre]

First, can you try connecting to those servers on port 80 and 8080 with a regular web browser? Can you try to tether the computer in question to mobile internet or use VPN temporarily to see if the stuck WUs upload? Alternatively try to use a HTTP proxy?

If you can visit them with a browser, but FAH transfers consistently fails, this looks like a deep packet inspection firewall somewhere along the way is interfering with your uploads, but not your downloads. There might also be an issue with the work servers or their network, but I'm consistently able to transfer WUs to 128.252.203.10, 128.252.203.11, 128.252.203.14

I'm not ruling out something being wrong at Wustl (e.g. their firewall blocking your network for some reason, or error at the work server) - but most of the time when this happens, it's due to deep packet inspection firewalls.

[toTOW]

The common source of these errors is that something is messing your TCP/IP packets ...

Kaspersky is known for this ... Fritzbox routers too ...

You have to find the correct option in your tool/router and disable it for FAHClient.

Here's an example for Kaspersky (in German) :

[Billy.Oneal]

>First, can you try connecting to those servers on port 80 and 8080 with a regular web browser?

No problems hitting them with a regular browser.

>Can you try to tether the computer in question to mobile internet or use VPN temporarily to see if the stuck WUs upload? Alternatively try to use a HTTP proxy?

Tether didn't appear to make any difference.

>this looks like a deep packet inspection firewall somewhere along the way is interfering with your uploads

I don't have any specific firewall software installed. I tried turning off Windows' firewall but that made no difference.

>Kaspersky is known for this ... Fritzbox routers too ...

No Kaspersky or Fritzbox here.

[Billy.Oneal]

OK we're up to 23 stuck WUs and at this point I think I'm just burning power for nothing and reducing your progress by holding WUs hostage, so I'm going to just shut it down for now. If you see other reports of something like this I'm happy to help run down the problem. Thanks folks <3

[bollix47]

Are you using a virus checker other than Windows Defender like Avast or AVG? Sometimes they can get a false positive for a virus ...

Another thought I had would be file size ... the examples you gave result in fairly large upploads and may be restricted by your settings or your ISP settings.

Sorry I can't be more helpful but you're experiencing something I've rarely seen with such frequency.

[gunnarre]

Tether didn't appear to make any difference.

Unless you get both broadband and phone access from the same provider, that suggests that the problem is either at WUStL or in your computer. Unless the server operator can match your logs to the server logs, it'll be hard to find the cause.

[KhaleedWalten01]

Same problem here. Mostly WUs tried to be uploaded to the mentioned servers just receive a zero byte response.

Also my comp has been directly hooked to the modem to no avail, Kaspersky has been shut down entirely, also to no avail.
Seems that these servers really appear to have a problem. the uploaded WUs are mostly 50 MB in size.

[gunnarre]

I wonder if you have the same ISP? Can you share your IP numbers with the server operators without posting them publicly, so that they can check the logs?

[toTOW]

Billy.Oneal is in the USA and KhaleedWalten01 is in Switzerland ...

[aetch]

Also my comp has been directly hooked to the modem to no avail, Kaspersky has been shut down entirely, also to no avail.

I believe Kaspersky is not completely shutdown until it has been removed.

Every anti-virus has an exclusions/exceptions list.
Have you added the folders "C:\ProgramData\FAHClient" and "C:\Program Files (x86)\FAHClient" to it?
For Windows - Start -> Settings -> Update & Security -> Windows Security -> Virus & Threat protection -> Manage settings (Virus & threat protection settings) -> Add or remove exclusions (Exclusions)
I don't know where to locate it in Kaspersky but it should be there somewhere.

FAH recommends adding the work folder, I add the program folder as well to make sure.

Windows requirements (see note at bottom of page) - https://foldingathome.org/support/faq/i ... ts/?lng=en
MAC requirements (see note) - https://foldingathome.org/support/faq/i ... ts/?lng=en

[KhaleedWalten01]

Added the entire FAH-Client directory to the exclusion list in Kaspersky -> Still stuck.

Interestingly the CPU-crunched WUs get uploaded without any problems.

I have another WU for the GPU mapping to another server now, let's see if that one also gets stalled and not accepted when completing...

I'll keep you updated later on as the WU still needs about 1h to complete.

Right now it looks like this:


Note how only the two WUs from PRCG_17257 appear to be stalled while no other WU is in this state.

[toTOW]

Added the entire FAH-Client directory to the exclusion list in Kaspersky -> Still stuck.

Did you check the option I mentioned in this post : viewtopic.php?p=354855#p354855 ?

[KhaleedWalten01]

Okay, after fiddling around with the settings I found Kaspersky Internet Security to be extremely invasive when it comes to data traffic.

For all those who also have stuck WUs and get the zero-byte response here's where you have to tell Kaspersky not to mess around with your data:

1. Like toTow already mentioned, check that you add FAHClient to your Program Exclusions. (will write in german here as I don't have the english KIS)
- Open your KIS and click the gear icon at the lower left
- Click on "Netzwerkeinstellungen" on the left, then on the main screen seek for "Vertrauenswürdige Programme" and click it.
- Add FAHClient.exe and FAHControl.exe to this list and check every item in the following screen. (2 rules)
- Note the arrow to the right at the "Verschlüsselten Datenverkehr nicht untersuchen" and click it to change this entry to "Gesamten Datenverkehr nicht untersuchen"
- Name the rule to your needs
- Make surem the status is "Aktiv"
- Click on OK to add the program to the exclusion list
- Okay, this should help the Applications not to be stopped to freely transfer data to and from the web.

2. Add the necessary servers of FAH to the list of "Vertrauenswürdige Adressen"
- I added the following addresse to that list:

• orkney.seas.wustl.edu
  brodgar.seas.wustl.edu
  highland1.engr.wustl.edu
  highland2.engr.wustl.edu
  highland3.engr.wustl.edu
  highland4.engr.wustl.edu
  islay.seas.wustl.edu

3. Add the following program folders to be entirely left alone from Kaspersky
- To do so, click on the left menu on "Gefahren und Ausnahmen"
- Click on "Ausnahmen anpassen" and add the following folders: (2 rules)

• <Path to your FAHClient>\FAHClient\
  <Path to ProgramData>\FAHClient\

- Type "*" into the "Objekt" textbox if not present
- Type "*" into the "Datei-Hash" textbox if not present
- Make sure to activate "Alle Schutzkomponenten"
- Name the rule to your needs
- Make sure "Aktiv" is marked and add it by pressing the OK button for each of the two exclusions
- Add that newly created rule using the OK button

After that, Kaspersky asks you to save your changes (don't forget to do that!) and after that, F@H should run smoothly.
For now, all my packages get transferred without flaws now.

I hope this helps you as it helped me.
For those having issues with FritzBox or other routers, maybe there's someone who could provide a step-by-step guide on how to allow certain traffic on FritzBox routers

[toTOW]

I'm not sure part 2 and 3 are mandatory, but the more the merrier

And yes, Kaspersky and all those tools that follow the fashion to mess with internet traffic are worse than what they're supposed to protect us from ...