Project 16525 not uploading

[TonyStewart14]

I have a WU that is not uploading despite other WUs uploading and downloading fine for my GPU. I get the following error for both the work and collection servers:

Code: Select all

15:12:10:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:16525 run:26 clone:4 gen:13 core:0x23 unit:0x0d000000040000001a0000008d400000
15:12:10:WU01:FS01:Uploading 83.12MiB to 128.104.69.82
15:12:10:WU01:FS01:Connecting to 128.104.69.82:8080
15:12:33:WU00:FS01:0x22:Completed 300000 out of 2000000 steps (15%)
15:12:33:WU00:FS01:0x22:Checkpoint completed at step 300000
15:12:40:WARNING:WU01:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
15:12:40:WU01:FS01:Trying to send results to collection server
15:12:40:WU01:FS01:Uploading 83.12MiB to 158.130.118.25
15:12:40:WU01:FS01:Connecting to 158.130.118.25:8080
15:13:10:ERROR:WU01:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0

If needed, system info is below:

Code: Select all

*********************** Log Started 2024-10-26T19:18:23Z ***********************
19:18:23:******************************* libFAH ********************************
19:18:23:           Date: Oct 20 2020
19:18:23:           Time: 13:36:55
19:18:23:       Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
19:18:23:         Branch: master
19:18:23:       Compiler: Visual C++ 2015
19:18:23:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:18:23:       Platform: win32 10
19:18:23:           Bits: 32
19:18:23:           Mode: Release
19:18:23:****************************** FAHClient ******************************
19:18:23:        Version: 7.6.21
19:18:23:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:18:23:      Copyright: 2020 foldingathome.org
19:18:23:       Homepage: https://foldingathome.org/
19:18:23:           Date: Oct 20 2020
19:18:23:           Time: 13:41:04
19:18:23:       Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
19:18:23:         Branch: master
19:18:23:       Compiler: Visual C++ 2015
19:18:23:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:18:23:       Platform: win32 10
19:18:23:           Bits: 32
19:18:23:           Mode: Release
19:18:23:           Args: --open-web-control
19:18:23:         Config: C:\ProgramData\FAHClient\config.xml
19:18:23:******************************** CBang ********************************
19:18:23:           Date: Oct 20 2020
19:18:23:           Time: 11:36:18
19:18:23:       Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
19:18:23:         Branch: master
19:18:23:       Compiler: Visual C++ 2015
19:18:23:        Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
19:18:23:       Platform: win32 10
19:18:23:           Bits: 32
19:18:23:           Mode: Release
19:18:23:******************************* System ********************************
19:18:23:            CPU: AMD Ryzen 9 7950X3D 16-Core Processor
19:18:23:         CPU ID: AuthenticAMD Family 25 Model 97 Stepping 2
19:18:23:           CPUs: 32
19:18:23:         Memory: 31.21GiB
19:18:23:    Free Memory: 19.14GiB
19:18:23:        Threads: WINDOWS_THREADS
19:18:23:     OS Version: 6.2
19:18:23:    Has Battery: false
19:18:23:     On Battery: false
19:18:23:     UTC Offset: -5
19:18:23:            PID: 17404
19:18:23:            CWD: C:\ProgramData\FAHClient
19:18:23:  Win32 Service: false
19:18:23:             OS: Windows 10 Enterprise
19:18:23:        OS Arch: AMD64
19:18:23:           GPUs: 2
19:18:23:          GPU 0: Bus:18 Slot:0 Func:0 AMD:2 Raphael [Ryzen 7000 Series iGPU]
19:18:23:          GPU 1: Bus:1 Slot:0 Func:0 NVIDIA:8 GA104 [GeForce RTX 3070 Ti]
19:18:23:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:8.6 Driver:12.6
19:18:23:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:3.0 Driver:560.94
19:18:23:OpenCL Device 1: Platform:1 Device:0 Bus:18 Slot:0 Compute:1.2 Driver:3608.0

Any ideas on what could be causing this WU upload to fail? I have tried disabling antivirus and it is still not working. I use Bitdefender and turned off both firewall and Bitdefender Shield. I also tried rebooting.

[Joe_H]

I am not seeing anything obvious to explain this. Both the WS and CS indicated in the log are up and responding from what I can see. Since you can upload and download WUs to other servers that makes some possible causes less likely. If you were posting from someplace outside the US that cold be an issue with some servers, but the IP you posted from indicates you are in the US. It is possible a router between you and these two servers is dropping packets under some conditions. Otherwise the symptoms look just like what we have seen in the past for a firewall blocking connections using HTTP over ports 80 or 8080 and not using a "known browser".

[TonyStewart14]

Thank you Joe for the response. I am on Ethernet and haven't had the same issue with other WUs, so packet drop shouldn't be an issue. I also followed Bitdefender's steps to disable AV completely since there isn't one button to completely turn everything off, but I still had the same issue.

[TonyStewart14]

I have a second WU giving the same error message (156,4,4 in addition to 26, 4, 13). I have been able to finish WUs on other projects just fine with some of the same servers, so I know that's not the issue.

[sirray]

I also had the same problem uploading projects 16525 and 16781 on a computer running BitDefender. The logs below was from when I had both WUs completed but neither one could be uploaded. So the two uploads were happening at the same time and the log entries were a little co-mingled.

15:30:33:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:16781 run:12 clone:4 gen:47 core:0x23 unit:0x2f000000040000000c0000008d410000
15:30:33:WU02:FS01:Uploading 65.43MiB to 128.104.69.82
15:30:33:ERROR:WU00:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
15:30:33:WU02:FS01:Connecting to 128.104.69.82:8080
15:30:33:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:16525 run:208 clone:2 gen:19 core:0x23 unit:0x1300000002000000d00000008d400000
15:30:33:WU00:FS01:Uploading 83.21MiB to 128.104.69.82
15:30:33:WU00:FS01:Connecting to 128.104.69.82:8080
15:31:04:WARNING:WU02:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0
15:31:04:WU02:FS01:Trying to send results to collection server
15:31:04:WU02:FS01:Uploading 65.43MiB to 128.174.73.78
15:31:04:WU02:FS01:Connecting to 128.174.73.78:8080
15:31:04:WARNING:WU00:FS01:Exception: Failed to send results to work server: 10002: Received short response, expected 512 bytes, got 0


However, the same computer was able to upload 12299 without a problem.
20:02:42:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:12299 run:33 clone:47 gen:159 core:0x23 unit:0x0000002f0000009f0000300b00000021
20:02:42:WU01:FS01:Uploading 22.86MiB to 206.223.170.146
20:02:42:WU01:FS01:Connecting to 206.223.170.146:8080
20:02:58:WU01:FS01:Upload complete
20:02:58:WU01:FS01:Server responded WORK_ACK (400)
20:02:58:WU01:FS01:Final credit estimate, 386053.00 points
20:02:58:WU01:FS01:Cleaning up


After much troubleshooting, I concluded that the problem is with my computer and not my network firewall. Firewall logs told me the TCP connection ended because client (my computer) had reset the connection rather than the normal TCP FIN, which means my computer didn't like the way the upload conversation was happening and unilaterally terminated the conversation. So it must have been caused by some settings or a piece of software on my computer. I tried disabling everything (yes everything not just anti virus) in BitDefender and it didn't make a difference. Finally I completely uninstalled BitDefender and voila both WUs uploaded immediately! There is no log entry in BitDefender that tells me what F@H did that made it mad.

Mystery is still unresolved. I'm still experimenting.

[Joe_H]

I also had the same problem uploading projects 16525 and 16781 on a computer running BitDefender. The logs below was from when I had both WUs completed but neither one could be uploaded. So the two uploads were happening at the same time and the log entries were a little co-mingled.

The symptoms do sound very like somewhere in BitDefender stateful packet filtering had been enabled on HTTP traffic. A common default filter is to deny HTTP connections when they are not to and from "known" browsers such as Chrome, Edge, Firefox, Safari and so on. When that happens an exception needs to be added allowing in this case FAHClient to do download and upload connections over ports 80 and 8080

[sirray]

@Joe_H...

Thanks for responding. Same computer just completed a CPU job and successfully uploaded the WU with BitDefender fully enabled. I'm running a GPU job on it now.

I hear your theory of BitDefender blocking the traffic due to fahclient not being a known browser but I don't think that was the case at least in my situation. The only place where BitDefender requires a browser to be specifically configured is the Privacy > Anti-tracker function, which installs a browser plug-in to your browser to request the web site you visit to not track you. Most browsers have that function already anyway. I don't have it enabled. I also made sure fahclient.exe is added to the known applications list and any outbounc TCP connections are allowed. Besides, I'm a network engineer and I routinely use non-browsers (curl, wget, telnet to port 80) to access web sites on my BitDefender computer without a problem.

Once the GPU job is done running, if it fails to upload, I will run a pcap and see if I can figure out something from the pcap.

[sirray]

Ok experiment done. Here are the observations:

1, The problem is spec

• • • ific to some WUs. Same computer with BitDefender fully enabled is able to turn in WUs successfully most of the time. But WUs 16525 and 16781 are the two that I am constantly having this problem with.
      
      2, The problem follows WUs and not work servers. Same WU different work server didn't make a difference. Still failed.
      
      3, Completely disabling BitDefender didn't help. Followed these instructions: https://www.bitdefender.com/consumer/su ... wer/28557/
      
      4, Only a full removal of

BitDefender made a difference.


When it didn't work, PCAP showed:
1, The completed WU was uploaded.
2, Work server responded with 200 HTTP_OK with TCP (FIN,ACK)
3, My computer sent back a TCP ACK, which was expected
4, Then without waiting for the server's response, my computer sent a TCP (RST,ACK) to the work server to end the connection abnormally. This was not expected.


When it worked (after I uninstalled BitDefender), PCAP showed:
1, The completed WU was uploaded.
2, Work server responded with 200 HTTP_OK with TCP (FIN,ACK)
3, My computer sent back a TCP ACK, which was expected
4, My computer then sent a FIN,ACK to the work server to gracefully end the TCP connection. This was an expected outcome given the server's request to end the connection in step 2 above.
5, Finally the work server responds with an ACK to my computer's FIN and it was all done.

It seems that something in BitDefender didn't like how these WUs were uploaded. And that something could not be disabled without a complete uninstall. I guess a workaround is to force small units with the max-packet-size=small setting. Does that work on 7.x clients?

[TonyStewart14]

Thanks sirray, I also tried steps 1-3 to no avail. I added max-packet-size=small to my config on 7.x and will run it to see if any other results get stuck. If so, I'll simply migrate away from BD once my subscription expires and probably just use Windows Defender... unless you recommend something else as a network engineer!

[toTOW]

When it didn't work, PCAP showed:
1, The completed WU was uploaded.
2, Work server responded with 200 HTTP_OK with TCP (FIN,ACK)
3, My computer sent back a TCP ACK, which was expected
4, Then without waiting for the server's response, my computer sent a TCP (RST,ACK) to the work server to end the connection abnormally. This was not expected.


When it worked (after I uninstalled BitDefender), PCAP showed:
1, The completed WU was uploaded.
2, Work server responded with 200 HTTP_OK with TCP (FIN,ACK)
3, My computer sent back a TCP ACK, which was expected
4, My computer then sent a FIN,ACK to the work server to gracefully end the TCP connection. This was an expected outcome given the server's request to end the connection in step 2 above.
5, Finally the work server responds with an ACK to my computer's FIN and it was all done.

It seems that something in BitDefender didn't like how these WUs were uploaded. And that something could not be disabled without a complete uninstall. I guess a workaround is to force small units with the max-packet-size=small setting. Does that work on 7.x clients?

We've seen this behaviour before on other servers, and network tweaks have been applied.

I'll try to get them ported to this one too ...

Note : antivirus, firewalls or routers with DPI features tends to interfere with FAH uploads and have to be disabled.

[sirray]

Thanks sirray, I also tried steps 1-3 to no avail. I added max-packet-size=small to my config on 7.x and will run it to see if any other results get stuck. If so, I'll simply migrate away from BD once my subscription expires and probably just use Windows Defender... unless you recommend something else as a network engineer!

@TonyStewart14 - IMHO, unless you intend to make granular tweak specific settings in your endpoint protection software (firewall, anti-virus, etc), Windows Defender is good enough. And the need to make granular tweaks are usually driven by how you use your use your computer. Unless you work in IT and have to use software whose behavior is similar to malware, you won't need to make these granular tweaks.

[Joe_H]

Thanks sirray, I also tried steps 1-3 to no avail. I added max-packet-size=small to my config on 7.x and will run it to see if any other results get stuck. If so, I'll simply migrate away from BD once my subscription expires and probably just use Windows Defender... unless you recommend something else as a network engineer!

You mentioned adding max-packet-size=small to your client. That does not affect the network packet size, but is a holdover from earlier version clients that included the setting to control the maximum size of WU uploads. This was for slower connections such as dial up modems or DSL. The max-packet-size=small option is supposed to indicate to the servers that WUs smaller than 10 MB for an upload are being requested as I recall. But in the last few years I think they have stopped checking that setting when assigning WUs.

[TonyStewart14]

IMHO, unless you intend to make granular tweak specific settings in your endpoint protection software (firewall, anti-virus, etc), Windows Defender is good enough. And the need to make granular tweaks are usually driven by how you use your use your computer. Unless you work in IT and have to use software whose behavior is similar to malware, you won't need to make these granular tweaks.

I ended up doing just that - I uninstalled BD and switched to Windows Defender. I also linked to your earlier post where you detail PCAP observations and how only an uninstall solved the problem.

You mentioned adding max-packet-size=small to your client. That does not affect the network packet size, but is a holdover from earlier version clients that included the setting to control the maximum size of WU uploads. This was for slower connections such as dial up modems or DSL. The max-packet-size=small option is supposed to indicate to the servers that WUs smaller than 10 MB for an upload are being requested as I recall. But in the last few years I think they have stopped checking that setting when assigning WUs.

Good to know - I was also apparently late to learn that the -advanced flag had been deprecated. Luckily, packet size is irrelevant now as I have removed BD.