155.247.166.220 downloads stalled

[vvoelz]

Thanks rickoic -- high traffic remains an issue, so we might put some assignment filters on to see if that helps.

[rickoic]

Well got woke up early by thunderstorms moving through so checked and found only 1 gpu frozen on download at 3.65% and had sit there for 5 hours or so.

[vvoelz]

p.s. Our head software dev is "working on a rewrite of the WS networking code that should help a lot
with uploads/downloads." Fingers crossed.

[rickoic]

Guru/witchdoctor/medicine man. Wish him heap big magic.

[info2x]

I wish them luck as well.

After several days of not being assigned to this server I got an assignment and failed to download again.

[HaloJones]

just found one and had to reboot to clear

[rickoic]

Sat had 1 to reboot.
Sun had 2 to reboot
Mon had 2 to reboot
Plus through out the day Sun caught 3 more that I had to reboot.

Tks
Rick

[rickoic]

Appears Tech Guru had heap big magic. Been 3 days or more since I've had a problem with the sergver.

Tks for the good work.

[cfhdev]

The last few days this has been stalling again.

[PantherX]

I can access the landing page without issues. Thus, would chalk it up to peak traffic potentially saturating the bandwidth. If the situation worsens, please post the log file so we can investigate it further

[cfhdev]

I can to. I don't think the issue is just a bandwidth issue. The client should have a time out period where it will either dump the process and try again or switch to another server.

Example from log of WU02:FS02 from one client. Before and after the reboot. Normally I dump the WU from

Code: Select all

/var/lib/fahclient/work

this time just rebooted

Code: Select all

*********************** Log Started 2020-05-26T01:13:35Z ***********************
******************************* Date: 2020-05-26 *******************************
07:44:25:WU02:FS02:Connecting to assign1.foldingathome.org:80
07:44:25:WARNING:WU02:FS02:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration
07:44:25:WU02:FS02:Connecting to assign2.foldingathome.org:80
07:44:26:WU02:FS02:Assigned to work server 155.247.166.220
07:44:26:WU02:FS02:Requesting new work unit for slot 02: RUNNING gpu:1:********* from 155.247.166.220
07:44:26:WU02:FS02:Connecting to 155.247.166.220:8080
07:44:26:WU02:FS02:Downloading 12.96MiB
******************************* Date: 2020-05-26 *******************************
******************************* Date: 2020-05-26 *******************************
*********************** Log Started 2020-05-26T15:22:32Z ***********************
*********************** Log Started 2020-05-26T15:22:32Z ***********************
15:23:19:WU02:FS02:Connecting to assign1.foldingathome.org:80
15:23:24:WARNING:WU02:FS02:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration
15:23:24:WU02:FS02:Connecting to assign2.foldingathome.org:80
15:23:25:WU02:FS02:Assigned to work server 128.252.203.10
15:23:25:WU02:FS02:Requesting new work unit for slot 02: READY gpu:1:********* from 128.252.203.10
15:23:25:WU02:FS02:Connecting to 128.252.203.10:8080
15:25:34:WARNING:WU02:FS02:WorkServer connection failed on port 8080 trying 80
15:25:34:WU02:FS02:Connecting to 128.252.203.10:80
15:26:28:WU02:FS02:Downloading 86.23MiB
15:26:34:WU02:FS02:Download 20.58%
15:26:40:WU02:FS02:Download 36.38%
15:26:46:WU02:FS02:Download 45.23%
15:26:52:WU02:FS02:Download 52.69%
15:26:58:WU02:FS02:Download 61.32%
15:27:04:WU02:FS02:Download 68.71%
15:27:10:WU02:FS02:Download 77.84%
15:27:16:WU02:FS02:Download 89.22%
15:27:22:WU02:FS02:Download 97.77%
15:27:22:WU02:FS02:Download complete

It just stops/hangs in the middle of the download and never tries anything again until the reboot 8 hours later.

[PantherX]

When you encounter the network issue, there's no need to delete any files. Simply restart FAHClient. It's a known bug: https://github.com/FoldingAtHome/fah-issues/issues/983

[cfhdev]

Thanks for that bug page. Its interesting that it only happens for us on that one IP across multiple machines.

[Sparkly]

This issue is still occurring a lot on this Server/IP – 155.247.166.220 by the looks of it and as far as I can see from my setups, which are currently running 12 slots total, it happens several times a day for me alone, thus needing a reboot of the process, since there is no recovery and the slot hangs in Ready.

[bruce]

Yes, it happens on specific servers. They do get overloaded and data transfer rates become slower and slower ... and evenually it starts to hang which leads to the need to restart FAHClient. Posting topics like this one often leads to a correction on the server side. (except at night when we can't contact the appropriate people if they happen to have a life).

155.247.166.220 (vav4) reportedly had two CSs, one of which has failed. We reported a problem last night and it looks like somebody is working on it. (It was rebooted 44 minutes ago.)