Fairly frequent timeouts downloading workunits

[BOB850123]

I've recently been noticing that I fairly often have these timeouts attempting to download work units. This is new within the last week or so. Most units do succeed, and do so immediately. I've observed this across multiple systems, so feels like something on the server side or somewhere in between. Mostly just curious if this is a known issue or something others are encountering as well.

Code: Select all

08:56:49:I1:WU1660:Requesting WU assignment for user BOB850123 team 37726
08:56:49:I1:WU1660:Received WU assignment yl5AmVs9VnlDnz2fXc2HKytfl1RkbYXPm3VcBxCLww8
08:56:49:I1:WU1660:Downloading WU
08:56:51:I1:WU1660:Retry #1 in 2 secs
08:56:53:I1:WU1660:Downloading WU
08:56:56:I1:WU1660:Retry #2 in 4 secs
08:57:00:I1:WU1660:Downloading WU
08:57:02:I1:WU1660:Retry #3 in 8 secs
08:57:10:I1:WU1660:Downloading WU
08:57:12:I1:WU1660:Retry #4 in 16 secs
08:57:28:I1:WU1660:Downloading WU
08:57:30:I1:WU1660:Retry #5 in 32 secs
08:58:02:I1:WU1660:Downloading WU
08:58:05:I1:WU1660:Retry #6 in 1 min 4 secs
08:59:09:I1:WU1660:Downloading WU
08:59:11:I1:WU1660:Retry #7 in 2 mins 8 secs
09:01:19:I1:WU1660:Downloading WU
09:01:22:I1:WU1660:Retry #8 in 4 mins 16 secs
09:05:38:I1:WU1660:Downloading WU
09:05:40:I1:WU1660:Retry #9 in 8 mins 32 secs
09:14:12:I1:WU1660:Downloading WU
09:14:15:I1:WU1660:Too many retries (9), failing WU

[muziqaz]

Yes, everyone is encountering them. Dev is aware of this

[Wedge009]

Thanks for the confirmation. I was also wondering what was going on when I noticed 'warnings' against hosts in the GUI. I've noticed them in the last few days, presumably due to downloads ultimately being given up on. I understand the time-sensitive requirements of F@h meaning that only one WU at a time is distributed, but it's times like these that I appreciate the queued-buffer approach of BOINC infrastructure.

[Wedge009]

Do we yet have an idea on what is causing this?

[muziqaz]

No

[BobWilliams757]

I've noticed the warnings when it was struggling to pick up work with repeated retries. Other than the warning that shows up, it keeps cycling with no ill impact that I could find, so I just ignore the warning.... beyond it making me notice that it's having issues.

[Wedge009]

I hadn't seen this issue for a while but it looks like I've got it again today, 2026-05-29 @ ~04:00 UTC. The usual `Failed response: EOF` message.

[Wedge009]

It's getting really bad again. Still no progress on a resolution or even identification of the problem?

[toTOW]

Since the client doesn't print the server address, we don't know which one is failing ...

If someone could catch this address while it is happening, it would help us.

It would also help to know what kind of hardware was requesting work.

[Wedge009]

In my case it's CPU work for Zen 2 & 3 hardware. Mix of 4 and 12 core counts. I have an old Skylake-X CPU but I don't recall that one having download issues. Presumably GPU work requests aren't affected or I imagine there would be a lot more complaints.

It's settled down since I last reported issues a couple of days ago.

[Wedge009]

If someone could catch this address while it is happening, it would help us.

This is something that needs to be captured on the server, isn't it? I tried having a look in the logs but there really isn't anything other than work requested followed by download failing repeatedly.

[Wedge009]

The lack of server information is logged in https://github.com/FoldingAtHome/fah-cl ... issues/396, with this issue with downloads also being reported in https://github.com/FoldingAtHome/fah-cl ... issues/443.

[muziqaz]

If someone could catch this address while it is happening, it would help us.

This is something that needs to be captured on the server, isn't it? I tried having a look in the logs but there really isn't anything other than work requested followed by download failing repeatedly.

No, it is possible to trace the connection to a server address with certain utilities in windows and of course in Linux