503 "Please wait"

[arisu]

Unlike the 503 "No appropriate assignment" error that is given by the assignment server when no work is available, this is given by the work server when an assignment is posted to it after a WU has already been found. However the v8 client will treat it as an unrecoverable error and will abandon the assignment altogether, which is probably not the right approach (but that's another topic).

What causes this? How can it be avoided? The server this happened with was pllwskifah1.mskcc.org/api/assign. The assignment given by the AS was 17658. The request made by my client was for WU 173 requested at 2025-05-05T10:15:27Z.

This is the log in question, note how an assignment is already found and issued, and how it's the work server that gives the error, not the assignment server:

Code: Select all

10:15:27:I1:Unit:WU173:Requesting WU assignment for user Grimoire_of_Lolice team 230362
10:15:27:I1:Request:OUT10:> CONNECT assign3.foldingathome.org:443 HTTP/1.1
10:15:28:I1:Request:OUT10:> POST https://assign3.foldingathome.org/api/assign HTTP/1.1
10:15:29:I1:Request:OUT10:< HTTP/1.1 200 HTTP_OK
10:15:29:I1:Unit:WU173:Received WU assignment ol61ZiTrElXXJTiDZiAhXBN5VDqx812eKOzb8_Ux4F4
10:15:29:I1:Unit:WU173:Downloading WU
10:15:29:I1:Request:OUT11:> CONNECT pllwskifah1.mskcc.org:443 HTTP/1.1
10:15:40:I1:Request:OUT11:> POST https://pllwskifah1.mskcc.org/api/assign HTTP/1.1
10:15:42:I1:Unit:WU173:DOWNLOAD 100% 46B of 46B
10:15:42:I1:Request:OUT11:< HTTP/1.1 503 HTTP_SERVICE_UNAVAILABLE
10:15:42:E :Request:OUT11:HTTP_SERVICE_UNAVAILABLE: {"error":{"message":"Please wait","code":503}}
10:15:42:I1:Unit:WU173:Retry #9 in 8 mins 32 secs

Compare that with the usual (and expected) error when no assignment can be found, which is returned by the assignment server rather than the work server:

Code: Select all

10:11:06:I1:Unit:WU173:Requesting WU assignment for user Grimoire_of_Lolice team 230362
10:11:06:I1:Request:OUT9:> CONNECT assign2.foldingathome.org:443 HTTP/1.1
10:11:07:I1:Request:OUT9:> POST https://assign2.foldingathome.org/api/assign HTTP/1.1
10:11:11:I1:Request:OUT9:< HTTP/1.1 503 HTTP_SERVICE_UNAVAILABLE
10:11:11:E :Request:OUT9:HTTP_SERVICE_UNAVAILABLE: {"error":"No appropriate assignment"}
10:11:11:I1:Unit:WU173:Retry #8 in 4 mins 16 secs

[toTOW]

"Please wait" indicates that the work server is busy and can't handle the request. This can happen when it's out of disk space or when it is busy generating new work units.

"No appropriate assignment" indicates that the assignment server can't find a work server that has WUs available for your hardware.

[arisu]

I knew what no appropriate assignment meant, just not please wait.

So does that mean that the client should request again in a few seconds instead of giving up and asking for an entirely new assignment? The servers it happens on are not out of disk space so I guess they are just busy generating other WUs. It's strange that it would give an HTTP status error instead of slightly delaying the download for a few seconds.

[CseppentoArpad]

Excuse me!

My English is very poor.

I thinking why my machine wait long-long hours for a work unit assigment....

01:29:31:I1:WU26:Uploading WU results
01:29:32:I1:WU27:Requesting WU assignment for user CseppentoArpad team 679
01:29:32:I1:OUT77:> POST https://assign1.foldingathome.org/api/assign HTTP/1.1
01:29:32:I1:OUT76:> POST https://fahserver1.flatironinstitute.org/api/results HTTP/1.1
01:29:33:I1:OUT77:< HTTP/1.1 503 HTTP_SERVICE_UNAVAILABLE
01:29:33:E :OUT77:HTTP_SERVICE_UNAVAILABLE: {"error":"No appropriate assignment"}
01:29:33:I1:WU27:Retry #1 in 2 secs
01:29:34:I1:WU26:UPLOAD 34% 9.60MiB of 28.12MiB
01:29:35:I1:WU26:UPLOAD 76% 21.52MiB of 28.12MiB
01:29:35:I1:WU27:Requesting WU assignment for user CseppentoArpad team 679
01:29:35:I1:OUT78:> POST https://assign2.foldingathome.org/api/assign HTTP/1.1
01:29:36:I1:OUT78:< HTTP/1.1 503 HTTP_SERVICE_UNAVAILABLE
01:29:36:E :OUT78:HTTP_SERVICE_UNAVAILABLE: {"error":"No appropriate assignment"}


And only wait and wait for new work unit....

Please help me! Why is this?

What can I do, to resolve this situation?

Please Moderators or technician leaders help for minimizing the waste time!

Thank you, from Europe,Hungary,Budapest

[toTOW]

So does that mean that the client should request again in a few seconds instead of giving up and asking for an entirely new assignment?

Yes, it was working perfectly fine with v7 client. It would just retry later.

[Joe_H]

Excuse me!

My English is very poor.

I thinking why my machine wait long-long hours for a work unit assigment....

01:29:31:I1:WU26:Uploading WU results
01:29:32:I1:WU27:Requesting WU assignment for user CseppentoArpad team 679
01:29:32:I1:OUT77:> POST https://assign1.foldingathome.org/api/assign HTTP/1.1
01:29:32:I1:OUT76:> POST https://fahserver1.flatironinstitute.org/api/results HTTP/1.1
01:29:33:I1:OUT77:< HTTP/1.1 503 HTTP_SERVICE_UNAVAILABLE
01:29:33:E :OUT77:HTTP_SERVICE_UNAVAILABLE: {"error":"No appropriate assignment"}
01:29:33:I1:WU27:Retry #1 in 2 secs
01:29:34:I1:WU26:UPLOAD 34% 9.60MiB of 28.12MiB
01:29:35:I1:WU26:UPLOAD 76% 21.52MiB of 28.12MiB
01:29:35:I1:WU27:Requesting WU assignment for user CseppentoArpad team 679
01:29:35:I1:OUT78:> POST https://assign2.foldingathome.org/api/assign HTTP/1.1
01:29:36:I1:OUT78:< HTTP/1.1 503 HTTP_SERVICE_UNAVAILABLE
01:29:36:E :OUT78:HTTP_SERVICE_UNAVAILABLE: {"error":"No appropriate assignment"}


And only wait and wait for new work unit....

Please help me! Why is this?

What can I do, to resolve this situation?

Please Moderators or technician leaders help for minimizing the waste time!

Thank you, from Europe,Hungary,Budapest

You have run into a different "503" message, "No appropriate assignment". That indicates the server does not have any WUs suitable for the hardware or configuration settings you have used. You could have an older GPU for which work is not as available as it once was. Or recently to work around a bug, most servers with CPU WUs have a thread count minimum of 3.

[arisu]

So does that mean that the client should request again in a few seconds instead of giving up and asking for an entirely new assignment?

Yes, it was working perfectly fine with v7 client. It would just retry later.

How long would it wait before retrying the WS? I'll patch my local copy of the client to get it to retry up to 3 times.

[toTOW]

As far as I remember, v7 client would retry indefinitely with increasing delay (capped to 1 hour) between attempts.

[arisu]

Even for a "Please wait"? It wouldn't just try to find a new assignment after a certain number of failures? Surely there is a point where the work server gives the WU to someone else.

What happens is that the assignment server finds a WU for me and gives my client the URL of the work server with that unit. The client contacts the work server which sends the wudata_01.dat but sometimes it responds with a 503 Please wait instead. If the v7 client retried indefinitely then I imagine it would get stuck in a loop once the work server no longer has the WU to give.

That's why I won't retry indefinitely with my patch. If the work server refuses the WU a few times in a row, it would probably be best to ask the assignment server for a new assignment.

[toTOW]

At some point, the AS should give up and assign something else ... unless you're unlucky and there's nothing else for this configuration.

[arisu]

Does the work server return a different error if the client sends it the assignment after the work server has already given it to someone else? Imagine the following sequence of events:

1. My client contacts the AS and is given an assignment $PRCG
2. My client contacts the WS asking for $PRCG and is told to wait
3. While waiting, the AS gives up and gives someone else $PRCG
4. That second person contacts the WS asking for $PRCG and receives it
5. My client asks the WS again for $PRCG

What error will it receive now, so that I can know when it should give up? If it just keeps asking the WS for the same old WU repeatedly and never goes back to the AS for a new assignment then it will get stuck.

[Joe_H]

No.

[arisu]

Got it. So it should give up on the WS and go back to the AS after a certain number of unsuccessful tries to avoid getting stuck. Thanks.