WU Oddities

[Juggy]

For the last 3-4 days I have struggled to get WU's at times.

For instance right now the client went through the cycle of looking for a WU, the 6th or 7th attempt, it didn't get one and 30 seconds later I rebooted my machine. Immediately after reboot it got a WU

Now this has happened 10+ times over the last few days. I watch the cycle of looking for a WU go by, nothing. Reboot the machine and immediately it gets a WU.

Why is this?

The issue I have is I have seen hourly cycles go by without work which is wasted cycles. Only after human intervention does it actually pick up work.

What must be noted is I always get CPU WU's

[ajm]

The client is sending requests for new WUs at increasing intervals. The first is sent when the present WU is at 99%, then maybe 1 minute later, then 2, 5, 10 minutes, and so on. This prevents the servers from being hammered by requests when they are overloaded.
When you reboot the computer, the request interval is reset and you thus have more chances to get a WU. You can get the same result by pausing the software or just the GPU slot (right clic in Advanced Control). But please don't overdo that. The shortage of those PGU WUs is affecting everybody at the moment... A good rule I think would be to wait until the interval is in the hours. It gives you better chances without overwhelming the servers.

[Juggy]

The client is sending requests for new WUs at increasing intervals. The first is sent when the present WU is at 99%, then maybe 1 minute later, then 2, 5, 10 minutes, and so on. This prevents the servers from being hammered by requests when they are overloaded.
When you reboot the computer, the request interval is reset and you thus have more chances to get a WU. You can get the same result by pausing the software or just the GPU slot (right clic in Advanced Control). But please don't overdo that. The shortage of those PGU WUs is affecting everybody at the moment... A good rule I think would be to wait until the interval is in the hours. It gives you better chances without overwhelming the servers.

Thanks, I understand that but as mentioned I reboot the machine immediately after a failed attempt at getting a WU for the 7th or 8th time. After reboot I get a WU immediately. Does the client get assigned to a new assignment server after a full client restart or machine reboot?

Another thing to be noted is that a PAUSE->FOLD does not get a new WU, only a reboot helps.

[ajm]

I just tried it on one of my idle GPUs. After the pause/fold, the interval is reset to one minute, then 2...
It's indeed not as "good" as a complete reboot, but it takes less time and doesn't disturb the other WUs.

You can easily follow this process in the Advanced Control:

[Joe_H]

Your client always tries assign1, 65.254.110.245, and then assign2, 18.218.241.186, in that order. Doesn't matter whether after a retry time has been reached, you pause and restart the folding slot, or you reboot. All the second two methods do is reset the retry time back to 0. By chance each time you ave managed to get a WS after rebooting, but that is no more likely than the other requests.

At the moment, there are relatively plenty of CPU jobs, the GPU projects are waiting on new WUs to be created. Partly his is related to full servers, they have to migrate off some of the data to create more space.

P.S. You need to wait about 30-60 seconds after selecting Pause for the client to update its settings before resuming.

[Juggy]

I just tried it on one of my idle GPUs. After the pause/fold, the interval is reset to one minute, then 2...
It's indeed not as "good" as a complete reboot, but it takes less time and doesn't disturb the other WUs.

You can easily follow this process in the Advanced Control:

Tried PAUSE-FOLD, doesn't work. The retry process just continues after shortening the time and doesn't find a WU.

PS, I always only use advanced controls.

Your client always tries assign1, 65.254.110.245, and then assign2, 18.218.241.186, in that order. Doesn't matter whether after a retry time has been reached, you pause and restart the folding slot, or you reboot. All the second two methods do is reset the retry time back to 0. By chance each time you ave managed to get a WS after rebooting, but that is no more likely than the other requests.

At the moment, there are relatively plenty of CPU jobs, the GPU projects are waiting on new WUs to be created. Partly his is related to full servers, they have to migrate off some of the data to create more space.

P.S. You need to wait about 30-60 seconds after selecting Pause for the client to update its settings before resuming.

Tried waiting 5 minutes before hitting FOLD again also after PAUSE-FOLD for another 3 or 4 cycles and I don't get a WU. I get one immediately after a reboot. My reboot happens in 15-25 seconds due to having an NVMe SSD

Seems odd, like there is a check and balance missing somewhere

[ajm]

OK, I tried to reboot one of my two machines. I got a GPU WU a few minutes later... but on the other one, the one that didn't reboot.
So it seems that Joe_H is right and that you got lucky with those fast reboots.

[Juggy]

Your client always tries assign1, 65.254.110.245, and then assign2, 18.218.241.186, in that order. Doesn't matter whether after a retry time has been reached, you pause and restart the folding slot, or you reboot. All the second two methods do is reset the retry time back to 0. By chance each time you ave managed to get a WS after rebooting, but that is no more likely than the other requests.

At the moment, there are relatively plenty of CPU jobs, the GPU projects are waiting on new WUs to be created. Partly his is related to full servers, they have to migrate off some of the data to create more space.

P.S. You need to wait about 30-60 seconds after selecting Pause for the client to update its settings before resuming.

Thanks again for the explanation. At one point it looked like I could re-create the behaviour however this morning a reboot doesn't help.

Do you know exactly what the retry intervals are in minutes? This morning it was on 2 hour 36 minutes to next retry. Seems a little long considering how fast modern hardware can get through a WU.

[PantherX]

...Do you know exactly what the retry intervals are in minutes? This morning it was on 2 hour 36 minutes to next retry. Seems a little long considering how fast modern hardware can get through a WU.

The retry attempts are based on an exponential back-off timer. This was done since historically, the reason why a Server wasn't functioning was because it required TLC (boots on ground to replace a hardware to other maintenance tasks). Since it takes a while, it was the best option. The current issue is friendly DDOS which wasn't expected (but glad to have it) and we are ramping up so the situation is a lot better now and will continue to improve as things progress down the pipeline.

[Juggy]

...Do you know exactly what the retry intervals are in minutes? This morning it was on 2 hour 36 minutes to next retry. Seems a little long considering how fast modern hardware can get through a WU.

The retry attempts are based on an exponential back-off timer. This was done since historically, the reason why a Server wasn't functioning was because it required TLC (boots on ground to replace a hardware to other maintenance tasks). Since it takes a while, it was the best option. The current issue is friendly DDOS which wasn't expected (but glad to have it) and we are ramping up so the situation is a lot better now and will continue to improve as things progress down the pipeline.

Thanks, so essentially I could be waiting days for a new WU if the back off timer goes that far?

Would it not make sense for the back off timer to reset itself after 10 or 15 tries.

[bruce]

There is a new version of the client which has been in development for what seems like a very long time. I can't predict when it will be ready to release. I do know that some changes to how the back-off timer works will be in that client.

[PantherX]

...Thanks, so essentially I could be waiting days for a new WU if the back off timer goes that far?

Would it not make sense for the back off timer to reset itself after 10 or 15 tries.

I would say that when the timers hits a few hours (generally it is after 10 or 12), you can pause/unpause the slot and the timer will reset.

However, as bruce mentioned, once a new version of the client is released, it may handle this situation too.

[Juggy]

There is a new version of the client which has been in development for what seems like a very long time. I can't predict when it will be ready to release. I do know that some changes to how the back-off timer works will be in that client.

...Thanks, so essentially I could be waiting days for a new WU if the back off timer goes that far?

Would it not make sense for the back off timer to reset itself after 10 or 15 tries.

I would say that when the timers hits a few hours (generally it is after 10 or 12), you can pause/unpause the slot and the timer will reset.

However, as bruce mentioned, once a new version of the client is released, it may handle this situation too.

Thanks for the information, it would be nice if it sorted itself out quicker without human intervention. It would be far more productive.