Page 1 of 1
Shortage of GPU work again?
Posted: Thu Jul 02, 2020 11:54 am
by Juggy
I am struggling to get GPU work units, is there a shortage again? *.mskcc.org/
Re: Shortage of GPU work again?
Posted: Thu Jul 02, 2020 12:36 pm
by Joe_H
Maybe. JohnChodera did announce within the last few weeks that the work for GPUs would tend to come in waves as each round of projects for the COVID Moonshot was released. Each round would take a period of time to gather data, and then then next round would need to be prepared for release.
Also, checking the Server Status page -
https://apps.foldingathome.org/serverstats, some of the servers show as being low or out of work to assign. As they receive WUs back, the next Gen WU can be generated, but they appear to be going out about as fast as they come in.
Re: Shortage of GPU work again?
Posted: Thu Jul 02, 2020 12:52 pm
by ajm
Also 140.163.4.231 seems to be struggling somehow. I get only connection errors from it and it doesn't load:
http://plfah1-1.mskcc.org/
Re: Shortage of GPU work again?
Posted: Fri Jul 03, 2020 2:31 am
by vmzy
ajm wrote:Also 140.163.4.231 seems to be struggling somehow. I get only connection errors from it and it doesn't load:
http://plfah1-1.mskcc.org/
AS server keeping assign me to 140.163.4.231, but it mustbe down now.
Code: Select all
01:55:53:WU01:FS01:Connecting to assign1.foldingathome.org:80
01:55:54:WU01:FS01:Assigned to work server 140.163.4.231
01:55:55:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:GP108 [GeForce MX150 (GT 1030) Max-Q] from 140.163.4.231
01:55:55:WU01:FS01:Connecting to 140.163.4.231:8080
01:56:59:ERROR:WU01:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
Re: Shortage of GPU work again?
Posted: Fri Jul 03, 2020 4:01 am
by aetch
The servers are waiting on a load of big units coming back from slow computers.
Seriously, I've had a big unit running on my laptop for almost 4 hours now with an ETA of another 18 hours. I'd love to be able to take the unit and just drop it into my desktop where it could be done in 4-5 hours.
Check the servers for yourself.
https://apps.foldingathome.org/serverstats
Joe_H - the Public/Beta jobs on the server stats page, are these units currently running on peoples computers or in a queue waiting to be assigned?
Re: Shortage of GPU work again?
Posted: Fri Jul 03, 2020 4:49 am
by JohnChodera
Thanks for the heads up! We certainly don't have a WU shortage---there's over 1.5 million WUs ready to go.
We've found an issue with plfah1-1.mskcc.org and think we have it temporarily resolved, and are investigating the root cause.
We've also increased the max assign rate on aws3.foldingathome.org.
~ John Chodera // MSKCC
Re: Shortage of GPU work again?
Posted: Fri Jul 03, 2020 5:13 am
by Juggy
JohnChodera wrote:Thanks for the heads up! We certainly don't have a WU shortage---there's over 1.5 million WUs ready to go.
We've found an issue with plfah1-1.mskcc.org and think we have it temporarily resolved, and are investigating the root cause.
We've also increased the max assign rate on aws3.foldingathome.org.
~ John Chodera // MSKCC
Great, thank you John
Re: Shortage of GPU work again?
Posted: Wed Jul 15, 2020 9:25 pm
by bumbel123
Hi,
any update here? Found my GPUs in a row doing almost nothing, since 2 days, just occasional 17200, 117xx ... rare spots here and there.
If there's nothing to do anymore and you have all of your results, I stop FAH client on my gear ...
Cheers
Re: Shortage of GPU work again?
Posted: Fri Jul 17, 2020 6:43 pm
by markdotgooley
I seem to have my two RTX2060 cards busy almost constantly, though the PPD has gone from ridiculously high levels (2.7 to 2.9 million) to lower than usual (1.7 million), to nearer normal (somewhat over 2 million up to 2.5). Nowadays I just shrug it off and wait for better times...
Re: Shortage of GPU work again?
Posted: Sat Jul 18, 2020 3:40 am
by kiore
markdotgooley wrote:I seem to have my two RTX2060 cards busy almost constantly, though the PPD has gone from ridiculously high levels (2.7 to 2.9 million) to lower than usual (1.7 million), to nearer normal (somewhat over 2 million up to 2.5). Nowadays I just shrug it off and wait for better times...
Indeed the work is coming in bursts, particular the COVID-19 'moonshot' stuff. The PPD is all over the place with these, extremely high and extremely low, averaging for me on the low end. But this is cutting edge science in a hurry, and I think we all need to accept these are exceptional times and tolerate some randomness for a bit.
Re: Shortage of GPU work again?
Posted: Sat Jul 18, 2020 4:39 pm
by bumbel123
Yes, PRCG 13416 and all 117xx WUs are causing a drop in PPD. Especially with 117xx WUx I can see an overall drop of solid 30% monitored over a couple of days now.
They are even worse to the already discussed PRCG 13416 ... where I could confirm a longer term drop of "just" 15% over all.
Re: Shortage of GPU work again?
Posted: Sat Jul 18, 2020 11:52 pm
by bruce
Some of the projects 134xx are known to produce lower PPDs on certain GPUs. To avoid those cases, assignments can be restricted at the server level. Then we get complaints that the GPUs can't get work. Right now, the projects that earn "normal" PPDs are in short supply so you have to choose between getting lower PPDs or shutting down those devices entirely.
I belive this to be a temporary condition but have no prediction when conditions may change.
Re: Shortage of GPU work again?
Posted: Sun Jul 19, 2020 10:33 am
by Tabaluga
Since a few days i'm having problems getting GPU WU.
The log shows, no WU availble for your configuration, so i change the settings to allow all type of GPU WU.
Rebooted the win10 machine and i received some WU.
Now again same message.
What is the problem with the GPU WU assignment ?
rgds
Tabaluga
Re: Shortage of GPU work again?
Posted: Sun Jul 19, 2020 4:05 pm
by Neil-B
Making the change you did wont have actually mage any difference to your likelihood of getting a WU as the choice of WU type is simply a preference and defaults to other types if none of your preference are available - pure luck something was available after you restarted ... If you post the first 200 lines of your log (see bruce sig block above for help) then someone may be able to assist ... otherwise any assistance will just be pure guesswork !!
Re: Shortage of GPU work again?
Posted: Tue Jul 21, 2020 6:20 pm
by bruce
Yes, there's a shortage of GPU projects.
Science run in spurts. Each scientist constructs projects to study particular aspects of a particular protein and when enough data is collected, it has to be analyzed. When somebody else proposes a new theory, a new study is launched. It takes scientific minds to do all those steps and FAH doesn't waste your time with "busy work" doing unneeded calculations.