Page 1 of 5
Project 13424 (Moonshot) very low PPD
Posted: Mon Aug 24, 2020 12:54 pm
by Shirty
Just an observation, the PPD on these units seems to be wildly lower then I would expect, around half of normal output. Currently running the following units:
GTX 1080Ti - P13424(R147,C32,G0) - 730,000 ppd - 3:05 tpf
RTX 2060 - P13424(R30,C27,G1) - 730,000 ppd - 3.05 tpf
RTX 2070 - P13424(R120,C5,G1) - 800,000 ppd - 2.57 tpf
RTX 2070 Super - P13424(R90,C12,G1) - 950,000 ppd - 2.38 tpf
RTX 2070 Super - P13424(R21,C18,G1) - 1,100,000 ppd - 2:22 tpf
RTX 2070 Super - P13424(R50,C49,G1) - 1,180,000 ppd - 2.16 tpf
Any idea why these are all running so slowly, utilising circa 60% of GPU power? It has literally shaved 6 million ppd off my daily output since I started picking these up this morning.
Once all 11 of my cards pick these up (as they invariably will) I will see my total output halved from around 21-22 million to 10-11 million ppd.
Re: Project 13424 (Moonshot) very low PPD
Posted: Mon Aug 24, 2020 4:44 pm
by HaloJones
Strange.My 1070s are down on 13422 but only by around 20% not 50%.
EDIT: Meant to write 13424
Re: Project 13424 (Moonshot) very low PPD
Posted: Mon Aug 24, 2020 5:13 pm
by Shirty
I've got no real issue with 13422, which fortuitously I seem to have gone back to on most of my cards since I posted the OP. I just thought it might benefit John to see my findings in case it's a wider issue.
Re: Project 13424 (Moonshot) very low PPD
Posted: Mon Aug 24, 2020 5:39 pm
by Joe_H
My understanding is this project has been pulled for now, problems were encountered with many WUs.
Re: Project 13424 (Moonshot) very low PPD
Posted: Mon Aug 24, 2020 6:31 pm
by Shirty
Cheers, hopefully it'll be back in New and improved form before long!
Re: Project 13424 (Moonshot) very low PPD
Posted: Mon Aug 24, 2020 7:05 pm
by JohnChodera
We had an issue in the preparation and quality control with the latest batch, so we've halted 13424-5 and rolled back to Sprint 2. We think we've fixed the issue and will re-launch tomorrow.
Thanks so much for your patience! These projects have produced incredibly useful data, and despite each week being nearly identical to the last, there always seems to be a new unanticipated challenge that sneaks in somehow!
~ John Chodera // MSKCC
Re: Project 13424 (Moonshot) very low PPD
Posted: Tue Aug 25, 2020 3:14 pm
by JohnChodera
Just an update on this: We've identified the issue here: A nearly co-linear bond between old and new atoms connected to a central heavy atom appears to be causing the OpenMM CCMA algorithm some pathological problems. Removing this single constraint speeds up the simulation by multiple times. This issue was likely the source of the significant RUN-to-RUN variation in earlier 134xx Moonshot projects.
We're tracking this issue here, for those that want to follow along:
https://github.com/openmm/openmm/issues/2814
While working on a long-term solution, we will re-prepare the system without constraints and start the sprint running shortly.
Thanks again for your patience!
~ John Chodera // MSKCC
Re: Project 13424 (Moonshot) very low PPD
Posted: Sat Aug 29, 2020 5:30 pm
by foldingfanmucde
Hi,
Observations on my side today, 29th August are:
Only project 13424 WUs seem to be currently being assigned to all my GPUs (NVIDIA and AMD) and PPD down by around 25% compared with e.g. WUs from project 13422 / 13423.
Maybe those symptoms are useful.
BR
Re: Project 13424 (Moonshot) very low PPD
Posted: Sat Aug 29, 2020 6:07 pm
by Neil-B
For the most part 13422/13423 PPD were very high (on my kit between 50%/100% depending on WU) to allow for the occasional low one ... now that the issue with low ones has been resolved I'd have expected PPDs on current sprint to be near normal ... I am seeing PPDs maybe 5% to 10% higher (if that) than I might have expected for my kit on this core so actually seem quite reasonable to me.
Re: Project 13424 (Moonshot) very low PPD
Posted: Sat Aug 29, 2020 7:33 pm
by koolpep
Confirming @foldingfanmuc.de - PPD down 25-30% with 13424 units compared to 13422 units. Actually base credit and bonus credit is down - processing time similar per WU.
Re: Project 13424 (Moonshot) very low PPD
Posted: Sat Aug 29, 2020 7:51 pm
by HaloJones
13424 is less than 13422 but 13422 was significantly above previous WU for my cards.
Re: Project 13424 (Moonshot) very low PPD
Posted: Sat Aug 29, 2020 7:53 pm
by PantherX
Welcome to the F@H Forum koolpep,
To paraphrase what Neil-B said, the Project 13424/13425 will now be using normal base points instead of the "higher base points" which previous Projects in 134XX series were using. The rational was in the current Project 13424/13425, there isn't much variation from RUN-to-RUN thus, there was no need to compensate for that. The previous Projects in 134XX series had a large variation between RUN-to-RUN so to ensure that the average PPD would be similar, a higher base points was temporarily used to compensate for the very low PPD in some RUNs.
Re: Project 13424 (Moonshot) very low PPD
Posted: Sat Aug 29, 2020 7:56 pm
by Neil-B
Re: Project 13424 (Moonshot) very low PPD
Posted: Sat Aug 29, 2020 9:18 pm
by JohnChodera
Just to echo @PantherX above: We've finally fixed the issues that were causing lots of heterogeneity, so we're now using a base credit more consistent with other projects. With previous 134xx projects, you may have been the lucky ones that never hit any slow RUNs---if so, you got extra points! If not, you probably had a more normal amount of PPD. Now that we were able to eliminate that RUN-to-RUN variation, we're back to more or less normal!
~ John Chodera // MSKCC
Re: Project 13424 (Moonshot) very low PPD
Posted: Sun Aug 30, 2020 2:40 am
by cine.chris
All my systems are down today at 82-83% of what I've been sustaining for several weeks. My goal was 300M pts for August which I'll make, but within hours now. User/Donor abandonment is already high, this could accelerate that rate. I'm still seeing normal levels of power consumption & GPU utilization, both a red flag to me that these WU are consuming resources (unlike some others I whined about) at high levels. Points don't cost anyone anything... except disgruntled & frustrated donors that abandon a project as that's the only metric & compensation they see.