Blog post: "Unified GPU/SMP benchmarking scheme ..."

[bruce]

By rigging the system to effectively penalize contributors with older hardware, they must realize that many dedicated folders are already frustrated and feeling that F@H is becoming an exclusive club for the techno-elite.

I agree with many of your statements .... and I'm CERTAINLY not suggesting that the new points system being proposed for GPUs is ideal, but it's election season here in the USA and if you were a politician, running for office, the other party would be calling you a liar.

In this context, the word "penalize" is a loaded word. Certainly nobody is being penalized. Everybody is getting MORE points that they did under the old system, and everyone is being incentivized to fold faster. While it is true that slower (older) hardware will receive a smaller incentive, the only thing you might call a penalty is if you fail to meet the Preferred Deadline (timeout) but even that use of the word is questionable.

[KWSN_PToT]

On the assumption that there are only so many points to go around (and if you just issue more to everyone it deflates their value, just like money), then which group would you want to provide the incentive to? I don't think you need to be a PhD at Stanford to figure out the right answer. Maybe you are right that just the higher output of the newer cards is enough of an incentive by itself, but the scientific value of the contribution can only be decided by PG.

I hope they get the right balance too, but just because a group of folders wants to get more points does not indicate to me that they have done it wrong.

I understand your point, and thank you for bring it up.

My point isn't about a "group of folders" wanting to get special treatment.
This is about how PG is defining simplification and fairness.

On the one hand, there are contributors who have been doing F@H for 10+ years.
Many of them have a legitimate complaint (that really should be addressed) that their continued contributions are being under-valued.

On the other hand, if a contributor is expecting to receive a "handicap" point system (like golf or league bowling, etc) then they need to think again.
Seriously old equipment (like a Pentium processor) cannot hold a candle to a modern Core i7.
Similarly, older GPUs are being seriously out-classed by brand-new GPUs.
But should a 3 year old GPU be penalized & obviated just because it can't keep up?
And I am not suggesting that we give any extra points as an artificial handicap.

Just as we are forced to upgrade OS versions every few years (and sometimes those upgrades come with 'minimum' hardware requirements), the older hardware - over time - becomes a more cost-effective paperweight than a computing device. That's just the way technology goes.

My point is about recognizing and rewarding long-term dedication & persistence as part of the whole point-system unification.
Let me give a practical example...

On my team, I am currently the #1 point-holder. But 6 months ago, I was quite farther down on that list (I have only been folding since Oct-2009).
In just 6 months (or less), using 3 GTX-560Ti's, a GTX-690, 2 AMD 8-cores, and 1 AMD 4-core, I was able to rocket to the top of the heap.
Now, there are a few of my team-mates that have completed MANY more WUs over the past 10 years than I have - and have less total points!

Let's say one of those 'veteran' folders acquires a new GTX-690.
I would be perfectly happy with giving them a few extra points for the same WU that I would complete on my GTX-690.
Why? Because they've been at it a lot longer than me.
But, over time (5 to 8 years?) - based on MY own dedication & persistence, I would expect that 'extra-points' gap to narrow with respect to the more senior folders (given the same hardware).

A nominal dividend should be paid for consistency & dedication (or seniority/tenure if you will).
There should - of course - be some limitations - that's why I suggested 5 to 8 years.
How this dividend would be computed is a separate discussion.
My real point is around deciding whether or not a dedication-dividend should be factored in.

[patonb]

Should I even inquire about the status of bigadv units in relations to this new gromacs that came seemingly out preform multi-socket systems?

[Zagen30]

The older GPU will complete the WU in about 7:30:00, but the newer GPU will complete that same WU in about 5:08:20.
Therein is the inherent QRB - the newer GPU will be able to complete more WUs over the same [longer] period of time than the older one.
More WUs = More points. Is my 'math' wrong???

No, your math is not wrong, but PG feels that getting in fewer WUs faster per computer is exponentially preferable, scientifically, to doing more WUs slower, rather than linearly preferable. Before the QRB, many people found that they could earn roughly twice the PPD on their quad-cores by running two SMP:2 clients. This was in large part because the SMP core at the time did not scale well to 4 cores, but PG saw the situation and wanted a way to emphasize that getting one WU done in X time was much better for them than getting 2 WUs done in 2X time, and a simple linear PPD increase did not adequately convey that. I don't think multi-clienting works on a single-GPU card, but at this point I think they feel that the QRB is doing its job of quantitatively expressing their preferences.

I understand that some projects are seemingly more urgent than others.
PG - with a little extra work - could do a better job with their assignment servers.
It has been my observation that when the client-app is attempting to get a new WU, it does/can transmit the "hardware info".
If some project is deemed to be a high-priority project requiring a minimum hardware level, then the assignment server(s) should not give one of those WUs to significantly slower hardware.
Additionally, just because BIGADV, etc are set on the client side, shouldn't automatically mean that the assignment server MUST honor that switch.
The assignment server should consider it a request that can be declined when necessary.

TWO:
The assertion that some professors or grad-students need results/data quickly in order to complete certain papers/theses - and are somehow entitled to preferential treatment - is a little hard to swallow.
PG and its project designers need to remember the scope and magnitude of the collective computing power that is being donated by hundreds of teams and many thousands of people -for free.
By rigging the system to effectively penalize contributors with older hardware, they must realize that many dedicated folders are already frustrated and feeling that F@H is becoming an exclusive club for the techno-elite.
The long term effect can be chilling. You risk losing a significant portion of long term contributors if you are hedging your bets on bolstering a smaller-but-better class of contributors.

The priority of projects in relation to others has nothing to do with the points. That's handled by assignment server weighting, where a higher-priority project will have more WUs sent out per 1000 than other projects. Getting work in quickly is something that benefits every researcher's work, hence why the bonus is applied to essentially all CPU projects and will soon apply to everything. A high-priority project will in theory earn the same PPD as a low-priority one, and any variances will be due to hardware scaling differences between your machine and the benchmark machine, not because it's higher priority.

Bigadv, like many of the other flags, is already a preference instead of an absolute setting, and it definitely does nothing if your machine is reporting fewer than 16 threads. I remember occasions where there was a shortage of bigadv work and a lot of bigadv folders were complaining about the low PPD that regular SMP work was getting them, as that was what was sent out in lieu of bigadv WUs. As for minimum hardware standards on regular SMP projects, there are already some projects that are restricted to 4 threads and above. There may be others that could use this as well, but I haven't seen much discussion on it.

I think a long-term contributor bonus has merit. It may have been brought up by the Donor Advisory Board* before, but we don't usually hear what's been discussed, so I don't know if it's already been rejected, still under consideration, or an idea no one's pitched. I think that's a discussion for another thread, however.

*A group consisting of Dr. Pande and a representative from several of the top teams, intended to give a more concise forum for discussing potential changes and complaints

[PinHead]

A nominal dividend should be paid for consistency & dedication (or seniority/tenure if you will).
There should - of course - be some limitations - that's why I suggested 5 to 8 years.

So something like 1% per year or 0.5% per year and a cap at 3%, 8% or 10%. Kinda like being vested!

If the coding is simple, then not a bad idea. If it involves a lot of maintenance and development work, then it distracts from the research. But I like the concept.

[JPinTO]

This should be interesting.

[k1wi]

Should I even inquire about the status of bigadv units in relations to this new gromacs that came seemingly out preform multi-socket systems?

Probably not at this stage... Particularly given the points they are trialling are subject to change and seeing as it is not public yet and this is a public forum, we are limited to what we can say...

The announcement does state that SMP and GPU are interoperable. But at this stage only one project is out and there are many other types of projects that are yet to be considered (implicit vs explicit, large proteins vs. small proteins etc etc). We have no confirmation that GPUs can even fold Bigadv type projects (v. large and explicit) at this stage...

We do know that GPUs have traditionally been only able to fold implicit projects and compared with CPUs they appear to be very efficient at doing so, not referring to any particular beta project. What we don't yet know if GPUs can fold a SMP project as effectively relative to a SMP client as they appear to do the GPU projects they are currently processing. How any of that will affect the benchmarking is anyone's guess as PG have the ability to modify the point system in order to reflect the scientific value.

At this point in time I'd certainly avoid making hardware decisions based on non-public projects.

[k1wi]

@KWSN_PToT - I think the mods should split your discussion to a new thread. Before you continue to develop your concept you should really have a browse through: viewtopic.php?f=16&t=21036. It will give you a taste of the opposition you will receive...

My method dealt with 'the inflationary effect' of hardware advancement by measuring the 'real' value of a point (i.e. removing the 'inflation'), while yours attempts to account for the inflation effect by 'inflation adjusting' previously earned points. Two quite different methods with two different outcomes but effectively they are attempting to solve the same problem of the relative value of previously earned points vs. currently earned points.

Mine kept previously earned points alone, but made earning new points harder at the same rate hardware improvements makes a given point 'easier' to earn. Yours allows for the continued increase in current points to be 'fair' by inflation adjusting existing points at the rate of inflation. The idea being in both the relative worth of an old point is effectively the same as a new point.

[KWSN_PToT]

By rigging the system to effectively penalize contributors with older hardware, they must realize that many dedicated folders are already frustrated and feeling that F@H is becoming an exclusive club for the techno-elite.

I agree with many of your statements .... and I'm CERTAINLY not suggesting that the new points system being proposed for GPUs is ideal, but it's election season here in the USA and if you were a politician, running for office, the other party would be calling you a liar.

In this context, the word "penalize" is a loaded word. Certainly nobody is being penalized. Everybody is getting MORE points that they did under the old system, and everyone is being incentivized to fold faster. While it is true that slower (older) hardware will receive a smaller incentive, the only thing you might call a penalty is if you fail to meet the Preferred Deadline (timeout) but even that use of the word is questionable.

Touché, mon ami. Point taken.

I did not intend to be this inflammatory - but this has got my hackles up a bit.

I am consistently trying to drum-up more active support (on my own team's forum) for F@H.
Unfortunately, I get a few actual responses that bemoan the perceived "unfairness" of the point system.
The most common themes from veteran folders are "...we don't feel welcome or appreciated anymore..." and "...being regarded as second-class..."
What is even more disheartening for me is the sound of "crickets"...the deafening silence of apathy and tacit rejection!

I also browse other F@H team forums occasionally - and there seems to be a similar undertow of growing discontent.
Many contributors have stated that they are so disillusioned with F@H that they intend to do BOINC, or even shutdown altogether.
For a couple of contributors who felt this way, I've actually encouraged them to go with BOINC instead of merely shutting-down.

I do try to do my part on the other side of this issue.
I won't molly-coddle those who want to over-rely on past performance (expecting a handicap score).
And although I haven't given it too much thought, I think I would be mildly disinclined to a one-time point re-valuation scheme.
But, I definitely think that past performance/contribution (especially when consistent over a long period of time) should count for something going forward!

[JimF]

My point isn't about a "group of folders" wanting to get special treatment.
This is about how PG is defining simplification and fairness.

On the one hand, there are contributors who have been doing F@H for 10+ years.
Many of them have a legitimate complaint (that really should be addressed) that their continued contributions are being under-valued.

Why are they legitimate? Why are they undervalued? If you found out that you could have taken a different job 10 years ago in a different city and have been paid more, do you think that fairness requires your current employer to make up the difference? I don't.

Seriously old equipment (like a Pentium processor) cannot hold a candle to a modern Core i7.
Similarly, older GPUs are being seriously out-classed by brand-new GPUs.
But should a 3 year old GPU be penalized & obviated just because it can't keep up?

Of course the newer equipment can do more. That is why we buy it, and why Intel spends billions developing it. But the old contributors are not being penalized at all. They are being given points commensurate with their contribution. Should the contributors with newer equipment be penalized by having to subsidize the points of the older ones?

Just as we are forced to upgrade OS versions every few years (and sometimes those upgrades come with 'minimum' hardware requirements), the older hardware - over time - becomes a more cost-effective paperweight than a computing device. That's just the way technology goes.

Yes, that is the way that technology goes, but in over 15 years of building my own PCs, no one has ever forced me to upgrade anything. I do it because it is more efficient and can do more, as you point out. And no one forces you to participate in distributed computing projects either. I don't think those interesting observations are at all relevant to your main point however.

My point is about recognizing and rewarding long-term dedication & persistence as part of the whole point-system unification.
Let me give a practical example...

On my team, I am currently the #1 point-holder. But 6 months ago, I was quite farther down on that list (I have only been folding since Oct-2009).
In just 6 months (or less), using 3 GTX-560Ti's, a GTX-690, 2 AMD 8-cores, and 1 AMD 4-core, I was able to rocket to the top of the heap.
Now, there are a few of my team-mates that have completed MANY more WUs over the past 10 years than I have - and have less total points!

Let's say one of those 'veteran' folders acquires a new GTX-690.
I would be perfectly happy with giving them a few extra points for the same WU that I would complete on my GTX-690.
Why? Because they've been at it a lot longer than me.
But, over time (5 to 8 years?) - based on MY own dedication & persistence, I would expect that 'extra-points' gap to narrow with respect to the more senior folders (given the same hardware).

A nominal dividend should be paid for consistency & dedication (or seniority/tenure if you will).
There should - of course - be some limitations - that's why I suggested 5 to 8 years.
How this dividend would be computed is a separate discussion.
My real point is around deciding whether or not a dedication-dividend should be factored in.

Why? Why? Why?
I use points to decide on what equipment to purchase, and rely on PG to provide the right metrics. If I have $400 to spend, should I get a new motherboard and CPU, or upgrade my video cards?

I don't see that feel-good notions of fairness do any good for that function, and only distort the metrics so that the wrong purchases will be made. You are building in a bias for older, less efficient equipment by reducing the incentives for cost-effective upgrades. That is the wrong way to go. Providing the wrong cost and benefit information can wreck any market. And it is even worse in high-tech markets, since the newer equipment generally uses less electrical energy. Why subsidize inefficiency?

[k1wi]

@KWSN_PToT - [snip]Before you continue to develop your concept you should really have a browse through: viewtopic.php?f=16&t=21036. It will give you a taste of the opposition you will receive...

Nevermind, you already got a taste of it here.

[jimerickson]

equal points for equal work, i am all for it. i have been folding for 10+ years and do not feel like a second class citizen. in fact with 8 gtx 480's folding for the last 2 years i feel as if my investment is finally being recognized. my first 3 years i donated anonymously so i know what it means to do it for the science not the points. and with unified gpu/smp benchmarking plus qrb for gpu i feel like i am being rewarded for all my work instead of just part of it. i too used to fold bigadv back when it was 8 core minumum and i begrudge those guys nothing with all their investments i hope they continue to get large qrb's. if it is more efficient and does more science then i say go for it. and more power to you pande group! looking foward to that 100x folding speed increase too. may we find the cures just that much faster. thats basically what i am after with all this equipment, cures for diseases. sorry to have rambled so long, didn't mean to take so much of your time. just felt i had to get that off my chest.

[alancabler]

AFAIK, no one on my team has ever publicly made an eloquent, but nevertheless emotional appeal filled with logical fallacies and inconsistencies in order to complain about getting left behind in the points race.
To the contrary, an original team member was so far ahead of everyone else that it took years for anyone else to catch him. He did eventually get passed in points, but kept folding. Alas, that gent has finally stopped folding, because he died.

It appears that some teams have complainers and now, one of the faster folders is bringing his resultant angst to the forum, trying to resolve his team conflicts.
Not to worry, he's merely a product of the times, having experienced throughout his young life, a constant barrage of illogical, but emotional argument from one political party famously parlaying a demonization of achievers in order to gain support for their failed utopian ideas of massive wealth distribution- equality of the masses. In reality, their ploy is just a method to gain advantage, power and control over others.

To that fast, but conflicted folding donor: to help you resolve your conflict, try this: for every work unit which gets credited to your donor name, start rotating credit for your work to other users on your team- one WU credit to you and then 5 to others with the highest "coefficient of hurt feelings"- in order to equalize the lowest performers. Sound reasonable to you? Willing to blindly make that jump, or do you want to think about it, for awhile?
I know that my proposal isn't exactly the same as yours, but they are mined from the same vein.

A 10- yr+ folder on my team recently discovered that he'd been folding without a passkey since the passkey/QRB introduction.
Know what he said? "They got folded, so what. Those points and $1.07 would have bought me a $1.07 cup of coffee down at 7-Eleven."

Thank you, Pavlov. You're welcome, Sigmund.

[JCM3500]

Regardless of points, I will continue contributing to FAH 24/7. Having seen my father suffer from Alzheimer's and Parkinson's in his last years, I will keep my ugly lavender box going with two GPU, and two uniprocessor tasks, one systray and one win32-x86 task. As I learn more, I will try to maximize the output from this PC, maybe adding Win7 64 bit. I recently bought my daughter a new AMD A8 based notebook for her postgrad studies and took over her 6-year old HP notebook which is processing two FAH tasks 24/7. It will run until it dies.

Happy crunching, everyone.

John

[Nathan_P]

I have no problem with a faster machine doing equal work getting more points.

However i do have an issue with a WU with only 900 atoms being folded in the same amount of time and getting 3 x PPD as the 77,113 atom WU that i am currently folding on a dual L5640 machine.

"Equal work means equal points" was the blog post - not on this 8057 WU its not.