Bigadv points change

[orion]

+2 Grandpa

[texinga]

I've seen the recent comments about larger than normal qtys of 2684's appearing lately. I always hope that people would not dump a WU simply because of points, but I expect it does happen to some degree. I'd guess that some people can dump 2684's (or other less desirable WUs) without really denting their 80% status that much. If my understanding is correct, a person who has successfully processed 10,000 WU's (for example) could (theoretically) have upwards of 2,000 WUs that could be dumped before crossing over the "bonus points" eligibility line. One concern I have about the recent overall Bigadv points reduction is that it may have given way to more aberrant Folding behavior like dumping certain WUs in attempt to getting WUs that still produce higher points (like the 6903). I dislike even the thought of such a thing, but it could be happening.

Grandpa, I hear ya and see this thing very similarly to your observations. My first impression after reading Kasson's statement was that Pande felt they had a problem with too many Bigadv Folders going after that particular area of Folding. I mean why make a statement like that if it wasn't an issue? I'm one of those Bigadv Folders and have no interest in running a bunch of GPU's and all the heat that they produce. So for me, if my desire to solely run the Bigadvs needs to be cut back, then I can do that on my own and stop Folding as much. I expect that we all can guess what really happened behind the scenes, but only a person that was involved in the decision could accurately answer our questions or interpretations of what Pande really wants. Take a break, go boating and come back when you feel the time is right. There is nothing wrong with taking a break (even from Folding).

[Grandpa_01]

Just one question: is it really possible to dump wu's and not affect your QRB eligibility? If that's the case, it's something which needs fixing fast.

Yes Very Possible. Any yes there is an easy fix =if(X>10,10,X)

[7im]

You and I both know that the point's for bigadv was readjusted because of people griping not scientific value. You can go back in this very thread and find a post by VJ saying the points were in line for the science done

As I said, donor feedback prompted a review. PG determined the outcome.

I really want you to highlight the part I missed.

Okay.

Kasson's post...

After much discussion, we are adjusting the points bonus for bigadv. Bigadv work units have been given a 50% base points bonus over standard SMP; the rationale for this was to compensate for the increased system load, increased memory requirements, and increased upload/download bandwidth requirements. As judged from the high demand for bigadv work units, this has been very much a success, perhaps a little too much so. We would like to continue to offer a bonus for bigadv to offset the above factors, but we don't want demand for bigadv to overwhelm the rest of the project or imbalance the points system.

We are therefore dropping the bigadv base points bonus from 50% to 20%, effective for all work units issued this time onwards.

We very much appreciate the donors who have volunteered to run bigadv work units; these projects add substantially to our scientific capabilities. We do important science with all classes of work units, however, and we want the points system to reflect that. Based on extensive feedback, we are considering renormalizing other parts of the system but have not finalized decisions in that regard.

Thanks again for folding!

Highlights from Vijay's News Post

For new bigadv WUs, we have changed how the points are calculated to bring them back into balance with the rest of the points in Folding@home.

After investigating the issues raised, we agreed that this was significantly out of balance, and made this change.

Vijay's forum posts...

This one was a tough call. After much donor concern on this issue, we did our own research into the matter which showed that bigadv was out of balance. However, we know that any sort of points change will be unpopular. This puts us in a very tough position. If we do nothing, we're ignoring the majority of donors (i.e. "Pande Group doesn't care.") and going against our own data, in favor of the high end FAH donors (and some have called FAH "elitist" because of this). If we make the change that's right based on our own numbers (i.e. the change we've just made), we get the bigadv donors naturally upset, which obviously is very bad too. No way to win here.

[Grandpa_01]

Come on 7im you did not highlighted anything that says the points were not right for the science being done. What you highlighted says they want to get them back into balance so it was not so attractive for people to hack the system. And that people are whining about the current pint system. Further statements from VJ below confirming the points for the science was correct. Public pressure played a very large role in what happened it is in this thread and really had nothing to do with the value of the bigadv WU being out of line. I myself said in the beta forum they needed to lower the value of the 6903, / 6904 WU's because I knew at the time there was going to be a public reaction to them. Stanford felt the value was right according to there formula (Black and White) which does not take the (H) factor into consideration. There is no (black and White for that)

* Report this post
* Reply with quote

Re: point system is getting ridiculous...

Postby VijayPande » Thu Jun 09, 2011 3:48 pm

MtM wrote:I don't think you can expect donors to come up with a point system and later on claim it's based on scientific value. That would be paradoxical.



My hope there was to see the candidates and choose the one which is closest to what's useful scientifically (or to suggest modifications to it that make it more useful scientifically). For example, a QRM is useful scientifically.

However, I don't want to push our current system if donors really hate it. While optimizing for science is my #1 concern, my #2 concern is to make sure that we have a lot of donors around to do the science, hence finding some compromise.
Prof. Vijay Pande, PhD
Departments of Chemistry, Structural Biology, and Computer Science
Chair, Biophysics
Director, Folding@home Distributed Computing Project
Stanford University

* Report this post
* Reply with quote

Re: point system is getting ridiculous...

Postby VijayPande » Thu Jun 09, 2011 4:12 pm

Russ_64 wrote:I think many will know this however some may not - the World Community Grid (WCG) project uses a points system that tries to be fair to all contibutors regardless of how powerful their hardware is (e.g. everyone will get the same points for the same unit), of course faster machines will always complete more work therefore there is an incentive for the user to upgrade.



Thanks for the feedback. Right now, QRB is only on a few projects, so most WUs have the same points value. However, I disagree with the characterization that QRB isn't "fair to all contributors." Returning a WU faster is much more beneficial, much like next day shipping is often very important compared to snail mail shipping. Of course, if you can ship a box next day, you'd expect to be paid more for it. QRB works like that. In a sense, the speed of WU return is part of completing the WU.

QRB is likely the biggest philosophical change to our points system ever and it was put in to "put our money where out mouth is". We asked donors to return WUs promptly if they can and the reply was "if it was so important, why doesn't the point system take that into account." We agreed and put it in.

Anyway, just some backstory here to keep in mind.
Prof. Vijay Pande, PhD
Departments of Chemistry, Structural Biology, and Computer Science
Chair, Biophysics
Director, Folding@home Distributed Computing Project
Stanford University

User avatar
VijayPande
Pande Group Member

Posts: 2401
Joined: Thu Nov 29, 2007 11:25 pm
Location: Stanford

* E-mail

[MtM]

But, it's not just 80% completion rate. PG should check for trends, it shouldn't be hard to spot dumping work unit's. Spot them, zero their accounts and be done with it. Yes I'm that harsh.

Let me add that if I knew a person was dumping I would report him to PG ( so it seems some people should be happy I don't read certain team forums ).

@Grandpa

Yes, you are right, the points were adjusted on the basis of (h) management, but (h) is a part of folding. Spreading out the donors over the available clients is part of the scientific value of work units, and giving to high a value to one part might cause less people to do the other parts.

Turn it around: if people hadn't complained nothing would change, points were in balance with science and stay in balance with science. Now scenario b: people complain, pg runs some numbers and finds out it's loosing donors on the other clients and decides this depreciates the scientific value of the whole project. To correct this, the high ppd client is penalized, allowing a more effective distribution of donors around the different clients. I always said I don't understand the why's involved with (h) management, but I'm starting to pick some of the how to's.

7im is trying to tell you that in the interest of the project the change was made to ensure the science which needs to be done using the other clients wouldn't be left waiting on the lone donor still interested in running an X client vs an Y client. I don't have the numbers they have, but I do believe they wouldn't penalize the top spenders without there being a real need.

Being out of balance indicates it should be balanced against another value, call it the scientific value of the other work units. If those work units are left alone because a certain work unit is worth more, this also devalueates the whole bigadv project.

Dr Pande's second quote which you did not highlight covers this. I know I said it is paradoxical, but I have to trust their numbers and intention. I happen to think PG does consider donors being their number one resource, and while I can't say I feel their decisions are right, I can't argue against it with proper content since I don't have the data needed to back up my gut feeling of it 'being wrong'.

[7im]

...

Further statements from VJ below confirming the points for the science was correct.

* Report this post
* Reply with quote

Re: point system is getting ridiculous...

Postby VijayPande » Thu Jun 09, 2011 3:48 pm

MtM wrote:I don't think you can expect donors to come up with a point system and later on claim it's based on scientific value. That would be paradoxical.



My hope there was to see the candidates and choose the one which is closest to what's useful scientifically (or to suggest modifications to it that make it more useful scientifically). For example, a QRM is useful scientifically.

However, I don't want to push our current system if donors really hate it. While optimizing for science is my #1 concern, my #2 concern is to make sure that we have a lot of donors around to do the science, hence finding some compromise.
Prof. Vijay Pande, PhD
Departments of Chemistry, Structural Biology, and Computer Science
Chair, Biophysics
Director, Folding@home Distributed Computing Project
Stanford University

* Report this post
* Reply with quote

Re: point system is getting ridiculous...

Postby VijayPande » Thu Jun 09, 2011 4:12 pm

Russ_64 wrote:I think many will know this however some may not - the World Community Grid (WCG) project uses a points system that tries to be fair to all contibutors regardless of how powerful their hardware is (e.g. everyone will get the same points for the same unit), of course faster machines will always complete more work therefore there is an incentive for the user to upgrade.



Thanks for the feedback. Right now, QRB is only on a few projects, so most WUs have the same points value. However, I disagree with the characterization that QRB isn't "fair to all contributors." Returning a WU faster is much more beneficial, much like next day shipping is often very important compared to snail mail shipping. Of course, if you can ship a box next day, you'd expect to be paid more for it. QRB works like that. In a sense, the speed of WU return is part of completing the WU.

QRB is likely the biggest philosophical change to our points system ever and it was put in to "put our money where out mouth is". We asked donors to return WUs promptly if they can and the reply was "if it was so important, why doesn't the point system take that into account." We agreed and put it in.

Anyway, just some backstory here to keep in mind.
Prof. Vijay Pande, PhD
Departments of Chemistry, Structural Biology, and Computer Science
Chair, Biophysics
Director, Folding@home Distributed Computing Project
Stanford University

User avatar
VijayPande
Pande Group Member

Posts: 2401
Joined: Thu Nov 29, 2007 11:25 pm
Location: Stanford

* E-mail

Please highlight in blue the words where Vijay says the points for science are correct or accurate.

You can't, because he didn't use those words. He said things like "useful" and "beneficial" which are very different in meaning, especially when you take them in context. Please note where those words were used, in response to a suggestion by Russ_64 that QRB be removed completely. NOT in response that QRB should be adjusted.

Please note what Vijay said was the #1 priority, part of what you quoted...

...optimizing for science is my #1 concern...

And how do you optimize for science? You align the goals of the project with the points awarded. So when people follow the points, as they naturally do, then they also follow the goals.

I'm not saying you are wrong, just that you are focusing too much on the wrong part of the message.

[Grandpa_01]

You are correct taken in context and you know what he meant as I do also. When he made that statement he was defending the QRB and all of us can choose the context we want but if you read the whole thread from the beginning you can figure it out. I myself whole heartedly agree with what MtM says his statement summarises what happened in the thread. You can try and twist it if you wish I do not care. It is there for everybody to read and make up there own mind. VJ suggests that the QRB is correct and that some people do not like the point system and he is listning Kasson says the system is working too well and based on public feed back the points are being changed. No where doe anybody agree with you that the points are out of line for the work untill after allot pressure was put on Stanford to lower the bigadv points.

And by the way MtM you are understanding the (H) factor better than you think.

I am done dealing with this for now it is time to load the boat and catch a crab or 10.

[Jester]

You are correct taken in context and you know what he meant as I do also. When he made that statement he was defending the QRB and all of us can choose the context we want but if you read the whole thread from the beginning you can figure it out. I myself whole heartedly agree with what MtM says his statement summarises what happened in the thread. You can try and twist it if you wish I do not care. It is there for everybody to read and make up there own mind. VJ suggests that the QRB is correct and that some people do not like the point system and he is listning Kasson says the system is working too well and based on public feed back the points are being changed. No where doe anybody agree with you that the points are out of line for the work untill after allot pressure was put on Stanford to lower the bigadv points.

And by the way MtM you are understanding the (H) factor better than you think.

I am done dealing with this for now it is time to load the boat and catch a crab or 10.

Another item that needs more clarification too is the reference to donor feedback, if it's from these forums alone then it's out in the open for all to see and make their own conclusions,
If it's from the "Donor Advisory Board" we are not able to see the descision process, and for obvious reasns do not know who makes up that DAB, but it would be useful to at least know
the makeup or selection criterea, or plainly, not by naming individuals, just to know where they come from and how are they chosen.

[bruce]

Another item that needs more clarification too is the reference to donor feedback, if it's from these forums alone then it's out in the open for all to see and make their own conclusions, If it's from the "Donor Advisory Board" we are not able to see the decision process, and for obvious reasons do not know who makes up that DAB, but it would be useful to at least know the makeup or selection criteria, or plainly, not by naming individuals, just to know where they come from and how are they chosen.

The DAB did discuss the issue but not in great detail. You can't build a conspiracy theory on what they've said.

7im was correct when he suggested the idea of making a change came from donor input and by far the greatest source of donor input is this forum. The decision(s) about what to change was exclusively the responsibility of the Pande Group in an effort to balance all of the science being done.

(Being scientists, they may have obtained other input based on server-based measurements, such as WUs being dumped, as somebody else suggested. I'd call that donor input, too, but it's hidden from all but the PG unless somebody chooses to post that they're doing it.)

[Jester]

Not trying to start any conspiracy theory, the more transparent things are the less likely it could even be thought of, let alone suggested,
As I said, I'm not interested in the individuals, more of a little history on how it came into being and how members are chosen,
It could be an encouragement to those who feel they have something to offer in that direction (no, not me) knowing if they may ever be considered.

[bruce]

It could be an encouragement to those who feel they have something to offer in that direction (no, not me) knowing if they may ever be considered.

This topic has been a mixture of highly emotional interchanges as well as a useful exchange of information. I'm certain that the Pande Group follows topics like this one. If I didn't believe that, I'd have closed it long ago when people were just venting their frustrations. There have been only a few suggestions that were concrete enough that they might be useful in revising the QRB (which is what Dr. Pande asked for many, many pages back) but there have been a variety of suggestions about general directions that have been or may be considered in the future.

Yes, you're being listened to, even when there are no PG responses.

[GreyWhiskers]

(Being scientists, they may have obtained other input based on server-based measurements, such as WUs being dumped, as somebody else suggested. I'd call that donor input, too, but it's hidden from all but the PG unless somebody chooses to post that they're doing it.)

Rather than rely on self-reported data, is there any reason that PG couldn't routinely distribute more meaningful server-based stats? e.g., correlation of WU completion time vs CPU type, or dumped WU count, etc. I know I'm totally left cold by the Client statistics by OS page.

[bruce]

I'd like to see more detailed statistics, too, but you're asking for the gathering of data to be automated and a regular report generated and publicized. I don't have any first-hand knowledge but my guess is that like most servers, you have to decide what information you want, figure out how to grep the logs, do a bunch of editing to get it to fit into an understandable format and once you've done that, you adjust something to improve the situation and then the data you've captured is no longer applicable.

Writing scripts that capture data from all the servers, combine the data, extract the kind of information that both you and I would like to see and display it periodically is a major task that's going to be so far down the priority list that we're unlikely to ever see it. Besides, although it might be interesting, it's unlikely to improve the science as much as any number of essential fixes to other types of code that are waiting for programmers to have time to work on them.

[MtM]

Another item that needs more clarification too is the reference to donor feedback, if it's from these forums alone then it's out in the open for all to see and make their own conclusions,
If it's from the "Donor Advisory Board" we are not able to see the descision process, and for obvious reasns do not know who makes up that DAB, but it would be useful to at least know
the makeup or selection criterea, or plainly, not by naming individuals, just to know where they come from and how are they chosen.

You do know who makes up the DAB -> http://www.evga.com/forums/tm.aspx?m=719517

Though I agree that on this forum there is little info to be found about the DAB, even while Bruce is 'our' representative.

About who they are/how they are chosen; I know xilikon has a proven track record both with his knowledge and dedication, I would think that he was indeed put forward by the rest of [H] but even if not chosen democratically he seems like the right person to be in a DAB. The same goes for the others, even if I don't know them ( as well ). Now, I'm a big advocate for democracy, so I would like it if every DAB representative was 'elected' in an open election in their respective teams, but alas who am I to suggest that to the teams involved? Bruce is the representative of the smaller teams and this forum in general, I don't think that warrants an election seeing Bruce's presence on these boards. Maybe that extends to the others on their respective teams.