[Debate] Revamped points system based on early return of WUs

[WangFeiHong]

Hi everyone, PG had previously mentioned that they intended to revamp the points system, but put it as low priority due to urgent issues on server/core codes.

So I’m thinking maybe we could have a debate thread where we can healthily discuss why and how this system may be implemented. Below are just some questions which I came up with to help guide the discussion.

May I also suggest that this discussion be limited to only this specific topic. If anyone has any dissent over other issues such as the benchmarking or ATI-Nvidia issues, feel free to post a new thread. And lets try to stay on topic so that this topic won't be locked . Thanks to uncle_fungus for his advice on this thread

A. Should we have a revamped points system based on time taken to return WUs?
A1. Importance of fast returns of WU (for progression of the science)
A2. Points are meant to be a reflection of the amount of science being done
A3. Discourages running multiple clients (particularly SMP) which may increase PPD but actually impede the progress of projects
A4. Why not just decrease deadlines (particularly for classic clients, currently it can be as long as 50 days) instead?

B. Problems associated with this new system
B1. Will it be unfair to long-time folders, folders with weaker hardware, folders who are not 24/7?
B2. Unfair to folders who receive WUs halfway through deadline (e.g. 10 days after a 20-day deadline project starts), or who receive WUs past preferred deadlines (which occur when a particular WU is not completed by the original folder by the preferred deadline)? Answer: Deadline for WUs are from the time they are assigned, so this is not an issue]
B3. What about High Performance Clients which have deadline of about 2 days? (They may return units with 50% time left instead of up to 95% time left on classic clients, so the bonus can be significantly lower)
B4. What about EUE-ed WUs?

C. How much bonus points?
C1. Linear or exponential (bonus drops rapidly to zero past preferred deadline) bonus?
C2. Formula (Points = mx + c) or stepped-based (50% time left = 1x bonus, 25% time left = 0.5x bonus, for example) bonus?


Others

If a trajectory consists of (say) 50 Gens, and the average turn-around time for a high performance client like SMP is 3 days, then a trajectory can take 5 months to complete. If every Gen required the intervention of the researcher, it might take a year or two. It makes more sense to me for a project to run for 5 months and the researcher to spend several months studying the data. If that produces publishable results, you'd have to add time to prepare a paper, get it peer-reviews, and eventually get it published. If it does not produce publishable results, (s)he may go back to the drawing board to design a new experiment.

Clearly during that first 5 months, partial results may be analyzed leading to early termination of a project or to the design of new projects that can run in parallel.

These are only my guesses for project timing so they may or may not be typical. Certainly the average turn-around time per Gen is MUCH longer for the CPU client and shorter for the PS3/GPU clients but the number of Gens may also be different.

More info will be added from other user's opinions later on.

[WangFeiHong]

Some Views (collected, contributed, and consolidated)


Why bonus points?
To me, the points system is already simple working quite well. The main reason why such a system was first conceived was because of hard-core (who, may I say, have many cores to fold with ) folders who, while trying to maximize PPD, delay the science; and because of people who feel that they put in more effort into folding than others (24/7 operation for example, which has higher electrical consumption vs. running FAH while the computer is on).

In order for a project to progress, most, if not all, of the WUs in a particular “Gen” needs to be completed, so as that the starting coordinates for the next set of WUs can be created. Explanation: (http://fahwiki.net/index.php/Runs%2C_Clones_and_Gens). Therefore, WUs should be returned as fast as possible.
We also should consider that there are always more WUs than folding processors, so this means that the total processing power of this distributed computer is still the same, whether users return units faster or not. Ultimately, the progress of the entire project depends on the total processing power of all folders.

However, on a micro-scale, I think that returning WUs faster simplifies the FAH research a lot, and ultimately one goal is probably to not have any WUs uncompleted after the preferred deadline (which would waste a lot of processing power, as the Assignment Server then assigns it to many folders, to process it ASAP).

Furthermore, I think most people would agree that this translates into more science being done, and since points are set so as to reflect the amount of science done, we should give bonus points for WUs returned early.

Theoretically, this could be implemented as a point bonus or a point penalty, but I think most people would prefer the carrot to the stick. Furthermore, if running more than 1 FAHcore per physical processor is undesirable, then more than verbal persuasion should be employed against such people.

Possible problems
While this will potentially create a PPD disparity between the haves and haves-not (people with faster hardware, or who fold 24/7, will get even more points than their slower counterparts), it can still be viewed as a win-win situation. The hard-core folders get their points, and for the “normal” folders, perhaps they are less concerned with points and just satisfied with folding for science.

*Paragraph deleted*

Nature of bonus
Lastly, on the technical part of the bonus, I propose that an exponential bonus that is formula-based be used, since there should be little difference in bonus from someone who returns work with 90% time left vs. 80% time left, but a large difference in bonus from someone who returns work with 10% time left vs. 0% time left.

What would be the maximum points bonus that is should be bestowed? Reference to: (viewtopic.php?f=46&t=6300), 2 SMP clients =1900PPD, 1 SMP client = 1300PPD, but 2 SMP clients return work almost twice as long as 1 SMP client. Assume WU has a deadline of 4 days, 1 SMP returns WUs in 2 days (50% to deadline), 2 SMP returns WUs in 4 days (0% to deadline). Assume we want to introduce a bonus such that the PPD (bonus inclusive) is about the same, so that the user will switch to 1 SMP client instead.

This means that the total points of the SMP1 WU needs to be about 50% more than that of the SMP2 WU. Assuming the SMP2 WU gets “x” points, this means that in two days: SMP2 = 2x points, SMP1 = 2(x+0.50x), which gives them equal PPD.

Using a formula of P = sqrt(0.25-0.25x²), where P = bonus points and x = proportion of time to deadline (e.g. finish 25% of time to deadline, x=0.25), the maximum bonus points is 50% (0.5) and the difference between x = 0.5 and x = 1.0 is 0.43=43% which kind of fits the scenario in the previous few paragraphs.

Note that the actual bonus amount should probably be much less, because A1 cores on Windows does not scale well across quad-cores and should be fixed in the future, resulting in higher PPD, so this is the worst-case scenario. Furthermore my formula is very primitive but… I’m just thinking.

[bruce]

Do you suppose we can combine this discussion with viewtopic.php?f=16&t=6949 ? (There are many more discussions, but they all wandered off-track, at some point.)

Do you propose to limit the disussions to
A) uniprocessor only
B) GPU only
C) SMP only
D) SP3 only
E) All of a,b,c,d simultaneously?
F) Some other combination?

Either I don't understand B2 or it's not possible.

[WangFeiHong]

With regards to clients I have no opinion on limiting in any way... but for the high performance clients (SMP/GPU), the extremely short deadlines already kind of reflects the need for faster return of WUs, so i'm not sure how a bonus here would work.


B2: Let's say a scientist starts a project/Gen with 10,000 WUs on 1 Jan, with a deadline of 1 March. I'm not sure about this, but i presume that the assignment servers can't possibly send out 10,000 WUs all at once? Like maybe on 1 Jan they'll be able to send out 1,000 WUs, and only on say 1 Feb can they send out the last WU of that particular Gen? Then the last guy who receives it would have 30 days left to deadline compared to some other guy at the start who has 60 days left?

And AFAIK, if a work server does not receive the WU back after the preferred deadline, it'll mass send that WU out to a lot of people (as it assumes that that WU was lost and it needs someone to fold it ASAP), so those people who receive it would have very little time left to fold, and if they get a bonus based on early return of WUs they'll receive very little bonus

[bruce]

Aha. There are three deadlines. The two that apply to individual WUs are called the Preferred and the Final deadline. You're talking about a third one which applies to the entire project.

Bonus points should logically only apply to that portion of the process that a donor can control. The researcher has to take responsibility for the rest of the total time and do what they can to minimize those portions. The scientific deadline of 1 March is only indirectly related to the deadlines on individual WUs.

It's possible that the total number of Generations in each trajectory can be estimated (I'm not sure about this) but ultimately you only know a protein has completely folded after it reaches that state so the number of Gens might change before 1 March. More importantly, there are a certain percentage of WUs that are downloaded and then not completed. Those must be reissued so in effect, there can be extra steps added into the trajectory during which time nothing was accomplished.

Of course the servers are less than 100% reliable and when they're down or too busy, delays are introduced into the uploading process.

I think we need to focus on the Preferred and the Final Deadline and let the researchers worry about the project completion deadline.

[WangFeiHong]

Hmm, i'm a little confused. Let me try to rephase, WRT classic clients.

Let's say WUs have a deadline of 60 days, and a bonus is accorded if a folder returns it within 10 days.
But if the day the group of WUs enter the work server is 1 Jan, it might not be able to distribute all the WUs to all the classic folders out there in time. So as a folder, i might receive that WU from the work server on 11 Jan, and return it on 12 Jan, would that make me not applicable for the bonus?

Unless of course the bonus is calculated from the time that particular WU is actually SENT out (and not when it enters the work server)...

[MtM]

More info will be added from other users opinions? Which parts where your own then?

[WangFeiHong]

If people have additional questions then i'll just add on and label their name.

[MtM]

I'd suggest reading all the other threads then

Besider the formula proposed, and the 'total project' based bonus you thought of which isn't actually there, everything has been mentioned before. I applaud the thread though

Bruce, the wu's all have a field which indicates when they have been assigned to a particular client ( queue.dat indicates this field exists? ). That's also how it calculates the percentage of time remaining to the preferred deadline.

[uncle_fungus]

Hmm, i'm a little confused. Let me try to rephase, WRT classic clients.

Let's say WUs have a deadline of 60 days, and a bonus is accorded if a folder returns it within 10 days.
But if the day the group of WUs enter the work server is 1 Jan, it might not be able to distribute all the WUs to all the classic folders out there in time. So as a folder, i might receive that WU from the work server on 11 Jan, and return it on 12 Jan, would that make me not applicable for the bonus?

Unless of course the bonus is calculated from the time that particular WU is actually SENT out (and not when it enters the work server)...

The deadlines you see on psummary relate to each individual WU assigned. I.e. you have 60 days to complete the WU from when it was assigned to you. It does not mean that the whole project has a 60 day window in which to complete all WUs (this figure is the hidden deadline that donors never see).
I.e. the B2 case above never happens.

[WangFeiHong]

Ah, i see!

[MtM]

Still not changed the paragraph about 'your views' ?

Nice to see you got all those ideas on your own, I don't believe any of it but still, if you did, I should just feel flatterd you thought of those same things I guess. Ow and not only me, I didn't invent the wheel either, but allot of people have brought the same points forward you're trying to pass of as your own here now. It's disrepespectfull don't you think to not mention that.

[WangFeiHong]

Changed the title.

If you

1. Feel that i plaguarized
2. Want credit for whatsoever reason (beta-testing?)
3. Feel that such an offensive tone is necessary

Feel free to help search for threads to acknowledge and quote everything i've said so far. The only reason why i put "My views" was because i typed that. Period. Changed.


Thanks.

[7im]

JAFBPDT. Let's try to save some time here.

A. Should we have a revamped points system? Yes, already answered by Pande Group. They have announced (a long time ago) a points system adjustment to better align the points and science. We all assumed it was a time of return based bonus, but no details were given at the time. And the short deadlines on the SMP and GPU clients are a similar reflection of this change in direction for the project.
A1. Importance of fast returns. This is a given, noting again the very short deadlines on the SMP and GPU clients.
A2. Points are meant to be a reflection of the amount of science being done. Correct, what's the question? A discussion of how the points are assigned are in each of the client's FAQs. See those for more detail.
A3. Discourages running multiple clients. See A1, which will resolve this, along with the coming fahcore A2 for SMP.
A4. Why not just decrease deadlines for CPU client? Because most people do turn in the work units very quickly, but shortening the deadline would exclude a large number of contributors with older hardware, and Pande Group does make an effort to enable as many people as possible to contribute to the project. A quick return would provide incentive to upgrade and do the right thing, but no exclude anyone, and PG avoids exclusions when possible.

I have also personally suggested re-using an older existing client switch to enable a short deadline bonus feature for the CPU clients. (re-using an old switch shortens development time and testing) Similar to small, normal, and big work units, it would enable standard or short deadline work units (with an associated points bonus), and breath new life in to the project from the many CPU contributors who feel slighted by the high points given to the new clients. A temporary version of this suggestion was already done with the -advmethods switch and the AMBER bonus projects from a few months back. I await a more permanent development.

B. Problems? There will always be points flame wars, but PG does try to minimize any impact, and has long deliberations befor making any changes.
B1. Will it be unfair to long-time folders? Yes and No. Yes, it dilutes previous contributions ever so slightly. However, new hardware like dual cores and the bonuses on SMP far outweigh any of those, so the bottom line is No, not really.
B2. Not an issue.
B3. What about High Performance Clients which have deadline of about 2 days? They already have a bonus program, so this gets a lower development priority, but it should eventually get a quick return bonus as well, to better align the points and science (modifying the points motive for running of multiple SMP clients into running 1 client faster)
B4. What about EUE-ed WUs? Fact of life, and we all get them, so no bonus on EUEs. Sure, an EUE may return some usable work, but the temptation to force EUEs is too great, and a no bonus on EUE eliminates that temptation, and is more motivation to run computers at a more stable overclocking speed.

C. How much bonus points?
C1. Linear or exponential (bonus drops rapidly to zero past preferred deadline) bonus?
C2. Formula (Points = mx + c) or stepped-based (50% time left = 1x bonus, 25% time left = 0.5x bonus, for example) bonus?

Good questions, but PG tends towards linear, but not always. Linear is simpler to reward on the fly, and easier for the user to calculate themselves. PG tends to use KISS methods, keep it super simple.

A quick return bonus has been a perpetual suggestion for many years. I'm not sure how debating this for the 101st time will be any more helpful than the 100th, but everyone is welcome to their own opinions, as am I. If I had to guess, PG already has most of this planned out to varying degrees, and is just waiting for several software developments to proceed, along with waiting for the GPU clients to become more stable and mature, so they can refocus on other developments. Pick the low hanging fruit first.