Bigadv points change

[Russ_64]

I think many will know this however some may not - the World Community Grid (WCG) project uses a points system that tries to be fair to all contibutors regardless of how powerful their hardware is (e.g. everyone will get the same points for the same unit), of course faster machines will always complete more work therefore there is an incentive for the user to upgrade.

How are points calculated?
Points are calculated in a two-step process which attempts to give a consistent number of points for similar amounts of research computation. First, the computational power/speed of the computer is determined by periodically running a benchmark calculation. Then, based on the central processing unit (CPU) time spent computing the research result for a work unit, the benchmark result is used to convert the time spent on a work unit into points. This adjusts the point value so that a slow computer or a fast computer would produce about the same number of points for calculating the research result for the same work unit. This value is the number of point credits "claimed" by the client. More information about that formula is available here.

Second, research results returned to the servers are validated in a manner which depends on the research project. Then the claimed points for valid results are examined for anomalous (excessively high or low compared to other machines computing the same or equivalent work unit) values and adjusted accordingly. The servers assign the resulting adjusted point values to the member (and team) for each of the returned work units. This process eliminates the ability for malicious users to tamper with results and artificially claim higher points for their work.

Even though I run relatively new, powerful hardware I can see a gradual decline in points production over time which doesn't seem to be a fair way to "reward" folders.

[Grandpa_01]

I don't think you can expect donors to come up with a point system and later on claim it's based on scientific value. That would be paradoxical.

My hope there was to see the candidates and choose the one which is closest to what's useful scientifically (or to suggest modifications to it that make it more useful scientifically). For example, a QRM is useful scientifically.

However, I don't want to push our current system if donors really hate it. While optimizing for science is my #1 concern, my #2 concern is to make sure that we have a lot of donors around to do the science, hence finding some compromise.

I do not know that donors hate the system, I believe the current system is a good guideline but needs to be a little more flexiable. I believe it is a workable system it just needs to be tweaked by you (points manually adjusted) from time to time when they appear to be out of line by the general public. We have had some good examples recientley with the 2684, 6700 and 6701 that were quite low they could have been manually adjusted to become more inline with the rest and now the 6903 which in public oppinion is a little high so you tweaked it down a little. I believe any point system you have will need some manual adjustment. If you feel a new system is needed then fine if you don't then fine also. Just look at the benchmarks and adjust the ones that are occasionally out of line.

[VijayPande]

I think many will know this however some may not - the World Community Grid (WCG) project uses a points system that tries to be fair to all contibutors regardless of how powerful their hardware is (e.g. everyone will get the same points for the same unit), of course faster machines will always complete more work therefore there is an incentive for the user to upgrade.

Thanks for the feedback. Right now, QRB is only on a few projects, so most WUs have the same points value. However, I disagree with the characterization that QRB isn't "fair to all contributors." Returning a WU faster is much more beneficial, much like next day shipping is often very important compared to snail mail shipping. Of course, if you can ship a box next day, you'd expect to be paid more for it. QRB works like that. In a sense, the speed of WU return is part of completing the WU.

QRB is likely the biggest philosophical change to our points system ever and it was put in to "put our money where out mouth is". We asked donors to return WUs promptly if they can and the reply was "if it was so important, why doesn't the point system take that into account." We agreed and put it in.

Anyway, just some backstory here to keep in mind.

[bruce]

Please clarify a couple of points for me, Leonardo.

If you want them scored separately, isn't the "flat rate" proposal equivalent to competing by using the number of WUs completed instead of points?

If you want them to be a combined score, isn't the baseline points essentially the same as your "flat rate" proposal -- or are you suggesting that the baseline points should be the same no matter how long a WU runs?

[P5-133XL]

It is very hard to construct a point system that is proportionally equivalent to the scientific value without having even a clue as to the scientific value of things. It would actually be useful to have something that rates the science value based of frame times even if it just for a single project. Once you have the curve as a set of points, then an equation can be constructed to match it or at least come close. Without those values, anyone constructing an equation is just guessing as to the shape of that scientific value curve and the likelihood that any new point system will match is relatively small. I am assuming that it is actually desired that the point system is somewhat proportional to the scientific value.

It should be possible to construct an equation with a linear portion and an exponential portion or even one that has some form of limit/cap. Perhaps I'm expecting too much and that scientific value isn't quantifiable or that it varies too much between projects that a single equation isn't possible. However, without even a clue as to what the curve should look like, I'm totally stumped as to how to form a point system that may work.

[MtM]

It should be possible to construct an equation with a linear portion and an exponential portion or even one that has some form of limit/cap. Perhaps I'm expecting too much and that scientific value isn't quantifiable or that it varies too much between projects that a single equation isn't possible. However, without even a clue as to what the curve should look like, I'm totally stumped as to how to form a point system that may work.

Nail on the head.

[Leonardo]

Bruce, my 'flat rate' proposal was given as a possible mitigating factor. The flat rate award would apply to every work unit, large or small, simple or complex. Yes, it would be the same points no matter what. It would be in addition to work unit-specific earnings. It would be recognition of dedication, which in the long run, is very important.

[GreyWhiskers]

re: timeliness and QRBs. I've spent a good amount of time looking at
- the arc of the 630+ WUs in Nvidia GPU p6801 series I've completed, and
- the logs for 130.237.232.141, the server for one series of -bigadv projects (http://fah-web.stanford.edu/logs/130.23 ... 1.log.html). After the shutdown last week, a group of WUs was loaded into the server, and is being fairly quickly given out, and after a while new WUs are added to the reservoir to continue project. I've looked at the WU RCV column of the log to see how a steady stream of WUs are returned. I've also noticed that the WU Avail number has been running in the high 200s, but that a few are added during every update cycle (twice an hour, I think) so the reservoir of WUs ready to be given up is slightly trending up - slightly more new WUs are added than ones assigned.

in both of these cases, I keep thinking about the importance of quick return of completed WUs back to Stanford, because of the use of completed WUs to create subsequent ones. But, much of this thread is about the scientific value of the completed WUs. And, to my mind, that is dependent on whether the flow of completed WUs is sufficient to keep the back end engine flowing to create the next sets of WUs, until the entire arc of the project has completely run its course. I would use a term like "time constant" to describe the process flow timing - how much of one gen of results is needed to make up for the next gen? How "wide" is the project - how many simultaneous arcs or threads of WUs can be out at one time?

We see that preferred and absolute deadlines are described for each project, going along with the time constant established. I would think that if the project donors were grossly underperforming against the natural time constant of the project, we would hear more from the FAH Project leads - they would be throwing "delay of game" flags onto the field, to use a bad analogy from US Football. I haven't seen that.

So - scientific value should be tuned to match each project's ability to cope with returns. Can the preferred and final deadlines, and the kFactors be tied not to the capabilities of the donor's hardware, but to the project's time constant needs? And, can the researchers include with the project descriptions process flows, or links to them, that show how "wide" the project is (how many parallel arcs or threads), and what kind of turn around cycle the project has, how many gen steps it will take to complete the computational phase on that project? The researchers obviously have this info to design the projects in the first place. Can that be shared with the community?

[7im]

...

We see that preferred and absolute deadlines are described for each project, going along with the time constant established. I would think that if the project donors were grossly underperforming against the natural time constant of the project, we would hear more from the FAH Project leads - they would be throwing "delay of game" flags onto the field, to use a bad analogy from US Football. I haven't seen that.

Just because you haven't seen it, doesn't mean it doesn't happen. Did you read Vijay's backstory post above?

QRB is likely the biggest philosophical change to our points system ever and it was put in to "put our money where out mouth is". We asked donors to return WUs promptly if they can and the reply was "if it was so important, why doesn't the point system take that into account." We agreed and put it in.

And they just rolled out the QRB for CPU work units with the new a4 core. They want their WUs back faster.

[mdk777]

GreyWiskers is asking for information to answer the question:

Return a 6901 in 2 days, get 41567.11 PPD. In 36 hours, get 63996.76 PPD. In 1 day's time, get 117569.55 PPD.

The first 12 hour jump is only 20K+ points. Are getting the results turned in another 12 hours sooner really worth 60K+ extra points?

What if you could complete the WU in 12 hours? 332,536.91 PPD. Is that next 12 hours really worth 200K+ PPD?

He is asking for a clarification of what are the benefits verses structural impediments to actually capitalizing on WU that are returned faster.

So - scientific value should be tuned to match each project's ability to cope with returns. Can the preferred and final deadlines, and the kFactors be tied not to the capabilities of the donor's hardware, but to the project's time constant needs? And, can the researchers include with the project descriptions process flows, or links to them, that show how "wide" the project is (how many parallel arcs or threads), and what kind of turn around cycle the project has, how many gen steps it will take to complete the computational phase on that project? The researchers obviously have this info to design the projects in the first place. Can that be shared with the community?

This is a rational question to determine how the bonus should be structured.

However, I have to agree with MTM and P5-133XL, this is a very hard thing for donors to quantify, much less describe with an equation.

[7im]

Same type of thinking here... it's very difficult to put specific numbers on this discussion because we don't know the scientific value of time.

If a WU is turned in after 4 days vs. 2 days, that is 2x as fast. Is the science helped 2x, 3x, 4x, more?

If we go from 4 days to 1 day, 4x as fast. Is the science helped 4x, 8x, 16x, more? How does it compare to the 2 day number?


Alternately (GreyWiskers's view), how helpful is a 48 core system when most everyone else is running a 4 core system? It's like having a top fuel dragster, but the track is only 50 feet long. It's like driving a Ferari in stop/go rush hour traffic. No matter how fast the car, it doesn't really help move things along that much faster. The race car is spinning it's wheels so to speak.


I suppose the counter point to Grey's view is that maybe a 48 core system should get even more points than before, because they are so unique, or sooo much faster than everyone else, it actually raised the average rate of return all by itself. It's like the one-eyed jack in a world of blind people. It's like the 1 geek in high school that didn't do anything but study and blew the bell curve for everyone.

So without knowing where we stand, I can't put numbers on anything.

[GreyWhiskers]

Excellent points, 7im.

I think that the 48 core "top fuel dragsters" make a major contribution by the sheer number of WUs they can crunch in a period of time - both shortening the average return time, and allowing creation of the next increment of WUs for all the WUs they produce all the faster. They help keep the reservoir of newly available WUs topped up so more WUs are available for the whole population of folders. I think that's a huge benefit for the whole project as it works through the entire arc of simulation steps.

EDIT: Not to mention they can uniquely tackle the really, really big problems.

EDIT: Parenthetically, the same affect comes from the GPU "fast folders" with 99+% performance factors. Even though they blow through a 5 day preferred WU in a couple of hours, that enormously improves the average project completion, given that there are enough separate "lanes" of WUs (runs/clones) to keep up with the project back end turn around.

But what I don't know is what the real simulation bottleneck is.

[MtM]

I remember something I talked about a long time ago with ( .. kasson or dan ensign.. not sure probably Dan though ) about if speedy returns could be exponentially important due to be able to selectively choose trajectories which seemed more promising based on the quicker returned results and drop trajectories which are very unlikely to be of interest.

It was just a thought but iirc he did say it was a possibility. If that would be true, speedy returns could do more then (re)move/lessen a bottleneck, it could maybe make folding more efficient? On the other hand, if the bottleneck moves to the PG server side which has to analyze and generate new wu's that would be funny for a DC project

[geokilla]

Who cares about the points? It's all about the science and health breakthroughs. If you're folding solely for the points, enjoy having a big e-penis.

[P5-133XL]

Who cares about the points? It's all about the science and health breakthroughs. If you're folding solely for the points, enjoy having a big e-penis.

If point are proportional to the science, as Stanford claims, then by optimizing the points, you are optimizing your contribution to the science. So points matter even if you don't care about your e-penis or competing at all.