Ideas for new point system

[Qinsp]

Like I said, it's not important, but saying it doesn't exist is weird.

PS - Please clue me in on the hardware that will get same PPD on P8101 and P8102. I'll go out and config a system to match exactly.

PPD ain't a big deal.

Correct response is:

Yes, the PPD varies a lot between WU's. While we try to keep it equal, it doesn't always work. The good news is that regardless of PPD, all the work is put to good use.

[mdk777]

That variation exists because your hardware is different than the standard benchmark machine. If you have the exact same system as the standard benchmark machine, you will always get the same PPD regardless of what Projects you are running.

Come now.

It is fine to express this as an ideal.

However, everyone who has folded for any period of time know that the ideal is not reality.

Over the years, I have posted examples of WU that were well beyond 30% variation.

Let's all agree that the point system is more art than science. It has many variables and many complexities.
There is nothing wrong with admitting that variation is inherent in the process.

I agree that starting a six sigma program to reduce that variation is probably not the best utilization of resources.

Like Qinsp implies, denying reality to defend an ideal is likewise a waste of time.

[Joe_H]

The reality is that the base points fairly easily can be shown to fall within that +/-10% range. The variation once you get away from the reference benchmark machine is getting the multiple parameters related to the differing projects for the QRB formula to give similar results far away from the baseline performance. Mathematically it is not easy, I am at times surprised they manage as well as they do. I mentioned the issue to my son who is a statistics major, he was glad it was someone else's problem.

[PantherX]

Last time I checked (a while back), there was a detailed specification of the benchmark machine. However, now it seems that it's very brief:

We have a single benchmark machine, its most important component is its processor: a Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz. The machine's OS is Linux

http://folding.stanford.edu/English/FAQ ... sNew#ntoc9

I am not saying that the current system is perfect nor I am denying the shortcomings. It exists and we all are aware of that. Having said that, they are working towards a points system that is more with their philosophy of equal points for equal work:

Note that GPU projects are now being benchmarked on the same machine, but using that machine's CPU. By using the same hardware, we want to preserve our goal of "equal pay for equal work". Our GPU methods have advanced to the point such that, with GPU FahCore 17, we can run any computation that we can do on the CPU on the GPU. Therefore we've unified the benchmarking scheme so that both GPU and CPU projects use the same "yardstick", which is our i5 benchmark CPU.

http://folding.stanford.edu/English/FAQ ... sNew#ntoc9

As Joe_H stated, the base points do fall within the 10% range. However, that difference is increased once the QRB comes in play and that might be an issue. If you do have valid solutions (which aren't already repeated here), I am sure that PG Members will consider it

[Grandpa_01]

The 10% is not really 10% it is plus or minus 10% at base which is actually 20% variation at base line which when you add QRB to some of these it can get quit out of range. I do believe the standard should be raised and maybe a little more time taken in benching, recently they have been listing to the beta team a little more and adjusting accordingly which is a good thing. All in all the point system works no matter what we all think about it.

I am pretty sure it does what it was designed to do which is give us some sort of direction as to what is needed and which way to go when we get new equipment. As far as I am concerned there is no need to change a system that works the way it was intended to work. It may need some fine tuning but that is about it.

[7im]

Points are the one topic fah gets the most feedback about.

Let's remember the QRB points curve is exponential, so even a very small variation in base points gets greatly enlarged in the QRB total over a very small time scale. This is especially exaggerated if your system is much faster than the benchmark computer because your system hits the slope on a much steeper part of the points curve. Even simple WU variations within the same project can cause points variations in the 100s to 1000s of PPD.

Those facts are not a denial of any potential problems. However, until those fact are understood very clearly (which often are not by new comers) it is very difficult to discuss/debate the points system on an equal and informed basis.

FAH has had 10+ years to find a better points system. And so far, no new suggested change to one part has been a greater good for all parts. A lot of smart people have tried. Most suggestions add to one group of folders while taking away from other groups. Most points changes are dictated by changes in the science and hardware, and science equating to points as a primary standard isn't about to change any time soon.

[Qinsp]

Points are the distance from the carrot to the mouth on the donkey. It controls the donkey speed. The donkey's concern about this distance is natural.

Did a quick check.

It's not the non-linear curve AFAIK that is at fault, nor PPD, nor variation in hardware.
Using 6 different kinds of HW, both Intel and AMD, there is a 19% to 28% variation in TPF on jobs with identical deadlines vs. the average, on BigAdv. This is using P8103 as average, and P8101 as the worse case scenario. When comparing the two extremes (8101-8102) the numbers are higher.

This is not unique to BigAdv, it's just easier to monitor.

And again, no biggie, but it's not the test hardware, not the non-linear QRB, or anything else but base points vs TPF vs deadline.

Yes, the points system works. No, it's not mission critical to tune it better.

Oddly enough, I use the 8101 as my metric for configuring hardware. It runs nice, and pops up often. But saying it does run ±10% from the average on some kind of system is probably false.

Sadly, while I have i5's I'm not smart enough to run a P8101 and P8103 on them. Nor would it represent the correct system for these jobs anyhow.

Bumping base points for 8101 should have been done during Beta. While it would suck for me, (all my calcs are on 8101), bumping base points would make a lot of folk happy.

[bruce]

Uniprocessor WUs were benchmarked on a P4. The FahCore went through several revisions, some of which altered the speed and others of which did not.

Recent SMP WUs have been benchmarked on the i5 and once again, the FahCore(s) have gone through several revisions.

In both cases, baseline points are established on the benchmark machine with whatever version of the FahCore is current. Now suppose a revised FahCore is faster. Those who are running WUs that were benchmarked under the old core will be pleased since their (baseline) PPD will go up. New projects will be added after that and they'll be benchmarked on the new FahCore so their benchmark will NOT get the bump in PPD. Eventually, old projects will finish and the "proper" PPD will apply to all active projects.

The same thing happens with GPU projects and BA projects. If a new FahCore is introduced that changes the speed, some projects will be faster than others. Old projects are (almost) never re-benchmarked using the new FahCore. [Oh, and a new FahCore can be either faster or slower ... depending on the scientific reason for the new FahCore, though most will be about the same speed.]

[johnerz]

Uniprocessor WUs were benchmarked on a P4. The FahCore went through several revisions, some of which altered the speed and others of which did not.

Recent SMP WUs have been benchmarked on the i5 and once again, the FahCore(s) have gone through several revisions.

In both cases, baseline points are established on the benchmark machine with whatever version of the FahCore is current. Now suppose a revised FahCore is faster. Those who are running WUs that were benchmarked under the old core will be pleased since their (baseline) PPD will go up. New projects will be added after that and they'll be benchmarked on the new FahCore so their benchmark will NOT get the bump in PPD. Eventually, old projects will finish and the "proper" PPD will apply to all active projects.

The same thing happens with GPU projects and BA projects. If a new FahCore is introduced that changes the speed, some projects will be faster than others. Old projects are (almost) never re-benchmarked using the new FahCore. [Oh, and a new FahCore can be either faster or slower ... depending on the scientific reason for the new FahCore, though most will be about the same speed.]

This implies that the BA work units are also bench marked, can you confirm this, and if they are not, how do you pull the base line points up?

Generaly I agree with Qinsp on this issue

[bruce]

BA and GPU projects are also benchmarked. I'm not sure about the details, though.

Baseline points are shown in http://fah-web.stanford.edu/psummary.html ("Credit")

[johnerz]

OK thanks, I was aware of the psummary page.

My understanding was (tbo I can't remember where I picked up the info) that BA units have not been Bench marked, and I was looking for some conformation about that, and if they had not been bench marked how the point were set. As you say they have been then thats a good part answered

[PantherX]

Regarding bigadv benchmark, here you go:

The deadlines were by benchmark numbers, but again the standard benchmark machine has been less than predictive for bigadv performance. What I did was set initial numbers via standard benchmarking (with bigadv numbers) and then adjust PPD to match 8103 on a bigadv-capable machine. Since PPD don't scale perfectly, we may have to adjust a bit more in testing.

Source -> viewtopic.php?p=240481#p240481