Ideas for new point system

[Perrosnk]

Hello everyone!
I am very new at the distributed computing world.
I was wondering if there is any way you can think of, to make a unified point system. For example, is there a way of real time measurement of the CPU or GPU work done, in flops or whatever else, and then get points based on flops or any other measure?
Is there a way of ensuring that nobody will be able to cheat the system?

[Zagen30]

Welcome to the forum, Perrosnk.

Raw FLOPs aren't always the best measurement of work done. For a long time GPUs weren't able to process the same type of work as CPUs, so while their FLOPs were much higher, the value of the work being done wasn't proportionally higher. This is set to change in the near future, as a new GPU core is in beta testing that will do the same work as CPU cores have been doing. This new core is expected to make GPUs earn significantly higher PPD since they do have so much more raw processing power than CPUs.

I seriously doubt they'll ever give you points mid-WU unless they overhaul how the whole process works. They don't benefit from your participation until they get a completed WU back, so they don't give you points until it's returned.

There are many security measures in place to prevent cheating. I know there's some serious encryption on each of the WUs that would be very difficult to break, as well as encryption on the client-server communications. The most anyone's cheated to my knowledge is setting up a botnet to fold for them, but the Pande Group takes that seriously and has zeroed out the points of individuals and teams that have done so. PG really doesn't want the validity of their results to be in doubt.

[P5-133XL]

It is my understanding that they are already heading towards a unified point system with all WU's being benchmarked against a standard SMP machine. That includes all GPU WU's, uniprocessor WU's and SMP/bigadv WU's.

As far as preventing cheating it is basically built into very vigorous validity checking that occurs with every WU returned. You get only points for successfully completing WU's that you were assigned and you won't get points of the data is bad or invalid. Theoretically, someone could game the data to make it appear valid but the best defense to that is the points are not worth real money. It would cost someone lots of time, and therefore real money, to reverse engineer the software and data so they could submit fake data that would pass the checks in place. Fake invalid data would invalidate the science being done so they are very careful with the validity checking. Also, since this is an electronic simulation of a real biological system, the predictions are checked against experimental data when possible.

[Perrosnk]

Thank you for your answer. I am pleased to be part of your community.

So, if flops can't work as a measurement, is there something else that could?
As far as I have understood, different projects give you different points, so a unified point system could be a very useful progress.

[mdk777]

s far as I have understood, different projects give you different points

Yes, a wide variety of math and science is being attempted. Different simulations require different amounts and types of compute utilization. In addition, the nature of distributed computing means that there has to be some logical packet formation...point at which a calculation can be broken down and dispersed.

This will not always be the same for every project, and trying to make them(WU) worth exactly the same amounts of points would require an inordinate amount of time and effort. (since each WU requires different amounts of resources, and takes varying amounts of time to complete a logical unit)

So, just FYI, the current practice is to set the value based on a standard benchmark machine comparison.

However, your general concern is one that is shared by many.

As mentioned above, while the current system may appear unnecessarily complex and arbitrary, there are plans to make it more uniform across all platforms.

[Perrosnk]

So what are the plans tO make it more uniform? Are there any suggestions? And which platforms do you mean?

[mdk777]

from oldest to recent:

http://folding.typepad.com/news/2012/10 ... -work.html
http://folding.typepad.com/news/2012/12 ... nted-.html
http://folding.typepad.com/news/2013/03 ... -core.html
http://folding.typepad.com/news/2013/03 ... pdate.html
http://folding.typepad.com/news/2013/03 ... re-17.html

discuss how the point system will be based on the work accomplished...using a single standard ...across CPU, GPU, or any future possible game consoles(my input...future PS and Microsoft game consoles will likely be open cl compliant, so the same standard can be applied)

[7im]

So what are the plans tO make it more uniform? Are there any suggestions? And which platforms do you mean?

Please understand the concept that the points ARE consistent and uniform on the benchmark computer. Similar projects all score the same points per day. The variations in points come in to play when YOUR computer has a faster processor, or slower memory, or you fold both CPU and GPU at the same time. The benchmark computer has no way to account for those variations, thus the variations in your scores.

[P5-133XL]

I told you: The plan is to assign point values by running all projects on the current SMP benchmark machine. That includes GPU projects. No longer will there be a separate GPU benchmark. Further, at some point GPU's are likely to start getting QRB (Quick return Bonus points) like SMP projects currently get.

Do note, that does not mean uniform points. Different WU's take different amount of time to run and therefore will be assigned more or less points. Further, some types of projects run very well on CPU's and totally other projects are much more suitable to GPU's but benchmarking both types on SMP will potentially make for some highly variable point values when projects are assigned to HW that is not optimum.

People generally do not like it when the PPD values move around much (especially lower) with the same HW but under a single uniform points calculation with a single benchmark machine it may be that there are different projects on the same identical HW that may vary in PPD by orders of magnitude. The first issue is that GPU performance is currently related to software (drivers + Cuda/OpenCL + Gromacs/OpenMM) all if which is likely to change significantly over time. Similarly, SMP performance is currently dependent upon Gromacs but that may also change over time. Then there is the simple fact is that what a CPU does well is not what a GPU does well and the difference could be 100x. Because the science demands it, Stanford could end up making projects that the CPU does well and assigning them to GPU's and visa-versa. That scientific priority is likely to play havoc with consistent PPD on the same HW. My point is the fact that the point calculation system is uniform and the benchmarking is uniform does not mean people are going to be happy with the result but it will be more rational and defensible.

Beware what you wish for -- You might get it!

[bruce]

Both the benchmark system and anything that might be base on, say, FLOPS assume that completing a WU is the only measurement of work done. For many distributed computing projects, this may be a valid assumption but not for FAH. Suppose you and I were assigned two WUs which required essentially the same calculations. Suppose you uploaded the completed work unit in 12 hours and I will not upload the finished WU until 12 days has passed. We did not do the same amount of work and we certainly will not be awarded the same number of points. The points system is based not only on the quantity of work completed, but the promptness with which it is completed. Your 12 hours of hard work is worth a lot more than my 12 days of rather slow work.

[Qinsp]

So what are the plans tO make it more uniform? Are there any suggestions? And which platforms do you mean?

Please understand the concept that the points ARE consistent and uniform on the benchmark computer. ...

Gotta call foul on that. I run a lot of different machines. Certain jobs are slow or fast all the way across the board. Unless they have custom hardware, their bench sees the same.

±10% WU difference is what I see across hardware types.

±30% is reality today. Two identical machines get 30% different PPD results depending on the WU.

Why do I care? It makes it hard to tune machine HW for max FAH performance if a slower config makes more PPD than a faster config.

Points are the only barometer we have as to how helpful our efforts are. It needs to be reasonably close, but has not been, and is not now.

Screw the points, gimme a tool for tuning for max FAH effort. OR using the existing points system more accurately.

This is intended to be helpful info. I'm going to run HW regardless of points. But having a broken metric is going slow the projects down in the long run.

[k1wi]

So what are the plans tO make it more uniform? Are there any suggestions? And which platforms do you mean?

Please understand the concept that the points ARE consistent and uniform on the benchmark computer. ...

Gotta call foul on that. I run a lot of different machines. Certain jobs are slow or fast all the way across the board. Unless they have custom hardware, their bench sees the same.

±10% WU difference is what I see across hardware types.

±30% is reality today. Two identical machines get 30% different PPD results depending on the WU.

Why do I care? It makes it hard to tune machine HW for max FAH performance if a slower config makes more PPD than a faster config.

Points are the only barometer we have as to how helpful our efforts are. It needs to be reasonably close, but has not been, and is not now.

Screw the points, gimme a tool for tuning for max FAH effort. OR using the existing points system more accurately.

This is intended to be helpful info. I'm going to run HW regardless of points. But having a broken metric is going slow the projects down in the long run.

Is one of your machines an "Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz"?

[Qinsp]

Yes, I have i5's. And i3's, and i7's, and Xeons.

I'd have to check the models and speeds. It's probably close.

Maybe 10 Intel machines, and 20 AMD machines. Never all at the same time though, but all have folded.

Slow jobs or fast jobs are consistent across hardware. This is not news. But it makes it hard to get maximum results.

Would you drive your car on the road if the speedometer was off 30%? Or punch in at work if the timeclock was 30% off?

It's not a serious issue. But to say it doesn't exist is weird. Really weird. That looks really bad. Everyone folding has seen it, but to say it's not there is... uh... worrisome.

[PantherX]

...Gotta call foul on that. I run a lot of different machines. Certain jobs are slow or fast all the way across the board. Unless they have custom hardware, their bench sees the same...

Not custom, just a standard benchmark machine(s) which is used across all Projects.

...±10% WU difference is what I see across hardware types...

That should be expected since there are different categories of hardware. High-end systems are more powerful than entry-level ones. Of course, WUs can be sensitive to other factors, like RAM speed, Cache, Clock speeds, threads, etc.

...±30% is reality today. Two identical machines get 30% different PPD results depending on the WU...

That variation exists because your hardware is different than the standard benchmark machine. If you have the exact same system as the standard benchmark machine, you will always get the same PPD regardless of what Projects you are running. Moreover, depending on the type of WU, it may scale better/worse with higher/lower number of threads.

...Why do I care? It makes it hard to tune machine HW for max FAH performance if a slower config makes more PPD than a faster config....

Humm... the only example that comes to mind is unoptimized FahCores for GPUs. However, that is being worked upon and things will improve. It would be nice if you could share some data regarding this.

...Points are the only barometer we have as to how helpful our efforts are. It needs to be reasonably close, but has not been, and is not now.

Screw the points, gimme a tool for tuning for max FAH effort. OR using the existing points system more accurately.

This is intended to be helpful info. I'm going to run HW regardless of points. But having a broken metric is going slow the projects down in the long run.

PG is aware of that and you do know that they are working on the equal points for equal work system.

Personally, I would prefer PG to dedicate all their resources to enure the most optimized FahCores for the supported hardware than shift their focus towards the points. The reason being is that regardless of how many points a WU is worth, the science being done by each WU is priceless in the big picture

[k1wi]

Yes, I have i5's. And i3's, and i7's, and Xeons.

I'd have to check the models and speeds. It's probably close.

Maybe 10 Intel machines, and 20 AMD machines. Never all at the same time though, but all have folded.

Slow jobs or fast jobs are consistent across hardware. This is not news. But it makes it hard to get maximum results.

Would you drive your car on the road if the speedometer was off 30%? Or punch in at work if the timeclock was 30% off?

It's not a serious issue. But to say it doesn't exist is weird. Really weird. That looks really bad. Everyone folding has seen it, but to say it's not there is... uh... worrisome.

I believe the benchmark system has a margin of 10%. With QRB, that means that this 10% is multiplied out by a substantial degree. If you have a Lynnfield i5 750 that you can compare projects on and find a margin of greater than +/-10% then you can always bring it to their attention...