Points system for SMP2 work units

[kasson]

The SMP2 Core A3 work units mark the debut of a new points system. We have been testing the key element of this system--early completion bonuses--in the bigadv work unit program. Please refer to this document with for a more detailed explanation of the points system. We are also changing our benchmark system over to a Core i5. Points have been calibrated against previous benchmarking setups, as described below.

Introduction
Points are a key aspect of distributed computing, such as Folding@home (FAH), as it both works to indicate to donors how much they have contributed, as well as foster friendly competition between donors that has always been an essential part of distributed computing. Folding@home’s point system is based on the concept of a benchmark machine, i.e. a particular class of hardware which we use as a standard to define how many points a given calculation should get. The choice of this benchmark machine can have implications for points for donors. Moreover, how we use this benchmark is important.

Our benchmarking philosophy tries to balance two elements: keeping a system reasonably simple (both for donors and for the FAH team to calculate) as well as keeping points in alignment with scientific value of a given calculation. Indeed, donors will optimize their machines (eg choice of hardware, choice of clients, etc) based on points, so it is important that points awarded be reflective of the scientific gain.

While our basic benchmark idea is pretty simple, this document is fairly long in order to give donors full details about how we have chosen the benchmark machine as well as giving detailed information of this machine and how this could impact points for donors.

Benchmark philosophy
Our philosophy is pretty simple: we would like to standardize benchmarks to a single machine and standardize and simplify the bonus schemes now employed. Bonuses have played a key role in aligning points with science and we will continue to use them. For example, returning work units (WUs) promptly can be very important for the science we’re doing, so we provide bonuses for this, especially with the high performance clients.

Machines used in comparison
We chose a 2.2 GHz E6600 as the prototype dual-core machine and a Q6600 at either 2.4 GHz or 3.2 GHz as the prototype quad-core.
The new benchmark machine is a Core i5-750 with Turbo Mode off. We compare single-core performance to the old benchmark machine, a 2.8 GHz Pentium 4.

FAH Projects used in the comparison
We base comparisons to the single-core benchmark machine on projects 4442 and 6315, comparing single-core speed on the 2.8 Ghz Pentium 4 to ideal quad-core speed on the 2.6 Ghz Core i5 machine.
We base comparisons to quad-core machines on project 2671.
We base comparisons to dual-core machines on project 6012.

Results.
Machine: Performance relative to Core i5:
P4 2.8 0.098 (on project 4442)
P4 2.8 0.12 (on project 6315)
E6600 0.30
Q6600-3.2 1.1
Q6600-2.4 0.82

Based on these multiplicative speed factors, we can project ppd output based on either the A1 or the A2 benchmarking standards.
Machine: A1 ppd: A2 ppd:
E6600 521 1663
Q6600-3.2 1933 6172
Q6600-2.4 1450 4629


Bonus point formula
Briefly summarizing our bonus formula, the bonus is applied for users who have a passkey, have successfully returned at least 10 bonus-eligible WU's, successfully return >=80% of assigned WU's, and return the WU before the preferred deadline. Bonus points do not apply to partial returns.

Our bonus formula calculates final points as follows:
final_points = base_points * max(1,sqrt(k*deadline_length/elapsed_time))
Note that the max(1,...) ensures that final_points are never lower than base_points.

We can convert this formula to points per day as follows:
ppd = base_ppd * speed_ratio * max(1,sqrt(x*speed_ratio)),
where speed_ratio is the machine speed relative to the Core i5, and x = k * deadline_length.

Parameter determination
If we set the new quad-core base ppd to 1024 and the parameter x to 30, we get the following results:

Machine: projected ppd:
E6600 903 (greater than A1, less than A2)
Q6600-3.2 6456 (greater than A2)
Q6600-2.4 4628 (approximately equal to A2)
P4 171 (on project 4442)
P4 228 (on project 6315)

Explanation of x parameter
We may vary the deadline length between projects (some projects require fast completion and thus have short deadlines). Each project has an associated k parameter that controls the bonus points yield. We standardize k as follows:
x * speed_ratio = k * deadline_length / elapsed_time
since we can express speed_ratio as Core_i5_time / elapsed_time:
x * Core_i5_time / elapsed time = k * deadline_length / elapsed_time
therefore:
x *Core_i5_time = k * deadline_length
solving for k, we obtain:
k = x * Core_i5_time / deadline_length
and since x is set to 30,
k = 30 * Core_i5_time / deadline_length, where Core_i5_time is the time to complete a work unit on our Core i5 benchmark machine.

Summary
According to our projections, this new benchmarking standard will result in points yield for a 2.8 GHz P4 that is slightly above the typical uniprocessor values, points yield for a 2.2 GHz E6600 that is greater than typical A1 core yields but less than typical A2 core yields, points yield for a 3.2 GHz Q6600 that is greater than typical A2 yields, and additional points yield rewards for faster systems. The crossover point between A3 and A2 ppd in speed falls approximately at a 2.4 GHz Q6600.

[kasson]

PS some users report substantially better points yield for certain dual-core machines on the first round of SMP2 projects. These are user data rather than what we use in our benchmarking calculations, but I have included them here because they may be of interest to many:

Core 2 Duo T9300 (45nm) 2.5Ghz: 2000 PPD on projects 6012, 6014, 6015
Core 2 Duo Xeon 3075 (65nm) 2.67 Ghz: 1950 PPD on projects 6012, 6014, 6015