Change in BA requirements

[TheWolf]

It appears that most everybody can agree that the decrease in points of 50% to 75% when switching from bigadv to smp is both disheartening and unpalatable to those affected by it. Further, it appears this far exceeds the design parameters of the bigadv program Dr. Kasson outlines, more than three times the intended 16.7% decline in points when switching from bigadv to smp WUs.

My initial hunch was that, due to the exponentially decreasing QRB, the large decrease in points was a failure of points system to properly account for the speed of a bigadv machine folding smp WUs much faster than bigadv WUs. A few quick back of the envelope calculations show that a bigadv machine getting 200,000 PPD on a smp WU is completing WUs 4.7 times faster than an "average" smp machine that gets 20,000 PPD on the same WU is only receives a QRB that is 2.2 times higher. The comparison between a 10,000 PPD and a 20,000 PPD machine is quite different. The faster machine is completing WUs 1.6 times faster and receives a QRB multiplier that is 1.3 times greater. It is readily apparent that additional speed increases on slow machines are rewarded much more than equal increases on fast machines. Return speed of WUs is initially highly valued, but adding additional return speed is subject to exponentially diminishing marginal returns..

Comparing bigadv vs smp on the same machine, however, provides an interesting result. The extremely large k-factor for bigadv WUs creates a roughly equal QRB multiplier for bigadv and smp WUs on the same machine across a wide range of WUs. If the k-factor is intentionally calculated to achieve this result (and consistency across a range of WUs leads me to believe it is intentional), the reason for the large decline in points when transitioning from bigadv to smp would not lie in the QRB. This leaves the assigned point as the most probable cause for the large deviation for the anticipated 17% decline in points. This is further supported by what Dr. Kasson indicates above, there are scaling issues both ways on bigadv vs smp (bigadv WUs fold much faster on a bigadv machine relative to the benchmark machine while the opposite is true for smp).

A potentially novel solution to the benchmark scaling issue may be to simply change the benchmark machine to a bigadv class machine. This would minimize the scaling issues (both up and down) for big advantage machines and provide stability through the bigadv to smp WU transition. Additionally, as a result of this change we can expect a bit of a point boost on non-bigadv machines running smp WUs that do not scale well to a large number of cores. Most people have argued that these lower smp WUs are under-valued and under-appreciated in regard to the points they receive, potentially making the point boost an ancillary benefit rather than a detriment. Regardless, at a minimum it would simplify the scaling issues by reducing them to one direction.

If this does not resolve the benchmarking issues, I fear the solution becomes a bit more complicated. The long-term solution may require that benchmarking procedures be changed and the golden standard of a single benchmark machine for all WUs be abandoned as the real world results indicate that it is not rewarding "equal pay for equal work." (I use the term work as a measurement of science). For bigadv class machines, the aforementioned scaling issues and observed data points suggest that, on a bigadv machine the bigadv points are over-inflated and the smp points are under-inflated relative to the theoretical construct of the points system. It would be extremely disheartening if PG chose to sacrifice the integrity of the observed results of the points system to maintain its underlying theoretical integrity.

In attempting to break away from the strict adherence to "equal pay for equal work" it is important to note that it appears bigadv was never based purely upon the premise. It is clear both from Dr. Kasson's description of bigadv and the observed comparison between bigadv and smp that the bigadv points premium rewards factors other than pure work/science, negating the direct application of equal pay for equal work to big adv WUs.

Given that actual results of bigadv vs smp PPD manifest large deviations from the intended behavior, the aforementioned difficulty in benchmarking bigadv WUs using current procedures and that equal pay for equal work only loosely applies to bigadv WUs; it may be time to consider revising benchmarking procedures to produce more consistent results. Several years ago, GPU WUs were benchmarked separately from smp WUs because it was not a valid comparison (and infeasible at the time). The description of how bigadv WUs are benchmarked makes it very much appear as if the integrity of the points system is being compromised for the sake of a maintaining its theoretical purity.

Other potential approaches to addressing the large decline in points transitioning from bigadv to smp can be achieved by modifying the points system. If benchmarking is at the heart of the problem, however, this approach will be a second-best solution. One possible approach is to alter the points formula to punish increased return speed less harshly than the square root function does (outlined above). I've briefly played around with a few functional forms that do so without drastically altering the points landscape. This approach would, however, reward increased points to all smp folders as well as QRB GPU folders. An alternate approach would be to create a smalladv WU type with a more appropriate decline in points relative to the bigadv level. Smalladv could be similar to bigadv but targeted at some of the larger SMP WUs that scale well with more cores and would benefit from the increased folding horsepower that no longer qualifies for bigadv.

Regardless of the cause, it is readily apparent that the bigadv point system is not functioning within the design parameters Dr. Kasson described, and this has understandably upset a large number of donors that will be negatively affected by the upcoming changes. Any changes to benchmarking procedures or points system will have to be carefully considered and probably will not be in place by the announced deadline for upcoming bigadv changes. As a show of good faith, it would be a nice gesture if PG delayed the impending changes until a solution is worked out or a temporary stop-gap measure can be put in place. This would prevent affected donors from experiencing a massive decline in points that it appears they were never intended to face if bigadv points functioned as originally outlined. While it is important that the folding community embraces change and continues to move forward, it is equally important to fix the issues at hand before doing so. The biennial cycle of bigadv crises has quickly grown wearisome.

+1

[mdk777]

Yes, and as you noted, I did not build a 4p rig.
point of fact, I did not OC a 4core 8 thread either.

How individuals choose to participate or not participate does not change the responsibility of PG to provide clear guidance.
ad hominem attacks add nothing to the topic.

This is what I wrote in 2010

The real question is whether PG will ever step up to the plate and issue reliable uniform and predictable clients/bonus systems from which donors can make informed decisions. If a donor wants to build a 4 socket machine to run -bigadv, he should be able to do so with the assurance that WU will be available and appropriately rated...

So far Dr. Kasson has not answered any of the substantive questions asked in this this thread.

I don't think anyone, not a single person...cares about the absolute value of BA WU.

What Everyone cares about is having their donation of considerable time, investment and ongoing electricity donation recognized.

To date I see no recognition on the part of PG that this has again been horribly handled.


Not a single question asked by a dozen donors has been seriously answered.
Instead of answering what is the logic of the timing, instead of answering the why?; Dr. Kasson merely states that he thinks it will happen sooner than later.
Again, my conclusion from 2010 has not been changed by Dr. Kassson's recent dismissive and condescending responses:

In my opinion, this attitude that "support" ranks a dead last on the list of priorities is the reason that participation retention is so abysmal

viewtopic.php?p=158575#p158575

Bruce has PM me several times...saying that we should hold judgement until PG can respond in more detail.

Well, I will not hold my breath.
I think my humor is entirely too accurate.

[bruce]

Many have stated that BA points are overinflated relative to SMP points -- which is the same as saying that SMP points are under-inflated relative to BA points. The two statements are equivalent except for the fact that they sort of suggest that the problem should be solved by increasing SMP points or decreasing BA points. Emotionally, those two solutions are not equivalent, but mathematically they are. I'm going to stick to mathematics and suggest that I don't care which approach is undertaken -- and that I'm abstaining from that discussion.

Nevertheless, about two pages back someone suggested that changing the benchmark machine would help solve the problem. I must disagree. Changing the benchmark machine won't do anything to align the points with the amount of work being done. Benchmarking a project that's earning 20K PPD and assigning it an arbitrary value of 20K PPD will still have a machine that earns 200K PPD on another project. Changing the benchmark machine on a project that currently earnsi 200K PPD and arbitrarily assigning it 200K PPD will still give you a machine that earns 20K PPD on a project that currently earns 20K. The bottom line is the relative points don't match the relative magnitude of the calculations (which is to say the calculations to not scale well). The choice of the benchmark machine doesn't change how well the calculations scale -- whether you scale them up or scale them down.

[Adak]

So far Dr. Kasson has not answered any of the substantive questions asked in this this thread.

I don't think anyone, not a single person...cares about the absolute value of BA WU.

What Everyone cares about is having their donation of considerable time, investment and ongoing electricity donation recognized.

To date I see no recognition on the part of PG that this has again been horribly handled.


Not a single question asked by a dozen donors has been seriously answered.

Dr. Kasson's post was only an assurance that our complaints were being heard, and that steps to fix the problem would be forthcoming soon.

Where did Kasson state that he was posting the solution, or the steps that PG would be taking, etc.? You are expecting an answer WAY too soon.

Instead of answering what is the logic of the timing, instead of answering the why?; Dr. Kasson merely states that he thinks it will happen sooner than later.

Again, my conclusion from 2010 has not been changed by Dr. Kassson's recent dismissive and condescending responses:

I believe we have a pretty good understanding of the problem, and don't need to bring up a problem from 2010. Let's work on today's problem.

Bruce has PM me several times...saying that we should hold judgement until PG can respond in more detail.

That is good advice.

Well, I will not hold my breath. I think my humor is entirely too accurate.

I don't believe "accurate" humor, is going to be helpful, in solving this problem. Keep breathing!

[Adak]

Changing the benchmark machine probably won't solve the problem, but removing the benchmark machine entirely, might. That is, design the point system so it is machine agnostic. HOW the bricks get carried up the hill, doesn't matter.

I know troy8d from EVGA and myself, have both designed good point systems. I'm sure there are a few others who could make a point system for FAH.

[Bill1024]

I only have a 9th grade education, I am not the sharpest knife in the chandelier. After 30 pages, I still can not figure out what the problem is in the first place.
Why are they changing anything at all? 8/16 core Intel desktops will not be out until Q3 or Q4 I have learned.
But why, is there a backlog of SMP or not? What exactly does move 16 and 24 core server to "enhance SMP" mean? Enhance as to make better? To fold more of?
Looks like -smp 24 or -smp 48 is not even folding the WUs that there is 180,000 of. Breaking it down to 4+8+12+24 made it so too many cores were left idle and the PPD or TPF was really out of line.
It's their project and they're going to do what they need to do, I understand that. But the posts are somewhat vague and not to the point.
Too much reading between the lines, guessing what PG is saying. Going to do exactly what "sooner rather than later".
I am not all that bright, I need clarity, say exactly what you mean in simple terms a simple man can understand.
With all due respect, say what you mean, mean what you say, say it so all of us understand exactly whats going on.
Please.

[k1wi]

My take is that as the number of fast BA machines are increasing they are effectively dropping the 'slower' machines from the pool, in the only way they know how (by limiting core counts) and also tightening up the deadlines. This should have the effect of decreasing the average turn around time, therefore increasing the speed at which trajectories are completed, or allowing trajectories to get further advanced in the period with which they have to get results.

Limiting by core counts limits the number of WUs that might be dumped from people trying unsuccessfully to meet the new deadlines (but is imperfect because old 4p systems might fail to meet the deadlines, while some newer 2p systems might still meet them).

If this is the case, alternative approaches might be to simply add more RGC to the mix, to satisfy donors demands for the BA 'bonus' but that might have zero scientific benefit if it is superfluous (and thus take resources best used elsewhere), or simply let BA'ers play musical chairs with a 'shortage' of BA WUs, neither of which would decrease the average turn around time and probably lead to complaints.

SMP should speed up (even if only slightly) by the 'pushing' of previously BA rigs back out of the big leagues, but that is of course dependent on donor behaviour. History probably provides a solid guide to what happens there.

Finding the right time to pull the trigger is pretty difficult, because there are consequences with leaving it too late and with going too early (I.e reducing the pool too much and leaving 'waiting' WUs), which may slow scientific progress. Further frequent revisions are difficult when core counts themselves tend to jump in steps (I.e 24 'cores' -> 32). An intermediate step from a 24 core minimum to 28 cores might have zero effect on BA donor #'s but require large amounts of resources to implement.

My experience with academic research is that things tend to crawl along and then get decided and happen pretty damn quickly. Predicting accurately in advance the resources required is dependent on a whole lot of factors out of control of the researcher - will she/he get the funding they are after to run the project? Or factors moderately in control of the researcher - what method will they use, based on what others have already tried? That's before computing advances such as hardware releases by your AMDs, NVidia's and Intel add uncertainty from the 'other side'. Again just my perspective from within academia and why I understand the adversion to predict far out. Of course it conflicts with what donors want.

The indicated 'coms officer' should improve the situation, but like above, sometimes these things take quite long periods of time from when they say 'let's get a communications officer' until that person starts fulfilling their role: how will they be funded? How do we spread it across multiple institutions with multiple grants/projects? When can we get them added to our budget - next financial year or next funding round or do we have enough in our operations budget now? What will their scope be? What do donors tell us we need to change (note this was done earlier with their survey)? Who out of the applicants is best for that role? What do the rest of the people in the cluster think of each decision? Collaboration is powerful, but can slow things down, as does bureaucracy!!!

Hopefully the position will allow the reallocation of resources in a more effective manner, that offsets the cost associated with the position. I imagine it will be just as much an experiment/cutting edge as the F@H project and DC is in general.

I apologise in post that I am just another folder. Hopefully my experience working in the university system might help shed some light?

[Adak]

I only have a 9th grade education, I am not the sharpest knife in the chandelier. After 30 pages, I still can not figure out what the problem is in the first place.
Why are they changing anything at all? 8/16 core Intel desktops will not be out until Q3 or Q4 I have learned.

PG knew that the requirements for BA work units would need to change, before long. They said that when they brought BA out, and they repeated that alert when they increased the threshold for BA, the last time. (over a year ago)

I don't know the ins and outs of the protein folding field, but I'm sure that PG knows it, and these changes are what they want or need. Some faith is required it seems, even dealing with a science project.

But why, is there a backlog of SMP or not?

I'm not sure. Whether there is a backlog or not, we need to adjust the point system for the BA PC's that will be cut out of BA, in a few months. We don't want them feeling like they're not a vital part of FAH anymore, because they are. Their ppd should reflect that.

What exactly does move 16 and 24 core server to "enhance SMP" mean? Enhance as to make better? To fold more of?
Looks like -smp 24 or -smp 48 is not even folding the WUs that there is 180,000 of. Breaking it down to 4+8+12+24 made it so too many cores were left idle and the PPD or TPF was really out of line.

Enhanced smp would be a new smp category, that would fit between regular SMP and BA. It's just an idea at this time. Doubt if it will be adopted.

It's their project and they're going to do what they need to do, I understand that. But the posts are somewhat vague and not to the point.
Too much reading between the lines, guessing what PG is saying. Going to do exactly what "sooner rather than later".
I am not all that bright, I need clarity, say exactly what you mean in simple terms a simple man can understand.
With all due respect, say what you mean, mean what you say, say it so all of us understand exactly whats going on.
Please.

Dr. Kasson was just letting us know that they had heard the complaints and suggestions in this thread, and will work on it soon. Nothing is being proposed as an answer by anyone at PG, yet. It is common for academics to take some vacation days in late December to early January, since the University is closed for classes, at that time.

These things take time. Be patient. Clarity and simplification of all the above, will arrive, but not right away.

[Adak]

My take is that as the number of fast BA machines are increasing they are effectively dropping the 'slower' machines from the pool, in the only way they know how (by limiting core counts) and also tightening up the deadlines. This should have the effect of decreasing the average turn around time, therefore increasing the speed at which trajectories are completed, or allowing trajectories to get further advanced in the period with which they have to get results.

Limiting by core counts limits the number of WUs that might be dumped from people trying unsuccessfully to meet the new deadlines (but is imperfect because old 4p systems might fail to meet the deadlines, while some newer 2p systems might still meet them).

Your take sounds quite accurate.

Dr. Kasson has stated that the core count threshold change will have to also have shortened deadlines for BA, as well.

[orion]

1) There is NO BOINC project that will give you nearly as many points for your BA rigs, as FAH does. Check it out!

Not speaking for Grandpa_01 because he is more than able to do so for himself but you can't compare F@H points to BOINC points, that's like comparing the yen to the dollar...at the end of the day the only thing that you can buy with any of it is more e-penis.

[Grandpa_01]

Adak

I was just running boinc last month and guess what I was #1 in Roseta and Poem on the same day for a 3 day period of time. I was #20 in WCG for the time I ran it. And guess what a GTX 680 makes more PPD over at boinc than 7 - 4P's make in a day. The point is more server class donor hardware runs here at FAH than at Boinc why do you think that is.

[Rattledagger]

1) There is NO BOINC project that will give you nearly as many points for your BA rigs, as FAH does. Check it out!

Since BOINC-projects doesn't give points but "credits" (or really Cobblestones but most call it credits) of course FAH gives more points/day than any BOINC-project.

While where's huge differences in crediting between BOINC-projects, within the same project it's normally fairly consistent over time. Meaning, my i7-920 running BOINC-project X gets roughly the same credit/day now as when I bought it back in March 2009(*). Running BOINC-project Y gives another credit/day than project X, but again it's small variations in credit/day now within project-Y for my i7-920 compared to back in 2009.

2) There is a HUGE disparity between SMP points and BA points. That gap needs to be closed, somewhat. Most of it should be closed by an increase in SMP points.

If there's one thing FAH doesn't need is even more inflated points.


(*): Note, this is by running win-64, where is instances where someone running Linux or Mac having an advantage or disadvantage for some time and this has been removed later on, meaning there can be large variations in credit/day over time for Linux/Mac-users.

[Adak]

Adak

I was just running boinc last month and guess what I was #1 in Roseta and Poem on the same day for a 3 day period of time. I was #20 in WCG for the time I ran it. And guess what a GTX 680 makes more PPD over at boinc than 7 - 4P's make in a day. The point is more server class donor hardware runs here at FAH than at Boinc why do you think that is.

That's EXACTLY what I am saying - your 4P's can't do any better on any other project, than they do folding on FAH BA wu's. I know that GPU's romp on BOINC, but not all projects have wu's for GPU's. Some are cpu only.

It sounded like he was indeed comparing BOINC project credits to FAH points, Orion.

[tear]

I should also point out that one could equally conclude that BA work units are overvalued rather than that SMP are undervalued. A number of people have made this point, and it is not clear (and not entirely in my hands) which direction the points scheme will move. I would personally be in favor of a larger realignment, but that is of course complicated.

Thank you for responding.

I would say that as far as the value of BA's go it is really up to you and the research that you are doing as to their real value, not other donors. Though the BA donors are not a large segment of F@H would you really want to alienate them more by cutting BA points. They have spent allot of money in hardware and electrical cost to help in your research.

This.

To reiterate orion's points:

First: research should be driving the value of work units, not donors.
Second: generally, lowering the reward is not taken kindly to, so inflation is inevitable.



Does inflation lessen the value of past contributions? Sure does. But a MacPro was state of the art.... once.
Similarly, today's top performers will eventually become mediocre... then poor.... then obsolete. Facts of life.
Folding@Home project should make sure that this... process is as smooth and as little surprising to donors
as possible.



I have no idea what the project desires due to repeated vague statements but let's hypothesize a bit (just
to demonstrate what's on my mind):

Vijay, you wanna do more SMP (or whatever) work, present a case before donors and create appropriate
incentive.

Peter, wanna make sure bigadv units get done faster, do the same. Explain where you're coming from,
make a case, have a plan (what was presented, sadly, doesn't qualify as such).

Yeah, it takes an effort, it also means you'll need to organize yourselves a bit better but the reward for the
project will be (should be) increased participation (or at least no participation losses -- sic!).

Otherwise you'll just alienate more and more donors. Not necessarily by action... but lack thereof.

I'm also concerned that the new position is just a way for you guys to avoid engagement. I can only hope
it won't be the case. Time will tell... it always does.

Anyway, I'll ask again --

What is the reason for cutting off the lower end of bigadv spectrum? We asked for explanation and more useful
statistics, got neither so far.

As the announcement does not help bigadv folders plan, when will new preferred deadlines be provided?

As every bigadv-capable machine will eventually "expire" (cease to be capable), how do you plan to handle
that with regard to (perceived or not) lower value of subsequent (non-bigadv) contributions?

[sbinh]

...

What Everyone cares about is having their donation of considerable time, investment and ongoing electricity donation recognized.

To date I see no recognition on the part of PG that this has again been horribly handled.

...

They give you points to recognize your donation.
What their points work? Nothing.
Can you use their points to pay for electric bill ? NO.
Can you claim for tax deduction? NO
Do they owe you anything? ... they think they don't. They let you USE their program for FREE -- ain't charge for a dime.

They are University Professors / Researchers , and you are (like you said 9th grade dropped out) -- Want to be respect? Never happened.


Today will be my last batch of BA WUs submission. Turned off my rigs until PG get things resolve. Could be that my rigs would be come obsolete with core requirement by then

P.S: I might be BANNED from this forum...