Change in BA requirements

[Khali]

Thank you for the post Dr. Pande. I am looking forward to more and meaningful communications with your group.

However, you still have not addressed what brought this whole episode into being. Just why is the BA requirements being changed? I also do not see any indication the change is being put on hold to be reevaluated. All I see is an admission the decision and the process might have been a misjudgement you want to avoid in the future. That's fine but it doesn't cover the current situation.

[Viper97]

Thank you for the post Dr. Pande. I am looking forward to more and meaningful communications with your group.

However, you still have not addressed what brought this whole episode into being. Just why is the BA requirements being changed? I also do not see any indication the change is being put on hold to be reevaluated. All I see is an admission the decision and the process might have been a misjudgement you want to avoid in the future. That's fine but it doesn't cover the current situation.

I believe the questions will be answered in a while. I'm quite certain this whole episode has awakened the sleeping Pande Bear (and I mean that in a fun way folks!).

I know that changes are difficult but I also understand that when we try to change things we don't send in the shock troops first rather we try less 'invasive' means. Let's face it we don't wield an axe to take out an appendix.

I'll give them time... and I hope if the corrections are meaningful and presented with an understanding that I as a simpleton could understand, I'd be willing to jump back in.

I am glad that the research we are aiding (or aiding and abetting) is public domain. I thank Dr. Kasson for that clarification in a sub-forum. (So... thank you sir!) The thought that the research we help with by donating our time and money and considerable effort would be rewarded by large pharma who would in turn use those discoveries in a patent action (and subsequently jack the cost of medications up for those most needy) would be extremely objectionable to me.

(It may happen but that's life and the courts will prevail.)

[VijayPande]

Sure, here are some more details. I think we've discussed this in the past but donors clearly aren't familiar with it, which I think has been at the root of a lot of the misunderstandings here. It seems we really need to make a bigadv FAQ so this information is in one place and easy for people to find. From the very beginning, our plan with BA was to change the BA requirements as time moves on. We have changed it in the past already. From the donor reaction, I think we have waited too long since this most recent change. It seems that many donors either have forgotten or just never knew about this history (and this also shows us just how much popular the BA experiment was and how passionate the donors are). Something that we weren't expecting.

BA has always been intended to be used for a few calculations/Projects in FAH that requires the most processing power; larger RAM, more cores, and more bandwidth than typical FAH calculations. However, as time goes on, the BA requirements are met by more and more donor machines due to technological advancements. If we wait too long, a large fraction of donor machines will become BA capable (and in terms of the computing power of FAH, a very large fraction of FAH would be in the BA class in that case). In that situation, in order to get any useful work done in FAH, we'd have to make all WUs in the same category as BA WUs (since most donor would not want to run any other type of WU given the point difference as many choose to optimize for highest PPD/Watt ratio). However, that change would just lead to a big inflation of FAH points and also wouldn't give donors with the most powerful machines any benefit for being part of BA. Something that we wanted to avoid.

So, the plan has been (as publicly discussed) to frequently increase the BA requirements. I should make it clear that I don't think it is a mistake to change the BA requirements (that has always been the advertised plan). However, I think we have made mistakes in terms of making sure donors knew this. We tried to give heads up in terms of a few months notice. Unfortunately, it's clear that donors were expecting more like 6 to 12 months notice and ideally a roadmap.

Regarding the increasing of core count from 24 to 32 over a two months gap, this was done to ensure a smoother transition (we felt that the jump from 16 cores to 32 cores was too drastic) and to reach our goal of limiting BA to only the top 5% of donor systems. Given the increase of BA system which exceeded our 5% criteria (which is already a significant expansion of BA beyond our original 1% threshold), this approached seemed to be one way of doing it. It's amazing to see the enthusiasm to build and maintain such powerful systems, the issue was we only had a small amount of BA work and a lot of non-BA work. We chose a simple method of increasing the core count to achieve our target (top 5% of donor systems).

Moreover, a lot of attention was given to BA work which we appreciate but unfortunately, the same amount of attention wasn't given to non-BA work. This created a somewhat unfair representation of non-BA work for some donors. While increasing the points for non-BA work (especially SMP) would seem an "easy fix", unfortunately, it isn't since we take any changes to the point system very seriously. Any changes to it, if not implemented properly, would upset donors across the board.

We've gone over this thread and we hear your concerns regarding the points difference so we are actively discussing it and are looking into methods to improve this to get a better representation of science and points. Unfortunately, it is too early to discuss this now but rest assured, we aren't ignoring those concerns and once we have come up with a plan, will disclose it publicly before it will be implemented. This will allow us to listen to your feedback regarding any proposed changes to the points system.

Once again, I really appreciate everyone who has donated their time and efforts into this F@H Project and I hope that they will continue to do so in the future.

[Jesse_V]

Regarding the increasing of core count from 24 to 32 over a two months gap, this was done to ensure a smoother transition (we felt that the jump from 16 cores to 32 cores was too drastic) and to reach our goal of limiting BA to only the top 5% of donor systems. Given the increase of BA system which exceeded our 5% criteria (which is already a significant expansion of BA beyond our original 1% threshold), this approached seemed to be one way of doing it. It's amazing to see the enthusiasm to build and maintain such powerful systems, the issue was we only had a small amount of BA work and a lot of non-BA work. We chose a simple method of increasing the core count to achieve our target (top 5% of donor systems).

Thanks for the enlightening explanation, that's all useful information and I hope everyone here reads it. I think one of the key pieces of information is that you're after the top 5% of machines. Having that specification is news to me (although it's always been known that it's supposed to be the very best of the best machines) but is helpful to know. When you readjust the requirements, perhaps you could also specify where this number is. For example, you could say "it looks like 10% of the machines have 16+ cores, which is above our threshold of +/- x% from 5%, so an adjustment is necessary to keep thing aligned with our goals of approximately 5% of user machines." A clear explanation and some details is always helpful when making adjustments.

Bigadv is an experimental program, and I think it's really, really neat that FAH has a class for exceptionally hard problems that are very difficult to solve anywhere else. Let's focus on the fact that we're tackling these big problems that really matter to medical researchers.

[Leonardo]

VJ, thank you for directly addressing the current concerns.

[Grandpa_01]

Hopefully this addresses everybody's concerns it does satisfy what was asked for by most in my opinion. I do not believe most forgot about the moving target of bigadv qualifications. I do believe however it is more like the last time we just choose to ignore them, I do not believe that was/is PG's fault while the method of regulation is questionable IMHO, it should just be a deadline adjustment and a published road map that is kept current, that should help alleviate the problem.

I would also like to thank Vijay and the FF staff for leaving this thread open and allowing the donors to voice there frustration without staff interference and moderation, for the most part anyway, that in itself is a positive sign of change and helps make us donors fell valued.

Thanks for the updates and please do not let them go stale.

Grandpa

[bruce]

Thank you Dr. Pande for taking time in responding.

Passions run deep on both sides, hopefully all this will bring out a better F@H experience for us all.

Thanks again to bruce and the other admin for letting this all play out, it must have been tuff.

"Tuff" to see clear statements of talking points and (for the most part) reasoned statements representing a variety of perspectives/opinions? Not at all.

Tuff to wade through all information that has been repeated over and over and over? Yes. Tough to endure the blatant anti-mod sentiment (aka trolling)? Yes.

[Viper97]

Thank you Dr. Pande for taking time in responding.

Passions run deep on both sides, hopefully all this will bring out a better F@H experience for us all.

Thanks again to bruce and the other admin for letting this all play out, it must have been tuff.

"Tuff" to see clear statements of talking points and (for the most part) reasoned statements representing a variety of perspectives/opinions? Not at all.

Tuff to wade through all information that has been repeated over and over and over? Yes. Tough to endure the anti-mod sentiment (trolling)? Yes.

As yourself this: If some totally unbiased editor were to process everything that has been said about the actual issues and they could boil it down from 40 pages to something concise, how many words would they need. If they were then to take all the material that at that point had been removed and process it into statements like: Apparently {UserName_X} feels Y and is extremely passionate about all this. thus giving everyone an opportunity to have vented their feelings, would anything really be lost?

Yes... it would be lost. The frustration, the venting and of course the concern over the extremely passionate feelings we have.

And hence a loss for FAH as is being evidenced by the daily machine count and of course the marginalization by moderators of the donors.

[kerryd]

I seen nothing in ether of VJs posts for me to start up my folders.So the power supply's will stay off for the time being.
I get the 5% part but not the 3 months for upping the core count,Or the big drop in ppd when changing a 4p over to smp<< its ether do that or leave them off.So to my way of thinking even if I do fire something up it will only be gpu's and only newest ones gtx 4xx and gtx 5xx are not worth the ppd to power same as smp not worth power to ppd to run them
Thanks VJ for a little more info wish it was more .

[ChristianVirtual]

Vijay,
thank you for your inputs and sharing of thoughts. Much appreciated (and also needed); knowing that you and your team got busy on multiple heavy topics these days to stabilise the project backbone.

What is good to know are some details now with the desired 5% cap for BA systems. I think many knew that BA requirements would be adjusted in theory; just not why, when and how. It getting a bit clearer now (still open the requirements on TPF or additional Core adjustment towards end of year. You might know the ultimate answer to everything: 42 ).

But put all that technical stuff beside I even more appreciate your statements on team weaknesses. Many people (including myself) are good in pointing on others but have difficulties with talking/committing on own (!) weaknesses (as such: communication is also not my most developed skill; so I better don't apply for the PR job ). But its a great sign of professionalism and the first step to overcome once they are understood.
With professionalism comes trust. And so I trust you and your team to come up with a good proposal we can review here; hopefully sooner then later.

I like your/our F@H project and looking forward to contribute/fold more. And I hope that some of the member right now "on vacation" come back soon.

[mdk777]

Regarding the increasing of core count from 24 to 32 over a two months gap, this was done to ensure a smoother transition (we felt that the jump from 16 cores to 32 cores was too drastic) and to reach our goal of limiting BA to only the top 5% of donor systems. Given the increase of BA system which exceeded our 5% criteria (which is already a significant expansion of BA beyond our original 1% threshold), this approached seemed to be one way of doing it. It's amazing to see the enthusiasm to build and maintain such powerful systems, the issue was we only had a small amount of BA work and a lot of non-BA work. We chose a simple method of increasing the core count to achieve our target (top 5% of donor systems).

Were these aim percentages ever made know? Aim top 1% and max 5%?

This is All I pushed for:
A clearly defined metric and a way to track.

Could the percentage of a particular BA core count of a machine be control charted verse the defined range?

Viewing said hypothetical control chart(s) a year or more ago:

"look, 24 core machines are approaching the upper control limit, but over the last year, 36 core machines have only increased 1/4 of their their range. So, if building a rig today, I know that a 36 core machine has 3/4 of its range remaining and I can guesstimate that it will take something like 2 years at the current rate of change(no certainty of future rate of change) before it will limit out." If I don't like that rate of depreciation, I can either look at the cost and commensurate life-span of a 48 core machine(again, using my own beliefs and knowledge{or lack of anticipation}of competitive CPU developments) or simply opt out of the BA competitive challenge .

Providing clear rules and continuous feedback pushes all the responsibility back to the donor where it belongs.

Thanks for the update.

[Rattledagger]

Regarding the increasing of core count from 24 to 32 over a two months gap, this was done to ensure a smoother transition (we felt that the jump from 16 cores to 32 cores was too drastic) and to reach our goal of limiting BA to only the top 5% of donor systems. Given the increase of BA system which exceeded our 5% criteria (which is already a significant expansion of BA beyond our original 1% threshold), this approached seemed to be one way of doing it. It's amazing to see the enthusiasm to build and maintain such powerful systems, the issue was we only had a small amount of BA work and a lot of non-BA work. We chose a simple method of increasing the core count to achieve our target (top 5% of donor systems).

If the client-statistics by OS is anything to go by, http://fah-web.stanford.edu/cgi-bin/mai ... e=osstats2 this doesn't add up, since BigAdv. is limited to Linux-only and even in the impossible scenario all Linux-computers runs BigAdv. this is only 2.6% (has only included cpu-clients in the count). It's impossible, since BigAdv. is atleast 16-core and Linux has 5.4 core/cpu on average, meaning if assumes all BigAdv. is 16-core and the rest is dual-core, the highest possible is 1073 BigAdv-capable computers and this equals 0.6% of all computers (counting only cpu-clients). If 5% of all computers has atleast 16-cores as indicated, this means roughly 90% of these isn't running Linux and therefore isn't BigAdv-capable.

[bruce]

Providing clear rules and continuous feedback pushes all the responsibility back to the donor where it belongs.

There's a lot to be said in favor of such a plan. The shift of responsibility for the details of the roadmap to the Donor community is a real plus.

The enthusiast community is much better equipped to predict the future growth rate of a particular type of donor hardware than the Pande Group ever would be. A detailed roadmap should be stated in FAH-measurable terms such as a percent of WUs or a percent of total PPD or a mean project turn-around-time (etc.) rather than something only indirectly related to their real goal (like core-count or deadline) -- even if the only way the might have to control the final outcome might be to alter the indirect values. Charting the variable(s) they need to control puts that responsibility on the donors.

This might work well for BA. I'm less confident it would be a workable solution to control the growth rate of GPU folding and it would be a totally ineffective way to control something like contributions from the old uniprocessor client or the old PS3 client. There are simply too many non-enthusiasts running uniprocessors who approach FAH from a set-it-and-forget-it orientation. I do doubt that control of those segments of the donor hardware is or ever will be as important to FAH as a good way to manage the growth of the BA resources so I'm not worried about that aspect of the plan.

[Grandpa_01]

Regarding the increasing of core count from 24 to 32 over a two months gap, this was done to ensure a smoother transition (we felt that the jump from 16 cores to 32 cores was too drastic) and to reach our goal of limiting BA to only the top 5% of donor systems. Given the increase of BA system which exceeded our 5% criteria (which is already a significant expansion of BA beyond our original 1% threshold), this approached seemed to be one way of doing it. It's amazing to see the enthusiasm to build and maintain such powerful systems, the issue was we only had a small amount of BA work and a lot of non-BA work. We chose a simple method of increasing the core count to achieve our target (top 5% of donor systems).

If the client-statistics by OS is anything to go by, http://fah-web.stanford.edu/cgi-bin/mai ... e=osstats2 this doesn't add up, since BigAdv. is limited to Linux-only and even in the impossible scenario all Linux-computers runs BigAdv. this is only 2.6% (has only included cpu-clients in the count). It's impossible, since BigAdv. is atleast 16-core and Linux has 5.4 core/cpu on average, meaning if assumes all BigAdv. is 16-core and the rest is dual-core, the highest possible is 1073 BigAdv-capable computers and this equals 0.6% of all computers (counting only cpu-clients). If 5% of all computers has atleast 16-cores as indicated, this means roughly 90% of these isn't running Linux and therefore isn't BigAdv-capable.

I am no mathamation by any means but they may be talking about 5% total work done. I believe it is possible that existing bigadv machines are capable of doing 5%+ of total work. I myself run 400+ cores which would be equal to 200 duel core machines then add the speadup of combining a large # of cores working on 1 task and I can see it being possible.

[troy8d]

If the client-statistics by OS is anything to go by, http://fah-web.stanford.edu/cgi-bin/mai ... e=osstats2 this doesn't add up, since BigAdv. is limited to Linux-only and even in the impossible scenario all Linux-computers runs BigAdv. this is only 2.6% (has only included cpu-clients in the count). It's impossible, since BigAdv. is atleast 16-core and Linux has 5.4 core/cpu on average, meaning if assumes all BigAdv. is 16-core and the rest is dual-core, the highest possible is 1073 BigAdv-capable computers and this equals 0.6% of all computers (counting only cpu-clients). If 5% of all computers has atleast 16-cores as indicated, this means roughly 90% of these isn't running Linux and therefore isn't BigAdv-capable.

I believe the difficulty you are having reconciling these numbers may be based on the assumption that all windows/os x/linux client are running SMP and neglecting to account for the fact that a large portion of these clients are uniprocessor. Correcting for this provides a much better estimate. This estimate still requires assumptions about the cores per machine for both bigadv and smp clients and I trust that PG has much more accurate measurements of these statistics that we can estimate from the client statistics.