Change in BA requirements

[Bill1024]

One major change in the the way bigadv is being handled with core count changes, is not a pattern make.
Two or three events make a pattern.

I just informed my wife from now on I will no longer give her money on the 1st and 15th to pay the bills.
Now I will give you money from time to time as I see fit.
That did not go over so well and now I have to sleep on the couch.
Thanks for the bright idea.

[bruce]

The question everyone is asking:

WHY DO YOU DEFEND AN OBVIOUSLY FAILED COMMUNICATION STANDARD?
In a thread that has moved on to a discussion on the best means to IMPROVE COMMUNICATION in the future.

Nobody is defending a failure, only questioning if it's absolutely a binary result. Everybody is agreeing that communications should be better (maybe on a scale of 0 to 100 rather than True/False)...(or like the last survey I took some number between 0 and 10 with a space for comments.)

As for my road-map, I get a lot of my predictive information from Simtk.org (for OpenMM developments) and from gromacs.org (for CPU-based developments). For the last ?? years, a lot of the changes to FahCores have been based on developments made by those two projects. Not everything done by those organizations does get incorporated into a FahCore nor does it happen soon, but many of the improvements they've released do become a new FahCore eventually. Only by guessing which of their changes are likely to make their way into a FahCore can I predict what might happen next.

In the past, Stanford has used other sources for the code in some now-defunct FahCores (like Tinker and AMBER) and I'm sure they're constantly looking for teams that are developing code that can be applied to the type of science being done by the PG ... provided it can be easily adapted to run as a FahCore, supporting the variations in hardware commonly found in donors' homes.

[mdk777]

Only by guessing which of their changes are likely to make their way into a FahCore can I predict what might happen next.

Yes and your analysis of these sources would fall into what is generally referred to as "expert"
I also enjoy following trends and reading "tealeaves".
However, it is not everyone's cup of tea.

It would be more helpful to the average donor, even the enthusiast donor willing to make substantial donations of capital and power; but maybe not time...to have a more direct PG development log/wiki/open to read forum.

I think a communications director/points co-coordinator/economist will be an invaluable addition to the project and I am sincerely looking forward to it.

Best wishes to all over the Holidays.

[DocJonz]

Core17 WUs drying up is/was a matter of supply and demand. The Pande Group tries to keep enough servers with enough WUs to satisfy everyone's demands but they are at the mercy of unpredictable demands. Once the supply drops below a critical level, it takes both time and effort to respond to that shortage. You can't pre-announce unpredictable events.

Surely by tracking returned WU's there is an element of prediction?? And surely with any planned project, testing of new WU's would be done before being things look like they are going to conk out? But it seems not.

On-topic posts are about "Change in BA requirements" There's another topic about Core17 shortages. Why should I have to tell you your question has already been answered here?

The whole point was - you didn't.

[bruce]

As for a better roadmap, yes, I would like to see that as well. But fah's ability to do that is very limited because we're all a slave to the PC hardware development cycle. [and] working and double the speed of GPU folding. Anyone here know those dates?

Like I explained earlier, I'm not asking about detailed roadmap but for something a donor can rely on.
Specifically, a de-facto guarantee of bigadv-usefulness of donor's hardware for a period of time. 6 months are not too much to ask, are they?

Moore's Law is something you can rely on, even though Moore, himself, said it was unsustainable. Nobody really knows why. Other methods don't work.

I'm going to guess that the planned change is a matter of balance. Suppose the Pande Group has determined that the top 3% or 4% of donor hardware should be allocated to projects in the BA category. How long can any BA donor expect his hardware to remain in the top 3% or 4%? You can't give me an answer that everybody can rely on.

FAH is not only slave to the hardware cycles, it's slave to the donor's build-plans and to add-on software that improves efficiency of one sector of FAH attracting more donors from other sectors. To me, it seems reasonable to guess that every project should get some fair share of the total FAH resources which means there must be constraints on projects in the BA sector. (Projects in the SMP category and projects in the GPU category most likely have quotas too.) Periodically the Pande Group notices that BA is exceeding it's quota significantly and SMP is below its quota. Maybe BA now represents 6% of the clients so they have to find some method trim the BA ranks in a way that reallocates perhaps 2% of the hardware to other projects while keeping donors happy. (Granted: that impossible.)

Ok. Lets postulate that BA must not continue to attracting more and more donors relative to the other projects (whether it's true or not). Let's explore a few ideas. Once they reduced the BA points relative to everything else. Twice now, they have chosen to tighten the deadlines. What else might work?

They can say "Please" but that won't work. Maybe they can figure out a hack-proof method (fat chance) of assigning one SMP project out of every three and forcing you to complete it before getting another BA bonus. Maybe the server can be programmed to simply stop assigning more BA WUs but they'd also have to quit granting credit for uploads too. They could do that periodically until things get back in balance. How about just dynamically reducing the BA bonus until balance is restored. It would make PPD unpredictable but it probably would keep BA from exceeding it's quota.

I'm sure there are other possibilities. Go ahead: Be creative. None of these options is going to be popular, but there has to be something that works with minimal unhappiness. No matter how it's accomplished, both now and the last time they made a change, they stated very clearly that the SMP projects need(ed) more resources which is consistent with my postulate. Projects that fall into both the BA sector and the SMP sector are important to FAH but keeping them in some sort of balance must be also critical.

I know you guys mean well, but you are working very hard to increase the imbalance. Set yourself a goal to increase SMP participation, too, it more-or-less equal proportions.

[mdk777]

well, your premise that balance is desirable or required is somewhat subject.

Why cannot BA research projects proceed at the rate of available donors?

These projects were designed to make use of high CPU core count in single systems. In theory, they benefit from this in-system consolidation to a greater degree than regular smp WU.

The degree that donors want to participate in BA WU should be entirely independent from the backlog in regular smp WU.

The backlog of regular smp WU had no impact on the recent project to utilize google for a specific research objective.

http://news.stanford.edu/news/2013/dece ... 21713.html

I don't see where donors should expect "tying" of the sort you are describing.

Did PG require Google to finish a set amount of backlogged smp WU as a prerequisite before starting this project?

Should GPU work units be limited until all smp WU are completed? Handicapping one research project because a different one is lagging hardly seems optimal.

Again, if the specific project(s) that benefited from high core count cpu no longer exists, then BA is indeed an artificial conceit.
Maintaining it on a false pretense does nothing for the donors or the integrity of the project.

[Bill1024]

They can say "Please" but that won't work. Maybe they can figure out a hack-proof method (fat chance) of assigning one SMP project out of every three and forcing you to complete it before getting another BA bonus. Maybe the server can be programmed to simply stop assigning more BA WUs but they'd also have to quit granting credit for uploads too. They could do that periodically until things get back in balance. How about just dynamically reducing the BA bonus until balance is restored. It would make PPD unpredictable but it probably would keep BA from exceeding it's quota.

I'm sure there are other possibilities. Go ahead: Be creative. None of these options is going to be popular, but there has to be something that works with minimal unhappiness. No matter how it's accomplished, both now and the last time they made a change, they stated very clearly that the SMP projects need(ed) more resources which is consistent with my postulate. Projects that fall into both the BA sector and the SMP sector are important to FAH but keeping them in some sort of balance must be also critical.

I know you guys mean well, but you are working very hard to increase the imbalance. Set yourself a goal to increase SMP participation, too, it more-or-less equal proportions.

bruce
Site Admin

Posts: 15993
Joined: November 29th, 2007, 5:13 pm
Location: So. Cal.

So now someone says that there is a need right now to get more SMP wus done as there is a backlog.
You say Please will not work. How do you know? Has PG put out the call and ask everyone to knock off the back log?
Here is the lack of communication, and no trust in the donors to step up to the plate and help out when needed.
What had happened had the opposite effect. Sadly this is the fifth day in a row I haven't folded any work unit at all. I am not the only one either.
I had not missed a day since 2006 unless the electric was off in my neighborhood.

[bruce]

OK.

Please aim your resources at SMP rather than BA.

(I've got to admit I feel pretty stupid doing this.)

[7im]

You say Please will not work. How do you know? Has PG put out the call and ask everyone to knock off the back log?

He says it, because he knows it. We all know it. PG has asked before and it didn't work. Some of us have been doing this for 10+ years. We know what works and what doesn't.

For example, when I suggested earlier that all the BA people run SMP WUs for the next 30 days so as to delay the next BA core hike they all laughed and joked. It's a serious suggestion but no one seems interested in serious solutions to serious problems. No 48 core people willing to "lose a few points" for their 24 core BA brothers and sisters.

Bill, you were so willing and cooperative to being asked to change BA settings in a few months that you turned off your systems. Ya, asking really worked good there. It's hard to take that seriously.

We're victims of our own human nature. As a result, Stick works, and Carrot works. Nothing else moves the masses. So be it.

[Bill1024]

OK.

Please aim your resources at SMP rather than BA.

(I've got to admit I feel pretty stupid doing this.)

Thank you Bruce.
I am going to go out on a limb and lets just see how the FAH community responds.
And no I for one am glad to see it. Really I am.

[bcavnaugh]

OK, I have 64 Cores on my BigAdv 4P Rig.
Should I run ONE SMP Client on all 64 Cores or break them down to 4 Clients 16 Cores each (1 CPU) or down to 8 Cores and 8 Clients?
Thanks

[PantherX]

Here are some of my thoughts:

As stated previously, I expected an increase of CPU for the bigadv but after Intel/AMD reveled their 8 Cores with 16 Threads CPUs. However, I can't understand why the jump from 24 CPUs to 32 CPUs happens in a span of 2 months with another possible change happening at the end of 2014. Thus, 2014 would have 2 confirmed changes in CPU count plus another potential announcement. From reading multiple threads/forums, I see that donors want to understand the rational behind this. Dr. Kasson did provide some statistical data but it needs further explanation from the researchers themselves so that donors can understand why the change is taking place. If there are 50 Server grade hardware instead of 20, maybe it shows that there is more interest and thus, can be utilized in a better manner instead of sticking to a number. For arguments sake, lets say that bigadv is 2% but there are 6% of 64 CPUs machines. If we go by "sticking by the number", would you prefer to raise the value to 128 or utilize these machines to increase the research being done? I prefer to utilize these machines.

If there is a backlog of a particular type of WUs, how about we have a supply and demand page. The supply would list all WUs in order of either FahCore or SMP, GPU, bigadv and the demand can list what WUs were assigned. The building blocks are present in some cases where we have WUs avail and other columns on the Server Status page. I guess that most donors would like to have some sort of idea so if that page isn't 100% accurate, it isn't an issue as long as it gives a rough picture of what the WU-scape is. Moreover, if Project 6742 has a huge backlog, let that researcher mention it on that page and/or blog about it. Those donors that have uber-systems or even just average systems and believe more in science than in points may configure their systems to fold Project 6742 WUs instead of their "daily" WUs. Maybe, you can develop a "science-enthusiast" group that will only fold WUs which are in the backlog. This benefits everyone. The researchers aren't worried that their research will be delayed and the donors know what the current scientific demand is. The current method of changing priority is too obscure for most donors and can have a negative impact on the donor if not done properly.

A road-map implies an "event" and "time". Since PG never gives ETAs, how about a Priority List? That will include items that list where most of efforts are dedicated to and how that impacts donors. The order of items can be changed as and when needed. Thus, donors can be better informed of what may potentially happen in the future. Currently, there are scattered posts across this Forum which indicates what the future of F@H may hold (bigadv changes, more FahCore_17 WUs, Nvidia JIT, AVX, etc). How about all of that information is gathered into a list and made available on the main site under News -> Future. Important items can be mentioned in the blog and the priority list be updated as and when information is available. Any major issues can be mentioned, e.g. CUDA implementation isn't possible due to lack of Nvidia JIT, etc. That way, new donors don't need to read though several threads to get to know the bigger picture. They simply visit that Future page and can see themselves what the future holds and what major issues are which are holding back the F@H performance. A detailed explanation isn't required. Just a simple easy-to-understand list would be sufficient to many donors.

When it comes to communications between donors, there is room for improvement. Researchers tend to focus more of their efforts on doing science. Developer's priority is on coding. Administrators/Moderators ensure that important information is passed between researchers/developers and donors. Forum members ensure that their teams are informed of changes and report to us any issues. Everyone tries to do their best from their perspective. However, the common ground amongst all of us is that this is usually our second priority, the first being real life. Thus, we all volunteering in our free time (I am grateful to everyone doing this) to help out this project and in the past, it may have worked out fine. However, this project is now bigger than before thus, now there is a need to have someone in real life to manage donor communication and coordinate information flow between different research groups within F@H. By having a person(s) who makes a living out of donor communication, would be invaluable to the project, both now and in the future. Of course, this doesn't mean that those that volunteer their free time aren't needed. Instead, they have an easier time passing along the information and the overall donor satisfaction would be higher than before. I honestly believe that effective communication between multiple parties is the way to a successful collaboration.

Even before 2014 has started, we are made aware of what changes will happen to F@H. I look forward to further advancements in F@H and honestly hope that we see some dedicated personal who can help increase the donor satisfaction which in turn will make the project bigger then before.

[PantherX]

...Should I run ONE SMP Client on all 64 Cores or break them down to 4 Clients 16 Cores each (1 CPU) or down to 8 Cores and 8 Clients?...

One SMP Client should work. If you want, you can break it down into:
1) 24 CPUs
2) 16 CPUs
3) 12 CPUs
4) 8 CPUs
5) 4 CPUs

The reason for the above breakdown is that some SMP Projects are limited to 12 CPUs or 16 CPUs, etc. The reason is that they can't scale up to bigger values since the atom count is too small or some other factors. With the above breakdown, you cover a huge range of SMP Projects.

[ChristianVirtual]

Two days ago I restarted two idle CPU slots; very little. CPU:2 (i3, 3,3GHz) and CPU:4 (i7, 2.8GHz) those CPU normally feed the GPU. But hey, if there is need, sure. That's now CPU:2, CPU:3 and CPU:4

Happy folding all together

[Bill1024]

You say Please will not work. How do you know? Has PG put out the call and ask everyone to knock off the back log?

He says it, because he knows it. We all know it. PG has asked before and it didn't work. Some of us have been doing this for 10+ years. We know what works and what doesn't.

For example, when I suggested earlier that all the BA people run SMP WUs for the next 30 days so as to delay the next BA core hike they all laughed and joked. It's a serious suggestion but no one seems interested in serious solutions to serious problems. No 48 core people willing to "lose a few points" for their 24 core BA brothers and sisters.

Bill, you were so willing and cooperative to being asked to change BA settings in a few months that you turned off your systems. Ya, asking really worked good there. It's hard to take that seriously.

We're victims of our own human nature. As a result, Stick works, and Carrot works. Nothing else moves the masses. So be it.

Well I did not get a heck of a lot of notice and I just spent hundreds and hundreds of dollars just a couple weeks before I found out.
With the colder weather setting in I fired up a coupe 6 core 1045T to work smp just before I heard the good news.
And I had the wife talked into buying me a GTX780 for Christmas, the Shelby GT500 was out of the question.
So I was mad, I was, and I do believe in DC projects so I switched my cores over to the WCG and for less points. Plus it is the Christmas challenge so I switched to help the team too.
The timing just happened to be at the right time,
It's more than points Tim, it's respect, being in on the loop. Having a plan in place as to what to buy and when.
Some times workers switch jobs if the pay or working environment is not to the liking.
Some times workers walk off the job and carry signs. Some times it is the only way to get the point across.