Proposal to the PG: Optimizing Performance using v7

[Jesse_V]

I have been thinking about the Pande Group's goals and policies, and the reasons behind their actions. There have been some recent changes to -bigadv that have led to extensive discussions about these reasons. It seemed that both the PG and the donors were debating about performance, which involved discussions of deadlines and minimum system requirements. I thought about its implications to lower classes of WUs. To me, it was emphasized by the v7 client auto-configuring a setup in an apparent attempt to gain the most scientific ground using that computer, while making F@h much easier for a layman user. This user may not necessarily pay attention to some of the more subtle aspects of F@h, such as the deadlines that come into play especially for high-performance clients, which the v7 client installs by default. I propose a compromise that I believe finds the middle ground and should maximum scientific production for the newcomer. As this is relevant to my proposal, let me first list where I believe the Pande Group stands, so that I can explain why I believe this could be in their best interest.

It seems that the Pande Group makes their decisions on the following grounds:
1. Computer should be utilized to the greatest reasonable degree possible (emphasizing user choice of course)
2. Each type of Work Units should be run on the most appropriate hardware available
3. Reasonable efforts should be made to minimize user interference or maintenance
4. Exceeding deadlines is detrimental to science and should be discouraged/prevented
5. Completing WUs significantly before the deadline is preferred
6. F@h should be as simple as possible for the donor
7. Changes to F@h should minimize donor disruptions while maintaining the above
8. F@h should be competitive, fun, and slightly addictive, while focusing on the above
9. Any proposed change must be examined in the long term for its cost-benefit ratio

I propose that the Pande Group implement a feature into v7 that allows the slot configuration to be automatically changed based on measured performance of a small set of WUs. My idea is similar to the "Performance Percentages" idea in v6 that simply divided the amount of time it took the user to complete the WU by the WU's deadline. However, I can see how that could be an inaccurate measurement, since it only measures one WU. My idea, described in detail below, aims to use multiple WUs, to achieve a better sample into the performance of the system. This performance is not only tied to raw hardware speed but also the system's availability hours. This measurement would all be on v7's end, and should not require a server upgrade, which makes implementation much easier.

Ever since F@h evolved from the screensaver, the uniprocessor has been the main recommended client. While it does do useful work, it acts as a set-and-forget client, and requires almost no interaction or maintenance by the user. However, if the new v7 client detects the appropriate hardware, it installs SMP+GPU. Compared to the uniprocessor, these two clients are very scientifically productive, use the most system resources, and have the shortest deadlines. As soon as v7 goes public, assuming the install policy does not change, high performance clients will now be installed. The v7 client does give a small description what SMP and GPU are, but does not explain about deadlines or system impact. It doesn’t need to, because that would be overly complicated for a user who just wants to donate to this interesting project he's heard about. I'm not implying that he should be forced into the uniprocessor, since 7im has informed me earlier that there are many people on the forums asking why their system aren't fully utilized. Thus, I think there is an easy choice.

I propose that the v7 client contain another option, the "Auto" feature, which is selected by default. Users who are familiar with F@h can still jump to "SMP+GPU" if they wish, but "Auto" tailors to newcomers. Although it knows that the computer system is capable of SMP+GPU, it installs only SMP. While SMP does have shorter deadlines, it is very scientifically productive so it would be useful on computers that have the performance and availability to meet the deadlines. The CPU does an excellent job at backing off for the user's other applications. The v7 client should run 10 SMP WUs in this way, and record which how many met the deadline. If 7 or less met the deadline, it uninstalls SMP and installs three uniprocessors (assuming the SMP is on 4 threads). I say "three" rather than "four" because it’s been my experience that three uniprocessors generate more heat (and thus a louder and faster fan) than -smp 4, and this makes sense to me because I can imagine the data overhead in moving information to/from the independent CPU cores. These uniprocessors have much longer deadlines, so it allows the donor to make scientific contributions based on their computer's performance. In addition, when this change happens there should be some sort of message displayed in the client (not a popup) illustrating that their configuration was automatically reconfigured based on their performance statistics. They should also have the option to easily undo the action if they feel their habits have changed. There's no way for v7 to determine if they later qualify for SMP and upgrade them, since SMP WUs are different, change over time, and it's difficult to determine how long it would take to run on their system without actually running it to completion. Benchmarks have been proposed before and then officially rejected. Thus it by far easier to downgrade than upgrade a configuration.

Now, if it is determined that 8 or more of the rolling 10 SMP WUs make the deadline, than GPU should be installed as well if it’s possible. Nevertheless, it too should be measured. So if 7 out of a rolling 10 don't make the deadline, then it should uninstall, again showing the aforementioned message. It's a good idea to delay the installation of the GPU slot like this, as donors may be caught off guard by every computing component suddenly generating heat and revving the fan up at the same time. Better to start at a reasonable level, and go up gradually from there.

Now, given its resource impact, -bigadv should remain completely optional as it stands, so at least now is rather outside this scope of this proposal.

Both of these automatic configurations should leave the computer with the best possible setup given their hardware usage. Back to my list of PG reasons, this completes #1. #2 is also covered because I'm sure there's been internal discussion which led to the SMP+GPU default installation by v7. #3 is covered by SMP, but to a lesser degree by GPU since they difficulty with priority processing. This setup also reduces the violations of #4, while still focusing on #1-3. It does not handle #5, but v7 could be modified to do this if it is a big enough issue. Although the automatic configuration changes could be a bit confusing to a donor who was carefully watching, I think that since most of the decision-making is automated, #6 is reasonably fulfilled. As I have mentioned, those who are familiar with F@h's clients can still choose the setup that is best for them. So long they are aware of that, #7 should still be fine. No real change to #8, although without changes to computer habits PPD is basically maximized, which would be encouraging since v7 now includes obvious links to the stats pages. As for #9, that's not really my call, but as I have noted the servers shouldn't need to be changed, and most of the work would be up to v7, which given its flexible design should be up the task.

I realize that I’ve said something close to this before, but after further consideration I feel that this is a more thorough idea. I hope that the Pande Group reviews and considers my proposal, and I welcome further comments and suggestions.

Thank you for your time in reading all of this,
Jesse V.

[codysluder]

This is a well thought out proposal. I can see a few minor details that still would have to be worked out, but that's one reason we should discuss things like this and offer CONSTRUCTIVE criticism. (I don't believe that any one person can see all the implications or how it might affect some strange setup that they have not considered.)

Is the initial dumping of 8 WUs considered acceptable? I doubt it. Can we find a better way to keep this simple without dumping so many WUs?

With a relatively few words, the Windows installer can give the donor the same simple choices. I don't remember exactly what it says right now, but "(highest performance)" and "(minimum impact)" should be part of the descriptions.

I would also add a new choice called "(computer does not run 24x7)" that chooses one less than the maximum number of uniprocessor slots,. (It should choose one slot if only has one "core".) It would be good to consider chips with 1c/2t or 2c/4t like the Atom which are too slow for smp even if run 24x7.

[Jesse_V]

This is a well thought out proposal. I can see a few minor details that still would have to be worked out, but that's one reason we should discuss things like this and offer CONSTRUCTIVE criticism. (I don't believe that any one person can see all the implications or how it might affect some strange setup that they have not considered.)

Is the initial dumping of 8 WUs considered acceptable? I doubt it. Can we find a better way to keep this simple without dumping so many WUs?

With a relatively few words, the Windows installer can give the donor the same simple choices. I don't remember exactly what it says right now, but "(highest performance)" and "(minimum impact)" should be part of the descriptions.

I would also add a new choice called "(computer does not run 24x7)" that chooses one less than the maximum number of uniprocessor slots,. (It should choose one slot if only has one "core".) It would be good to consider chips with 1c/2t or 2c/4t like the Atom which are too slow for smp even if run 24x7.

Thank you for your response. I chose to use the initial dumpings of eight WUs because the alternative in my mind was to include another option in the installation. Also, if the 8 WUs are completed on time, then it's scientifically beneficial. Note that the 8 also coincides with the minimum percentage needed to get bonus points, so clearly there's something about that number the PG likes. I like your idea about the high-performance to minimum-impact choices, but we have to be REALLY careful that we don't overly complicate a setup. It's supposed to be simple after all. As an example, I proposed on the first page of the v7.1.38 Release topic that the install screens should ask the user if their computer is a desktop or a laptop, so that it can make a choice about battery usage. 7im responded with "Please no additional prompting! The install needs to stay simple. ..." And this is what I've been aiming for. I'd like to achieve both high performance and minimum impact. We just have to decided if your additional prompting idea is less confusing for newcomers than my automated reconfiguration. Are you sure it would be?

[VijayPande]

I'll pass this along to our programmers. We have frozen features for the v7 client (you have to do that, or otherwise it will never come out), but this is very good feedback for future feedback.

[Jesse_V]

I'll pass this along to our programmers. We have frozen features for the v7 client (you have to do that, or otherwise it will never come out), but this is very good feedback for future feedback.

Thank you very much Dr. Pande!

[bruce]

This is the sort of ideas that can potentially improve V7 significantly (if you've considered all the possible ramifications). I don't think a feature-frozen client should interfere with additional discussion.

Assuming that a 20% failure rate is acceptable (and I'm not convinced that's what the PG wants), if the first four SMP WUs are not returned before the preferred deadline, reconfigure. (Three out of four is 75% which is already pretty close to indicating it should be a non-SMP installation.)

[Jesse_V]

This is the sort of ideas that can potentially improve V7 significantly (if you've considered all the possible ramifications). I don't think a feature-frozen client should interfere with additional discussion.

Assuming that a 20% failure rate is acceptable (and I'm not convinced that's what the PG wants), if the first four SMP WUs are not returned before the preferred deadline, reconfigure. (Three out of four is 75% which is already pretty close to indicating it should be a non-SMP installation.)

If there are important ramifications which I have not considered and which are publicly known, please let me know. Yes, perhaps five SMP WUs would be better. Less of a sample, but there would be less WUs that potentially exceed the deadline. I guess I was going for 10 because that's what you need to complete to get bonus points (completing 10 shows you're serious about running SMP) but perhaps five would work better in this case. It should be either five or ten, both have pros and cons behind them. We also have to make sure that we don't downgrade a configuration prematurely, since as I've stated it's hard to go back. With the variations in SMP WUs, is five enough of a sample?

[MtM]

Depends on assignment logic/hw detection. Some people can run all fahcore's except one which always gives troubles. If the client could indicate that a certain core keeps failing and prevents assignments of work units which use that core, the threshold would change.

With current logic in place, this doesn't apply as afaik the client can't influence assignments in this way.

Also, some wu's are bad, you would have to wait for confirmation of it being a individual hardware issue and not a bad wu by waiting on if another person is able to submit the completed work unit. Think I said before it would be usefull if the 'problems with a particular wu' section's 'intended usage scenario' would be automated. All wu's which fail are marked as potentially bad, not requiring moderator involvement, and after reaching a preset limit of failures for multiple donors, take the wu out of circulation and if needed inform the project owner. That would fall nicely into place with the 'autoconfigure' suggestion.

I hope btw that this suggestion won't make it impossible for users to configure as they want as allot of people don't need a babysitter. I would like a 'simple' and an 'advanced' setup wizard if it comes to that. Infact the installer I had in mind for when my project is much closer to completion ( as I heard it would be allowed to make custom installers as well, I think only in the sense that you could install the official client with an option to not configure FAHClient and install FAHControl idk for sure atm ) will consist of these options. If any more good suggestions are done in this thread I will probably attempt to incorporate them into the project as well.

[bruce]

MtM: I think you missed one important point. As I see it, this would only apply to new installations and only of the donor accepts the default installer option. Experienced donors would still be able to choose other options, and if it's a reinstall, you probably would save your old config.

Jesse_V: Did you intend to do an automatic configuration change if this client has completed 100 WUs and only 79 of them were successful? That's something that might or might not be a desirable feature.

In any case, the local client would have to keep a count of bonus WUs and bonus WU failures. V6 installations do count locally produced WUs but it never distinguished between success and failure. Certainly it's only local WUs that count. Success or failure on another machine won't be considered even though the QRB looks at totals.

[jcoffland]

Adding this logic to the client could significantly complicate things. I foresee a lot of corner cases and am not convinced that we can always get a good measure of performance in this way. I don't like the idea of automatically changing the configuration and potentially breaking things several days after the software is installed. This would also be difficult to test since you have to wait for 8 WUs to complete before the reconfiguration happens.

@Jesse_V, If I understand you correctly your main concern is that low-end clients will run SMP WUs that they cannot complete in time with the default configuration. This could also be solved by the AS and by offering SMP WUs that make more sense for low-end clients. SMP is still probably better than uni-processor for low-end multi-CPU systems but the current points and bonuses probably don't reflect this. There is seldom if ever a good reason to run multiple uni-processor WUs as opposed to SMP.

We have discussed running a performance measuring core periodically for brief periods. Using this performance data the AS could make better decisions and assign better performing clients to high priority projects and avoid assigning WUs that cannot be completed in time. One of our biggest difficulties is correctly assigning points to projects.

The other part is that we need good GPU throttling before running GPU WUs by default makes sense.

[Jesse_V]

Jesse_V: Did you intend to do an automatic configuration change if this client has completed 100 WUs and only 79 of them were successful? That's something that might or might not be a desirable feature.

In any case, the local client would have to keep a count of bonus WUs and bonus WU failures. V6 installations do count locally produced WUs but it never distinguished between success and failure. Certainly it's only local WUs that count. Success or failure on another machine won't be considered even though the QRB looks at totals.

No I do not, so I agree with you that it's not desirable. My plan was to reconfigure if and only if 7 or less of the first 10 WUs after installation were not completed before the Preferred Deadline. Once it passes 10, the configuration should stay the way it is, and v7 continues as normal. It should do this reconfiguring on the one machine it's installed on (count of local WUs) but shouldn't need to get information from other machines or some other database, unless you think that would be beneficial somehow.

Adding this logic to the client could significantly complicate things. I foresee a lot of corner cases and am not convinced that we can always get a good measure of performance in this way. I don't like the idea of automatically changing the configuration and potentially breaking things several days after the software is installed. This would also be difficult to test since you have to wait for 8 WUs to complete before the reconfiguration happens.

@Jesse_V, If I understand you correctly your main concern is that low-end clients will run SMP WUs that they cannot complete in time with the default configuration. This could also be solved by the AS and by offering SMP WUs that make more sense for low-end clients. SMP is still probably better than uni-processor for low-end multi-CPU systems but the current points and bonuses probably don't reflect this. There is seldom if ever a good reason to run multiple uni-processor WUs as opposed to SMP.

We have discussed running a performance measuring core periodically for brief periods. Using this performance data the AS could make better decisions and assign better performing clients to high priority projects and avoid assigning WUs that cannot be completed in time. One of our biggest difficulties is correctly assigning points to projects.

The other part is that we need good GPU throttling before running GPU WUs by default makes sense.

I'm sorry that it would be difficult to implement. In it's reconfiguration, can't v7 run through the same sequence that a user would regularly use to change a slot configuration? If that regular sequence of button-pressing and menu selection run fine, then some sort of behind-the-scenes action that does the same thing should work as well. I didn't consider the possibility of bugs/errors in the reconfiguration process.

I'm a sophomore studying Computer Science at Utah State University, and nearly all of the computers around me are laptops. I'm running F@h quite well on one myself. But not all of my friends and neighbors leave their laptop in one spot 24/7 as I do. I got one of my friends to join F@h, and he ran into a bit of trouble with pause-on-battery not being selected by default, and I came to realize that that his laptop usage would be appropriate for uniprocessors but not SMP+GPU as it had default installed to. I should have given him further instructions regarding both of those issues, but I shouldn't need to. I understand that we need to keep the installation simple, and at the same time universities power a good portion of F@h's research (BYU is one of the top teams) so I thought this was the best middle ground there.

I thought that even though a system may have enough cores to make it look to v7 and the servers that it works for SMP, it may simply not be available enough of the time. My concerns rest not only on low-end machines, but more on normal machines that simply aren't available for enough time to complete SMP WUs before they expire. For those who don't adjust this for F@h (they just want set-and-forget like the Main client) they won't be able to complete the SMP WUs on time. I thought this could be easiest on v7 since server changes can be disruptive and difficult, and I thought that it would end up being less disruptive to have v7 handle this whole thing.

The problem that I see with brief benchmarking like that is that it only measures performance over a short period of time, kind of like LINPACK. Running a small set of WUs measures the system performance over real-world tasks that it will continue to get. Also, running that set of WUs won't be detrimental at all if the system is up enough and powerful enough to complete them just fine.

I'm also looking forward to GPU throttling. Thanks for the response.

[bruce]

There is seldom if ever a good reason to run multiple uni-processor WUs as opposed to SMP.

Actually, the one good reason is deadlines. Multiple uniprocessor WUs probably have deadlines 5x to 10x as long as SMP WUs. Most of the people who post on the forum run their computers 24x7 but there are a lot of computers which are used for email and browsing an hour or two a day. SMP projects assigned to those computers will all expire while that hardware with that pattern of usage can successfully complete uniprocessor projects.

Computers with that pattern of usage need a question about "How many hours per week will this computer fold?" together with a determination of FAH's processing speed. The results of that determination would be one of the following
A) The computer should not be used for FAH. (A FAH processing total of less than about 400 MHz x 60 x 60 x 24 per day)
B) The computer can finish WUs with long deadlines relative to the processing demands (traditionally uniprocessor WUs)
C) The computer can meet shorter deadlines relative to the processing demands (traditionally SMP WUs)
... and for that matter...
D) The computer can meet the really large processing demands relative to the deadlines (traditionally the Bigadv WUs)

It's certainly not clear whether the choice between B and C can or should be made by the V7 installer (automatically? or not?) or by the donor or by some other piece of software somewhere else, but there's a good reason for the choice to be made. How it is handled has a lot to be said about simplifying the process of getting a new FAH donor started along the right path. Jesse_V's suggestion is KISS aimed at helping the newest of donors to make the right choices without a long series of errors or a complex educational process. V7 goes a really long way toward that goal, but there is still room for improvement, though the cost of doing it needs to be considered, too, compared to the cost of not doing it.

That's why I support continued discussion.

[Jesse_V]

There is seldom if ever a good reason to run multiple uni-processor WUs as opposed to SMP.

Actually, the one good reason is deadlines. Multiple uniprocessor WUs probably have deadlines 5x to 10x as long as SMP WUs. Most of the people who post on the forum run their computers 24x7 but there are a lot of computers which are used for email and browsing an hour or two a day. SMP projects assigned to those computers will all expire while that hardware with that pattern of usage can successfully complete uniprocessor projects.

Exactly Bruce. That's precisely what I was talking about. Of course, Mr. Coffland is correct, since if a system can successfully handle SMP it ought to run that instead of uniprocessors. But the thing is, we don't know whether a system qualifies for SMP or not. In the current setup, if the machine simply doesn't have enough cores, then clearly it can't handle SMP. However, that's just part of the picture. Bruce is right, the deadlines play a large role here, and that's exactly what my post was targeting. I'm trying to carefully put myself in a newcomer's shoes, and think about what they are looking for. They want to participate in this interesting project, and they want to do so with very little to no inside knowledge into how F@h works, and they don't want F@h to be technically complex. Bruce said it well, but I'll say it again: v7 makes impressive headway here, but I'd like to take it one step further. My approach tries to make things simple with no extra complex installation questions or anything like that. Just because a system runs 24/7 and has the necessary number of cores doesn't guarentee that it can meet SMP deadlines, since the user could be running computationally-intensive applications (games, tools, etc) or there could be some unusual architecture setup that slows down computations in some way.(I've seen bottlenecks in some Dell machines) Both of these circumstances are caught by my automatic reconfiguration idea, but are difficult to determine by v7 or during its installation.

Neither BOINC nor our own uniprocessors have this issue, so I feel like we're really pioneering here. Thus I offer a setup that gets things all squared away for the user, and everything's close to set-and-forget, while still maintaining as much scientific production as reasonably possible.

[jrweiss]

Laptops are a significant subset of computers where multiple Uni slots/clients are better than a single SMP. While a laptop might complete SMPs when plugged in, a single travel period may miss a deadline and eliminate enough work/points to make the multiple Uni slots a preferable option.

[7im]

It is bad form (bad software etiquette) to change software configurations behind the scenes (without user intervention). There is a small minority of people that would complain about this very loudly.

The only option is to shutdown the client when it fails to meet the deadline on "N" number if SMP WUs, and prompt the user to switch configurations, even provide a link to a guide to do so.

As you adequately put, the problem is choice.