Increasing efficiency of FAH

[naapi]

I have an idea to increase efficiency of certain FAH clients FAH by some substantial factor.

I think Stanford should consider to add an option to clients that would specify what is the preference of WUs to the particular client/user. This could enhance productivity - possibly by as much as 5-10%, as users would get WUs that better fit their hardware/sofware/connection.

How this would work:
1) At the time of client configuration, the user would specify the project number (such as 2653, 3065) that is most preferred, second most preferred and, say third most preferred.
2) This information would then be supplied to Stanford assignment servers which would then take this into account when chosing which WU to supply to any given user/client.

Notice that I'm thinking about user SUGGESTING a work unit, not CHOOSING the WU. The choice would still be in the hands of Stanford. If there are no WUs that are most preferred, second and then third choice should be supplied. This could also help the revisions of points for WUs that nobody wants. If nobody wants a WU that means that the project has been benchmarked on a machine that is not representative of the folding community and the points should be changed.

What is the reason for such a change?

- The clients operate on different systems and different types of software.
- The proportional times to finish certain WU differ by sytem.
- As (I understand this is the case) the WUs are currently asigned randomly, this creates inefficiency as the WUs are not assigned optimally.
- This inefficiency can be eliminated, or shrunk by assigning the WUs that fit best to system (as determined by user).

Lets assume we have two systems/clients (1 and 2) and projects/WUs (X and Y). If the efficiency of the systems are as follows (say in PPD):

X Y
1 1200 1500
2 1200 1000

then by giving each system (client) a WU that it prefers (Y for 1 and X for 2) we can increase the efficiency by roughly 10% ---- [1200+1500] / [(1200+1500)/2+(1200+1000)/2]-1. This assumes the clients get supplied the WUs at 50/50 rate.

From my experience with SMP there are easily discrepancies of 30-50% in PPD between my system and points assigned by Stanford on their benchmark machine, so 5-10% increase in efficiency could easily be expected.

This would work for any client that uses different types of hardware (SMP, GPU, etc.), but not for PS3.

M.

[dschief]

I have an idea to increase efficiency of certain FAH clients FAH by some substantial factor.

I think Stanford should consider to add an option to clients that would specify what is the preference of WUs to the particular client/user. This could enhance productivity - possibly by as much as 5-10%, as users would get WUs that better fit their hardware/sofware/connection.

How this would work:
1) At the time of client configuration, the user would specify the project number (such as 2653, 3065) that is most preferred, second most preferred and, say third most preferred.
2) This information would then be supplied to Stanford assignment servers which would then take this into account when chosing which WU to supply to any given user/client.

Notice that I'm thinking about user SUGGESTING a work unit, not CHOOSING the WU. The choice would still be in the hands of Stanford. If there are no WUs that are most preferred, second and then third choice should be supplied. This could also help the revisions of points for WUs that nobody wants. If nobody wants a WU that means that the project has been benchmarked on a machine that is not representative of the folding community and the points should be changed.

What is the reason for such a change?

- The clients operate on different systems and different types of software.
- The proportional times to finish certain WU differ by sytem.
- As (I understand this is the case) the WUs are currently asigned randomly, this creates inefficiency as the WUs are not assigned optimally.
- This inefficiency can be eliminated, or shrunk by assigning the WUs that fit best to system (as determined by user).

Lets assume we have two systems/clients (1 and 2) and projects/WUs (X and Y). If the efficiency of the systems are as follows (say in PPD):

X Y
1 1200 1500
2 1200 1000

then by giving each system (client) a WU that it prefers (Y for 1 and X for 2) we can increase the efficiency by roughly 10% ---- [1200+1500] / [(1200+1500)/2+(1200+1000)/2]-1. This assumes the clients get supplied the WUs at 50/50 rate.

From my experience with SMP there are easily discrepancies of 30-50% in PPD between my system and points assigned by Stanford on their benchmark machine, so 5-10% increase in efficiency could easily be expected.

This would work for any client that uses different types of hardware (SMP, GPU, etc.), but not for PS3.

M.

The assignment servers already do a really good job of screening;

The number of Processor cores is identified, the amount of available memory is noted.
And after every completed Wu the performance factor is updated, indicating how fast it was completed in reference to the benchmark.
And as far as requesting, well my friend that will never happen. there is an old saying we used to throw around
when dealing with Uncle Sam. just substitute " Stanford" where applicable

" The needs of the "Navy/ARMY/ Stanford" preclude members wishes!

[John Naylor]

Plus if users could select a preferred project then any projects which are thought to give lower PPD values will take much longer to complete as seemingly "fast" projects are completed first as users choose them. The v6 clients can also detect the processor in the machine they are being run on, and eventually the Assignment Server code will be updated to allow this information to be used to better pair work units with machines, which would make your proposal, however good, redundant - the Pande Group's method would mean that "slow" projects will also get completed within a reasonable time.

[^w^ing]

the assignment system needs some serious updates, its nothing unusual that my amd gets double gromacs WUs and my intel gets amber WUs :/

[7im]

the assignment system needs some serious updates, its nothing unusual that my amd gets double gromacs WUs and my intel gets amber WUs :/

Other than the known and purposeful DGromacs legacy issue, can you provide another example?

[^w^ing]

you want more? this is quite enough because it leads to a major slowdown of these projects. when i switch these wus between computers, one is done cca 50% faster and the second one is done 30% faster. is that the purpose you were talking about? i thought rapid work turnaround was important. and i also thought thats why there are high performance clients i guess i was wrong.

[naapi]

Other than the known and purposeful DGromacs legacy issue, can you provide another example?

7im:

One more example:

I have 3 Q6600 quads - 2 SLACR and 1 SL9UM. The SL9UM cannot finish ANY WU belonging to 2665 project. It goes to 100% then finds an error and deletes the WU. It happens regardless of Run, Generation or Clone. It finishes any other SMP WU - 2653, 2605, 3062 or 3064 with no problem. It is not overclocked, in fact the memory is even underclocked.

If I could suggest a project to choose I would likely increase this quad's efficiency by 20-30%. I even had a week that I wouldn't finish a single WU - I got only 2665s and all were lost

M.

[naapi]

Plus if users could select a preferred project then any projects which are thought to give lower PPD values will take much longer to complete as seemingly "fast" projects are completed first as users choose them.

John,

I did not say "SELECT", I said "SUGGEST", there's big, big difference between supplying additional information to Stanford and deciding for them.

Additionally - if my computers crunch one project WU for a better PPD vs. another project's, they are not "seemingly" faster, this is a proof that my system crunches one project vs. another relatively faster than Stanford's benchmark computer. Whatever Stanford does with this useful information is up to them, I'm claiming that there are 3 choices:

1) Ignore the information, as currently - which leads to lower overall performance of the system and mildly annoys the contributors that feel that something is not fully optimized or fair.
2) Embrace the information by changing the assignment of the units, which improves the efficiency of the system and improves perception of fairness.
3) Change the points awarded to different projects, which does not change the efficiency of the system, but increases the sense of fairness.

As you see the current choice (lack of action) seems to be the worst solution of all.

As to the argument that some projects would be finished later than others, if the preference of folders is taken into account I have the following comments:
1) It could be assured by proper algorithm that all the projects are finished at the same time or faster, even when folders suggest the project for them to fold.
2) It is WRONG for Stanford to assume that their one "benchmark" machine properly reflects the universum of all the hundreds of thousands of folders around the world. This wrong assumption creates inefficiency in the whole system and it can be eliminated, not sacrificing any of the projects.

I would understand, if Stanford replies that they have more urgent projects to concentrate on, such as adding new clients. I see that new clients, such as NVIDIA GPU can increase overall FAH productivity by 30% or more, so that's clearly a priority. But if they do not have more urgent productivity-enhancing projects, that seems to me to be an area to at least look at in details.

Regards,

M.

[naapi]

Other than the known and purposeful DGromacs legacy issue, can you provide another example?

7im,

I just recalled one more example - Core_a2 for Linux is supposedly crunching project 2662 much faster (PPDwise) than other projects crunched by Core_a1 (2605, 2653, 306x). I have tried this and realized that it is folding much slower on my systems - contrary to what many folders claim. So I cancelled "advmethods" flag and returned to a1. As some claimed that 2662 crunches even faster than the venerable 2605/2653, I feel that my system was at least 30-35% slower than their system, as 2662 was taking ~50% longer to finish (PPDwise). 2605 takes on one of my systems 27.5 mins to finish 1% of a 2605 WU (for 1760 points), while 2662 was taking about 47 minutes (for 1920 points).

M.

[DriveEuro]

Other than the known and purposeful DGromacs legacy issue, can you provide another example?

I have 3 Q6600 quads - 2 SLACR and 1 SL9UM. The SL9UM cannot finish ANY WU belonging to 2665 project. It goes to 100% then finds an error and deletes the WU. It happens regardless of Run, Generation or Clone. It finishes any other SMP WU - 2653, 2605, 3062 or 3064 with no problem. It is not overclocked, in fact the memory is even underclocked.

Obviously somthing wrong with the B3 stepping on your end. Reinstall or update the client, that's an operator error/install error there.

[naapi]

Obviously somthing wrong with the B3 stepping on your end. Reinstall or update the client, that's an operator error/install error there.

DriveEuro,

I have reinstalled the client with the Aug 1 upgrade - there was no change in behavior. Just yesterday I got another 2665, it crunched all the way to 100%, then deleted the WU for no points awarded whatsoever and loaded antother one. The client desn't seem to be the issue.

M.

[John Naylor]

Plus if users could select a preferred project then any projects which are thought to give lower PPD values will take much longer to complete as seemingly "fast" projects are completed first as users choose them.

John,

I did not say "SELECT", I said "SUGGEST", there's big, big difference between supplying additional information to Stanford and deciding for them.

I know, hence why I said "preferred" which still imples that Stanford have the final say - I did read you properly first time around.

There is also another problem - projects do eventually complete, and every time the preferred ones complete, the client should be reconfigured (I know it doesn't have to be as this is still a preference rather than a demand but still) - a major pain for those of us running large numbers of clients (I'm not such a person but I would imagine it would be annoying for those that do.)

[naapi]

There is also another problem - projects do eventually complete, and every time the preferred ones complete, the client should be reconfigured (I know it doesn't have to be as this is still a preference rather than a demand but still) - a major pain for those of us running large numbers of clients (I'm not such a person but I would imagine it would be annoying for those that do.)

That's a fair comment. From my experience, the SMP projects run for longer than 6 months - I guess adding 1 number to each of the config files every 6-12 months would not be an issue for most folders. For me it would be less than 1% of effort that I spent controlling my rigs...

M.

[Tigerbiten]

My thinking is that this idea does not take Murphys Law into account.

What happens if you set this flag/switch and something borks in the servers.

I still remember the complaints from when the QMD servers let their work-units into the wild.

Luck ...............