Universal Client

[kzaske]

A problem I see with a universal client that optimizes for you is that some folders will not want 'optimization' for various reasons.
How will the server recognize other factors like environmental heat issues, internet connectivity, other uses you may have planned or such local issues as poweroutages? There would need to be a considerable user interface built in to allow the user to reconfig the client for local conditions.
For example my i7 920 is setup up and capable of doing big adv, and I have done a few of them, but I have frequent power and internet connection issues that make the adv methods A3 a better choice for me personally. Likewise I prefer running at -smp 7 rather than -smp 8 because I also have 2 gpus on this system.
How would the server recognise that a system was fine heat wise running 3 x uniprocessors but 4, or an smp would push it over the edge, despite having the resources available? The end result would be a scaling back of optimizations to the lowest common denominator.
I do support the idea of more easily installed clients as long as they maintain a high level of user configurability not just at setup. I know that the complexity of some of the advanced setups puts people off, and I have struggled with this myself frequently, but it does reduce the risk of people setting up things that they just can't run, or worse the project loading a configuration that the donor doesn't want.

I would hope the same way it is done now, a editable local configuration file.

[7im]

The local file is not supposed to be edited. It's not a simple text file, it's easily corrupted because people make this bad assumption, and you break the EULA when you do that. Enter at your own risk.

Run the configuration again, as was designed. And it's also bad assumption that making a universal client would take away customization options. Fah has always moved forwards, not backwards. I don't expect that to change any time soon.

[k1wi]

I'd like to thank Vijay for weighing in here - it confirms my already strong belief that the client and it's usability is significant importance to the F@H team, even though the core focus of their work is on the science being produced.

I think this thread has encouraged myself to throw out the mindset that can become prevalent once someone has used a particular style/type of software for a period of time, and rather take a 'fresh' , or blue sky approach; just because a client has been run one particular way, doesn't mean it has to continue to run that way.

To that end, I'm going to go down a path of blue sky dreaming, about what I would like to see from a 'universal' client. I want to stress that this isn't a criticism of the current generation of clients, nor of the process behind developing each generation of clients (in fact, the opposite, as an end-user, I am in full praise and 100% in awe of the work of the Pande Group). For the sake of this exercise, I'm describing my own vision of a universal folding client - lets say "future-FAH", which provides all the usability benefits of CLI and/or GUI.

To pick up on Bruce's comments, "This happens simply because at some point during the installation, they start the client with an incorrect setting", he lists two solutions, maintain the status quo, or remove the ability to adjust the settings. But there is a third option - make it easier for the end-user to correctly adjust the settings, in a manner which will give them a better chance of correctly configuring their client. How would I like the future-FAH client differ from current version?

1. Remove flags; I'll be honest, they've caused me headaches and then some - not just for me, but for other members of my team. How would I go about doing this? For the GUI:

a) Create a step-by-step installation process that allows you to select, through a tabbed process, how you want to run your client ie SMP, uni, GPU.

b) In this installation process, allow users to stipulate how many cores they want to use & the % of each CPU they want to use, regardless of SMP or uni-proc (as the OP mentions, allow the client to spawn multiple single core processes if desired.

c) Perhaps let the user add a value (estimated) of the % of hours in a day they generally leave their machine on. Allow the client to monitor the true proportion and update, if the user so wishes, but allow the user to disable auto-maintenance if they realllyyy want.

d) For post-installation adjustments, utilise a separate configuration app to adjust settings once installed, make the default 'end option' ensure that the client finishes off each currently operational work unit and then restart with the new settings/configuration, but in just in case, provide a 'restart client now' for any emergency situations (giving lots of warnings/are you really sure you want to do this to discourage it unless absolutely necessary!).

d) Make 'advanced methods' discrete. Some applications require you to press F5 (and provides warnings etc to ensure you really want to do it) to access advanced options; do the same for folding.

Remove flags the CLI:

a) Replace -config with the above configuration app, obviously designed for CLI (like you get when you're configuring other CLI applications (cant remember the name of the style!), provide all the same settings etc and options. Design the client so it won't run if it hasn't been configured by the config app, but allow for users to roll out a client pre-configured (or copies of a configured client) on multiple machines.

For both GUI & CLI:

1. Webui?
My inspiration for this comes straight out of Linux_FAH's VM; develop a stand-alone app that allows for a webui configuration page, to allow both clients the ability to be configured via webpage. Linux_FAH's is out of the box outstanding.

2. Work unit suitable for a computer - small,normal,big - for an entire computer system.
Create a matrix of variables based on a user's components to spawn a code, which falls within a particular 'bin'. A c2d with 4GBs of ram might fall in bin "3", which reflects work units that requires a comparable amount of computational power. These bins can be quite broad and clients would be able, if necessary, to run smaller binned WUs, so that if there is a shortage of "3"s, it could run a "2" work unit, until another "3" turns up; but it won't run a "4". As the bottom rung of work units becomes 'obsolete' (and this could be run extremely conservatively), the client would be informed that the system is unfortunately not powerful enough to continue providing work to the project.

More, perhaps, controversially, ditch the viewer for computer (ie not PS3) FAH altogether . Spin it out as a 3rd party open-source/community built and supported application or encourage a 100% community built version. (This may actually be happening, I'm not sure!).

Moreover, encourage/facilitate a 3rd-party open-source developers centre. At the moment, there is a wide range of 3rd party apps (FAHmon, HFM.net etc) that can be quite difficuilt to find exist/track down, but some of them are really really quite good and allows those with the skills to contribute to FAH in a way that allows the Scientists to do the Science. A single location for open-source, 3rd party apps governed by, but for which Stanford is not responsible for in any way, and provides no guarantees over etc, in my mind would provide a community for developing these 'enchancement' apps in the same way that these forums provide a community for Folding@home. Perhaps just as there is a team of dedicated and hard working community Moderators here, a team of appropriately skilled community Moderators can oversee the 3rd-party app centre.

So there you have my future-FAH client

Also, one final thing I'd love to see is a user/account/passkey system, where users with passkeys can keep track (through a password system) things like their passkey ratio (to ensure it's above .8) and other information that's not available for the whole world to see in public pages!

Once again, this is just an exercise and in no way a criticism of the current clients/plans. Rather, it's end-user's point of view. You know you have a dedicated community of users when one ponders this over at 3am in the morning (well, it was actually 4am as the clocks had just changed back here ) and I will continue to push my team to continue working our way up the charts! 9.6 million points and counting .

Thank-you for letting me think creatively this quiet Easter weekend, I'm going to go back to working on my insane watercooling system project now; 10m loop and all.

k1wi

[bruce]

We're no longer on-topic, but your post makes it difficult to figure out a good place to split the discussion between "Universal Client" and "3rd Party Developments" or something like that.

More, perhaps, controversially, ditch the viewer for computer (ie not PS3) FAH altogether . Spin it out as a 3rd party open-source/community built and supported application or encourage a 100% community built version. (This may actually be happening, I'm not sure!).

The API / toolkit and the source for at least one viewer were published quite some time back but as far as I know, nobody has taken up the challenge to produce something useful. That leaves FAH managemnt with a difficult choice -- continue to distribute the current viewer or remove the capability entirely. I'm sure it's still around because they hope to improve it someday but there continue to be higher priority work that keeps everyone busy, but it's hard to abandon that dream unless somebody produces an alternative.

Moreover, encourage/facilitate a 3rd-party open-source developers centre. At the moment, there is a wide range of 3rd party apps (FAHmon, HFM.net etc) that can be quite difficult to find exist/track down, but some of them are really really quite good and allows those with the skills to contribute to FAH in a way that allows the Scientists to do the Science. A single location for open-source, 3rd party apps governed by, but for which Stanford is not responsible for in any way, and provides no guarantees over etc, in my mind would provide a community for developing these 'enhancement' apps in the same way that these forums provide a community for Folding@home. Perhaps just as there is a team of dedicated and hard working community Moderators here, a team of appropriately skilled community Moderators can oversee the 3rd-party app centre.

Hmmm. Who would manage such a center? ... or would it be just an un-managed repository where 3rd party developers are encouraged to archive their source?

So there you have my future-FAH client

Also, one final thing I'd love to see is a user/account/passkey system, where users with passkeys can keep track (through a password system) things like their passkey ratio (to ensure it's above .8) and other information that's not available for the whole world to see in public pages!

Once again, this is just an exercise and in no way a criticism of the current clients/plans. Rather, it's end-user's point of view. You know you have a dedicated community of users when one ponders this over at 3am in the morning (well, it was actually 4am as the clocks had just changed back here ) and I will continue to push my team to continue working our way up the charts! 9.6 million points and counting .

Thank-you for letting me think creatively this quiet Easter weekend, I'm going to go back to working on my insane watercooling system project now; 10m loop and all.

k1wi[/quote]

[k1wi]

I would maintain that I'm on-topic: The third party app situation is only part of my future-FAH client, in which I outline, using rose tinted glasses, what I'd like to see included, and what I'd like to see excluded (or more correctly divested). It's just you've chosen to critique the what I'd like to see divested aspect of my vision rather than comment on the things I'd like to see included.

To address your first comment: I agree it is a difficult decision and it easier for me to hypothesise from the sidelines than make the decisions. I see three main 'directions' for it: maintain the status quo and divert coding time/effort/money from the core science to fix the not-quite-100% viewer. 2. Remove the viewer and somehow release what they have as a stand-alone app. In both cases, eventually down the road, perhaps the single standalone/inbuilt app can display all the clients from the one instance in some fancy way). 3. Encourage the community to come together and build a open-source alternative. How to encourage the community? Well I don't know - run a competition, make it a high profile process (say by using a combined forum announcement/forum thread/forum group/blog/twitter) to engage those with the skills. I'm sure there are others from the open source world who know how to pull peoples strings to encourage them to contribute. Maybe the strings are the same as encourages people to participate/contribute in the forums.

To address your second comment, which I'll answer second part first: The simplest and perhaps best method I envision would be a link to a 'Third Party App' page from the download page on folding.stanford. This page could list in categories the different third party apps that provide extra non-core features. Perhaps my future-client GUI would contain a link to the page in a 'link' tab. Made in a way that anyone downloading the client could then easily go to the third party app page. The apps wouldn't necessarily be hosted @ stanford, in fact I don't think they should. But they could link to the sourceforge/googlecode/whoever-they-currently-are-hosted-by homepage of the application. Maybe a requirement is that they're hosted by an appropriate hosting site such as sourceforge (once again, I don't know enough about how easy/able it is for opensource apps to be hosted at sourceforge/googlecode etc).

Who would manage such a centre? Once again, there are a hundred different flavours of how to go about developing the centre, and there would be a cost in setting it up and maintaining it, but so there is too in maintaining the forums, or the wiki. I think the forums work, plain and simple. Through the hard work of those such as yourself, they provide a great conduit between the community/donors and Stanford. So it would probably be easiest to leverage off the forums:

Off the top of my head, perhaps a group of 'third-party developer' (in the same way there are/were? beta-testers) is made for people who have a proven track record of applications, along with a position a long the lines of 'third-party app moderator' made for a community member(s) to oversee the maintenance with the initial oversight of a current mod(s)/administrator(s).

They would then work with whoever it is within Stanford who is best placed to make periodic updates to the third-party app page. Once a developer has met the standards of the moderators, they can request, perhaps through a specific section within the third-party apps section of the forums, that their app, with a brief outline of what it does/how it works, be added the third-party app page by the appropriate liaison person within Stanford. As I said, there is an upfront cost of bringing aboard extra community people/moderators, just as there was in setting up the forums. In my own opinion, as this future-FAH exercise is, I believe it would be justifiable, particularly if it shifts the energy and cost of developing non-central features from the science people, to the community (and from the number of third-party monitoring apps, which is a great example of this, I'd say it is). I for one would put my hand up to help moderate such a section.

So yes, the above is still discussing my universal client a 'future-FAH' client, specifically what I think should be divested from it, and answering your question of 'how do we divest them and what is the justification for doing so?'

k1wi

[PantherX]

As long as it is about FF@H (Future F@H) here are some suggestions that I have:

1) There should be an option within the client itself to start with windows automatically and also an option to include a delay of X minutes.

2) Rather than remove the flags, in the Advance Tab, make all the flags available in a check-box style and each flag should have its own description.

3) A plug-in should be available so that FF@H Clients can have access to SpeedFan data (or any other temperature monitoring software) and there is a constant monitoring of the temperatures of the CPU. GPU, HDD, etc. If any of the temperature crosses a threshold (either default settings or user configured) than that client which is causing the overheating problem should either get paused or should exit (depends on the user's choice). If the client is paused, once the temperature reaches a safe level, it can automatically resume its work. If the client exits, then a message should be displayed on screen willing the user what client was exited at what time.

4) The client displays real-time stats about you.

5) There should be an option called "testing" where new users who just want to test the client and are not sure that they want to continue. With this option, "dummy" WUs should be assigned so that the user will see the effect the client has on the system. The dummy WUs should be available for GPU, SMP, UNI and also in Small, Medium, Large so that all possible combination are covered and the user will know what to expect form their system.

6) Rather than downloading and installing 3rd party apps, you can introduce the add-on feature just like Firefox. The main client is provided by PG and the add-ons are up to the user which to install from the available list that PG knows won't have a negative impact on the science.

7) The homepage of the FF@H should include a "virtual client" that will be installed upon user request and will tell you if your current system is capable enough of running F@H (like virtualmark) if not, a list of hardware that is preventing you from joining F@H should be displayed.

I really hope to see some of these features in the v7 Client

[Nathan_P]

Another suggestion from me - get the client to compress the WU before it goes back to stanford - saves upload time and bandwidth and allows machines to get more work done

[Wrish]

I used WinRAR to compress several wuresults files:

Three A3's shrank from 20.5 MB to 17.4 MB.
Four Nvidia GPU units at ~100-150k each did not compress at all.
One A2 bigadv shrank from 99 MB to 52 MB.

I also compressed incoming wudata files:

An A3 incoming file at ~1.8 MB did not compress at all. (Client always reports decompressing downloads.)
An Nvidia GPU incoming file at 65,142 bytes compressed to 64,479 bytes.
A bigadv incoming file shrank from 30 MB to 26 MB. (Client never reports decompressing downloads.)

Both A3 and bigadv checkpoint files compressed to 85% of their original size.

it seems one improvement of A3 over A2 is that downloads from Stanford are compressed by ~15%. Uploads don't appear to be compressed, but the available compression room for A3 uploads as reported by WinRAR is only about 15%, in contrast to the near 50% achievable for my A2 bigadv sample.

[7im]

There is always a trade off. Higher compression slows down the client, delaying the uploads. It also has to be uncompressed at the other end too. Since they already use compression, would there really be that much improvement?

But again, we're getting off topic again. This is the one client runs all client types thread, not the all client types laundry list for all improvements every conceived thread. Feel free to start a new thread for that laundry list, if you like.

[kzaske]

Perhaps we should rename this thread "Features we would like to see."

[Nathan_P]

We are moving to eventually have a universal client. Our v7 client will be our first step, but it will take a while to get all the way there. Our goal is to make it easy for people to donate computer time, but there are several new developments which have to be done to get there.

Excellent news Vijay - the guys on [H]ardforum have just been discussing this very topic. Please keep us posted with updates.

PS any news on a possible date for v7

[theteofscuba]

There is always a trade off. Higher compression slows down the client, delaying the uploads. It also has to be uncompressed at the other end too. Since they already use compression, would there really be that much improvement?

If it is already compressed there is no improvement.


I believe the general idea of compression is that the time it takes to compress and send the compressed data will be less than the time and banwidth it would normally take without compression. Even when decompressing huge files it isn't all that cpu intensive. I suppose it makes more sense for people on dialup, and then there are people like me who despite having a reasonable download speed, our DSL upload speed is around 30kb/s which will take a SMP2 WU or Protomol WU about 10-15 minutes to upload the results. If that is already compressed, oh well. I'll live.

but this brings an idea to me. it takes a while to upload and the cpu sits mostly idle through the duration of the upload. i'm not saying to use BOINC but im just saying that the way BOINC handles this is really quite interesting to me. In a BOINC world, the work units that need to be uploaded are queued.. we want to try to be running a new WU on the cpu while the finished WU is benig uploaded. the 10-15 minutes of idle time is kind of wasteful in a world where every secound counts.

[Nathan_P]

There is always a trade off. Higher compression slows down the client, delaying the uploads. It also has to be uncompressed at the other end too. Since they already use compression, would there really be that much improvement?

If it is already compressed there is no improvement.


I believe the general idea of compression is that the time it takes to compress and send the compressed data will be less than the time and banwidth it would normally take without compression. Even when decompressing huge files it isn't all that cpu intensive. I suppose it makes more sense for people on dialup, and then there are people like me who despite having a reasonable download speed, our DSL upload speed is around 30kb/s which will take a SMP2 WU or Protomol WU about 10-15 minutes to upload the results. If that is already compressed, oh well. I'll live.

but this brings an idea to me. it takes a while to upload and the cpu sits mostly idle through the duration of the upload. i'm not saying to use BOINC but im just saying that the way BOINC handles this is really quite interesting to me. In a BOINC world, the work units that need to be uploaded are queued.. we want to try to be running a new WU on the cpu while the finished WU is benig uploaded. the 10-15 minutes of idle time is kind of wasteful in a world where every secound counts.

i was thinking more for -bigadv, 10 minutes isn't much but a big unit can take an hour to upload, thats a big waste of time for the kind of machine running bigadv

[theteofscuba]

i was thinking more for -bigadv, 10 minutes isn't much but a big unit can take an hour to upload, thats a big waste of time for the kind of machine running bigadv

It wouldn't hurt to experiment and verify this. But then, maybe if we had the CPU working on a new WU while previous is uploading, it will be better. but to keep my upload speed maxed out for too long, i can't really use the computer -- can't play games, and can't even really visit web sites. it is all too slow.

[bruce]

In the early days of FAH, the results were rather small, so that even at modem speed, they didn't take all that long to upload. [My how things have changed.] When the projects started to grow in size, an option was added to distinguish Small WUs from Bigger ones and we recommended that those with dial-up use the Small setting.

Most folks now have broadband connections of some sort but the size of uploads continue to grow. The client now supports Small/Normal/Big and there's even another classification called bigadv.

There have been a number of enhancement requests suggesting that the client's logic needs to do a better job of simultaneously processing a WU and uploading results. The Pande Group makes all decisions about enhancements. This enhancement has repeatedly gotten pushed down the priority list but I sincerely hope that it gets incorporated into V7. The communications issues that are currently being discussed elsewhere on the forum are getting a lot of attention right now and they accentuate the need for overlapping uploading with processing. (The client is already capable of doing both at the same time, but on it's first couple of upload attempts it doesn't use that option.)

One fact that most people do not appreciate when they compare BOINC with FAH is that it's very important to minimize the total time starting when the WU is downloaded and ending when the result is uploaded. That means that the objective of the original design of the client was to get the results returned FIRST and only then to download a new WU. If you download a new WU earlier than that or upload a result later than that, you'll have two or more WUs assigned to you at the same time, which actually extends the time that Stanford considers critical.

BOINC's goal is to keep the CPU busy, no matter how many WUs you have sitting on your machine at any one time. Deadlines are not as important as they are in FAH. BOINC's scheduling logic turns out to be rather wasteful if you need to minimize the turn-around time for every WU. I think this probably has something to do with the original goals of S@H where there always were plenty of WUs and all could be processed in parallel. Note that FAH has many WUs that can be processed in parallel, but there are very important parts of the FAH analysis which must be processed serially. If you have been assigned Gen (N) of a FAH trajectory, nobody can work on Gen (N+1) until you turn in your results.