Is F@h the 1st DC project to use SMP and PS3?

[Jesse_V]

Hi there. I'd like to know if Folding@home really is the first distributed computing project use write a client to use a multi-core processor and also the first project to utilize the PS3. I've heard this in several places, most notably from 7im:

viewtopic.php?f=16&t=1164&start=135#p137893

Yes, I suppose anything can fail. And yet no other project has succeeded as well as FAH. From world record holder, to first ever GPU client, to first ever PS3 client, to first ever true SMP client... shall I go on?

viewtopic.php?f=67&t=19541#p194945

On the other hand, FAH has build a customized, specialized, and optimized client to do ONE specific task. Fold proteins on various types of hardware. Their goal is to get the best performance while folding. They have no obligation to change their client to allow other projects to do other types of research. That's why FAH had the first client on the PS3. And the first client on GPUs. And the first true SMP client to support multiple processors. FAH leads the way for others to follow.

As of 2007, it looks like BOINC is unable to do SMP processing: http://boincfaq.mundayweb.com/index.php ... 188757c695 but I don't know if that's still true, and there are likely other DC projects that are outside of BOINC. I haven't found any website or scientific publication that says "Yes F@h really did pioneer SMP and PS3 for DC". I mean I trust 7im since he's very experienced and has above-average access to the inner workings of the PG (website, v7 bug tracker, etc) but I'm a bit skeptical. I'm just trying to remain objective and really confirm this. Is this true?

In my quest to resolve this from those who know best, I emailed Dr. Pande a while back and asked him this question, and he said "I'm not sure myself about the precedence here. I mainly pay attention to what we do, so I'm not keep track of the other projects." Which is why I'm now opening this up for input from the community.

I already have a scientific publication and one of Dr. Pande's blog posts that says that F@h was the first to use the GPU, so I've got that pretty well answered. The SMP and PS3 I'm just not sure of. When one is enthusiastic about a project, it can be easy to brag and say that it is, I just need to be sure. I mean a statement by a scientist or an admin could be sufficient, but if anyone knows of any other sources that answer this I'd sure appreciate a link. With F@h's accomplishments, I'd always like to be able to make a claim knowing full well that it is true. Thoughts?

[Jonazz]

I have no idea about SMP, but there are no other distributed computing projects on the PS3 besides F@H. But, through Linux you were able to indirectly run several other DC project.

[Jesse_V]

I read once that there used to be a way to put Linux or something on the PS3 so you could run other DC projects, but there was some sort of update/patch released which prevented that. I'm still looking for the article, would be nice to pull up. Anyway, I've been searching some more and came across some interesting material from 2007 which seemed to support the idea that F@h was the first on the PS3: http://www.foxnews.com/story/0,2933,260748,00.html which was then promptly quoted by 7im: http://arstechnica.com/civis/viewtopic. ... &start=680 Defending F@h like a boss... But it looks like FoxNews just recopied that text from somewhere, so I'm still not sure I trust it. Looking for the original or something better...

EDIT: Actually, maybe people copied it from FoxNews. That seems more reasonable considering that their article has "Published March 23, 2007 Associated Press" at the top and it wouldn't make sense for them to copy/paste from some other website. Hmm.

[bruce]

The Linux that you could install on the PS3 was called YellowDog linux because it was dreadfully slow, with access to most of the CPU resources disabled. FAH uses a Sony designed application which gives it access to multiple cores. I can't say for sure, but FAH might be unique in that regard.

[Jesse_V]

Ahh I see. http://en.wikipedia.org/wiki/OtherOS Thanks so much. How about the SMP? Any ideas on that as well?

[Rattledagger]

I have no idea about SMP, but there are no other distributed computing projects on the PS3 besides F@H. But, through Linux you were able to indirectly run several other DC project.

GPUGRID.net did initially run only on PS3, but since FAH in practice has an exclusive deal for running DC on the PS3, they dropped support for the PS3 and started using GPU instead.

As for GPU & SMP-support, this is included with BOINC v6.4.x released December 2008, one or two GPU-projects was active before this using beta-clients. Aqua@home used SMP but has shut-down now.

For most projects developing a SMP-application would be a waste of time, since SMP always comes with a small extra overhead, and for most projects all work needs to be done, not just a single wu done very fast. While climatepredicition.net did test-out SMP internally, they decided against releasing it since they needs many finished models, and not just a couple quickly-finished models. Getting 10000 finished models in 6 months time is therefore much more important than getting 100 finished models in 1 months time.

[Ivoshiee]

I have no idea about SMP, but there are no other distributed computing projects on the PS3 besides F@H. But, through Linux you were able to indirectly run several other DC project.

GPUGRID.net did initially run only on PS3, but since FAH in practice has an exclusive deal for running DC on the PS3, they dropped support for the PS3 and started using GPU instead.

As for GPU & SMP-support, this is included with BOINC v6.4.x released December 2008, one or two GPU-projects was active before this using beta-clients. Aqua@home used SMP but has shut-down now.

For most projects developing a SMP-application would be a waste of time, since SMP always comes with a small extra overhead, and for most projects all work needs to be done, not just a single wu done very fast. While climatepredicition.net did test-out SMP internally, they decided against releasing it since they needs many finished models, and not just a couple quickly-finished models. Getting 10000 finished models in 6 months time is therefore much more important than getting 100 finished models in 1 months time.

If all other DC projects have dropped supporting SMP or never had one then the FAH is in unique position indeed.

[Jesse_V]

Thanks for the information. I'll be further researching for supporting information. I'm eager to solve this not only for me, but if I make the bold statement online I should provide citations. It seems odd to me that if F@h was the first, surely it would be more obvious and there would be some statement on the website or something. Dr. Pande's blog says F@h was the first to use MPI: http://folding.typepad.com/news/2008/03 ... s-cli.html but obviously there are other ways to utilize multicore processors (like threads) so that's only halfway supportive. Quite a puzzle. I'm not sure what happened to 7im recently, but I'm hoping he'll come back and enter this discussion as well.

[rwh202]

I believe that Milkyway@home is still doing smp type processing.

The N-body simulations use openMT (since early 2010) and certainly uses all available cores.

Obviously, this is after multi-processor support appeared on F@h, but F@h doesn't appear unique.

[Jesse_V]

Well I understand F@h isn't unique in regard to those clients now, since others are doing the same thing now. I'm just trying to determine if it was the first, if it really did pioneer those clients. I've read that PrimeGrid uses GPUs (first link from a Google search: http://www.primegrid.com/forum_thread.php?id=3672) and there's now PS3Grid and GPUGrid. Looks like the PG first released the SMP client in November 2006, so we just need to find out where ALL the DC projects were at that time.

EDIT: Compounding the problem is that I'm not even sure when the first SMP was released! Wow. http://folding.stanford.edu/English/FAQ-highperformance is a bit ambiguous, but indisputably puts it between November 2006 and March 2007. Then on this page http://folding.stanford.edu/English/FAQ-SMP it says 2006. Hmm I'm really looking forward to a revamp of the website, it would make learning and information retrieval SO much easier, especially for newcomers.

[Ivoshiee]

If it shall help you a bit then the finstall script taped out with SMP support December 10-th year 2006:

Code: Select all

#
# Version 5.0
#
# * Added SMP Linux client support. Use the "smp" flag for the SMP Linux
# client installation.
# Note: The FAH SMP client will use up to 4 CPUs. If you are running new 
# installation over the old one then you may need to delete some CPU# directories
# or you end up with numerous FAH SMP clients set up on your system.
#

It is likely around that time there was official SMP release, but betas were available earlier.
I didn't find earlier mention of the Linux SMP either: http://webcache.googleusercontent.com/s ... =firefox-a

[bruce]

As for GPU & SMP-support, this is included with BOINC v6.4.x released December 2008, one or two GPU-projects was active before this using beta-clients. Aqua@home used SMP but has shut-down now.

For most projects developing a SMP-application would be a waste of time, since SMP always comes with a small extra overhead, and for most projects all work needs to be done, not just a single wu done very fast. While climatepredicition.net did test-out SMP internally, they decided against releasing it since they needs many finished models, and not just a couple quickly-finished models. Getting 10000 finished models in 6 months time is therefore much more important than getting 100 finished models in 1 months time.

The statement that "all work needs to be done, not just a single WU done very fast" is inaccurate. First, all of the WUs that FAH distributes need to be done, and second, there is no single WU that has to be done very fast. The fundamental difference is the type of data that is being analyzed.

Take, as an example, the original Seti@home project. A great deal of data was gathered from radio telescopes. Every segment of the sky and every frequency band represented a WU which needed to be analyzed. Data was fully parallelizeable, meaning that it didn't matter which WU was finished first. Work could be assigned in any order and completed in any order. The only real requirement was that there be enough computers to complete the required analysis. (This neglects ongoing collection of radiotelescope data, but that really doesn't change anything.) Speeding up the analysis of any block of data was meaningless. The data has been pre-gathered and is always available to be issued to the next computer. The analysis was complete whenever the last block of data was completed. Ten slow computers were just as useful as one computer that was ten times as fast.

Folding@home (and a few other DC project) do not analyze fully-parallel data. They analyze data that is both parallel and serial. First, they have to have enough WUs identified by Project/Run/Clone which can be run in parallel so that one can be assigned to every computer that participates. When the science permits more parallel WUs to be generated, that's an added benefit, but that's not always easy to do. The critical difference is that WUs are generated directly from a previous WU so the MUST be analyzed serially. You cannot assign Gen N+1 of a particular WU until the results of Gen N have been completed. Each successive Gen represents the next time segment of a particular trajectory and you can only predict the future one step at a time based on the current positions and velocities (etc.). It's this predictive nature of the analysis that forces FAH to employ SMP and GPUs and PS3s.

Every FAH WU is important, not just because they want need the results, but because they also need the end-conditions of that WU to generate the next WU.

The combinations of Project/Run/Clone are called a Trajectory which starts from certain preestablished conditions and proceeds for a "LONG" time, but is broken up into bite-sized WUs that are analyzed serially. If each WU takes an average of N days to process, and you establish that for this project, LONG means M successive bite-sized WUs, then the project must run for N*M days. If N*M works out to be a few months, the project doesn't need any special methods of acceleration. If it works out to be a few years or many, many years, then some serious steps need to be take to shorten that calendar time. SMP/GPU/PS3 are significant tools to shortening that calendar time. In the early days of SMP, a quad processor could reduce the elapsed time from N days to ~0.23*N which ultimately reduced "a few years" to "~.23*(a few years)" Newer hardware continues to increase the number of parallel threads available to FAH, so the value of 0.23 continues to get smaller and smaller, but there still are projects that will require "many, many years" to process, even with the fastest hardware.

Not every protein can benefit from high numbers of threads. SMP is designed to split the protein into multiple segments, one for each available processor. One timestep is analyzed by each processor and then the new positions and velocities are synchronized to that new forces can be calculated for the next timestep. That synchronization process interchanges a lot of data very, very frequently. The entire simulation proceeds at the speed that the slowest processor can manage its segment of the atoms. The number of atoms and the data interchange requirements do put some limitations on what can actually be accomplished.

[Jesse_V]

If it shall help you a bit then the finstall script taped out with SMP support December 10-th year 2006:
...
It is likely around that time there was official SMP release, but betas were available earlier.
I didn't find earlier mention of the Linux SMP either: http://webcache.googleusercontent.com/s ... =firefox-a

Thanks once again. I also found this: http://web.archive.org/web/200710160551 ... sh/FAQ-SMP which is dated October 16th 2007 but implies that SMP was released in November 4th or 6th 2006 because that's the date displayed in the Linux Installation Instructions. But that's really weak, and your link is much better since its proof that it was available on December 10th. Still no idea where BOINC and non-BOINC DC projects were at that time, so I'll have to keep hunting around.

EDIT: All right. So I believe I have it figured out for the GPU. I found Dr. Pande's blog post here: http://folding.typepad.com/news/2008/05 ... pu1-g.html and a scientific publication from non-F@h scientists titled "Accelerating the MilkyWay@Home volunteer computing project with GPUs" which both specifically state F@h was the first for that. For the PS3, I've got an article from the Pittsburgh Post Gazette here: http://www.post-gazette.com/pg/07085/772011-96.stm which "It's the first distributed computing project to take advantage of the PS3's zippy Cell processor, and Sony promises it won't be the last." but I have not yet found the original statement from Sony that backs this statement. But I'm still searching around for definitive proof on the SMP, but we have some good clues thus far. Other than those, the best that I have is 7im's post viewtopic.php?f=67&t=19541#p194945. Hopefully we can find something that says something definitive about the SMP and some more info about the PS3.

[Rattledagger]

The statement that "all work needs to be done, not just a single WU done very fast" is inaccurate. First, all of the WUs that FAH distributes need to be done, and second, there is no single WU that has to be done very fast.

Well, a wu that needs atleast a 16-core-computer atleast to me goes under the heading "has to be done very fast"...

In the early days of SMP, a quad processor could reduce the elapsed time from N days to ~0.23*N which ultimately reduced "a few years" to "~.23*(a few years)" Newer hardware continues to increase the number of parallel threads available to FAH, so the value of 0.23 continues to get smaller and smaller, but there still are projects that will require "many, many years" to process, even with the fastest hardware.

1/4 of a quad is 0.25, I find it unlikely you'll manage a speed-up to 0.23. More realistic is 0.3 or something.

A quick example, let's say a quad-core uses 4 days for a wu if single-threaded. For 100 generations you'll need 400 days to finish a trajectory. For SMP you'll use 120 days (with 0.3). This looks like a nice speed-up.

But, let's say a project has 10000 wu's, and you've got 2k active quad-core computers. The quad-cores will use 8 days to finish a generation, but at the same time part of the wu's has also finished a 2nd. generation. You'll use 20 days for 4 generations, and 500 days for all. The SMP on the other hand will use... 5 * (4 * 0.3) = 6 days for one generation, and 600 days for all.

If you've got more than 2500 active computers, single-threaded will land on 400 days for a single project. SMP on the other hand will continue to increase upto 10k there it will use 120 days for all work. If you've not got more work in other projects to do, the single-threaded application will leave some computers idle if between 2500 and 10k active computers.

If enough work is available in other projects on the other hand 10k active computers will with SMP finish a single project in 120 days, but will use 480 days to finish 4 projects. For single-threaded on the other hand, the computers will use 400 days for a single project but at the same time finish 4 projects in 400 days.

So, SMP will example finish abdulwahidc project-7028 faster than single-core, but abdulwahidc will likely need to wait longer using SMP to finish the 29 projects 7000 - 7028. Getting a single project finished very fast can be an advantage, but if most or all 29 projects needs to be done SMP is likely a disadvantage...


Anyway, for most DC-projects, developing a SMP-application would be a waste of time, since less work will be done per day, and this not even counting some projects that's likely to land roughly at 1 instead of close to 0.25. Only for a few projects like CPDN could this be an advantage, but atleast for now CPDN has decided against it, since getting users to give-up a whole computer for a single wu for 1+ months isn't so popular, getting users to use one or more cores for the same amount of time is easier to get users to do.

BTW, as FAH's SMP-application clearly has shown, the second you start using computer for something, even some "light" things like surfing the web, can give a large performance-hit. For single-threaded applications, a quad-core is likely to still continue at 75% of it's capasity, while FAH's SMP-application at the same time drops-down to 10% or something...

[Jesse_V]

The statement that "all work needs to be done, not just a single WU done very fast" is inaccurate. First, all of the WUs that FAH distributes need to be done, and second, there is no single WU that has to be done very fast.

Well, a wu that needs atleast a 16-core-computer atleast to me goes under the heading "has to be done very fast"...

The SMP require 2+ cores and bigadv require 8/16+ cores. I guess you could classify them as being completed very fast, but I see it more as those WUs are so computationally demanding that you need these machines to get them done in a reasonable amount of time. In my mind, they're not regular SMP WUs that have some massive priority, they are massive WUs that have regular SMP priority. In that sense, we just need to get all the WUs done. The QRB urges us to complete the WU so the next gen can be created, as Bruce mentioned.

In the early days of SMP, a quad processor could reduce the elapsed time from N days to ~0.23*N which ultimately reduced "a few years" to "~.23*(a few years)" Newer hardware continues to increase the number of parallel threads available to FAH, so the value of 0.23 continues to get smaller and smaller, but there still are projects that will require "many, many years" to process, even with the fastest hardware.

1/4 of a quad is 0.25, I find it unlikely you'll manage a speed-up to 0.23. More realistic is 0.3 or something.

I'm pretty sure Bruce said 0.23 for a reason. At first I too was a bit confused, but I thought about it a bit more and realized that it might have something to do with overhead and the architecture. In fact, I'll bet that if you ran 4 uniprocessors you wouldn't be exactly 4x as productive. The 0.23 is reasonable considering all the data transfers and core synchonizations that Bruce discussed, and that can slow down production. In that sense, its difficult to get the strong scaling needed to get to 0.25*N. With a quad-core and considering Bruce's discussion, I'm not sure how you could get over 0.25.

So, SMP will example finish abdulwahidc project-7028 faster than single-core, but abdulwahidc will likely need to wait longer using SMP to finish the 29 projects 7000 - 7028. Getting a single project finished very fast can be an advantage, but if most or all 29 projects needs to be done SMP is likely a disadvantage...

Which is why we just need to get WUs efficiently done rather than a single trajectory extremely quickly. Quad-cores give a near-4x speedup overall.

Anyway, for most DC-projects, developing a SMP-application would be a waste of time, since less work will be done per day, and this not even counting some projects that's likely to land roughly at 1 instead of close to 0.25. Only for a few projects like CPDN could this be an advantage, but atleast for now CPDN has decided against it, since getting users to give-up a whole computer for a single wu for 1+ months isn't so popular, getting users to use one or more cores for the same amount of time is easier to get users to do.

BTW, as FAH's SMP-application clearly has shown, the second you start using computer for something, even some "light" things like surfing the web, can give a large performance-hit. For single-threaded applications, a quad-core is likely to still continue at 75% of it's capasity, while FAH's SMP-application at the same time drops-down to 10% or something...

Hmm. Well each DC projects has to balance the development and support of an SMP client against its benefits. In F@h's case where there's serial calculations involved, getting things done quickly is an advantage. As Bruce mentioned, things like SETI@home don't need this, and 4 uniprocessors is probably fine. And by "performance hit" I'm assuming you mean the DC project, which is true. I don't know about you, but when I browse the web I used like 5% of my quad-core's power, which means that I can watch FahCore_a4.exe drop down to 95% usage. Even when I run some CPU-intensive program, F@h drops down to only 75%. The point of these run-in-the-background clients is that people don't use that much CPU all the time, and even when they do there's usually plenty left over.