Is distributed computing worth it?

[abombthecoder]

Looking at this link at F@H's wiki article: Folding@home_and_Supercomputer_Computational_Powers.png copy and paste this into google if you want to see the graph.

it seems that the faster super computer speeds are dominating distributed computing in performance. Maybe I'm not understanding the graph correctly, is both the red and blue curves different folding clients, so you add them together to see the total TFflops? But even then it seems that we've hit a plateu. I was going to buy an uncessary $250 video card simply to contribute to a better world, by folding, but could i make a bigger contribution by simply donating to some standford supercomputer fund?

Why is folding lagging the fastest super computers? Is distributed computing just too inefficient when compared to one dedicated super computer? Can distriubted computing improve more so that it's relevant? Maybe it's because I hangout @ ocn, but people are spending a lot of money on dedicated rigs to help with folding because they want to make a difference in the world, so I extrapolate there a lot of people like myself willing to do this. SHould we all just be pooling our money into buying a huge super computer to make a bigger difference in the world then buying some computer with a relatively minuscule impact when compared to a super computer?

[Jesse_V]

Hi abombthecoder, welcome to the forum.

Since you're new, I'll post the link for you: http://en.wikipedia.org/wiki/File:Foldi ... Powers.png

Thanks for checking out the article there. A number of volunteers here put that graph together, and I remember uploading it to Wikipedia. Yes, supercomputers are much faster than Folding@home, at the moment. You'll also notice that F@h was significantly faster for a number of years.

The red and blue lines represent different measurements of Folding@home's performance. There are two methods because CPUs and GPUs are two completely different types of hardware, and they perform computations differently. The standard x86 CPU, for example, is commonly used as the universal "yardstick" for measuring performance; one could say "how fast is this system if we were using x86 CPUs?" This is necessary because Graphics Processing Units (GPUs) contain many hardware optimizations that a CPU does not. For example, sine, cosine, tangent, square root, and a 4-by-4 matrix multiply each take one floating-point operation (FLOP) on a modern GPU, but it would take about 30 operations on an x86 CPU. So if a system can does 1,000 square root operations per second on a GPU, (the "native" performance) one could convert that and report the corresponding 30,000 operations/sec on the x86 CPU, (the "x86" performance) even though all the operations took place on the GPU. Long story short, the two measurements are different, and Folding@home is powered by both CPUs and GPUs, so you can't add the two together.

As explained in the article, supercomputers are extremely expensive, and typically researchers don't get to use the whole machine by themselves. Both of those are significant disadvantages for using a supercomputer. So things are more complicated than a simple performance comparison. Furthermore, as noted in the footnotes in the article, supercomputers are benchmarked using LINPACK, an application that scales extremely well across all the supercomputer's CPUs. There are many scientific tools which aren't so lucky; often there's a great deal of difficulty in getting a program to use thousands of CPU cores efficiently at the same time. Molecular simulation programs are no exception to this (there are numerous scientific papers on this). Certain problems just don't scale that well. So the performance from LINPACK may be a high indication of the supercomputer's performance. Folding@home's performance is measured on the completion rate of its WUs, which is a real-world measurement.

Stanford does have a Folding@home fund, but such money is used towards things like servers, development, etc. Their donate page says "Some people have asked how they could donate money to help Folding@home. First and foremost, you can help by downloading and running our client software." Thus, running the software is an excellent way to contribute. If you want to spend money to help F@h, you can follow their recommendation even further by buying a high-end GPU.

Folding@home's performance is driven by the computational capabilities of its users, and how often people spread the word. It's a volunteer-driven effort, so it changes more dynamically than supercomputers do. There are even some indications that it's speed changes depending on the time of year. However, Folding@home is still extremely relevant, and (I'm not exaggerating here) it continues to be very groundbreaking and important to the scientific community. The volume and citation counts on the 109 publications are evidence of this. Dr. Pande, the director of Folding@home, currently has 12,985 citations to his papers (he co-authors all of F@h's publications) and 169 of his publications have been cited at least 10 times by other scientists. (reference) During my research for the F@h Wikipedia article, I ran across dozens of publications on molecular dynamics that described in high regard the work done by F@h.

Folding@home, like all distributed computing projects, is designed to use the idle processing power of current personal computers. Under that paradigm, researchers gain significant computational power from volunteers (and most support/debugging from volunteers like those on this forum as well) for basically the cost of running a server and the cost of developing the software. There's certainly no requirement to buy any equipment for F@h, so those who do spend the money do so because they believe in the cause and want to accelerate it beyond their current capabilities. I've never heard of any volunteer being that excited about upgrading a supercomputer.

[bruce]

Some problems are ideally suited for DC, some are not.

The first type of problem is to analyze prerecorded information. Record a lot of data (weather observations, seismic data, radio signals from space, etc.) and create some equations that ask "Have we recorded anything interesting?" Each observation can be analyzed and most of the data can be put in the "not interesting" pile (discarded) while a small fraction can be put in the "probably interesting" pile for further analysis. There's no inherent order to how the data must be analyzed so each node works independently of the others as long as eventually all the data is analyzed. Gather enough (slow?) computers, and the job will get done by brute force.

A second type of problem is one of predicting the future from observations. If we know that an earthquake or a hurricane was recorded, what patterns existed leading up to that time and could it have been predicted?

A third type of problem is one of predicting the future from equations. If you know the current velocity and position of everything in a system and the equations of motion (aka laws of physics) can predict what will happen a short time in the future, repeated keep applying those laws to predict the velocity and position at any time in the future.

(There are other types, too.)

Case 1 is ideal for DC. A massive problem can be processes as a lot of tiny problems.

Cases 2 and 3 may or may not be ideal for DC.

In case 2, it's not clear how much data we have to look at to achieve the desired predictions. A home computer may not be able to handle that much data simultaneously.

FAH falls into case 3, and the same kinds of limitation as case 2 apply. Just how many atoms does FAH need to analyze simultaneously and how long will it take to predict the system characteristics far enough in the future to be interesting? Stanford has done an excellent job of solving larger and larger proteins for longer and longer time periods in tandem with the progressive increases of the capabilities of home computers.

Other interesting proteins probably do need supercomputers, but a great deal of important research is being done with FAH at a much lower cost. The key here is that a supercomputer features very rapid communications between a massive number of the nodes. FAH uses a massive number of nodes, too, but they focus on problems that can be contained within a single node, avoiding the cost and the need for rapid inter-node communications.

[Napoleon]

But even then it seems that we've hit a plateu. I was going to buy an uncessary $250 video card simply to contribute to a better world, by folding, but could i make a bigger contribution by simply donating to some standford supercomputer fund?

My understanding is that the plateau has very little to do with FAH infrastructure hitting its limits. I recall Dr Pande mentioning in various interviews that they'd have the capacity to do a lot more, given a larger donor base. Maybe not in those exact words, but the message seems pretty clear and consistent to me.

With the new OpenCL core_17, GPUs are no longer restricted to doing specific types of FAH simulations, the GPUs can do just about all the same things FAH as the (previously) more general purpose CPUs - and a whole lot faster, too. Sources: GPU QRB update, Welcome to FahCore 17!

Currently, FAH has about 20000 active GPUs, and about 15000 active GPUs capable of core_17 work (source: http://fah-web.stanford.edu/cgi-bin/mai ... e=osstats2). That represents only a tiny fraction of core_17 capable GPUs all around the world, so if the new core manages to "lure" (a lot) more people to folding, things could change dramatically in terms FLOPS once again. In fact, http://folding.stanford.edu/home/ has this plea at the time I'm writing this:

Help us reach 1,000,000
Today we are 262,040 computers strong and growing fast. Share. Tweet. Email your friends and let them know.

That hardly sounds like they're having scaling problems or something like that.

After all that FAH specific stuff, I too would like to reiterate that while not diametrically opposite (IMHO), Distributed Computing and supercomputers are somewhat different concepts, and both have their uses. What makes supercomputer a supercomputer are the fast and low latency interconnects between nodes. Your average interweb certainly is neither. AFAIK, in supercomputers terms, even gigabit ethernet is considered (too) slow. So, I'd call Distributed Computing and SuperComputing specific solutions to specific problems, as opposed to directly interchangeable methods.

[7im]

The graph represents how governments and large institutions have continued to spend money on the latest technology to increase the theoretical speeds of but a handful of supercomputers and not all supercomputers.

The graph also shows how a lackluster economy has slowed down the computer upgrade process of a typical fah donor and how a general lack of disposable income has curtailed philanthropic donations to all good causes including fah.

The graph is a comment on the economy not a comment on fah's worth as a cause, and not a comment on the best way to process work units.

The graph does say, "Hey, look at that! A bunch of home users can pool their resources and rival the power of even the fastest super computers and make a difference in the world."

[Jesse_V]

But even then it seems that we've hit a plateu. I was going to buy an uncessary $250 video card simply to contribute to a better world, by folding, but could i make a bigger contribution by simply donating to some standford supercomputer fund?

My understanding is that the plateau has very little to do with FAH infrastructure hitting its limits. I recall Dr Pande mentioning in various interviews that they'd have the capacity to do a lot more, given a larger donor base. Maybe not in those exact words, but the message seems pretty clear and consistent to me.

Yes, I recall that as well. F@h does have the ability to scale much further, it's simply a matter of volunteers spreading the word and accelerating it further. With more resources, F@h would be able to launch more simulations and complete current simulations much faster. Speed is an important factor here: there are some important molecular problems so computationally challenging that they take an unacceptable amount of time even on F@h. A grad student has a limited timeframe in which they want to get their simulations done. With a larger userbase, F@h can tackle these problems.

If anyone is interested, here are two talks from Dr. Pande that describe scaling and other subjects. Both are excellent. I think that the statement Napoleon mentioned is in the second link.
http://folding.typepad.com/news/2010/10 ... ghome.html
http://folding.typepad.com/news/2012/09 ... t-fah.html

Along the lines of the original topic, here's a graph for you. This one comes from this paper, published in Current Opinion in Structural Biology in February 2013.

Here's the image: http://tinyurl.com/my2o3d7 (it's too large to embed here)

The graph compares simulation results from some of the notable researchers and their molecular dynamics simulators. As you can see, F@h continues to handle simulations that are significantly more computationally demanding than the others (including from D. E. Shaw's Anton supercomputer, also described in the F@h Wikipedia article). That vertical scale is logarithmic. The NTL9 protein, for example, has a folding time of 1.5 milliseconds. The previous record for the longest simulation of the folding of a protein was in the range of nanoseconds to microseconds. (publication and description).

Last year, Dr. Pande published these statistics on F@h's exponentially increasing capabilities: http://folding.typepad.com/news/2012/06 ... -come.html Software optimizations and new scientific methods are likely a significant factor here when combined with F@h's computational capabilities.

[bruce]

After all that FAH specific stuff, I too would like to reiterate that while not diametrically opposite (IMHO), Distributed Computing and supercomputers are somewhat different concepts, and both have their uses. What makes supercomputer a supercomputer are the fast and low latency interconnects between nodes. Your average interweb certainly is neither. AFAIK, in supercomputers terms, even gigabit ethernet is considered (too) slow. So, I'd call Distributed Computing and SuperComputing specific solutions to specific problems, as opposed to directly interchangeable methods.

To support this statement. we should mention the concept of a cluster. Few people have them, but we do get questions about whether FAH runs on a cluster or not. For those of you who may be unfamiliar with the concept, a cluster is a bit like a miniature supercomputer -- a number of (relatively) slow computers are interconnected and run under a single operating system, enabling them to work cooperatively on a single problem, much like the nodes of a supercomputer.

FAH is not designed to run on a cluster. On the other hand, FAH is designed to make use of a CPU with multiple cores using SMP. These two statements may seem to be in conflict. There are several reasons why FAH does not support clusters, but one of the critical ones is that the nodes of a cluster generally communicate with each other at ethernet speeds (i.e.-slowly) whereas the SMP cores perform data exchanges at memory-to-memory speeds (many times faster).

Your multicore CPU system, operating in SMP mode, is a miniature supercomputer with 2 or 4 or ... 12 or ... 48 nodes with high speed, low latency interconnections. Developing FAH for a cluster would be technically possible but not worth the investment. It would not be useful because the speed of the interconnections would be a very serious handicap for them to operate cooperatively on a single WU. Moreover it would probably add very few new clients.

[Napoleon]

If anyone is interested, here are two talks from Dr. Pande that describe scaling and other subjects. Both are excellent. I think that the statement Napoleon mentioned is in the second link.
http://folding.typepad.com/news/2010/10 ... ghome.html
http://folding.typepad.com/news/2012/09 ... t-fah.html

Actually, I had the first one in mind. Maybe it's just me, but I had problems with the streamed version, so I downloaded the entire .flv file (192MB) from http://www.gputechconf.com/gtcnew/on-de ... pic=34#158 in order to jog my memory. The part explaining parallelism, conventional scaling limitations and the FAH approach starts around 21:30. Most interesting, and not "embarrassingly mathematical" either.

31:25: "Scalability to billion cores would not be a big deal." Priceless!