Folding Forum

Posted: **Sun Aug 13, 2017 5:23 pm**

Hi everyone,

I have been contributing to this wonderful project since about a year or so.
I have liked this project a lot. (I have somewhere close to half a million points currently)

I decided to put in some money in cloud computing if possible and was exploring possibilities. But, I was surprised to find out that folding@home does not run with similar hardwares for some reason. I thought of starting with Heroku free dynos and checking out how that works to ultimately shell out some effective money to contribute to the project. (I am also thinking of getting my friends contribute and I really want to make this whole process simple)

I have bench-marked the results:

Code: Select all

								Windows Chrome Browser:		Heroku:										Ubuntu 17.04 (VM):					Docker (Windows):
								23 seconds for 1%				20-25 minutes per core for 1%			Client no longer detected			15 minutes for 1%
																		(shutting down; file not read)		(8 minutes 9 seconds for 1%)
Number of cores:					4 								8 													1									2 		(1)
Operating System:			Windows 10 system 			Modified Ubuntu 							Ubuntu Desktop							Ubuntu 	(Ubuntu)
RAM:								8 GB 							60 GB (limited to 512MB)				2 GB 											2 GB 		(2 GB)
Probable CPU Frequency: 		2.8 GHz 							2.5 GHz 								2.8 GHz 										2.8 GHz 	(2.8 GHz)
Cache: 							512KB (L2) 4MB (L3)						25600 KB 							4MB 										4MB 		(4MB)
Disk Speed (approx):			210 MB/s (not-isolated)		170 MB/s 								346 MB/s 										210 MB/s 	(166 MB/s)

Note: I have forgotten Docker's benchmark results (results were obtained some 15 days ago); Hyper-V is not getting enabled and I do not have administrative privileges of Windows; I have tried to replicate that by running Ubuntu 17.04 64-bit Desktop in VM of Windows 10 and running Docker in that VM.

But, nevertheless, from the results observed above, it is definitely not working well for Heroku or Docker or VM even though the hardware is not that bad.

Can somebody be requested to let me know the cause of the problem?
Thanks for your time and consideration.

I take this opportunity to thank the developers for the wonderful softwares and the contributors for making everyone's lives better.

Regards,
Prasad N R

Posted: **Sun Aug 13, 2017 7:08 pm**

I have not personally tried FAH on a cloud system but here's what I understand. Please correct me if this doesn't match your results.

FAH is resource intensive. It requires a very significant amount of computing power and only makes use of unused resources.

1) GPU folding (FAHCore_1x or FAHCore_2x) does not work unless you have access to a local GPU with sufficient resources to meet the deadlines. It will NOT work using any (local) Virtual Machine system that I know of ... the characteristics of a virtual GPU are insufficient.
2) CPU folding (FAHCore_Ax) will run in a virtual environment. The resources of one or more fractional CPUs may or may not be sufficient to meet the required deadlines depending on what percentage of the CPU's processing power is devoted to others who are sharing the same CPU(s).

Both of these satements apply to a local VM environment and they probably also apply to a shared cloud resource.

It should be noted that there are positive financial considerations for the companies who sell cloud computing to sell as much of their resources as they can. During the introductory period of a new Cloud Computing servers, it is likely that there is some idle time available for low-priority tasks. If the service is to survive (financially) they need to sell those idle resources and create sufficient backlog that there's nothing left for them to sell. Applications that use small percentages of a CPU are given priority over applications which "hog" resources like FAH does.

Also, FAH is specifically designed to run an an extremely low priority, making use of unused resources -- i.e. - using only those resources that are typically wasted when the CPU can't find anything else to run. This plan works very well if a you're running on a system that's idle except when you press a key or move a mouse -- and you wouldn't want to run FAH if it hogged sufficient foreground resources to create a lag when you type something or move the mouse.

In other words, Cloud computing and FAH are not designed for each other.

Posted: **Sun Aug 13, 2017 8:05 pm**

As cloud computing with GPUs is expensive there is the curecoin group which setups some GPUs to run for folding and you can pay them a cheap price for that.
You can rent GPUs for a month or a year and running a gtx 1070 one year costs 875 dollar. https://curecoin.net/shop/

Posted: **Mon Aug 14, 2017 1:15 pm**

foldy wrote:As cloud computing with GPUs is expensive there is the curecoin group which setups some GPUs to run for folding and you can pay them a cheap price for that.
You can rent GPUs for a month or a year and running a gtx 1070 one year costs 875 dollar. https://curecoin.net/shop/

Yeah. You are right regarding that. But, I have not seen any benchmarks for GPU vs CPU. I do not want to risk my money on some cloud provider still experimenting with that. (GPU IO on cloud seems to be pretty awful from what I have heard and they continuously try to refine GPU transfer methods)

But, on a plain CPU, I simply run the FAHClient application either with config.xml or through command-line.

Code: Select all

heroku run bash
wget --no-check-certificate https://fah-web.stanford.edu/file-releases/public/release/fahclient/debian-testing-64bit/v7.4/fahclient_7.4.4-64bit-release.tar.bz2
tar -xvf fahclient_7.4.4-64bit-release.tar.bz2
cd fahclient_7.4.4-64bit-release
./FAHClient --gpu=false --power=full

bruce wrote:1) GPU folding (FAHCore_1x or FAHCore_2x) does not work unless you have access to a local GPU with sufficient resources to meet the deadlines. It will NOT work using any (local) Virtual Machine system that I know of ... the characteristics of a virtual GPU are insufficient.
2) CPU folding (FAHCore_Ax) will run in a virtual environment. The resources of one or more fractional CPUs may or may not be sufficient to meet the required deadlines depending on what percentage of the CPU's processing power is devoted to others who are sharing the same CPU(s).

Both of these satements apply to a local VM environment and they probably also apply to a shared cloud resource.

It should be noted that there are positive financial considerations for the companies who sell cloud computing to sell as much of their resources as they can. During the introductory period of a new Cloud Computing servers, it is likely that there is some idle time available for low-priority tasks. If the service is to survive (financially) they need to sell those idle resources and create sufficient backlog that there's nothing left for them to sell. Applications that use small percentages of a CPU are given priority over applications which "hog" resources like FAH does.

Also, FAH is specifically designed to run an an extremely low priority, making use of unused resources -- i.e. - using only those resources that are typically wasted when the CPU can't find anything else to run. This plan works very well if a you're running on a system that's idle except when you press a key or move a mouse -- and you wouldn't want to run FAH if it hogged sufficient foreground resources to create a lag when you type something or move the mouse.

I am tempted to agree on that. But, I am still unsure as to why it cannot run on cloud when it can run well on desktop computer.
I decided to check that and installed Chrome in VM and then ran

Code: Select all

google-chrome --allow-insecure-content

I was able to fold upto 1% within 130 seconds with full power. (It is just 5 times slower than running natively; But, this is fine too; This isn't 15 minutes)

But, I do not have this sort of 'liberty' to install Chrome software in cloud.

I have tried to benchmark with the following config.xml files (Benchmark results are as mentioned in the 'table').

minimal config.xml would look like:

Code: Select all

<config>
    <!-- Folding Slot Configuration -->
    <gpu v='false'/>

    <!-- Slot Control -->
    <power v='full'/>
    
    <!-- User Information -->
    <passkey v=''/>
    <team v='1'/>
    <user v=''/>

    <!-- Folding Slots -->
    <slot id='0' type='CPU'/>
</config>

On cloud, I am running the following:

Code: Select all

<config>

    <!-- Folding Slot Configuration -->
    <gpu v='false'/>

    <!-- Slot Control -->
    <power v='full'/>
    
    <!-- User Information -->
    <passkey v=''/>
    <team v='1'/>
    <user v=''/>
    
    <!-- Folding Slots -->
    <slot id='0' type='CPU'/>
    <slot id='1' type='CPU'/>
    <slot id='2' type='CPU'/>
    <slot id='3' type='CPU'/>
    <slot id='4' type='CPU'/>
    <slot id='5' type='CPU'/>
</config>

bruce wrote:In other words, Cloud computing and FAH are not designed for each other.

That is depressing. Because, I cannot spend huge money on hardwares which will turn out to be obsolete in 3-4 years. Instead, I can use that to scale cloud instances and simply let it run without my intervention.

Summary: I still need to know why this is not possible; I have similar hardwares and yet cloud seems to be 'different'. I would love to know technical description regarding the same if it is not possible.

Thanks for your responses.

Thanks for your time and consideration.

Regards,
Prasad N R

Posted: **Mon Aug 14, 2017 3:03 pm**

For CPUs you want them to support AVX instructions for fastest speed, a 4 core CPU with AVX can get 100k PPD.
For GPUs the newest generation with most shaders is best with upto 1000k PPD.

Maybe you want to try the Amazon Cloud which also have a free trial. There you can also get dedicated GPUs.
https://aws.amazon.com/ec2/instance-types/?nc1=h_ls
e.g. an instance p2.xlarge gives you 1 GPU and virtual CPU 4 core. It may deliver 450k PPD but cost $0.9 per hour, this is 7884 dollar a year.

On the other hand curecoin cloud is there for several years and many folders are part of the curecoin team which makes 30% of the whole FAH performance.
There you get 450k PPD for 875 dollar a year and this is why I believe that is your best option.

Posted: **Mon Aug 14, 2017 3:34 pm**

Cloud computing is not designed to run compute intensive applications lie FAH.

If you want to rent something efficient, get a dedicated server.

The best solution is always to build you own computer ... you can get some cheap build now with only PSU/MB/CPU/RAM ...

Posted: **Thu Aug 17, 2017 4:24 pm**

You will find that the cost of a particular level of performance in a "cloud" environment adds up fast to a TON MORE than buying the hardware yourself.
Amazon Google and Microsoft are PARTICULARLY bad that way - you'll pay more in fees IN A MONTH than the equivalent hardware would cost for a high-utilization program like Folding.

Posted: **Fri Aug 18, 2017 6:29 am**

Right!

If you need a few minutes of processing time per month, it's cheaper to rent it from a cloud -- just like if you rarely need transportation, it's cheaper to take a taxi, and let somebody else pay a large percentage of the costs of ownership while making a profit for themselves. On the other hand, if you drive/compute more than a moderate amount (up to and including all the time) it's cheaper to own the hardware yourself.

Ride sharing services and clouds have their place in the world, but they're NOT affordable for FAH.

Posted: **Sat Aug 19, 2017 4:19 pm**

Thank you for the time spent in answering and thanks for the insights.

I have taken some time to experiment before posting here so that your precious time is not wasted.

1. However, I am not satisfied with the answers which suggest that FAH is not designed to be run on cloud.
(It is like me being told that my sickness cannot be cured and I have to live with it. It is not satisfying and I would continue learning more about that sickness myself)

2. I do not want to put in money initially and none of my friends would. We would wish to have a 'trial' to check everything works and then, when people are happy enough about it, they and I would surely pitch in some money.
I am thinking of running it in Heroku, Amazon AWS, Google Cloud, Oracle Cloud and Microsoft Azure and then checking the best one.
(Mining curecoin comes in later; It is definitely on the list)

3. I would love to run this with Heroku first as it is really simple to sign up. (even on corporate laptops without having to deal with Credit Card numbers)

4. I get that AVX instructions are better. But, I can see huge difference in performance. On computers with similar performance, (given same instruction set and similar FLOPS as calculated with https://stackoverflow.com/q/6289745 ) I am getting a perfomance difference of 100x (One computer is taking 23 seconds and another one is taking more than 1800 seconds to compute 1%) So, I have to get similar performance (2-3x difference is OK. 100x is not) before I switch over to AVX. I even tried benchmarking with a simple Python code (Python runs fine a loop of 100000 count takes about 1 second in both desktop and Heroku)
(I even tried changing the niceness with the following command:
nice -n 0 ./FAHClient --gpu=false --power=full )

5. I would like to inspect the source code of FAHClient and compile it natively if possible.
But, I have seen that the source code is not open. (As in viewtopic.php?f=67&t=21026#p210315 )
If that is the case, (to the developers expecting from contributors: I am not sure if I would support the project anymore if the source code is not released.
I would also like to know why this software cannot be open-sourced)
I also tried searching repositories.

(http://fah-web.stanford.edu/svn/pub/trunk/control/ appears to be empty.
https://github.com/FoldingAtHome does not appear to have the code of FAH Client which can be found at http://folding.stanford.edu/releases/be ... se.tar.bz2 )

6. If nothing works, I would like to see the core code and implement the Python or Java code by myself. (using GROMACS as library or API)

Additional question: I would wish to know how this is run in AWS. (Does AWS allow super-user privileges?)

I understand that there are inherent limitations of the cloud (experimented with Heroku so far).
Heroku cannot display dyno minutes and dyno seconds; Only dyno hours are displayed.
Heroku kernel is locked. No sudo or root privileges are available.
Heroku server can be reset intermittently (and every 24 hours).
Heroku does not have terminal programs like gnome-terminal installed.
Only one shell is available and that shell spawns a fresh isolated linux instance everytime.

But, I also suspect that there are inherent limitations in FAHClient.
FAHClient binary maybe poorly built; It cannot run with 7 or more cores. (As in http://en.fah-addict.net/news/5-folding ... ger-folds/ and viewtopic.php?f=55&t=24891 )
It sometimes get stuck while reading files. (As in https://github.com/FoldingAtHome/fah-is ... -217752272 )

Thanks for your time and consideration. I am grateful to you.

Regards,
Prasad N R

Posted: **Sat Aug 19, 2017 7:15 pm**

5. The source code to FAHClient is closed source for the reasons given here - http://folding.stanford.edu/faqs/projec ... urce-code/ & here - http://folding.stanford.edu/support/faq/opensource/. They need to maintain the scientific integrity of the results returned to them as well as the points awarded for work turned in. If that is a deal breaker for you, then there are plenty of other ways to contribute to science.

PrasadNR wrote:FAHClient binary maybe poorly built; It cannot run with 7 or more cores.

You are misunderstanding both issues brought up in the linked posts.

In the first the problem resides in the Gromacs code. In the past it was determined that it did not work well on multiples of "large" prime numbers. That was determined at the time to be primes that were 11 or larger. The code reassigned requests to numbers that were known to be okay. Over time it was found that multiples of 7 and sometimes 5 were also problematical. The released version of FAHClient 7.4.4 does not have code to handle that, but the public beta version 7.4.16 does.

For the second the issue resides in the OS or other software taking control of files that FAHClient and its related programs need to access. In any case, again it is a problem that should be fixed in the public beta version.

Posted: **Sun Aug 20, 2017 7:47 am**

To 4) There may be 2 explanations for the performance difference you see:
a) Different FAH work units from different projects may take different times to fold 1%
b) If you do not have a dedicated CPU in the cloud then it is shared with others which impacts performance randomly.

For benchmarking there is the tool FahBench which may help you http://fahbench.github.io/

For CPU to GPU performance comparison: a 4 core CPU may have 100 GFlops while a gtx 1070 GPU has 6600 Gflops.

Posted: **Sun Aug 20, 2017 10:14 am**

Joe_H wrote:They need to maintain the scientific integrity of the results returned to them as well as the points awarded for work turned in.
In any case, again it is a problem that should be fixed in the public beta version.

Thank you so much for the confirmation.

I appreciate it. I am eager to contribute. To circumvent this problem of source code, I would suggest a solution that would greatly simplify the compilation problem to a mere passkey generation and approval.

I would suggest open-sourcing the code so that computational resources are not wasted and Stanford researchers need not code on all softwares and operating systems of all devices on the planet (developers like me will take care of those).

I suggest the inclusion of these passkeys:

1. Admin Passkey (available only to Stanford researchers)
2. Developer Passkey (available only for development to anyone)
3. Contributor Passkey (available for normal contributors just for the collection of points; the exisiting passkey and this is completely optional)

One Developer Passkey and one Contributor passkey would be available to those who sign up (two passkeys as opposed to one passkey that is being used).

I would suggest two databases:

1. Developer database (Automatically routed when developer passkey is used and is available only for development)
This would contain the sample data before folding and sample results obtained after folding (sample data needs to be generated by Stanford developers)

2. Contributor database (containing actual data that needs to be processed; the data that is being used by FAHClient as of now)

Now, the entire problem of development of Cloud or running on Raspberry Pi Zero efficiently reduces to the developer downloading the source code which would contain an empty string that developer should fill in before compiling.
String developerPasskey = "";

Now, the next idea is simple; Just make sure that the adminPasskey is verified by the Contributor Database's controller and if that is incorrect or is not present, then that data is immediately discarded and the contributors' database is not filled with junk data. So, only the data uploaded by the binaries compiled by the Stanford researchers using the adminPasskey would pass through the Contributor Database's controller.

In case of developers, any data that is uploaded without Developer passkey would be discarded by Developer database.
Once developers are satisfied with the performance they obtain, they are supposed to upload the code and hand a copy of their source code to Stanford researchers. Then, the results of the final data uploaded can be compared with the existing test data of Stanford researchers on Developer database using some metrics (like 3D euclidean distance).

When Stanford researchers are happy about it, they would just replace the Developer Passkey by Admin Passkey and compile it with encryption so that the binary cannot be easily decompiled.

If you are unhappy about this setup, you can consider using different sort of authentication. But, what I am trying to say is that it is possible to let developers develop once the authentication is in place.

Conclusion: It is not the problem of malicious source code uploading junk data. It is the problem of authentication and authorisation.
(People will also have more confidence regarding the proper science behind this project if the source code is shared)

foldy wrote:For benchmarking there is the tool FahBench which may help you http://fahbench.github.io/

For CPU to GPU performance comparison: a 4 core CPU may have 100 GFlops while a gtx 1070 GPU has 6600 Gflops.

Thank you so much. I will try those and I hadn't checked that.

Regards,
Prasad N R

Posted: **Sun Aug 20, 2017 11:47 am**

PrasadNR wrote:1. However, I am not satisfied with the answers which suggest that FAH is not designed to be run on cloud.
(It is like me being told that my sickness cannot be cured and I have to live with it. It is not satisfying and I would continue learning more about that sickness myself)

2. I do not want to put in money initially and none of my friends would. We would wish to have a 'trial' to check everything works and then, when people are happy enough about it, they and I would surely pitch in some money.
I am thinking of running it in Heroku, Amazon AWS, Google Cloud, Oracle Cloud and Microsoft Azure and then checking the best one.
(Mining curecoin comes in later; It is definitely on the list)

1. It is more that "the cloud" is not designed to run F@H. "The cloud" is made up of servers that have lots of storage (not needed for F@H), limited CPU resources (usable by F@H), and no graphics capability (needed for best F@H productivity). While some server farms may offer limited hosting for Folding, it is a VERY small niche market.

2. That is another incompatibility with "the Cloud". "The Cloud" is made up of for-profit companies; F@H is designed to maximize use of free, volunteered resources.

While Curecoin is a noble initiative, I don't know that it is viable in the long run.

Posted: **Sun Aug 20, 2017 1:43 pm**

While you are seeking to "understand your fatal disease" you'll need to understand the difference between FAHClient and FAHCore_xx.

FAHClient is essentially a browser. It's function is to download a block of data, wait for it to be analyzed, and upload the results. Even if it is poorly writted (which it may or may not be so what. It contributes almost nothing to the time spent processing FAH. It's job is to provide secure communications with FAH's servers plus monitor when the status of what else is going on and to be guided by some configuration settings.

It is supported by a number of FAHCores which do the scientific processing. They do all the real work, and they are all open source. For CPU processing, check with gromacs.org about the limitation for running with 7 CPU-cores. Check with OpenMM about the code used on GPUs.

Posted: **Sun Aug 20, 2017 2:26 pm**

Some Clouds where you have no dedicated CPU or GPU will also kill FAH process after some time because 100% permanent usage of resources is against the rules. Getting a Cloud dedicated CPU or GPU will be expensive. (except at the curecoin cloud for folding only)

Folding Forum

folding@home performance in cloud

folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud

Re: folding@home performance in cloud