Do you need help?

[mikeestacio]

Hey F@H! I've noticed that the servers are slammed and people are having trouble getting WUs or uploading their results. However I don't see any of us reaching out to you to see if there's any way for us to offer our talents and services to assist you. I'm sure some of us here work in IT and could offer an extra hand to get those servers up and running to face the demands of this pandemic we're facing. I just wanted to personally reach out and see if there's any other way I could help F@H other than running the client and open up the line of communication for others in our community that feel the same way.

[Joe_H]

The people in charge are currently in talks with some large providers, the requirements are steep. A Work Server needs to handle high levels of I/O, high network traffic, have about 50-100 TB of fast storage. Those are the requirements I remember.

If there are specific other areas, I can pass that info on.

[tulanebarandgrill]

I think the question "Do you need help" may still be unanswered here. If you're looking into hosting this elsewhere that's probably going to cost money. Maybe folders could send a small donation to help get it started sooner. If everybody sent $10 that could be a good sized chunk. Just a thought. Or maybe there's another way to help ?

[bruce]

Traditionally, FAH has purchased enough servers to meet the research needs of Grad Students at the various campuses that are part of the Consortium. The whole world is suddenly interested in COVAID-19 and FAH has received unanticipated advertising from several popular websites so suddenly we need 6x of the infrastructure and of the staff. Such rapid growth could not really be anticipated or planned for.

FAH doesn't bug people for money but the information is available if you look for it.
https://foldingathome.org/about/donate/
Unrestricted funds for "infrastructure" are harder to come by than research funds, too.

[tedder]

The people in charge are currently in talks with some large providers, the requirements are steep. A Work Server needs to handle high levels of I/O, high network traffic, have about 50-100 TB of fast storage. Those are the requirements I remember.

If there are specific other areas, I can pass that info on.

Or devops help, or dev help? For instance, can I help by reviewing and seeing how to reduce server load? Look at the logs and try to fix the "passphrase emailer"? Work on log rotation and archiving unused data?

[mikeestacio]

Traditionally, FAH has purchased enough servers to meet the research needs of Grad Students at the various campuses that are part of the Consortium. The whole world is suddenly interested in COVAID-19 and FAH has received unanticipated advertising from several popular websites so suddenly we need 6x of the infrastructure and of the staff. Such rapid growth could not really be anticipated or planned for.

FAH doesn't bug people for money but the information is available if you look for it.
https://foldingathome.org/about/donate/
Unrestricted funds for "infrastructure" are harder to come by than research funds, too.

Do you have a rough estimate of how much it's going to cost to be able to keep up with the current surge? Maybe now would be a good time to hold a telethon or fundraiser. Maybe we could reach out to Nvidia to help since they put out the call to action.

[mikeestacio]

Anybody reading this willing to volunteer?

[PairedPrototype]

I would like to volunteer to help as well.
I'm primarily an ASP dotNET Core software engineer, but I've dabbled with other languages too and I also enjoy BASHing a few things out in a terminal.
I guess it's a matter of could you find the time to collate a list of things that need to be improved and create a process that would allow, essentially, untrusted people to contribute. Which I would imagine would be difficult for DevOps based task.

[HouseCat]

I have already reached out as a developer to Bruce. I emailed the person in question. I might be able to assist with the load distribution and optimizing most of the user facing systems. However, it would be arrogant of me to assume I can learn their entire setup faster than they can fix it or that they don't have a brilliant team on it and its just a matter of paying to scale up.

As a bit of a mentor myself - I know they also may not be able to dedicate time necessary to on board somebody in the midst of the current issues. My extension of assistance does extend beyond Corvid-19 though.

Fans/users just have to be patient.

[bruce]

I'm here to provide help-desk kinds of support from volunteers like yourself and coordinate those kinds of help to people who are unfamiliar with FAH and it's infrastructure. I've spent some significant time with software development and systems analysis ... but like @housecat suggests, I'd just be in the way trying to facilitate the creation/distribution of biochemical research or even get involved with attempting to make changes to any of the the server-based components that are being over-stressed by the sudden influx of thousands of new Donors who are simply offering their home computing resources to fight COVAID119.

I suppose most of you would be in the same boat. I did say (elsewhere) that things are getting better at the server level. There's hope.

At this point, it's best to understand whether you can say

My extension of assistance does extend beyond Corvid-19 though.

I don't think Management was looking to hire somebody, or what kind of talents they might be looking for some time in the future. We really have been hit by a tidal wave of unexpected computer resources without the resources to deal with it easily.

[fangfufu]

Have you guys thought about contacting Linus Tech Tip? They are literally holding a competition for running Folding@Home. Linus himself has a few server with 100TB storage space.

https://linustechtips.com/main/topic/11 ... -covid-19/

[ghstridr]

Has anyone approached AWS or Microsoft Azure? They have the infrastructure at the scales need, ready to deploy to. These loads are ready made for this. On top of that, it can be scaled quite easily. I, personally, work with AWS and Google Cloud. On AWS, keep the work on EFS in case a server 'goes away' and the next server launched can connect with the data. Sure it would take a little engineering, but not much. Some quick python and some locking methods to ensure data integrity. SQS calls to automatically generate more WU's. Additional SQS calls to automatically perform ETL/ETA (ExtractTransformAnalysis) and push it further through a data pipeline. Approach it with the vendors as doing pro bono work like lawyers and doctors do. When covid19 is defeated, demand will drop off, but you can autoscale back or drop it altogether, but still have the ability to fire up a fleet at a near moments notice because it's all pre-configured.

[Joe_H]

Take a look at the Server Status page for a couple new additions.

[RGB_Lights]

Please let us know what the need is. Many of us have serious tech connections. That same interest that overran the capacity due to publicity is ready to overrun you with resources, but there needs to be an authoritative ask.

Please think big, ask for what you could put to use in a massive surge scenario, not what you think you could get. We really can help. With an authoritative description of a big request, we can also save you the trouble of assembling many small offers. You have an unparalleled case for support and a big ask at this a solid strategy.

[GingeraMan]

Get corporate sponsorship they are all desperate to show off their stuff so let them brand the rescue all they like. We saved FaH blah blah blah. Has anyone approached Amazon etc?