Help building out a folding machine (motherboard specifically)

[bourbonbelly]

Looking for buildout suggestions for a build that will be used for (mostly) folding@home. Of note is that I'd like to be able to have a second internal GPU. I don't plan on building it myself, but rather going through CyperPowerPC which builds customs to spec.

What needs to be prioritized is the first GPU (thinking 3060TI), the PSU (going with 1000W to support 2x GPUs), and cooling (figure need to max this out).

Beyond that though I'm at a total loss as to what motherboard to select for example that would support a 2nd internal GPU, so that's what I could really use the most help with.

Any thoughts/help appreciated!

[hrsetrdr]

Well first the usual question: AMD or Intel? The AMD Ryzen series are excellent CPUs, as are the Intel i5/i7/i9 processors. As for motherboard I'd recommend an ATX(as opposed to an MATX), so that you have a bit for space for 2 dual slot GPUs. 32 GB of RAM is a good starting point...
So, pick which processor type(AMD or Intel) then you can focus on specific motherboards for that platform.

[bourbonbelly]

Thanks. My current machine is is an AMD Ryzen 7 3700X w/ a RTX 3070, so was planning on going with AMD again just for familiarities sake.

Thanks for the info about the motherboard as that's where I've been at a loss, i.e. what boards have space for a second GPU vs having to go the riser route , or accidentally find myself in a position where my purchase doesn't even allow for a second at all.

For reference here's sort of my starting point:

I've been looking at a case that supports E-ATX boards, AMD Ryzen™ 5 5600G 3.9GHz [4.4GHz Turbo] 6C/12T 19MB L3 Cache, RTX 3060TI, 32GB (8GBx4) DDR4/3200MHz Memory, Corsair RMx RM1000X 80 PLUS Gold certified PSU.

Thinking ASUS PRIME X570-PRO ATX for the motherboard if that works...

[MeeLee]

My experience with PCs, is that Intel is more capable of running long and intense computations, without crashing.
Like, I still own 40 Atomic Pi units, where about 1 to 3 units per month end up in a crash, or cease working.
I also have 4 AMD systems, containing of 3900x and 3950x CPUs, and they fail about between once a day, to once a week, with a record of 1 system nearly completing 3 weeks before needing a hard reset.

For CPUs, it's actually better to have a dual core CPU than a 8+ core CPU, if you only will be folding on the GPUs anyway.
The CPU you mention, is much slower than a 5 years old GPU, and most people who have both, just focus on GPU folding.

But if you have the hardware already, and are still looking for a motherboard, I'd suggest a B550 motherboard.
You don't need all the advanced features of an X570 motherboard if all you will do is just fold.
Most of the extras are built in Bluetooth, or wifi, extra slots, etc...

Just find a board with 2 slots and plug in the GPUs.
Cooling is always an issue with 2 GPUs running close next to one another.
Either go watercooling, or use a riser to move the bottom GPU one or two slots lower (if your case allows for it).

I wouldn't spend any money on a Gold PSU. Just get a Bronze (80+) PSU, that runs the whole system at approx 80% load.
So if you purchase a 1000W PSU, your system should ideally use between 50-80% of that (500-800W), and no more.
Bronze PSUs are pretty much like gold PSUs, so long you run em at those efficiencies.
Where Gold PSUs shine, is that they are more efficient above 80% (up to 90% or so), and below (like, are very efficient at very low wattages).

Since your system is a binary state (either running idle, or running under full load), you'll be better off choosing the PSU correctly.
the 3060TI uses around 200W, the CPU and Motherboard will use around 200W max.
That leaves you between 100W to 400W for a second GPU.

The rest I'd invest in water cooling. At least for the CPU, which if you can choose a 240mm AIO, will even help hot air exit the case; and then find some good case fans to suck out any hot air.
Ducting is also very important. You don't want a GPU to suck in hot air, so using some plastic tape or cardboard in the right places, can sometimes help your system keep cooler.

With regular PC cases running dual GPUs, your main enemy is heat.
It'll be hard to combat, and in many cases you'll have to limit the TDP of the GPU to reduce heat, for it to remain cool.
You can also choose a case you can open on the side for better cooling!

[bourbonbelly]

Thanks for the reply. This is extremely helpful! I'm still in the process of pricing things out / choosing components and this gives me a lot to think about and will likely help me keep costs down!

[kiore]

I run both AMD and Intel boards for folding and have built more than 10 folding rigs over the years. My advice is to build it for your primary use and look for extra capacity for folding, if folding primary use build in capacity for other future uses. If primary use is gaming build for gaming but look to keep extra capacity for your folding, so motherboard with extra PCIE lanes spaced to allow an extra GPU to be added and really having run many multi gpu rigs you need lots of space.. If you need a PSU at 650 gold think about making it more if $$$ allow suggest at least another 200 in case that new GPU or old GPU looking for a spare slot is added. CPU, get the one you want but low power and lots of cores seems like a good general idea.
No matter what people say there is no future proofing, but anticipating at least 30% more of everything currently needed is probably a minimum.

Addit: for GPUs especially if thinking of multiples, air cooling nolonger really cuts it for 24/7 work, my first dedicated folding rig had 4 x GT 9800's but there is really no way 4 modern aircooled GPUs could live in the same box without serious environmental mitigation. Consider hybrid GPUs, the last RTX card I bought was 2080ti aircooled and really really regret not waiting for the hybrid version as this card despite being in a high airflow case cannot coexist easily with another card.

[Mxyzptlk]

Just for some ideas, take a look at my folding rigs here: https://w6nikreport.com/foldinghome-setup

Notice that I do decently well with GPU’s with 2 - 2.2 slot spacing, but there are 2.75 slot GPU’s that will restrict the placement of a second GPU.

[PaulTV]

going with 1000W to support 2x GPUs

If you mean 'to support 2x GPU cards' (asides from a internal GPU), be sure they have enough room for airflow. Many GPU cards these days are 2.5 slots wide. On quite a number of motherboards, the GPU slots will be three slots apart. If there is only a centimeter or even less between the cards, the top card won't be able to get enough cooling. Depending on the case, there may also be very little room between the bottom card and the bottom of the case / top of the PSU.

[gunnarre]

Water cooling is a hobby in itself, and you could build a custom loop which cools both GPUs and CPUs, or just one or the other. You might use AIO coolers in creative ways too. It is not very cost effective, but it's a nice DIY project. Don't forget that water cooling means that a water pump has to run in addition to the fans, which does make some extra noise when running on low power. If you're going to run this box as a heater in an unoccupied room you might just get a good airflow case and stick some Corsair ML fans or even some server grade fans in it. The ML fans can run both quietly on lower RPMs, and push a lot of air with more noise at high RPMs.

[gordonbb]

I’m a bit late to the party but as others have mentioned slot spacing is problematic these days with the prevalence of 2.75 and fatter cards. Asrock has the x570 Steel Legend and Pro4 models with 4-slot spacing but then the lower card can get choked by the power supply shroud in most tower ATX cases. So with a 4-slot spacing a Lian-Li 011 Air or similar with an open basement might be an option.

Really for folding you want to run the cards at the minimum power limits for best efficiency and better thermals.

Many 4-slot spaced motherboards though can have clearance issues with dual tower CPU air coolers.

I run 5 dual GPU rigs in Fractal Meshify S2s and similar cases. With a Hybrid card in the top slot and a dual axial fan card in the lower but the hybrid cards these days tend to come with 240mm radiators rather than the 120s I’m using.

[MeeLee]

I personally use PCIE risers. Small x16 (POWERED) risers are usually the best for this purpose.
The powered part helps with voltage stability, and the risers themselves allow for GPUs to be mounted apart from one another.
Preferably you will want to suspend the GPUs in the air, allowing them to run with about 0,5 to 1ft of space between them.

Single GPU systems are recommended, when talking about 200W running watts on GPUs, like a RTX 3080Ti or bigger, simply because the heat dissipation of a case does not allow for much more than 500 total watts.
To solve this issue, you can use Risers to mount the GPU further from one another, or even outside of the case; or use an open design case.
Ideally you will want to keep all GPUs on risers, and run no more than 2 GPUs on a home computer.
A quad core CPU with HT disabled is probably your best bet, as it has enough computing power to run up to 3 GPUs just fine or up to 4 GPUs under Linux, and usually consumes a lot less power than higher thread count CPUs.

Aside from heat buildup, CPU throughput, there's also issues wattage wise.
Many modern PSUs won't run well pushing more than 2 GPUs, as the most affordable ones run from 500W to 1000W. For 3 GPUs you ideally have a 1200-1500W PSU, but these are extremely expensive, and may trip your breaker, as efficiencies will drop at those high power levels, and now also the copper wires inside your wall will start consuming power (they'll turn into heating elements).

So it really depends on what you're wanting to make.
A single GPU system, or a dual GPU system; both of these with the most powerful GPU you can afford, and a small but fast CPU.

Then there's the issue of compatibility. Many Linux versions run well with only 1 GPU. Once 2 GPUs run, there are all kinds of display issues.
Older versions run well with 2 GPUs.
Most motherboards don't allow for more than 2 GPUs, with the exception of some (like MSI MPG), which has enough bar addresses for 3 GPUs.
I believe they support 2 GPUs, but if you equip it with an Intel CPU without GPU (the -F series), I believe it may be possible that the bar addresses reserved for the IGP are used for a third GPU. But you need an Intel CPU for that, without IGP.
And the CPU needs to be able to offer the IGP PCIE lanes to the board, meaning unlike most Intel CPUs who have 16 PCIE lanes, it should support 24 PCIE lanes; usually in an x8, x8, and x4 configuration.
This is especially important if your system will run only at PCIE 3.0 speeds.
If your GPUs support PCIE 4.0, the PCIE bus won't be a bottle neck, so available PCIE lanes will become less of an issue.
You could run a lot of powerful GPUs on smaller slotted PCIE 4.0 lanes (like run a powerful GPU like an RTX 3090 on an x4 slot without too much performance penalty).

Right now, most WUs are not made to run on the highest end GPUs.
If you're looking for GPUs, Nvidia GPUs are considered the best option.
The highest end GPUs have many shaders, but as a result, those shaders will run at lower boost frequencies.
Which is why it's sometimes better to get a cheaper RTX3080 running at say 1850Mhz boost, than a 3090 running at only 1500Mhz boost.
Once FAH will have more WUs that can run on higher end GPUs, you can buy such GPUs, but for now, the price doesn't warrant it's performance.

Just remember:
- If you run Windows, as main OS, there should be no issues running 2 GPUs (or 3) so long the Bios accepts it; though Folding under windows, is a bit slower than under Linux.
- Don't go for mining boards, that use multiple PCIE x1 slots, because they are just too slow for folding.
- Your motherboard and BIOS will determine how many GPUs you can run maximum on your system, even if it has multiple PCIE slots still available.
- Fast RAM is good, PCIE 4.0 is good, CPUs should be dual core (Windows 1 GPU, or linux 2GPUs), or quadcores (for running the OS with double the GPUS), and have a boost speed of at least 3Ghz for GPUs up to a 3080, and 4Ghz if you want to be future proof.
- A simple 128-256GB SATA 3 or SATA 6 SSD is good enough to fold on for many years. It's fast enough too!
- Find a case which is either open, or has plenty of air flow.
- Space out your GPUs as far as possible. If you can use risers, to mount the bottom GPU below the motherboard, so it can more easily blow the hot air away from itself (prevent from sucking in hot exhaust air for cooling).
- Power Limiting GPUs is almost a must, and will result in slightly lower boost clocks, but more efficient folding.
- Choose GPUs with as many fans as possible (3fan GPUs are better than 2 fan GPUs, and 2 fans are better than 1), and as thick as possible (usually 2 slot GPUs or more are recommended for 200W GPUs).
- You can always mount a case fan to cool the back of the GPUs.

[MARSTG]

I agree with @MeeLee, you would need a sort of mining rig setup if you aim for more GPUs then 2, unless they are single slot Quadros.
@hrsetrdr : you do that kind of PPD with just 2 Xeons?

[PaulTV]

... you would need a sort of mining rig setup

Red flag here A mining rig usually is not suitable for folding. F@h needs a proper CPU (one thread per GPU), and proper PCIe bandwidth. I suppose you're not suggesting to use a mining rig, but a comparable setup, but just to be sure.

[MARSTG]

Yes that's what I meant. Proper pcie bandwidth not really : I had 290x folding on a 1x slot and it lost 2-3% of the ppd.

[PaulTV]

The 290x is released in 2013. Maybe that'd be happy with a 1x slot, but that won't work for something like an RTX card.