Guidance for building a folding dedicated computer

[boristsybin]

AMD FX series are a lot lower cost for the same number of cores,
...
Ignore "hyperthreaded" cores - they do NOT work well to support folding GPUs, they MIGHT be worth a QUARTER of a real core.
...
In fact, ignore AMD

AMD FX supports only pci-e v2, that, in theory, may limit gpu productivity

i3 4130 with HT enabled is enough for three 1080ti, so HT does work well to support folding GPUs

amd gpus loads CPU less than nVidia`s :
with amd gpu, FahCore uses 100% per thread on task start and and drops to ~60% per thread on main time of task count,
with nVidia, FahCore uses 100% of CPU thread whole folding time.

[rwh202]

RX Vega 64 makes 600-850k PPD so that is still reasonable too but uses more power.

Unless your electricity rates are very low, I think that performance is not good considering the power consumption of that GPU.

I think it's worth pointing out to those that are concerned by power use, that both manufacturers ship their cards past the optimal point in the power / performance curve.
Good results can be had by setting lower power targets than those that the cards ship with.
I have a rig with a pair of 1080 FE with enforced power limits of 130 W on them (down from 180 W). Performance is still good (650 - 700 k PPD per card) and just less than 300 W from the wall.
Overclocked 1080s might do 900 k PPD, but for double the power draw.

[foldy]

GTX 1080 ti with 55% power limit 1000k PPD at 250 watts (incl. CPU) instead of 1200k PPD at 350 watts.

[Leonardo]

Thanks, Foldy. I've never tried adjusting power downward, at least not for Folding. I may just give that a try this weekend and see how it shakes out.

[SteveWillis]

I have a couple of GPUs that run hot, around 90C that I reduce power on to cool them off and slow their fans. I don't like to run much over 80C and 75% fan.
For anyone running nvidia cards on Linux:

To display the current power

Code: Select all

nvidia-smi 

To set the power for GPU 0

Code: Select all

sudo -n nvidia-smi -i 0 --persistence-mode=1
sudo -n nvidia-smi -i 0 --power-limit=115

[SombraGuerrero]

So, in reading this thread, taking some new power readings on my rig, and doing some of my own quick calculations, I realize that folding with my CPU adds about 60-70 watts to my power load. For that extra load, I'm getting roughly about 43 - 77 PPD/watt/WU. This pales in comparison to my single GPU, which gives me about 140 - 300 PPD/watt/WU. So, this means I'm getting not even 1 PPD/watt on my CPU. From an energy cost perspective, it seems like I'd be better off not folding with my CPU. Is that a fair assessment, or are my numbers too wishy-washy?

[Leonardo]

Fair assessment. That is exactly why so many of us don't bother with setting up CPU slots.

[SombraGuerrero]

I suppose there is an argument that the CPU cores obviously focus on different projects than the GPU cores, so I would hate to think that those get less attention, but I suppose there will always be plenty of other CPU folders out there.....

[bruce]

The "general rule" does not apply to everybody. I can see several possibles.

1) You have a Windows Gaming machine or the equivalent Linux machine with one or more powerful GPUs and a CPU that supports many threads.
> 1A) You pay for power.
> 1B) You live in a home or a school Dorm where somebody else pays the electric bill -- or you have a large enough roof-top solar system so that you rarely have an electric bill.
2) you have a laptop with a weak iGPU that can't handle FAH so the CPU is your only choice.
3) You run Mac OS-X (where FAH does not support GPUs) so again, the CPU is your only choice..

You guys in group 1A decide CPUs aren't practical.
Those in groups 1B, 2, and 3 run with their CPUs.

Let's not denigrate the contributions that others can make just because you happen have a powerful system. All contributions to science are valuable.

[SombraGuerrero]

My feelings exactly, Bruce. I pay for electricity, but I wanted to run these numbers in the interest of awareness. Although it is an interesting exercise in conservation to run these types of numbers, I also feel that unless there is a grave need to cut consumption, it is sort've a slight to the CPU core project owners to ignore their work in the interest of "practicality."

[QuintLeo]

AMD FX series are a lot lower cost for the same number of cores,
...
Ignore "hyperthreaded" cores - they do NOT work well to support folding GPUs, they MIGHT be worth a QUARTER of a real core.
...
In fact, ignore AMD

AMD FX supports only pci-e v2, that, in theory, may limit gpu productivity

i3 4130 with HT enabled is enough for three 1080ti, so HT does work well to support folding GPUs

amd gpus loads CPU less than nVidia`s :
with amd gpu, FahCore uses 100% per thread on task start and and drops to ~60% per thread on main time of task count,
with nVidia, FahCore uses 100% of CPU thread whole folding time.

PCI 2.0 16x is more than enough for any GPU - even the 1080ti.
8x can lose a little bit, but commonly only a few percent at most even on the 1080ti.
Any quad core should be ENOUGH to support 4 x 1080ti - but have you priced Intel quad-cores vs the FX 6300 6-core or even the 8320e 8-core lately - and given my results as I talk about below, I'm not sure a single core is enough to support a 1080ti fully even WITH Hyperthreading enabled unless it's a very HIGH end CPU.

I tried to run *2* 1080 ti with a G4600 (Kaby Lake 2 core WITH hyperthreading, the SAME as a low-to-midrange Kaby Lake i3 except less cache) - the setup was DID CPU LIMIT on some work units, and was right on the edge most of the time on the rest.
Dedicated folding machine that wasn't running anything else, one 1080ti on a PCI-E 3.0 16x slot the other on a PCI-E 3.0 4x slot.
I can't recommend trying to run a 1080 ti on a SINGLE core - even with hyperthreading - as it gets very marginal.

AMD GPUs load the CPU a lot less in general because they're doing less work - FAH is not AMD GPU friendly, something about the differences between AMD and Nvidia GPU design make FAH work a LOT better on Nvidia GPUs.

[QuintLeo]

The nvidia driver can only use one CPU core per GPU.
Looks like really old slow CPUs limit the new fast GPUs.

"My short-lived rig with a pair of GTX 1080ti running on a dual-core Intel G4600 (Kaby Lake 3.5 Ghz) WAS CPU LIMITING NOTICEABLY on both GPUs"
That is interesting, did anyone else had this problem with a new fast GPU but only dual core for 2 GPUs?

The pcie bandwidth limit is only a big problem on Windows. Using Linux you loose only some % and so using x1 risers on Linux is possible.

Machine in question was running Ubuntu 14.04 64-bit.

[Joe_H]

I tried to run *2* 1080 ti with a G4600 (Kaby Lake 2 core WITH hyperthreading, the SAME as a low-to-midrange Kaby Lake i3 except less cache) - the setup was DID CPU LIMIT on some work units, and was right on the edge most of the time on the rest.
Dedicated folding machine that wasn't running anything else, one 1080ti on a PCI-E 3.0 16x slot the other on a PCI-E 3.0 4x slot.

That is marketed as a Pentium. Limitation you did not mention is that it only supports a total of 16 PCIe lanes, so depending on what else needed PCIe lanes such as for access to storage, there would not have been enough to run one GPU at 16x and another at 4x. Quite likely your 16x GPU was actually running at 8x at best, that would depend on your board.

[rwh202]

I tried to run *2* 1080 ti with a G4600 (Kaby Lake 2 core WITH hyperthreading, the SAME as a low-to-midrange Kaby Lake i3 except less cache) - the setup was DID CPU LIMIT on some work units, and was right on the edge most of the time on the rest.
Dedicated folding machine that wasn't running anything else, one 1080ti on a PCI-E 3.0 16x slot the other on a PCI-E 3.0 4x slot.

That is marketed as a Pentium. Limitation you did not mention is that it only supports a total of 16 PCIe lanes, so depending on what else needed PCIe lanes such as for access to storage, there would not have been enough to run one GPU at 16x and another at 4x. Quite likely your 16x GPU was actually running at 8x at best, that would depend on your board.

Actually, the 16 lanes are just the separate dedicated graphics lanes and are typically tied to one PCI-E slot (unless you have an SLI board with switches etc.). There are further lanes provided by the chipset that provide storage and the other slots, so 16x and 4x is quite normal on pentiums. They are then connected to the CPU over the DMI.

[v00d00]

I have a question on PCI-E utilisation between Gen 2 and 3.

On my AMD FX i have Gen 2, my GPU PCIE utilisation is at 3-4% at any given time according to nvidia-smi, running a GTX 970.
On my Intel Skylake i7, my GPU PCIE utilisation is at 2-3% at any given time according to nvidia-smi, running on a different GTX 970.
Both systems running Slackware 14.x with a modern kernel. But I also got this same amount last year on Debian.

PPD is essentially identical between the two.

Does PCI-E gen 3 actually make any difference, because it looks like the amount of data being sent and received comes nowhere near to saturating the lanes?

As for what computer to build for dedicated, its an apples and oranges discussion. In reality it doesnt really matter. If it has a relatively modern quad core cpu, a board with at least 2 PCIE slots, a 4GB stick of ram and a decent 650-850W PSU, you are good to go for most modern graphics card from either manufacturer. I would say to go with SSD, not for the fact you need the speed, but that it uses very little power and you can generally pick up a cheap industrial grade drive (16GB is fine for OS and folding) for very little money from ebay. If your budget is unlimited you could go with 1080 Ti with either water cooling or hybrid cooling, or maybe a couple of 1060's.

Key things to consider include Power Usage, Noise and Ambient Temperature. If electricity costs a lot, then aim for something thats much more streamlined on the PPD/watt. If you live in a place where the average outside temperature is 30-40C then you may end up having to use a lot more power to keep your home cool when running folding gpu. Similarly dont run cpu client. 6000PPD on avg for an FX. My 970 uses only 60w more and does about 270k/day. Their are enough people running cpu/smp that dedicate server level systems to it. If you do it for the science and you live at the North Pole, then running cpu client is probably going to help keep your house warm. Otherwise I see it as a waste of power that could be used to run gpu.

As ever, YMMV.