Rack-Mount 8-GPU Dedicated Folding Rig

[treckin]

My 2c: systems with a PLX or Microsemi PCIE switch are pretty horrible to maintain, and often inject pretty gnarly bugs. I would personally stick with a traditional [CPU<--PCIE-->Device] architecture if it were my hardware.

If you have a fancy third party software layer and a bunch of support to make it all play nice, it can work. I just dont typically see any advantage in a folding environment, especially seeing as GPU memory utilization (and therefore bus I/O requirements) seems to be super low. You could likely fold against a PCIE G3x1 or G3x2 link with minimal PPD impact.

[r0k0]

CPU should be 2.5+Ghz to feed fast GPUs. On Windows it needs minimum pcie 3.0 x4 for fast GPUs and on Linux pcie 3.0 x1 or else it bottlenecks.

is this why my gtx 1060 doesnt give me good ppd with my hp z220 xeon e3 1245 v2 ? it runs only pcie 2.0

[foldinghomealone2]

CPU should be 2.5+Ghz to feed fast GPUs. On Windows it needs minimum pcie 3.0 x4 for fast GPUs and on Linux pcie 3.0 x1 or else it bottlenecks.

is this why my gtx 1060 doesnt give me good ppd with my hp z220 xeon e3 1245 v2 ? it runs only pcie 2.0

Probably yes, but define 'doesn't give me good ppd' just to be sure...

[HaloJones]

PCIE 2.0 vs 3.0 makes next to no difference in my experience.

[Nathan_P]

CPU should be 2.5+Ghz to feed fast GPUs. On Windows it needs minimum pcie 3.0 x4 for fast GPUs and on Linux pcie 3.0 x1 or else it bottlenecks.

is this why my gtx 1060 doesnt give me good ppd with my hp z220 xeon e3 1245 v2 ? it runs only pcie 2.0

What PPD are you getting from that 1060?

[Nathan_P]

You'll only find PCIe bifurication on HEDT, workstation or server boards, its not a thing on consumer boards, especially on intel who segment their product stack so much. If you are looking at this seriously I would possibly look at a X399 board with a good amount of expansion and a cheap 1900x. You usually get 2-3 x16 PCIe 3.0 slots, plus a couple of m.2 and plenty of hi speed USB3.1

[gordonbb]

You'll only find PCIe bifurication on HEDT, workstation or server boards, its not a thing on consumer boards, especially on intel who segment their product stack so much. If you are looking at this seriously I would possibly look at a X399 board with a good amount of expansion and a cheap 1900x. You usually get 2-3 x16 PCIe 3.0 slots, plus a couple of m.2 and plenty of hi speed USB3.1

It is is thing on Consumer Boards - perhaps just not on the Intel z370/z390 Platform
Gigabyte has it in almost all of their mid to high-end x470 and x570 boards. The best option these days IMHO is the Gigabyte Gaming-X. You get a PCIex16 slot capable of bi-furcation (x8/x8; x4/x4/x4/x4) and two m.2 slots as well. HEDT are nice but you can build a really nice folding machine with a x570, 3900x or 3950x, and a PCIe m.2 adapter for a LOT less money than going with a HEDT motherboard and processor.

[gordonbb]

CPU should be 2.5+Ghz to feed fast GPUs. On Windows it needs minimum pcie 3.0 x4 for fast GPUs and on Linux pcie 3.0 x1 or else it bottlenecks.

is this why my gtx 1060 doesnt give me good ppd with my hp z220 xeon e3 1245 v2 ? it runs only pcie 2.0

I use an EVGA GTX 1060 6GB SC (Short-Board) in the PCIe3 x4 slot of my SuperMicro X10SL7-F with a Xeon e3-1231v3 on Win 10 Pro 64 all the time and see little to no degradation of performance compared to running it in a PCIe3 x16 slot running at x8 on any of my Linux Systems. It gets between 440-580kPPD on Core 22 WUs.

I would estimate that as long as it is PCIe Gen2 and at least x4 it shouldn't bottle-neck unless that slot is off the ChipSet and the DMI bus is choked.

[r0k0]

i dont know how to check where PPD is calculated, still new to this but i have a :

HP Z220 - Xeon E3 1245 V2 / 16GB DDR3-1333 ECC - GTX 1060 6GB
HP Z800 - 2x Xeon X5675 / 24GB DDR3-1333 ECC - Quadro K4000 ( Waiting of 8 pin dongle to arrive threw mail for a GTX 1070Ti )
Gaming PC - Intel i7 6850K / 32GB DDR4-3000 - RX 5700XT.

now with 93 Work Unit done, im only at 389 093 pts, both my HP workstations have been running 24/7 for almost 2 weeks now...

[Nathan_P]

do you have a passkey? your hardware should be getting the following:-

1060 - 400k PPD
K4000 - ~100k PPD
5700XT - not sure but around 600k PPD seems reasonable
1070Ti - 700-800k PPD

[r0k0]

yes i got one, but they didnt accept my current name ( r0k0 ) so they made a new one and lost all my points and wu count

[nivedita]

This is going to be very expensive and noisy ...

You can't mount gaming GeForce in such case. You'll probably need very expensive Quadro or Tesla ...

For instance, nothing like these will fit :
https://fr.msi.com/Graphics-card/GeForc ... VENTUS-11G
https://fr.msi.com/Graphics-card/GeForc ... UKE-11G-OC

You usually need something like this :
http://www.pny.eu/fr/professional/explo ... datacenter

It depends on the GeForce. Eg the NVIDIA originals will all fit. Anything that doesn't use more than 2 slots will work.

The server airflow is designed for passive cards it is true, but that should be fine I'd think as the server fans are going to be generating a lot more airflow than gaming cards with their builtin fans need. If you can stick one of these in a basement so you don't hear it you should be fine.

[gordonbb]

It depends on the GeForce. Eg the NVIDIA originals will all fit. Anything that doesn't use more than 2 slots will work.

The server airflow is designed for passive cards it is true, but that should be fine I'd think as the server fans are going to be generating a lot more airflow than gaming cards with their builtin fans need. If you can stick one of these in a basement so you don't hear it you should be fine.

Just "Fitting" in the case is different than getting good airflow.
Servers chassis are designed for front-to back airflow and with 2-slot GPU clearances with the assumption you will be using a card that intakes air from the rear of the card and exhausts it out the bracket.

The vast majority of Consumer cards will choke (starve for air) if you install them directly beside each other as their is no air gap to provide a channel for fresh air to be fed to the card. Most cards have a hard enough time keeping cool in a large chassis with a 1 slot gap between them and usually require custom fan curves or running at a fixed manual setting.

You could remove the shroud and fans leaving just the heatsink assembly but most of the heatsink fins run up and down with the exception of blower cards.