Many GPUs on one machine

[Peter_Hucker]

I've just purchased the following which (by my calculations, including CPU speed) should allow Folding to run flat out on up to 12 GPUs:

Esonic B250 BTC Gladiator motherboard (TWELVE PCI express sockets)
i3-6100 4-core CPU
256GB NVME.
(Borrowing RAM from another machine. Found case, cooler, power supply lying in garage).

I'm doing this because connecting 4 GPUs to one socket in my current machine causes folding to slow down. The CPU is not the limitation, sharing a PCI-express lane is. The GPUs are Tahitis (Radeon R9 280X) and will be connected with PCI-Express 2 USB risers. If you have faster cards, you can use ribbons to get PCI-Express 3.

[Vester]

I ran eight GPUs on a similar motherboard using PCIe x1 risers. In BIOS, hyperthreading should be disabled and you need to enable above 4k decoding.

[Peter_Hucker]

I leave HT on, it gets more CPU work done. It only uses it if it needs to.

As for 4K decoding, I'll turn that on only if there's a problem, last time I put that on the computer wouldn't boot and I had to reset the CMOS. And that was a better machine - i5 8600K.

If it's too slow, I can get 1x to 16x ribbons, and just have to build a physical support structure to put the cards closer instead of just sitting on the shelf next to it. 3 foot USB cables make things much easier.

[Peter_Hucker]

Correction, I've just read up on above 4G decoding (along with the newer "resizeable bar") and it sounds like they boost performance (about 15%). I'm gonna switch it on for all my machines. If you never see me again, it broke.

[CarlHolmberg]

My pardons for jumping in months later. Re using various PCIe x1/x4/x8/x16 risers. My subjective experience suggests you may not want to get too carried away with the narrower connections.

My first setup used a Supermicro x8 dual socket, Extended ATX motherboard, mounted in a DIY plywood case. I had room for and used up to five GPU cards with risers attached wherever I could find room. Using NVTOP on Linux as my PCI performance monitor, I didn't appear to lose a noticeable amount of throughput with x8 risers compared to x16. Using x4, it noticeably sagged. Using the x1 riser via a USB cable, throughput was pathetic, this while I was using GTX1060s.

Unfortunately, I didn't write down any numbers back in 2020, and this was a 2012 mobo.

[Peter_Hucker]

It depends on the card's power and the PCI express version. One lane of PCI Express version 2 will support 3 280X cards on Folding at full speed.

I'm putting more than one card per lane, not more than one lane per card.

[nchowlett]

I'm having issues with Ubuntu recognising my last GPU (others are picked up by the OS fine). Has anyone had any issue of the GPU looking like it was physically working (lighting up and fans moving), but once in Ubuntu it didn't appear?

[Peter_Hucker]

Frequently. But in Windows and with old cards. They do wear out. I dismantle them and give them a good clean, redo all the heatsink gunk, change any sticky fans, then they magically work. While I'm in there, I check for visible damage with a lit magnifying glass - blown up or burst or leaking transistors or capacitors.

If you fail to fix it or don't want to, they have a decent resale value on Ebay when broken. Just describe the fault and take photos.

[nchowlett]

Thanks Peter! Just swapped in another GPU and my OS sees it, so I'll give your suggestions a go, thanks again.

[nchowlett]

... then they magically work.

I moved my case about 4 cm and suddenly it was booting (without infinitely rebooting). Go figure.

[Peter_Hucker]

Are you saying you fixed the GPU by moving something?

[nchowlett]

Are you saying you fixed the GPU by moving something?

Nope, but I was having issues where I couldn't boot into Ubuntu until I touched (moved) the motherboard (I think). Maybe a poor connection, they're off Ebay.

[Peter_Hucker]

I had a machine where the SSD suddenly failed to be detected. I pushed the SATA lead's connection on the motherboard end (lightly, not violently), and then I couldn't boot either. The motherboard and the SSD are now broken. Assuming it wasn't me as I was careful, I'm guessing the disk controller was shorting out and blew the SSD, then managed to blow the motherboard when I touched it. Oh well, it was an old quad core board and I got a new one for £10, and an SSD for £7. And then found things were incompatible, continued to buy more pieces, and ended up with two machines. More GPU sockets!

[nchowlett]

The motherboard and the SSD are now broken.

Dam, luckily I haven't had any such issues.

I'm trying to prototype a new configuration, your comments makes me think it needs to be almost production-quality at the same time (e.g. pre-designed and properly mounted to avoid human-induced errors). Annoying! But thanks for the insight, nonetheless.

[Peter_Hucker]

No, my stuff is cobbled together from what's available and cheap. My GPUs aren't even inside the cases, I use mining risers on USB cables.