Many GPUs on one machine
Moderator: Site Moderators
Forum rules
Please read the forum rules before posting.
Please read the forum rules before posting.
-
- Posts: 308
- Joined: Wed Feb 16, 2022 1:18 am
Many GPUs on one machine
I've just purchased the following which (by my calculations, including CPU speed) should allow Folding to run flat out on up to 12 GPUs:
Esonic B250 BTC Gladiator motherboard (TWELVE PCI express sockets)
i3-6100 4-core CPU
256GB NVME.
(Borrowing RAM from another machine. Found case, cooler, power supply lying in garage).
I'm doing this because connecting 4 GPUs to one socket in my current machine causes folding to slow down. The CPU is not the limitation, sharing a PCI-express lane is. The GPUs are Tahitis (Radeon R9 280X) and will be connected with PCI-Express 2 USB risers. If you have faster cards, you can use ribbons to get PCI-Express 3.
Esonic B250 BTC Gladiator motherboard (TWELVE PCI express sockets)
i3-6100 4-core CPU
256GB NVME.
(Borrowing RAM from another machine. Found case, cooler, power supply lying in garage).
I'm doing this because connecting 4 GPUs to one socket in my current machine causes folding to slow down. The CPU is not the limitation, sharing a PCI-express lane is. The GPUs are Tahitis (Radeon R9 280X) and will be connected with PCI-Express 2 USB risers. If you have faster cards, you can use ribbons to get PCI-Express 3.
-
- Posts: 71
- Joined: Sun Dec 02, 2007 4:29 pm
- Hardware configuration: Gigabyte Aorus Z590 Pro AX, Intel i9-10850K, 32GB Crucial Ballistix DDR4-2600, Samsung NVMe EVO 980 Pro 256GB, CoolerMaster liquid cooler ML360, Nvidia Titan X (Pascal), Dell Nvidia RTX 3080 10GB 4Y12V, Pop!_OS.
- Location: Fair Play, SC
Re: Many GPUs on one machine
I ran eight GPUs on a similar motherboard using PCIe x1 risers. In BIOS, hyperthreading should be disabled and you need to enable above 4k decoding.
-
- Posts: 308
- Joined: Wed Feb 16, 2022 1:18 am
Re: Many GPUs on one machine
I leave HT on, it gets more CPU work done. It only uses it if it needs to.
As for 4K decoding, I'll turn that on only if there's a problem, last time I put that on the computer wouldn't boot and I had to reset the CMOS. And that was a better machine - i5 8600K.
If it's too slow, I can get 1x to 16x ribbons, and just have to build a physical support structure to put the cards closer instead of just sitting on the shelf next to it. 3 foot USB cables make things much easier.
As for 4K decoding, I'll turn that on only if there's a problem, last time I put that on the computer wouldn't boot and I had to reset the CMOS. And that was a better machine - i5 8600K.
If it's too slow, I can get 1x to 16x ribbons, and just have to build a physical support structure to put the cards closer instead of just sitting on the shelf next to it. 3 foot USB cables make things much easier.
-
- Posts: 308
- Joined: Wed Feb 16, 2022 1:18 am
Re: Many GPUs on one machine
Correction, I've just read up on above 4G decoding (along with the newer "resizeable bar") and it sounds like they boost performance (about 15%). I'm gonna switch it on for all my machines. If you never see me again, it broke.
-
- Posts: 2
- Joined: Mon May 18, 2020 1:20 am
Re: Many GPUs on one machine
My pardons for jumping in months later. Re using various PCIe x1/x4/x8/x16 risers. My subjective experience suggests you may not want to get too carried away with the narrower connections.
My first setup used a Supermicro x8 dual socket, Extended ATX motherboard, mounted in a DIY plywood case. I had room for and used up to five GPU cards with risers attached wherever I could find room. Using NVTOP on Linux as my PCI performance monitor, I didn't appear to lose a noticeable amount of throughput with x8 risers compared to x16. Using x4, it noticeably sagged. Using the x1 riser via a USB cable, throughput was pathetic, this while I was using GTX1060s.
Unfortunately, I didn't write down any numbers back in 2020, and this was a 2012 mobo.
My first setup used a Supermicro x8 dual socket, Extended ATX motherboard, mounted in a DIY plywood case. I had room for and used up to five GPU cards with risers attached wherever I could find room. Using NVTOP on Linux as my PCI performance monitor, I didn't appear to lose a noticeable amount of throughput with x8 risers compared to x16. Using x4, it noticeably sagged. Using the x1 riser via a USB cable, throughput was pathetic, this while I was using GTX1060s.
Unfortunately, I didn't write down any numbers back in 2020, and this was a 2012 mobo.
-
- Posts: 308
- Joined: Wed Feb 16, 2022 1:18 am
Re: Many GPUs on one machine
It depends on the card's power and the PCI express version. One lane of PCI Express version 2 will support 3 280X cards on Folding at full speed.
I'm putting more than one card per lane, not more than one lane per card.
I'm putting more than one card per lane, not more than one lane per card.
-
- Posts: 48
- Joined: Tue Feb 28, 2023 4:05 am
- Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
- Location: Melbourne, Australia.
Re: Many GPUs on one machine
I'm having issues with Ubuntu recognising my last GPU (others are picked up by the OS fine). Has anyone had any issue of the GPU looking like it was physically working (lighting up and fans moving), but once in Ubuntu it didn't appear?
-
- Posts: 308
- Joined: Wed Feb 16, 2022 1:18 am
Re: Many GPUs on one machine
Frequently. But in Windows and with old cards. They do wear out. I dismantle them and give them a good clean, redo all the heatsink gunk, change any sticky fans, then they magically work. While I'm in there, I check for visible damage with a lit magnifying glass - blown up or burst or leaking transistors or capacitors.
If you fail to fix it or don't want to, they have a decent resale value on Ebay when broken. Just describe the fault and take photos.
If you fail to fix it or don't want to, they have a decent resale value on Ebay when broken. Just describe the fault and take photos.
-
- Posts: 48
- Joined: Tue Feb 28, 2023 4:05 am
- Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
- Location: Melbourne, Australia.
Re: Many GPUs on one machine
Thanks Peter! Just swapped in another GPU and my OS sees it, so I'll give your suggestions a go, thanks again.
-
- Posts: 48
- Joined: Tue Feb 28, 2023 4:05 am
- Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
- Location: Melbourne, Australia.
Re: Many GPUs on one machine
I moved my case about 4 cm and suddenly it was booting (without infinitely rebooting). Go figure.
-
- Posts: 308
- Joined: Wed Feb 16, 2022 1:18 am
Re: Many GPUs on one machine
Are you saying you fixed the GPU by moving something?
-
- Posts: 48
- Joined: Tue Feb 28, 2023 4:05 am
- Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
- Location: Melbourne, Australia.
Re: Many GPUs on one machine
Nope, but I was having issues where I couldn't boot into Ubuntu until I touched (moved) the motherboard (I think). Maybe a poor connection, they're off Ebay.
-
- Posts: 308
- Joined: Wed Feb 16, 2022 1:18 am
Re: Many GPUs on one machine
I had a machine where the SSD suddenly failed to be detected. I pushed the SATA lead's connection on the motherboard end (lightly, not violently), and then I couldn't boot either. The motherboard and the SSD are now broken. Assuming it wasn't me as I was careful, I'm guessing the disk controller was shorting out and blew the SSD, then managed to blow the motherboard when I touched it. Oh well, it was an old quad core board and I got a new one for £10, and an SSD for £7. And then found things were incompatible, continued to buy more pieces, and ended up with two machines. More GPU sockets!
-
- Posts: 48
- Joined: Tue Feb 28, 2023 4:05 am
- Hardware configuration: Modified 'mining' rig with 4 parallel GPUs. For more info check out this article: https://tessellate.science/microscoped- ... ercomputer
- Location: Melbourne, Australia.
Re: Many GPUs on one machine
Dam, luckily I haven't had any such issues.The motherboard and the SSD are now broken.
I'm trying to prototype a new configuration, your comments makes me think it needs to be almost production-quality at the same time (e.g. pre-designed and properly mounted to avoid human-induced errors). Annoying! But thanks for the insight, nonetheless.
-
- Posts: 308
- Joined: Wed Feb 16, 2022 1:18 am
Re: Many GPUs on one machine
No, my stuff is cobbled together from what's available and cheap. My GPUs aren't even inside the cases, I use mining risers on USB cables.