AMD CPU systems & PCIE 4.0
Posted: Fri May 24, 2019 10:03 am
I've finished my personal research on AMD CPUs for powering GPUs for folding.
Is it worth jumping ship from Intel to AMD?
Short answer:
Depends...
Intel CPUs are limited to 16 PCIE lanes, which provides up to 3 GPUs, at PCIE 4x speeds or higher (in other words, no PCIE bandwidth limitation for GPUs in Linux).
Even the power hungriest RTX 2080 ti, at 300W (230W in power cap mode), would use no more than 1000W on the wall (800W when power limiting).
So there's clearly room for more! Though the 16 lanes aren't sufficient to add a 4th GPU without PCIE bandwidth limitation.
AMD CPUs give you access to 24 PCIE lanes instead of 16, which allows you to run an extra GPU at PCIE 4x speed, where as some slots on Intel boards would fall back to PCIE 1x speeds when adding more than 3 dedicated GPUs.
The long answer is:
It seems that the most interesting CPU to choose for folding is the AMD Ryzen 5 2600 CPU @ USD $148; or the Ryzen 3 1200 or 1300x.
Which ever CPU you choose, depends on how many GPUs you plan on running.
The Ryzen 3 CPUs have only 4 cores and are limited to 4 GPUs in Linux (3 in Windows). Great, if you don't plan on expanding beyond this. Also great that it can run each card at PCIE 4x or higher speeds.
The Ryzen 5 also allows for 4 GPUs at PCIE 4x, but allows for up to 9-10 GPUs (CPU thread limited) at reduced PCIE 1x speeds for some of the cards.
The Ryzen 7, I found, has a too high TDP (95W), and too many threads (16). Finding a motherboard that runs more than 7 GPUs is not efficient, nor realistic; so I won't choose this CPU for folding.
AMD CPUs have 32 PCIE lanes, and just like with Intel, only 16 PCIE lanes are directed towards PCIE slots, and good for up to 4x GPUs at the full PCIE 4x speed; 4 go to the chipset, which provides USB and Sata connections, etc...
But AMD also has an additional 4x PCIE lane for an m.2 slot, (This m.2 slot, with the right adapter can host a full size GPU at PCIE 4x speed, and in theory this is good for a fifth* GPU), where as Intel boards (being limited to only 16 lanes) will steal these PCIE lanes from the other GPU slots.
* Sadly, for the 5 GPU system, no manufacturer makes motherboards with 4x PCIE 4 lane slots, AND a 4x speed m.2 slot.
Nearly all motherboards provide the primary PCIE slot with 16 or 8 lanes. Not 4; which makes that the maximum you'll find is 3 GPUs at PCIE 4x or faster, PLUS one GPU fitted in the m.2 slot!
Any additional card beyond this 4 GPU setup, and some of the slots, as well as the extra slots, will fall back to (or are already at) PCIE 1x speed.
I said AMD has 32 PCIE slots, so the 8 missing PCIE lanes (16 go to PCIE slots, 4 to the chipset, and 4 go to the m.2 adapter) appear to be designated for the integrated graphics.
The Ryzen 5 (and Ryzen 3 1200/1300x) don't have the integrated graphics so the lanes are internally sealed, and supposedly inaccessible (unless someone can find the correct CPU pins and tap off the PCIE lanes from there).
So the benefit AMD offers over Intel is basically 4 PCIE lanes extra, on a 3 GPU system, or 1 extra GPU at PCIE 4x speed.
It's something, but maybe not enough to jump ship.
If you're looking to expand your Intel system beyond the 3 GPUs at PCIE 4x speed or higher, and want to eliminate a PCIE 1x speed bottleneck, perhaps it's good to wait for PCIE 4.0 (or even PCIE 5.0) to come out both on Motherboard and GPU; as PCIE 4.0 1x speed roughly equals PCIE 3.0 2x speed, and PCIE 5.0 roughly equals PCIE 3.0 4x speed.
And PCIE 3.0 2x speed for folding, is about where the bottleneck significantly reduces on RTX 2080 cards or lower in Linux.
PCIE 3.0 4x is still recommended for RTX 2080 ti cards in Linux.
Then again, if you'd run 4x RTX 2080 cards on a Linux based Ryzen 3 1200 or 1300x system, you'd probably be running THE most efficient system you could possibly run for folding, without major PCIE bottlenecks.
The only factor I haven't yet factored in, is that AMD motherboards might not provide sufficient power over their motherboard slots, to power 3 or 4 graphics cards with the 35W required to run a PCIE 4x slot at. Powered risers like these will be required!
Another one, is power consumption, and performance of a Ryzen 3 quadcore chip, VS a quadcore i3 or i5 Intel chip; and see if Ryzen offers the same performance as Intel in this matter, or if 2Ghz AMD CPUs will bottleneck a 2060, and 3Ghz a 2080 ti.
I hope some people with more experience can chime in. Perhaps compare results on Intel VS AMD CPUs, and how they affect folding and performance, as well as if AMD is as power efficient as Intel when it comes to folding?
Is it worth jumping ship from Intel to AMD?
Short answer:
Depends...
Intel CPUs are limited to 16 PCIE lanes, which provides up to 3 GPUs, at PCIE 4x speeds or higher (in other words, no PCIE bandwidth limitation for GPUs in Linux).
Even the power hungriest RTX 2080 ti, at 300W (230W in power cap mode), would use no more than 1000W on the wall (800W when power limiting).
So there's clearly room for more! Though the 16 lanes aren't sufficient to add a 4th GPU without PCIE bandwidth limitation.
AMD CPUs give you access to 24 PCIE lanes instead of 16, which allows you to run an extra GPU at PCIE 4x speed, where as some slots on Intel boards would fall back to PCIE 1x speeds when adding more than 3 dedicated GPUs.
The long answer is:
It seems that the most interesting CPU to choose for folding is the AMD Ryzen 5 2600 CPU @ USD $148; or the Ryzen 3 1200 or 1300x.
Which ever CPU you choose, depends on how many GPUs you plan on running.
The Ryzen 3 CPUs have only 4 cores and are limited to 4 GPUs in Linux (3 in Windows). Great, if you don't plan on expanding beyond this. Also great that it can run each card at PCIE 4x or higher speeds.
The Ryzen 5 also allows for 4 GPUs at PCIE 4x, but allows for up to 9-10 GPUs (CPU thread limited) at reduced PCIE 1x speeds for some of the cards.
The Ryzen 7, I found, has a too high TDP (95W), and too many threads (16). Finding a motherboard that runs more than 7 GPUs is not efficient, nor realistic; so I won't choose this CPU for folding.
AMD CPUs have 32 PCIE lanes, and just like with Intel, only 16 PCIE lanes are directed towards PCIE slots, and good for up to 4x GPUs at the full PCIE 4x speed; 4 go to the chipset, which provides USB and Sata connections, etc...
But AMD also has an additional 4x PCIE lane for an m.2 slot, (This m.2 slot, with the right adapter can host a full size GPU at PCIE 4x speed, and in theory this is good for a fifth* GPU), where as Intel boards (being limited to only 16 lanes) will steal these PCIE lanes from the other GPU slots.
* Sadly, for the 5 GPU system, no manufacturer makes motherboards with 4x PCIE 4 lane slots, AND a 4x speed m.2 slot.
Nearly all motherboards provide the primary PCIE slot with 16 or 8 lanes. Not 4; which makes that the maximum you'll find is 3 GPUs at PCIE 4x or faster, PLUS one GPU fitted in the m.2 slot!
Any additional card beyond this 4 GPU setup, and some of the slots, as well as the extra slots, will fall back to (or are already at) PCIE 1x speed.
I said AMD has 32 PCIE slots, so the 8 missing PCIE lanes (16 go to PCIE slots, 4 to the chipset, and 4 go to the m.2 adapter) appear to be designated for the integrated graphics.
The Ryzen 5 (and Ryzen 3 1200/1300x) don't have the integrated graphics so the lanes are internally sealed, and supposedly inaccessible (unless someone can find the correct CPU pins and tap off the PCIE lanes from there).
So the benefit AMD offers over Intel is basically 4 PCIE lanes extra, on a 3 GPU system, or 1 extra GPU at PCIE 4x speed.
It's something, but maybe not enough to jump ship.
If you're looking to expand your Intel system beyond the 3 GPUs at PCIE 4x speed or higher, and want to eliminate a PCIE 1x speed bottleneck, perhaps it's good to wait for PCIE 4.0 (or even PCIE 5.0) to come out both on Motherboard and GPU; as PCIE 4.0 1x speed roughly equals PCIE 3.0 2x speed, and PCIE 5.0 roughly equals PCIE 3.0 4x speed.
And PCIE 3.0 2x speed for folding, is about where the bottleneck significantly reduces on RTX 2080 cards or lower in Linux.
PCIE 3.0 4x is still recommended for RTX 2080 ti cards in Linux.
Then again, if you'd run 4x RTX 2080 cards on a Linux based Ryzen 3 1200 or 1300x system, you'd probably be running THE most efficient system you could possibly run for folding, without major PCIE bottlenecks.
The only factor I haven't yet factored in, is that AMD motherboards might not provide sufficient power over their motherboard slots, to power 3 or 4 graphics cards with the 35W required to run a PCIE 4x slot at. Powered risers like these will be required!
Another one, is power consumption, and performance of a Ryzen 3 quadcore chip, VS a quadcore i3 or i5 Intel chip; and see if Ryzen offers the same performance as Intel in this matter, or if 2Ghz AMD CPUs will bottleneck a 2060, and 3Ghz a 2080 ti.
I hope some people with more experience can chime in. Perhaps compare results on Intel VS AMD CPUs, and how they affect folding and performance, as well as if AMD is as power efficient as Intel when it comes to folding?