GPU folders: check your ReBAR...

[Albuquerquefx]

Last week I upgraded my 14 year-old i7-3930k Fedora 42 folding rig to a far more modern Ryzen 5 5500. The CPU isn't folding, instead I'm leaving the protein mangling to a 4070 Super (power limited to 200W) and a 4090 (PL=350W.) Both of these cards mildly underperformed compared to LAR Systems data, with an average of 9MPPD and 22MPPD respectively. I chalked up the mildly lesser scores to the reduced power limits, and of course to the ancient CPU and motherboard.

After moving the cards to the new CPU + Motherboard + DDR4 memory, and installing a fresh copy of Fedora and the NVIDIA 580 drivers. I got everything configured and running, and to my dismay the cards were cranking out the same underwhelming 9MPPD and 22MPPD. Even with a CPU which should be easily double the performance, and a motherboard with full PCIe 4.0 x8/x8 support, I couldn't understand why they underperformed.

I spent a day futzing with memory overclock, CPU overclock, stability testing, benchmarks, trying to understand what I was missing, FInally, I accidentally stumbled into the BIOS -> Advanced -> PCI settings and discovered Above 4G Decode and ReBAR were disabled, because the compatibility service module (CSM) was still enabled for boot. CSM is necessary for really old operating systems that don't understand UEFI, which is absolutely not a problem for any OS built in about the last seven to ten years. I disabled CSM, which allowed me to enable Above 4G Decode and ReBAR, and I booted back into Fedora.

To my satisfaction, the 4070 Super performance jumped to about 11.5MPPD and the 4090 started cranking out about 25MPPD, both are still power limited as before, and now both slightly outperform the datapoints on LAR Systems. After a bunch of searching online, it looks like a lot (maybe most? not all...) AM4 motherboards shipped from the factory with CSM enabled, which also means ReBAR is disabled. It also looks like most (maybe not all?) AM5 boards shipped with ReBAR fully enabled.

It's worth checking, especially if you feel like you're not getting the folding performance you might have expected from your GPUs.