CPU folding without losing GPU PPD (a quick guide)

[arisu]

It's no secret that Nvidia GPUs are notorious for being unhappy if CPU folding is going on at the same time. Even if a core or two is reserved for the GPU's thread on the CPU, Nvidia's folding PPD will drop and the CPU folding PPD won't make up for it. I've solved this issue on Linux and I'm writing down what I did in case anyone else is interested.

I've been doing testing on two RTX 4090 Mobile GPUs, one RTX 3090, and one GTX 770M and I've managed to eliminate this issue, making it possible to fold on an Nvidia GPU and on the CPU without upsetting the GPU. Here are the requirements:

• All threads must only run on P-cores, never E-cores
• The initial GPU thread (the one using 100% CPU) must be pinned to a single core
• The CPU folding threads must be pinned to their own cores
• The core that the GPU thread is on must have its maximum clock rate limited (*)
• No folding threads may use the HyperThreading/SMT pair that the GPU thread runs on
• Enough cores must be left unused that the GPU thread is not thermally throttled (*)

(* if your chipset can sustain its all-core boost indefinitely, these two steps can be skipped)

The most important parts are pinning the threads to cores, avoiding thermal throttling, and not loading the core that is the SMT pair of the one that the GPU feeding thread is running on.

Locking the clock of the core the GPU thread is on is not strictly necessary, but it is important for for finding out how fast it has to be. For an RTX 4090 Mobile, I find that limiting one Intel i9 14900HX P-core to 2.5 GHz is the lowest it can go before the GPU starts being starved (on some projects it can go lower, as low as 2.2 GHz). You'll want to test this out yourself. If in doubt, just lock it to the base clock.

Then when you start up CPU folding, wait 10 seconds for the cores to get up to temperature and then check the core that the GPU thread is running on, and make sure that it is still running at the maximum you gave it. If it is being throttled, then reduce the number of cores for CPU folding by 2 and check again.

If all these requirements are met, you'll achieve GPU folding and CPU folding at the same time without performance loss. For an i9 14900HX with an RTX 4090 Mobile, these are the steps I use to fold on 12 cores, plus GPU:

Code: Select all

# taskset -acp 0-11 $(pgrep FahCore_a)
# taskset -acp 0-15 $(pgrep FahCore_2)
# taskset -cp 15 $(pgrep FahCore_2)
# echo 2500000 > /sys/devices/system/cpu/cpu15/cpufreq/scaling_max_freq

That pins all CPU folding threads to cores 0-11, all GPU threads to 0-15 (to speed up the periodic sanity checks done at each checkpoint), and then switches the first GPU thread (the one that uses 100% CPU and feeds the GPU) to only core 15. It leaves cores 12 and 13 idle to prevent thermal throttling (I should probably leave 0 and 1 idle instead because 0 is the "main" core and is the one the scheduler favors), and it leaves core 14 idle because it is the SMT pair for core 15 (that is, cores 14 and 15 are actually one physical core. Which cores share the same physical core can be found with the command "lscpu --extended"). It also reduces core 15's clock speed to 2.5 GHz.

When I do this, the core running the GPU thread maintains 2.5 GHz and the cores doing CPU folding run at 2.6-2.7 GHz. Compared to only pinning the GPU thread and not folding on the CPU, I lose up to 1% GPU PPD at most (about 100k PPD out of 10M total), but gain more than 300k PPD from the CPU folding. In most cases, I don't lose any noticible PPD at all.

In quick tests, I believe a dedicated Linux folding machine can gain fairly significant PPD by performing a few extra steps that involve isolating certain CPUs from the scheduler and disabling timer interrupts on them.

[enroscado]

Thank you arisu.

I imagine there must have been a LOT of testing combinations of different settings to get this. Respect!

[jfb9301]

So I know:
1) This hasn't been posted on in 8 months, and it never got many posts to begin with.
2) I just created my account today, and this would be my first post, so I hardly have any "cred" here.
3) I am working with windows on this, not Linux.

For reference, I've been with Team 33 [H]ardOCP for 20 years now. I know GPU folding makes the most points, but I want to be able to Fold CPU along with GPU without compromising my GPU folding. #allWUsmater.

I started with N-2, where N is the number of REAL CPU cores. So for my 9950X3D and RTX5080, I set my folding to 16-2 at 14 cores. The GPU gained far more PPD than the CPU lost. There were losses though. Somewhat following what it says above, I added a step to my thinking reserving X CPU cores. I wanted to reserve 1 real core for the GPU at all times, and the general guidance was to reserve 1 CPU core for system tasks.
I wanted those cores to be real cores, and I also reserved their associated HT cores. Also now that I am counting HT cores so N is now 32 not 16.

In summary N-2 becomes N-2X. 32-2(2)=28.

9950X3D - CPU0-3 for Core 0x27 (GPU), and CPU4-31 for Core 0xa8 and oxa9(CPU)
10700K - CPU0-3 for Core 0x27 (GPU), and CPU4-15 for Core 0xa8 and oxa9(CPU)

Now I know it's relatively simple to assign permanent affinities in Linux, under windows, things become tougher. You can use Task Manager or better yet download the Power Tool Process Explorer. However, you can only set the affinity for the current running processes. With GPUs finishing WUs in 45 minutes, that would be a new process with the new WU, and also when that happens, the CPU core restarts (god only knows why, it's annoying), so you loose all of your settings, and is a PITA when you have 32 cores to assign. Right now I trying things out with Process Lasso (https://bitsum.com/tools/cpu-affinity-calculator/).

I am seeing the significant gains of reserved REAL CPU cores for my GPU Folding
I am keeping most of my speed of my CPU folding.

My RTX5080 is now 21-25MPPD, around a 1-3MPPD gain over no reserved cores.
My 9950X3d is now 1.0-1.4MPPD, only giving up 100KPPD vs no reserved cores, but gaining as much as 500KPPD vs my original N-2 method.
AND it can be done in Windows using Process Lasso.

I would love for someone to check this out and confirm what I am seeing.

Also to the OP could you say how you use the code in the code box above on Linux. I will be putting together an Intel 10700K RTX3070 Ubuntu 2404 LTS together later and want to apply your work to it.

[calxalot]

The CPU core gets interrupted because available cpus fluctuates.
To avoid this, use resource groups in web control config.
(click padlock at lower right of config panel)

Set cpus to zero in a separate group for your gpus.
Only allocate cpus in one group, which has no gpus.

[jfb9301]

Well, thanks for that advice, I will give it a shot on my next day off.

[jonault]

I replaced Windows with Linux on my dedicated folding PC this weekend. I changed the setup just now to move the CPU cores to their own group, hopefully that will give an extra boost beyond the OS swap.

[jfb9301]

I am thinking that there will not be the same gains to be had. I believe you still need to constrain the CPU folding to limited cores. I am having luck with letting GPU have all available cores, and setting affinity for the CPU to limited cores.