Folding Forum

Hi and sorry if this has been through before!

I have a number of different rigs all folding as GPU and am confused by the relative performance of Linux and Windows.

An i7 Windows 10 machine with a Titan X running a 94xx struggles to get much over 500K ppd
An i3 Linux Mint machine with a 980ti running a 94xx gets over 600K.

Can anyone explain why Linux is so much faster?#


Other units types like 11xxx there's no Windows impact so it seems to be specific WU types are particularly crappy on Windows.

Though both on Linux and Windows FAH uses common OpenCL API, the GPU drivers implementation are different.

Currently for Nvidia GPUs and the most common work units, Linux is faster than Windows.

I would have expected it the other way round, that nvidia drivers for Windows would be more optimized but that maybe only true for games.

Between the F@H software and the actual hardware is a Hardware Abstraction Layer (HAL) The efficiency of the HAL will depend on the OS. Some OSes hide more of the hardware behind layers of software than others.

https://en.wikipedia.org/wiki/Hardware_abstraction

Windows in particular has to assume a graphics card may be MGA, CGA, EGA, or VGA on a ISA, VESA,, EISA, PCI or PCIE bus. that is a lot of abstraction.

There are some strange things happening to some people with p94xx on Windows failing to engage full throttle on those WUs ... Do clocks look normal when running those WU ? Did you try to select "Prefer maximum performance" in NV drivers ? Do clocks look normal when running those WU ?

But yes, there's naturally a small difference in performance between Linux and Windows (mostly because of WDDM being conservative on what's allowed to do with the GPU to avoid BSOD).

JimboPalmer wrote:Windows in particular has to assume a graphics card may be MGA, CGA, EGA, or VGA on a ISA, VESA,, EISA, PCI or PCIE bus. that is a lot of abstraction.

I think that you're forgetting that Win10 also runs on Android & iOS. I'll bet that takes some additional abstractions. Though that''s not necessarily using the GPU drivers we use, they probably still run through many components of WDDM.

toTOW wrote:There are some strange things happening to some people with p94xx on Windows failing to engage full throttle on those WUs ... Do clocks look normal when running those WU ? Did you try to select "Prefer maximum performance" in NV drivers ? Do clocks look normal when running those WU ?

But yes, there's naturally a small difference in performance between Linux and Windows (mostly because of WDDM being conservative on what's allowed to do with the GPU to avoid BSOD).

Clocks look normal.

The Titan with Windows10 is on a faster motherboard, faster CPU, runs water-cooled at 1402, standard RAM. GPU power at 76% and GPU usage 84% but my current 9414 unit is pushing at 1:11 TPF for <500K.

The 980ti with Linux Mint is also water-cooled at 1418, but has a TPF of 1:00 and is pushing >600K.

To get the most out of the Titan it looks as if I will have to move it to another Linux rig.

The 94xx units are horrible for the high end cards, and windows honestly just sucks when it comes to folding. Windows is a resource hog for folding, and should not be used at all. For folding in windows you need super high end hardware as you need to have at least 1 full core per card, in Linux, you can literally have a toaster, and the card will perform awesome (currently have a 1080Ti in a AMD A5200 pulling 15% more PPD than it was in my Xeon rig) Nvidia doesn't seem to care about the fact that windows is so bad for folding, as they still haven't addressed the fact that 372.90 is the best driver for non Ti Pascal cards, a driver that is over a year old. Also, linux doesn't use anywhere near as much PCIE bandwidth as windows. Put your Titan on Linux and watch your scores soar.

With the release of the new Titan V, I am hoping Stanford will start to address the fact that we need larger work units, as the current set does not address the sheer power that some of the cards have.

Unfortunately, the driver model (WDDM) in Windows is to blame for this situation ... nVidia can't do much about this. It considers that a GPU is mainly made to display things and that people will be angry if something goes wrong (freeze, GPU reset, ...). So you can't push it too hard with GPGPU load.

The only solution would be to use the TCC drivers from NV, but they won't install unless you have a Tesla card, which is not recognized as a display GPU in Windows ...

Maybe the fah core code could be optimized to better adapt to the nvidia driver behavior on Windows. But that would be extra work to do.

The only improvement we can expect is to free the CPU core from active wait used in OpenCL by moving to CUDA passive wait, but the driver will still prioritize graphics and stability over GPGPU performance.

For those who know GPUGRID in BOINC, which use a software called ACEMD that uses CUDA on Windows, they have the same issue as ours. On Windows it can't use the GPU at its full potential because of driver limitations. The only good point is that it only use a couple of CPU % to feed the GPU.

NVidia is putting their resources into Windows drivers that deliver a high frame-rate in Windows games. That's where the sell the most chips. GPGPU processing on home computers is a small market for them.

CUDA is also a key market for NVidia, with less emphasis on OpenCL. FAH develops for OpenCL because it supports both AMD/ATI and NVidia GPUs. Developing two versions of the FAHCore, one for OpenCL and another for CUDA, would increase FAH's overall performance slightly, but it's difficult for them to devote the extra resources into supporting a second version with their limited development resources. (Sometimes in the past, they have, and they might again in the future.)

HaloJones wrote:Can anyone explain why Linux is so much faster?#

It's all about GPU utilization under OpenCL loads.
I have an old s1366 system - 6 core Xeon @ 3800 MHz, PCIe 2.0 x16 slots and GTX 1070 clocked 2050 MHz.
With Windows 10 GPU utilization most of the time was 82-86%, nearly the same with any of new drivers (with some very rare WUs gives GPU load spikes to 90%), bus load was 40-44% (by GPU-Z), PPD was from 570k to 640k and average was 600k.
Linux Mint 18.1 with Xorg drivers 370.28 gives GPU utilization most of the time no lower than 95% - with some WUs it peaking to 97-98%, bus load is 1% (by Psensor), with PPD from 770k to 870k and average above 800k.
Same GPU clocks of 2050 MHz in both cases, with Win 10 was set by EVGA precision, with Linux set by editing xorg.conf with 'coolbits 31' line and adding MHz in Nvidia settings panel.
In both OSes most of current WUs takes single CPU-thread loaded up to the roof - I think faster modern CPU may give even slightly higher GPU utilization level under Linux - with PPD average around 900k with GTX 1070@2050.

Frontiers wrote:I think faster modern CPU may give even slightly higher GPU utilization level under Linux

I tried i5-2500k 3.3Ghz vs 4.4Ghz on Windows and there was no difference in GPU utilization, but 1.6Ghz was too slow and PPD goes down.

foldy wrote:I tried i5-2500k 3.3Ghz vs 4.4Ghz on Windows and there was no difference in GPU utilization, but 1.6Ghz was too slow and PPD goes down.

and
I'm guessing that the number "1.6GHz" would scale with the bandwidth of your PCIe slot(s)?
... and it might scale with the particular GPU that's in that slot.

The idea is that throughput should be limited only by the charistics of the data path, not by the CPU. The CPU has to be fast enough saturate the slot. (Also, the GPU has to be fast enough to process the data as fast as it gets it and main RAM has to be fast enough to keep up with that bandwidth.)

This would also answer the question "What happens if I put a * GPU in a x* slot?" and we could make a list of slow GPUs that work fine in slow slots.

bruce wrote:I'm guessing that the number "1.6GHz" would scale with the bandwidth of your PCIe slot(s)?

It was the same CPU i5-2500k and GPU gtx 1080ti and work unit but I limited CPU clock speed in Windows using maximum CPU power state = 50% (tried to save some watts, but PPD goes down).