PCIE x speeds for F@H gpu crunching

rickoic · Post by **rickoic** » Thu Jun 11, 2020 6:23 pm

After reading your suggestions, and they were all helpful in one way or another. My decision is this:
I like the idea of using a mining frame. With an adaptor I can use 2 power supplies as the frame is designed for that (some have the adaptor already installed).
With ribbon risers I can install my gpu's with plenty of apace between them for better cooling. As well as having a fan far to increase air flow.
Easy to see any dust buildup and whisk it away with a small broom or compressed air.
And the frame is cheaper than a regular case.

Tks to all for the help and advice.
Rick

rickoic · Post by **rickoic** » Sat Jun 13, 2020 12:35 pm

OK, been too long and I haven't been able to find the instructions on how to get the NVIDIA drivers installed.. As well as the FAHClient install, and what are the changes needed for FAHDisplay?

Got UBUNTU 20.04 installed and running.

Much TKS
Rick

Post by **PantherX** » Sat Jun 13, 2020 12:56 pm

This is what I have read so far:
1) Ensure that Ubuntu is fully updated.
2) Install the proprietary drivers by Nvidia
3) Install OpenCL package (sudo apt-get install ocl-icd-opencl-dev)
4) Install FAHClient

If you mean FAHViewer or FAHControl instead of FAHDisplay, I am aware that FAHControl has python 2 dependencies and that you may need to modify a file to get that sorted out (someone with more Linux experience will advise you).

rickoic · Post by **rickoic** » Sat Jun 13, 2020 2:00 pm

As I said its been a full 10 years or more since I've played with Linux. So step by step instructions would really be a BIG help.

I tried the Install OpenCL instruction and got Unable to locate package...
I have downloaded NVIDIA-Linux-x86_64-440.82.run in my Downloads folder but what are the instructions to install it?

Much thanks
Rick

ajm · Post by **ajm** » Sat Jun 13, 2020 2:25 pm

There is a good how-to here: https://linustechtips.com/main/topic/10 ... lding-rig/

Now, this how-to is for Ubuntu 18.04. For 20.4 and the python 2 problem (only concerns FAHControl, so not absolutely necessary), you may want to try this: viewtopic.php?p=321446#p321446

MeeLee · Post by **MeeLee** » Sat Jun 13, 2020 4:05 pm

Nvidia updates it's drivers all the time.
Installation procedures change by the minute.
My last known procedure was the following:
After downloading the .run files, make the file executable (rightclick menu, or do chmod +x NvidiaDriver.run).
then reboot, go into recovery mode (grub, advanced boot, boot into terminal).
I usually do:
sudo init 3
sudo service sddm stop
(sddm can be gddm3 in your case, just look up how to disable the desktop manager of your distribution).
Then go to the nvidiaDriver.run folder.
Execute by:
sudo ./NVidiaDriver.run
(with the appropriate Nvidia driver file there)
Aknowledge/overwrite everything if the popups ask anything.
Some drivers crash at the 32 bit stage.
You don't need to install the 32 bit libraries.
It saves installation trouble, and isn't really needed anyway.

then reboot.

Your PC can also have fan curve, power settings and over/under-clocking options enabled by doing:
sudo nvidia-xconfig --cool-bits=28
Then reboot.
Do this only if you have 1 Nvidia GPU

If you have multiple Nvidia GPUs, it's strongly recommended to run 18.04 instead of 20.04, and input the next commands instead:
sudo nvidia-xconfig --enable-all-gpus
sudo nvidia-xconfig --cool-bits=28

Last I remember, the above 2 lines will crash the desktop on 20.04.
Not sure if the crash is limited to SDDM (as 18.04 uses LightDM), and if Ubuntu 19.04 and up (gddm2/gddm3) is affected by it or not.
You can give it a try. I just haven't successfully found a way to undo the desktop crashes '--enable-all-gpus' causes on SDDM, other than a complete uninstall of the drivers (or OS).

Post by **Joe_H** » Sat Jun 13, 2020 4:28 pm

rickoic wrote:I have downloaded NVIDIA-Linux-x86_64-440.82.run in my Downloads folder but what are the instructions to install it?

For most Linux distributions you can connect to a repository that will have the nVidia drivers packaged up for proper installation on that version of Linux. That is usually easier than installing from the .run file provided for download from nVidia.

The_Bad_Penguin · Post by **The_Bad_Penguin** » Wed Jun 17, 2020 2:27 pm

Just as a follow-up / FYI.

Don't know what the policy is about putting in links, so I will just note that I came across a new (June 16, yesterday) YouTube video entitled "TWO Graphics Cards in ONE Slot?!" that discusses essentially pcie bifurcation.

You can search under that title or the content creator, "Techquickie".

FWIW.

Post by **bruce** » Wed Jun 17, 2020 3:13 pm

Links are prohibited until you've used the forum for a while.

MeeLee · Post by **MeeLee** » Fri Jun 19, 2020 12:31 am

I've only once tried a $200 PCIE x16 to 4x x4 slots, and it wasn't supported by the bios.
The other time, I tried a PCIE x1 splitter to 4x USB (for USB risers) and even then it wasn't supported (PCIE error from bios).
The only time it worked, was when one of my motherboards told me there are more PCIE lanes than the CPU could supply (I believe it was an MSI MPG board).
And it wouldn't have made sense either, even on core 21, to split an x1 slot in anything.
PCIE 3.0 x1 was barely enough for core 21. Anything slower would not have worked well...Let alone on core 22.

The_Bad_Penguin · Post by **The_Bad_Penguin** » Sat Jun 27, 2020 2:31 pm

I have a tangential question, and figured it was close enough to this thread not to warrant a new thread.

What is really the maximum number of gpu's you can run off a motherboard?

For this question, ignore power consumption, physical spacing, etc.

Just concern ourselves with PCIE lanes.

I understand that a motherboard's chip set determines the number of available PCIE lanes, e.g., 16, 36, 40.

So, if the chipset has 40 lanes, and has 3 pcie-x16's (electrically), that alone is 48 lanes, without taking into account any other peripherals wanting PCIE lanes.

What happens then?

Can you F@H crunch with 48 lanes on a 40 lane chipset?

Are there chipsets which will allow for more than 40 PCIE lanes? A motherboard for AMD threadripper 39xx cpu's perhaps?

Post by **bruce** » Sat Jun 27, 2020 3:43 pm

FAHClient allows you to configure 10 GPUs numbered 0 through 9.

Post by **Joe_H** » Sat Jun 27, 2020 3:46 pm

For F@h, the answer is 10. The client can enumerate from 0-9, it currently will not utilize more than 10 GPUs.

As for the rest, it will be up to the logic board and its chipset. The lanes would need to be coming from a combination of the ones supported directly by the processor and others supplied by one or more bridge chips. In the example you propose, if there are 40 lanes and three 16x slots at least one of them will be running at 8x or less.

The_Bad_Penguin · Post by **The_Bad_Penguin** » Sat Jun 27, 2020 7:58 pm

OK, so let me see if I am correctly understanding.

For the sake of this example, let us assume an ASUS P9X79-E WS motherboard has an Intel X79 chipset with 48 lanes of PCIE. Let us also assume that the motherboard also has 7 PCIE slots: 4 are PCIE-x16 (electrically) and 3 are PCIE-x8 (electrically). Let us assume (using risers) we are able to put modern gpu's in each of the 7 slots. By my math, that is 4x16lanes + 3x8lanes; 64 lanes + 24 lanes = 88 lanes. 88 > 48, thus all the x16's would run at x8, and all of the x8's would run at x4 (32+12 = 44), So 44 out of 48 lanes would be used by the 7 gpu's, each running at 50% of their pcie-x electrical abilities, with 4 pcie lanes left over for other system use.

I think I saw that the chipset for current AMD ThreadRipper motherboards is TRX40, and does have 88 PCIE lanes.

Have to look to see if what sTRX4 motherboards offer as far as maximum number of physical slots of pcie-x16 (electrical) and pcie-x8 (electrical).

If I understand everything, if such a motherboard exists with 5 pcie-x16 electrical slots (and 88 pcie lanes), it could gpu F@H crunch with no downgrading / degradation.

EDIT --> Apparently there are only 72 usable of the 88 total pcie lanes.

"... because we’re also increasing the CPU<->chipset link from 4x Gen4 to 8x Gen4—quadruple the bandwidth vs. 2nd Gen TR. Extra data pins between the chipset and CPU make this possible, so you’ll be able to hang more I/O off the motherboard at full performance.”

ajm · Post by **ajm** » Sat Jun 27, 2020 8:12 pm

It's not that "simple". The manufacturers only give general indications and how the board will split the lanes when they are too scarce can be a bit of a mystery.
Gigabyte propose an online system for that, where you can choose your MB, your CPU and how you intend to populate the PCI slots (and shared M.2 slots) - the system then tells you exactly where you'll get what: https://www.gigabyte.com/FileUpload/glo ... /x299.html
But I don't know for other manufacturers. If you don't find such a system, it would be safer to ask them directly before the purchase.

Folding Forum

PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching

Re: PCIE x speeds for F@H gpu crunching