PCI-e bandwidth/capacity limitations

[Nathan_P]

How do I know if my system has enough lanes to add a 1x device without degrading the primary slot?

Some motherboard manuls include a block diagram that shows what is connected to what. On my Asus boards the x1 slots are fed from the PCH, not the cpu. The cpu lanes are used to connect the x16 slots - usually 2 or 3 and depending on socket choice will switch between x16, x8 and x4 as needed

[boristsybin]

How do I know if my system has enough lanes to add a 1x device without degrading the primary slot?

1. Go to http://ark.intel.com/#@Processors and look for your processor specs -PCI Express Revision -PCI Express Configurations
for example, for ordinary desktop intel CPUs any of third version of i3 and lower supports only pci-e 2.0 with 16 lines maximum; and third version of i5 and higher supports pci-e 3.0 with 16 lines max.

2. Go to mb manufacturer site, and look for motherboard specs
for example, for asus X99-E WS https://www.asus.com/Motherboards/X99E_ ... fications/

"Expansion Slots
40-Lane CPU-
7 x PCIe 3.0/2.0 x16 (single x16 or dual x16/x16 or triple x16/x16/x16 or quad x16/x16/x16/x16 or seven x16/x8/x8/x8/x8/x8/x8)"

or for GA-X99-Ultra Gaming http://www.gigabyte.com/products/produc ... id=5809#sp

"Expansion Slots
2 x PCI Express x16 slots, running at x16 (PCIEX16_1, PCIEX16_2)
* For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install it in the PCIEX16_1 slot; if you are installing two PCI Express graphics cards, it is recommended that you install them in the PCIEX16_1 and PCIEX16_2 slots.
* When an i7-5820K or i7-6800K CPU is installed, the PCIEX16_2 slot operates at up to x8 mode.
2 x PCI Express x16 slots, running at x8 (PCIEX8_1, PCIEX8_2)
* The PCIEX8_1 slot shares bandwidth with the PCIEX16_1 slot and the PCIEX8_2 slot shares bandwidth with the PCIEX16_2 slot. When the PCIEX8_1/PCIEX8_2 slot is populated, the PCIEX16_1/PCIEX16_2 slot operates at up to x8 mode.
* When an i7-5820K or i7-6800K CPU is installed, the PCIEX8_2 slot becomes unavailable.
(All of the PCI Express x16 slots conform to PCI Express 3.0 standard.)
1 x PCI Express x1 slot
(The PCI Express x1 slot conforms to PCI Express 2.0 standard.)
1 x M.2 Socket 1 connector for the wireless communication module (M2_WIFI)"

if you feel not enough clearance with webinfo about mb, then download pdf manual for it.

It is realy a challenge to choose the right cpu and mb for even just two GPUs at one system.

[foldy]

I propose that there's a direct relationship between the minimum speed of the PCIe bus and the GFLOPS of the GPU. [...] If so [and adding x1 slot GPU], at what point would the "really low performance GPU" become too fast to be worthwhile?

We had the tests with a gtx 1070 with 6.5 Tflops which lost 50% on x1 compared to x16 on Windows not Linux.
And a R9 280X with 4Tflops which lost 10% on x1 compared to x16 on Windows not Linux.
So a first assumption could be a 3 Tflops GPU can do without performance loss on Windows.
I guess all GPUs doing around 100k PPD or less do not reach the pcie limit on Windows.
And even with 10% loss the 100k PPD is still worth it.

So to simplify the pcie limits discussion is only relevant for the current fast GPUs above 3Tflops on Windows.

[des1957]

Thank you for all the work PS3EdOlkkola. I see you are using Win 7 on all your systems. Have you tried Linux. From my experience with Nvidia cards, I show a significant increase in points on Linux with 2 GTX 1070s. With all your systems, that could be a major increase in points. Just a thought.

[ComputerGenie]

We had the tests with a gtx 1070 with 6.5 Tflops which lost 50% on x1 compared to x16 on Windows....

Had the same experience with 1080s; with 3 1080s on 1x-16x USB3 risers, I was only getting ~1.2M

[PS3EdOlkkola]

@des1957 Yes, all systems -- except one -- are Win7 x64. One system is running the linux distro CentOS 7 on an Intel Xeon Phi 7210 CPU (256 threads), and in the past ran a 2x AMD 6274 (32 cores) Opteron on Ubuntu. Installing and maintaining GPU drivers on Linux appears a fair bit more complicated than using Windows. Now that I have some experience using Linux with the Phi box, I may build a system just to experiment with adding and updating GPUs so I can learn the ins-and-outs of how to do it efficiently and with repeatability. I could be wrong, but it also seems the number of test, diagnostic and management tools for GPUs is more comprehensive on Windows than Linux. I know I have to get off of Win7 eventually so beginning the process of migrating certainly does make sense.

[des1957]

I run Mint 18.1 and Nvidia driver 370.28. Driver installation has become a lot better. The xserver that comes with driver does a pretty good job of displaying quite a bit of info for diagnostics. PCIE GEN,whether the card is at 16x or 8x, bandwith usage,temp and clocks. It will also allow you to overclock. You have done a great job of informing us of multi gpu setup and proper motherboards to run. Eventually I will get up to 4 gpus with your guidance. Thanx

[boristsybin]

my little experience
rx480@pci-e2.0x1 ~250k ppd
rx480@pci-e2.0x4 ~320k ppd
rx480@pci-e2.0x16 ~320k ppd

[Aurum]

my little experience
rx480@pci-e2.0x1 ~250k ppd
rx480@pci-e2.0x4 ~320k ppd
rx480@pci-e2.0x16 ~320k ppd

I've never gotten that high with my RX480s. Do you overclock

The proportional drop to x1 is about what I see.

[boristsybin]

I've never gotten that high with my RX480s. Do you overclock
The proportional drop to x1 is about what I see.

for now my rx480 displays 262k@13206 and 279k@10496
may be those were some speedy tasks

[leggazoid]

GPU drivers in linux the easy way...

Install Ubuntu 16.10

sudo apt-add-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-370 nvidia-cuda-toolkit
sudo nvidia-xconfig -a --cool-bits=31 --allow-empty-initial-configuration
wget http://launchpadlibrarian.net/109052632 ... 15_all.deb
sudo apt install python-support_1.0.15_all.deb
wget https://folding.stanford.edu/releases/p ... _amd64.deb
sudo apt install fahclient_7.4.4_amd64.deb
wget https://folding.stanford.edu/releases/p ... -1_all.deb
sudo apt install fahcontrol_7.4.4-1_all.deb
wget http://fah-web.stanford.edu/file-releas ... c/GPUs.txt
sudo cp GPUs.txt /var/lib/fahclient/

Edit by Mod:
corrected the spelling from "add-apt-repository"
(No such command)

[boristsybin]

I've never gotten that high with my RX480s. Do you overclock

Now getting 313k @ 11707 , 323k @ 11710, 312k @ 9178
no overclocking

[Aurum]

I have an MSI 870-G45 mb with two x16 PCIe slots. Their spec says with 2 cards installed it should run x16x4 @ 2.0 but it only runs x16@2.0 x4@1.1
The x16 is a GTX 1070 and the x4 is a GTX 970. When they started folding the speed upshifted on x16 from 1.1 to 2.0, as I watched GPU-z, but it does not change for the x4 card. Any ideas on why? I have a dual-core AMD Athlon II X2 440 (unlocked Sempron 140) and 8GB DDR3 RAM.
I just swapped out a 1070 from that 2nd slot because it was choked down ~50%. The 2nd slot 1070 never upshifted from x4@1.1 either. Should I try a better CPU

[bruce]

I suggest you ask the manufacturer of your motherboard.

[boristsybin]

is here any expereance with GPUs folding on pci-e risers?
so, questions:

1. what is the best gpu to use in folding on win7x64 with pci-e x1 riser on 1.1 slot?
2.1. is cheap non-shielded pci-e x16-x16 or pci-e x16-x4 20cm riser enough to provide stable folding at pci-e 3.0 standard? if not, does handmade aluminium foil shield helps?
2.2. is "premium" Termaltake riser realy good for folding?

i know about 3M risers, they are perfect, but too expensive for just risers