PCI-e bandwidth/capacity limitations

[Aurum]

...on an Intel Xeon Phi 7210 CPU (256 threads)...

Hi Ed, I'd love to hear more about your experience with Intel Xeon Phi

[GPU timpster]

Hello,

I'd like to talk about how PCI-e speeds affect the folding speed of the GPU. I'm thinking of building a system with an AM4 motherboard, that does x8 + x8 and it also have another PCI-e slow that is PCI-e 2 at x4 speed, which is x2 speed in pci e 3, but x8 speed in gen 1 of PCI-e.

Does anyone know if this bandwidth makes any difference at all for folding@home on the GPU, and if I choose a motherboard with a bunch of additions that had a few PCI-e x1 speed 3.0 and also 2.0 x1 speed slots, would this hinder the performance of the GPU significantly?

There are now "mining" motherboards, that have a BUNCH of these PCI-e ports, but they are very very low bandwidth ports, like x1 as stated above. I'm wanting to just get a huge 1200+ watt PSU and throw in a bunch of old AMD cards once the prices go back to normal. I'll also have an 8 core, 16 thread AM4 CPU. that will work on 12 threads, four for several GPUs.

[bruce]

The influence of PCIe bandwidth on FAH depends on the project as well as the GPU that's doing the folding. The time required for a system to complete a WU is a combination of the time spent computing, the time spend sending and receiving data, and the time spend coordinating various distributed aspects of the processing. With a slow GPU, the time is dominated by the time to do the computations and the PCIe speed "doesn't matter much". On the other hand, a fast GPU can outrun the abilityh of the PCIe bus to maintain a prompt supply of data and it will spend more time waiting on that data to transfer. Then, too, there's a lot of variation between projects that influences these processes.

Nevertheless, we have a responsibility to be realistic. I have a GTX 560 Ti which was not being used. I connected it to a 1x powered riser and configured it to fold. Even though the GPU is not able to run at 100% GPU utilization, it's doing a lot more folding than the extra CPU thread that no longer folding because it's needed to drive the data through the PCIe bus.

[Aurum]

Hi GPU timpster, I have a few of those mining MBs, e.g. ASRock H81 Pro BTC. They have one x16 2.0 slot and some x1 2.0 slots. I just use them for mining with my legacy cards. Only the single x16 2.0 slot is capable of folding. Now they're coming out with versions that have even more x1 slots.

Why an AMD CPU and AMD cards? If you already have them and want to make the best of them for folding that's one thing, but, if you're building a folding rig from scratch you can do much better. Check the manual for an AMD CPU based MB and I believe the block diagram shows the CPU interfaces with the PCIe slots via the Northbridge chip, i.e. a switch which slows it down. The AMD GPU energy efficiency is less than half as good as Nvidia.

Here's some data on GPU performance and energy efficiency:
https://docs.google.com/spreadsheets/d/ ... edit#gid=0
https://docs.google.com/spreadsheets/d/ ... utput=html

If you want to build a smoking hot folding rig from scratch here's what I'm doing:
Gigabyte GA-X99-UD4 X99 mb ($149) http://store.gigabyte.us/refurbished-ga ... therboard/
The amazing thing about these Gigabyte X99 MBs is they use all 40 PCIe lanes from the CPU to run four x16 slots as x8x8x16x8 or with 3 cards as x16x16x8. Look at its web page and scroll about a third down and also read the section in its manual. It supports the x1 slots, SATA ports etc via the Northbridge.
http://www.gigabyte.us/Motherboard/GA-X99-UD4-rev-10#ov
I have 2 of these in the mail and have deployed variants GA-X99-UD3P and GA-X99-SLI.
These MBs have the Intel 2011-3 socket and can run Xeon CPUs. I buy my Xeons from bloommax on eBay and they've all worked great. Be sure to get at least 10 cores with hyperthreading (do not use the bottom Xeons like E5-2603). My E5-4640 v3's arrived today.
http://stores.ebay.com/bloommax/
My plan is to put an EVGA GTX 1080 Ti in the 3rd slot (x16) with 1070s and/or 1080s in the other three slots and three 1060s up on 1x risers. I expect it to fold 4.1 to 4.5 million PPD.

[foldy]

For fast GPUs like gtx 1070 and better you want pcie 3.0 x4 or pcie 2.0 x8 at least and still loose 10% speed on Windows. On slower pcie slots you may loose 50% speed and more on Windows.
That is not the case with much slower GPUs or when using Linux.

[GPU timpster]

Sorry everyone--when I made this topic, I didn't even think anyone would have created it, then I got taken back to the list of threads, and saw this one. I'm glad it's added inline, but there's more than enough information here to answers the questions that I originally had.

I'm wanting to go AMD because I want theme to not go out of business. You say that AMD GPUs are less efficient--that's because they just seriously do not have the R&D money as Nvidia, they have about half the R&D, and it's split between both CPU and GPUs. I really like what they're doing with Radeon Chill, and also the newer RX 500 series has three instead of just two power states.

I'm very glad you all still promote used (or slightly older) hardware, like the 2011 v3 chipset boards, as I was considering them in the past. I know if I buy used, I'm not directly supporting Intel, but AMD also loses my business and they will not be able to compete as well against Nvidia and Intel. It's really a David v Goliath game here to me.

I don't quite have the cash for such a system, but if I sell off my current hardware, I plan to have $700 - 800 to play with. Total. So, I probably want to go a bit below that.

I would like to wait for thread ripper from AMD, as they have 16 core, 32 thread CPUs, and the kicker is 64 PCI-e lanes so that will prevent any GPU limitations. Unfortunately, I probably won't be able to afford such a CPU, so I'll settle for just 24 lanes of PCI-e with their current 16 core CPU. It's only $300 at the moment on certain stores, and that leaves about 200 for the GPUs, considering I'll need a motherboard, ram, and I'd like another hard drive.

If AMD's very late "Vega" doesn't show up around .... Fall? this year, then I'll go all out with Nvidia GPUs, and the AMD CPUs.

One last thing, another reason I want to go AM4, is because they are working on their PCI-e passthrough, which will pass through those connections to a virtual machine. Hopefully they will have it all setup right, so I can fold on Linux, which as this thread has CLEARLY shown, results in lower PCI-e bandwidth used, and I can play games in a Windows virtual machine, seriously the BEST of both systems!

[Aurum]

I don't quite have the cash for such a system, but if I sell off my current hardware, I plan to have $700 - 800 to play with. Total. So, I probably want to go a bit below that.

Selling may not be your best option to start. There's an old thread I started about my HD 5970s that used to be the fastest BTC miners. Folks said I was nuts but I used earnings to upgrade gear and have since donated all my 5970s to a local computer charity. What gear do you have now? Tell us your specific model numbers and you'll find someone here has tried every possibility there is.

...thread ripper from AMD, as they have 16 core, 32 thread CPUs, and the kicker is 64 PCI-e lanes so that will prevent any GPU limitations. Unfortunately, I probably won't be able to afford such a CPU, so I'll settle for just 24 lanes of PCI-e with their current 16 core CPU. It's only $300 at the moment on certain stores, and that leaves about 200 for the GPUs, considering I'll need a motherboard, ram, and I'd like another hard drive.

Depending on how they implement the MB there could be eight x8 3.0 slots (my choice) or four x16 3.0 slots. Thanks for the tip, hadn't heard about it and will have to read up on it.
BTW, AMD CPUs have the most computing power for the buck and the Wraith Cooler works really well. I have AM3 and AM3+ MBs but know nothing about the AM4 Ryzen series. Also, the pins on an AMD CPU are far superior to the flimsy unreliable filaments used in the Intel 2011-3 sockets (look at them cross-eyed and they bend and short).
What's the highest BW AM4 MB?

[SteveWillis]

Aurum, what motherboard are you using now?

[Aurum]

These are my best: Gigabyte GA-X99-UD3P and GA-X99-SLI.

[SteveWillis]

thanks

[boristsybin]

i`ve bought riser on ebay from http://www.ebay.com/usr/liheat48
TypeA, 150 mm. It works fine for more than week now, gpu-z detects respective gpu as connected to pci-e x4 3.0, and gpu (1080Ti) folds with fine ppd, up to 1200 kppd on 132xx job

also, GTX1050Ti tested with pci-e x1 2.0 riser. it folds with 150 kppd on 132xx and 120 kppd on 114xx (17 hours for single job). Sad, it also uses full cpu thread

[Aurum]

Boris that sure looks like it's a lot higher quality than what I had.

[boristsybin]

and price is comparible to EZDYI and TT. and shipping is not free.
But it seems that leheat48 produces those risers, and can make them any length, and straight or angled. And may be with power connector (I`m already add power connector to my riser to reduce power impact on mb)

[GPU timpster]

...thread ripper from AMD, as they have 16 core, 32 thread CPUs, and the kicker is 64 PCI-e lanes so that will prevent any GPU limitations. Unfortunately, I probably won't be able to afford such a CPU, so I'll settle for just 24 lanes of PCI-e with their current 16 core CPU. It's only $300 at the moment on certain stores, and that leaves about 200 for the GPUs, considering I'll need a motherboard, ram, and I'd like another hard drive.

Depending on how they implement the MB there could be eight x8 3.0 slots (my choice) or four x16 3.0 slots. Thanks for the tip, hadn't heard about it and will have to read up on it.
BTW, AMD CPUs have the most computing power for the buck and the Wraith Cooler works really well. I have AM3 and AM3+ MBs but know nothing about the AM4 Ryzen series. Also, the pins on an AMD CPU are far superior to the flimsy unreliable filaments used in the Intel 2011-3 sockets (look at them cross-eyed and they bend and short).
What's the highest BW AM4 MB?

I'm not interested in BTC mining of any kind, it has taxes involved and I really don't want to deal with it, I don't think I'd like the effort of keeping up with everything. I know it's currently in a state of high value, but I'm not willing to risk it dropping so quickly, it's just not my idea of investment.

I'm currently using a GTX 960, so nowhere near 1,000,000 PPD, but I'll be looking for AMD hardware that can provide such performance. If Vega doesn't deliver--seriously, AMD is under some INSANE pressure--then I'll (and everyone else looking for high performing vega) go to Nvidia.

[Aurum]

@des1957 Yes, all systems -- except one -- are Win7 x64. One system is running the linux distro CentOS 7 on an Intel Xeon Phi 7210 CPU (256 threads)...

I was about to get an Intel Xeon Phi Coprocessor 7120P and started reading the documentation and DLing the MPSS drivers for Windows. I'm running all rigs on Win7-64 but it does not list it as supported. The docs definitely do not look like plug'n play.