Looking for a multi-gpu setup guide

[dirtybit]

I have a number of bitcoin mining rigs that are no longer profitable and I want to re-purpose them as folding machines.

The rigs are made up of the following hardware:
AMD Sempron 145 CPU
1 GB RAM
6x AMD/ATI 5850s
40GB Hard drive

Will I need to upgrade/add anything? Like more RAM? I don't mind buying some sticks of RAM to make this work.

I do not have Windows licenses for them and I read that there's some trouble with ATI drivers. Does anyone have a step-by-step on how to setup a folding rig with this hardware on Linux? My rigs run on Ubuntu 12.04.2, but of course I'll do fresh installs if I need to.

Some of the GPUs are connected via 1x-to-16x PCIe riser cables. Will that bandwidth throttling hurt my folding processing time?

Any tips or advice from anyone who has done this will be greatly appreciated.

[P5-133XL]

I don't know that there is such a guide. That being said, perhaps I can give you some useful helps.

1. For AMD, currently there is no Linux GPU folding for the drivers need work. Currently only Windows works.
2. Folding is not particularly PCI-e bandwidth limited. The only testing that occurred was very early in the PCI-e offerings and the result was that running at PCI-e v1.0 at 4x caused a small productivity drop of 5-10% as opposed to x16. Perhaps you can extrapolate the net effect of your 1x slots or you can test and report newer numbers of what you are getting as opposed to on identical HW folding on a 16x slot.
3. The 5850's are relatively low end cards compared to what currently exists and is therefore not going to be a high point producer.
4. On Windows the last good AMD driver was v12.8 for the older Core_16. It needed 1 CPU core per video card which is a problem when your CPU only has one CPU core. With that in mind, you will need to run Core_17 to run more than one video card.
5. Core_17 WU's do not need a full CPU core to run AMD video cards (so you can run more than one video card per machine) and can make use of recent video drivers. To request Core_17 you currently need to set client-type to be advanced.
6. GPU Folding is not RAM nor HD intensive but 1 GB is not very much. With 6x video cards folding I'd be more comfortable with 2GB. Really upgrading the video cards is the most valuable upgrade and if you do, the power supply is likely to need to be upgraded too. It is probably better on a PPD/Watt basis to run one big video card than lots of low end cards and long-term the cost of electricity dwarfs the up front cost of the video card.
7. While Core_17 does not use lots of CPU per video card, it does use some. I really do not have a good feeling as to using a single CPU core for 6 video cards but I can't say that you are going to run into problems either. I'd just be more comfortable with another CPU core.

I suggest trying it on a machine and seeing what happens. With hard data you will likely be able to make better judgments as to possible changes in your setup.

[dirtybit]

I don't know that there is such a guide. That being said, perhaps I can give you some useful helps.

1. For AMD, currently there is no Linux GPU folding for the drivers need work. Currently only Windows works.
2. Folding is not particularly PCI-e bandwidth limited. The only testing that occurred was very early in the PCI-e offerings and the result was that running at PCI-e v1.0 at 4x caused a small productivity drop of 5-10% as opposed to x16. Perhaps you can extrapolate the net effect of your 1x slots or you can test and report newer numbers of what you are getting as opposed to on identical HW folding on a 16x slot.
3. The 5850's are relatively low end cards compared to what currently exists and is therefore not going to be a high point producer.
4. On Windows the last good AMD driver was v12.8 for the older Core_16. It needed 1 CPU core per video card which is a problem when your CPU only has one CPU core. With that in mind, you will need to run Core_17 to run more than one video card.
5. Core_17 WU's do not need a full CPU core to run AMD video cards (so you can run more than one video card per machine) and can make use of recent video drivers. To request Core_17 you currently need to set client-type to be advanced.
6. GPU Folding is not RAM nor HD intensive but 1 GB is not very much. With 6x video cards folding I'd be more comfortable with 2GB. Really upgrading the video cards is the most valuable upgrade and if you do, the power supply is likely to need to be upgraded too. It is probably better on a PPD/Watt basis to run one big video card than lots of low end cards and long-term the cost of electricity dwarfs the up front cost of the video card.
7. While Core_17 does not use lots of CPU per video card, it does use some. I really do not have a good feeling as to using a single CPU core for 6 video cards but I can't say that you are going to run into problems either. I'd just be more comfortable with another CPU core.

I suggest trying it on a machine and seeing what happens. With hard data you will likely be able to make better judgments as to possible changes in your setup.

The first point is a total show stopper for me. I don't own any windows licenses, unless someone wants to make a donation of two serial numbers to me.

I'm also not interested at all in whatever points you're talking about, unless I can cash them in for beer somehow. Anyway, I'll check back in when Linux multi-gpu is working.

[P5-133XL]

Linux multi-GPU works fine for Nvidia, just not AMD/ATI. The problem with Nvidia is that with Core_17 they take one CPU core per video card.

[bruce]

Predictions seem to suggest that there will be Linux drivers for ATI GPUs soon, but nobody has suggested what kind of a time-frame is being called "soon" so make whatever you want to out of the prediction. I certainly would check back "soon" and see if anything has happened.

The Sempron 145 is a single-core computer which is capable of running FAH (at a rather slow rate) unless it's devoted to supporting a GPU. That would allow you to get started with the installation of FAH V7 and to familiarize yourself with FAH.

Linux multi-GPU works fine for Nvidia, just not AMD/ATI. The problem with Nvidia is that with Core_17 they take one CPU core per video card.

While this is an expected behavior, it's not a showstopper. Nobody has reported what happens if you have two GPUs and one CPU but I can guess they'd compete for the CPU. My guess is that both GPUs would run somewhat slower than if each had a dedicated CPU, but what somebody could determine is whether it's approaching 100% or 50% or some other number. We do know that it's possible for a GPU to fold very nicely with almost no CPU load if the drivers are written in a way that supports that sort of activity.

[ChristianVirtual]

I'm running a mixed setup with one GTX 780 and one GTX 660 Ti driven now by an old Quad-C2 on ubuntu 13.04. The only issue is a bug in nvidia driver which freeze the system every 36 hours; NV found a solution and will provide update soon.
Else very happy with that setup. Get around 200kPPD with no issues while folding.
The setup was as easy as plug in, install driver from xorg-ppa and let FAH figure out the GPU slots.

[PantherX]

...

Linux multi-GPU works fine for Nvidia, just not AMD/ATI. The problem with Nvidia is that with Core_17 they take one CPU core per video card.

While this is an expected behavior, it's not a showstopper. Nobody has reported what happens if you have two GPUs and one CPU but I can guess they'd compete for the CPU. My guess is that both GPUs would run somewhat slower than if each had a dedicated CPU, but what somebody could determine is whether it's approaching 100% or 50% or some other number. We do know that it's possible for a GPU to fold very nicely with almost no CPU load if the drivers are written in a way that supports that sort of activity.

We do have a report that running 4 GTX 780 on 2 physical cores will not produce any loss in performance in Ubuntu 13.10 (viewtopic.php?p=246950#p246950). However, this may not apply for Windows or may change with driver versions.

[bruce]

We do have a report that running 4 GTX 780 on 2 physical cores will not produce any loss in performance in Ubuntu 13.10 (viewtopic.php?p=246950#p246950). However, this may not apply for Windows or may change with driver versions.

If this is a general result, it probably makes sense to work out a plan to map all copies of FahCore_17 to affinity for a single core and run a CPU client on the remaining CPUs. We're talking about one WU per GPU so there's no worry about synchronizing threads like with SMP.

[Napoleon]

We do have a report that running 4 GTX 780 on 2 physical cores will not produce any loss in performance in Ubuntu 13.10 (viewtopic.php?p=246950#p246950). However, this may not apply for Windows or may change with driver versions.

Based on my personal experience with Win7 64bit and affinity tweaking with 3rd party tools, FahCore_17/NVidia driver/OS scheduler seems to work in some sort of "co-operative spin-wait" mode if you force the issue, so I'm not completely taken by surprise with affinity-mapping two GPUs per core not causing a significant performance loss. One might be able to do even more.

[PantherX]

AFAIK, the sweet spot is 2 GPUs per Physical Core. If you do more that that, performance will take a hit. I am curios if you can run 2 GPUs on a Virtual Core without any performance loss. I am hoping to experiment with it soon.

[Napoleon]

Well, if my Atom330, ION iGPU & GT430 count for anything... you can, with plenty of room to spare, CPU performance wise. I'd expect, say, i3 2C/4T and two high end GPUs to work just fine if affinity-locked (with some 3rd party tool) to a single physical core.

PX, what do you actually mean by "Virtual Core", by the way? With hyperthreaded CPUs, all the cores (seen/shown by OS) are virtual (aka logical CPUs). They just happen to be mapped to physical cores. On Windows side, you can check the mapping with e.g Coreinfo.

[PantherX]

The information that I learnt about Hyperthreading was from an Intel video and presentation quite some time ago. Since I haven't heard any difference in implementation, I still assume that it's correct.

My understanding is that each physical core supports two threads. With that, a dual core would be like this:
Core 0 -> Thread 0, Thread 1
Core 1 -> Thread 2, Thread 3
Now, by Virtual Core, I refer to the threads. When it comes to mapping, I would use 1 Thread from each core since I don't want to use 1 Core (Thread 0 and Thread 1) to feed the GPU. If I use Thread 1 and Thread 3 to feed my GPUs, the SMP can still run on 2 Cores by using Thread 0 and Thread 2.

[bruce]

A CPU is not a monolithic entity. There's a pipeline which (simplified) loads instructions, decodes operations, loads data, performs instructions, stores results. Sometimes one step has to wait longer than usual for the preceding step so performance is less than ideal even when the CPU is busy 100% of the time. Actually, there are multiple parallel pipelines so if one execution unit can't get an instruction to perform, it may get one from another pipeline to stay busy. The potential ways that a circuit can stay busy vary depending on the hardware design, and HTT is only one of the tools that a CPU designer can use to facilitate higher throughput.

Looking ONLY at the ALU circuitry, multiple operations are performed concurrently with or without HTT. HTT just gives the preceding circuitry more options to supply work than the circuit can do. In other words, 100% doesn't mean 100% with or without HTT. The FPU circuitry can perform fewer instructions per clock and many programs never use the i387 instructions, so having fewer FPU units saves costs. The instructions in Cores 11/15/16 (and a lot of 17) are (almost) exclusively for the ALU and it can handle a lot more parallelism than the FPU so running a GPU core on a logical processor makes a lot of sense because it's pair is completely free to run a CPU thread without contending for the FPU.

[Napoleon]

PX, Bruce. I agree. That is why I like to do my affinity/priority management with WinAFC. I've tried various configs with my 2C/4T Atom330 & two GPUs, but looks like this is what I am returning to, time after time,

Code: Select all

# TestMode=1 should only provide console output of
# what WinAFC *would* do with TestMode=0
TestMode=0

##################################################
##          Application Profile lines
##################################################
#
# An application profile is specified on a single line. 
# An application profile includes the following information: an application
# name, a CPU mask, and optional attributes in the following format:
# C:\Path\To\Application := CPU0+CPU1 [attr1=val1,attr2=val2]
#
# Check the documentation and the provided examples for more 
# information about these fields.

#
# Napoleon:
# For Hyperthreaded 2C/4T Intel Atom330 + ION iGPU + NVidia GT430
# 2x CPU:1 for FahCore_78.exe
# 1x distributed.net OGR-27 cruncher (non-FPU workload)
#

# Distributed.net OGR-27 integer work (low priority by default)
C:\Users\*\dnetc.exe := CPU3+CPU0

# Einstein@Home on ION iGPU (retired from FAH)
C:\BOINC\BoincData\*\einsteinbinary*.exe := CPU0 [priority=High]

# Folding@Home on Atom330 & GT430 (FahCore_15.exe & FahCore_17.exe)
C:\FAH\Data\cores\*\FahCore_1?.exe := CPU3 [priority=High]
C:\FAH\Data\cores\*\FahCore_78.exe := CPU2+CPU1 [priority=BelowNormal,assign=1,policy=PSEUDOBALANCED,resource=MEMUSE]

# Miscellaneous apps with priority and affinity adjustment
C:\Users\*\procexp*.exe := CPU3+CPU0 [priority=AboveNormal]
C:\Users\*\WinAFC64.exe := CPU3+CPU0 [priority=AboveNormal]
C:\Windows\*\taskmgr.exe := CPU3+CPU0 [priority=AboveNormal]

# Daily grind
C:\BOINC\boinc*.exe := CPU3+CPU0
C:\FAH\Client\FAH*.exe := CPU3+CPU0
C:\Program Files\Opera*\opera*.exe := CPU3+CPU0
C:\Program Files (x86)\HP\*\HPTLBXFX.exe := CPU3+CPU0
C:\Program Files (x86)\Mozilla*\thunderbird.exe := CPU3+CPU0
C:\Users\*\HFM.exe := CPU3+CPU0

based on Coreinfo:

Code: Select all

Logical to Physical Processor Map:
**--  Physical Processor 0 (Hyperthreaded)
--**  Physical Processor 1 (Hyperthreaded)

Logical Processor to Socket Map:
****  Socket 0

Logical Processor to NUMA Node Map:
****  NUMA Node 0

Logical Processor to Cache Map:
**--  Data Cache          0, Level 1,   24 KB, Assoc   6, LineSize  64
**--  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**--  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
--**  Data Cache          1, Level 1,   24 KB, Assoc   6, LineSize  64
--**  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--**  Unified Cache       1, Level 2,  512 KB, Assoc   8, LineSize  64

Logical Processor to Group Map:
****  Group 0

[dirtybit]

Thanks for the info, but if all of this is true, I'm not going to bother with folding, my rigs need so much upgrading that I may as well buy new rigs. I was hoping to make a transition with minimal upgrades.