New Folding Hardware

[SandyG]

Just got some low cost mining hardware (seems to be a good thing for FAH folks). My cluster of small Brix computers are getting old and slow (a mix of 20 or so celerons, and i5, none with gpu's) so time to start unraveling those.

Putting together a mining computer for the sole purpose of FAH.

Picked up the following (All from Amazon)

• Biostar TZ590-BTC Duo Mining Motherboard with 8 x PCIe 3.0 slots Support, 1 x PCIe 4.0 x16 Slot x8 mode ($199)

• Corsair Vengeance LPX 16GB (2x8GB) DDR4 DRAM 3200MHz

• Had an el-cheepo 256g Silicon Power SSD that has Windows 11 Pro on it.

• EVGA Supernova 1600 watt G+ Power Supplies (2x)

• 240v Power plugs (20amp 240v wiring and outlet in garage!)

• ZOTAC Gaming GeForce RTX 3060 Twin Edge OC (Testing and for a spare)

• Intel Core i5-11400 Desktop Processor 6 Cores

• BTBcoin 6 Pack PCI-E Riser 010S GPU Riser Adapter Card PCI-Express 1X to 16X Riser Card

• Antec PWM PC Fans, 120mm Fan, High Performance Case Fan, 4-pin PWM Connector, Computer Fans with 2000 RPM, Storm Series 5 Pack

• ARCTIC Case Fan Hub - 10-fold PWM Fan Distributor with SATA Power

• Funplaysmart V4 6-GPU Aluminum Miner Case, Stackable Cryptocurrency Mining Frame, Surface Miner Rack with Fan Bracket (was $36!)

Set up the computer with Win11 Pro, nothing special, Same for the Bios, basically stock. Hooked up a PCIe adapter , tossed on the video card and hooked up the rest of the wires. Came up quick, did a long round of window 11 updated, Nvidia drivers, and was Folding! I did turn down the number of CPU cores to 6 for folding, will likely go lower as I increase video cards, but will see.

Some numbers for comparison, both with the same video card (RTX3060) and same work project 18601.

i9 SH370R8 Shuttle computer with PCI-E x 16 3.0 slot
9% complete showed a daily points of 2,219,979
70% complete showed a daily points of 2,184,844
i5 FAH mining computer with PCI-E x 1 at
6% complete showed a daily points of 1,825,384
47% complete showed a daily points of 1,918,667

So looks like average about a 15% hit for using the PCIe 1x slots vs. the Shuttle's 3.0 x16 slot

SOME QUESTIONS...

• Anything Special in setting up windows 11?

• Anything Special for setting up FAH?

• Suggestions for cost effective RTX GPU's? 4x+ 3070Ti's are what I'm looking at plus possibly a RTX3090 for the full slot on the Mobo)

• Can FAH support 6 GPU's?

I'm hoping that GPU prices start to drop a bit after the new year, but might splurge on a RTX3090 to get things rolling in the main PCIe slot which should have the least impact on performance, the rest of the slots likely looking at some RTX3070ti's or OC's.
New to the mining style of computing, but seems like it could really work well as the mining game has taken a turn for the worse it seems, but good for FAH hardware prices I guess

[Mxyzptlk]

Pretty sure those pcie riser cables are going to limit the 3070 ppd (by a lot). You will need at lest 3.0 x 8 pcie slots for those.

I run an X99 MoBo with an intel 5960X w/ 40 pcie lanes to accomadate all of the cards (up to 5). (I'm actually not really running it at the moment but I was for a while. Never mined, just a healthy obsession for F@H)

Then there is the power draw of 5 3070's and a 3090. You will honestly get more ppd and use way less power just getting a single 4090. Plus be cheaper out on fornt end costs...

[SandyG]

Thanks for the comments, looked at your work and all I can say is meticulous!

Long time ham operator WN6CTY, 73's

In running the RTX3060 card seems to be about 15% cost for using the usb riser cards, not sure how much more the cost goes up with faster cards, but expect some cost. I had the 3060 as a spare for 2 of the other computers so gave it a shot as a direct comparison, but not sure how direct even with the same work done on the same project (not sure if points for a work unit are consistent even within the same project).

Having multiple home run 240v outlets wired in the 'Ham Shack' for an amplifier leaves plenty of power for FAH

For the faster and larger cards, the cost of and single source of fail on an expensive card keeps me a bit away from going that direction, but like to see some better value on the 3060/3070 cards as they seem to be really coming down in price after the holiday's. I like the idea of buy bits as I can afford vs. a big outlay, that's how I built out the other cluster of Brix computers. Finally had one possibly die after 2 years of processing Seti@home when it shut and then then FAH.

I didn't know they made a riser/extension for a full PCIe slot may look at that as makes mounting a bigger card on a motherboard without a chassis a lot easier (looks like what you are using on your rig). Was going to have to make some bracketing for it if mounting on the mobo, but in seeing how you did it, might be better to mount on the top rail as It looks like you have a similar mining chassis too.

[Joe_H]

Besides 16x risers you can also find 4x risers. A PCIe 3 4x data path is enough to get most of the performance of all but the highest end cards on folding.

[SandyG]

The riser extensions look good for the one full card slot, but only would use one for the open full slot on the motherboard. I need to get my hands on a faster video card to see how the single lane interfaces messes with performance on a 3070ti or better. I'm thinking lots of returns after the holidays will yield some amazon warehouse returns with good prices.

What's the max GPU's FAH on one windows 11 machine can run if anyone knows?

[Joe_H]

The V7 folding client maxes out at 10 GPUs, it can only enumerate from 0 to 9. People have set up the client in servers with more GPUs, but only the first 10 are recognized as being configurable. As far as I recall the limit is the same for both Linux, where these installations were, and Windows.

[Mxyzptlk]

Nice to meet you Sandy,

Regardless of the risers, look into the power draw of so many cards. It's great that you have 230vac in your shack (wish I did and soon will!), but its the overall power bill that is making me liquidate most of my 3000 series cards for fewer 4000 series. I have solar, and I am still paying it to PG&E when I crank up the rigs.

If you aren't familiar with it, check out this website for GPU PPD and power draws: https://folding.lar.systems/

73,

Mxy

[SandyG]

Yeah, have been up looking at the GPU PPD/KW site.

Looks like so far bang for buck is 3080ti or 4080, but not many cards out yet in the 4080 serices, these need to drop to under $1k before I would get one. Also looking at cost for the card as cost of operations so if the card is 300 bucks more, that's a year of electricity. Given cost is the factor, and less so time to process, I'm OK giving up some time efficiencies if cost for 1m points is still better.

That all being said, I'll starting to watch 4080 prices and at least get one of those online to see how it would work with the full PCIe slot vs. the Riser slot in my set up since I have it all going, not likely going to swap the motherboard so may have to live with the cost of the 1x riser for a while.

Thanks for your input!

Sandy

[SandyG]

The V7 folding client maxes out at 10 GPUs, it can only enumerate from 0 to 9. People have set up the client in servers with more GPUs, but only the first 10 are recognized as being configurable. As far as I recall the limit is the same for both Linux, where these installations were, and Windows.

That should be more then enough, was looking for at least 7 if the penalty for the x1 slots are not too bad (so far seems like 15%).

Sandy

[FaaR]

What exactly is FaH using all this PCIe bandwidth for? CPU usage when GPU folding is absolutely negligible, at least on AMD platform - one or two percent of one CPU thread is what I'm typically seeing.

[Joe_H]

The F@h GPU folding core is continuously preparing blocks of data using the CPU to transmit to the GPU for processing and retrieving the processed blocks of data. Those blocks of data are dependent on the results of previously processed blocks. That is what uses all that PCIe bandwidth. This is different for example to mining which mostly passes some numeric data and processing instructions that continue for a long time before needing new instructions and data from the CPU.

Somewhat oversimplified, the F@h folding core is using the GPU as a large vector processor. Mining can use the available GPU commands to load an algorithm to continue processing from some seed values for some time before CPU intervention of data passing is required over the PCIe bus.

[SandyG]

Ended up picking up a pair of used EVGA GeForce RTX 3090 FTW3 Ultra Cards.One is in the full PCI slot and the other is in the x1 with the extender card. Catching some numbers, hopefully not too bad but I'll be expecting the faster the card the more penalty for the performance.

As the glut of used cards and mining rigs being sold off cheap would be awesome if a driver could optimize better for the slower bus speed when using a mining mother board with a bunch of x1 slots.

Will see how they do and if I can catch them with the same Project ID being worked on. Hopefully points from the same card running the same projecct work unit is a fair comparison.

Sandy

[thiblec71]

Could you share the difference of PPD between the card on the full PCI slot and on the X1?

[SandyG]

Could you share the difference of PPD between the card on the full PCI slot and on the X1?

I don't have specifics, but it was always around 15 to 20% on like work on the 3090's, it seems to have less effect on the 3060 card (makes sense). I have since removed the mining motherboard from use and replaced it with an older MSI-MW51H0 server motherboard that had a lot of PCIe slots. I'm not sure if you can run over 4 cards due to power draw from the mobo, not sure if that makes sense, but I tried adding a 3060 to the mix of 4 other cards and the board would not boot. The card is good and in use on this computer I'm typing on so not a card issue. Not sure if draw from the mobo with the reset of the cards was too much, but I always assumed most of the power was from the 8 pin connectors on the top of the cards.

If you have a 1x mining board, it will work, but will waste some performance for sure, if they are older cards fire it up! If newer cards as other have also said, wastes some performance.

Currently running 2x 3090's and 2x 4090's all with PCI extension cables to the cards, seems to work well but costly at about 40-50 for each cable. Also switched up the O/S to Linux Mint (Ubuntu variant) as some folks mentioned it's it a bit better for performance due to the way it works vs. Windows. I have not benchmarked that, but seems logical.

A bit off topic but here are power and heat use from the 3090 vs 4090 from my linux box monitored with 'nvidia-smi1' this is all in a very warm room with NO A/C in the Los Angeles area.

Code: Select all

+
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:17:00.0 Off |                  N/A |
| 82%   82C    P2   387W / 420W |    416MiB / 24576MiB |     93%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:65:00.0 Off |                  N/A |
| 89%   84C    P2   369W / 420W |    461MiB / 24576MiB |     97%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:B6:00.0 Off |                  Off |
| 39%   65C    P2   277W / 450W |    574MiB / 24564MiB |     91%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:B8:00.0 Off |                  Off |
| 39%   65C    P2   271W / 450W |    574MiB / 24564MiB |     95%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Sandy

[nchowlett]

Hi Sandy! Just found this thread, seems we're trying same thing.

I tried adding a 3060 to the mix of 4 other cards and the board would not boot

My system with 4 GPUs sometimes reboots while booting, were you getting this behaviour? If so I just kept booting and eventually Linux Ubuntu started for me.