Guidance for building a folding dedicated computer

[ComputerGenie]

...
4 to 7 1080Ti
...
watercooling

Now there's a good bit of wasted $.

[ollie1983]

Ive got a water cooling loop in my existing PC. It is good because heat from the CPU is dumped straight out of the case and not recycled etc.

I will definitely look at the X79 Asus/SW platform. Thanks.

[Nathan_P]

I would go either x99 or wait for Threadripper (TR) to launch. Not only will you get more PCIe lanes with TR, (important if you want to get the maximum performance from your cards), it should be cheaper than an intel rig. Despite what people say PCIe speed does affect F@H performance and although in some cases its not much, 2-3% longer on a GTX1070/1080 is 6k-8k PPD less. GPU's would depend on budget but I would look at 1080 as a minimum.

Personally I would look at 12C/24t TR, high end mobo, upto 3 GPU's and a 1000-1200w PSU in a large case with lots of fans along with a cheap HDD and Linux as the OS, buy decent branded parts - especially Ram, Mobo and PSU. If you want to fold on both CPU and GPU I would go for 16C/32T TR or dual processor intel xeon or AMD EYPC. I'm currently running a CPU rig on dual 12c/24t xeon's and I'm getting anything from 250-380k PPD.

Once you have al the parts the only thing that may need to be upgraded are the GPU's as newer ones are faster/more power efficient but the base components should last 5-6 years

[boristsybin]

...
4 to 7 1080Ti
...
watercooling

Now there's a good bit of wasted $.

there was request for

a fairly brutish PC to fold with

[Aurum]

...looking to build or buy a fairly brutish PC to fold with. It might get some VR gaming...

Hi Ollie, I don't game but I think there's a dichotomy in approaches. I think gaming benefits from a fast CPU RAM combination making an i7 look great or a Ryzen (more bang for the buck) look great, but these only have 24 PCIe lanes. Folding benefits from more cores (Xeon E5 with 40 PCIe lanes) and especially more PCIe lanes, not so much on RAM, but is often PCIe throughput limited. The best info I can recall to get into the nitty gritty is the thread foldy posted a couple back on PCIe bandwidth.
AMD MBs run all PCIe lanes through their Northbridge switch and this slows it down. Most Intel MBs route at least 8 lanes thru Northbridge. The only MBs I've found that route all 40 lanes directly from the CPU to the graphics card slots are the Gigabyte GA-X99 boards. Take a look at the overview web page and especially in the manual and find the block diagram of the MB. Where this matters is when you use four 1080 Ti's you can get x8x8x16x8 or with 3 cards x16x8x16 or x16x16x8.
I'm watching to see how MB makers implement the AMD Thread Ripper with 64 3.0 PCIe lanes. Sadly they always design these MBs for gamers and not scientists so no great expectations. I'd like to have either x16x16x16x16 or x8x8x8x8x8 directly from CPU to graphics cards and fewer bells 'n whistles and blinky lights.
http://wccftech.com/amd-ryzen-threadrip ... arks-leak/

[Aurum]

When considering 5 to 7 GPU cards keep in the mind the total at-the-wall power consumption of a rig. I'm in the US and most homes have 120 V 15 amp circuit breakers. I wired in additional 20-amp breakers to supply my folding rigs. I can put one 4-GPU rig on a 15-amp breaker or two 4-GPU rigs on a 20-amp breaker. You may have more head room in 240 Volt countries but do the math and be safe as this can be a serious fire hazard.

[Aurum]

...the base components should last 5-6 years.

I predict this is not true of the M.2 SSD cards that are buried under the second graphics cards on all MBs I've seen. Why on Earth any competent engineer would allow that design layout is beyond me. That's a blackbody cavity and means the M.2 SSD will operate at 66C to 80C. Heat exponentially accelerates electromigration and electronic failure. I'll be amazed if they make it to 2 years under these severe conditions. Also if you overload your PSU you'll shorten its life as well. I only mention PSU because I can't forget some of the cryptominers I've read that bragged about how they got 1,300 Watts at the wall on a 1,200 Watt PSU.

[boristsybin]

got 1,300 Watts at the wall on a 1,200 Watt PSU

That`s normal for 87+ gold certified PSU. 1200/0.87=1379

[Nathan_P]

...the base components should last 5-6 years.

I predict this is not true of the M.2 SSD cards that are buried under the second graphics cards on all MBs I've seen. Why on Earth any competent engineer would allow that design layout is beyond me. That's a blackbody cavity and means the M.2 SSD will operate at 66C to 80C. Heat exponentially accelerates electromigration and electronic failure. I'll be amazed if they make it to 2 years under these severe conditions. Also if you overload your PSU you'll shorten its life as well. I only mention PSU because I can't forget some of the cryptominers I've read that bragged about how they got 1,300 Watts at the wall on a 1,200 Watt PSU.

You do not need an M2 ssd card for an F@H rig, a normal HDD from WD or Seagate et al is perfectly adequate. M2 is a pointless expense. PSU wise 3x 1080Ti is 750w TDP, add 130w for your cpu & mobo and a 1000-1200w PSU is still more than enough.

A 1070 and 1080 + CPU, Mobo, HDD and fans only pulls 450w from the wall.

Also crypto miners run everything balls to the walls to make as much cash as possible, when a PSU goes pop everything is vulnerable, a 100mhz gain on the core clock speed is irrelevant when you kill the gpu and have to shell out for a replacement

[Nathan_P]

...looking to build or buy a fairly brutish PC to fold with. It might get some VR gaming...

Hi Ollie, I don't game but I think there's a dichotomy in approaches. I think gaming benefits from a fast CPU RAM combination making an i7 look great or a Ryzen (more bang for the buck) look great, but these only have 24 PCIe lanes. Folding benefits from more cores (Xeon E5 with 40 PCIe lanes) and especially more PCIe lanes, not so much on RAM, but is often PCIe throughput limited. The best info I can recall to get into the nitty gritty is the thread foldy posted a couple back on PCIe bandwidth.
AMD MBs run all PCIe lanes through their Northbridge switch and this slows it down. Most Intel MBs route at least 8 lanes thru Northbridge. The only MBs I've found that route all 40 lanes directly from the CPU to the graphics card slots are the Gigabyte GA-X99 boards. Take a look at the overview web page and especially in the manual and find the block diagram of the MB. Where this matters is when you use four 1080 Ti's you can get x8x8x16x8 or with 3 cards x16x8x16 or x16x16x8.
I'm watching to see how MB makers implement the AMD Thread Ripper with 64 3.0 PCIe lanes. Sadly they always design these MBs for gamers and not scientists so no great expectations. I'd like to have either x16x16x16x16 or x8x8x8x8x8 directly from CPU to graphics cards and fewer bells 'n whistles and blinky lights.
http://wccftech.com/amd-ryzen-threadrip ... arks-leak/

Expect at least x16x16x8x8 for GPU's on threadripper. the bigger problem is slot spacing. I would like to see high end boards take advantage of the 8-9 slot spacing you now get in high end cases to let the cards breathe better

[Aurum]

Hi Nathan, I was responding to a comment about building 5 to 7 card rigs not 3 to 4.
But those M.2 SSDs are so cute and fast and get rid of a brick and two SATA cables.
I don't put my rigs in coffins so a more spacious motherboard would be great. Sadly MB designers work for gamers and couldn't care less what's good for science.

[ollie1983]

Hello again guys

What would you do, so far I have penciled a build in for about £3000

Haswell-E X99, 6850K under a Corsair 240mm liquid cooler.

16GB of DDR4 RAM from Kingston

The big I am not sure about is the GPU department.

Would you recommend 2 x GTX 1080 or a single GTX 1080ti by itself?

[Aurum]

Hello again guys. What would you do, so far I have penciled a build in for about £3000
Haswell-E X99, 6850K under a Corsair 240mm liquid cooler.
16GB of DDR4 RAM from Kingston
The big I am not sure about is the GPU department.
Would you recommend 2 x GTX 1080 or a single GTX 1080ti by itself?

Definitely go with the 1080 Ti, I use EVGA 11G-P4-6393 and can fit 4 of them directly on the MB. The 1080 Ti has better energy efficiency 5.2 PPD/Watt versus 4.7 for a 1080.
The price of that CPU has come way down recently.
Is that 16 GB RAM a dual-kit? I've read they're faster.
What MB are you looking at?
You've got the makings of a fast rig.

[Aurum]

I see they're taking pre-orders for Threadrippers and X399 MBs. Way too overpriced for me. The MB only uses 48 of 64 lanes for graphics cards (x16x8x16x8), sad, so sad. TR looks like a major disappointment for folding. I'll stick with my GA-X99 plus Xeon E5 approach, $500 versus $1,400.
http://www.ebay.com/itm/Intel-Xeon-E5-2 ... 2009339940
https://www.newegg.com/Product/Product. ... 813128772R

[bruce]

GTX 1080
GFLOPS: 8228 (8873)
TDP: 180
Price: $600

2 x GTX 1080
GFLOPS: 16456
TDP: 360
Price: $1200
GFLOPS/Watt: 46
GFLOPS/$: 13.7

GTX 1080ti
GFLOPS: 10609 (11340)
TDP: 250
Price: $700
GFLOPS/Watt: 42
GFLOPS/$: 15

I'd get the Ti and consider a second Ti when I can afford it.