Guidance for building a folding dedicated computer

A forum for discussing FAH-related hardware choices and info on actual products (not speculation).

Moderator: Site Moderators

Forum rules
Please read the forum rules before posting.
ComputerGenie
Posts: 230
Joined: Mon Dec 12, 2016 4:06 am

Re: Please advise, hoping to build a dedicated folding machi

Post by ComputerGenie »

boristsybin wrote:...
4 to 7 1080Ti
...
watercooling
Now there's a good bit of wasted $. :e?:
ollie1983
Posts: 7
Joined: Sat Jul 29, 2017 8:12 pm

Re: Guidance for building a folding dedicated computer

Post by ollie1983 »

Ive got a water cooling loop in my existing PC. It is good because heat from the CPU is dumped straight out of the case and not recycled etc.

I will definitely look at the X79 Asus/SW platform. Thanks.
Nathan_P
Posts: 1164
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: Guidance for building a folding dedicated computer

Post by Nathan_P »

I would go either x99 or wait for Threadripper (TR) to launch. Not only will you get more PCIe lanes with TR, (important if you want to get the maximum performance from your cards), it should be cheaper than an intel rig. Despite what people say PCIe speed does affect F@H performance and although in some cases its not much, 2-3% longer on a GTX1070/1080 is 6k-8k PPD less. GPU's would depend on budget but I would look at 1080 as a minimum.

Personally I would look at 12C/24t TR, high end mobo, upto 3 GPU's and a 1000-1200w PSU in a large case with lots of fans along with a cheap HDD and Linux as the OS, buy decent branded parts - especially Ram, Mobo and PSU. If you want to fold on both CPU and GPU I would go for 16C/32T TR or dual processor intel xeon or AMD EYPC. I'm currently running a CPU rig on dual 12c/24t xeon's and I'm getting anything from 250-380k PPD.

Once you have al the parts the only thing that may need to be upgraded are the GPU's as newer ones are faster/more power efficient but the base components should last 5-6 years
Image
boristsybin
Posts: 50
Joined: Mon Jan 16, 2017 11:40 am
Hardware configuration: 4x1080Ti + 2x1050Ti
Location: Russia, Moscow

Re: Please advise, hoping to build a dedicated folding machi

Post by boristsybin »

ComputerGenie wrote:
boristsybin wrote:...
4 to 7 1080Ti
...
watercooling
Now there's a good bit of wasted $. :e?:
there was request for
ollie1983 wrote:a fairly brutish PC to fold with
Image
Aurum
Posts: 292
Joined: Sat Oct 03, 2015 3:15 pm
Location: The Great Basin

Re: Please advise, hoping to build a dedicated folding machi

Post by Aurum »

ollie1983 wrote:...looking to build or buy a fairly brutish PC to fold with. It might get some VR gaming...
Hi Ollie, I don't game but I think there's a dichotomy in approaches. I think gaming benefits from a fast CPU RAM combination making an i7 look great or a Ryzen (more bang for the buck) look great, but these only have 24 PCIe lanes. Folding benefits from more cores (Xeon E5 with 40 PCIe lanes) and especially more PCIe lanes, not so much on RAM, but is often PCIe throughput limited. The best info I can recall to get into the nitty gritty is the thread foldy posted a couple back on PCIe bandwidth.
AMD MBs run all PCIe lanes through their Northbridge switch and this slows it down. Most Intel MBs route at least 8 lanes thru Northbridge. The only MBs I've found that route all 40 lanes directly from the CPU to the graphics card slots are the Gigabyte GA-X99 boards. Take a look at the overview web page and especially in the manual and find the block diagram of the MB. Where this matters is when you use four 1080 Ti's you can get x8x8x16x8 or with 3 cards x16x8x16 or x16x16x8.
I'm watching to see how MB makers implement the AMD Thread Ripper with 64 3.0 PCIe lanes. Sadly they always design these MBs for gamers and not scientists so no great expectations. I'd like to have either x16x16x16x16 or x8x8x8x8x8 directly from CPU to graphics cards and fewer bells 'n whistles and blinky lights.
http://wccftech.com/amd-ryzen-threadrip ... arks-leak/
Last edited by Aurum on Sun Jul 30, 2017 1:41 pm, edited 1 time in total.
In Science We Trust Image
Aurum
Posts: 292
Joined: Sat Oct 03, 2015 3:15 pm
Location: The Great Basin

Re: Guidance for building a folding dedicated computer

Post by Aurum »

When considering 5 to 7 GPU cards keep in the mind the total at-the-wall power consumption of a rig. I'm in the US and most homes have 120 V 15 amp circuit breakers. I wired in additional 20-amp breakers to supply my folding rigs. I can put one 4-GPU rig on a 15-amp breaker or two 4-GPU rigs on a 20-amp breaker. You may have more head room in 240 Volt countries but do the math and be safe as this can be a serious fire hazard.
In Science We Trust Image
Aurum
Posts: 292
Joined: Sat Oct 03, 2015 3:15 pm
Location: The Great Basin

Re: Guidance for building a folding dedicated computer

Post by Aurum »

Nathan_P wrote:...the base components should last 5-6 years.
I predict this is not true of the M.2 SSD cards that are buried under the second graphics cards on all MBs I've seen. Why on Earth any competent engineer would allow that design layout is beyond me. That's a blackbody cavity and means the M.2 SSD will operate at 66C to 80C. Heat exponentially accelerates electromigration and electronic failure. I'll be amazed if they make it to 2 years under these severe conditions. Also if you overload your PSU you'll shorten its life as well. I only mention PSU because I can't forget some of the cryptominers I've read that bragged about how they got 1,300 Watts at the wall on a 1,200 Watt PSU.
In Science We Trust Image
boristsybin
Posts: 50
Joined: Mon Jan 16, 2017 11:40 am
Hardware configuration: 4x1080Ti + 2x1050Ti
Location: Russia, Moscow

Re: Guidance for building a folding dedicated computer

Post by boristsybin »

Aurum wrote:got 1,300 Watts at the wall on a 1,200 Watt PSU
That`s normal for 87+ gold certified PSU. 1200/0.87=1379 :)
Image
Nathan_P
Posts: 1164
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: Guidance for building a folding dedicated computer

Post by Nathan_P »

Aurum wrote:
Nathan_P wrote:...the base components should last 5-6 years.
I predict this is not true of the M.2 SSD cards that are buried under the second graphics cards on all MBs I've seen. Why on Earth any competent engineer would allow that design layout is beyond me. That's a blackbody cavity and means the M.2 SSD will operate at 66C to 80C. Heat exponentially accelerates electromigration and electronic failure. I'll be amazed if they make it to 2 years under these severe conditions. Also if you overload your PSU you'll shorten its life as well. I only mention PSU because I can't forget some of the cryptominers I've read that bragged about how they got 1,300 Watts at the wall on a 1,200 Watt PSU.
You do not need an M2 ssd card for an F@H rig, a normal HDD from WD or Seagate et al is perfectly adequate. M2 is a pointless expense. PSU wise 3x 1080Ti is 750w TDP, add 130w for your cpu & mobo and a 1000-1200w PSU is still more than enough.

A 1070 and 1080 + CPU, Mobo, HDD and fans only pulls 450w from the wall.

Also crypto miners run everything balls to the walls to make as much cash as possible, when a PSU goes pop everything is vulnerable, a 100mhz gain on the core clock speed is irrelevant when you kill the gpu and have to shell out for a replacement
Last edited by Nathan_P on Sun Jul 30, 2017 8:27 pm, edited 2 times in total.
Image
Nathan_P
Posts: 1164
Joined: Wed Apr 01, 2009 9:22 pm
Hardware configuration: Asus Z8NA D6C, 2 x5670@3.2 Ghz, , 12gb Ram, GTX 980ti, AX650 PSU, win 10 (daily use)

Asus Z87 WS, Xeon E3-1230L v3, 8gb ram, KFA GTX 1080, EVGA 750ti , AX760 PSU, Mint 18.2 OS

Not currently folding
Asus Z9PE- D8 WS, 2 E5-2665@2.3 Ghz, 16Gb 1.35v Ram, Ubuntu (Fold only)
Asus Z9PA, 2 Ivy 12 core, 16gb Ram, H folding appliance (fold only)
Location: Jersey, Channel islands

Re: Please advise, hoping to build a dedicated folding machi

Post by Nathan_P »

Aurum wrote:
ollie1983 wrote:...looking to build or buy a fairly brutish PC to fold with. It might get some VR gaming...
Hi Ollie, I don't game but I think there's a dichotomy in approaches. I think gaming benefits from a fast CPU RAM combination making an i7 look great or a Ryzen (more bang for the buck) look great, but these only have 24 PCIe lanes. Folding benefits from more cores (Xeon E5 with 40 PCIe lanes) and especially more PCIe lanes, not so much on RAM, but is often PCIe throughput limited. The best info I can recall to get into the nitty gritty is the thread foldy posted a couple back on PCIe bandwidth.
AMD MBs run all PCIe lanes through their Northbridge switch and this slows it down. Most Intel MBs route at least 8 lanes thru Northbridge. The only MBs I've found that route all 40 lanes directly from the CPU to the graphics card slots are the Gigabyte GA-X99 boards. Take a look at the overview web page and especially in the manual and find the block diagram of the MB. Where this matters is when you use four 1080 Ti's you can get x8x8x16x8 or with 3 cards x16x8x16 or x16x16x8.
I'm watching to see how MB makers implement the AMD Thread Ripper with 64 3.0 PCIe lanes. Sadly they always design these MBs for gamers and not scientists so no great expectations. I'd like to have either x16x16x16x16 or x8x8x8x8x8 directly from CPU to graphics cards and fewer bells 'n whistles and blinky lights.
http://wccftech.com/amd-ryzen-threadrip ... arks-leak/
Expect at least x16x16x8x8 for GPU's on threadripper. the bigger problem is slot spacing. I would like to see high end boards take advantage of the 8-9 slot spacing you now get in high end cases to let the cards breathe better
Image
Aurum
Posts: 292
Joined: Sat Oct 03, 2015 3:15 pm
Location: The Great Basin

Re: Guidance for building a folding dedicated computer

Post by Aurum »

Hi Nathan, I was responding to a comment about building 5 to 7 card rigs not 3 to 4.
But those M.2 SSDs are so cute and fast and get rid of a brick and two SATA cables.
I don't put my rigs in coffins so a more spacious motherboard would be great. Sadly MB designers work for gamers and couldn't care less what's good for science.
In Science We Trust Image
ollie1983
Posts: 7
Joined: Sat Jul 29, 2017 8:12 pm

Re: Guidance for building a folding dedicated computer

Post by ollie1983 »

Hello again guys

What would you do, so far I have penciled a build in for about £3000

Haswell-E X99, 6850K under a Corsair 240mm liquid cooler.

16GB of DDR4 RAM from Kingston

The big I am not sure about is the GPU department.

Would you recommend 2 x GTX 1080 or a single GTX 1080ti by itself?
Aurum
Posts: 292
Joined: Sat Oct 03, 2015 3:15 pm
Location: The Great Basin

Re: Guidance for building a folding dedicated computer

Post by Aurum »

ollie1983 wrote:Hello again guys. What would you do, so far I have penciled a build in for about £3000
Haswell-E X99, 6850K under a Corsair 240mm liquid cooler.
16GB of DDR4 RAM from Kingston
The big I am not sure about is the GPU department.
Would you recommend 2 x GTX 1080 or a single GTX 1080ti by itself?
Definitely go with the 1080 Ti, I use EVGA 11G-P4-6393 and can fit 4 of them directly on the MB. The 1080 Ti has better energy efficiency 5.2 PPD/Watt versus 4.7 for a 1080.
The price of that CPU has come way down recently.
Is that 16 GB RAM a dual-kit? I've read they're faster.
What MB are you looking at?
You've got the makings of a fast rig.
In Science We Trust Image
Aurum
Posts: 292
Joined: Sat Oct 03, 2015 3:15 pm
Location: The Great Basin

Re: Guidance for building a folding dedicated computer

Post by Aurum »

I see they're taking pre-orders for Threadrippers and X399 MBs. Way too overpriced for me. The MB only uses 48 of 64 lanes for graphics cards (x16x8x16x8), sad, so sad. TR looks like a major disappointment for folding. I'll stick with my GA-X99 plus Xeon E5 approach, $500 versus $1,400.
http://www.ebay.com/itm/Intel-Xeon-E5-2 ... 2009339940
https://www.newegg.com/Product/Product. ... 813128772R
In Science We Trust Image
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Guidance for building a folding dedicated computer

Post by bruce »

GTX 1080
GFLOPS: 8228 (8873)
TDP: 180
Price: $600

2 x GTX 1080

GFLOPS: 16456
TDP: 360
Price: $1200
GFLOPS/Watt: 46
GFLOPS/$: 13.7

GTX 1080ti
GFLOPS: 10609 (11340)
TDP: 250
Price: $700
GFLOPS/Watt: 42
GFLOPS/$: 15

I'd get the Ti and consider a second Ti when I can afford it.
Post Reply