Page 1 of 1
Dell R720 and T620 with GPU
Posted: Thu Mar 19, 2020 6:52 am
by JacquesVoogt
Hi Folders
I have some old servers lying about.
R720 and T620
They have decent RAM. 64GB or more. And dual socket Xeon E5-26xx CPUs
Would they be useful?
What OS would you recommend? I can do Windows, Ubuntu or CentOs
Anyone added GPU's to such machines? The T620 has lots of space and the manual says it supports up to 4 GPU, so it looks promising
Nvidia C2075, K4000, Q6000, K20A
ATI FirePro V7800
AMD FirePro W7000
I could add a few GPU but I'd have to buy them.
Has anyone tried other cards like GTX 1060, 1080, 2060 on a Dell T620 and have some confidence getting it to work?
Re: Dell R720 and T620 with GPU
Posted: Thu Mar 19, 2020 9:42 am
by foldy
I did not try Dell T620 but it should be able to run any GPU so you don't need expensive Quadro or FirePro GPUs but can use the cheaper GTX GPUs.
The only problem could be cooling if you mount 4 GPUs because they must be dual slot and then 2 GPUs would be directly next to each other.
And the power supply must provide enough watts at 12V to feed your GPUs.
Maybe just get 2 fast GPUs which can be put at distance and try to put them next to each other to test temp and performance in comparison.
If that works good then you can get 2 more GPUs.
Linux is faster than Windows for FAH, so I would go with Ubuntu which most users have. But CentOS also seems supported by FAH but did not try that myself.
This may give a short comparison of GPUs you could use. Currently Nvidia GPUs are better for FAH than AMD GPUs
https://docs.google.com/spreadsheets/d/ ... Ek/pubhtml
I would recommend RTX 2060 because it gives best performance/price/watts and you can easily resell to gamers if not needed anymore
Re: Dell R720 and T620 with GPU
Posted: Thu Mar 19, 2020 12:25 pm
by JacquesVoogt
Thanks foldy.
I did some more searching and opening. Adding this here for other folders looking to use a T620 or R720.
T620 supports 4 double width GPU cards. 2 on each CPU. But then there is no space for the Perc Raid controller or Network interface card.
One would have to use single width cards to retain additional functionality.
In addition the T620 should be bought with the additional GPU enablement kit which adds a power distribution board and cabling.
Dell says it can not be added later, but it just means you would have to strip out everything to get to it's location behind the motherboard.
My T620 does not have this factory option installed, therefore it is a full strip down and additional hardware to add in 2 GPU cards and still retain some other functionality.
The R720 supports 2 double width GPU cards, but again requires a GPU enablement kit. This kit contains power cables, low profile heat zinks and low profile RAM shrouds to make things fit. You may also have to get an additional PCI express 16 riser, depending on your current config. This kit is probably less work to install than the T620 kit but only allow 2 cards and will likely cost more given the additional components.
Dell supports specific GPU cards. That does not mean that other cards will not work, it just means they have not tested them and can not say they will or will not work. Some people have reported their IDRAC sometimes reporting GPU card failure with unsupported cards.
So it is doable, but I think it might be easier, cheaper and quieter to achieve on a regular PC as a base.
Now the question is, what is most needed? GPU or CPU power?
If there is value in a dual CPU set up with total 12 cores or 24 virtual processors, then I'd rather persist don the server path and use it as a file server too. If the need is mostly GPU, then I'll sell the server and buy a regular PC instead. I think the latter may be more energy efficient in the long run too.
Re: Dell R720 and T620 with GPU
Posted: Thu Mar 19, 2020 2:19 pm
by Jesse_V
GPUs are more valuable to Folding@home because they are orders of magnitude faster than CPUs, although they can do a more limited range of calculations. Every GPU you can give goes a long ways. It's up to you, because a 24-core CPU is very helpful too.
Re: Dell R720 and T620 with GPU
Posted: Thu Mar 19, 2020 5:58 pm
by Joe_H
But CPUs are also important. They are putting out a number of projects connected to COVID-19 as CPU based.
Re: Dell R720 and T620 with GPU
Posted: Fri Mar 20, 2020 6:21 am
by JacquesVoogt
Don't know how to post images, but the T620 is now folding on all 24 cores at 99%
Does not seem to be using much in the way of memory.
No GPUs yet, but I might be able to add another 3 R720 servers with similar CPU specs over the weekend.
Re: Dell R720 and T620 with GPU
Posted: Fri Mar 20, 2020 6:52 am
by Joe_H
Memory usage will depend on the size of the protein system being done. Most will run in 2 GB of RAM, but there are one that need the expanded address space from running in 64-bit mode. They do try to use memory efficient algorithms, otherwise the RAM usage could be larger.
Re: Dell R720 and T620 with GPU
Posted: Fri Mar 20, 2020 12:10 pm
by JacquesVoogt
Thanks Joe. 64GB would be a bit excessive then.
It got me thinking...would it be better to have multiple VMs running, eg.
12 instances, each with 2 cores and 4GB RAM or
6 instances, each with 4 cores and 8GB RAM or
3 instances, each with 8 cores and 16GB RAM or
2 instances, each with 12 cores and 32GB RAM or
1 instance, with all 24 cores and all 64GB RAM
I guess the question is, are there routines that would benefit from multi-core capabilities or is the work split up into multiple single threads.
On the VM host it will typically wait for all the allocated cores to be available before letting the guest clock pulse. Any interrupt on any core would put all cores on hold until all the cores are available again. This can degrade total performance. This would become more apparent on shared hosts where other processes would compete for CPU time, specifically where the CPU may become oversubscribed. Another process would only need to hold down one core to affect performance of all cores.
With smaller instances of say 2 or 4 cores, it would only need any 2 or 4 cores to be available to continue clock stepping. Therefore less waiting.
Just wondering if anyone has done any tests to see if there is a sweet spot or if there are routines that would benefit from larger allocations.
Re: Dell R720 and T620 with GPU
Posted: Fri Mar 20, 2020 12:15 pm
by scerbera
I'm running my 3990x single threaded, split into 32 cores and 28 cores, with 4 left for 2 gpu slots and general. When I get a cpu task all cores are used so i'd suggest running with 24.
Re: Dell R720 and T620 with GPU
Posted: Fri Mar 20, 2020 12:47 pm
by foldy
FAH CPU work units are optimized for multithreading so it is better to complete one work unit with 24 cores than to split it up. There are some conditions though e.g. when you have more than 32 threads then if you get issues it makes sense to split e.g. a 64 threads CPU into 2 FAH CPU slots with 32 threads each.
For a GPU server that may be interesting to you:
viewtopic.php?f=38&t=32233
viewtopic.php?f=38&t=31991
Re: Dell R720 and T620 with GPU
Posted: Sun Mar 22, 2020 12:09 am
by JacquesVoogt
Interesting builds indeed.
Each of the Xeon CPU's have 40 PCI-e lanes for a total of 80 and the motherboard has 7 slots. 4 slots are x16 and 3 are x8
Single slot GPUs are expensive or old and inefficient by today's standard. In addition Dell recons it supports 4 of those and then recons there should be some space around the RAID card for cooling of the backup battery, so that is a no for me.
When it comes to 2 slot cards, it can do 4 but without any RAID card. So I'd have to boot off the NIC or SD card or combination of those.
Since my original plans for this machine was to be a file server, I'd probably keep the RAID controller attached to the 12 disk bays and add 2 GPU for a start.
It certainly seems capable of adding another 4 cards externally later through some extension cables. (Thanks for that info foldy) But I'd probably fill the other 3x R720s I have with 2 cards each before I go that route.
That is still 8 GPU total before I get to that point.
Almost done with adding the other 3x R720s as CPU only clients.