RTX Super cards.

[foldy]

Another option might be to be able to run multiple WUs on a single GPU but that could impede finishing generations within projects and likely would be way too much work to code and/or may not be possible with the libraries in use.

User can configure 2 GPU slots running on same GPU. Maybe that helps with super fast GPUs and small WU.
I tried it on my gtx 1080ti which is saturated with only one GPU slot already. The 2nd GPU slot slowed things down a little to 1000k PPD and it needs 2 CPU threads to feed it. Maybe someone with RTX 2080 ti wants to try that? You need to set the gpu slot index values manually to the same.

[foldy]

https://apps.foldingathome.org/psummary
In psummary for OPENMM_21/22 we have projects with atom counts 5k to 287k

Category 1 = 0 to 35k atoms
Category 2 = 35k to 75k atoms
Category 3 = 75k to 150k atoms
Category 4 = 150k to 300k atoms
...
I choose each category to have an atoms interval of 2x.

https://apps.foldingathome.org/GPUs.txt
Should fit GPUs ranging from x PPD to 2x PPD.

Category 1 = 0k to 300k PPD
Category 2 = 300k to 600k PPD
Category 3 = 600k to 1200k PPD
Category 4 = 1200k to 2400k PPD
...
I choose each category to have an PPD interval of 2x.

What does the number at the end mean in GPUs.txt?
0x10de:0x1e84:2:8:TU104 [GeForce RTX 2070 Super] 8218
0x10de:0x1e87:2:8:TU104 [GeForce RTX 2080 Rev. A] 10068
0x10de:0x1e04:2:8:TU102 [GeForce RTX 2080 Ti] M 13448

Maybe we don't need categories but get all into a continous formula.

IF SHADER_COUNT>K1 AND ATOMS < K1
Do we have more data variables at assignment server than SHADER_COUNT? e.g. clock Ghz or even PPD? Or the ids from GPUs.txt?

[bruce]

Big WUs were not run on GPU, (super or not) but rather on multi-CPU systems. That's off-topic for this discussion.

Then, too, science studies specific proteins, not a specific number of atoms. The size of a study is a dependent variable.

[MeeLee]

CUDA could allocate GPU resources from a low atom count card to a higher atom card.
However that doesn't solve if both cards are running low counts.

Smaller atom counts should be doubled.
Most GPUs have 2-8GB of vram, and each WU takes less than 500MB.

[MeeLee]

..From here on DGPUs will only get bigger; and unless IGPs are supported, or small atom count work is shifted to CPUs, I think it would benefit if folding software is more able to predict card utilization, and do multiple WUs when possible.
When assigning 2 WUs to 1 GPU, one doesn't have control on what type of WU one will get.
When you'll get 2 large atom count WUs, your score will decrease drastically, as the GPU will deliver the result in about half the time (in case of 2 identically large and complex WUs being processed simultaneous).

[bruce]

CUDA could allocate GPU resources from a low atom count card to a higher atom card.
However that doesn't solve if both cards are running low counts.

So could OpenCL, but that's not the way FAH is designed to work. FAHClient assigns one WU to each GPU. In the early days of FAH, the speed of every WU was limited by the total number of shaders and their clock rate. That assumption no longer makes sense due to the phenomenal growth in GPU speed. Now the GPU could do so much more if more computations were moved to it across the PCIe bus.

One (small) WU isn't enough to keep a RTX 20xx busy.

[MeeLee]

@Foldy, how do you select the right GPU slot, on a multi GPU system?



As far as the estimates on best GPU go, based on gordonbb's numbers, taken a little more conservative,

Code: Select all

Name = Performnc Watt Price RecW P/CW P/USD
2080Ti=2,24M PPD 250W $1100 180W 13.3 2.036
2080s= 1,70M PPD 250W $720  180W 9.44 2.361
2070s= 1,50M PPD 215W $500  140W 10.7 3.000
2080 = 1,44M PPD 215W $700  140W 10.3 2.057
2060s= 1,35M PPD 175W $400  127W 10.6 3.375
2070 = 1,30M PPD 175W $500  127W 10.2 2.600 (mine does 1,27M PPD)
2060 = 1.04M PPD 170W $350  125W 8.32 2.971

* RecW= Recommended power cap on the card
* P/CW= PPD per capped watt

Super vs RTX 2060

Using the numbers from above, as well as pricing found online today:
3x 2060s performs about the same as 4x 2060, and saves you $200 initial at purchase, plus an additional ~$120 a year on electricity!
2x RTX 2070s has similar performance and cost to 3x RTX 2060, and saves you about $275/year in electricity.
3x 2080s gets roughly the same performance as 5x 2060, and pays you back in about 5 years. Not sure GPUs will run well for 5 years 24/7 though.



10x 2060 Super vs 9x 2070 Super
Since both perform very close to one another, I had to ramp up the amount of cards, to do a close comparison.
Both are equal in performance (13.5M PPD), as well as nearly identical power consumption.
9x 2070s costs about $500 more than 10x 2060s, which will take a good 25 years to break even with a 2060s.

In this case, the 2060 Super will be the better purchase, unless you're power or slot constricted.

[bruce]

Not sure GPUs will run well for 5 years 24/7 though.

It depends on your definition of "well". I'm running some 5 year old GPUs, and they've been running 24x7 the whole time. If you compare them to current GPUs, I should sell them and buy newer ones with better power efficiency, but they still draw the original power, which was good enough when I originally bought them. Is that "well"?

[foldy]

@Foldy, how do you select the right GPU slot, on a multi GPU system?

First when you start FAH it configures one GPU slot for each GPU and all GPUs are listed in beginning of FAH logfile as OpenCL devices.
e.g. Your first GPU listed is GPU index 0, the 2nd GPU listed is GPU index 1, ...

So if your 2nd listed GPU is a RTX 2080 ti and wants 2 work units then you create a new GPU slot with GPU index 1.

If you have issues with this setup then you can define all slots indexes by yourself
e.g.

Code: Select all

  <slot id='0' type='GPU'>
    <gpu-index v='0'/>
    <opencl-index v='0'/>
  </slot>
  <slot id='1' type='GPU'>
    <gpu-index v='1'/>
    <opencl-index v='1'/>
  </slot>
  <slot id='2' type='GPU'>
    <gpu-index v='2'/>
    <opencl-index v='2'/>
  </slot>

Then add additional slot for GPU 1 as slot 3

Code: Select all

  <slot id='3' type='GPU'>
    <gpu-index v='1'/>
    <opencl-index v='1'/>
  </slot>

[MeeLee]

Not sure GPUs will run well for 5 years 24/7 though.

It depends on your definition of "well". I'm running some 5 year old GPUs, and they've been running 24x7 the whole time. If you compare them to current GPUs, I should sell them and buy newer ones with better power efficiency, but they still draw the original power, which was good enough when I originally bought them. Is that "well"?

They might physically still run well (as long as the fans survive the long runtime), however current turnover rate is probably closer to 3 years. Meaning, Nvidia just introduced 2 GPU generations in 1 year. The RTX and RTX super cards, and is about to release a next gen about 6 months from now.
At the rate of progress, a 5 year old card will fold at half, to a third the speed of a new one that uses the same amount of power .

That being said, current smallest stable die size is 7nm, or a 5+nm using ULV litography. Meaning, Nvidia will continue to ride the 10nm train, until AMD pushes them to go smaller.
Until now, they had no reason to.
So 5 to 10 years is about the best realistic forecast in large gains of product improvements. After that, the increases in performance using current day technology, will be too small to notice. (Think Intel, who has been rebranding all their core I CPUs for the past 10 years or so).

I think Nvidia will go down to 5+nm nodes in the next 5 years, as well as a 30-100% increase in GPU cores.

From there on, I think progress in hardware will go slower again.
But an initial investment into RTX2060S or 2070 supers, should pay you back over older gen cards in no time, not only in electricity but in PPD and amount of work done per time.

[MeeLee]

@Foldy, how do you select the right GPU slot, on a multi GPU system?

First when you start FAH it configures one GPU slot for each GPU and all GPUs are listed in beginning of FAH logfile as OpenCL devices.
e.g. Your first GPU listed is GPU index 0, the 2nd GPU listed is GPU index 1, ...

So if your 2nd listed GPU is a RTX 2080 ti and wants 2 work units then you create a new GPU slot with GPU index 1.

If you have issues with this setup then you can define all slots indexes by yourself
e.g.

Code: Select all

  <slot id='0' type='GPU'>
    <gpu-index v='0'/>
    <opencl-index v='0'/>
  </slot>
  <slot id='1' type='GPU'>
    <gpu-index v='1'/>
    <opencl-index v='1'/>
  </slot>
  <slot id='2' type='GPU'>
    <gpu-index v='2'/>
    <opencl-index v='2'/>
  </slot>

Then add additional slot for GPU 1 as slot 3

Code: Select all

  <slot id='3' type='GPU'>
    <gpu-index v='1'/>
    <opencl-index v='1'/>
  </slot>

I can find the file in Windows, but would need to look around in Linux to find it.
If you remove a GPU, and can't start FAH, because of the GPU error, stop fahclient service, edit out the GPU in the config file, and restart the service, will fah work? Or will it try to download a configuration file from the server first, and still give the error?

[foldy]

In Linux Ubuntu I have it at /etc/init.d/config.xml where I start the FAHclient manually in console without service to see the errors easily.

[MeeLee]

You could type in console /etc/init.d/FAHClient log
It'll show you the same log as the client.
You can also open your logs, located in /var/lib/fahclient, with nano or a word editor

[MeeLee]

Seems like Nvidia is about to release super cards for the 1650 and 1660 cards as well.
The 1660 equivalent has lower cuda cores, but higher RAM speed than the 1660Ti, and would have given the 1660Ti a run for it's money, if they weren't priced nearly the same...
Since prices aren't out yet on the 1650 and 1660 Super cards, the current 1660 and 1660 Ti go for $229 and $279; we can estimate the 1660 Super to be priced around $259.

I would guess that the 1660 Ti would be a better buy for folding, due to the higher core count. GDDR speed doesn't matter as much for folding.
I would estimate that the 1660 Super might be more energy efficient than a standard 1660, based on their upgraded RAM, which allows them to run cooler.

[Akaanc]

I'm looking for the assigned wus and most of the time my cpu gets wus wit higher atom numbers. I have rtx 2080 and ryzen 9 3900x 12 core 24 threads. If gpus are faster why cpu gets wus with higher atoms? Why cant we run all work units on cpu or all on gpu?