GPU CORE22 0.0.2 coming to FAH - p11737-9 feedback thread

[squads]

yeah it seems a Radeon VII should be able to do better than 1.8-1.9M PPD on 11738, so your riser is probably costing you there. I have a Vega 56 that does 1.4M on those WU in a PCI3 x16 slot

[Prettz]

Seeing fairly wild fluctuating PPD on 11738 (0, 346, 39) on a GTX 1070 Ti. It's not pushing the GPU very hard, power and heat-wise. Jumping around between roughly 730k and 810k, which unfortunately happens to be the range of PPD I get from all of the core 21 projects. Not used to seeing fluctuations like that.

Large atom-count projects generally need wider bandwidth so they tend to have degraded performance on Windows and/or with 1x risers.

What's the connection between Windows and PCI bandwidth?

[bruce]

There's really no direct connection. It also depends on what project we're talking about.

For a long time, FAH contended that PCIe bandwidth wasn't particularly important ... but projects have grown, requiring more data to move across the bus ... and GPUs have gotten faster, requiring a better supply of data (i.e.-bandwidth) to keep up with the speed of the computations.

1x risers or even a 4x slot can slow down a fast GPU when working on a large-atom-count-protein. Windows is less effective than Linux at utilizing all of the bandwidth that's actually avaiable.

If you've got a 16x 3.0 slot the bandwidth probably won't limit your GPU's computation capabilities but successive limitations become more an more important if you want to see progress limited entirely by the computational capabilities of your top-of-the-line GPU. Adding an additional moderate speed GPU on a moderate speed slot with the limitations imposed by Windows will be a positive benefit, but your mileage may vary.

[Prettz]

Windows is less effective than Linux at utilizing all of the bandwidth that's actually avaiable.

Adding an additional moderate speed GPU on a moderate speed slot with the limitations imposed by Windows will be a positive benefit, but your mileage may vary.

What limitations?

[bruce]

What limitations?

I don't think we need to get into a discussion of why MacOS or Linux or Windows is better than the other two. That's mostly a matter of preferences and this forum is aimed at helping people with difficulties getting FAH to work. Your choice of OS is a personal discision (aka "prejudice") and I'm sure we can find peopole who make other choices for equally prejudiced reasons, but we're not going to do that here.

The internal construction of how Windows handles moving data to/from the GPU is different than the internal construction of Linux. Device drivers, interrupt handlers, task schedulers, etc. are inherent parts of an OS and you're pretty much stuck with the performance of the aggregated results. Measurements show that Windows is less effective than Linux at utilizing all of the bandwidth that the hardware can actually provide.

Why? Because that's the way it was written.

[n_w95482]

I took a look at my logs yesterday and noticed that I'm now receiving core 22 WUs - this is without using the advanced flag. I received my first core 22 WU (p11737) on January 31st, with a steady trickle until about February 9th. Since then, I've mostly received core 22 work on both my GTX 1080 Ti and RX 580, all of it being p11737.

I haven't noticed much difference yet with the RX 580 in terms of performance or power consumption, but the 1080 Ti has had a marked increase in both PPD (~1.2-1.4m -> ~1.65m) and power consumption (80-90% TDP (250w) -> 98-111%).

The 1080 Ti is running 441.87 drivers from late December, and the RX 580 is running 20.1.1 from mid-January. Both cards have been perfectly stable, even with a custom clock/voltage curve on the 1080 Ti. I did have the RX 580 underclocked 7% for lower noise, but reverted to stock (1366 MHz) just now for testing. The 1080 Ti runs at 1999-2012 MHz.

I just enabled the advanced flag on both machines to see if p11738/11739 show up, as well as the new core 21 WUs. Fun stuff .

Edit: Forgot to add that the 1080 Ti is sitting at 20-21% PCIe bus utilization (3.0 @ x16).

[bruce]

Thanks for the report.

FAHCore_22 is expected to provide improvements over what FAHCore_21 has provided, both scientifically and in GPU utilization. I expect that the current research being done with Core_21 will gradually be completed and new projects will be initiated on Core_22, so that Core_21 will gradually be phased out. (That's consistent with your report.)

I don't have a much data associated with GPU utilization factors associated with specific projects on specific GPUs. What GPU utilization do you observe, tabulated by project and GPU? Is the increase in PPD directly related to increased utilization percentage and power/heat? If more work is being done, PPD should increase. If not, then it shouldn't.

[n_w95482]

Unfortunately, I don't have specific numbers for prior projects as I wasn't paying too much attention to all of the details then, just that the 1080 Ti seemed to be averaging 80-90% of rated TDP most of the time, with GPU utilization % close to those numbers as well.

I'm unsure of the exact power consumption of the RX 580 as the numbers reported by AMD's API are usually all over the place for me, same with GPU utilization.

Here's what I've seen thus far from the projects that are the subject of this thread (also, this is my first time using HFM's Work Unit History viewer, very neat!):
1080 Ti

• p11737: 69 WUs thus far (1 in progress), 1.57-1.66m PPD, 98-111% rated TDP, GPU utilization was usually about 97-98% IIRC. The power consumption exception is the one running right now (R6, C9, G126), which appears to be lighter on power consumption (88-92% TDP) but particularly heavy on memory bandwidth utilization (42-48%)
• p11738: 2 WUs thus far, 1.36-1.4m PPD, unknown power/GPU utilization (I wasn't home when they ran)
• p11739: 3 WUs thus far, 1.62-2.15m PPD, unknown power/GPU utilization (I wasn't home when they ran)

RX 580

• p11737: 38 WUs thus far, 299k-313k PPD (most of that was underclocked)
• p11738: 1 WU thus far (1 in progress), 565k PPD on the one that finished - the one in progress was paused for a couple of hours, currently at 474k. Power consumption/heat doesn't seem any higher than normal, at least based on the fan noise.
• p11739: No WUs thus far

One other thing that I've noticed is that both with these core 22 WUs and the core 21 ones in advanced, my 1080 Ti's coil whine has been greatly reduced .

[snapshot]

A brief snapshot of what's currently running here. I have two similar GTX 1660 Tis which are capped at 71C to keep the noise and heat down.

Project PPD TDP GPU
11739 939,000 85% 96%
14291 497,000 86% 96%
14294 544,000 84% 97%

So very similar power & GPU usage for much better results. I have to admit that that's an exceptional 11739 and around 850,000 PPD would be more typical. The 142xx WUs are also lower scoring than whatever the previous core 21 series was as that gave around 620,000 PPD.

[Frontiers]

The Radeon VIIs really seem to like the 11738 and 11739 WUs, but it should be noted that (based on my results) the 11737 has disappointing PPD (~780,000) on the Radeon VII.

Don't know any other card halving PPD on small atom-count projects, so in this case R-VII may be temperature limited, because it may use hot-spot sensor data for clock on the fly clock adjustment.
If it cycling between 1300 and 1750 MHz on this projects - it's not enough cooling, you may look in GPU-Z during folding 11737 WUs.
Possible solutions are here: https://www.igorslab.de/rtg-radeon-twea ... -igorslab/
I also saw at folding@evga team forum 1.9-2.05M PPD for watercooled R-VII @1850 MHz for 11738 and 11739 projects.
Sadly in GPUs.txt R-VII rated at only 5 while some Nvidia cards with twice less core-counts rated at 7.
If all 11737 WUs would be redirected by assignment servers to GTX 1070's - it would be fine, because 1070 do them with pretty small difference compared to 11738 and 11739, lower only 50k or so.

[Joe_H]

Sadly in GPUs.txt R-VII rated at only 5 while some Nvidia cards with twice less core-counts rated at 7.

The numbers you are looking at are not ratings, but "species", and are specific to the brand of GPU.

In this case 5 for the Radeon-VII indicates it supports both single and double precision calculations, AMD GPU's that don't support DP are species 4. Species 6 is for the new Navi based cards from AMD. There is no species 7 yet for AMD.

The species classes for nVIdia are similar, but not the same as for AMD.

[bruce]

Don't know any other card halving PPD on small atom-count projects, so in this case R-VII may be temperature limited, because it may use hot-spot sensor data for clock on the fly clock adjustment.

That's one possibility, but I'd guess it's dependent on the number of atoms compared to the number of shaders that the GPU can use. As the number of atoms increases. it gets harder and harder for all the shanders to be kept busy all of the time, so efficiency goes down. Small atom count proteins can keep a small GPU very busy but large proteins are less effective at keeping all the shaders busy. When that condition is discovered during benchmarking, the smaller proteins are designated for GPUs that aren't so big.

[Frontiers]

Another cause of low PPD and underutilization with low atom-count projects may be ULPS enabled, everyone with Vega/5700XT/R-VII - can try to disable this thing and after rebooti see to PPD with low atom-count projects:
https://community.amd.com/thread/176003

[JimF]

Small atom count proteins can keep a small GPU very busy but large proteins are less effective at keeping all the shaders busy. When that condition is discovered during benchmarking, the smaller proteins are designated for GPUs that aren't so big.

That is interesting. I always thought that they did not segregate work units by size. Maybe it came in with the new client?

[bruce]

We've been working on creating that kind of segregation. It's not a perfect solution, but it can certainly help. Also it will need to be tweaked in a few years as many of the smaller GPU get upgraded to big one. (I don't anticipate that small projects will vanish and somebody needs to process them.)