Folding Forum

Hi Everyone,

It started getting worse sometime during the last year or so, I don't remember exactly when it started getting worse, but my system's PPD is FAR lower now than it used to be.

My system config is as follows:

AMD 3950X
ASUS Crosshair VIII Hero
64GB RAM
NVIDIA GTX 1080Ti
2TB NVME Gen 4 Boot Drive
Other drives
FAH 7.6.13

Even with NOTHING else running on my system, I'm only getting ~30K-225K PPD from my 3950x. Within the last year or so, I used to regularly get ~450K PPD from this system.

With the 3950x getting ~450K PPD, I would regularly get 2.3-2.8M PPD from my GTX 1080 Ti. Now I get 1.8-2.3M PPD.

Combined, I've hit as much as 3+M PPD per day, with averages of 2.5-2.8M PPD.

I've even tried changing the CPU cores used from the default -1 to 30, 28, 16, 15, with no improvement in either CPU or GPU performance. Right now, I'm at 15 cores and it is doing the same CPU as with -1. The GPU also doesn't vary with the number of cores used.

I can see the amount of CPU usage changing as I change cores (~50% with 15 cores), but the PPD doesn't change. It seems to be fixed.

This is REALLY strange.

Of course, all of these numbers are with nothing running in the background. My temps on both the CPU and GPU are in the 68-78C range, usually in the low 70s (70-74C).

I've tried restarting my machine many times and no joy. I've uninstalled and reinstalled FAH a couple times also.

I know that my passkey is correct, so I should still be getting the same PPD as I did a year ago.

I've looked around the forums and see that some other AMD users have had problems with the latest FAHCORE. I can get it to run consistently for about 2-4 days before I start seeing FAHCORE crashes and then have to reboot my system.

Does anyone have any idea short of a complete reinstall of Windows to improve this? I'm at my a complete loss.

Any help would be appreciated.

Scott

OK, this is WEIRD!

Just for grins, I changed the number of CPU cores from 15 down to 10 and now, with only "~35%" of my CPU working, I'm getting about 425K PPD out of my CPU, with no change on my GPU.

Does anyone have any idea what is going on?!?

Thanks.

Scott

PS. I'd upload a screenshot, but I don't see that option...

It has been suggested that the optimal smp client setup to have 24 cores allocated, no more. As for the 1080ti there is some fluctuation over time, look here.https://folding.lar.systems/gpu_ppd/bra ... 0_ti_11380
Seems you could get better results by installing Linux.

The forums don't host images, you need to host them somewhere else and then link to them.

A few thoughts:-
GPU - suggest updating your geforce drivers. Cuda was only fully supported with the release of Core_22 0.0.18 in November of 2021 and this is when you should have seen a sustained increase in PPD. It requires at least geforce driver 472 to operate.
viewtopic.php?t=37391
I'm currently running 511.79 and there are a couple of threads here have highlighted issues with slightly newer drivers.
Cuda support was initially pushed out with the release of core_22 0.0.14 in July 2021 but required support that wasn't flagged during testing, so was pulled until it could be further developed. There were a couple of versions pushed in the interim but I believe they didn't support some of the newer RTX cards.
I also noted that the species for GTX 1080 Ti was deranked from 8 to 7 a few months ago, this shouldn't affect the cards PPD, only the upper limit of which work units it receives.

CPU - do a prime 95 stress test on your CPU, you want small FFT with maximum heat. Run the test for at least a few hours, preferably 24. I found on my system I had a weak core and would error out. Initially I increased the VRM voltage to stabilise it but I eventually had to swap out the motherboard for one with a stronger VRM.

It probably worth noting that a couple of other users here with have trouble with their 3950x have never been able to resolve their issues but it might be worth a look for ideas.
viewtopic.php?t=37535

Core/Thread affinity - OK, to briefly explain, this is tying specific processes to specific cores/threads of the CPU with the intention of segregating folding processes from normal background processes on the CPU. The suggestion is to look at a program called process lasso. Personally, I do this through a power shell script.

Thanks for the replies!

However, most of your suggestions won't address my issue. For one, I can't change to another OS and REALLY don't want to do a clean install as it'll probably take me the better part of a week's worth of time to make sure I have all my data/apps backed up AND get everything set up again. I've done this before and there is always something "lost in the translation" (for better or worse).

Again, I'll state that my system was working FINE up until about a year ago and has been getting progressively worse.

My temps are reasonable on both the CPU and GPU. That was my first thought, but they've been good. I've been using (and still do) use CPU Temp to keep an eye on the CPU temps. I think it just reports the hottest core, so I should be good. It's reported power draw is about 105-110W.

I'm up to date on all of my drivers and update them regularly, when notified of new drivers. I did notice a bump in GPU performance around the end of last year, but it didn't last.

Also, I'm REALLY confused on why my CPU PPD went UP when I decreased my core count from -1 down to 10. I'm getting a significantly better CPU PPD at a core count of 10 on my 3950x than on another system with a 3600x set at 10 cores. Neither of these results make sense to me.

FWIW, my current CPU PPD is about 285K PPD (reported by the Advanced Control Screen).

I'll take a look at that 3950x thread.

Do any of the FAH devs ever come here to take a look at strange issues? It seems like the AMD part of the FAHCore has more issues than the Intel part. And AMD's GPUs appear to be slower than expected, even though they have competitive benchmarks.

Thanks again and any other suggestions would be great!

Scott

AMD systems are getting the same exact FAHCores as the Intel systems, the only CPU hardware that gets a different core is ARM. There are separate core downloads for Nvidia, AMD, and Intel GPUs.

There is a performance "knee" that will be seen by high core count processors. It will depend on the CPU project, those WUs with higher atom counts will have it occur at a higher CPU thread count than WUs from projects on lower number of atoms. They do try during internal testing to determine thread counts that result in poor performance, but don't always catch all of them. There are some other influences on performance that show up on high core/thread count processors as well.

GlueFactoryBJJ wrote: ↑Sat Jun 11, 2022 6:51 amAlso, I'm REALLY confused on why my CPU PPD went UP when I decreased my core count from -1 down to 10. I'm getting a significantly better CPU PPD at a core count of 10 on my 3950x than on another system with a 3600x set at 10 cores.

My best guess? It's the difference between running something on 10 physical cores and 10 pretend cores.

The 3600x does not have 10 physical cores, it has 6 physical cores pretending to be 12. Intel call it hyper-threading, AMD call it multi-threading. While it does give an uplift in performance it is nowhere near the equivalent of adding physical cores.
The 3950x does have more than 10 physical cores (it is a 16 core/32 thread CPU). When you set the slot size to 10 Windows is more than likely dedicating the performance of 9 or 10 physical cores to FAH. As to why you're not getting better performance at higher thread counts, partly the size of the protein (see Joe_H on that one) and possibly too much contention within the CPU, beyond that I don't know.

I don't run my cpu slot set to -1 and I advise others to avoid it as well. I find that it impacts folding in other areas. By default the client will assign a single cpu thread/core to each gpu and grab the rest for the cpu slot. I normally advise to manually set the cpu slot at least 1 core, or 2 threads, lower to give the operating system room to breath.

There are a few things you could try:-
*). enable Hardware-accelerated GPU scheduling in Windows (Start -> Settings -> System -> Display -> Graphics settings)
*). disable SMT (multi-threading) in the bios
*). just run with a smaller cpu slot size
*). set a couple of cpu slots each with a smaller number of threads/cores

A note of thread/core size - when you adjust the thread/core size of a slot the direction affects when the adjustment kicks in. Increases kick in when the next work unit starts, decreases kick in immediately.

I will say, I run my 3900x with a 20 thread slot and I run about 300K PPD (depending upon project the swing is 250-400K PPD), so with your 10 core slot, you're outstripping me.

At the end of the day, your computer is in front of you and I'm just an anonymous nobody behind a keyboard half the world away.

This post reminds me of a problem I had. PC would run great for hours and then slow down for no reason. It took me weeks to figure out the power supply unit was going bad. I'm sure his problem isn't that simple.

Hey Everyone!

Thanks for all of the replies!

I'm sure that there are a number of reasons for the problem, but I feel I need to reiterate one point:

I previously got CPU performance of ~425K PPD and 2.4-2.8M PPD (CPU and GPU, respectively) and up to 3.1M PPD combined, UNTIL about a year ago when the PPD slowly started dropping until the PPD got to its current levels (775K-2.4M PPD, combined). And that was with both the CPU and GPU slots set at -1.

I don't have any strange system crashes, so I don't think it is the PS, though if I were having them, that would be the first place I'd look to, just from past experience.

I don't think it is the difference between "real cores" (non-hyperthread) and "pretend cores" (hyperthread). My reason for saying this is that my understanding is that FAHCore creates and assigns threads rather than picking actual cores/hyperthreads. However, I could be wrong.

Regardless, I seem to be hitting a wall at about 300K PPD on the 3950x now. My CPU is running about 4.0 - 4.1+ GHz, all cores, which should help, at least compared to the 3.5GHz "stock" speed. None of the cores are getting over 80 degrees C (the max rating is supposed to be 95C), whether running at "10 cores" or "-1" cores (which I believe is supposed to be the same as "30 cores", with my 16 core/32 thread CPU.

Also, I don't know anything about how the WUs are programmed and if poor WU programming may account for some decreases. Of course, if few other AMD users are seeing a decline, then that probably isn't the issue.

Other than that, I'm still at a loss.

I'm still experimenting, but my results of that experimentation are really strange. I tried going from 15 to 18 cores, but my PPD for both my CPU, and especially my GPU, dropped significantly (over 500K PPD, combined). Strangely, even at "18 cores", my cpu utilization (via Task Manager) wouldn't go over 50-55%.

I can get the CPU utilization to hit near to, or at, 100% if I change it to "-1".

I'm just trying to get back to what I used to be able to do for FAH, but something is holding my system back and I can't think of any significant changes/additions to my PC around that time. Just "normal" updates of apps and the OS.

I'm REALLY trying to avoid doing a Win 10 Pro reinstall or replacing the motherboard, CPU, etc.

Frankly, I'd like to try whatever clients were in use in, say the summer of 2020 through the end of 2021, when I was getting my highest PPDs, but don't know where to find them. At least that would let me know it isn't a newer client that's the problem.

Thanks again for the help and I'd appreciate any suggestions, if you have them!

Scott

Ok, this is just nuts.

After taking the hit at 18 cores on the CPU, I changed it back to 14 cores (still -1 on the GPU).

Now my CPU is reporting 1.4M PPD on the CPU, with an appropriate finishing time. My GPU has cut back a bit (1.9-2.0M PPD), but my combined is currently being reported at ~3.4M PPD.

What the heck is going on?

Scott

And now it just dropped down to about 250K PPD on the CPU and about 2.0M PPD on the GPU.

UGH!

Scott

I wouldn't pay too much attention to it. Some WUs run lower PPD scores on CPU.
Also, you need to wait at least before 5% of the WU is done, before reading results.
Third, the PPD scores only work when the CPU is folding 100% of the time, and doing nothing else.

I don't think that limiting the number of cores will fix the problem, at least not on a permanent basis. But as Joe_H points out, there is a performance "knee" beyond which more cores do not do much good. But occasionally they do. My Ryzen 5950X will normally get only slightly more PPD than my 5900X, but at the moment it is getting 903 k PPD on P16959, one of the high-yielding ones (Ubuntu 20.04.4).

As for the 3950X, I used to run one on Folding also, but have never seen such swings. I do know that the CPU operating characteristics do change over time however, and a memory speed (for example) that was stable when it was new may no longer be stable after a couple of years. So I would try various fixed CPU frequencies and voltages to see if you can find a stable one.

PS - Since you are on Windows, there is always the possibility that the anti-virus is interfering with it. I would try disabling it. Just excluding the folders does not always prevent the real-time protection from inspecting the packets.

One other thought, similar to the power supply idea mentioned above. And that is the voltage regulator on the motherboard is getting flaky. That could cause the CPU speed to vary. (Do you have the latest BIOS?).

I have had several Ryzen motherboards fail with bad voltage regulators in the last three years. Usually, they just fail outright. But maybe they could get unstable? Replacing a MB is no fun, but I have gotten used to it by now.

@Jim, it's not only folding that proves that the 3900 series is memory bandwidth limited to roughly 24-28 cores.
Other programs also have indicated that there's an internal bottleneck (most likely RAM wise, or Infinity fabric wise).

You can always try to flash your Bios, make sure you have thermal paste applied to CPUs at least on a Bi-Annually timeframe for units that run 24/7 at 100%, and hopefully you have a quad channel capable motherboard (not 4 memory slots in dual channel configuration).

I find that most Ryzen 3000 series and their motherboards are incredibly unstable.
I find my Intel systems with GPU to run much more stable.

As far as the VRMs, some get way too hot (burning hot).
I'd use something like an 80mm case fan to cool them down, especially if you are using AIO solutions on your CPU.