Productivity Suddenly Dropped

[DrBB1]

I replaced my PC in September, and until about two weeks ago I had been earning about 400,000 points/day. Since then, however, I am averaging only about 30,000, more than a 90% drop. I checked my settings and they haven't changed: 10 cores available, always folding. I noticed that only four cores seem to be chugging away at 100% utilization when I fold; unfortunately, I don't know if more cores were active when I was being more productive. As far as I can tell, nothing new was installed on my PC around the time of the productivity drop.

Could this drop-off be due to the nature of the projects being made available? If not, I'd appreciate any further troubleshooting suggestions. TIA

[AZBrandon]

Any chance you can post the header of the log file when starting a new WU?

[muziqaz]

I replaced my PC in September, and until about two weeks ago I had been earning about 400,000 points/day. Since then, however, I am averaging only about 30,000, more than a 90% drop. I checked my settings and they haven't changed: 10 cores available, always folding. I noticed that only four cores seem to be chugging away at 100% utilization when I fold; unfortunately, I don't know if more cores were active when I was being more productive. As far as I can tell, nothing new was installed on my PC around the time of the productivity drop.

Could this drop-off be due to the nature of the projects being made available? If not, I'd appreciate any further troubleshooting suggestions. TIA

Mind reading ability does not grow on trees. Please provide full system specs

[DrBB1]

I'm not a hi-tech person, so excuse me if I'm not understanding all the jargon or need more clarity on attempts to help me. Is this the information requested?

CPU
Intel Core
Cores 10
Threads 10
Name Intel Core
Code Name Arrow Lake
Specification Intel Core Ultra 5 225
Family 6
Extended Family 6
Model 6
Extended Model C6
Stepping 2
Instructions MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, Intel 64, NX, AES, AVX, AVX2, FMA3
Virtualization Not supported
Hyperthreading Not supported
Bus Speed 99.7 MHz
Average Temperature 38 °C
Caches
L1 Data Cache Size 6 x 48 KBytes
L1 Instructions Cache Size 6 x 64 KBytes
L2 Unified Cache Size 6 x 3072 KBytes
L3 Unified Cache Size 20480 KBytes
Cores
Core Speed Multiplier Bus Speed Temperature Threads
Core 0 2094.1 MHz x 21.0 99.7 MHz 38 °C APIC ID: 0
Core 1 1595.5 MHz x 16.0 99.7 MHz 38 °C APIC ID: 8
Core 2 1495.8 MHz x 15.0 99.7 MHz 38 °C APIC ID: 16
Core 3 2891.8 MHz x 29.0 99.7 MHz 38 °C APIC ID: 18
Core 4 1495.8 MHz x 15.0 99.7 MHz 40 °C APIC ID: 20
Core 5 2891.8 MHz x 29.0 99.7 MHz 38 °C APIC ID: 22
Core 6 2193.8 MHz x 22.0 99.7 MHz 40 °C APIC ID: 32
Core 7 2094.1 MHz x 21.0 99.7 MHz 36 °C APIC ID: 40
Core 8 1794.9 MHz x 18.0 99.7 MHz 38 °C APIC ID: 48
Core 9 2094.1 MHz x 21.0 99.7 MHz 38 °C APIC ID: 56

08:37:53:I1:WU526: CUDA: OFF
08:37:53:I1:WU526: Args: -dir sndtPBCkiy6o9xiT9xAD1IHThhrbk_IKhkqDRPNYle0 -suffix 01
08:37:53:I1:WU526: -version 8.4.9 -lifeline 25496 -np 10
08:37:53:I1:WU526:************************************ libFAH ************************************
08:37:53:I1:WU526: Date: Jan 16 2021
08:37:53:I1:WU526: Time: 11:24:13
08:37:53:I1:WU526: Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
08:37:53:I1:WU526: Branch: master
08:37:53:I1:WU526: Compiler: Visual C++ 2019 16.7
08:37:53:I1:WU526: Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
08:37:53:I1:WU526: Platform: win32 10
08:37:53:I1:WU526: Bits: 64
08:37:53:I1:WU526: Mode: Release
08:37:53:I1:WU526:************************************ CBang *************************************
08:37:53:I1:WU526: Date: Jan 16 2021
08:37:53:I1:WU526: Time: 11:23:53
08:37:53:I1:WU526: Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
08:37:53:I1:WU526: Branch: master
08:37:53:I1:WU526: Compiler: Visual C++ 2019 16.7
08:37:53:I1:WU526: Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
08:37:53:I1:WU526: Platform: win32 10
08:37:53:I1:WU526: Bits: 64
08:37:53:I1:WU526: Mode: Release
08:37:53:I1:WU526:************************************ System ************************************
08:37:53:I1:WU526: CPU: Intel(R) Core(TM) Ultra 5 225
08:37:53:I1:WU526: CPU ID: GenuineIntel Family 6 Model 198 Stepping 2
08:37:53:I1:WU526: CPUs: 10
08:37:53:I1:WU526: Memory: 15.46GiB
08:37:53:I1:WU526:Free Memory: 3.96GiB
08:37:53:I1:WU526: Threads: WINDOWS_THREADS
08:37:53:I1:WU526: OS Version: 6.2
08:37:53:I1:WU526:Has Battery: false
08:37:53:I1:WU526: On Battery: false
08:37:53:I1:WU526: UTC Offset: -5
08:37:53:I1:WU526: PID: 68744
08:37:53:I1:WU526: CWD: C:\ProgramData\FAHClient\work
08:37:53:I1:WU526:********************************************************************************
08:37:53:I1:WU526:Project: 19229 (Run 4335, Clone 9, Gen 4)
08:37:53:I1:WU526:Unit: 0x00000000000000000000000000000000
08:37:53:I1:WU526:Reading tar file core.xml
08:37:53:I1:WU526:Reading tar file md4.tpr
08:37:53:I1:WU526:Digital signatures verified
08:37:53:I1:WU526:Calling: mdrun -c md4.gro -s md4.tpr -x md4.xtc -cpt 5 -nt 10 -ntmpi 1
08:37:53:I1:WU526:Steps: first=2000000 total=2500000
08:38:16:I1:WU526:Completed 1 out of 500000 steps (0%)
08:53:21:I1:WU526:Completed 5000 out of 500000 steps (1%)
09:08:25:I1:WU526:Completed 10000 out of 500000 steps (2%)
09:23:29:I1:WU526:Completed 15000 out of 500000 steps (3%)

[muziqaz]

Your CPU has performance (6) and eco (4) cores.
Go into fah settings an set the CPU slider to 6, to fold on 6 Performance cores, and let eco cores be.
On top of that write to your local intel representative to express your opinion on how their big.shittle CPU architecture is crap
Well, skip that last bit, they won't listen.
See how those 6 cores perform

[foxpy]

Windows really likes to schedule background jobs on smaller CPUs. FAH has a fix for MacOS, where is asks MacOS to run stuff on big CPUs, but there is unfortunately nothing like that for Windows. Simply setting thread count to 6 (like in MacOS) won't suffice, you will have to install Process Lasso and configure it to pin FAH core to big CPUs. Alternatively, if you don't want to be bothered, just set the threads slider to 4 and fold on small CPUs only, but that would obviously not be super fast.

[DrBB1]

Your CPU has performance (6) and eco (4) cores.
Go into fah settings an set the CPU slider to 6, to fold on 6 Performance cores, and let eco cores be.... See how those 6 cores perform

Sorry, that didn't have any effect. I reset to use all 10 cores. I'm wondering now if there is a Windows 11 setting that I unknowingly changed that might have been the cause of my 90% loss in efficiency. Please keep in mind when responding that while I am not a Luddite (I've been folding over 18 years now), I'm not a high tech person either, so clear and simple suggestions are ideal. TIA.

[muziqaz]

Set it to 6 cores, and use Process lasso to tie fahcore_a8 and fahcore_a9.exe to those 6 P cores.
It is also possible, your CPU is overheating

[DrBB1]

... It is also possible, your CPU is overheating

Not overheating, but I checked the Windows Power settings and found the Power Mode set to "Best Efficiency." When I changed it to "Best Performance," TPF dropped from about 16.5 minutes/frame to about 4 minutes per frame. Switching to "Balanced" did not increase TPF, so that's where I'm leaving it. I'll give it a few days and see if PPD returns to it's earlier volume. All 10 cores are chugging away like they are supposed to. I'll give it some more time, then dial back to 6 cores to see if performance is affected.

Thanks for your help. Much appreciated!

[foxpy]

Yeah, the worst thing for FAH is folding on more threads than physically available, your issue was 10 threads crammed into 4 CPUs, leading to almost zero PPD. Glad to hear it works well on your side now!

You might also want to benchmark and compare how well 6 threads perform V/s 10 threads. E-cores on newer Intel CPUs are a lot better than in older ones, but performance gap is still quite high. There is a chance 6 threads will yield even more PPD than 10.

[DrBB1]

Yeah, the worst thing for FAH is folding on more threads than physically available, your issue was 10 threads crammed into 4 CPUs, leading to almost zero PPD. Glad to hear it works well on your side now!

You might also want to benchmark and compare how well 6 threads perform V/s 10 threads. E-cores on newer Intel CPUs are a lot better than in older ones, but performance gap is still quite high. There is a chance 6 threads will yield even more PPD than 10.

Good idea. I tested it BRIEFLY, and based on a very small sample of 10 frames in one WU, 6 CPUs took about 20% longer to fold than using all 10 CPUs in the same WU. I'm going to let it run until the end on 6 and see if the time moderates with a larger sample. I wonder: has anyone has done a more carefully controlled study, with multiple WUs and multiple PC models, to see if there is a general rule for selecting the optimum number of CPUs to fold with?

[foxpy]

Good idea. I tested it BRIEFLY, and based on a very small sample of 10 frames in one WU, 6 CPUs took about 20% longer to fold than using all 10 CPUs in the same WU. I'm going to let it run until the end on 6 and see if the time moderates with a larger sample. I wonder: has anyone has done a more carefully controlled study, with multiple WUs and multiple PC models, to see if there is a general rule for selecting the optimum number of CPUs to fold with?

I haven't seen anyone do such a study, but according to my own experience, it shouldn't be that difficult, there a few simple rules to this:
1. the more threads the better (obviously), as long as there are physical threads to run them
2. FAH cores are developed in such a way, that all threads run at the speed of the slowest one, and this creates challenges with SMT (Intel call that HT in their marketing) and hybrid architectures, for example, using 4 threads on 4 core 8 thread CPU is more productive than using 5 threads, but less productive than using 8 threads
3. there is a limit to how much FAH cores scale (I think it is something around 40 threads? I never had a chance to test that on a high core count system, but this limit was mentioned somewhere on this forum)

Now, speaking about Intel's hybrid CPU design, I am surprised 10 threads yield more PPD than 6 threads on your setup. Keep in mind, that due to FAH core design, all 10 threads run at the speed of the slowest one, so your performance is essentially equal to the performance of 10 E-cores. Maybe Intel's Skymont design is already so good it can compete with their P-cores now? It is certainly not the case with Gracemont though. I have i5-14500 and I have to run a separate WU on E-cores and a separate one on P-cores, because if I utilize all of them together with a single WU, productivity will fall behind of using P-cores only.

[DrBB1]

I ran a few quick tests (5-10 frames) using 6, 8, and 10 cores on the same WU. 10 cores was consistently about 20% faster per frame than 6 cores, with 8 cores about half way in between (possibly a bit closer in speed to 1the 10 -core run). I certainly wouldn't want to draw any firm conclusions from this quasi pilot study (limited observations, one machine, one WU), but someone with more interest and time (and perhaps funding!) might want to go for it!

[foxpy]

I ran a few quick tests (5-10 frames) using 6, 8, and 10 cores on the same WU. 10 cores was consistently about 20% faster per frame than 6 cores, with 8 cores about half way in between (possibly a bit closer in speed to 1the 10 -core run). I certainly wouldn't want to draw any firm conclusions from this quasi pilot study (limited observations, one machine, one WU), but someone with more interest and time (and perhaps funding!) might want to go for it!

I've just realized. Did you make sure all FAH threads were scheduled on P-cores during your tests? It is not hard to check again:
1. run a WU and set threads count to 6
2. open Task Manager, navigate to the tab "Details", find FahCore_a8.exe or FahCore_a9.exe, right click on it, select "Set affinity" and make sure that only the first 6 CPUs are checked (from CPU0 to CPU5)
3. let WU run for a while
4. observe the productivity difference (if any)

[Joe_H]

2. open Task Manager, navigate to the tab "Details", find FahCore_a8.exe or FahCore_a9.exe, right click on it, select "Set affinity" and make sure that only the first 6 CPUs are checked (from CPU0 to CPU5)

Except Windows doesn't necessarily number the P-cores in that order. From the details in this post - viewtopic.php?p=372614#p372614 - I would guess the P-cores are 0, 3, 5, 6, 7 and 9

Code: Select all

Cores
Core Speed Multiplier Bus Speed Temperature Threads
Core 0 2094.1 MHz x 21.0 99.7 MHz 38 °C APIC ID: 0
Core 1 1595.5 MHz x 16.0 99.7 MHz 38 °C APIC ID: 8
Core 2 1495.8 MHz x 15.0 99.7 MHz 38 °C APIC ID: 16
Core 3 2891.8 MHz x 29.0 99.7 MHz 38 °C APIC ID: 18
Core 4 1495.8 MHz x 15.0 99.7 MHz 40 °C APIC ID: 20
Core 5 2891.8 MHz x 29.0 99.7 MHz 38 °C APIC ID: 22
Core 6 2193.8 MHz x 22.0 99.7 MHz 40 °C APIC ID: 32
Core 7 2094.1 MHz x 21.0 99.7 MHz 36 °C APIC ID: 40
Core 8 1794.9 MHz x 18.0 99.7 MHz 38 °C APIC ID: 48
Core 9 2094.1 MHz x 21.0 99.7 MHz 38 °C APIC ID: 56