FAH underperforming on new system

[tom.w.2277]

I have been running FAH since ~2020. Initially on the first 2 systems listed below. I currently have 4 systems running FAH, all are running Windows - From oldest to newest:

1) i3-2105, 16GB DDR3 RAM, Windows 10 64 bit with latest updates, 14K PPD Average(over 1 day - This system was offline for close to a year, and I just upgraded to FAH 8.4.9)
2) i5-6600K, 32GB DDR4 RAM, Windows 10 64 bit with latest updates, 64K PPD Average(over 33 days)
3) i3-12100, 32GB DDR4 RAM, Windows 10 64 bit with latest updates, 165K PPD Average(over 33 days)
4) i5-13500, 16GB DDR4 RAM, Initially running Windows 10 64 bit with latest updates, but upgraded to Windows 11 64 bit with latest updates as of 5/25/2025, 37K PPD Average(over 16 days, with 6 of 8 jobs incomplete the first 7 days)

I am trying to understand why the i5-13500 system, which has a benchmark rating 2.5 times higher than the i3-12100 and 5 times the i5-6600K, is considerably underperforming both. When FAH is running on the other 3 systems, total CPU utilization is normally in the range of 90-100%, and FAH CPU utilization is in the range of 80-90+%. On the i5-13500 system, total CPU Utilization is normally around 60% with FAH CPU utilization in the mid 50s%. I was anticipating PPD for this new system to be on the order of 300K+, and getting around 10% of that, even with 50+% CPU utilization is rather disheartening.

When I run the PassMark Burnin Test - Max CPU Temp test, CPU utilization goes to 100% and the test passes with no errors - including CPU core temps, which stabilize around 60 degrees C. I have run every applicable Burnin Test and they all passed, except for the first time I ran the Max CPU Temp test, which failed with a Max CPU Temp of 90 C because the CPU Thermal Paste had just been applied - All subsequent Max CPU Temp tests have passed.

All 4 systems are running FAH 8.4.9 and are configured to utilize all but one of the available CPUs in their respective system. Any ideas as to what is causing the poor FAH performance on this system, and what I can do to get it performing as expected?

Here is the current Machine Log. I don't see any other log files to attach.

Code: Select all

23:08:22:I1: Version: 8.4.9
23:08:22:I1: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
23:08:22:I1: Org: foldingathome.org
23:08:22:I1: Copyright: 2023-2024, foldingathome.org
23:08:22:I1: Homepage: https://foldingathome.org/
23:08:22:I1: License: GPL-3.0-or-later
23:08:22:I1: URL: https://v8-4.foldingathome.org/
23:08:22:I1: Date: Nov 20 2024
23:08:22:I1: Time: 14:47:46
23:08:22:I1: Revision: 360fe71b1bd05bb89814bfb97b73a5bda84802d6
23:08:22:I1: Branch: master
23:08:22:I1: Compiler: Visual C++
23:08:22:I1: Options: $( /TP $) /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2
23:08:22:I1: /Zc:throwingNew /MT
23:08:22:I1: Platform: win32 10
23:08:22:I1: Bits: 64
23:08:22:I1: Mode: Release
23:08:22:I1:****************************** CBang ******************************
23:08:22:I1: Version: 1.7.2
23:08:22:I1: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
23:08:22:I1: Org: Cauldron Development
23:08:22:I1: Copyright: Cauldron Development, 2003-2024
23:08:22:I1: Homepage: https://cauldrondevelopment.com/
23:08:22:I1: License: LGPL-2.1-or-later
23:08:22:I1: Date: Nov 20 2024
23:08:22:I1: Time: 09:04:40
23:08:22:I1: Revision: 443c54e909eb8d8994405a18fb328b5b05a623a5
23:08:22:I1: Branch: master
23:08:22:I1: Compiler: Visual C++
23:08:22:I1: Options: $( /TP $) /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2
23:08:22:I1: /Zc:throwingNew /MT
23:08:22:I1: Platform: win32 10
23:08:22:I1: Bits: 64
23:08:22:I1: Mode: Release
23:08:22:I1:***************************** System ******************************
23:08:22:I1: CPU: 13th Gen Intel(R) Core(TM) i5-13500
23:08:22:I1: CPU ID: GenuineIntel Family 6 Model 191 Stepping 2
23:08:22:I1: CPUs: 20
23:08:22:I1: Memory: 15.77GiB
23:08:22:I1:Free Memory: 10.59GiB
23:08:22:I1: OS Version: 6.2
23:08:22:I1:Has Battery: false
23:08:22:I1: On Battery: false
23:08:22:I1: Hostname: DESKTOP-3RC0LJ7
23:08:22:I1: UTC Offset: -4
23:08:22:I1: PID: 17472
23:08:22:I1: CWD: C:\ProgramData\FAHClient
23:08:22:I1: Exec: C:\Program Files\FAHClient\FAHClient.exe
23:08:22:I1:*******************************************************************
23:08:22:I2:<config/>
23:08:22:I1:Opening Database
23:08:22:I1:F@H ID = 0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc
23:08:22:I3:Loading default resource group
23:08:22:I1:Listening for HTTP on 127.0.0.1:7396
23:08:22:I3:WU16:Loading work unit 16 with ID jZ_3IreNyFLn54aX7X631c2cniW0hTjuQOjcQLqJiSw
23:08:22:I3:Loaded 1 wus.
23:08:22:I1:Started Windows systray control
23:08:22:I3:gpus = {
23:08:22:I3: "gpu:00:02:00": {
23:08:22:I3: "vendor": 32902,
23:08:22:I3: "type": "intel",
23:08:22:I3: "description": "Intel(R) UHD Graphics 770",
23:08:22:I3: "uuid": "86808046-0c00-0000-0002-000000000000",
23:08:22:I3: "opencl": {"platform": 0, "device": 0, "compute": "3.0", "driver": "31.0"},
23:08:22:I3: "device": 18048,
23:08:22:I3: "supported": false
23:08:22:I3: }
23:08:22:I3:}
23:08:22:I1:OUT2:> GET https://api.foldingathome.org/gpus HTTP/1.1
23:08:22:I1:OUT1:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
23:08:23:I1:OUT1:< HTTP/1.1 200 HTTP_OK
23:08:23:I1:OUT2:< HTTP/1.1 200 HTTP_OK
23:08:23:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
23:08:23:I1:OUT3:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
23:08:23:I1:Logging into node account
23:12:22:I1:Machine state fold
23:12:22:I1:Loaded cores/fahcore-a8-win-64bit-avx2_256-0.0.12/FahCore_a8.exe
23:12:22:I3:Running FahCore: C:\ProgramData\FAHClient\cores/fahcore-a8-win-64bit-avx2_256-0.0.12/FahCore_a8.exe -dir jZ_3IreNyFLn54aX7X631c2cniW0hTjuQOjcQLqJiSw -suffix 01 -version 8.4.9 -lifeline 17472 -np 19
23:12:23:I3:WU16:Started FahCore on PID 7052
23:12:23:I1:WU16:*********************** Log Started 2025-05-24T23:12:23Z ***********************
23:12:23:I1:WU16:************************** Gromacs Folding@home Core ***************************
23:12:23:I1:WU16: Core: Gromacs
23:12:23:I1:WU16: Type: 0xa8
23:12:23:I1:WU16: Version: 0.0.12
23:12:23:I1:WU16: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
23:12:23:I1:WU16: Copyright: 2020 foldingathome.org
23:12:23:I1:WU16: Homepage: https://foldingathome.org/
23:12:23:I1:WU16: Date: Jan 16 2021
23:12:23:I1:WU16: Time: 12:29:40
23:12:23:I1:WU16: Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
23:12:23:I1:WU16: Branch: master
23:12:23:I1:WU16: Compiler: Visual C++ 2019 16.7
23:12:23:I1:WU16: Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
23:12:23:I1:WU16: Platform: win32 10
23:12:23:I1:WU16: Bits: 64
23:12:23:I1:WU16: Mode: Release
23:12:23:I1:WU16: SIMD: avx2_256
23:12:23:I1:WU16: OpenMP: ON
23:12:23:I1:WU16: CUDA: OFF
23:12:23:I1:WU16: Args: -dir jZ_3IreNyFLn54aX7X631c2cniW0hTjuQOjcQLqJiSw -suffix 01
23:12:23:I1:WU16: -version 8.4.9 -lifeline 17472 -np 19
23:12:23:I1:WU16:************************************ libFAH ************************************
23:12:23:I1:WU16: Date: Jan 16 2021
23:12:23:I1:WU16: Time: 11:24:13
23:12:23:I1:WU16: Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
23:12:23:I1:WU16: Branch: master
23:12:23:I1:WU16: Compiler: Visual C++ 2019 16.7
23:12:23:I1:WU16: Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
23:12:23:I1:WU16: Platform: win32 10
23:12:23:I1:WU16: Bits: 64
23:12:23:I1:WU16: Mode: Release
23:12:23:I1:WU16:************************************ CBang *************************************
23:12:23:I1:WU16: Date: Jan 16 2021
23:12:23:I1:WU16: Time: 11:23:53
23:12:23:I1:WU16: Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
23:12:23:I1:WU16: Branch: master
23:12:23:I1:WU16: Compiler: Visual C++ 2019 16.7
23:12:23:I1:WU16: Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
23:12:23:I1:WU16: Platform: win32 10
23:12:23:I1:WU16: Bits: 64
23:12:23:I1:WU16: Mode: Release
23:12:23:I1:WU16:************************************ System ************************************
23:12:23:I1:WU16: CPU: 13th Gen Intel(R) Core(TM) i5-13500
23:12:23:I1:WU16: CPU ID: GenuineIntel Family 6 Model 191 Stepping 2
23:12:23:I1:WU16: CPUs: 20
23:12:23:I1:WU16: Memory: 15.77GiB
23:12:23:I1:WU16:Free Memory: 9.00GiB
23:12:23:I1:WU16: Threads: WINDOWS_THREADS
23:12:23:I1:WU16: OS Version: 6.2
23:12:23:I1:WU16:Has Battery: false
23:12:23:I1:WU16: On Battery: false
23:12:23:I1:WU16: UTC Offset: -4
23:12:23:I1:WU16: PID: 7052
23:12:23:I1:WU16: CWD: C:\ProgramData\FAHClient\work
23:12:23:I1:WU16:********************************************************************************
23:12:23:I1:WU16:Project: 12474 (Run 19, Clone 12, Gen 65)
23:12:23:I1:WU16:Unit: 0x00000000000000000000000000000000
23:12:23:I1:WU16:Digital signatures verified
23:12:23:I1:WU16:Calling: mdrun -c frame65.gro -s frame65.tpr -x frame65.xtc -cpi state.cpt -cpt 5 -nt 19 -ntmpi 1
23:12:23:I1:WU16:Steps: first=162500000 total=165000000
23:12:26:I1:WU16:Completed 37882 out of 2500000 steps (1%)
23:28:21:I1:WU16:Completed 50000 out of 2500000 steps (2%)
23:57:43:I1:WU16:Completed 75000 out of 2500000 steps (3%)
00:26:52:I1:WU16:Completed 100000 out of 2500000 steps (4%)
00:55:56:I1:WU16:Completed 125000 out of 2500000 steps (5%)
01:26:00:I1:WU16:Completed 150000 out of 2500000 steps (6%)
01:55:06:I1:WU16:Completed 175000 out of 2500000 steps (7%)
02:24:12:I1:WU16:Completed 200000 out of 2500000 steps (8%)
02:53:16:I1:WU16:Completed 225000 out of 2500000 steps (9%)
03:22:20:I1:WU16:Completed 250000 out of 2500000 steps (10%)
03:51:27:I1:WU16:Completed 275000 out of 2500000 steps (11%)
04:20:30:I1:WU16:Completed 300000 out of 2500000 steps (12%)
04:49:32:I1:WU16:Completed 325000 out of 2500000 steps (13%)
05:18:34:I1:WU16:Completed 350000 out of 2500000 steps (14%)
05:47:35:I1:WU16:Completed 375000 out of 2500000 steps (15%)
06:16:40:I1:WU16:Completed 400000 out of 2500000 steps (16%)
06:45:42:I1:WU16:Completed 425000 out of 2500000 steps (17%)
07:14:43:I1:WU16:Completed 450000 out of 2500000 steps (18%)
07:43:41:I1:WU16:Completed 475000 out of 2500000 steps (19%)
08:12:41:I1:WU16:Completed 500000 out of 2500000 steps (20%)
08:41:42:I1:WU16:Completed 525000 out of 2500000 steps (21%)
09:10:44:I1:WU16:Completed 550000 out of 2500000 steps (22%)
09:39:46:I1:WU16:Completed 575000 out of 2500000 steps (23%)
10:08:46:I1:WU16:Completed 600000 out of 2500000 steps (24%)
10:37:46:I1:WU16:Completed 625000 out of 2500000 steps (25%)
11:06:46:I1:WU16:Completed 650000 out of 2500000 steps (26%)
11:35:48:I1:WU16:Completed 675000 out of 2500000 steps (27%)
12:04:50:I1:WU16:Completed 700000 out of 2500000 steps (28%)
12:33:53:I1:WU16:Completed 725000 out of 2500000 steps (29%)
13:02:55:I1:WU16:Completed 750000 out of 2500000 steps (30%)
13:31:58:I1:WU16:Completed 775000 out of 2500000 steps (31%)
14:01:01:I1:WU16:Completed 800000 out of 2500000 steps (32%)
14:30:07:I1:WU16:Completed 825000 out of 2500000 steps (33%)
14:59:19:I1:WU16:Completed 850000 out of 2500000 steps (34%)
15:28:25:I1:WU16:Completed 875000 out of 2500000 steps (35%)
15:57:34:I1:WU16:Completed 900000 out of 2500000 steps (36%)
16:26:42:I1:WU16:Completed 925000 out of 2500000 steps (37%)
16:55:49:I1:WU16:Completed 950000 out of 2500000 steps (38%)
17:24:56:I1:WU16:Completed 975000 out of 2500000 steps (39%)
17:54:05:I1:WU16:Completed 1000000 out of 2500000 steps (40%)
18:23:08:I1:WU16:Completed 1025000 out of 2500000 steps (41%)
18:52:14:I1:WU16:Completed 1050000 out of 2500000 steps (42%)
19:21:19:I1:WU16:Completed 1075000 out of 2500000 steps (43%)
19:50:24:I1:WU16:Completed 1100000 out of 2500000 steps (44%)
20:20:16:I1:WU16:Completed 1125000 out of 2500000 steps (45%)
20:49:59:I1:WU16:Completed 1150000 out of 2500000 steps (46%)
21:19:44:I1:WU16:Completed 1175000 out of 2500000 steps (47%)
21:45:04:I1:Account websocket closed: PROTOCOL msg=Failed to read header start
21:45:04:I1:OUT5:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:45:04:E :OUT5:Failed response: EOF
21:45:19:I1:OUT6:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:45:20:I1:OUT6:< HTTP/1.1 200 HTTP_OK
21:45:27:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:45:46:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:45:46:I1:OUT7:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:45:46:I1:OUT7:< HTTP/1.1 200 HTTP_OK
21:46:08:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:46:08:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:46:08:I1:OUT8:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:46:08:I1:OUT8:< HTTP/1.1 200 HTTP_OK
21:46:23:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:46:42:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:46:42:I1:OUT9:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:46:43:I1:OUT9:< HTTP/1.1 200 HTTP_OK
21:46:43:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:47:53:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:47:54:I1:OUT10:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:47:54:I1:OUT10:< HTTP/1.1 200 HTTP_OK
21:48:15:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:48:15:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:48:15:I1:OUT11:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:48:15:I1:OUT11:< HTTP/1.1 200 HTTP_OK
21:48:16:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:49:17:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:49:17:I1:OUT12:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:49:18:I1:OUT12:< HTTP/1.1 200 HTTP_OK
21:49:31:I1:WU16:Completed 1200000 out of 2500000 steps (48%)
21:49:33:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:50:22:I1:OUT3:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
21:50:22:I1:Logging into node account
21:51:22:I1:Account websocket closed: PROTOCOL msg=Failed to read header start
21:51:22:I1:OUT13:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:51:23:I1:OUT13:< HTTP/1.1 200 HTTP_OK
21:51:44:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:51:44:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:51:44:I1:OUT14:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:51:44:I1:OUT14:< HTTP/1.1 200 HTTP_OK
21:51:59:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:53:01:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:53:01:I1:OUT15:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:53:02:I1:OUT15:< HTTP/1.1 200 HTTP_OK
21:53:05:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:54:05:I1:OUT3:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
21:54:05:I1:Logging into node account
21:55:05:I1:Account websocket closed: PROTOCOL msg=Failed to read header start
21:55:05:I1:OUT16:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:55:06:I1:OUT16:< HTTP/1.1 200 HTTP_OK
21:55:06:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:56:07:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:56:07:I1:OUT17:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:56:07:I1:OUT17:< HTTP/1.1 200 HTTP_OK
21:56:07:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:57:08:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:57:08:I1:OUT18:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:57:09:I1:OUT18:< HTTP/1.1 200 HTTP_OK
21:57:24:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:57:45:I1:OUT3:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
21:57:45:I1:Logging into node account
21:58:45:I1:Account websocket closed: PROTOCOL msg=Failed to read header start
21:58:45:I1:OUT19:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:58:47:I1:OUT19:< HTTP/1.1 200 HTTP_OK
21:59:08:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:59:08:I1:Account websocket closed: NONE msg=Connection failed: EOF
21:59:08:I1:OUT20:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
21:59:09:I1:OUT20:< HTTP/1.1 200 HTTP_OK
21:59:09:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
22:00:10:I1:Account websocket closed: NONE msg=Connection failed: EOF
22:00:10:I1:OUT21:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
22:00:11:I1:OUT21:< HTTP/1.1 200 HTTP_OK
22:00:11:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
22:01:12:I1:Account websocket closed: NONE msg=Connection failed: EOF
22:01:12:I1:OUT22:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
22:01:13:I1:OUT22:< HTTP/1.1 200 HTTP_OK
22:01:13:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
22:01:23:I1:OUT3:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
22:01:23:I1:Logging into node account
22:02:23:I1:Account websocket closed: PROTOCOL msg=Failed to read header start
22:02:23:I1:OUT23:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
22:02:23:I1:OUT23:< HTTP/1.1 200 HTTP_OK
22:02:45:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
22:02:45:I1:Account websocket closed: NONE msg=Connection failed: EOF
22:02:45:I1:OUT24:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
22:02:45:I1:OUT24:< HTTP/1.1 200 HTTP_OK
22:03:06:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
22:03:06:I1:Account websocket closed: NONE msg=Connection failed: EOF
22:03:06:I1:OUT25:> GET https://api.foldingathome.org/machine/0cdkpYr-oDiW2KAPy-DvQN9TZce9ArDLzlI0CSVCgZc HTTP/1.1
22:03:07:I1:OUT25:< HTTP/1.1 200 HTTP_OK
22:03:07:I1:OUT3:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
22:03:23:I1:OUT3:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
22:03:23:I1:Logging into node account
22:19:17:I1:WU16:Completed 1225000 out of 2500000 steps (49%)
22:49:49:I1:WU16:Completed 1250000 out of 2500000 steps (50%)
23:20:30:I1:WU16:Completed 1275000 out of 2500000 steps (51%)
23:51:27:I1:WU16:Completed 1300000 out of 2500000 steps (52%)
00:21:58:I1:WU16:Completed 1325000 out of 2500000 steps (53%)
01:09:03:I1:WU16:Completed 1350000 out of 2500000 steps (54%)
01:40:28:I1:WU16:Completed 1375000 out of 2500000 steps (55%)
02:11:53:I1:WU16:Completed 1400000 out of 2500000 steps (56%)
02:43:17:I1:WU16:Completed 1425000 out of 2500000 steps (57%)
03:13:35:I1:WU16:Completed 1450000 out of 2500000 steps (58%)
03:43:41:I1:WU16:Completed 1475000 out of 2500000 steps (59%)
04:13:43:I1:WU16:Completed 1500000 out of 2500000 steps (60%)
04:43:45:I1:WU16:Completed 1525000 out of 2500000 steps (61%)
05:13:47:I1:WU16:Completed 1550000 out of 2500000 steps (62%)
05:43:50:I1:WU16:Completed 1575000 out of 2500000 steps (63%)
06:13:54:I1:WU16:Completed 1600000 out of 2500000 steps (64%)
06:43:58:I1:WU16:Completed 1625000 out of 2500000 steps (65%)
07:14:07:I1:WU16:Completed 1650000 out of 2500000 steps (66%)
07:44:14:I1:WU16:Completed 1675000 out of 2500000 steps (67%)
08:14:21:I1:WU16:Completed 1700000 out of 2500000 steps (68%)
08:44:29:I1:WU16:Completed 1725000 out of 2500000 steps (69%)
09:14:40:I1:WU16:Completed 1750000 out of 2500000 steps (70%)
09:44:49:I1:WU16:Completed 1775000 out of 2500000 steps (71%)
10:15:15:I1:WU16:Completed 1800000 out of 2500000 steps (72%)
10:45:20:I1:WU16:Completed 1825000 out of 2500000 steps (73%)
11:15:22:I1:WU16:Completed 1850000 out of 2500000 steps (74%)
11:45:13:I1:WU16:Completed 1875000 out of 2500000 steps (75%)
12:15:22:I1:WU16:Completed 1900000 out of 2500000 steps (76%)
12:45:14:I1:WU16:Completed 1925000 out of 2500000 steps (77%)
13:15:07:I1:WU16:Completed 1950000 out of 2500000 steps (78%)
13:44:59:I1:WU16:Completed 1975000 out of 2500000 steps (79%)
14:15:27:I1:WU16:Completed 2000000 out of 2500000 steps (80%)
14:46:41:I1:WU16:Completed 2025000 out of 2500000 steps (81%)
15:17:54:I1:WU16:Completed 2050000 out of 2500000 steps (82%)
15:49:07:I1:WU16:Completed 2075000 out of 2500000 steps (83%)
16:20:24:I1:WU16:Completed 2100000 out of 2500000 steps (84%)
16:51:44:I1:WU16:Completed 2125000 out of 2500000 steps (85%)

[calxalot]

You need to not use e-cores.
Set cpus no higher than p-core threads.
Use Process Lasso to pin the FahCore_* to p-cores.

Make sure your system clock is correct.
Do not have spaces in the machine name.

[toTOW]

I am trying to understand why the i5-13500 system, which has a benchmark rating 2.5 times higher than the i3-12100 and 5 times the i5-6600K, is considerably underperforming both.

big.LITTLE shitty architecture.

Set the number of folding threads to match with you P core threads and use Process Lasso to pin the FahCore_* to P cores automatically. And next time you build a computer, think about AMD CPUs.

[arisu]

I think GROMACS may improve their load balancing in the future to support heterogeneous core architectures. The current problem is that the load balancer cannot cope with such drastic differences in core performance, so the E-cores will all be running at max speed while the P-cores spend much of their time idle, just waiting on the E-cores to complete their task. It ends up with less work being done than if the E-cores just stayed out of the way and let the P-cores do their thing alone.

If your computer has sufficient cooling so that the P-cores won't thermally throttle too badly if the E-cores are all in use, you can set up an extra resource group so that you get two WUs at once, and use Process Lasso to bind one WU to all the P-cores and one WU to all the E-cores. That will let you use every core.

I am trying to understand why the i5-13500 system, which has a benchmark rating 2.5 times higher than the i3-12100 and 5 times the i5-6600K, is considerably underperforming both.

big.LITTLE shitty architecture.

Set the number of folding threads to match with you P core threads and use Process Lasso to pin the FahCore_* to P cores automatically. And next time you build a computer, think about AMD CPUs.

Sadly, there are rumors that AMD is going to go the direction of heterogeneous cores as well for Zen 7, even for desktop processors.

[muziqaz]

Rumours are just musings of some delusional "leakers".
AMD already has "heterogenous" arch in place. With compact cores. Intel as always does brute force, while AMD manages to do things properly.
There is no indication Gromacs will support Intel's shittle design. Even if they do, it will be a while till FAH rebuilds the CPU core to support that.
Current AMD CPUs work fine on FAH

[arisu]

I thought there was an effort in GROMACS to improve the load balancer so that it could distribute load effectively even among a mix of E- and P-cores. I could be mistaken.

But you're right, FAH would take a while before it would adopt that change regardless of whether/when it happens. The best thing now would be for the client to launch the GROMACS cores with CPU affinity set to restrict it to the P-cores. My fork of the client does that, but only for Linux because I do not have a Windows system to test it on.

[Joe_H]

Yes, there was an effort to do that, but it didn't work very well when they used it in the development of Core_A8. The developer ended up choosing an alternate method of thread synchronization that worked better. Part of the reason was Core_A8 was also going to support ARM processors with big-little architecture. But that is still not very good, just not as bad as the original method they intended to use.

From what I gather Core_A9 follows essentially the same choices, just uses a later GROMACS version. The biggest difficulty is that the core would have to dynamically resize the partitions used to assign parts of the system to each thread so that processing on an E-core would complete in about the same time as on a P-core.

[arisu]

Yes, I don't see anything in the git log between the GROMACS versions used by a8 and a9 that would indicate any major changes in the load balancer or decomposition.

The hybrid architecture is getting more common, so something has to be done. But I suspect it will be done on the client by hiding E-cores, because that would be easier and would work without releasing an entirely new folding core.

[foxpy]

The biggest issue in my eyes is that GROMACS threads for some reason just sit in infinite busy poll and wait until all threads finish their slice of work, instead of yielding and waiting on e.g. a mutex. This design choice denies slower threads a chance to access a free/quicker CPU and finish job faster, because the other CPU is already busy with useless polling.

My guess is that mutex isn't as fast as busy wait in ideal scenario where all CPUs are equally fast and there is no interference from other stuff executing on the same system, but still it feels like a poor design decision. It is literally considered as an anti pattern to implement spinlocks in userspace, and yet they have done exactly that.

[arisu]

They don't sit in a busy loop. If you artificially throttle one of the threads, the other threads will slow down to around its speed. This is most apparent in natural situations when you have a single GPU thread running at 100%, and its hyperthreading twin, running GROMACS, will be using only 80% CPU because it is faster than the other GROMACS threads (because it is not sharing the core with a thread that makes heavy use of SIMD, just a GPU thread that does a dumb busy wait). But all the other threads running GROMACS will be using 100% CPU because they are "slower". The thread waiting on the slower threads uses mutexes for synchronization and so will only use around 80%.

You can also see it if try to fold on both P-cores and E-cores. The threads on the P-cores will be heavily throttled as they wait on the slower E-cores. The throttling is caused by it waiting on a synchronization mutex to be released.

On Linux, it uses a type of mutex called a futex ("fast userspace mutex"). Tracing the syscalls shows it using it for synchronization. The trace, showing it use mutexes for synchronization, looks like this:

Code: Select all

283061 futex(0x6645b29831c4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
283060 <... futex resumed>)             = -1 ETIMEDOUT (Connection timed out)
283061 futex(0x6730fab42084, FUTEX_WAKE_PRIVATE, 2147483647 <unfinished ...>
283060 futex(0x65d7d7aa4f40, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
283061 <... futex resumed>)             = 0
283060 <... futex resumed>)             = 0
283061 futex(0x6645b29831c4, FUTEX_WAKE_PRIVATE, 2147483647 <unfinished ...>
283060 futex(0x65d7d7aa406c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1748438663, tv_nsec=82242965}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
283061 <... futex resumed>)             = 0
283061 futex(0x6730fab42084, FUTEX_WAKE_PRIVATE, 2147483647) = 0
283061 futex(0x6645b29831c4, FUTEX_WAKE_PRIVATE, 2147483647) = 0

[foxpy]

Well, it definitely uses a mutex under the hood for something, but the main question I have: if the quicker threads slow down (which they definitely do, I agree), why do they still run 100% of the time instead of sleeping on a mutex?

[arisu]

They don't run 100% of the time. It yields each time that futex() is called, so the quicker threads aren't using 100% of the CPU if the slower threads can't keep up.

[foxpy]

Sadly I don't have a Linux machine to test it right now, but try running a WU with 4 threads, and pin them in the following configuration:
thread 0 - cpu 0
thread 1 - cpu 1
thread 2 - cpu 2
thread 3 - cpu 2

You will quickly notice that threads 0 and 1 still keep using 100% of CPU time, although common sense says they should only consume around ~50%

[toTOW]

I thought there was an effort in GROMACS to improve the load balancer so that it could distribute load effectively even among a mix of E- and P-cores.

The issue is not with Gromacs but with OS scheduler (mostly Windows one) that will assign all threads to E core because they run at low priority ...

And yes, allowing FAH to handle all combinations automatically will be a pain.

[arisu]

Sadly I don't have a Linux machine to test it right now, but try running a WU with 4 threads, and pin them in the following configuration:
thread 0 - cpu 0
thread 1 - cpu 1
thread 2 - cpu 2
thread 3 - cpu 2

You will quickly notice that threads 0 and 1 still keep using 100% of CPU time, although common sense says they should only consume around ~50%

When I pin all threads but one to a single core, the remaining thread drops to around 70%. I believe that is because it exceeds the limit of the load balancer. I'll check the source code to make sure, but I don't think it is using spinlocks.