Error: No Matching Core on v8.1.11 client

[JL_678]

I am using the beta and just installed it on a second machine which worked fine. The first two units are reasonably powerful (11th gen i5), and now I am considering running on FAH other systems. Specifically the following:

• Celeron J3160

• Celeron 3215U

• Celeron N3450

I have installed the 8.1.11 client on the 3215U and N3450 and constantly get the above-referenced error. Here is the exact log entry:

Code: Select all

19:19:10:E ::WU15:HTTP_SERVICE_UNAVAILABLE: {"error":"No matching core"}

Previously, I found that I needed at least 2 cores or I saw this message and so in all cases the client (an LXC container) is configured with 2 cores and sometimes more. However, I keep getting the same error.

Am I doing something wrong? Could this be related to the fact that these are older and less powerful CPUs?

TIA for any thoughts.

[Joe_H]

The message could be indirectly connected with the Celeron CPUs, possibly indicating that the proper identification of resources was not made by the client and transmitted to the servers. Or the project could be misconfigured on the server, and the WU is calling for a nonexistent version of the folding core.

Information on the WU would be needed to see which core it is supposed to be using. At this point CPU projects are mostly using Core_A8, but some are using Core_A9. I know Core_A8 has folding cores for both SSE enabled processors and those with AVX. I don't recall if there is a SSE core for the projects that use the A9 core. These two Celerons only support up to SSE4.2, no AVX.

A project on the server can be set to require a specific revision of a folding core. Sometimes this is a minimum, and a higher level version is okay. But there can also be typos in the setup, and that has happened in the past where Was from a project were looking to download a revision that did not exist. Not quite certain where to see which core is being requested with v8, it was explicitly shown in the log with the v7 client.

[calxalot]

I don't know what the released A9 supports, but future builds currently are for sse4_1, avx_256, avx2_256.

There is also the matter of which cores have been validated and deployed for v8 use.

[calxalot]

To see the assignment request in the log, I think you need to run the debug build at verbosity 3.

[JL_678]

I don't know what the released A9 supports, but future builds currently are for sse4_1, avx_256, avx2_256.

There is also the matter of which cores have been validated and deployed for v8 use.

Does this mean that to use A9 a CPU must have one of these things or all of them? This raises a bigger question about whether I should even bother with Celeron CPUs. I would love to support the project with them and they have free cycles that could be used; however; if Celeron's are not really supported (or are supported in a very limited way) then maybe I shouldn't bother.

I don't understand the project architecture fully, but it seems a shame to exclude older CPUs like these.

[Joe_H]

No, that means there are separate versions of the core for SSE 4.1, AVX-256, and AVX2-256. The client is supposed to detect the correct CPU and availability of hardware support for these levels of SIMD calculations and download the correct core. For example the older A8 core still has support for SSE 2, and can be used with even older processors.

But until we know which folding core is being requested it is not possible to determine what is going wrong here. I would have to look through old v7 logs to find an example of a core download request, but can provide an example of how much is defined at run time:

Code: Select all

 08:50:40:WU00:FS00:Running FahCore: /usr/local/bin/FAHCoreWrapper "/Library/Application Support/FAHClient/cores/cores.foldingathome.org/osx/64bit-avx-256/a8-0.0.12/Core_a8.fah/

On this system there is support for AVX-256, so that version of the core is being used. It also specifies revision 0.0.12, if there was a project that still used the older version it might specify 0.0.11. A typo on the server supplying the WU might result in a request for a non-existent 0.0.13 revision of the A8 core.

[JL_678]

Thank you for the clarification. I will install the debug version of v8 client and switch to verbosity 3 and see if I can find and post that information.

[JL_678]

I moved to the debug version and do not see that. I then upped the debug verbosity to 10. I let a few WU searches happen and grep'd the resulting log for "avx" or "sse" and found nothing. For good measure, I also searched for "a8" and "a9" and neither of those showed up either. The closest thing I saw was this:

Code: Select all

15:02:27:I1::WU21:Requesting WU assignment
15:02:27:D3::WU21:{
15:02:27:D3:  "time": "2023-01-29T15:02:27Z",
15:02:27:D3:  "wu": 21,
15:02:27:D3:  "version": "8.1.11",
15:02:27:D3:  "id": "Gtc7hTQ2N2TQsKnUTTV+zYxDsRIfTmBUIVrhZ7bPsWI=",
15:02:27:D3:  "user": "<uname>",
15:02:27:D3:  "team": 229500,
15:02:27:D3:  "passkey": "<passkey>",
15:02:27:D3:  "os": {"version": "5.15", "type": "linux", "memory": 3398156288},
15:02:27:D3:  "project": {"cause": "any"},
15:02:27:D3:  "resources": {
15:02:27:D3:    "cpu": {"cpu": "amd64", "cpus": 2, "vendor": "GenuineIntel", "signature": 263876, "family": 6, "model": 76, "stepping": 4, "features": 4888907808208845823, "ext
ended": 8834, "80000001": 1104478734336}
15:02:27:D3:  }
15:02:27:D3:}
15:02:27:D4:REQ0:created
15:02:27:D5:OUT0:Connecting to assign2.foldingathome.org:443
15:02:27:D4:FD-1:cb::Event::FD::FD() fd=-1
15:02:27:D4:CON5:Connection opened

[Joe_H]

Just a historical note, older versions of Gromacs on which the CPU folding cores are based used to include support for conditional execution of code paths. So a single executable could be created that for example supported both SSE enabled processors and those which did not have SSE.

When conditional execution was removed from Gromacs, with the existing client they could either create an executable that supported the lowest common denominator or start dropping support for older CPUs. F@h decided to support the newer levels of SSE and then AVX by adding support in the client to identify and download different versions for selected types of SSE or AVX.

Not all variations of SSE and AVX have dedicated executables, but they can run on CPUs that have later levels of support. For example a CPU with SSE 3 support can use a SSE 2 executable with little loss of performance compared to a dedicated SSE 3 core.

[JL_678]

Thank you for the clarification and history. A couple of questions to consider:

• Where is the core download request attribute in the beta build? It would appear to be missing if it is not in the debug build with verbosity 10.

• Could the issue I am seeing be related to a bug?

Out of curiousity, I am going to install a v7 client on this same host and see what happens.

[JL_678]

Update: I installed v7 on the SAME client that is given the "No cores found" error in v8, and it is happily folding away. I will open a Github ticket as this is definitely a v8 bug.

For future reference is here a direct link to the GitHub ticket: https://github.com/FoldingAtHome/fah-cl ... issues/103

[calxalot]

Which core is running for v7?

[JL_678]

I just looked at this and cannot find the entry in the log mentioned above. Maybe because verbosity was set at 1? I upped it to 5 and restarted the service and still don't see where I can find the core. Is it somewhere else? The system is still folding away....

[JL_678]

Update, this was just populated in the log. Here is the entry.

Code: Select all

21:38:57:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-sse2/a8-0.0.12/Core_a8.fah/FahCore_a8 -dir 00 -suffix 01 -version 704 -lifeline 1845 -checkpoint 15

[Joe_H]

Are you asking where to find it in the v7 or the v8 log entries? For v7 it will be a few lines before the WU is started, assuming you are using the log created by the client.

Here is an example from one of my Mac's:

Code: Select all

07:09:50:WU00:FS00:Starting
07:09:50:WU00:FS00:Running FahCore: /usr/local/bin/FAHCoreWrapper "/Library/Application Support/FAHClient/cores/cores.foldingathome.org/osx/64bit-avx-256/a8-0.0.12/Core_a8.fah/FahCore_a8" -dir 00 -suffix 01 -version 706 -lifeline 71 -checkpoint 15 -np 2
07:09:50:WU00:FS00:Started FahCore on PID 97206
07:09:54:WU00:FS00:Core PID:97207
07:09:54:WU00:FS00:FahCore 0xa8 started
07:10:01:WU00:FS00:0xa8:*********************** Log Started 2023-01-29T07:10:01Z ***********************
07:10:01:WU00:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
07:10:01:WU00:FS00:0xa8:       Core: Gromacs
07:10:01:WU00:FS00:0xa8:       Type: 0xa8
07:10:01:WU00:FS00:0xa8:    Version: 0.0.12
07:10:01:WU00:FS00:0xa8:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:10:01:WU00:FS00:0xa8:  Copyright: 2020 foldingathome.org
07:10:01:WU00:FS00:0xa8:   Homepage: https://foldingathome.org/
07:10:01:WU00:FS00:0xa8:       Date: Jan 16 2021
07:10:01:WU00:FS00:0xa8:       Time: 21:24:14
07:10:01:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
07:10:01:WU00:FS00:0xa8:     Branch: master
07:10:01:WU00:FS00:0xa8:   Compiler: GNU Apple LLVM 12.0.0 (clang-1200.0.32.28)
07:10:01:WU00:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -stdlib=libc++ -O3
07:10:01:WU00:FS00:0xa8:             -funroll-loops -fno-pie -mmacosx-version-min=10.7
07:10:01:WU00:FS00:0xa8:   Platform: darwin 20.2.0
07:10:01:WU00:FS00:0xa8:       Bits: 64
07:10:01:WU00:FS00:0xa8:       Mode: Release
07:10:01:WU00:FS00:0xa8:       SIMD: avx_256
07:10:01:WU00:FS00:0xa8:     OpenMP: ON
07:10:01:WU00:FS00:0xa8:       CUDA: OFF
07:10:01:WU00:FS00:0xa8:       Args: -dir 00 -suffix 01 -version 706 -lifeline 97206 -checkpoint 15 -np
07:10:01:WU00:FS00:0xa8:             2
07:10:01:WU00:FS00:0xa8:************************************ libFAH ************************************
07:10:01:WU00:FS00:0xa8:       Date: Jan 16 2021
07:10:01:WU00:FS00:0xa8:       Time: 21:21:45
07:10:01:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
07:10:01:WU00:FS00:0xa8:     Branch: master
07:10:01:WU00:FS00:0xa8:   Compiler: GNU Apple LLVM 12.0.0 (clang-1200.0.32.28)
07:10:01:WU00:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -stdlib=libc++ -O3
07:10:01:WU00:FS00:0xa8:             -funroll-loops -fno-pie -mmacosx-version-min=10.7
07:10:01:WU00:FS00:0xa8:   Platform: darwin 20.2.0
07:10:01:WU00:FS00:0xa8:       Bits: 64
07:10:01:WU00:FS00:0xa8:       Mode: Release
07:10:01:WU00:FS00:0xa8:************************************ CBang *************************************
07:10:01:WU00:FS00:0xa8:       Date: Jan 16 2021
07:10:01:WU00:FS00:0xa8:       Time: 21:21:02
07:10:01:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
07:10:01:WU00:FS00:0xa8:     Branch: master
07:10:01:WU00:FS00:0xa8:   Compiler: GNU Apple LLVM 12.0.0 (clang-1200.0.32.28)
07:10:01:WU00:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -stdlib=libc++ -O3
07:10:01:WU00:FS00:0xa8:             -funroll-loops -fno-pie -mmacosx-version-min=10.7 -fPIC
07:10:01:WU00:FS00:0xa8:   Platform: darwin 20.2.0
07:10:01:WU00:FS00:0xa8:       Bits: 64
07:10:01:WU00:FS00:0xa8:       Mode: Release
07:10:01:WU00:FS00:0xa8:************************************ System ************************************
07:10:01:WU00:FS00:0xa8:        CPU: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
07:10:01:WU00:FS00:0xa8:     CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
07:10:01:WU00:FS00:0xa8:       CPUs: 4
07:10:01:WU00:FS00:0xa8:     Memory: 16.00GiB
07:10:01:WU00:FS00:0xa8:Free Memory: 9.03GiB
07:10:01:WU00:FS00:0xa8:    Threads: POSIX_THREADS
07:10:01:WU00:FS00:0xa8: OS Version: 10.14
07:10:01:WU00:FS00:0xa8:Has Battery: true
07:10:01:WU00:FS00:0xa8: On Battery: false
07:10:01:WU00:FS00:0xa8: UTC Offset: -5
07:10:01:WU00:FS00:0xa8:        PID: 97207
07:10:01:WU00:FS00:0xa8:        CWD: /Library/Application Support/FAHClient/work
07:10:01:WU00:FS00:0xa8:********************************************************************************
07:10:01:WU00:FS00:0xa8:Project: 16977 (Run 1, Clone 1883, Gen 302)
07:10:01:WU00:FS00:0xa8:Unit: 0x00000000000000000000000000000000
07:10:01:WU00:FS00:0xa8:Reading tar file core.xml
07:10:01:WU00:FS00:0xa8:Reading tar file frame302.tpr
07:10:01:WU00:FS00:0xa8:Digital signatures verified
07:10:01:WU00:FS00:0xa8:Calling: mdrun -c frame302.gro -s frame302.tpr -x frame302.xtc -cpt 15 -nt 2 -ntmpi 1
07:10:01:WU00:FS00:0xa8:Steps: first=755000000 total=757500000
07:10:03:WU00:FS00:0xa8:Completed 1 out of 2500000 steps (0%)

This is for a WU that was downloaded just before the "Starting" message and the core was downloaded sometime in the past. The next line is the command used by the client to launch the folding core inside the Wrapper process, it includes the full path to the core executable which includes the core type and revision number. The beginning of the processing log after that also includes the core identification:

Code: Select all

07:10:01:WU00:FS00:0xa8:       Core: Gromacs
07:10:01:WU00:FS00:0xa8:       Type: 0xa8
07:10:01:WU00:FS00:0xa8:    Version: 0.0.12
...
07:10:01:WU00:FS00:0xa8:       SIMD: avx_256

From an older log when I was doing testing of an early version of the A9 core, here is the first WU download and core download:

Code: Select all

18:15:09:WU00:FS00:Requesting new work unit for slot 00: cpu:2 from 130.237.11.145
18:15:09:WU00:FS00:Connecting to 130.237.11.145:8080
18:15:09:WU00:FS00:Downloading 8.01MiB
18:15:15:WU00:FS00:Download 94.44%
18:15:15:WU00:FS00:Download complete
18:15:15:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16819 run:0 clone:5 gen:0 core:0xa9 unit:0x0000000500000000000041b300000000
18:15:15:WU00:FS00:Downloading core from http://cores.foldingathome.org/gromacs-core-a9/macos-12-64bit/cpu-avx_256-release/0xa9-0.0.5/Core_a9.fah
18:15:15:WU00:FS00:Connecting to cores.foldingathome.org:80
18:15:16:WU00:FS00:FahCore a9: Downloading 8.27MiB
18:15:17:WU00:FS00:FahCore a9: Download complete
18:15:17:WU00:FS00:Valid core signature
18:15:18:WU00:FS00:Unpacked 25.63MiB to cores/cores.foldingathome.org/gromacs-core-a9/macos-12-64bit/cpu-avx_256-release/0xa9-0.0.5/Core_a9.fah/FahCore_a9
18:15:18:WU00:FS00:Unpacked 2.66KiB to cores/cores.foldingathome.org/gromacs-core-a9/macos-12-64bit/cpu-avx_256-release/0xa9-0.0.5/Core_a9.fah/nbnxm_ocl_kernels.cl
18:15:18:WU00:FS00:Unpacked 7.71KiB to cores/cores.foldingathome.org/gromacs-core-a9/macos-12-64bit/cpu-avx_256-release/0xa9-0.0.5/Core_a9.fah/pme_program.cl
18:15:18:WU00:FS00:Starting
18:15:18:WU00:FS00:Running FahCore: /usr/local/bin/FAHCoreWrapper "/Library/Application Support/FAHClient/cores/cores.foldingathome.org/gromacs-core-a9/macos-12-64bit/cpu-avx_256-release/0xa9-0.0.5/Core_a9.fah/FahCore_a9" -dir 00 -suffix 01 -version 706 -lifeline 71 -checkpoint 15 -np 2
18:15:18:WU00:FS00:Started FahCore on PID 21458
18:15:19:WU00:FS00:Core PID:21459
18:15:19:WU00:FS00:FahCore 0xa9 started
18:15:19:WU00:FS00:0xa9:*********************** Log Started 2022-07-22T18:15:19Z ***********************
18:15:19:WU00:FS00:0xa9:************************** Gromacs Folding@home Core ***************************
18:15:19:WU00:FS00:0xa9:       Core: Gromacs
18:15:19:WU00:FS00:0xa9:       Type: 0xa9
18:15:19:WU00:FS00:0xa9:    Version: 0.0.5
18:15:19:WU00:FS00:0xa9:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:15:19:WU00:FS00:0xa9:  Copyright: 2022 foldingathome.org
18:15:19:WU00:FS00:0xa9:   Homepage: https://foldingathome.org/

There is a slightly different sequence if a version of a core is already present, but not the required version. But the downloading and unpacking is pretty much the same. Some of the details will be slightly different on Windows or Linux, but the basics remain the same.

The default verbosity for logging on v7 is 3, 5 mainly shows all of the default settings that are otherwise not included in the log.