Receiving extra CPU WUs after a 503 "please wait" error

[muziqaz]

Finish with old WU, then continue with new one. And make sure to not have same device twice in two different resource groups

[arisu]

It looks like it is letting me set all 8 cores for the CPU. Would it be inadvisable for me to allow it 8/8 cores for the CPU? Because the iGPU is only using around 25% of a core to keep itself fed, which I guess would not be the case if it was much more powerful. Is this a case where "not reserving" a core for the GPU is not a bad idea?

[muziqaz]

GPUs need a CPU core regardless of how much they are using it

[arisu]

It looks like it doesn't, because it's letting me set all 8 cores for the CPU and "0" cores for the iGPU (although the iGPU is still using about 25% of a core of course, it's just spamming ioctls on the /dev/kfd character device which I assume is it feeding the iGPU), which was not possible when both CPU and iGPU were in the same resource group. It seems to be working fine, and now all 8 cores are saturated and it's not forcing the CPU WU to only use up to 7 cores.

[muziqaz]

GPU needs 1 core. I don't care how much it looks like it is using, o don't care what it is being used for, it needs a core. Trust me.
You will not lose much of the FAH performance if that 1 core is sitting being used by GPU 25%. However you will lose some performance on the GPU if it does not have spare core.
This will be more impactful once we move to HIP instead of OpenCL. And nVidia already works best when CPU is not even folding at all.

[arisu]

That's interesting, so you're saying that 7 cores at 100% and 1 core to feed the GPU at 25% gives better performance than all cores being shared? I would love to learn the technical reasons behind that. There's not much I can glean with just strace.

If I was to guess, would it be that the GPU's thread does work cannot be distributed among multiple cores, meaning that essentially the system would have 7 "full performance" cores and 1 "three quarters full performance" core? If that is the case then I can see why it would reduce performance in the same way that spreading one CPU WU among both E-cores and P-cores on Intel processors is bad. Or I might be completely wrong.

I have a background in CS and systems programming so I can handle technical details.

[Joe_H]

Is it true that prime numbers of cores is bad for performance, since I have 7 in use? I've heard that around but I can't imagine how that could possibly be.

It was true for older versions of the CPU folding core. After decomposing the WU into slices based on the number of CPU threads to be used, processing on each slice was done in one thread. Then there is a reconciliation pass to handle forces between atoms in adjacent slices. The code that was used for that tended to break down with higher number primes present in the number of threads used. Original testing found problems with decompositions with primes 11 and larger, 7 usually worked, except when it didn't. Eventually they ran into some small systems where 5 was identified as a problem.

Currently used CPU folding cores use a different method to reconcile the forces between slices or threads. That was chosen because the newer cores were also coded to be used on ARM where Big-little architecture would also have threads being processed at varying speeds. It didn't work as well as desired between threads running on slower efficiency cores and performance cores, but did mostly eliminate issues related to prime numbers being used in the WU decomposition. There can still be some drops in efficiency as more threads are used, and on smaller WU systems too many threads can actually run slower.

[muziqaz]

That's interesting, so you're saying that 7 cores at 100% and 1 core to feed the GPU at 25% gives better performance than all cores being shared? I would love to learn the technical reasons behind that. There's not much I can glean with just strace.

If I was to guess, would it be that the GPU's thread does work cannot be distributed among multiple cores, meaning that essentially the system would have 7 "full performance" cores and 1 "three quarters full performance" core? If that is the case then I can see why it would reduce performance in the same way that spreading one CPU WU among both E-cores and P-cores on Intel processors is bad. Or I might be completely wrong.

I have a background in CS and systems programming so I can handle technical details.

FAH in itself does not like competing compute processes. If you have 8 cores assigned to CPU work, and then start rendering on one of those cores, FAH performance will drop massively and not just by a margin of one lost core. If you select 7 cores in the beginning, then fahcore will not be bothered by whatever is happening on that 1 free core you left out because it expects 7 cores and it gets 7 cores.
This is just the way fahcore is built or actually how gromacs work.

[arisu]

FAH in itself does not like competing compute processes. If you have 8 cores assigned to CPU work, and then start rendering on one of those cores, FAH performance will drop massively and not just by a margin of one lost core. If you select 7 cores in the beginning, then fahcore will not be bothered by whatever is happening on that 1 free core you left out because it expects 7 cores and it gets 7 cores.
This is just the way fahcore is built or actually how gromacs work.

Is this because the process is only as fast as its slowest thread?

To go back to the original issue, the bug actually looks exactly like https://github.com/FoldingAtHome/fah-cl ... issues/144:

I think the problem now is that the client reserves one CPU for each GPU but the AS says it's not using that reserved CPU so the client requests a one CPU WU and gets it. The zero CPU assignment problem described above should be fixed.

Although it was closed saying it was fixed, but I am having the issue nonetheless. Any advice on fixing it that does not require setting up multiple resource groups? Using a different resource group for GPU and CPU has one big downside: When there is no GPU WU available, the number of CPU cores that the CPU resource group uses stays at 7. It does not increase itself to 8 like it does when both the CPU and GPU are in the same resource group.

[muziqaz]

You can post your issue in that GitHub issue with full evidence that this is actually happening.

And yes, the process is as fast as it's slowest thread