Pause/Resume gpu core 17 takes 2.5min with high CPU usage

[foldy]

When i pause the client and resume again, then i realize a high cpu usage with gpu core 17 workunits for about 2.5 min (
only one cpu core is used). This was not the case some weeks ago, where on resume high cpu usage was only for some seconds.

[7im]

It may be the updated core 17. Or it may be which frame number you resumed from. Some core 17 WUs write check points every 2 frames, some every 5.

[bruce]

On Linux, the OpenCL drivers reportedly take a full CPU constantly. On Windows, it's only for a few minutes every checkpoint. This seems to be the way AMD designed their new drivers.

Is it possible that you're not comparing the CPU utilization of Core_17 with your memory of previous drivers and/or previous FahCores?

[7im]

To answer your next question, no, it's not broken.

[foldy]

OK, i checked previous core_17 version and previous driver and it is the same behavior.
So this is how it works and i had a false memory...

[prjindigo]

Aye, hash and maths check to verify it will run the same way in a predictable manner.

[Eddy]

OpenCL needs a "GPU kernel" to be compiled before it can run, and it takes that CPU time.
I think it's can be optimized because it has to be compiled just 1 time per GPU, and I have seen openCL softwares that saves and re-use the same kernel without recompiling.

[Jim Saunders]

It does that for me too if that helps.

Jim

[7im]

OpenCL needs a "GPU kernel" to be compiled before it can run, and it takes that CPU time.
I think it's can be optimized because it has to be compiled just 1 time per GPU, and I have seen openCL softwares that saves and re-use the same kernel without recompiling.

Have you confirmed this compiling in Fah GPU? Or is this just a guess based on other programs?

[bruce]

Core_17 on Windows does run with high CPU utilization when start a WU. Then it runs with low CPU utilization until it reached the next checkpoint (commonly two 1% frames, but it depends on the project). Then I see high CPU utilization for a period of time again. On linux, it uses one CPU core continuously with no intervening periods of low CPU utilization

Without detailed knowledge of the actual code internal to FahCore_17.exe, allocating the Windows utilization between compiling a kernel, searching the data for inconsistencies, converting/preparing data from one form to another (such a generating data for the viewer) os any number of other possibilities is virtually impossible. I might assume one thing and you might assume something else and neither of us would have the facts required to convince the other of the accuracy of our guess.

I have asked Proteneer (the Developer) if anything can be done to reduce this CPU processing without compromising the results. Or maybe there's some CPU-based code that can be moved to the GPU efficiently. I'm sure they'll take another look at it, but based on external symptoms we can see, there's no assurance that we will notice any changes.

Over the course of a long WU, it seems that the CPU portion of the analysis takes about 5%. of the time (During about 20 hours on my system, 1 CPU hour, which is a lot more than the 2.5 min that you're seeing just at the beginning.)

[Eddy]

Have you confirmed this compiling in Fah GPU? Or is this just a guess based on other programs?

It's just a guess, I've seen programs "compiling" a kernel the first time for looong minutes, then saving it in a cache file (named against gpu core and drivers or something) and never doing that again at least until configuration or parameter changes.

On Linux, the OpenCL drivers reportedly take a full CPU constantly.

I think (by my linux experience) that it depends on how or which program do you use to check CPU usage. Linux divides CPU percentage in system time (kernel and drivers - gpu drivers too), user time (user programs) I/O wait (time spent waiting for I/O), sometimes also Soft and Hard interrupts time. Take a look at "man vmstat" and ran "vmstat 1" to see which one is it calculating. IO wait is not real cpu calculation, nor it spend power, it's really just waiting, I guess that it's your case.

I have asked Proteneer (the Developer) if anything can be done to reduce this CPU processing without compromising the results. Or maybe there's some CPU-based code that can be moved to the GPU efficiently. I'm sure they'll take another look at it, but based on external symptoms we can see, there's no assurance that we will notice any changes.

Please ask specially for the OpenCL kernel caching thing, because if a 5% CPU time is acceptable above much time, It's not acceptable in a configuration where the gpu slot is set to start when idle, and paused when the computer is no longer idle.. it can became 100%,200% etc.. of cpu wasted time.
I'll try to figure out and remember which programs did this, but if I'm not wrong, one of them was blender cycles render ... I'm not sure though .. maybe I saw it on some OpenCL benchmarck or something...

[foldy]

In this post by user Napoleon he found a way to pause and resume the gpu core without the 2.5min high cpu usage.
viewtopic.php?f=14&t=26619

Using a process tool you can suspend the FahCore_17.exe process, which means FahControl still sees the gpu core running,
but it isn't, the GPU usage goes to 0%. I then can start a 3D game without conflict.

When i resume the FahCore_17.exe process using a process tool then immidiatly gpu usage goes to 99% again, no extra cpu usage is seen.
So it looks like the OpenCL kernel cache survived in the gpu.

Maybe this is a hint how a fast gpu core pause and resume could be implemented officially.
But still the question: Is it safe to suspend and resume the FahCore_17.exe process

[FaaR]

But still the question: Is it safe to suspend and resume the FahCore_17.exe process

It "should be*", with the caveat that since FaH doesn't know you've paused its GPU core it won't write any checkpoints during the time it is paused, meaning if you shut your PC or FaH down, you'll lose the work done since the last checkpoint save.

*Caveat: I have no idea if FaH periodically challenges its various cores running to see if they're healthy; if you suspend its process, it obviously can't respond to any challenge... FaH might, possibly (speculation) decide the core has crashed and proceed to shut it down/restart it... *shrug* Who knows.

[Napoleon]

*Caveat: I have no idea if FaH periodically challenges its various cores running to see if they're healthy; if you suspend its process, it obviously can't respond to any challenge... FaH might, possibly (speculation) decide the core has crashed and proceed to shut it down/restart it... *shrug* Who knows.

Maybe not, see https://fah.stanford.edu/projects/FAHClient/ticket/1059
But that's still just maybe...

[bruce]

If the PC sleeps, contents of video memory are lost while main ram is checkpointed and restored. Images on the screen can be refreshed but fah may hang or crash. The FahCore should be smart enough to restart from the previous ckpt but it's not yet that smart. The same problem happens if use MS rdt.