Dumping CPU work units when GPU WUs are completed - no fix worked

[Tropical Snowman]

Hello there, I've got a problem since several weeks.
When a GPU work units completes, sometimes the CPU work unit is dumped or appears as failed. A new one starts.

I'm on Ryzen 7 7700, 32gb ram DDR5, graphics RX 6700. Windows 11 IoT LTSC.

I've tried lowering CPU threads (currently using 12 of 16), adding FAH to DefenderUI exclusions, uninstalling DefenderUI and setting Windows Defender to default, reinstalling Visual C++, deleting cores folder... the problem persists. Only deleting cores folder fixed it for a couple days and I had no error.

Here's the log of the last error:

Code: Select all

01:59:01:I1:WU571:Completed 2475000 out of 2500000 steps (99%)
02:00:09:I1:WU573:Completed 1200000 out of 2500000 steps (48%)
02:00:51:I1:WU571:Completed 2500000 out of 2500000 steps (100%)
02:00:51:I1:WU571:Average performance: 39.095 ns/day
02:00:52:I1:WU571:Checkpoint completed at step 2500000
02:00:55:I1:WU571:Saving result file ..\logfile_01.txt
02:00:55:I1:WU571:Saving result file checkpointIntegrator.xml
02:00:55:I1:WU571:Saving result file checkpointState.xml.bz2
02:00:55:I1:WU571:Saving result file positions.xtc
02:00:55:I1:WU571:Saving result file science.log
02:00:55:I1:WU571:Saving result file xtcAtoms.csv.bz2
02:00:55:I1:WU571:Folding@home Core Shutdown: FINISHED_UNIT
02:00:55:I1:WU571:Core returned FINISHED_UNIT (100)
02:00:55:I1:Default:Added new work unit: cpus:0 gpus:gpu:03:00:00
02:00:55:I1:WU571:Uploading WU results
02:00:55:I1:WU574:Requesting WU assignment for user acwvfqvlj2ui4vcpalbdq5trcfsdtnqpwu5ypuip63uueh4zkvt2h34augfii team 1066107
02:00:55:I1:WU573:WARNING:Console control signal 1 on PID 3656
02:00:55:I1:WU573:Exiting, please wait. . .
02:00:56:I1:OUT13:> POST https://fah1.innovatr.ca/api/results HTTP/1.1
02:00:56:I1:OUT14:> POST https://assign4.foldingathome.org/api/assign HTTP/1.1
02:00:56:I1:WU573:Folding@home Core Shutdown: INTERRUPTED
02:00:56:I1:WU573:Core returned INTERRUPTED (102)
02:00:56:I3:Running FahCore: C:\ProgramData\FAHClient\cores/fahcore-a8-win-64bit-avx2_256-0.0.12/FahCore_a8.exe -dir NVieAQm711ou_6LhSiKtPvGX6Uy9c9PHXJhioeK5jh0 -suffix 01 -version 8.4.9 -lifeline 13384 -np 12
02:00:56:I3:WU573:Started FahCore on PID 4020
02:00:56:I1:OUT14:< HTTP/1.1 200 HTTP_OK
02:00:56:I1:WU574:Received WU assignment v4FfFHJpQPDOPhYDBL3HgZqMFBxg0SaJmAbLAHlscsQ
02:00:56:I1:WU574:Downloading WU
02:00:56:I1:OUT15:> POST https://fah1.innovatr.ca/api/assign HTTP/1.1
02:00:57:E :WU573:Core exited with Windows unhandled exception code 0xc000013a.  See https://bit.ly/2CXgWkZ for more information.
02:00:57:E :WU573:Core returned FAILED_1 (0)
02:00:57:E :WU573:The folding core did not produce any log output.  This indicates that the core is not functional on your system.  Check for missing libraries or GPU drivers.  Make a post about your issue on https://foldingforum.org/ to get more help.
02:00:57:E :WU573:Run did not produce any results. Dumping WU
02:00:57:I1:WU573:Sending dump report
02:00:58:I1:OUT16:> POST https://vav22.fah.temple.edu/api/results HTTP/1.1
02:00:59:I1:OUT16:< HTTP/1.1 200 HTTP_OK
02:00:59:I1:WU573:Dumped
02:01:01:I1:OUT13:< HTTP/1.1 200 HTTP_OK
02:01:01:I1:WU571:Credited
02:01:08:I1:OUT15:< HTTP/1.1 200 HTTP_OK
02:01:08:I1:WU574:Received WU P18265 R0 C108 G60
02:01:08:I1:Default:Added new work unit: cpus:11 gpus:
02:01:08:I3:Running FahCore: C:\ProgramData\FAHClient\cores/openmm-core-27/windows-10-64bit/release/fahcore-27-windows-10-64bit-release-8.2.1/FahCore_27.exe -dir v4FfFHJpQPDOPhYDBL3HgZqMFBxg0SaJmAbLAHlscsQ -suffix 01 -version 8.4.9 -lifeline 13384 -gpu-platform cuda -gpu-vendor amd -opencl-platform 0 -opencl-device 1 -gpu 1
02:01:08:I3:WU574:Started FahCore on PID 17496
02:01:08:I1:WU575:Requesting WU assignment for user

[muziqaz]

https://apps.foldingathome.org/wu#proje ... 108&gen=60

It seems it was counted as completed by you

[Tropical Snowman]

Project 18265 is the finished GPU work unit. The dumped one is the CPU assigned one, from project 16969.
Here's the log when it was assigned:

Code: Select all

00:37:40:I1:WU573:Requesting WU assignment for user acwvfqvlj2ui4vcpalbdq5trcfsdtnqpwu5ypuip63uueh4zkvt2h34augfii team 1066107
00:37:40:I1:OUT11:> POST https://assign3.foldingathome.org/api/assign HTTP/1.1
00:37:40:I1:OUT10:> POST https://vav17.fah.temple.edu/api/results HTTP/1.1
00:37:41:I1:OUT11:< HTTP/1.1 200 HTTP_OK
00:37:41:I1:WU573:Received WU assignment NVieAQm711ou_6LhSiKtPvGX6Uy9c9PHXJhioeK5jh0
00:37:41:I1:WU573:Downloading WU
00:37:41:I1:OUT12:> POST https://vav22.fah.temple.edu/api/assign HTTP/1.1
00:37:46:I1:OUT12:< HTTP/1.1 200 HTTP_OK
00:37:46:I1:WU573:Received WU P16969 R2 C237 G541
00:37:46:I3:Running FahCore: C:\ProgramData\FAHClient\cores/fahcore-a8-win-64bit-avx2_256-0.0.12/FahCore_a8.exe -dir NVieAQm711ou_6LhSiKtPvGX6Uy9c9PHXJhioeK5jh0 -suffix 01 -version 8.4.9 -lifeline 13384 -np 11
00:37:46:I3:WU573:Started FahCore on PID 3656

Then, the GPU WU was completed and this CPU WU was dumped (see previous log). This happens seemingly intermittently around every 5 or 6 completed WUs, always when a GPU WU is finished. Log says "This indicates that the core is not functional on your system. Check for missing libraries or GPU drivers." but I've got latest version AMD driver and I've just uninstalled and reinstalled Visual C++. GPU work units never fail.

I don't know what to do. If there's a more useful piece of log I can paste, let me know.

[calxalot]

Maybe it has something to do with the available cpu count varying when a GPU WU finishes.

You might try making a separate resource group for cpu. Set cpus to zero in the GPU group.

[muziqaz]

Project 18265 is the finished GPU work unit. The dumped one is the CPU assigned one, from project 16969.
Here's the log when it was assigned:

Code: Select all

00:37:40:I1:WU573:Requesting WU assignment for user acwvfqvlj2ui4vcpalbdq5trcfsdtnqpwu5ypuip63uueh4zkvt2h34augfii team 1066107
00:37:40:I1:OUT11:> POST https://assign3.foldingathome.org/api/assign HTTP/1.1
00:37:40:I1:OUT10:> POST https://vav17.fah.temple.edu/api/results HTTP/1.1
00:37:41:I1:OUT11:< HTTP/1.1 200 HTTP_OK
00:37:41:I1:WU573:Received WU assignment NVieAQm711ou_6LhSiKtPvGX6Uy9c9PHXJhioeK5jh0
00:37:41:I1:WU573:Downloading WU
00:37:41:I1:OUT12:> POST https://vav22.fah.temple.edu/api/assign HTTP/1.1
00:37:46:I1:OUT12:< HTTP/1.1 200 HTTP_OK
00:37:46:I1:WU573:Received WU P16969 R2 C237 G541
00:37:46:I3:Running FahCore: C:\ProgramData\FAHClient\cores/fahcore-a8-win-64bit-avx2_256-0.0.12/FahCore_a8.exe -dir NVieAQm711ou_6LhSiKtPvGX6Uy9c9PHXJhioeK5jh0 -suffix 01 -version 8.4.9 -lifeline 13384 -np 11
00:37:46:I3:WU573:Started FahCore on PID 3656

Then, the GPU WU was completed and this CPU WU was dumped (see previous log). This happens seemingly intermittently around every 5 or 6 completed WUs, always when a GPU WU is finished. Log says "This indicates that the core is not functional on your system. Check for missing libraries or GPU drivers." but I've got latest version AMD driver and I've just uninstalled and reinstalled Visual C++. GPU work units never fail.

I don't know what to do. If there's a more useful piece of log I can paste, let me know.

hm, yes, sorry. 16969 indeed has not been recorded yet.
As calxalot suggested, try creating separate resource groups for CPU and GPU.
Go to the cog looking icon for that system in web ui. right hand corner you should see padlock looking icon. click on that, click yes. on left hand side you should see Resource Group section with Default under it. Click on Default, untick your gpu, set CPU slider to number of threads your CPU has minus 1.
On the left hand side, click Add+, the name the resource group: GPU, click ok, the tick your GPU, and make sure CPU slider is set to 0. Click Save at the top right corner. Now your web ui should have 2 resource groups shown. One named Default, other named GPU. Both now can be controlled separately. Try folding now, and see how it behaves. Each group will have their own logs as well.

[Tropical Snowman]

Thanks to both of you. I'm trying this method, let's see if it holds uo. No error in the first work unit.

I'm curious: how does this resource group thing work? When I first started folding, I selected GPU only and it kept failing, so I figured it needed at least one CPU core. But it's working now, even tho it's GPU only.

[muziqaz]

GPUs do not need a CPU assigned to them. It just needs one core/thread free doing nothing left for it.
Resource groups just compartmentalize each device to its own group, that they do not interfere with each other

[Tropical Snowman]

No dumped or failed WUs so far. The issue seems resolved. Thanks a lot!