Folding Forum

Community driven support forum for Folding@home
https://foldingforum.org/

CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

https://foldingforum.org/viewtopic.php?t=43270

Page 1 of 1

CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Sat Oct 11, 2025 6:44 pm
by ivy47
Hi, I installed F@H yesterday on my new linux mint setup after ditching windows a few months back. It worked perfectly out of the box on day 1; I let it finish the work unit and allow the PC to sleep overnight afterwards. It's configured (on the web interface) to not do any CPU folding, just the GPU. Now, attempting to fold always spits out aborted work units. Example log:

Code: Select all

18:13:52:I3:WU25:Started FahCore on PID 71334
18:13:52:I1:WU25:*********************** Log Started 2025-10-11T18:13:52Z ***********************
18:13:52:I1:WU25:*************************** Core27 Folding@home Core ***************************
18:13:52:I1:WU25:       Core: Core27
18:13:52:I1:WU25:       Type: 0x27
18:13:52:I1:WU25:    Version: 8.2.1
18:13:52:I1:WU25:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:13:52:I1:WU25:  Copyright: 2022 foldingathome.org
18:13:52:I1:WU25:   Homepage: https://foldingathome.org/
18:13:52:I1:WU25:       Date: Mar 27 2025
18:13:52:I1:WU25:       Time: 04:28:24
18:13:52:I1:WU25:   Revision: 84019442387121f2bed4ed52e05c53b7b7b52a72
18:13:52:I1:WU25:     Branch: HEAD
18:13:52:I1:WU25:   Compiler: GNU 7.5.0
18:13:52:I1:WU25:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
18:13:52:I1:WU25:             -fdata-sections -O3 -funroll-loops -fno-pie
18:13:52:I1:WU25:             -DOPENMM_VERSION="\"8.2.0\""
18:13:52:I1:WU25:   Platform: linux 6.8.0-1021-azure
18:13:52:I1:WU25:       Bits: 64
18:13:52:I1:WU25:       Mode: Release
18:13:52:I1:WU25:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
18:13:52:I1:WU25:             <peastman@stanford.edu>
18:13:52:I1:WU25:       Args: -dir wKA28vt3mkPHlPz0X7LoB3kCwPpuqC-xDb0QBw9f0Uo -suffix 01
18:13:52:I1:WU25:             -version 8.4.9 -lifeline 1410 -gpu-uuid
18:13:52:I1:WU25:             6d368011-d0d1-a715-29bf-4c7760f61e08 -gpu-platform cuda -gpu-vendor
18:13:52:I1:WU25:             nvidia -cuda-platform 0 -cuda-device 0
18:13:52:I1:WU25:************************************ libFAH ************************************
18:13:52:I1:WU25:       Date: Mar 27 2025
18:13:52:I1:WU25:       Time: 04:23:18
18:13:52:I1:WU25:   Revision: c7d2824a47eb025fa8cda8968c7a5e971585d90c
18:13:52:I1:WU25:     Branch: HEAD
18:13:52:I1:WU25:   Compiler: GNU 7.5.0
18:13:52:I1:WU25:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
18:13:52:I1:WU25:             -fdata-sections -O3 -funroll-loops -fno-pie
18:13:52:I1:WU25:   Platform: linux 6.8.0-1021-azure
18:13:52:I1:WU25:       Bits: 64
18:13:52:I1:WU25:       Mode: Release
18:13:52:I1:WU25:************************************ CBang *************************************
18:13:52:I1:WU25:    Version: 1.7.2
18:13:52:I1:WU25:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:13:52:I1:WU25:        Org: Cauldron Development LLC
18:13:52:I1:WU25:  Copyright: Cauldron Development LLC, 2003-2024
18:13:52:I1:WU25:   Homepage: https://cauldrondevelopment.com/
18:13:52:I1:WU25:    License: LGPL-2.1-or-later
18:13:52:I1:WU25:       Date: Mar 27 2025
18:13:52:I1:WU25:       Time: 04:22:53
18:13:52:I1:WU25:   Revision: f1cd4c791e8c40a35dcfeab3ab85d910949cc0cb
18:13:52:I1:WU25:     Branch: HEAD
18:13:52:I1:WU25:   Compiler: GNU 7.5.0
18:13:52:I1:WU25:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
18:13:52:I1:WU25:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
18:13:52:I1:WU25:   Platform: linux 6.8.0-1021-azure
18:13:52:I1:WU25:       Bits: 64
18:13:52:I1:WU25:       Mode: Release
18:13:52:I1:WU25:************************************ System ************************************
18:13:52:I1:WU25:        CPU: 12th Gen Intel(R) Core(TM) i5-12600KF
18:13:52:I1:WU25:     CPU ID: GenuineIntel Family 6 Model 151 Stepping 2
18:13:52:I1:WU25:       CPUs: 16
18:13:52:I1:WU25:     Memory: 46.88GiB
18:13:52:I1:WU25:Free Memory: 2.11GiB
18:13:52:I1:WU25: OS Version: 6.8
18:13:52:I1:WU25:Has Battery: false
18:13:52:I1:WU25: On Battery: false
18:13:52:I1:WU25:   Hostname: MS-7E06
18:13:52:I1:WU25: UTC Offset: -4
18:13:52:I1:WU25:        PID: 71334
18:13:52:I1:WU25:        CWD: /var/lib/fah-client/work
18:13:52:I1:WU25:       Exec: /var/lib/fah-client/cores/openmm-core-27/centos-7.9.2009-64bit/release/fahcore-27-centos-7.9.2009-64bit-release-8.2.1/FahCore_27
18:13:52:I1:WU25:************************************ OpenMM ************************************
18:13:52:I1:WU25:    Version: 8.2.0
18:13:52:I1:WU25:********************************************************************************
18:13:52:I1:WU25:Project: 18274 (Run 0, Clone 143, Gen 23)
18:13:52:I1:WU25:Reading tar file core.xml
18:13:52:I1:WU25:Reading tar file integrator.xml
18:13:52:I1:WU25:Reading tar file state.xml.bz2
18:13:52:I1:WU25:Reading tar file system.xml.bz2
18:13:52:I1:WU25:Digital signatures verified
18:13:52:I1:WU25:Folding@home GPU Core27 Folding@home Core
18:13:52:I1:WU25:Version 8.2.1
18:13:52:I1:WU25:  Checkpoint write interval: 50000 steps (2%) [50 total]
18:13:52:I1:WU25:  JSON viewer frame write interval: 25000 steps (1%) [100 total]
18:13:52:I1:WU25:  XTC frame write interval: 20000 steps (0.8%) [125 total]
18:13:52:I1:WU25:  TRR frame write interval: disabled
18:13:52:I1:WU25:  Global context and integrator variables write interval: disabled
18:13:52:I1:WU25:There are 3 platforms available.
18:13:52:I1:WU25:Platform 0: Reference
18:13:52:I1:WU25:Platform 1: CPU
18:13:52:I1:WU25:Platform 2: CUDA
18:13:52:I1:WU25:  cuda-device 0 specified
18:13:54:I1:WU25:Attempting to create CUDA context:
18:13:54:I1:WU25:  Configuring platform CUDA
18:13:54:I1:WU25:Failed to create CUDA context:
18:13:54:I1:WU25:Error initializing CUDA: CUDA_ERROR_UNKNOWN (999) at /home/conda/workspace/openmm/platforms/cuda/src/CudaContext.cpp:92
18:13:54:I1:WU25:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
18:13:54:I1:WU25:Saving result file ../logfile_01.txt
18:13:54:I1:WU25:Saving result file science.log
18:13:54:I1:WU25:Folding@home Core Shutdown: BAD_WORK_UNIT
[91m18:13:55:E :WU25:Core returned BAD_WORK_UNIT (114)[0m
18:13:55:I1:WU25:Uploading WU results
18:13:55:I1:OUT98:> POST https://highland4.seas.upenn.edu/api/results HTTP/1.1
18:13:55:I1:OUT98:< HTTP/1.1 200 HTTP_OK
18:13:55:I1:WU25:Credited
This is fixed by restarting the PC, but it would be nice to not have to resort to that. Is it possible for me to restart the process manually (evidently the fold/pause buttons still leave it running in a suspended state)?

Re: CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Sat Oct 11, 2025 6:53 pm
by calxalot

Code: Select all

sudo systemctl restart fah-client

Re: CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Sun Oct 12, 2025 1:39 pm
by toTOW
First thing to do on a folding machine : disable power saving features.

Re: CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Mon Oct 13, 2025 9:33 pm
by muziqaz
Second thing in computer world: disable sleep modes. In all these years no one came up with stable sleep mode

Re: CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Tue Oct 14, 2025 9:36 pm
by Breach
Sleep destroys CUDA context. If I understand correctly, you "sleep" after finishing a WU, correct? So normally the old context won't be re-used. You may try this, which allows in theory faster recovery:

Try (as root or sudo):
systemctl enable --now nvidia-persistenced
nvidia-smi -pm 1

If you need to reinit the gpu: nvidia-smi --gpu-reset

[Edit: Also you may want to try this script: https://www.reddit.com/r/linuxmint/comm ... ent_after/]

Re: CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Thu Oct 16, 2025 9:56 pm
by ivy47
Sorry for the wait; I didn't get any e-mail alert about my account getting verified so quickly!
calxalot wrote: ↑Sat Oct 11, 2025 6:53 pm

Code: Select all

sudo systemctl restart fah-client
Thank you; this does restart the process, but now the folding interface refuses to fetch any new work units...

Code: Select all

21:51:18:I1:Opening Database
21:51:18:I1:F@H ID = XYIbIBNAw5FJYvOQieSxY0GJZ_nC8j2l6ecoXLdF2S8
21:51:18:I3:Loading default resource group
21:51:18:I1:Listening for HTTP on 127.0.0.1:7396
21:51:18:I3:Loaded 0 wus.
21:51:18:I3:gpus = {
21:51:18:I3: "gpu:01:00:00": {"vendor": 4318, "device": 7809, "type": "nvidia", "supported": false, "description": "TU104 [GeForce RTX 2080 SUPER]"}
21:51:18:I3:}
21:51:18:I1:OUT1:> GET https://api.foldingathome.org/machine/XYIbIBNAw5FJYvOQieSxY0GJZ_nC8j2l6ecoXLdF2S8 HTTP/1.1
21:51:19:I1:OUT1:< HTTP/1.1 200 HTTP_OK
21:51:19:I1:OUT2:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:51:20:I1:OUT2:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
21:51:20:I1:Logging into node account
I can press "Fold all" but it just logs another line of "Machine state fold" without starting any download.
Breach wrote: ↑Tue Oct 14, 2025 9:36 pm Sleep destroys CUDA context. If I understand correctly, you "sleep" after finishing a WU, correct? So normally the old context won't be re-used. You may try this, which allows in theory faster recovery:

Try (as root or sudo):
systemctl enable --now nvidia-persistenced
nvidia-smi -pm 1

If you need to reinit the gpu: nvidia-smi --gpu-reset

[Edit: Also you may want to try this script: https://www.reddit.com/r/linuxmint/comm ... ent_after/]
I appreciate the extra functionality available here. I'd like to at least get the manual restart working just for simplicity's sake and then plausibly delve into this later for a more robust solution.

Not putting the PC to sleep at all is a bit woeful of a prospect. I'd rather not have this caveat when trying to convince friends to ditch windows lol.

Re: CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Fri Oct 17, 2025 6:58 pm
by Breach
I'm not running FAH on Linux myself (I'm one of these weird people who moved from UNIX to Windows for productivity ;-), but I think that there is something wrong with your nVidia driver. If you restart the client, it should properly work. It looks that CUDA is unusable after the driver exits from sleep, which rather points at an nVidia driver issue.

Re: CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Fri Oct 17, 2025 7:04 pm
by muziqaz
ivy47 wrote: ↑Thu Oct 16, 2025 9:56 pm Sorry for the wait; I didn't get any e-mail alert about my account getting verified so quickly!
calxalot wrote: ↑Sat Oct 11, 2025 6:53 pm

Code: Select all

sudo systemctl restart fah-client
Thank you; this does restart the process, but now the folding interface refuses to fetch any new work units...

Code: Select all

21:51:18:I1:Opening Database
21:51:18:I1:F@H ID = XYIbIBNAw5FJYvOQieSxY0GJZ_nC8j2l6ecoXLdF2S8
21:51:18:I3:Loading default resource group
21:51:18:I1:Listening for HTTP on 127.0.0.1:7396
21:51:18:I3:Loaded 0 wus.
21:51:18:I3:gpus = {
21:51:18:I3: "gpu:01:00:00": {"vendor": 4318, "device": 7809, "type": "nvidia", "supported": false, "description": "TU104 [GeForce RTX 2080 SUPER]"}
21:51:18:I3:}
21:51:18:I1:OUT1:> GET https://api.foldingathome.org/machine/XYIbIBNAw5FJYvOQieSxY0GJZ_nC8j2l6ecoXLdF2S8 HTTP/1.1
21:51:19:I1:OUT1:< HTTP/1.1 200 HTTP_OK
21:51:19:I1:OUT2:> GET wss://node1.foldingathome.org/ws/client HTTP/1.1
21:51:20:I1:OUT2:< HTTP/1.1 101 HTTP_SWITCHING_PROTOCOLS
21:51:20:I1:Logging into node account
I can press "Fold all" but it just logs another line of "Machine state fold" without starting any download.
Breach wrote: ↑Tue Oct 14, 2025 9:36 pm Sleep destroys CUDA context. If I understand correctly, you "sleep" after finishing a WU, correct? So normally the old context won't be re-used. You may try this, which allows in theory faster recovery:

Try (as root or sudo):
systemctl enable --now nvidia-persistenced
nvidia-smi -pm 1

If you need to reinit the gpu: nvidia-smi --gpu-reset

[Edit: Also you may want to try this script: https://www.reddit.com/r/linuxmint/comm ... ent_after/]
I appreciate the extra functionality available here. I'd like to at least get the manual restart working just for simplicity's sake and then plausibly delve into this later for a more robust solution.

Not putting the PC to sleep at all is a bit woeful of a prospect. I'd rather not have this caveat when trying to convince friends to ditch windows lol.
Your log shows that your GPU driver is not showing up. I will repeat myself: sleep to this day is the PITA for many users (Linux or Windows). This is not just FAH issue. Modern GPU drivers seem to have no clue how to handle sleep mode. This is true on AMD, and same with nVidia. If you want to contribute to FAH you will have to disable sleep, otherwise you will be wasting your own time (let alone researchers work) trying to get things going after your PC wakes up from sleep

Re: CUDA_ERROR_UNKNOWN when trying to fold after waking up from sleep

Posted: Sat Oct 18, 2025 4:47 pm
by ivy47
muziqaz wrote: ↑Fri Oct 17, 2025 7:04 pm Your log shows that your GPU driver is not showing up. I will repeat myself: sleep to this day is the PITA for many users (Linux or Windows). This is not just FAH issue. Modern GPU drivers seem to have no clue how to handle sleep mode. This is true on AMD, and same with nVidia. If you want to contribute to FAH you will have to disable sleep, otherwise you will be wasting your own time (let alone researchers work) trying to get things going after your PC wakes up from sleep
Every other part of the GPU including gaming still works fine in this environment. I don't understand why this one specific issue would be irreparable. It's a bit sad to see.
As for being courteous, yes, I turned off the folding as soon as I noticed the spam of failed work unit uploads lol.
All times are UTC
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited