Nvidia GPU error INTERRUPTED (102 = 0x66)

[goodyca]

Version 22-0.0.13 has been downloaded and installed on all 3 of my computers. All of the clients are running Fedora 32 and the Nvidia driver 450.66-2.fc32.x86_64 and CUDA . The clients will no longer fold GPU units. Here is an example from one of the log files.

Code: Select all

*********************** Log Started 2020-09-29T12:38:18Z ***********************
12:38:31:FS01:Unpaused
12:38:31:WU01:FS01:Downloading core from http://cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah
12:38:31:WU01:FS01:Connecting to cores.foldingathome.org:80
12:38:31:WU01:FS01:FahCore 22: Downloading 79.02MiB
12:38:37:WU01:FS01:FahCore 22: 72.45%
12:38:38:WU01:FS01:FahCore 22: Download complete
12:38:39:WU01:FS01:Valid core signature
12:38:39:WU01:FS01:Unpacked 5.21MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22
12:38:39:WU01:FS01:Unpacked 65.05KiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMPME.so
12:38:39:WU01:FS01:Unpacked 2.72MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMOpenCL.so
12:38:39:WU01:FS01:Unpacked 32.98KiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMCudaCompiler.so
12:38:39:WU01:FS01:Unpacked 2.54MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMCUDA.so
12:38:39:WU01:FS01:Unpacked 84.05MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libcufft.so.9.2
12:38:39:WU01:FS01:Unpacked 570.08KiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMMCPU.so
12:38:39:WU01:FS01:Unpacked 3.21MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libnvrtc-builtins.so
12:38:39:WU01:FS01:Unpacked 3.22MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libOpenMM.so
12:38:39:WU01:FS01:Unpacked 30.73KiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libfftw3f_threads.so.3
12:38:39:WU01:FS01:Unpacked 1.57MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libfftw3f.so.3
12:38:39:WU01:FS01:Unpacked 19.32MiB to cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/libnvrtc.so.9.2
12:38:39:WU01:FS01:Starting
12:38:39:WU01:FS01:Removing old file 'work/01/logfile_01-20200929-113748.txt'
12:38:39:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 706 -lifeline 14522 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
12:38:39:WU01:FS01:Started FahCore on PID 14686
12:38:39:WU01:FS01:Core PID:14690
12:38:39:WU01:FS01:FahCore 0x22 started
12:38:39:WU01:FS01:0x22:*********************** Log Started 2020-09-29T12:38:39Z ***********************
12:38:39:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
12:38:39:WU01:FS01:0x22:       Core: Core22
12:38:39:WU01:FS01:0x22:       Type: 0x22
12:38:39:WU01:FS01:0x22:    Version: 0.0.13
12:38:39:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
12:38:39:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
12:38:39:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
12:38:39:WU01:FS01:0x22:       Date: Sep 19 2020
12:38:39:WU01:FS01:0x22:       Time: 01:10:35
12:38:39:WU01:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
12:38:39:WU01:FS01:0x22:     Branch: core22-0.0.13
12:38:39:WU01:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
12:38:39:WU01:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
12:38:39:WU01:FS01:0x22:             -funroll-loops -DOPENMM_GIT_HASH="\"189320d0\""
12:38:39:WU01:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
12:38:39:WU01:FS01:0x22:       Bits: 64
12:38:39:WU01:FS01:0x22:       Mode: Release
12:38:39:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
12:38:39:WU01:FS01:0x22:             <peastman@stanford.edu>
12:38:39:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 14686 -checkpoint 15
12:38:39:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
12:38:39:WU01:FS01:0x22:             0 -gpu 0
12:38:39:WU01:FS01:0x22:************************************ libFAH ************************************
12:38:39:WU01:FS01:0x22:       Date: Sep 15 2020
12:38:39:WU01:FS01:0x22:       Time: 05:14:43
12:38:39:WU01:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
12:38:39:WU01:FS01:0x22:     Branch: HEAD
12:38:39:WU01:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
12:38:39:WU01:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
12:38:39:WU01:FS01:0x22:             -funroll-loops
12:38:39:WU01:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
12:38:39:WU01:FS01:0x22:       Bits: 64
12:38:39:WU01:FS01:0x22:       Mode: Release
12:38:39:WU01:FS01:0x22:************************************ CBang *************************************
12:38:39:WU01:FS01:0x22:       Date: Sep 15 2020
12:38:39:WU01:FS01:0x22:       Time: 05:11:04
12:38:39:WU01:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
12:38:39:WU01:FS01:0x22:     Branch: HEAD
12:38:39:WU01:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
12:38:39:WU01:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
12:38:39:WU01:FS01:0x22:             -funroll-loops -fPIC
12:38:39:WU01:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
12:38:39:WU01:FS01:0x22:       Bits: 64
12:38:39:WU01:FS01:0x22:       Mode: Release
12:38:39:WU01:FS01:0x22:************************************ System ************************************
12:38:39:WU01:FS01:0x22:        CPU: Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
12:38:39:WU01:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
12:38:39:WU01:FS01:0x22:       CPUs: 12
12:38:39:WU01:FS01:0x22:     Memory: 15.56GiB
12:38:39:WU01:FS01:0x22:Free Memory: 10.17GiB
12:38:39:WU01:FS01:0x22:    Threads: POSIX_THREADS
12:38:39:WU01:FS01:0x22: OS Version: 5.8
12:38:39:WU01:FS01:0x22:Has Battery: false
12:38:39:WU01:FS01:0x22: On Battery: false
12:38:39:WU01:FS01:0x22: UTC Offset: -5
12:38:39:WU01:FS01:0x22:        PID: 14690
12:38:39:WU01:FS01:0x22:        CWD: /var/lib/fahclient/work
12:38:39:WU01:FS01:0x22:************************************ OpenMM ************************************
12:38:39:WU01:FS01:0x22:   Revision: 189320d0
12:38:39:WU01:FS01:0x22:********************************************************************************
12:38:39:WU01:FS01:0x22:Project: 17102 (Run 7, Clone 1502, Gen 0)
12:38:39:WU01:FS01:0x22:Unit: 0x0000000012bc7d9a5f728a8c2dfc2f39
12:38:39:WU01:FS01:0x22:Reading tar file core.xml
12:38:39:WU01:FS01:0x22:Reading tar file integrator.xml.bz2
12:38:39:WU01:FS01:0x22:Reading tar file state.xml.bz2
12:38:39:WU01:FS01:0x22:Reading tar file system.xml.bz2
12:38:39:WU01:FS01:0x22:Digital signatures verified
12:38:39:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
12:38:39:WU01:FS01:0x22:Version 0.0.13
12:38:40:WU01:FS01:FahCore returned: INTERRUPTED (102 = 0x66)

[PantherX]

Just wondering if you have the OpenCL Package installed (sudo apt-get install ocl-icd-opencl-dev). Also, if you have CUDA toolkit installed, what happens when you uninstall it? What's your GPU model?

EDIT: I made a mistake, it should have been uninstalled, not installed.

[goodyca]

The OpenCL is installed, and I was successfully folding on the GPU's prior to the update to version 22.0.0.13. I have one GeForce GTX 1650 Super and two GeForce GTX 1050 Ti.

The CUDA toolkit is not installed.

[goodyca]

Installing the CUDA toolkit made no difference.

[bruce]

Without the top of FAH's log, I can only guess how your GPU(s) are configured when FAHClient initializes and what drivers are available to each GPU. Development is working on issues surrounding that initialization process and the client may see future updates to resolve them.

In the meantime we may be able to help you with a work-around. (See below)

[pannu]

This has happened to me too. Was working perfectly prior to 22-0.0.13. Looks like I got similar setup as OP, Fedora 32 driver 450.66 etc..

Code: Select all

04:02:54:Trying to access database...
04:02:54:Successfully acquired database lock
04:02:54:Read GPUs.txt
04:02:54:Enabled folding slot 01: READY gpu:0:GM206 [GeForce GTX 950] 1572
04:02:54:****************************** FAHClient ******************************
04:02:54:        Version: 7.6.13
04:02:54:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:02:54:      Copyright: 2020 foldingathome.org
04:02:54:       Homepage: https://foldingathome.org/
04:02:54:           Date: Apr 28 2020
04:02:54:           Time: 04:20:27
04:02:54:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
04:02:54:         Branch: master
04:02:54:       Compiler: GNU 4.9.4
04:02:54:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
04:02:54:                 -funroll-loops
04:02:54:       Platform: linux2 4.19.0-5-amd64
04:02:54:           Bits: 64
04:02:54:           Mode: Release
04:02:54:           Args: --user=**** --team=***
04:02:54:                 --passkey=******************************** --gpu=true
04:02:54:                 --power=medium --cause=COVID_19
04:02:54:         Config: /home/****/config.xml
04:02:54:******************************** CBang ********************************
04:02:54:           Date: Apr 25 2020
04:02:54:           Time: 00:07:55
04:02:54:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
04:02:54:         Branch: master
04:02:54:       Compiler: GNU 4.9.4
04:02:54:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
04:02:54:                 -funroll-loops -fPIC
04:02:54:       Platform: linux2 4.19.0-5-amd64
04:02:54:           Bits: 64
04:02:54:           Mode: Release
04:02:54:******************************* System ********************************
04:02:54:            CPU: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
04:02:54:         CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
04:02:54:           CPUs: 4
04:02:54:         Memory: 7.71GiB
04:02:54:    Free Memory: 714.13MiB
04:02:54:        Threads: POSIX_THREADS
04:02:54:     OS Version: 5.8
04:02:54:    Has Battery: false
04:02:54:     On Battery: false
04:02:54:     UTC Offset: 3
04:02:54:            PID: 38820
04:02:54:            CWD: /home/****
04:02:54:             OS: Linux 5.8.11-200.fc32.x86_64 x86_64
04:02:54:        OS Arch: AMD64
04:02:54:           GPUs: 1
04:02:54:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:5 GM206 [GeForce GTX 950] 1572
04:02:54:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:5.2 Driver:11.0
04:02:54:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:450.66
04:02:54:******************************* libFAH ********************************
04:02:54:           Date: Apr 15 2020
04:02:54:           Time: 21:43:27
04:02:54:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
04:02:54:         Branch: master
04:02:54:       Compiler: GNU 4.9.4
04:02:54:        Options: -std=c++11 -ffunction-sections -fdata-sections -O3
04:02:54:                 -funroll-loops
04:02:54:       Platform: linux2 4.19.0-5-amd64
04:02:54:           Bits: 64
04:02:54:           Mode: Release
04:02:54:***********************************************************************
04:02:54:<config>
04:02:54:  <!-- Folding Slots -->
04:02:54:  <slot id='1' type='GPU'/>
04:02:54:</config>
04:02:54:WU00:FS01:Starting
04:02:54:WU00:FS01:Removing old file 'work/00/logfile_01-20201001-033105.txt'
04:02:54:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /home/****/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 38820 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
04:02:54:WU00:FS01:Started FahCore on PID 38835
04:02:55:WU00:FS01:Core PID:38839
04:02:55:WU00:FS01:FahCore 0x22 started
04:02:55:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)

[PantherX]

Installing the CUDA toolkit made no difference.

Apologies, I meant if you had CUDA installed, then uninstall it. Using CUDA toolkit isn't required for F@H. I have edited my original post.

[PantherX]

Welcome to the F@H Forum pannu,

Please note that Nvidia has released a newer version of their Linux drivers, version 450.80.02 which might resolve that issue. Can you please look into updating it if possible?

If that still fails, can you please follow these instructions and see what happens:

Can you try a few things to help us debug?

Run the core stand-alone: First, go to the path where the core has been downloaded, something like cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/. Can you ls -l that directory and paste the output here? Then try running the core directly to make sure it doesn't segfault:

LD_LIBRARY_PATH="." ./FahCore_22 -info
Report the output here.

Delete the core and have it re-download: It's possible one of the dynamic libraries was accidentally deleted by an uninstall, in which case deleting the whole directory (carefully rm -rf cores/cores.foldingathome.org/lin/64bit/22-0.0.13 or equivalent) will force the core to be re-downloaded and unpacked.

Add the -disable-cuda flag: Assuming that the previous step ran fine, can you try to configure the extra-core-args for your GPU folding slot to add -disable-cuda, which should allow you to go back to OpenCL. Does this work?

We have a few more things we can do to debug from here after we get this info back.

Source: https://github.com/FoldingAtHome/fah-is ... -701640944

[pannu]

I only use repositories to update these type of drivers and 450.80 hasn't made it there yet as it was apparently released yesterday.

Code: Select all

[Core_22.fah]$ ls -l
total 125496
-rwxr-xr-x. 1 **** ****  5458944 Sep 30 23:46 FahCore_22
-rw-r--r--. 1 **** **** 88134064 Sep 30 23:46 libcufft.so.9.2
-rw-r--r--. 1 **** ****  1647691 Sep 30 23:46 libfftw3f.so.3
-rw-r--r--. 1 **** ****    31467 Sep 30 23:46 libfftw3f_threads.so.3
-rw-r--r--. 1 **** ****  3362248 Sep 30 23:46 libnvrtc-builtins.so
-rw-r--r--. 1 **** **** 20257792 Sep 30 23:46 libnvrtc.so.9.2
-rw-r--r--. 1 **** ****   583767 Sep 30 23:46 libOpenMMCPU.so
-rw-r--r--. 1 **** ****    33769 Sep 30 23:46 libOpenMMCudaCompiler.so
-rw-r--r--. 1 **** ****  2665529 Sep 30 23:46 libOpenMMCUDA.so
-rw-r--r--. 1 **** ****  2855961 Sep 30 23:46 libOpenMMOpenCL.so
-rw-r--r--. 1 **** ****    66614 Sep 30 23:46 libOpenMMPME.so
-rw-r--r--. 1 **** ****  3380650 Sep 30 23:46 libOpenMM.so

Code: Select all

[Core_22.fah]$ LD_LIBRARY_PATH="." ./FahCore_22 -info
*************************** Core22 Folding@home Core ***************************
       Core: Core22
       Type: 0x22
    Version: 0.0.13
     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
  Copyright: 2020 foldingathome.org
   Homepage: https://foldingathome.org/
       Date: Sep 19 2020
       Time: 01:10:35
   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
     Branch: core22-0.0.13
   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
             -funroll-loops -DOPENMM_GIT_HASH="\"189320d0\""
   Platform: linux2 4.19.76-linuxkit
       Bits: 64
       Mode: Release
Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
             <peastman@stanford.edu>
       Args: -info
************************************ libFAH ************************************
       Date: Sep 15 2020
       Time: 05:14:43
   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
     Branch: HEAD
   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
             -funroll-loops
   Platform: linux2 4.19.76-linuxkit
       Bits: 64
       Mode: Release
************************************ CBang *************************************
       Date: Sep 15 2020
       Time: 05:11:04
   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
     Branch: HEAD
   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
             -funroll-loops -fPIC
   Platform: linux2 4.19.76-linuxkit
       Bits: 64
       Mode: Release
************************************ System ************************************
        CPU: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
     CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
       CPUs: 4
     Memory: 7.71GiB
Free Memory: 6.51GiB
    Threads: POSIX_THREADS
 OS Version: 5.8
Has Battery: false
 On Battery: false
 UTC Offset: 3
        PID: 8355
        CWD: /home/****/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah
************************************ OpenMM ************************************
   Revision: 189320d0
********************************************************************************

Neither delete/re-download of core or disable-cuda flag helped.

[hus]

Me too ... Several machines with Fedora 32 & 33(beta), GeForce 1060 and driver 450.66 from rpmfusion. Uninstalled the rpmfusion driver and installed 450.80.02 directly from Nvidia - now it works again.

[pannu]

Unfortunately I'm still getting the same error with 450.80.02 driver.

[foldy]

You can add the setting -disable-cuda to extra core options to disable CUDA and use OpenCL instead which is a little slower

[pannu]

-disable-cuda doesn't help. Everything worked just fine with version 0.0.12.

Code: Select all

16:12:35:             OS: Linux 5.8.13-200.fc32.x86_64 x86_64
16:12:35:        OS Arch: AMD64
16:12:35:           GPUs: 1
16:12:35:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:5 GM206 [GeForce GTX 950] 1572
16:12:35:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:5.2 Driver:11.0
16:12:35:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:450.80


16:12:35:<config>
16:12:35:  <!-- Folding Slots -->
16:12:35:  <slot id='0' type='CPU'/>
16:12:35:  <slot id='1' type='GPU'>
16:12:35:    <extra-core-args v='-disable-cuda'/>
16:12:35:  </slot>
16:12:35:</config>


16:14:00:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
16:14:00:WU01:FS01:0x22:Version 0.0.13
16:14:00:WU01:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
16:14:59:WU01:FS01:Starting
16:14:59:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /home/*****/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 706 -lifeline 135904 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0 -disable-cuda

[PantherX]

That's unexpected behavior (FahCore_22 fails to run on OpenCL even when -disable-cuda is used) so I have reported it. Let's see what happens

[pannu]

Just to update, no change with next set of drivers:

Code: Select all

20:25:53:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:5.2 Driver:11.1
20:25:53:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:455.28

20:26:03:WU01:FS01:FahCore 0x22 started
20:26:04:WU01:FS01:FahCore returned: INTERRUPTED (102 = 0x66)

This is the output from journalctl:

Code: Select all

kernel: FahCore_22[7588]: segfault at 0 ip 0000000000000000 sp 00007ffd069d1e38 error 14 in FahCore_22[400000+4e9000]
kernel: Code: Bad RIP value.