FahCore returned: INTERRUPTED (102 = 0x66)

It seems that a lot of GPU problems revolve around specific versions of drivers. Though NVidia has their own support structure, you can often learn from information reported by others who fold.

Moderators: Site Moderators, FAHC Science Team

Post Reply
daves_nt_here
Posts: 6
Joined: Thu Dec 22, 2022 6:21 pm

FahCore returned: INTERRUPTED (102 = 0x66)

Post by daves_nt_here »

Googled the error and most posts are from back in 2020 about a bad Fahcore so I don't this this is my issue.
Fresh install of Debian 11 and current FaH. Ran for 3 days processing a bunch of WU's then this morning I get this error.
I tried deleting the GPU work unit to see if it was bad. That didn't help.

Code: Select all

*********************** Log Started 2023-08-20T19:01:03Z ***********************
19:01:03:******************************* libFAH ********************************
19:01:03:           Date: Oct 20 2020
19:01:03:           Time: 20:36:39
19:01:03:       Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
19:01:03:         Branch: master
19:01:03:       Compiler: GNU 8.3.0
19:01:03:        Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:03:                 -fdata-sections -O3 -funroll-loops -fno-pie
19:01:03:       Platform: linux2 5.8.0-1-amd64
19:01:03:           Bits: 64
19:01:03:           Mode: Release
19:01:03:****************************** FAHClient ******************************
19:01:03:        Version: 7.6.21
19:01:03:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:01:03:      Copyright: 2020 foldingathome.org
19:01:03:       Homepage: https://foldingathome.org/
19:01:03:           Date: Oct 20 2020
19:01:03:           Time: 20:39:00
19:01:03:       Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
19:01:03:         Branch: master
19:01:03:       Compiler: GNU 8.3.0
19:01:03:        Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:03:                 -fdata-sections -O3 -funroll-loops -fno-pie
19:01:03:       Platform: linux2 5.8.0-1-amd64
19:01:03:           Bits: 64
19:01:03:           Mode: Release
19:01:03:           Args: --child /etc/fahclient/config.xml --run-as fahclient
19:01:03:                 --pid-file=/var/run/fahclient.pid --daemon
19:01:03:         Config: /etc/fahclient/config.xml
19:01:03:******************************** CBang ********************************
19:01:03:           Date: Oct 20 2020
19:01:03:           Time: 18:37:59
19:01:03:       Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
19:01:03:         Branch: master
19:01:03:       Compiler: GNU 8.3.0
19:01:03:        Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:03:                 -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
19:01:03:       Platform: linux2 5.8.0-1-amd64
19:01:03:           Bits: 64
19:01:03:           Mode: Release
19:01:03:******************************* System ********************************
19:01:03:            CPU: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
19:01:03:         CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
19:01:03:           CPUs: 9
19:01:03:         Memory: 125.83GiB
19:01:03:    Free Memory: 22.87GiB
19:01:03:        Threads: POSIX_THREADS
19:01:03:     OS Version: 5.15
19:01:03:    Has Battery: false
19:01:03:     On Battery: false
19:01:03:     UTC Offset: 0
19:01:03:            PID: 309
19:01:03:            CWD: /var/lib/fahclient
19:01:03:             OS: Linux 5.15.108-1-pve x86_64
19:01:03:        OS Arch: AMD64
19:01:03:           GPUs: 1
19:01:03:          GPU 0: Bus:5 Slot:0 Func:0 NVIDIA:3 GK110 [Tesla K20C] 3524
19:01:03:  CUDA Device 0: Platform:0 Device:0 Bus:5 Slot:0 Compute:3.5 Driver:11.2
19:01:03:OpenCL Device 0: Platform:0 Device:0 Bus:5 Slot:0 Compute:1.2 Driver:460.106
19:01:03:***********************************************************************
19:01:03:<config>
19:01:03:  <!-- Client Control -->
19:01:03:  <fold-anon v='true'/>
19:01:03:
19:01:03:  <!-- HTTP Server -->
19:01:03:  <allow v='127.0.0.1 192.168.52.0/24'/>
19:01:03:
19:01:03:  <!-- Network -->
19:01:03:  <proxy v=':8080'/>
19:01:03:
19:01:03:  <!-- Remote Command Server -->
19:01:03:  <command-allow-no-pass v='127.0.0.1 192.168.52.0/24'/>
19:01:03:
19:01:03:  <!-- Slot Control -->
19:01:03:  <power v='full'/>
19:01:03:
19:01:03:  <!-- User Information -->
19:01:03:  <passkey v='*****'/>
19:01:03:  <team v='223518'/>
19:01:03:  <user v='Daves_nt_here'/>
19:01:03:
19:01:03:  <!-- Web Server -->
19:01:03:  <web-allow v='127.0.0.1 192.168.52.0/24'/>
19:01:03:
19:01:03:  <!-- Folding Slots -->
19:01:03:  <slot id='0' type='CPU'/>
19:01:03:  <slot id='1' type='GPU'>
19:01:03:    <pci-bus v='5'/>
19:01:03:    <pci-slot v='0'/>
19:01:03:  </slot>
19:01:03:</config>
19:01:03:Trying to access database...
19:01:03:Successfully acquired database lock
19:01:03:FS00:Initialized folding slot 00: cpu:8
19:01:03:FS01:Initialized folding slot 01: gpu:5:0 GK110 [Tesla K20C] 3524
19:01:03:WU01:FS00:Starting
19:01:03:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx-256/a8-0.0.12/Core_a8.fah/FahCore_a8 -dir 01 -suffix 01 -version 706 -lifeline 309 -checkpoint 15 -np 8
19:01:03:WU01:FS00:Started FahCore on PID 319
19:01:03:WU01:FS00:Core PID:323
19:01:03:WU01:FS00:FahCore 0xa8 started
19:01:03:WU00:FS01:Starting
19:01:03:WU00:FS01:Removing old file 'work/00/logfile_01-20230820-182337.txt'
19:01:03:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 309 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
19:01:03:WU00:FS01:Started FahCore on PID 327
19:01:03:WU00:FS01:Core PID:331
19:01:03:WU00:FS01:FahCore 0x22 started
19:01:03:WU01:FS00:0xa8:*********************** Log Started 2023-08-20T19:01:03Z ***********************
19:01:03:WU01:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
19:01:03:WU01:FS00:0xa8:       Core: Gromacs
19:01:03:WU01:FS00:0xa8:       Type: 0xa8
19:01:03:WU01:FS00:0xa8:    Version: 0.0.12
19:01:03:WU01:FS00:0xa8:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:01:03:WU01:FS00:0xa8:  Copyright: 2020 foldingathome.org
19:01:03:WU01:FS00:0xa8:   Homepage: https://foldingathome.org/
19:01:03:WU01:FS00:0xa8:       Date: Jan 16 2021
19:01:03:WU01:FS00:0xa8:       Time: 19:23:19
19:01:03:WU01:FS00:0xa8:   Compiler: GNU 8.3.0
19:01:03:WU01:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
19:01:03:WU01:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
19:01:03:WU01:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
19:01:03:WU01:FS00:0xa8:       Bits: 64
19:01:03:WU01:FS00:0xa8:       Mode: Release
19:01:03:WU01:FS00:0xa8:       SIMD: avx_256
19:01:03:WU01:FS00:0xa8:     OpenMP: ON
19:01:03:WU01:FS00:0xa8:       CUDA: OFF
19:01:03:WU01:FS00:0xa8:       Args: -dir 01 -suffix 01 -version 706 -lifeline 319 -checkpoint 15 -np 8
19:01:03:WU01:FS00:0xa8:************************************ libFAH ************************************
19:01:03:WU01:FS00:0xa8:       Date: Jan 16 2021
19:01:03:WU01:FS00:0xa8:       Time: 19:21:38
19:01:03:WU01:FS00:0xa8:   Compiler: GNU 8.3.0
19:01:03:WU01:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
19:01:03:WU01:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
19:01:03:WU01:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
19:01:03:WU01:FS00:0xa8:       Bits: 64
19:01:03:WU01:FS00:0xa8:       Mode: Release
19:01:03:WU01:FS00:0xa8:************************************ CBang *************************************
19:01:03:WU01:FS00:0xa8:       Date: Jan 16 2021
19:01:03:WU01:FS00:0xa8:       Time: 19:21:24
19:01:03:WU01:FS00:0xa8:   Compiler: GNU 8.3.0
19:01:03:WU01:FS00:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
19:01:03:WU01:FS00:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
19:01:03:WU01:FS00:0xa8:   Platform: linux2 4.15.0-128-generic
19:01:03:WU01:FS00:0xa8:       Bits: 64
19:01:03:WU01:FS00:0xa8:       Mode: Release
19:01:03:WU01:FS00:0xa8:************************************ System ************************************
19:01:03:WU01:FS00:0xa8:        CPU: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
19:01:03:WU01:FS00:0xa8:     CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
19:01:03:WU01:FS00:0xa8:       CPUs: 9
19:01:03:WU01:FS00:0xa8:     Memory: 125.83GiB
19:01:03:WU01:FS00:0xa8:Free Memory: 22.87GiB
19:01:03:WU01:FS00:0xa8:    Threads: POSIX_THREADS
19:01:03:WU01:FS00:0xa8: OS Version: 5.15
19:01:03:WU01:FS00:0xa8:Has Battery: false
19:01:03:WU01:FS00:0xa8: On Battery: false
19:01:03:WU01:FS00:0xa8: UTC Offset: 0
19:01:03:WU01:FS00:0xa8:        PID: 323
19:01:03:WU01:FS00:0xa8:        CWD: /var/lib/fahclient/work
19:01:03:WU01:FS00:0xa8:********************************************************************************
19:01:03:WU01:FS00:0xa8:Project: 16959 (Run 38, Clone 114, Gen 653)
19:01:03:WU01:FS00:0xa8:Unit: 0x00000000000000000000000000000000
19:01:03:WU01:FS00:0xa8:Digital signatures verified
19:01:03:WU01:FS00:0xa8:Calling: mdrun -c frame653.gro -s frame653.tpr -x frame653.xtc -cpi state.cpt -cpt 15 -nt 8 -ntmpi 1
19:01:03:WU01:FS00:0xa8:Steps: first=1632500000 total=1635000000
19:01:03:WU00:FS01:0x22:*********************** Log Started 2023-08-20T19:01:03Z ***********************
19:01:03:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:01:03:WU00:FS01:0x22:       Core: Core22
19:01:03:WU00:FS01:0x22:       Type: 0x22
19:01:03:WU00:FS01:0x22:    Version: 0.0.20
19:01:03:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:01:03:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
19:01:03:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
19:01:03:WU00:FS01:0x22:       Date: Jan 20 2022
19:01:03:WU00:FS01:0x22:       Time: 00:57:52
19:01:03:WU00:FS01:0x22:   Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
19:01:03:WU00:FS01:0x22:     Branch: HEAD
19:01:03:WU00:FS01:0x22:   Compiler: GNU 9.4.0
19:01:03:WU00:FS01:0x22:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:03:WU00:FS01:0x22:             -fdata-sections -O3 -funroll-loops -fno-pie
19:01:03:WU00:FS01:0x22:             -DOPENMM_VERSION="\"7.7.0\""
19:01:03:WU00:FS01:0x22:   Platform: linux 5.11.0-1025-azure
19:01:03:WU00:FS01:0x22:       Bits: 64
19:01:03:WU00:FS01:0x22:       Mode: Release
19:01:03:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:01:03:WU00:FS01:0x22:             <peastman@stanford.edu>
19:01:03:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 327 -checkpoint 15
19:01:03:WU00:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
19:01:03:WU00:FS01:0x22:             nvidia -gpu 0 -gpu-usage 100
19:01:03:WU00:FS01:0x22:************************************ libFAH ************************************
19:01:03:WU00:FS01:0x22:       Date: Jan 20 2022
19:01:03:WU00:FS01:0x22:       Time: 00:57:22
19:01:03:WU00:FS01:0x22:   Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
19:01:03:WU00:FS01:0x22:     Branch: HEAD
19:01:03:WU00:FS01:0x22:   Compiler: GNU 9.4.0
19:01:03:WU00:FS01:0x22:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:03:WU00:FS01:0x22:             -fdata-sections -O3 -funroll-loops -fno-pie
19:01:03:WU00:FS01:0x22:   Platform: linux 5.11.0-1025-azure
19:01:03:WU00:FS01:0x22:       Bits: 64
19:01:03:WU00:FS01:0x22:       Mode: Release
19:01:03:WU00:FS01:0x22:************************************ CBang *************************************
19:01:03:WU00:FS01:0x22:       Date: Jan 20 2022
19:01:03:WU00:FS01:0x22:       Time: 00:57:00
19:01:03:WU00:FS01:0x22:   Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
19:01:03:WU00:FS01:0x22:     Branch: HEAD
19:01:03:WU00:FS01:0x22:   Compiler: GNU 9.4.0
19:01:03:WU00:FS01:0x22:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:03:WU00:FS01:0x22:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
19:01:03:WU00:FS01:0x22:   Platform: linux 5.11.0-1025-azure
19:01:03:WU00:FS01:0x22:       Bits: 64
19:01:03:WU00:FS01:0x22:       Mode: Release
19:01:03:WU00:FS01:0x22:************************************ System ************************************
19:01:03:WU00:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
19:01:03:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
19:01:03:WU00:FS01:0x22:       CPUs: 9
19:01:03:WU00:FS01:0x22:     Memory: 125.83GiB
19:01:03:WU00:FS01:0x22:Free Memory: 22.86GiB
19:01:03:WU00:FS01:0x22:    Threads: POSIX_THREADS
19:01:03:WU00:FS01:0x22: OS Version: 5.15
19:01:03:WU00:FS01:0x22:Has Battery: false
19:01:03:WU00:FS01:0x22: On Battery: false
19:01:03:WU00:FS01:0x22: UTC Offset: 0
19:01:03:WU00:FS01:0x22:        PID: 331
19:01:03:WU00:FS01:0x22:        CWD: /var/lib/fahclient/work
19:01:03:WU00:FS01:0x22:************************************ OpenMM ************************************
19:01:03:WU00:FS01:0x22:    Version: 7.7.0
19:01:03:WU00:FS01:0x22:********************************************************************************
19:01:03:WU00:FS01:0x22:Project: 12010 (Run 0, Clone 3, Gen 49)
19:01:03:WU00:FS01:0x22:Digital signatures verified
19:01:03:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:01:03:WU00:FS01:0x22:Version 0.0.20
19:01:03:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:01:03:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:01:03:WU00:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
19:01:03:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
19:01:03:WU00:FS01:0x22:There are 4 platforms available.
19:01:03:WU00:FS01:0x22:Platform 0: Reference
19:01:03:WU00:FS01:0x22:Platform 1: CPU
19:01:03:WU00:FS01:0x22:Platform 2: OpenCL
19:01:03:WU00:FS01:0x22:  opencl-device 0 specified
19:01:03:WU00:FS01:0x22:Platform 3: CUDA
19:01:03:WU00:FS01:0x22:  cuda-device 0 specified
19:01:04:WU01:FS00:0xa8:Completed 37752 out of 2500000 steps (1%)
19:01:14:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
19:01:14:WU00:FS01:Starting
19:01:14:WU00:FS01:Removing old file 'work/00/logfile_01-20230820-182437.txt'
19:01:14:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 309 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
19:01:14:WU00:FS01:Started FahCore on PID 346
19:01:14:WU00:FS01:Core PID:350
19:01:14:WU00:FS01:FahCore 0x22 started
19:01:15:WU00:FS01:0x22:*********************** Log Started 2023-08-20T19:01:14Z ***********************
19:01:15:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
19:01:15:WU00:FS01:0x22:       Core: Core22
19:01:15:WU00:FS01:0x22:       Type: 0x22
19:01:15:WU00:FS01:0x22:    Version: 0.0.20
19:01:15:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
19:01:15:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
19:01:15:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
19:01:15:WU00:FS01:0x22:       Date: Jan 20 2022
19:01:15:WU00:FS01:0x22:       Time: 00:57:52
19:01:15:WU00:FS01:0x22:   Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
19:01:15:WU00:FS01:0x22:     Branch: HEAD
19:01:15:WU00:FS01:0x22:   Compiler: GNU 9.4.0
19:01:15:WU00:FS01:0x22:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:15:WU00:FS01:0x22:             -fdata-sections -O3 -funroll-loops -fno-pie
19:01:15:WU00:FS01:0x22:             -DOPENMM_VERSION="\"7.7.0\""
19:01:15:WU00:FS01:0x22:   Platform: linux 5.11.0-1025-azure
19:01:15:WU00:FS01:0x22:       Bits: 64
19:01:15:WU00:FS01:0x22:       Mode: Release
19:01:15:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
19:01:15:WU00:FS01:0x22:             <peastman@stanford.edu>
19:01:15:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 346 -checkpoint 15
19:01:15:WU00:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
19:01:15:WU00:FS01:0x22:             nvidia -gpu 0 -gpu-usage 100
19:01:15:WU00:FS01:0x22:************************************ libFAH ************************************
19:01:15:WU00:FS01:0x22:       Date: Jan 20 2022
19:01:15:WU00:FS01:0x22:       Time: 00:57:22
19:01:15:WU00:FS01:0x22:   Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
19:01:15:WU00:FS01:0x22:     Branch: HEAD
19:01:15:WU00:FS01:0x22:   Compiler: GNU 9.4.0
19:01:15:WU00:FS01:0x22:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:15:WU00:FS01:0x22:             -fdata-sections -O3 -funroll-loops -fno-pie
19:01:15:WU00:FS01:0x22:   Platform: linux 5.11.0-1025-azure
19:01:15:WU00:FS01:0x22:       Bits: 64
19:01:15:WU00:FS01:0x22:       Mode: Release
19:01:15:WU00:FS01:0x22:************************************ CBang *************************************
19:01:15:WU00:FS01:0x22:       Date: Jan 20 2022
19:01:15:WU00:FS01:0x22:       Time: 00:57:00
19:01:15:WU00:FS01:0x22:   Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
19:01:15:WU00:FS01:0x22:     Branch: HEAD
19:01:15:WU00:FS01:0x22:   Compiler: GNU 9.4.0
19:01:15:WU00:FS01:0x22:    Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
19:01:15:WU00:FS01:0x22:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
19:01:15:WU00:FS01:0x22:   Platform: linux 5.11.0-1025-azure
19:01:15:WU00:FS01:0x22:       Bits: 64
19:01:15:WU00:FS01:0x22:       Mode: Release
19:01:15:WU00:FS01:0x22:************************************ System ************************************
19:01:15:WU00:FS01:0x22:        CPU: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
19:01:15:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
19:01:15:WU00:FS01:0x22:       CPUs: 9
19:01:15:WU00:FS01:0x22:     Memory: 125.83GiB
19:01:15:WU00:FS01:0x22:Free Memory: 22.71GiB
19:01:15:WU00:FS01:0x22:    Threads: POSIX_THREADS
19:01:15:WU00:FS01:0x22: OS Version: 5.15
19:01:15:WU00:FS01:0x22:Has Battery: false
19:01:15:WU00:FS01:0x22: On Battery: false
19:01:15:WU00:FS01:0x22: UTC Offset: 0
19:01:15:WU00:FS01:0x22:        PID: 350
19:01:15:WU00:FS01:0x22:        CWD: /var/lib/fahclient/work
19:01:15:WU00:FS01:0x22:************************************ OpenMM ************************************
19:01:15:WU00:FS01:0x22:    Version: 7.7.0
19:01:15:WU00:FS01:0x22:********************************************************************************
19:01:15:WU00:FS01:0x22:Project: 12010 (Run 0, Clone 3, Gen 49)
19:01:15:WU00:FS01:0x22:Digital signatures verified
19:01:15:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
19:01:15:WU00:FS01:0x22:Version 0.0.20
19:01:15:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
19:01:15:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
19:01:15:WU00:FS01:0x22:  XTC frame write interval: 25000 steps (2.5%) [40 total]
19:01:15:WU00:FS01:0x22:  Global context and integrator variables write interval: disabled
19:01:15:WU00:FS01:0x22:There are 4 platforms available.
19:01:15:WU00:FS01:0x22:Platform 0: Reference
19:01:15:WU00:FS01:0x22:Platform 1: CPU
19:01:15:WU00:FS01:0x22:Platform 2: OpenCL
19:01:15:WU00:FS01:0x22:  opencl-device 0 specified
19:01:15:WU00:FS01:0x22:Platform 3: CUDA
19:01:15:WU00:FS01:0x22:  cuda-device 0 specified
19:01:24:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
Top
daves_nt_here
Posts: 6
Joined: Thu Dec 22, 2022 6:21 pm

Re: FahCore returned: INTERRUPTED (102 = 0x66)

Post by daves_nt_here »

Well my impatience payed off. Just posting this encase anyone else can see a deeper issue or needs to resolve this in the future.

I had deleted /var/lib/fahclient/work/01 folder and contents. That didn't work but deleting the whole /var/lib/fahclient/work folder and contents fixed the issues.
For now ;)
Image
Top
daves_nt_here
Posts: 6
Joined: Thu Dec 22, 2022 6:21 pm

Re: FahCore returned: INTERRUPTED (102 = 0x66)

Post by daves_nt_here »

Spoke too soon. It processed one WU, uploaded it, grabbed another then went back to the same error. Looping every minute.
Image
Top
toTOW
Site Moderator
Posts: 6349
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France
Contact:
Contact toTOW

Re: FahCore returned: INTERRUPTED (102 = 0x66)

Post by toTOW »

This is an old Kepler GPU ... any chances that it is showing some signs of its age ?

Note : the WU mentioned in first post has been completed successfully ...
daves_nt_here wrote: ↑Sun Aug 20, 2023 10:21 pm Spoke too soon. It processed one WU, uploaded it, grabbed another then went back to the same error. Looping every minute.
Please provide some WU references so that we could check it and think about potential links between the cases ...
Image

Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
Top
daves_nt_here
Posts: 6
Joined: Thu Dec 22, 2022 6:21 pm

Re: FahCore returned: INTERRUPTED (102 = 0x66)

Post by daves_nt_here »

Found the problem after a couple days of rest. I am running it in a Proxmox VE container. Checked the Proxmox logs and found the container kept maxing out the ram I had given it. I had only allocated 1gb of ram. Seems happy now with 3gb. Peeks out around 2.2gb
Image
Top
toTOW
Site Moderator
Posts: 6349
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France
Contact:
Contact toTOW

Re: FahCore returned: INTERRUPTED (102 = 0x66)

Post by toTOW »

It is indeed one of the possible cause of such errors.

The recommendation is to have at least 4 GB of system RAM per GPU slot.
Image

Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
Top
Bsb5068
Posts: 28
Joined: Tue Sep 05, 2017 10:00 am

Re: FahCore returned: INTERRUPTED (102 = 0x66)

Post by Bsb5068 »

i've got same errors too. used 4 CPU cores for this slot:

Code: Select all

13:50:44:WU03:FS03:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx2-256/a8-0.0.12/Core_a8.fah/FahCore_a8 -dir 03 -suffix 01 -version 706 -lifeline 3776 -checkpoint 6 -np 6
13:50:44:WU03:FS03:Started FahCore on PID 13067
13:50:44:WU03:FS03:Core PID:13071
13:50:44:WU03:FS03:FahCore 0xa8 started
13:50:44:WU03:FS03:0xa8:*********************** Log Started 2024-01-06T13:50:44Z ***********************
13:50:44:WU03:FS03:0xa8:************************** Gromacs Folding@home Core ***************************
13:50:44:WU03:FS03:0xa8:       Core: Gromacs
13:50:44:WU03:FS03:0xa8:       Type: 0xa8
13:50:44:WU03:FS03:0xa8:    Version: 0.0.12
13:50:44:WU03:FS03:0xa8:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
13:50:44:WU03:FS03:0xa8:  Copyright: 2020 foldingathome.org
13:50:44:WU03:FS03:0xa8:   Homepage: https://foldingathome.org/
13:50:44:WU03:FS03:0xa8:       Date: Jan 16 2021
13:50:44:WU03:FS03:0xa8:       Time: 19:24:44
13:50:44:WU03:FS03:0xa8:   Compiler: GNU 8.3.0
13:50:44:WU03:FS03:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
13:50:44:WU03:FS03:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
13:50:44:WU03:FS03:0xa8:   Platform: linux2 4.15.0-128-generic
13:50:44:WU03:FS03:0xa8:       Bits: 64
13:50:44:WU03:FS03:0xa8:       Mode: Release
13:50:44:WU03:FS03:0xa8:       SIMD: avx2_256
13:50:44:WU03:FS03:0xa8:     OpenMP: ON
13:50:44:WU03:FS03:0xa8:       CUDA: OFF
13:50:44:WU03:FS03:0xa8:       Args: -dir 03 -suffix 01 -version 706 -lifeline 13067 -checkpoint 6 -np 6
13:50:44:WU03:FS03:0xa8:************************************ libFAH ************************************
13:50:44:WU03:FS03:0xa8:       Date: Jan 16 2021
13:50:44:WU03:FS03:0xa8:       Time: 19:21:38
13:50:44:WU03:FS03:0xa8:   Compiler: GNU 8.3.0
13:50:44:WU03:FS03:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
13:50:44:WU03:FS03:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie
13:50:44:WU03:FS03:0xa8:   Platform: linux2 4.15.0-128-generic
13:50:44:WU03:FS03:0xa8:       Bits: 64
13:50:44:WU03:FS03:0xa8:       Mode: Release
13:50:44:WU03:FS03:0xa8:************************************ CBang *************************************
13:50:44:WU03:FS03:0xa8:       Date: Jan 16 2021
13:50:44:WU03:FS03:0xa8:       Time: 19:21:24
13:50:44:WU03:FS03:0xa8:   Compiler: GNU 8.3.0
13:50:44:WU03:FS03:0xa8:    Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
13:50:44:WU03:FS03:0xa8:             -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
13:50:44:WU03:FS03:0xa8:   Platform: linux2 4.15.0-128-generic
13:50:44:WU03:FS03:0xa8:       Bits: 64
13:50:44:WU03:FS03:0xa8:       Mode: Release
13:50:44:WU03:FS03:0xa8:************************************ System ************************************
13:50:44:WU03:FS03:0xa8:        CPU: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
13:50:44:WU03:FS03:0xa8:     CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
13:50:44:WU03:FS03:0xa8:       CPUs: 16
13:50:44:WU03:FS03:0xa8:     Memory: 15.49GiB
13:50:44:WU03:FS03:0xa8:Free Memory: 9.87GiB
13:50:44:WU03:FS03:0xa8:    Threads: POSIX_THREADS
13:50:44:WU03:FS03:0xa8: OS Version: 5.4
13:50:44:WU03:FS03:0xa8:Has Battery: false
13:50:44:WU03:FS03:0xa8: On Battery: false
13:50:44:WU03:FS03:0xa8: UTC Offset: 3
13:50:44:WU03:FS03:0xa8:        PID: 13071
13:50:44:WU03:FS03:0xa8:        CWD: /var/lib/fahclient/work
13:50:44:WU03:FS03:0xa8:********************************************************************************
13:50:44:WU03:FS03:0xa8:Project: 12444 (Run 8, Clone 0, Gen 4)
13:50:44:WU03:FS03:0xa8:Unit: 0x00000000000000000000000000000000
13:50:44:WU03:FS03:0xa8:Digital signatures verified
13:50:44:WU03:FS03:0xa8:Calling: mdrun -c frame4.gro -s frame4.tpr -x frame4.xtc -cpi state.cpt -cpt 6 -nt 6 -ntmpi 1
13:50:44:WU03:FS03:0xa8:Steps: first=10000000 total=12500000
13:50:45:WU03:FS03:0xa8:Completed 1174351 out of 2500000 steps (46%)
13:50:48:WU03:FS03:FahCore returned: INTERRUPTED (102 = 0x66)
gmesg log:

Code: Select all

[  191.718002] Segment Routing with IPv6
[112664.505015] FahCore_a8[9667]: segfault at 5179ffc ip 000000000071b374 sp 00007ffff9a336f0 error 4
[112664.505022] Code: 58 0c 88 4c 8b 44 24 40 c5 ea 5c d0 4f 63 04 90 4c 8b 54 24 48 c5 fa 11 48 04 c5 fa 11 50 08 44 89 01 4f 63 0c 8e 44 89 49 04 <49> 63 3c ba 89 79 08 41 83 fb 01 7e 2c 4c 8b 54 24 10 47 8b 04 82
[115911.521639] FahCore_a8[5347]: segfault at fffffffe01e70e30 ip 000000000071b329 sp 00007f92964abda0 error 5
[115911.521647] Code: 6a 59 d1 c4 41 7a 2c cd c5 7a 2c c1 c5 fa 2c fa c4 c1 7a 2a c1 4d 63 d1 4c 8b 4c 24 30 c4 41 0a 2a f0 c5 12 5c e0 c5 f8 57 c0 <c4> 01 1a 58 2c 91 4d 63 c8 c5 fa 2a c7 4c 8b 44 24 38 c4 41 72 5c
# free
total used free shared buff/cache available
Mem: 16239932 5422072 8701400 593200 2116460 9750048
Swap: 0 0 0

may be its to low memory too?

upd: i make 16gb swap partition, reload FAHClient,

Code: Select all

[149859.208178] EXT4-fs (sda1): resizing filesystem from 1280000 to 2303744 blocks
[149859.235816] EXT4-fs (sda1): resized filesystem to 2303744
[149897.616466] Adding 16777212k swap on /dev/sda2.  Priority:-2 extents:1 across:16777212k FS
nothing changes :(
Top
toTOW
Site Moderator
Posts: 6349
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France
Contact:
Contact toTOW

Re: FahCore returned: INTERRUPTED (102 = 0x66)

Post by toTOW »

It's not the same issue. Your error is from the CPU core.

You seem to have plenty of free memory and current CPU work isn't very memory intensive ...

It could be some slightly unstable WU ... according to the WU database, a user called AKoo returned it with an error, but I completed it successfully.
Image

Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
Top
Post Reply

Return to “Problems with NVidia drivers”