Page 1 of 1
FAULTY project:18450 (Multiple errors)
Posted: Sat Nov 19, 2022 4:53 am
by n1np
What went wrong here? Failed in this way ten times before stopping.
(This is why I previously stopped running GPUs, too much maintenance, too needy
)
HP DL380g8 rack-mount server with 2x Quadro P4000 GPUs, Kubuntu 22 LTS, nvidia 520 driver, previously worked OK, now has issues with slot 2. Is this a fluke, a hardware problem, or something else?
Log pruned to fit forum rules.
Code: Select all
*********************** Log Started 2022-11-18T22:36:24Z ***********************
22:36:24:******************************* libFAH ********************************
22:36:24: Date: Oct 20 2020
22:36:24: Time: 20:36:39
22:36:24: Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
22:36:24: Branch: master
22:36:24: Compiler: GNU 8.3.0
22:36:24: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:36:24: -fdata-sections -O3 -funroll-loops -fno-pie
22:36:24: Platform: linux2 5.8.0-1-amd64
22:36:24: Bits: 64
22:36:24: Mode: Release
22:36:24:****************************** FAHClient ******************************
22:36:24: Version: 7.6.21
22:36:24: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:36:24: Copyright: 2020 foldingathome.org
22:36:24: Homepage: https://foldingathome.org/
22:36:24: Date: Oct 20 2020
22:36:24: Time: 20:39:00
22:36:24: Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
22:36:24: Branch: master
22:36:24: Compiler: GNU 8.3.0
22:36:24: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:36:24: -fdata-sections -O3 -funroll-loops -fno-pie
22:36:24: Platform: linux2 5.8.0-1-amd64
22:36:24: Bits: 64
22:36:24: Mode: Release
22:36:24: Args: --child /etc/fahclient/config.xml --run-as fahclient
22:36:24: --pid-file=/var/run/fahclient.pid --daemon
22:36:24: Config: /etc/fahclient/config.xml
22:36:24:******************************** CBang ********************************
22:36:24: Date: Oct 20 2020
22:36:24: Time: 18:37:59
22:36:24: Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
22:36:24: Branch: master
22:36:24: Compiler: GNU 8.3.0
22:36:24: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:36:24: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:36:24: Platform: linux2 5.8.0-1-amd64
22:36:24: Bits: 64
22:36:24: Mode: Release
22:36:24:******************************* System ********************************
22:36:24: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
22:36:24: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
22:36:24: CPUs: 32
22:36:24: Memory: 15.60GiB
22:36:24: Free Memory: 14.36GiB
22:36:24: Threads: POSIX_THREADS
22:36:24: OS Version: 5.15
22:36:24: Has Battery: false
22:36:24: On Battery: false
22:36:24: UTC Offset: -5
22:36:24: PID: 1469
22:36:24: CWD: /var/lib/fahclient
22:36:24: OS: Linux 5.15.0-53-generic x86_64
22:36:24: OS Arch: AMD64
22:36:24: GPUs: 2
22:36:24: GPU 0: Bus:4 Slot:0 Func:0 NVIDIA:5 GP104GL [Quadro P4000]
22:36:24: GPU 1: Bus:33 Slot:0 Func:0 NVIDIA:5 GP104GL [Quadro P4000]
22:36:24: CUDA Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:6.1 Driver:11.8
22:36:24: CUDA Device 1: Platform:0 Device:1 Bus:33 Slot:0 Compute:6.1 Driver:11.8
22:36:24:OpenCL Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:3.0 Driver:520.56
22:36:24:OpenCL Device 1: Platform:0 Device:1 Bus:33 Slot:0 Compute:3.0 Driver:520.56
22:36:24:***********************************************************************
22:36:24:<config>
22:36:24: <!-- Client Control -->
22:36:24: <fold-anon v='true'/>
22:36:24:
22:36:24: <!-- HTTP Server -->
22:36:24: <allow v='192.168.0/24'/>
22:36:24:
22:36:24: <!-- Network -->
22:36:24: <proxy v=':8080'/>
22:36:24:
22:36:24: <!-- Remote Command Server -->
22:36:24: <password v='*****'/>
22:36:24:
22:36:24: <!-- User Information -->
22:36:24: <passkey v='*****'/>
22:36:24: <team v='12912'/>
22:36:24: <user v='n1np'/>
22:36:24:
22:36:24: <!-- Folding Slots -->
22:36:24: <slot id='0' type='CPU'>
22:36:24: <cpus v='28'/>
22:36:24: <paused v='true'/>
22:36:24: </slot>
22:36:24: <slot id='1' type='GPU'>
22:36:24: <paused v='true'/>
22:36:24: <pci-bus v='33'/>
22:36:24: <pci-slot v='0'/>
22:36:24: </slot>
22:36:24: <slot id='2' type='GPU'>
22:36:24: <paused v='true'/>
22:36:24: </slot>
22:36:24:</config>
22:36:24:Trying to access database...
22:36:24:Successfully acquired database lock
22:36:24:FS00:Initialized folding slot 00: cpu:28
22:36:24:FS01:Initialized folding slot 01: gpu:33:0 GP104GL [Quadro P4000]
22:36:24:FS02:Initialized folding slot 02: gpu:4:0 GP104GL [Quadro P4000]
22:37:03:FS00:Unpaused
22:37:03:WU00:FS00:Starting
22:37:03:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx-256/a8-0.0.12/Core_a8.fah/FahCore_a8 -dir 00 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -np 28
22:37:03:WU00:FS00:Started FahCore on PID 2056
22:37:03:WU00:FS00:Core PID:2060
22:37:03:WU00:FS00:FahCore 0xa8 started
22:37:04:WU00:FS00:0xa8:*********************** Log Started 2022-11-18T22:37:04Z ***********************
22:37:04:WU00:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
22:37:04:WU00:FS00:0xa8: Core: Gromacs
22:37:04:WU00:FS00:0xa8: Type: 0xa8
22:37:04:WU00:FS00:0xa8: Version: 0.0.12
22:37:04:WU00:FS00:0xa8: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:37:04:WU00:FS00:0xa8: Copyright: 2020 foldingathome.org
22:37:04:WU00:FS00:0xa8: Homepage: https://foldingathome.org/
22:37:04:WU00:FS00:0xa8: Date: Jan 16 2021
22:37:04:WU00:FS00:0xa8: Time: 19:23:19
22:37:04:WU00:FS00:0xa8: Compiler: GNU 8.3.0
22:37:04:WU00:FS00:0xa8: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
22:37:04:WU00:FS00:0xa8: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:04:WU00:FS00:0xa8: Platform: linux2 4.15.0-128-generic
22:37:04:WU00:FS00:0xa8: Bits: 64
22:37:04:WU00:FS00:0xa8: Mode: Release
22:37:04:WU00:FS00:0xa8: SIMD: avx_256
22:37:04:WU00:FS00:0xa8: OpenMP: ON
22:37:04:WU00:FS00:0xa8: CUDA: OFF
22:37:04:WU00:FS00:0xa8: Args: -dir 00 -suffix 01 -version 706 -lifeline 2056 -checkpoint 15 -np
22:37:04:WU00:FS00:0xa8: 28
22:37:04:WU00:FS00:0xa8:************************************ libFAH ************************************
22:37:04:WU00:FS00:0xa8: Date: Jan 16 2021
22:37:04:WU00:FS00:0xa8: Time: 19:21:38
22:37:04:WU00:FS00:0xa8: Compiler: GNU 8.3.0
22:37:04:WU00:FS00:0xa8: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
22:37:04:WU00:FS00:0xa8: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:04:WU00:FS00:0xa8: Platform: linux2 4.15.0-128-generic
22:37:04:WU00:FS00:0xa8: Bits: 64
22:37:04:WU00:FS00:0xa8: Mode: Release
22:37:04:WU00:FS00:0xa8:************************************ CBang *************************************
22:37:04:WU00:FS00:0xa8: Date: Jan 16 2021
22:37:04:WU00:FS00:0xa8: Time: 19:21:24
22:37:04:WU00:FS00:0xa8: Compiler: GNU 8.3.0
22:37:04:WU00:FS00:0xa8: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
22:37:04:WU00:FS00:0xa8: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:37:04:WU00:FS00:0xa8: Platform: linux2 4.15.0-128-generic
22:37:04:WU00:FS00:0xa8: Bits: 64
22:37:04:WU00:FS00:0xa8: Mode: Release
22:37:04:WU00:FS00:0xa8:************************************ System ************************************
22:37:04:WU00:FS00:0xa8: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
22:37:04:WU00:FS00:0xa8: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
22:37:04:WU00:FS00:0xa8: CPUs: 32
22:37:04:WU00:FS00:0xa8: Memory: 15.60GiB
22:37:04:WU00:FS00:0xa8:Free Memory: 13.61GiB
22:37:04:WU00:FS00:0xa8: Threads: POSIX_THREADS
22:37:04:WU00:FS00:0xa8: OS Version: 5.15
22:37:04:WU00:FS00:0xa8:Has Battery: false
22:37:04:WU00:FS00:0xa8: On Battery: false
22:37:04:WU00:FS00:0xa8: UTC Offset: -5
22:37:04:WU00:FS00:0xa8: PID: 2060
22:37:04:WU00:FS00:0xa8: CWD: /var/lib/fahclient/work
22:37:04:WU00:FS00:0xa8:********************************************************************************
22:37:04:WU00:FS00:0xa8:Project: 16996 (Run 4, Clone 70, Gen 127)
22:37:04:WU00:FS00:0xa8:Unit: 0x00000000000000000000000000000000
22:37:04:WU00:FS00:0xa8:Digital signatures verified
22:37:04:WU00:FS00:0xa8:Calling: mdrun -c frame127.gro -s frame127.tpr -x frame127.xtc -cpi state.cpt -cpt 15 -nt 28 -ntmpi 1
22:37:04:WU00:FS00:0xa8:Steps: first=317500000 total=320000000
22:37:05:WU00:FS00:0xa8:Completed 33152 out of 2500000 steps (1%)
22:37:09:FS01:Unpaused
22:37:09:WU01:FS01:Starting
22:37:09:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
22:37:09:WU01:FS01:Started FahCore on PID 2091
22:37:09:WU01:FS01:Core PID:2095
22:37:09:WU01:FS01:FahCore 0x22 started
22:37:10:WU01:FS01:0x22:*********************** Log Started 2022-11-18T22:37:09Z ***********************
22:37:10:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
22:37:10:WU01:FS01:0x22: Core: Core22
22:37:10:WU01:FS01:0x22: Type: 0x22
22:37:10:WU01:FS01:0x22: Version: 0.0.20
22:37:10:WU01:FS01:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:37:10:WU01:FS01:0x22: Copyright: 2020 foldingathome.org
22:37:10:WU01:FS01:0x22: Homepage: https://foldingathome.org/
22:37:10:WU01:FS01:0x22: Date: Jan 20 2022
22:37:10:WU01:FS01:0x22: Time: 00:57:52
22:37:10:WU01:FS01:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
22:37:10:WU01:FS01:0x22: Branch: HEAD
22:37:10:WU01:FS01:0x22: Compiler: GNU 9.4.0
22:37:10:WU01:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:10:WU01:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:10:WU01:FS01:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
22:37:10:WU01:FS01:0x22: Platform: linux 5.11.0-1025-azure
22:37:10:WU01:FS01:0x22: Bits: 64
22:37:10:WU01:FS01:0x22: Mode: Release
22:37:10:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
22:37:10:WU01:FS01:0x22: <peastman@stanford.edu>
22:37:10:WU01:FS01:0x22: Args: -dir 01 -suffix 01 -version 706 -lifeline 2091 -checkpoint 15
22:37:10:WU01:FS01:0x22: -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
22:37:10:WU01:FS01:0x22: nvidia -gpu 1 -gpu-usage 100
22:37:10:WU01:FS01:0x22:************************************ libFAH ************************************
22:37:10:WU01:FS01:0x22: Date: Jan 20 2022
22:37:10:WU01:FS01:0x22: Time: 00:57:22
22:37:10:WU01:FS01:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
22:37:10:WU01:FS01:0x22: Branch: HEAD
22:37:10:WU01:FS01:0x22: Compiler: GNU 9.4.0
22:37:10:WU01:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:10:WU01:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:10:WU01:FS01:0x22: Platform: linux 5.11.0-1025-azure
22:37:10:WU01:FS01:0x22: Bits: 64
22:37:10:WU01:FS01:0x22: Mode: Release
22:37:10:WU01:FS01:0x22:************************************ CBang *************************************
22:37:10:WU01:FS01:0x22: Date: Jan 20 2022
22:37:10:WU01:FS01:0x22: Time: 00:57:00
22:37:10:WU01:FS01:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
22:37:10:WU01:FS01:0x22: Branch: HEAD
22:37:10:WU01:FS01:0x22: Compiler: GNU 9.4.0
22:37:10:WU01:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:10:WU01:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:37:10:WU01:FS01:0x22: Platform: linux 5.11.0-1025-azure
22:37:10:WU01:FS01:0x22: Bits: 64
22:37:10:WU01:FS01:0x22: Mode: Release
22:37:10:WU01:FS01:0x22:************************************ System ************************************
22:37:10:WU01:FS01:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
22:37:10:WU01:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
22:37:10:WU01:FS01:0x22: CPUs: 32
22:37:10:WU01:FS01:0x22: Memory: 15.60GiB
22:37:10:WU01:FS01:0x22:Free Memory: 13.26GiB
22:37:10:WU01:FS01:0x22: Threads: POSIX_THREADS
22:37:10:WU01:FS01:0x22: OS Version: 5.15
22:37:10:WU01:FS01:0x22:Has Battery: false
22:37:10:WU01:FS01:0x22: On Battery: false
22:37:10:WU01:FS01:0x22: UTC Offset: -5
22:37:10:WU01:FS01:0x22: PID: 2095
22:37:10:WU01:FS01:0x22: CWD: /var/lib/fahclient/work
22:37:10:WU01:FS01:0x22:************************************ OpenMM ************************************
22:37:10:WU01:FS01:0x22: Version: 7.7.0
22:37:10:WU01:FS01:0x22:********************************************************************************
22:37:10:WU01:FS01:0x22:Project: 18601 (Run 4814, Clone 2, Gen 20)
22:37:10:WU01:FS01:0x22:Digital signatures verified
22:37:10:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
22:37:10:WU01:FS01:0x22:Version 0.0.20
22:37:10:WU01:FS01:0x22: Checkpoint write interval: 25000 steps (2%) [50 total]
22:37:10:WU01:FS01:0x22: JSON viewer frame write interval: 12500 steps (1%) [100 total]
22:37:10:WU01:FS01:0x22: XTC frame write interval: 20000 steps (1.6%) [62 total]
22:37:10:WU01:FS01:0x22: Global context and integrator variables write interval: disabled
22:37:10:WU01:FS01:0x22:There are 4 platforms available.
22:37:10:WU01:FS01:0x22:Platform 0: Reference
22:37:10:WU01:FS01:0x22:Platform 1: CPU
22:37:10:WU01:FS01:0x22:Platform 2: OpenCL
22:37:10:WU01:FS01:0x22: opencl-device 1 specified
22:37:10:WU01:FS01:0x22:Platform 3: CUDA
22:37:10:WU01:FS01:0x22: cuda-device 1 specified
22:37:16:FS02:Unpaused
22:37:16:WU02:FS02:Connecting to assign1.foldingathome.org:80
22:37:16:WU02:FS02:Assigned to work server 129.32.209.202
22:37:16:WU02:FS02:Requesting new work unit for slot 02: gpu:4:0 GP104GL [Quadro P4000] from 129.32.209.202
22:37:16:WU02:FS02:Connecting to 129.32.209.202:8080
22:37:17:WU02:FS02:Downloading 41.72MiB
22:37:23:WU02:FS02:Download 51.53%
22:37:25:Removing old file 'configs/config-20221115-035206.xml'
22:37:25:Saving configuration to /etc/fahclient/config.xml
22:37:25:<config>
22:37:25: <!-- Client Control -->
22:37:25: <fold-anon v='true'/>
22:37:25:
22:37:25: <!-- HTTP Server -->
22:37:25: <allow v='192.168.0/24'/>
22:37:25:
22:37:25: <!-- Network -->
22:37:25: <proxy v=':8080'/>
22:37:25:
22:37:25: <!-- Remote Command Server -->
22:37:25: <password v='*****'/>
22:37:25:
22:37:25: <!-- User Information -->
22:37:25: <passkey v='*****'/>
22:37:25: <team v='12912'/>
22:37:25: <user v='n1np'/>
22:37:25:
22:37:25: <!-- Folding Slots -->
22:37:25: <slot id='0' type='CPU'>
22:37:25: <cpus v='28'/>
22:37:25: </slot>
22:37:25: <slot id='1' type='GPU'>
22:37:25: <pci-bus v='33'/>
22:37:25: <pci-slot v='0'/>
22:37:25: </slot>
22:37:25: <slot id='2' type='GPU'>
22:37:25: <pci-bus v='4'/>
22:37:25: <pci-slot v='0'/>
22:37:25: </slot>
22:37:25:</config>
22:37:28:WU02:FS02:Download complete
22:37:28:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:18447 run:3 clone:81 gen:275 core:0x22 unit:0x00000051000001130000480f00000003
22:37:28:WU02:FS02:Starting
22:37:28:WU02:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
22:37:28:WU02:FS02:Started FahCore on PID 2099
22:37:28:WU02:FS02:Core PID:2103
22:37:28:WU02:FS02:FahCore 0x22 started
22:37:28:WU02:FS02:0x22:*********************** Log Started 2022-11-18T22:37:28Z ***********************
22:37:28:WU02:FS02:0x22:*************************** Core22 Folding@home Core ***************************
22:37:28:WU02:FS02:0x22: Core: Core22
22:37:28:WU02:FS02:0x22: Type: 0x22
22:37:28:WU02:FS02:0x22: Version: 0.0.20
22:37:28:WU02:FS02:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:37:28:WU02:FS02:0x22: Copyright: 2020 foldingathome.org
22:37:28:WU02:FS02:0x22: Homepage: https://foldingathome.org/
22:37:28:WU02:FS02:0x22: Date: Jan 20 2022
22:37:28:WU02:FS02:0x22: Time: 00:57:52
22:37:28:WU02:FS02:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
22:37:28:WU02:FS02:0x22: Branch: HEAD
22:37:28:WU02:FS02:0x22: Compiler: GNU 9.4.0
22:37:28:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:28:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:28:WU02:FS02:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
22:37:28:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
22:37:28:WU02:FS02:0x22: Bits: 64
22:37:28:WU02:FS02:0x22: Mode: Release
22:37:28:WU02:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
22:37:28:WU02:FS02:0x22: <peastman@stanford.edu>
22:37:28:WU02:FS02:0x22: Args: -dir 02 -suffix 01 -version 706 -lifeline 2099 -checkpoint 15
22:37:28:WU02:FS02:0x22: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
22:37:28:WU02:FS02:0x22: nvidia -gpu 0 -gpu-usage 100
22:37:28:WU02:FS02:0x22:************************************ libFAH ************************************
22:37:28:WU02:FS02:0x22: Date: Jan 20 2022
22:37:28:WU02:FS02:0x22: Time: 00:57:22
22:37:28:WU02:FS02:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
22:37:28:WU02:FS02:0x22: Branch: HEAD
22:37:28:WU02:FS02:0x22: Compiler: GNU 9.4.0
22:37:28:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:28:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:28:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
22:37:28:WU02:FS02:0x22: Bits: 64
22:37:28:WU02:FS02:0x22: Mode: Release
22:37:28:WU02:FS02:0x22:************************************ CBang *************************************
22:37:28:WU02:FS02:0x22: Date: Jan 20 2022
22:37:28:WU02:FS02:0x22: Time: 00:57:00
22:37:28:WU02:FS02:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
22:37:28:WU02:FS02:0x22: Branch: HEAD
22:37:28:WU02:FS02:0x22: Compiler: GNU 9.4.0
22:37:28:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:28:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:37:28:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
22:37:28:WU02:FS02:0x22: Bits: 64
22:37:28:WU02:FS02:0x22: Mode: Release
22:37:28:WU02:FS02:0x22:************************************ System ************************************
22:37:28:WU02:FS02:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
22:37:28:WU02:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
22:37:28:WU02:FS02:0x22: CPUs: 32
22:37:28:WU02:FS02:0x22: Memory: 15.60GiB
22:37:28:WU02:FS02:0x22:Free Memory: 12.50GiB
22:37:28:WU02:FS02:0x22: Threads: POSIX_THREADS
22:37:28:WU02:FS02:0x22: OS Version: 5.15
22:37:28:WU02:FS02:0x22:Has Battery: false
22:37:28:WU02:FS02:0x22: On Battery: false
22:37:28:WU02:FS02:0x22: UTC Offset: -5
22:37:28:WU02:FS02:0x22: PID: 2103
22:37:28:WU02:FS02:0x22: CWD: /var/lib/fahclient/work
22:37:28:WU02:FS02:0x22:************************************ OpenMM ************************************
22:37:28:WU02:FS02:0x22: Version: 7.7.0
22:37:28:WU02:FS02:0x22:********************************************************************************
22:37:28:WU02:FS02:0x22:Project: 18447 (Run 3, Clone 81, Gen 275)
22:37:28:WU02:FS02:0x22:Reading tar file core.xml
22:37:28:WU02:FS02:0x22:Reading tar file integrator.xml
22:37:28:WU02:FS02:0x22:Reading tar file state.xml
22:37:28:WU02:FS02:0x22:Reading tar file system.xml
22:37:29:WU02:FS02:0x22:Digital signatures verified
22:37:29:WU02:FS02:0x22:Folding@home GPU Core22 Folding@home Core
22:37:29:WU02:FS02:0x22:Version 0.0.20
22:37:29:WU02:FS02:0x22: Checkpoint write interval: 50000 steps (2%) [50 total]
22:37:29:WU02:FS02:0x22: JSON viewer frame write interval: 25000 steps (1%) [100 total]
22:37:29:WU02:FS02:0x22: XTC frame write interval: 2500000 steps (1e+02%) [1 total]
22:37:29:WU02:FS02:0x22: Global context and integrator variables write interval: disabled
22:37:29:WU02:FS02:0x22:There are 4 platforms available.
22:37:29:WU02:FS02:0x22:Platform 0: Reference
22:37:29:WU02:FS02:0x22:Platform 1: CPU
22:37:29:WU02:FS02:0x22:Platform 2: OpenCL
22:37:29:WU02:FS02:0x22: opencl-device 0 specified
22:37:29:WU02:FS02:0x22:Platform 3: CUDA
22:37:29:WU02:FS02:0x22: cuda-device 0 specified
22:37:35:WU02:FS02:0x22:Attempting to create CUDA context:
22:37:35:WU02:FS02:0x22: Configuring platform CUDA
22:37:35:WU01:FS01:0x22:Attempting to create CUDA context:
22:37:35:WU01:FS01:0x22: Configuring platform CUDA
22:37:50:WU02:FS02:0x22: Using CUDA and gpu 0
22:37:50:WU02:FS02:0x22:Completed 0 out of 2500000 steps (0%)
22:37:50:WU02:FS02:0x22:Checkpoint completed at step 0
22:38:04:WU01:FS01:0x22: Using CUDA and gpu 1
22:38:04:WU01:FS01:0x22:Completed 0 out of 1250000 steps (0%)
22:38:06:WU01:FS01:0x22:Checkpoint completed at step 0
22:39:11:WU00:FS00:0xa8:Completed 50000 out of 2500000 steps (2%)
22:39:27:WU02:FS02:0x22:Completed 25000 out of 2500000 steps (1%)
22:41:05:WU02:FS02:0x22:Completed 50000 out of 2500000 steps (2%)
22:41:06:WU02:FS02:0x22:Checkpoint completed at step 50000
22:42:11:WU00:FS00:0xa8:Completed 75000 out of 2500000 steps (3%)
22:42:15:WU01:FS01:0x22:Completed 12500 out of 1250000 steps (1%)
22:42:44:WU02:FS02:0x22:Completed 75000 out of 2500000 steps (3%)
22:44:22:WU02:FS02:0x22:Completed 100000 out of 2500000 steps (4%)
22:44:23:WU02:FS02:0x22:Checkpoint completed at step 100000
22:45:12:WU00:FS00:0xa8:Completed 100000 out of 2500000 steps (4%)
22:45:30:FS00:Finishing
*********
cut to fit
*********
01:12:32:WU02:FS02:0x22:Checkpoint completed at step 2400000
01:12:45:WU01:FS01:0x22:Completed 462500 out of 1250000 steps (37%)
01:14:09:WU02:FS02:0x22:Completed 2425000 out of 2500000 steps (97%)
01:14:59:WU00:FS00:0xa8:Completed 1400000 out of 2500000 steps (56%)
01:15:45:WU02:FS02:0x22:Completed 2450000 out of 2500000 steps (98%)
01:15:45:WU02:FS02:0x22:Checkpoint completed at step 2450000
01:16:53:WU01:FS01:0x22:Completed 475000 out of 1250000 steps (38%)
01:16:56:WU01:FS01:0x22:Checkpoint completed at step 475000
01:17:21:WU02:FS02:0x22:Completed 2475000 out of 2500000 steps (99%)
01:17:22:WU03:FS02:Connecting to assign1.foldingathome.org:80
01:17:22:WU03:FS02:Assigned to work server 129.32.209.202
01:17:22:WU03:FS02:Requesting new work unit for slot 02: gpu:4:0 GP104GL [Quadro P4000] from 129.32.209.202
01:17:22:WU03:FS02:Connecting to 129.32.209.202:8080
01:17:23:WU03:FS02:Downloading 94.22MiB
01:17:29:WU03:FS02:Download 20.17%
01:17:35:WU03:FS02:Download 42.39%
01:17:41:WU03:FS02:Download 63.09%
01:17:47:WU03:FS02:Download 77.48%
01:17:52:WU00:FS00:0xa8:Completed 1425000 out of 2500000 steps (57%)
01:17:53:WU03:FS02:Download 93.33%
01:17:55:WU03:FS02:Download complete
01:17:55:WU03:FS02:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:18450 run:0 clone:75 gen:148 core:0x22 unit:0x0000004b000000940000481200000000
01:18:57:WU02:FS02:0x22:Completed 2500000 out of 2500000 steps (100%)
01:18:57:WU02:FS02:0x22:Average performance: 45 ns/day
01:18:57:WU02:FS02:0x22:Checkpoint completed at step 2500000
01:19:00:WU02:FS02:0x22:Saving result file ../logfile_01.txt
01:19:00:WU02:FS02:0x22:Saving result file checkpointIntegrator.xml
01:19:00:WU02:FS02:0x22:Saving result file checkpointState.xml
01:19:00:WU02:FS02:0x22:Saving result file positions.xtc
01:19:00:WU02:FS02:0x22:Saving result file science.log
01:19:00:WU02:FS02:0x22:Saving result file xtcAtoms.csv.bz2
01:19:00:WU02:FS02:0x22:Folding@home Core Shutdown: FINISHED_UNIT
01:19:01:WU02:FS02:FahCore returned: FINISHED_UNIT (100 = 0x64)
01:19:01:WU02:FS02:Sending unit results: id:02 state:SEND error:NO_ERROR project:18447 run:3 clone:81 gen:275 core:0x22 unit:0x00000051000001130000480f00000003
01:19:01:WU02:FS02:Uploading 21.04MiB to 129.32.209.202
01:19:01:WU02:FS02:Connecting to 129.32.209.202:8080
01:19:01:WU03:FS02:Starting
01:19:01:WU03:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 03 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
01:19:01:WU03:FS02:Started FahCore on PID 3205
01:19:01:WU03:FS02:Core PID:3209
01:19:01:WU03:FS02:FahCore 0x22 started
01:19:02:WU03:FS02:0x22:*********************** Log Started 2022-11-19T01:19:01Z ***********************
01:19:02:WU03:FS02:0x22:*************************** Core22 Folding@home Core ***************************
01:19:02:WU03:FS02:0x22: Core: Core22
01:19:02:WU03:FS02:0x22: Type: 0x22
01:19:02:WU03:FS02:0x22: Version: 0.0.20
01:19:02:WU03:FS02:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:19:02:WU03:FS02:0x22: Copyright: 2020 foldingathome.org
01:19:02:WU03:FS02:0x22: Homepage: https://foldingathome.org/
01:19:02:WU03:FS02:0x22: Date: Jan 20 2022
01:19:02:WU03:FS02:0x22: Time: 00:57:52
01:19:02:WU03:FS02:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
01:19:02:WU03:FS02:0x22: Branch: HEAD
01:19:02:WU03:FS02:0x22: Compiler: GNU 9.4.0
01:19:02:WU03:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:19:02:WU03:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
01:19:02:WU03:FS02:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
01:19:02:WU03:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:19:02:WU03:FS02:0x22: Bits: 64
01:19:02:WU03:FS02:0x22: Mode: Release
01:19:02:WU03:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
01:19:02:WU03:FS02:0x22: <peastman@stanford.edu>
01:19:02:WU03:FS02:0x22: Args: -dir 03 -suffix 01 -version 706 -lifeline 3205 -checkpoint 15
01:19:02:WU03:FS02:0x22: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
01:19:02:WU03:FS02:0x22: nvidia -gpu 0 -gpu-usage 100
01:19:02:WU03:FS02:0x22:************************************ libFAH ************************************
01:19:02:WU03:FS02:0x22: Date: Jan 20 2022
01:19:02:WU03:FS02:0x22: Time: 00:57:22
01:19:02:WU03:FS02:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
01:19:02:WU03:FS02:0x22: Branch: HEAD
01:19:02:WU03:FS02:0x22: Compiler: GNU 9.4.0
01:19:02:WU03:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:19:02:WU03:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
01:19:02:WU03:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:19:02:WU03:FS02:0x22: Bits: 64
01:19:02:WU03:FS02:0x22: Mode: Release
01:19:02:WU03:FS02:0x22:************************************ CBang *************************************
01:19:02:WU03:FS02:0x22: Date: Jan 20 2022
01:19:02:WU03:FS02:0x22: Time: 00:57:00
01:19:02:WU03:FS02:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
01:19:02:WU03:FS02:0x22: Branch: HEAD
01:19:02:WU03:FS02:0x22: Compiler: GNU 9.4.0
01:19:02:WU03:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:19:02:WU03:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
01:19:02:WU03:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:19:02:WU03:FS02:0x22: Bits: 64
01:19:02:WU03:FS02:0x22: Mode: Release
01:19:02:WU03:FS02:0x22:************************************ System ************************************
01:19:02:WU03:FS02:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
01:19:02:WU03:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
01:19:02:WU03:FS02:0x22: CPUs: 32
01:19:02:WU03:FS02:0x22: Memory: 15.60GiB
01:19:02:WU03:FS02:0x22:Free Memory: 10.81GiB
01:19:02:WU03:FS02:0x22: Threads: POSIX_THREADS
01:19:02:WU03:FS02:0x22: OS Version: 5.15
01:19:02:WU03:FS02:0x22:Has Battery: false
01:19:02:WU03:FS02:0x22: On Battery: false
01:19:02:WU03:FS02:0x22: UTC Offset: -5
01:19:02:WU03:FS02:0x22: PID: 3209
01:19:02:WU03:FS02:0x22: CWD: /var/lib/fahclient/work
01:19:02:WU03:FS02:0x22:************************************ OpenMM ************************************
01:19:02:WU03:FS02:0x22: Version: 7.7.0
01:19:02:WU03:FS02:0x22:********************************************************************************
01:19:02:WU03:FS02:0x22:Project: 18450 (Run 0, Clone 75, Gen 148)
01:19:02:WU03:FS02:0x22:Reading tar file core.xml
01:19:02:WU03:FS02:0x22:Reading tar file integrator.xml
01:19:02:WU03:FS02:0x22:Reading tar file state.xml
01:19:02:WU03:FS02:0x22:Reading tar file system.xml
01:19:02:WU03:FS02:0x22:Digital signatures verified
01:19:02:WU03:FS02:0x22:Folding@home GPU Core22 Folding@home Core
01:19:02:WU03:FS02:0x22:Version 0.0.20
01:19:02:WU03:FS02:0x22: Checkpoint write interval: 50000 steps (2%) [50 total]
01:19:02:WU03:FS02:0x22: JSON viewer frame write interval: 25000 steps (1%) [100 total]
01:19:02:WU03:FS02:0x22: XTC frame write interval: 2500000 steps (1e+02%) [1 total]
01:19:02:WU03:FS02:0x22: Global context and integrator variables write interval: disabled
01:19:02:WU03:FS02:0x22:There are 3 platforms available.
01:19:02:WU03:FS02:0x22:Platform 0: Reference
01:19:02:WU03:FS02:0x22:Platform 1: CPU
01:19:02:WU03:FS02:0x22:Platform 2: CUDA
01:19:02:WU03:FS02:0x22: cuda-device 0 specified
01:19:02:WU03:FS02:0x22:opencl-device was set but OpenCL platform could not be found.
01:19:07:WU02:FS02:Upload 17.52%
01:19:13:WU02:FS02:Upload 35.34%
01:19:16:WU03:FS02:0x22:Attempting to create CUDA context:
01:19:16:WU03:FS02:0x22: Configuring platform CUDA
01:19:17:WU03:FS02:0x22:Failed to create CUDA context:
01:19:17:WU03:FS02:0x22:Error initializing CUDA: CUDA_ERROR_UNKNOWN (999) at /home/conda/feedstock_root/build_artifacts/openmm_1640541150081/work/platforms/cuda/src/CudaContext.cpp:139
01:19:17:WU03:FS02:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
01:19:17:WU03:FS02:0x22:Saving result file ../logfile_01.txt
01:19:17:WU03:FS02:0x22:Saving result file science.log
01:19:17:WU03:FS02:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
[93m01:19:17:WARNING:WU03:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)[0m
01:19:17:WU03:FS02:Sending unit results: id:03 state:SEND error:FAULTY project:18450 run:0 clone:75 gen:148 core:0x22 unit:0x0000004b000000940000481200000000
01:19:17:WU03:FS02:Uploading 11.00KiB to 129.32.209.202
01:19:17:WU03:FS02:Connecting to 129.32.209.202:8080
01:19:18:WU04:FS02:Connecting to assign1.foldingathome.org:80
01:19:18:WU03:FS02:Upload complete
01:19:19:WU03:FS02:Server responded WORK_ACK (400)
01:19:19:WU03:FS02:Cleaning up
01:19:19:WU02:FS02:Upload 51.98%
01:19:20:WU04:FS02:Assigned to work server 129.32.209.202
01:19:20:WU04:FS02:Requesting new work unit for slot 02: gpu:4:0 GP104GL [Quadro P4000] from 129.32.209.202
01:19:20:WU04:FS02:Connecting to 129.32.209.202:8080
01:19:21:WU04:FS02:Downloading 94.21MiB
01:19:25:WU02:FS02:Upload 68.01%
01:19:27:WU04:FS02:Download 2.06%
01:19:31:WU02:FS02:Upload 84.35%
01:19:33:WU04:FS02:Download 4.84%
01:19:37:WU02:FS02:Upload 99.50%
01:19:38:WU02:FS02:Upload complete
01:19:38:WU02:FS02:Server responded WORK_ACK (400)
01:19:38:WU02:FS02:Final credit estimate, 114293.00 points
01:19:38:WU02:FS02:Cleaning up
01:19:39:WU04:FS02:Download 14.79%
01:19:45:WU04:FS02:Download 30.05%
01:19:51:WU04:FS02:Download 43.78%
01:19:57:WU04:FS02:Download 56.92%
01:20:03:WU04:FS02:Download 68.26%
01:20:09:WU04:FS02:Download 83.05%
01:20:15:WU04:FS02:Download 94.13%
01:20:17:WU04:FS02:Download complete
01:20:17:WU04:FS02:Received Unit: id:04 state:DOWNLOAD error:NO_ERROR project:18450 run:0 clone:305 gen:37 core:0x22 unit:0x00000131000000250000481200000000
01:20:17:WU04:FS02:Starting
01:20:17:WU04:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 04 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
01:20:17:WU04:FS02:Started FahCore on PID 3247
01:20:17:WU04:FS02:Core PID:3251
01:20:17:WU04:FS02:FahCore 0x22 started
01:20:18:WU04:FS02:0x22:*********************** Log Started 2022-11-19T01:20:17Z ***********************
01:20:18:WU04:FS02:0x22:*************************** Core22 Folding@home Core ***************************
01:20:18:WU04:FS02:0x22: Core: Core22
01:20:18:WU04:FS02:0x22: Type: 0x22
01:20:18:WU04:FS02:0x22: Version: 0.0.20
01:20:18:WU04:FS02:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:20:18:WU04:FS02:0x22: Copyright: 2020 foldingathome.org
01:20:18:WU04:FS02:0x22: Homepage: https://foldingathome.org/
01:20:18:WU04:FS02:0x22: Date: Jan 20 2022
01:20:18:WU04:FS02:0x22: Time: 00:57:52
01:20:18:WU04:FS02:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
01:20:18:WU04:FS02:0x22: Branch: HEAD
01:20:18:WU04:FS02:0x22: Compiler: GNU 9.4.0
01:20:18:WU04:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:20:18:WU04:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
01:20:18:WU04:FS02:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
01:20:18:WU04:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:20:18:WU04:FS02:0x22: Bits: 64
01:20:18:WU04:FS02:0x22: Mode: Release
01:20:18:WU04:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
01:20:18:WU04:FS02:0x22: <peastman@stanford.edu>
01:20:18:WU04:FS02:0x22: Args: -dir 04 -suffix 01 -version 706 -lifeline 3247 -checkpoint 15
01:20:18:WU04:FS02:0x22: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
01:20:18:WU04:FS02:0x22: nvidia -gpu 0 -gpu-usage 100
01:20:18:WU04:FS02:0x22:************************************ libFAH ************************************
01:20:18:WU04:FS02:0x22: Date: Jan 20 2022
01:20:18:WU04:FS02:0x22: Time: 00:57:22
01:20:18:WU04:FS02:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
01:20:18:WU04:FS02:0x22: Branch: HEAD
01:20:18:WU04:FS02:0x22: Compiler: GNU 9.4.0
01:20:18:WU04:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:20:18:WU04:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
01:20:18:WU04:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:20:18:WU04:FS02:0x22: Bits: 64
01:20:18:WU04:FS02:0x22: Mode: Release
01:20:18:WU04:FS02:0x22:************************************ CBang *************************************
01:20:18:WU04:FS02:0x22: Date: Jan 20 2022
01:20:18:WU04:FS02:0x22: Time: 00:57:00
01:20:18:WU04:FS02:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
01:20:18:WU04:FS02:0x22: Branch: HEAD
01:20:18:WU04:FS02:0x22: Compiler: GNU 9.4.0
01:20:18:WU04:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:20:18:WU04:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
01:20:18:WU04:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:20:18:WU04:FS02:0x22: Bits: 64
01:20:18:WU04:FS02:0x22: Mode: Release
01:20:18:WU04:FS02:0x22:************************************ System ************************************
01:20:18:WU04:FS02:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
01:20:18:WU04:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
01:20:18:WU04:FS02:0x22: CPUs: 32
01:20:18:WU04:FS02:0x22: Memory: 15.60GiB
01:20:18:WU04:FS02:0x22:Free Memory: 10.89GiB
01:20:18:WU04:FS02:0x22: Threads: POSIX_THREADS
01:20:18:WU04:FS02:0x22: OS Version: 5.15
01:20:18:WU04:FS02:0x22:Has Battery: false
01:20:18:WU04:FS02:0x22: On Battery: false
01:20:18:WU04:FS02:0x22: UTC Offset: -5
01:20:18:WU04:FS02:0x22: PID: 3251
01:20:18:WU04:FS02:0x22: CWD: /var/lib/fahclient/work
01:20:18:WU04:FS02:0x22:************************************ OpenMM ************************************
01:20:18:WU04:FS02:0x22: Version: 7.7.0
01:20:18:WU04:FS02:0x22:********************************************************************************
01:20:18:WU04:FS02:0x22:Project: 18450 (Run 0, Clone 305, Gen 37)
01:20:18:WU04:FS02:0x22:Reading tar file core.xml
01:20:18:WU04:FS02:0x22:Reading tar file integrator.xml
01:20:18:WU04:FS02:0x22:Reading tar file state.xml
01:20:18:WU04:FS02:0x22:Reading tar file system.xml
01:20:19:WU04:FS02:0x22:Digital signatures verified
01:20:19:WU04:FS02:0x22:Folding@home GPU Core22 Folding@home Core
01:20:19:WU04:FS02:0x22:Version 0.0.20
01:20:19:WU04:FS02:0x22: Checkpoint write interval: 50000 steps (2%) [50 total]
01:20:19:WU04:FS02:0x22: JSON viewer frame write interval: 25000 steps (1%) [100 total]
01:20:19:WU04:FS02:0x22: XTC frame write interval: 2500000 steps (1e+02%) [1 total]
01:20:19:WU04:FS02:0x22: Global context and integrator variables write interval: disabled
01:20:19:WU04:FS02:0x22:There are 3 platforms available.
01:20:19:WU04:FS02:0x22:Platform 0: Reference
01:20:19:WU04:FS02:0x22:Platform 1: CPU
01:20:19:WU04:FS02:0x22:Platform 2: CUDA
01:20:19:WU04:FS02:0x22: cuda-device 0 specified
01:20:19:WU04:FS02:0x22:opencl-device was set but OpenCL platform could not be found.
01:20:33:WU04:FS02:0x22:Attempting to create CUDA context:
01:20:33:WU04:FS02:0x22: Configuring platform CUDA
01:20:33:WU04:FS02:0x22:Failed to create CUDA context:
01:20:33:WU04:FS02:0x22:Error initializing CUDA: CUDA_ERROR_UNKNOWN (999) at /home/conda/feedstock_root/build_artifacts/openmm_1640541150081/work/platforms/cuda/src/CudaContext.cpp:139
01:20:33:WU04:FS02:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
01:20:33:WU04:FS02:0x22:Saving result file ../logfile_01.txt
01:20:33:WU04:FS02:0x22:Saving result file science.log
01:20:33:WU04:FS02:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
[93m01:20:33:WARNING:WU04:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)[0m
01:20:33:WU04:FS02:Sending unit results: id:04 state:SEND error:FAULTY project:18450 run:0 clone:305 gen:37 core:0x22 unit:0x00000131000000250000481200000000
01:20:33:WU04:FS02:Uploading 11.00KiB to 129.32.209.202
01:20:33:WU04:FS02:Connecting to 129.32.209.202:8080
01:20:33:WU04:FS02:Upload complete
01:20:34:WU04:FS02:Server responded WORK_ACK (400)
01:20:34:WU04:FS02:Cleaning up
01:20:34:WU02:FS02:Connecting to assign1.foldingathome.org:80
01:20:34:WU02:FS02:Assigned to work server 206.223.170.146
01:20:34:WU02:FS02:Requesting new work unit for slot 02: gpu:4:0 GP104GL [Quadro P4000] from 206.223.170.146
01:20:34:WU02:FS02:Connecting to 206.223.170.146:8080
01:20:35:WU02:FS02:Downloading 22.05MiB
01:20:40:WU00:FS00:0xa8:Completed 1450000 out of 2500000 steps (58%)
01:20:41:WU02:FS02:Download 63.50%
01:20:47:WU02:FS02:Download 99.22%
01:20:47:WU02:FS02:Download complete
01:20:47:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:18601 run:12530 clone:2 gen:30 core:0x22 unit:0x000000020000001e000048a9000030f2
01:20:47:WU02:FS02:Starting
01:20:47:WU02:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
01:20:47:WU02:FS02:Started FahCore on PID 3294
01:20:47:WU02:FS02:Core PID:3298
01:20:47:WU02:FS02:FahCore 0x22 started
01:20:47:WU02:FS02:0x22:*********************** Log Started 2022-11-19T01:20:47Z ***********************
01:20:47:WU02:FS02:0x22:*************************** Core22 Folding@home Core ***************************
01:20:47:WU02:FS02:0x22: Core: Core22
01:20:47:WU02:FS02:0x22: Type: 0x22
01:20:47:WU02:FS02:0x22: Version: 0.0.20
01:20:47:WU02:FS02:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:20:47:WU02:FS02:0x22: Copyright: 2020 foldingathome.org
01:20:47:WU02:FS02:0x22: Homepage: https://foldingathome.org/
01:20:47:WU02:FS02:0x22: Date: Jan 20 2022
01:20:47:WU02:FS02:0x22: Time: 00:57:52
01:20:47:WU02:FS02:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
01:20:47:WU02:FS02:0x22: Branch: HEAD
01:20:47:WU02:FS02:0x22: Compiler: GNU 9.4.0
01:20:47:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:20:47:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
01:20:47:WU02:FS02:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
01:20:47:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:20:47:WU02:FS02:0x22: Bits: 64
01:20:47:WU02:FS02:0x22: Mode: Release
01:20:47:WU02:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
01:20:47:WU02:FS02:0x22: <peastman@stanford.edu>
01:20:47:WU02:FS02:0x22: Args: -dir 02 -suffix 01 -version 706 -lifeline 3294 -checkpoint 15
01:20:47:WU02:FS02:0x22: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
01:20:47:WU02:FS02:0x22: nvidia -gpu 0 -gpu-usage 100
01:20:47:WU02:FS02:0x22:************************************ libFAH ************************************
01:20:47:WU02:FS02:0x22: Date: Jan 20 2022
01:20:47:WU02:FS02:0x22: Time: 00:57:22
01:20:47:WU02:FS02:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
01:20:47:WU02:FS02:0x22: Branch: HEAD
01:20:47:WU02:FS02:0x22: Compiler: GNU 9.4.0
01:20:47:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:20:47:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
01:20:47:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:20:47:WU02:FS02:0x22: Bits: 64
01:20:47:WU02:FS02:0x22: Mode: Release
01:20:47:WU02:FS02:0x22:************************************ CBang *************************************
01:20:47:WU02:FS02:0x22: Date: Jan 20 2022
01:20:47:WU02:FS02:0x22: Time: 00:57:00
01:20:47:WU02:FS02:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
01:20:47:WU02:FS02:0x22: Branch: HEAD
01:20:47:WU02:FS02:0x22: Compiler: GNU 9.4.0
01:20:47:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
01:20:47:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
01:20:47:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
01:20:47:WU02:FS02:0x22: Bits: 64
01:20:47:WU02:FS02:0x22: Mode: Release
01:20:47:WU02:FS02:0x22:************************************ System ************************************
01:20:47:WU02:FS02:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
01:20:47:WU02:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
01:20:47:WU02:FS02:0x22: CPUs: 32
01:20:47:WU02:FS02:0x22: Memory: 15.60GiB
01:20:47:WU02:FS02:0x22:Free Memory: 11.00GiB
01:20:47:WU02:FS02:0x22: Threads: POSIX_THREADS
01:20:47:WU02:FS02:0x22: OS Version: 5.15
01:20:47:WU02:FS02:0x22:Has Battery: false
01:20:47:WU02:FS02:0x22: On Battery: false
01:20:47:WU02:FS02:0x22: UTC Offset: -5
01:20:47:WU02:FS02:0x22: PID: 3298
01:20:47:WU02:FS02:0x22: CWD: /var/lib/fahclient/work
01:20:47:WU02:FS02:0x22:************************************ OpenMM ************************************
01:20:47:WU02:FS02:0x22: Version: 7.7.0
01:20:47:WU02:FS02:0x22:********************************************************************************
01:20:47:WU02:FS02:0x22:Project: 18601 (Run 12530, Clone 2, Gen 30)
01:20:47:WU02:FS02:0x22:Reading tar file core.xml
01:20:47:WU02:FS02:0x22:Reading tar file integrator.xml
01:20:47:WU02:FS02:0x22:Reading tar file state.xml.bz2
01:20:47:WU02:FS02:0x22:Reading tar file system.xml.bz2
01:20:47:WU02:FS02:0x22:Digital signatures verified
01:20:47:WU02:FS02:0x22:Folding@home GPU Core22 Folding@home Core
01:20:47:WU02:FS02:0x22:Version 0.0.20
01:20:47:WU02:FS02:0x22: Checkpoint write interval: 25000 steps (2%) [50 total]
01:20:47:WU02:FS02:0x22: JSON viewer frame write interval: 12500 steps (1%) [100 total]
01:20:47:WU02:FS02:0x22: XTC frame write interval: 20000 steps (1.6%) [62 total]
01:20:47:WU02:FS02:0x22: Global context and integrator variables write interval: disabled
01:20:47:WU02:FS02:0x22:There are 3 platforms available.
01:20:47:WU02:FS02:0x22:Platform 0: Reference
01:20:47:WU02:FS02:0x22:Platform 1: CPU
01:20:47:WU02:FS02:0x22:Platform 2: CUDA
01:20:47:WU02:FS02:0x22: cuda-device 0 specified
01:20:47:WU02:FS02:0x22:opencl-device was set but OpenCL platform could not be found.
01:21:06:WU01:FS01:0x22:Completed 487500 out of 1250000 steps (39%)
01:21:10:WU02:FS02:0x22:Attempting to create CUDA context:
01:21:10:WU02:FS02:0x22: Configuring platform CUDA
01:21:10:WU02:FS02:0x22:Failed to create CUDA context:
01:21:10:WU02:FS02:0x22:Error initializing CUDA: CUDA_ERROR_UNKNOWN (999) at /home/conda/feedstock_root/build_artifacts/openmm_1640541150081/work/platforms/cuda/src/CudaContext.cpp:139
01:21:10:WU02:FS02:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
01:21:10:WU02:FS02:0x22:Saving result file ../logfile_01.txt
01:21:10:WU02:FS02:0x22:Saving result file science.log
01:21:10:WU02:FS02:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
[93m01:21:11:WARNING:WU02:FS02:FahCore returned: BAD_WORK_UNIT (114 = 0x72)[0m
01:21:11:WU02:FS02:Sending unit results: id:02 state:SEND error:FAULTY project:18601 run:12530 clone:2 gen:30 core:0x22 unit:0x000000020000001e000048a9000030f2
01:21:11:WU02:FS02:Uploading 11.00KiB to 206.223.170.146
01:21:11:WU02:FS02:Connecting to 206.223.170.146:8080
01:21:11:WU02:FS02:Upload complete
01:21:11:WU02:FS02:Server responded WORK_ACK (400)
01:21:11:WU02:FS02:Cleaning up
01:21:11:WU03:FS02:Connecting to assign1.foldingathome.org:80
01:21:11:WU03:FS02:Assigned to work server 129.32.209.202
01:21:11:WU03:FS02:Requesting new work unit for slot 02: gpu:4:0 GP104GL [Quadro P4000] from 129.32.209.202
01:21:11:WU03:FS02:Connecting to 129.32.209.202:8080
01:21:12:WU03:FS02:Downloading 41.65MiB
01:21:18:WU03:FS02:Download 19.51%
01:21:24:WU03:FS02:Download 38.57%
01:21:30:WU03:FS02:Download 55.07%
01:21:36:WU03:FS02:Download 76.23%
01:21:41:WU03:FS02:Download complete
Ben N1NP
Re: FAULTY project:18450 (Multiple errors)
Posted: Sat Nov 19, 2022 6:01 am
by n1np
Again, same machine, slot 1.
Again, what went wrong here?
Kubuntu 22 LTS, nvidia 520 driver, 2x Quadro P4000.
Code: Select all
*********************** Log Started 2022-11-18T22:36:24Z ***********************
22:36:24:******************************* libFAH ********************************
22:36:24: Date: Oct 20 2020
22:36:24: Time: 20:36:39
22:36:24: Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
22:36:24: Branch: master
22:36:24: Compiler: GNU 8.3.0
22:36:24: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:36:24: -fdata-sections -O3 -funroll-loops -fno-pie
22:36:24: Platform: linux2 5.8.0-1-amd64
22:36:24: Bits: 64
22:36:24: Mode: Release
22:36:24:****************************** FAHClient ******************************
22:36:24: Version: 7.6.21
22:36:24: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:36:24: Copyright: 2020 foldingathome.org
22:36:24: Homepage: https://foldingathome.org/
22:36:24: Date: Oct 20 2020
22:36:24: Time: 20:39:00
22:36:24: Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
22:36:24: Branch: master
22:36:24: Compiler: GNU 8.3.0
22:36:24: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:36:24: -fdata-sections -O3 -funroll-loops -fno-pie
22:36:24: Platform: linux2 5.8.0-1-amd64
22:36:24: Bits: 64
22:36:24: Mode: Release
22:36:24: Args: --child /etc/fahclient/config.xml --run-as fahclient
22:36:24: --pid-file=/var/run/fahclient.pid --daemon
22:36:24: Config: /etc/fahclient/config.xml
22:36:24:******************************** CBang ********************************
22:36:24: Date: Oct 20 2020
22:36:24: Time: 18:37:59
22:36:24: Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
22:36:24: Branch: master
22:36:24: Compiler: GNU 8.3.0
22:36:24: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:36:24: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:36:24: Platform: linux2 5.8.0-1-amd64
22:36:24: Bits: 64
22:36:24: Mode: Release
22:36:24:******************************* System ********************************
22:36:24: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
22:36:24: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
22:36:24: CPUs: 32
22:36:24: Memory: 15.60GiB
22:36:24: Free Memory: 14.36GiB
22:36:24: Threads: POSIX_THREADS
22:36:24: OS Version: 5.15
22:36:24: Has Battery: false
22:36:24: On Battery: false
22:36:24: UTC Offset: -5
22:36:24: PID: 1469
22:36:24: CWD: /var/lib/fahclient
22:36:24: OS: Linux 5.15.0-53-generic x86_64
22:36:24: OS Arch: AMD64
22:36:24: GPUs: 2
22:36:24: GPU 0: Bus:4 Slot:0 Func:0 NVIDIA:5 GP104GL [Quadro P4000]
22:36:24: GPU 1: Bus:33 Slot:0 Func:0 NVIDIA:5 GP104GL [Quadro P4000]
22:36:24: CUDA Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:6.1 Driver:11.8
22:36:24: CUDA Device 1: Platform:0 Device:1 Bus:33 Slot:0 Compute:6.1 Driver:11.8
22:36:24:OpenCL Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:3.0 Driver:520.56
22:36:24:OpenCL Device 1: Platform:0 Device:1 Bus:33 Slot:0 Compute:3.0 Driver:520.56
22:36:24:***********************************************************************
22:36:24:<config>
22:36:24: <!-- Client Control -->
22:36:24: <fold-anon v='true'/>
22:36:24:
22:36:24: <!-- HTTP Server -->
22:36:24: <allow v='192.168.0/24'/>
22:36:24:
22:36:24: <!-- Network -->
22:36:24: <proxy v=':8080'/>
22:36:24:
22:36:24: <!-- Remote Command Server -->
22:36:24: <password v='*****'/>
22:36:24:
22:36:24: <!-- User Information -->
22:36:24: <passkey v='*****'/>
22:36:24: <team v='12912'/>
22:36:24: <user v='n1np'/>
22:36:24:
22:36:24: <!-- Folding Slots -->
22:36:24: <slot id='0' type='CPU'>
22:36:24: <cpus v='28'/>
22:36:24: <paused v='true'/>
22:36:24: </slot>
22:36:24: <slot id='1' type='GPU'>
22:36:24: <paused v='true'/>
22:36:24: <pci-bus v='33'/>
22:36:24: <pci-slot v='0'/>
22:36:24: </slot>
22:36:24: <slot id='2' type='GPU'>
22:36:24: <paused v='true'/>
22:36:24: </slot>
22:36:24:</config>
22:36:24:Trying to access database...
22:36:24:Successfully acquired database lock
22:36:24:FS00:Initialized folding slot 00: cpu:28
22:36:24:FS01:Initialized folding slot 01: gpu:33:0 GP104GL [Quadro P4000]
22:36:24:FS02:Initialized folding slot 02: gpu:4:0 GP104GL [Quadro P4000]
22:37:03:FS00:Unpaused
22:37:03:WU00:FS00:Starting
22:37:03:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx-256/a8-0.0.12/Core_a8.fah/FahCore_a8 -dir 00 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -np 28
22:37:03:WU00:FS00:Started FahCore on PID 2056
22:37:03:WU00:FS00:Core PID:2060
22:37:03:WU00:FS00:FahCore 0xa8 started
22:37:04:WU00:FS00:0xa8:*********************** Log Started 2022-11-18T22:37:04Z ***********************
22:37:04:WU00:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
22:37:04:WU00:FS00:0xa8: Core: Gromacs
22:37:04:WU00:FS00:0xa8: Type: 0xa8
22:37:04:WU00:FS00:0xa8: Version: 0.0.12
22:37:04:WU00:FS00:0xa8: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:37:04:WU00:FS00:0xa8: Copyright: 2020 foldingathome.org
22:37:04:WU00:FS00:0xa8: Homepage: https://foldingathome.org/
22:37:04:WU00:FS00:0xa8: Date: Jan 16 2021
22:37:04:WU00:FS00:0xa8: Time: 19:23:19
22:37:04:WU00:FS00:0xa8: Compiler: GNU 8.3.0
22:37:04:WU00:FS00:0xa8: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
22:37:04:WU00:FS00:0xa8: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:04:WU00:FS00:0xa8: Platform: linux2 4.15.0-128-generic
22:37:04:WU00:FS00:0xa8: Bits: 64
22:37:04:WU00:FS00:0xa8: Mode: Release
22:37:04:WU00:FS00:0xa8: SIMD: avx_256
22:37:04:WU00:FS00:0xa8: OpenMP: ON
22:37:04:WU00:FS00:0xa8: CUDA: OFF
22:37:04:WU00:FS00:0xa8: Args: -dir 00 -suffix 01 -version 706 -lifeline 2056 -checkpoint 15 -np
22:37:04:WU00:FS00:0xa8: 28
22:37:04:WU00:FS00:0xa8:************************************ libFAH ************************************
22:37:04:WU00:FS00:0xa8: Date: Jan 16 2021
22:37:04:WU00:FS00:0xa8: Time: 19:21:38
22:37:04:WU00:FS00:0xa8: Compiler: GNU 8.3.0
22:37:04:WU00:FS00:0xa8: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
22:37:04:WU00:FS00:0xa8: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:04:WU00:FS00:0xa8: Platform: linux2 4.15.0-128-generic
22:37:04:WU00:FS00:0xa8: Bits: 64
22:37:04:WU00:FS00:0xa8: Mode: Release
22:37:04:WU00:FS00:0xa8:************************************ CBang *************************************
22:37:04:WU00:FS00:0xa8: Date: Jan 16 2021
22:37:04:WU00:FS00:0xa8: Time: 19:21:24
22:37:04:WU00:FS00:0xa8: Compiler: GNU 8.3.0
22:37:04:WU00:FS00:0xa8: Options: -faligned-new -std=c++14 -fsigned-char -ffunction-sections
22:37:04:WU00:FS00:0xa8: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:37:04:WU00:FS00:0xa8: Platform: linux2 4.15.0-128-generic
22:37:04:WU00:FS00:0xa8: Bits: 64
22:37:04:WU00:FS00:0xa8: Mode: Release
22:37:04:WU00:FS00:0xa8:************************************ System ************************************
22:37:04:WU00:FS00:0xa8: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
22:37:04:WU00:FS00:0xa8: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
22:37:04:WU00:FS00:0xa8: CPUs: 32
22:37:04:WU00:FS00:0xa8: Memory: 15.60GiB
22:37:04:WU00:FS00:0xa8:Free Memory: 13.61GiB
22:37:04:WU00:FS00:0xa8: Threads: POSIX_THREADS
22:37:04:WU00:FS00:0xa8: OS Version: 5.15
22:37:04:WU00:FS00:0xa8:Has Battery: false
22:37:04:WU00:FS00:0xa8: On Battery: false
22:37:04:WU00:FS00:0xa8: UTC Offset: -5
22:37:04:WU00:FS00:0xa8: PID: 2060
22:37:04:WU00:FS00:0xa8: CWD: /var/lib/fahclient/work
22:37:04:WU00:FS00:0xa8:********************************************************************************
22:37:04:WU00:FS00:0xa8:Project: 16996 (Run 4, Clone 70, Gen 127)
22:37:04:WU00:FS00:0xa8:Unit: 0x00000000000000000000000000000000
22:37:04:WU00:FS00:0xa8:Digital signatures verified
22:37:04:WU00:FS00:0xa8:Calling: mdrun -c frame127.gro -s frame127.tpr -x frame127.xtc -cpi state.cpt -cpt 15 -nt 28 -ntmpi 1
22:37:04:WU00:FS00:0xa8:Steps: first=317500000 total=320000000
22:37:05:WU00:FS00:0xa8:Completed 33152 out of 2500000 steps (1%)
22:37:09:FS01:Unpaused
22:37:09:WU01:FS01:Starting
22:37:09:WU01:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 01 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
22:37:09:WU01:FS01:Started FahCore on PID 2091
22:37:09:WU01:FS01:Core PID:2095
22:37:09:WU01:FS01:FahCore 0x22 started
22:37:10:WU01:FS01:0x22:*********************** Log Started 2022-11-18T22:37:09Z ***********************
22:37:10:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
22:37:10:WU01:FS01:0x22: Core: Core22
22:37:10:WU01:FS01:0x22: Type: 0x22
22:37:10:WU01:FS01:0x22: Version: 0.0.20
22:37:10:WU01:FS01:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:37:10:WU01:FS01:0x22: Copyright: 2020 foldingathome.org
22:37:10:WU01:FS01:0x22: Homepage: https://foldingathome.org/
22:37:10:WU01:FS01:0x22: Date: Jan 20 2022
22:37:10:WU01:FS01:0x22: Time: 00:57:52
22:37:10:WU01:FS01:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
22:37:10:WU01:FS01:0x22: Branch: HEAD
22:37:10:WU01:FS01:0x22: Compiler: GNU 9.4.0
22:37:10:WU01:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:10:WU01:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:10:WU01:FS01:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
22:37:10:WU01:FS01:0x22: Platform: linux 5.11.0-1025-azure
22:37:10:WU01:FS01:0x22: Bits: 64
22:37:10:WU01:FS01:0x22: Mode: Release
22:37:10:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
22:37:10:WU01:FS01:0x22: <peastman@stanford.edu>
22:37:10:WU01:FS01:0x22: Args: -dir 01 -suffix 01 -version 706 -lifeline 2091 -checkpoint 15
22:37:10:WU01:FS01:0x22: -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
22:37:10:WU01:FS01:0x22: nvidia -gpu 1 -gpu-usage 100
22:37:10:WU01:FS01:0x22:************************************ libFAH ************************************
22:37:10:WU01:FS01:0x22: Date: Jan 20 2022
22:37:10:WU01:FS01:0x22: Time: 00:57:22
22:37:10:WU01:FS01:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
22:37:10:WU01:FS01:0x22: Branch: HEAD
22:37:10:WU01:FS01:0x22: Compiler: GNU 9.4.0
22:37:10:WU01:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:10:WU01:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:10:WU01:FS01:0x22: Platform: linux 5.11.0-1025-azure
22:37:10:WU01:FS01:0x22: Bits: 64
22:37:10:WU01:FS01:0x22: Mode: Release
22:37:10:WU01:FS01:0x22:************************************ CBang *************************************
22:37:10:WU01:FS01:0x22: Date: Jan 20 2022
22:37:10:WU01:FS01:0x22: Time: 00:57:00
22:37:10:WU01:FS01:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
22:37:10:WU01:FS01:0x22: Branch: HEAD
22:37:10:WU01:FS01:0x22: Compiler: GNU 9.4.0
22:37:10:WU01:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:10:WU01:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:37:10:WU01:FS01:0x22: Platform: linux 5.11.0-1025-azure
22:37:10:WU01:FS01:0x22: Bits: 64
22:37:10:WU01:FS01:0x22: Mode: Release
22:37:10:WU01:FS01:0x22:************************************ System ************************************
22:37:10:WU01:FS01:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
22:37:10:WU01:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
22:37:10:WU01:FS01:0x22: CPUs: 32
22:37:10:WU01:FS01:0x22: Memory: 15.60GiB
22:37:10:WU01:FS01:0x22:Free Memory: 13.26GiB
22:37:10:WU01:FS01:0x22: Threads: POSIX_THREADS
22:37:10:WU01:FS01:0x22: OS Version: 5.15
22:37:10:WU01:FS01:0x22:Has Battery: false
22:37:10:WU01:FS01:0x22: On Battery: false
22:37:10:WU01:FS01:0x22: UTC Offset: -5
22:37:10:WU01:FS01:0x22: PID: 2095
22:37:10:WU01:FS01:0x22: CWD: /var/lib/fahclient/work
22:37:10:WU01:FS01:0x22:************************************ OpenMM ************************************
22:37:10:WU01:FS01:0x22: Version: 7.7.0
22:37:10:WU01:FS01:0x22:********************************************************************************
22:37:10:WU01:FS01:0x22:Project: 18601 (Run 4814, Clone 2, Gen 20)
22:37:10:WU01:FS01:0x22:Digital signatures verified
22:37:10:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
22:37:10:WU01:FS01:0x22:Version 0.0.20
22:37:10:WU01:FS01:0x22: Checkpoint write interval: 25000 steps (2%) [50 total]
22:37:10:WU01:FS01:0x22: JSON viewer frame write interval: 12500 steps (1%) [100 total]
22:37:10:WU01:FS01:0x22: XTC frame write interval: 20000 steps (1.6%) [62 total]
22:37:10:WU01:FS01:0x22: Global context and integrator variables write interval: disabled
22:37:10:WU01:FS01:0x22:There are 4 platforms available.
22:37:10:WU01:FS01:0x22:Platform 0: Reference
22:37:10:WU01:FS01:0x22:Platform 1: CPU
22:37:10:WU01:FS01:0x22:Platform 2: OpenCL
22:37:10:WU01:FS01:0x22: opencl-device 1 specified
22:37:10:WU01:FS01:0x22:Platform 3: CUDA
22:37:10:WU01:FS01:0x22: cuda-device 1 specified
22:37:16:FS02:Unpaused
22:37:16:WU02:FS02:Connecting to assign1.foldingathome.org:80
22:37:16:WU02:FS02:Assigned to work server 129.32.209.202
22:37:16:WU02:FS02:Requesting new work unit for slot 02: gpu:4:0 GP104GL [Quadro P4000] from 129.32.209.202
22:37:16:WU02:FS02:Connecting to 129.32.209.202:8080
22:37:17:WU02:FS02:Downloading 41.72MiB
22:37:23:WU02:FS02:Download 51.53%
22:37:25:Removing old file 'configs/config-20221115-035206.xml'
22:37:25:Saving configuration to /etc/fahclient/config.xml
22:37:25:<config>
22:37:25: <!-- Client Control -->
22:37:25: <fold-anon v='true'/>
22:37:25:
22:37:25: <!-- HTTP Server -->
22:37:25: <allow v='192.168.0/24'/>
22:37:25:
22:37:25: <!-- Network -->
22:37:25: <proxy v=':8080'/>
22:37:25:
22:37:25: <!-- Remote Command Server -->
22:37:25: <password v='*****'/>
22:37:25:
22:37:25: <!-- User Information -->
22:37:25: <passkey v='*****'/>
22:37:25: <team v='12912'/>
22:37:25: <user v='n1np'/>
22:37:25:
22:37:25: <!-- Folding Slots -->
22:37:25: <slot id='0' type='CPU'>
22:37:25: <cpus v='28'/>
22:37:25: </slot>
22:37:25: <slot id='1' type='GPU'>
22:37:25: <pci-bus v='33'/>
22:37:25: <pci-slot v='0'/>
22:37:25: </slot>
22:37:25: <slot id='2' type='GPU'>
22:37:25: <pci-bus v='4'/>
22:37:25: <pci-slot v='0'/>
22:37:25: </slot>
22:37:25:</config>
22:37:28:WU02:FS02:Download complete
22:37:28:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:18447 run:3 clone:81 gen:275 core:0x22 unit:0x00000051000001130000480f00000003
22:37:28:WU02:FS02:Starting
22:37:28:WU02:FS02:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
22:37:28:WU02:FS02:Started FahCore on PID 2099
22:37:28:WU02:FS02:Core PID:2103
22:37:28:WU02:FS02:FahCore 0x22 started
22:37:28:WU02:FS02:0x22:*********************** Log Started 2022-11-18T22:37:28Z ***********************
22:37:28:WU02:FS02:0x22:*************************** Core22 Folding@home Core ***************************
22:37:28:WU02:FS02:0x22: Core: Core22
22:37:28:WU02:FS02:0x22: Type: 0x22
22:37:28:WU02:FS02:0x22: Version: 0.0.20
22:37:28:WU02:FS02:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:37:28:WU02:FS02:0x22: Copyright: 2020 foldingathome.org
22:37:28:WU02:FS02:0x22: Homepage: https://foldingathome.org/
22:37:28:WU02:FS02:0x22: Date: Jan 20 2022
22:37:28:WU02:FS02:0x22: Time: 00:57:52
22:37:28:WU02:FS02:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
22:37:28:WU02:FS02:0x22: Branch: HEAD
22:37:28:WU02:FS02:0x22: Compiler: GNU 9.4.0
22:37:28:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:28:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:28:WU02:FS02:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
22:37:28:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
22:37:28:WU02:FS02:0x22: Bits: 64
22:37:28:WU02:FS02:0x22: Mode: Release
22:37:28:WU02:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
22:37:28:WU02:FS02:0x22: <peastman@stanford.edu>
22:37:28:WU02:FS02:0x22: Args: -dir 02 -suffix 01 -version 706 -lifeline 2099 -checkpoint 15
22:37:28:WU02:FS02:0x22: -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
22:37:28:WU02:FS02:0x22: nvidia -gpu 0 -gpu-usage 100
22:37:28:WU02:FS02:0x22:************************************ libFAH ************************************
22:37:28:WU02:FS02:0x22: Date: Jan 20 2022
22:37:28:WU02:FS02:0x22: Time: 00:57:22
22:37:28:WU02:FS02:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
22:37:28:WU02:FS02:0x22: Branch: HEAD
22:37:28:WU02:FS02:0x22: Compiler: GNU 9.4.0
22:37:28:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:28:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
22:37:28:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
22:37:28:WU02:FS02:0x22: Bits: 64
22:37:28:WU02:FS02:0x22: Mode: Release
22:37:28:WU02:FS02:0x22:************************************ CBang *************************************
22:37:28:WU02:FS02:0x22: Date: Jan 20 2022
22:37:28:WU02:FS02:0x22: Time: 00:57:00
22:37:28:WU02:FS02:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
22:37:28:WU02:FS02:0x22: Branch: HEAD
22:37:28:WU02:FS02:0x22: Compiler: GNU 9.4.0
22:37:28:WU02:FS02:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
22:37:28:WU02:FS02:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
22:37:28:WU02:FS02:0x22: Platform: linux 5.11.0-1025-azure
22:37:28:WU02:FS02:0x22: Bits: 64
22:37:28:WU02:FS02:0x22: Mode: Release
22:37:28:WU02:FS02:0x22:************************************ System ************************************
22:37:28:WU02:FS02:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
22:37:28:WU02:FS02:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
22:37:28:WU02:FS02:0x22: CPUs: 32
22:37:28:WU02:FS02:0x22: Memory: 15.60GiB
22:37:28:WU02:FS02:0x22:Free Memory: 12.50GiB
22:37:28:WU02:FS02:0x22: Threads: POSIX_THREADS
22:37:28:WU02:FS02:0x22: OS Version: 5.15
22:37:28:WU02:FS02:0x22:Has Battery: false
22:37:28:WU02:FS02:0x22: On Battery: false
22:37:28:WU02:FS02:0x22: UTC Offset: -5
22:37:28:WU02:FS02:0x22: PID: 2103
22:37:28:WU02:FS02:0x22: CWD: /var/lib/fahclient/work
22:37:28:WU02:FS02:0x22:************************************ OpenMM ************************************
22:37:28:WU02:FS02:0x22: Version: 7.7.0
22:37:28:WU02:FS02:0x22:********************************************************************************
22:37:28:WU02:FS02:0x22:Project: 18447 (Run 3, Clone 81, Gen 275)
22:37:28:WU02:FS02:0x22:Reading tar file core.xml
22:37:28:WU02:FS02:0x22:Reading tar file integrator.xml
22:37:28:WU02:FS02:0x22:Reading tar file state.xml
22:37:28:WU02:FS02:0x22:Reading tar file system.xml
22:37:29:WU02:FS02:0x22:Digital signatures verified
22:37:29:WU02:FS02:0x22:Folding@home GPU Core22 Folding@home Core
22:37:29:WU02:FS02:0x22:Version 0.0.20
22:37:29:WU02:FS02:0x22: Checkpoint write interval: 50000 steps (2%) [50 total]
22:37:29:WU02:FS02:0x22: JSON viewer frame write interval: 25000 steps (1%) [100 total]
22:37:29:WU02:FS02:0x22: XTC frame write interval: 2500000 steps (1e+02%) [1 total]
22:37:29:WU02:FS02:0x22: Global context and integrator variables write interval: disabled
22:37:29:WU02:FS02:0x22:There are 4 platforms available.
22:37:29:WU02:FS02:0x22:Platform 0: Reference
22:37:29:WU02:FS02:0x22:Platform 1: CPU
22:37:29:WU02:FS02:0x22:Platform 2: OpenCL
22:37:29:WU02:FS02:0x22: opencl-device 0 specified
22:37:29:WU02:FS02:0x22:Platform 3: CUDA
22:37:29:WU02:FS02:0x22: cuda-device 0 specified
22:37:35:WU02:FS02:0x22:Attempting to create CUDA context:
22:37:35:WU02:FS02:0x22: Configuring platform CUDA
22:37:35:WU01:FS01:0x22:Attempting to create CUDA context:
22:37:35:WU01:FS01:0x22: Configuring platform CUDA
22:37:50:WU02:FS02:0x22: Using CUDA and gpu 0
22:37:50:WU02:FS02:0x22:Completed 0 out of 2500000 steps (0%)
22:37:50:WU02:FS02:0x22:Checkpoint completed at step 0
22:38:04:WU01:FS01:0x22: Using CUDA and gpu 1
22:38:04:WU01:FS01:0x22:Completed 0 out of 1250000 steps (0%)
22:38:06:WU01:FS01:0x22:Checkpoint completed at step 0
01:26:19:WU00:FS00:0xa8:Completed 1500000 out of 2500000 steps (60%)
01:29:05:WU00:FS00:0xa8:Completed 1525000 out of 2500000 steps (61%)
01:29:36:WU01:FS01:0x22:Completed 512500 out of 1250000 steps (41%)
01:31:51:WU00:FS00:0xa8:Completed 1550000 out of 2500000 steps (62%)
01:33:56:WU01:FS01:0x22:Completed 525000 out of 1250000 steps (42%)
01:33:59:WU01:FS01:0x22:Checkpoint completed at step 525000
01:34:40:WU00:FS00:0xa8:Completed 1575000 out of 2500000 steps (63%)
01:37:26:WU00:FS00:0xa8:Completed 1600000 out of 2500000 steps (64%)
01:38:17:WU01:FS01:0x22:Completed 537500 out of 1250000 steps (43%)
01:40:11:WU00:FS00:0xa8:Completed 1625000 out of 2500000 steps (65%)
01:42:29:WU01:FS01:0x22:Completed 550000 out of 1250000 steps (44%)
01:42:32:WU01:FS01:0x22:Checkpoint completed at step 550000
01:42:59:WU00:FS00:0xa8:Completed 1650000 out of 2500000 steps (66%)
01:45:44:WU00:FS00:0xa8:Completed 1675000 out of 2500000 steps (67%)
01:46:44:WU01:FS01:0x22:Completed 562500 out of 1250000 steps (45%)
01:48:28:WU00:FS00:0xa8:Completed 1700000 out of 2500000 steps (68%)
01:50:55:WU01:FS01:0x22:Completed 575000 out of 1250000 steps (46%)
01:50:58:WU01:FS01:0x22:Checkpoint completed at step 575000
01:51:17:WU00:FS00:0xa8:Completed 1725000 out of 2500000 steps (69%)
01:54:04:WU00:FS00:0xa8:Completed 1750000 out of 2500000 steps (70%)
01:55:07:WU01:FS01:0x22:Completed 587500 out of 1250000 steps (47%)
01:56:51:WU00:FS00:0xa8:Completed 1775000 out of 2500000 steps (71%)
01:59:24:WU01:FS01:0x22:Completed 600000 out of 1250000 steps (48%)
01:59:27:WU01:FS01:0x22:Checkpoint completed at step 600000
01:59:41:WU00:FS00:0xa8:Completed 1800000 out of 2500000 steps (72%)
02:02:28:WU00:FS00:0xa8:Completed 1825000 out of 2500000 steps (73%)
02:03:40:WU01:FS01:0x22:Completed 612500 out of 1250000 steps (49%)
02:05:15:WU00:FS00:0xa8:Completed 1850000 out of 2500000 steps (74%)
02:07:55:WU01:FS01:0x22:Completed 625000 out of 1250000 steps (50%)
02:07:57:WU01:FS01:0x22:Checkpoint completed at step 625000
02:08:05:WU00:FS00:0xa8:Completed 1875000 out of 2500000 steps (75%)
02:10:52:WU00:FS00:0xa8:Completed 1900000 out of 2500000 steps (76%)
02:12:10:WU01:FS01:0x22:Completed 637500 out of 1250000 steps (51%)
02:13:39:WU00:FS00:0xa8:Completed 1925000 out of 2500000 steps (77%)
02:16:24:WU01:FS01:0x22:Completed 650000 out of 1250000 steps (52%)
02:16:27:WU01:FS01:0x22:Checkpoint completed at step 650000
02:16:29:WU00:FS00:0xa8:Completed 1950000 out of 2500000 steps (78%)
02:19:15:WU00:FS00:0xa8:Completed 1975000 out of 2500000 steps (79%)
02:20:43:WU01:FS01:0x22:Completed 662500 out of 1250000 steps (53%)
02:22:02:WU00:FS00:0xa8:Completed 2000000 out of 2500000 steps (80%)
02:24:49:WU00:FS00:0xa8:Completed 2025000 out of 2500000 steps (81%)
02:24:57:WU01:FS01:0x22:Completed 675000 out of 1250000 steps (54%)
02:25:00:WU01:FS01:0x22:Checkpoint completed at step 675000
02:27:38:WU00:FS00:0xa8:Completed 2050000 out of 2500000 steps (82%)
02:29:14:WU01:FS01:0x22:Completed 687500 out of 1250000 steps (55%)
02:30:24:WU00:FS00:0xa8:Completed 2075000 out of 2500000 steps (83%)
02:33:10:WU00:FS00:0xa8:Completed 2100000 out of 2500000 steps (84%)
02:33:27:WU01:FS01:0x22:Completed 700000 out of 1250000 steps (56%)
02:33:30:WU01:FS01:0x22:Checkpoint completed at step 700000
02:35:59:WU00:FS00:0xa8:Completed 2125000 out of 2500000 steps (85%)
02:37:42:WU01:FS01:0x22:Completed 712500 out of 1250000 steps (57%)
02:38:45:WU00:FS00:0xa8:Completed 2150000 out of 2500000 steps (86%)
02:41:32:WU00:FS00:0xa8:Completed 2175000 out of 2500000 steps (87%)
02:41:57:WU01:FS01:0x22:Completed 725000 out of 1250000 steps (58%)
02:42:00:WU01:FS01:0x22:Checkpoint completed at step 725000
02:44:21:WU00:FS00:0xa8:Completed 2200000 out of 2500000 steps (88%)
02:46:14:WU01:FS01:0x22:Completed 737500 out of 1250000 steps (59%)
02:47:08:WU00:FS00:0xa8:Completed 2225000 out of 2500000 steps (89%)
02:49:54:WU00:FS00:0xa8:Completed 2250000 out of 2500000 steps (90%)
02:50:27:WU01:FS01:0x22:Completed 750000 out of 1250000 steps (60%)
02:50:30:WU01:FS01:0x22:Checkpoint completed at step 750000
02:52:43:WU00:FS00:0xa8:Completed 2275000 out of 2500000 steps (91%)
02:54:51:WU01:FS01:0x22:Completed 762500 out of 1250000 steps (61%)
02:55:29:WU00:FS00:0xa8:Completed 2300000 out of 2500000 steps (92%)
02:58:14:WU00:FS00:0xa8:Completed 2325000 out of 2500000 steps (93%)
02:59:08:WU01:FS01:0x22:Completed 775000 out of 1250000 steps (62%)
02:59:11:WU01:FS01:0x22:Checkpoint completed at step 775000
03:01:02:WU00:FS00:0xa8:Completed 2350000 out of 2500000 steps (94%)
03:03:23:WU01:FS01:0x22:Completed 787500 out of 1250000 steps (63%)
03:03:48:WU00:FS00:0xa8:Completed 2375000 out of 2500000 steps (95%)
03:06:34:WU00:FS00:0xa8:Completed 2400000 out of 2500000 steps (96%)
03:07:40:WU01:FS01:0x22:Completed 800000 out of 1250000 steps (64%)
03:07:43:WU01:FS01:0x22:Checkpoint completed at step 800000
03:09:23:WU00:FS00:0xa8:Completed 2425000 out of 2500000 steps (97%)
03:11:57:WU01:FS01:0x22:Completed 812500 out of 1250000 steps (65%)
03:12:09:WU00:FS00:0xa8:Completed 2450000 out of 2500000 steps (98%)
03:14:54:WU00:FS00:0xa8:Completed 2475000 out of 2500000 steps (99%)
03:16:06:WU01:FS01:0x22:Completed 825000 out of 1250000 steps (66%)
03:16:09:WU01:FS01:0x22:Checkpoint completed at step 825000
03:17:42:WU00:FS00:0xa8:Completed 2500000 out of 2500000 steps (100%)
03:17:43:WU00:FS00:0xa8:Saving result file ../logfile_01.txt
03:17:43:WU00:FS00:0xa8:Saving result file frame127.gro
03:17:43:WU00:FS00:0xa8:Saving result file frame127.xtc
03:17:43:WU00:FS00:0xa8:Saving result file md.log
03:17:43:WU00:FS00:0xa8:Saving result file science.log
03:17:43:WU00:FS00:0xa8:Saving result file state.cpt
03:17:43:WU00:FS00:0xa8:Folding@home Core Shutdown: FINISHED_UNIT
03:17:43:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
03:17:43:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16996 run:4 clone:70 gen:127 core:0xa8 unit:0x000000460000007f0000426400000004
03:17:43:WU00:FS00:Uploading 5.15MiB to 129.32.209.205
03:17:43:WU00:FS00:Connecting to 129.32.209.205:8080
03:17:49:WU00:FS00:Upload 72.88%
03:17:52:WU00:FS00:Upload complete
03:17:52:WU00:FS00:Server responded WORK_ACK (400)
03:17:52:WU00:FS00:Final credit estimate, 40770.00 points
03:17:52:WU00:FS00:Cleaning up
03:20:23:WU01:FS01:0x22:Completed 837500 out of 1250000 steps (67%)
03:24:38:WU01:FS01:0x22:Completed 850000 out of 1250000 steps (68%)
03:24:41:WU01:FS01:0x22:Checkpoint completed at step 850000
03:28:56:WU01:FS01:0x22:Completed 862500 out of 1250000 steps (69%)
03:33:09:WU01:FS01:0x22:Completed 875000 out of 1250000 steps (70%)
03:33:12:WU01:FS01:0x22:Checkpoint completed at step 875000
03:37:26:WU01:FS01:0x22:Completed 887500 out of 1250000 steps (71%)
03:41:38:WU01:FS01:0x22:Completed 900000 out of 1250000 steps (72%)
03:41:41:WU01:FS01:0x22:Checkpoint completed at step 900000
03:45:48:WU01:FS01:0x22:Completed 912500 out of 1250000 steps (73%)
03:50:00:WU01:FS01:0x22:Completed 925000 out of 1250000 steps (74%)
03:50:02:WU01:FS01:0x22:Checkpoint completed at step 925000
03:54:12:WU01:FS01:0x22:Completed 937500 out of 1250000 steps (75%)
03:58:20:WU01:FS01:0x22:Completed 950000 out of 1250000 steps (76%)
03:58:23:WU01:FS01:0x22:Checkpoint completed at step 950000
04:02:35:WU01:FS01:0x22:Completed 962500 out of 1250000 steps (77%)
04:06:46:WU01:FS01:0x22:Completed 975000 out of 1250000 steps (78%)
04:06:49:WU01:FS01:0x22:Checkpoint completed at step 975000
04:11:05:WU01:FS01:0x22:Completed 987500 out of 1250000 steps (79%)
04:15:18:WU01:FS01:0x22:Completed 1000000 out of 1250000 steps (80%)
04:15:21:WU01:FS01:0x22:Checkpoint completed at step 1000000
04:19:35:WU01:FS01:0x22:Completed 1012500 out of 1250000 steps (81%)
04:23:52:WU01:FS01:0x22:Completed 1025000 out of 1250000 steps (82%)
04:23:55:WU01:FS01:0x22:Checkpoint completed at step 1025000
04:28:10:WU01:FS01:0x22:Completed 1037500 out of 1250000 steps (83%)
04:32:24:WU01:FS01:0x22:Completed 1050000 out of 1250000 steps (84%)
04:32:27:WU01:FS01:0x22:Checkpoint completed at step 1050000
******************************* Date: 2022-11-19 *******************************
04:36:37:WU01:FS01:0x22:Completed 1062500 out of 1250000 steps (85%)
04:40:44:WU01:FS01:0x22:Completed 1075000 out of 1250000 steps (86%)
04:40:47:WU01:FS01:0x22:Checkpoint completed at step 1075000
04:44:57:WU01:FS01:0x22:Completed 1087500 out of 1250000 steps (87%)
04:49:09:WU01:FS01:0x22:Completed 1100000 out of 1250000 steps (88%)
04:49:12:WU01:FS01:0x22:Checkpoint completed at step 1100000
04:53:19:WU01:FS01:0x22:Completed 1112500 out of 1250000 steps (89%)
04:57:27:WU01:FS01:0x22:Completed 1125000 out of 1250000 steps (90%)
04:57:30:WU01:FS01:0x22:Checkpoint completed at step 1125000
05:01:36:WU01:FS01:0x22:Completed 1137500 out of 1250000 steps (91%)
05:05:41:WU01:FS01:0x22:Completed 1150000 out of 1250000 steps (92%)
05:05:43:WU01:FS01:0x22:Checkpoint completed at step 1150000
05:09:55:WU01:FS01:0x22:Completed 1162500 out of 1250000 steps (93%)
05:14:08:WU01:FS01:0x22:Completed 1175000 out of 1250000 steps (94%)
05:14:11:WU01:FS01:0x22:Checkpoint completed at step 1175000
05:18:13:WU01:FS01:0x22:Completed 1187500 out of 1250000 steps (95%)
05:22:16:WU01:FS01:0x22:Completed 1200000 out of 1250000 steps (96%)
05:22:19:WU01:FS01:0x22:Checkpoint completed at step 1200000
05:26:31:WU01:FS01:0x22:Completed 1212500 out of 1250000 steps (97%)
05:30:35:WU01:FS01:0x22:Completed 1225000 out of 1250000 steps (98%)
05:30:38:WU01:FS01:0x22:Checkpoint completed at step 1225000
05:34:45:WU01:FS01:0x22:Completed 1237500 out of 1250000 steps (99%)
05:34:46:WU00:FS01:Connecting to assign1.foldingathome.org:80
05:34:47:WU00:FS01:Assigned to work server 129.32.209.202
05:34:47:WU00:FS01:Requesting new work unit for slot 01: gpu:33:0 GP104GL [Quadro P4000] from 129.32.209.202
05:34:47:WU00:FS01:Connecting to 129.32.209.202:8080
05:34:47:WU00:FS01:Downloading 17.43MiB
05:34:51:WU00:FS01:Download complete
05:34:51:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:18454 run:20 clone:7 gen:143 core:0x22 unit:0x000000070000008f0000481600000014
05:38:51:WU01:FS01:0x22:Completed 1250000 out of 1250000 steps (100%)
05:38:51:WU01:FS01:0x22:Average performance: 8.76268 ns/day
05:38:54:WU01:FS01:0x22:Checkpoint completed at step 1250000
05:39:10:WU01:FS01:0x22:Saving result file ../logfile_01.txt
05:39:10:WU01:FS01:0x22:Saving result file checkpointIntegrator.xml
05:39:10:WU01:FS01:0x22:Saving result file checkpointState.xml.bz2
05:39:11:WU01:FS01:0x22:Saving result file positions.xtc
05:39:11:WU01:FS01:0x22:Saving result file science.log
05:39:11:WU01:FS01:0x22:Saving result file xtcAtoms.csv.bz2
05:39:11:WU01:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
05:39:12:WU01:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:39:12:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:18601 run:4814 clone:2 gen:20 core:0x22 unit:0x0000000200000014000048a9000012ce
05:39:12:WU01:FS01:Uploading 22.93MiB to 206.223.170.146
05:39:12:WU01:FS01:Connecting to 206.223.170.146:8080
05:39:12:WU00:FS01:Starting
05:39:12:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
05:39:12:WU00:FS01:Started FahCore on PID 3780
05:39:12:WU00:FS01:Core PID:3784
05:39:12:WU00:FS01:FahCore 0x22 started
05:39:13:WU00:FS01:0x22:*********************** Log Started 2022-11-19T05:39:12Z ***********************
05:39:13:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
05:39:13:WU00:FS01:0x22: Core: Core22
05:39:13:WU00:FS01:0x22: Type: 0x22
05:39:13:WU00:FS01:0x22: Version: 0.0.20
05:39:13:WU00:FS01:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
05:39:13:WU00:FS01:0x22: Copyright: 2020 foldingathome.org
05:39:13:WU00:FS01:0x22: Homepage: https://foldingathome.org/
05:39:13:WU00:FS01:0x22: Date: Jan 20 2022
05:39:13:WU00:FS01:0x22: Time: 00:57:52
05:39:13:WU00:FS01:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
05:39:13:WU00:FS01:0x22: Branch: HEAD
05:39:13:WU00:FS01:0x22: Compiler: GNU 9.4.0
05:39:13:WU00:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:39:13:WU00:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
05:39:13:WU00:FS01:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
05:39:13:WU00:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:39:13:WU00:FS01:0x22: Bits: 64
05:39:13:WU00:FS01:0x22: Mode: Release
05:39:13:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
05:39:13:WU00:FS01:0x22: <peastman@stanford.edu>
05:39:13:WU00:FS01:0x22: Args: -dir 00 -suffix 01 -version 706 -lifeline 3780 -checkpoint 15
05:39:13:WU00:FS01:0x22: -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
05:39:13:WU00:FS01:0x22: nvidia -gpu 1 -gpu-usage 100
05:39:13:WU00:FS01:0x22:************************************ libFAH ************************************
05:39:13:WU00:FS01:0x22: Date: Jan 20 2022
05:39:13:WU00:FS01:0x22: Time: 00:57:22
05:39:13:WU00:FS01:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
05:39:13:WU00:FS01:0x22: Branch: HEAD
05:39:13:WU00:FS01:0x22: Compiler: GNU 9.4.0
05:39:13:WU00:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:39:13:WU00:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
05:39:13:WU00:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:39:13:WU00:FS01:0x22: Bits: 64
05:39:13:WU00:FS01:0x22: Mode: Release
05:39:13:WU00:FS01:0x22:************************************ CBang *************************************
05:39:13:WU00:FS01:0x22: Date: Jan 20 2022
05:39:13:WU00:FS01:0x22: Time: 00:57:00
05:39:13:WU00:FS01:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
05:39:13:WU00:FS01:0x22: Branch: HEAD
05:39:13:WU00:FS01:0x22: Compiler: GNU 9.4.0
05:39:13:WU00:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:39:13:WU00:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
05:39:13:WU00:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:39:13:WU00:FS01:0x22: Bits: 64
05:39:13:WU00:FS01:0x22: Mode: Release
05:39:13:WU00:FS01:0x22:************************************ System ************************************
05:39:13:WU00:FS01:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
05:39:13:WU00:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
05:39:13:WU00:FS01:0x22: CPUs: 32
05:39:13:WU00:FS01:0x22: Memory: 15.60GiB
05:39:13:WU00:FS01:0x22:Free Memory: 13.37GiB
05:39:13:WU00:FS01:0x22: Threads: POSIX_THREADS
05:39:13:WU00:FS01:0x22: OS Version: 5.15
05:39:13:WU00:FS01:0x22:Has Battery: false
05:39:13:WU00:FS01:0x22: On Battery: false
05:39:13:WU00:FS01:0x22: UTC Offset: -5
05:39:13:WU00:FS01:0x22: PID: 3784
05:39:13:WU00:FS01:0x22: CWD: /var/lib/fahclient/work
05:39:13:WU00:FS01:0x22:************************************ OpenMM ************************************
05:39:13:WU00:FS01:0x22: Version: 7.7.0
05:39:13:WU00:FS01:0x22:********************************************************************************
05:39:13:WU00:FS01:0x22:Project: 18454 (Run 20, Clone 7, Gen 143)
05:39:13:WU00:FS01:0x22:Reading tar file core.xml
05:39:13:WU00:FS01:0x22:Reading tar file integrator.xml
05:39:13:WU00:FS01:0x22:Reading tar file state.xml
05:39:13:WU00:FS01:0x22:Reading tar file system.xml
05:39:13:WU00:FS01:0x22:Digital signatures verified
05:39:13:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
05:39:13:WU00:FS01:0x22:Version 0.0.20
05:39:13:WU00:FS01:0x22: Checkpoint write interval: 100000 steps (2%) [50 total]
05:39:13:WU00:FS01:0x22: JSON viewer frame write interval: 50000 steps (1%) [100 total]
05:39:13:WU00:FS01:0x22: XTC frame write interval: 500000 steps (10%) [10 total]
05:39:13:WU00:FS01:0x22: Global context and integrator variables write interval: disabled
05:39:13:WU00:FS01:0x22:There are 3 platforms available.
05:39:13:WU00:FS01:0x22:Platform 0: Reference
05:39:13:WU00:FS01:0x22:Platform 1: CPU
05:39:13:WU00:FS01:0x22:Platform 2: CUDA
05:39:13:WU00:FS01:0x22: cuda-device 1 specified
05:39:13:WU00:FS01:0x22:opencl-device was set but OpenCL platform could not be found.
05:39:14:WU00:FS01:0x22:Attempting to create CUDA context:
05:39:14:WU00:FS01:0x22: Configuring platform CUDA
05:39:14:WU00:FS01:0x22:Failed to create CUDA context:
05:39:14:WU00:FS01:0x22:Error initializing CUDA: CUDA_ERROR_UNKNOWN (999) at /home/conda/feedstock_root/build_artifacts/openmm_1640541150081/work/platforms/cuda/src/CudaContext.cpp:139
05:39:14:WU00:FS01:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
05:39:14:WU00:FS01:0x22:Saving result file ../logfile_01.txt
05:39:14:WU00:FS01:0x22:Saving result file science.log
05:39:14:WU00:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
[93m05:39:14:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)[0m
05:39:14:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:18454 run:20 clone:7 gen:143 core:0x22 unit:0x000000070000008f0000481600000014
05:39:14:WU00:FS01:Uploading 11.00KiB to 129.32.209.202
05:39:14:WU00:FS01:Connecting to 129.32.209.202:8080
05:39:15:WU00:FS01:Upload complete
05:39:15:WU02:FS01:Connecting to assign1.foldingathome.org:80
05:39:15:WU00:FS01:Server responded WORK_ACK (400)
05:39:15:WU00:FS01:Cleaning up
05:39:17:WU02:FS01:Assigned to work server 206.223.170.146
05:39:17:WU02:FS01:Requesting new work unit for slot 01: gpu:33:0 GP104GL [Quadro P4000] from 206.223.170.146
05:39:17:WU02:FS01:Connecting to 206.223.170.146:8080
05:39:18:WU01:FS01:Upload 16.90%
05:39:18:WU02:FS01:Downloading 22.04MiB
05:39:24:WU02:FS01:Download 19.28%
05:39:24:WU01:FS01:Upload 31.89%
05:39:30:WU02:FS01:Download 41.96%
05:39:30:WU01:FS01:Upload 46.61%
05:39:36:WU02:FS01:Download 68.61%
05:39:36:WU01:FS01:Upload 61.60%
05:39:40:WU02:FS01:Download complete
05:39:40:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:18601 run:6829 clone:2 gen:30 core:0x22 unit:0x000000020000001e000048a900001aad
05:39:40:WU02:FS01:Starting
05:39:40:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
05:39:40:WU02:FS01:Started FahCore on PID 3822
05:39:40:WU02:FS01:Core PID:3826
05:39:40:WU02:FS01:FahCore 0x22 started
05:39:40:WU02:FS01:0x22:*********************** Log Started 2022-11-19T05:39:40Z ***********************
05:39:40:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
05:39:40:WU02:FS01:0x22: Core: Core22
05:39:40:WU02:FS01:0x22: Type: 0x22
05:39:40:WU02:FS01:0x22: Version: 0.0.20
05:39:40:WU02:FS01:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
05:39:40:WU02:FS01:0x22: Copyright: 2020 foldingathome.org
05:39:40:WU02:FS01:0x22: Homepage: https://foldingathome.org/
05:39:40:WU02:FS01:0x22: Date: Jan 20 2022
05:39:40:WU02:FS01:0x22: Time: 00:57:52
05:39:40:WU02:FS01:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
05:39:40:WU02:FS01:0x22: Branch: HEAD
05:39:40:WU02:FS01:0x22: Compiler: GNU 9.4.0
05:39:40:WU02:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:39:40:WU02:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
05:39:40:WU02:FS01:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
05:39:40:WU02:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:39:40:WU02:FS01:0x22: Bits: 64
05:39:40:WU02:FS01:0x22: Mode: Release
05:39:40:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
05:39:40:WU02:FS01:0x22: <peastman@stanford.edu>
05:39:40:WU02:FS01:0x22: Args: -dir 02 -suffix 01 -version 706 -lifeline 3822 -checkpoint 15
05:39:40:WU02:FS01:0x22: -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
05:39:40:WU02:FS01:0x22: nvidia -gpu 1 -gpu-usage 100
05:39:40:WU02:FS01:0x22:************************************ libFAH ************************************
05:39:40:WU02:FS01:0x22: Date: Jan 20 2022
05:39:40:WU02:FS01:0x22: Time: 00:57:22
05:39:40:WU02:FS01:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
05:39:40:WU02:FS01:0x22: Branch: HEAD
05:39:40:WU02:FS01:0x22: Compiler: GNU 9.4.0
05:39:40:WU02:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:39:40:WU02:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
05:39:40:WU02:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:39:40:WU02:FS01:0x22: Bits: 64
05:39:40:WU02:FS01:0x22: Mode: Release
05:39:40:WU02:FS01:0x22:************************************ CBang *************************************
05:39:40:WU02:FS01:0x22: Date: Jan 20 2022
05:39:40:WU02:FS01:0x22: Time: 00:57:00
05:39:40:WU02:FS01:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
05:39:40:WU02:FS01:0x22: Branch: HEAD
05:39:40:WU02:FS01:0x22: Compiler: GNU 9.4.0
05:39:40:WU02:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:39:40:WU02:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
05:39:40:WU02:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:39:40:WU02:FS01:0x22: Bits: 64
05:39:40:WU02:FS01:0x22: Mode: Release
05:39:40:WU02:FS01:0x22:************************************ System ************************************
05:39:40:WU02:FS01:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
05:39:40:WU02:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
05:39:40:WU02:FS01:0x22: CPUs: 32
05:39:40:WU02:FS01:0x22: Memory: 15.60GiB
05:39:40:WU02:FS01:0x22:Free Memory: 13.37GiB
05:39:40:WU02:FS01:0x22: Threads: POSIX_THREADS
05:39:40:WU02:FS01:0x22: OS Version: 5.15
05:39:40:WU02:FS01:0x22:Has Battery: false
05:39:40:WU02:FS01:0x22: On Battery: false
05:39:40:WU02:FS01:0x22: UTC Offset: -5
05:39:40:WU02:FS01:0x22: PID: 3826
05:39:40:WU02:FS01:0x22: CWD: /var/lib/fahclient/work
05:39:40:WU02:FS01:0x22:************************************ OpenMM ************************************
05:39:40:WU02:FS01:0x22: Version: 7.7.0
05:39:40:WU02:FS01:0x22:********************************************************************************
05:39:40:WU02:FS01:0x22:Project: 18601 (Run 6829, Clone 2, Gen 30)
05:39:40:WU02:FS01:0x22:Reading tar file core.xml
05:39:40:WU02:FS01:0x22:Reading tar file integrator.xml
05:39:40:WU02:FS01:0x22:Reading tar file state.xml.bz2
05:39:40:WU02:FS01:0x22:Reading tar file system.xml.bz2
05:39:40:WU02:FS01:0x22:Digital signatures verified
05:39:40:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
05:39:40:WU02:FS01:0x22:Version 0.0.20
05:39:40:WU02:FS01:0x22: Checkpoint write interval: 25000 steps (2%) [50 total]
05:39:40:WU02:FS01:0x22: JSON viewer frame write interval: 12500 steps (1%) [100 total]
05:39:40:WU02:FS01:0x22: XTC frame write interval: 20000 steps (1.6%) [62 total]
05:39:40:WU02:FS01:0x22: Global context and integrator variables write interval: disabled
05:39:40:WU02:FS01:0x22:There are 3 platforms available.
05:39:40:WU02:FS01:0x22:Platform 0: Reference
05:39:40:WU02:FS01:0x22:Platform 1: CPU
05:39:40:WU02:FS01:0x22:Platform 2: CUDA
05:39:40:WU02:FS01:0x22: cuda-device 1 specified
05:39:40:WU02:FS01:0x22:opencl-device was set but OpenCL platform could not be found.
05:39:42:WU01:FS01:Upload 75.50%
05:39:48:WU01:FS01:Upload 91.58%
05:39:52:WU01:FS01:Upload complete
05:39:52:WU01:FS01:Server responded WORK_ACK (400)
05:39:52:WU01:FS01:Final credit estimate, 287206.00 points
05:39:52:WU01:FS01:Cleaning up
05:40:00:WU02:FS01:0x22:Attempting to create CUDA context:
05:40:00:WU02:FS01:0x22: Configuring platform CUDA
05:40:00:WU02:FS01:0x22:Failed to create CUDA context:
05:40:00:WU02:FS01:0x22:Error initializing CUDA: CUDA_ERROR_UNKNOWN (999) at /home/conda/feedstock_root/build_artifacts/openmm_1640541150081/work/platforms/cuda/src/CudaContext.cpp:139
05:40:00:WU02:FS01:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
05:40:00:WU02:FS01:0x22:Saving result file ../logfile_01.txt
05:40:00:WU02:FS01:0x22:Saving result file science.log
05:40:00:WU02:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
[93m05:40:00:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)[0m
05:40:00:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:18601 run:6829 clone:2 gen:30 core:0x22 unit:0x000000020000001e000048a900001aad
05:40:00:WU02:FS01:Uploading 11.00KiB to 206.223.170.146
05:40:00:WU02:FS01:Connecting to 206.223.170.146:8080
05:40:00:WU02:FS01:Upload complete
05:40:00:WU02:FS01:Server responded WORK_ACK (400)
05:40:00:WU02:FS01:Cleaning up
05:40:00:WU00:FS01:Connecting to assign1.foldingathome.org:80
05:40:01:WU00:FS01:Assigned to work server 206.223.170.146
05:40:01:WU00:FS01:Requesting new work unit for slot 01: gpu:33:0 GP104GL [Quadro P4000] from 206.223.170.146
05:40:01:WU00:FS01:Connecting to 206.223.170.146:8080
05:40:01:WU00:FS01:Downloading 22.07MiB
05:40:07:WU00:FS01:Download 94.60%
05:40:07:WU00:FS01:Download complete
05:40:07:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:18601 run:13163 clone:2 gen:25 core:0x22 unit:0x0000000200000019000048a90000336b
05:40:07:WU00:FS01:Starting
05:40:07:WU00:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.20/Core_22.fah/FahCore_22 -dir 00 -suffix 01 -version 706 -lifeline 1469 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
05:40:07:WU00:FS01:Started FahCore on PID 3864
05:40:07:WU00:FS01:Core PID:3868
05:40:07:WU00:FS01:FahCore 0x22 started
05:40:07:WU00:FS01:0x22:*********************** Log Started 2022-11-19T05:40:07Z ***********************
05:40:07:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
05:40:07:WU00:FS01:0x22: Core: Core22
05:40:07:WU00:FS01:0x22: Type: 0x22
05:40:07:WU00:FS01:0x22: Version: 0.0.20
05:40:07:WU00:FS01:0x22: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
05:40:07:WU00:FS01:0x22: Copyright: 2020 foldingathome.org
05:40:07:WU00:FS01:0x22: Homepage: https://foldingathome.org/
05:40:07:WU00:FS01:0x22: Date: Jan 20 2022
05:40:07:WU00:FS01:0x22: Time: 00:57:52
05:40:07:WU00:FS01:0x22: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
05:40:07:WU00:FS01:0x22: Branch: HEAD
05:40:07:WU00:FS01:0x22: Compiler: GNU 9.4.0
05:40:07:WU00:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:40:07:WU00:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
05:40:07:WU00:FS01:0x22: -DOPENMM_VERSION=\"\\\"7.7.0\\\"\"
05:40:07:WU00:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:40:07:WU00:FS01:0x22: Bits: 64
05:40:07:WU00:FS01:0x22: Mode: Release
05:40:07:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
05:40:07:WU00:FS01:0x22: <peastman@stanford.edu>
05:40:07:WU00:FS01:0x22: Args: -dir 00 -suffix 01 -version 706 -lifeline 3864 -checkpoint 15
05:40:07:WU00:FS01:0x22: -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
05:40:07:WU00:FS01:0x22: nvidia -gpu 1 -gpu-usage 100
05:40:07:WU00:FS01:0x22:************************************ libFAH ************************************
05:40:07:WU00:FS01:0x22: Date: Jan 20 2022
05:40:07:WU00:FS01:0x22: Time: 00:57:22
05:40:07:WU00:FS01:0x22: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
05:40:07:WU00:FS01:0x22: Branch: HEAD
05:40:07:WU00:FS01:0x22: Compiler: GNU 9.4.0
05:40:07:WU00:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:40:07:WU00:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie
05:40:07:WU00:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:40:07:WU00:FS01:0x22: Bits: 64
05:40:07:WU00:FS01:0x22: Mode: Release
05:40:07:WU00:FS01:0x22:************************************ CBang *************************************
05:40:07:WU00:FS01:0x22: Date: Jan 20 2022
05:40:07:WU00:FS01:0x22: Time: 00:57:00
05:40:07:WU00:FS01:0x22: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
05:40:07:WU00:FS01:0x22: Branch: HEAD
05:40:07:WU00:FS01:0x22: Compiler: GNU 9.4.0
05:40:07:WU00:FS01:0x22: Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
05:40:07:WU00:FS01:0x22: -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
05:40:07:WU00:FS01:0x22: Platform: linux 5.11.0-1025-azure
05:40:07:WU00:FS01:0x22: Bits: 64
05:40:07:WU00:FS01:0x22: Mode: Release
05:40:07:WU00:FS01:0x22:************************************ System ************************************
05:40:07:WU00:FS01:0x22: CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
05:40:07:WU00:FS01:0x22: CPU ID: GenuineIntel Family 6 Model 45 Stepping 7
05:40:07:WU00:FS01:0x22: CPUs: 32
05:40:07:WU00:FS01:0x22: Memory: 15.60GiB
05:40:07:WU00:FS01:0x22:Free Memory: 13.42GiB
05:40:07:WU00:FS01:0x22: Threads: POSIX_THREADS
05:40:07:WU00:FS01:0x22: OS Version: 5.15
05:40:07:WU00:FS01:0x22:Has Battery: false
05:40:07:WU00:FS01:0x22: On Battery: false
05:40:07:WU00:FS01:0x22: UTC Offset: -5
05:40:07:WU00:FS01:0x22: PID: 3868
05:40:07:WU00:FS01:0x22: CWD: /var/lib/fahclient/work
05:40:07:WU00:FS01:0x22:************************************ OpenMM ************************************
05:40:07:WU00:FS01:0x22: Version: 7.7.0
05:40:07:WU00:FS01:0x22:********************************************************************************
05:40:07:WU00:FS01:0x22:Project: 18601 (Run 13163, Clone 2, Gen 25)
05:40:07:WU00:FS01:0x22:Reading tar file core.xml
05:40:07:WU00:FS01:0x22:Reading tar file integrator.xml
05:40:07:WU00:FS01:0x22:Reading tar file state.xml.bz2
05:40:07:WU00:FS01:0x22:Reading tar file system.xml.bz2
05:40:07:WU00:FS01:0x22:Digital signatures verified
05:40:07:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
05:40:07:WU00:FS01:0x22:Version 0.0.20
05:40:07:WU00:FS01:0x22: Checkpoint write interval: 25000 steps (2%) [50 total]
05:40:07:WU00:FS01:0x22: JSON viewer frame write interval: 12500 steps (1%) [100 total]
05:40:07:WU00:FS01:0x22: XTC frame write interval: 20000 steps (1.6%) [62 total]
05:40:07:WU00:FS01:0x22: Global context and integrator variables write interval: disabled
05:40:07:WU00:FS01:0x22:There are 3 platforms available.
05:40:07:WU00:FS01:0x22:Platform 0: Reference
05:40:07:WU00:FS01:0x22:Platform 1: CPU
05:40:07:WU00:FS01:0x22:Platform 2: CUDA
05:40:07:WU00:FS01:0x22: cuda-device 1 specified
05:40:07:WU00:FS01:0x22:opencl-device was set but OpenCL platform could not be found.
05:40:27:WU00:FS01:0x22:Attempting to create CUDA context:
05:40:27:WU00:FS01:0x22: Configuring platform CUDA
05:40:27:WU00:FS01:0x22:Failed to create CUDA context:
05:40:27:WU00:FS01:0x22:Error initializing CUDA: CUDA_ERROR_UNKNOWN (999) at /home/conda/feedstock_root/build_artifacts/openmm_1640541150081/work/platforms/cuda/src/CudaContext.cpp:139
05:40:27:WU00:FS01:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
05:40:27:WU00:FS01:0x22:Saving result file ../logfile_01.txt
05:40:27:WU00:FS01:0x22:Saving result file science.log
05:40:27:WU00:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
[93m05:40:27:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)[0m
05:40:27:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:18601 run:13163 clone:2 gen:25 core:0x22 unit:0x0000000200000019000048a90000336b
05:40:27:WU00:FS01:Uploading 11.00KiB to 206.223.170.146
05:40:27:WU00:FS01:Connecting to 206.223.170.146:8080
05:40:27:WU00:FS01:Upload complete
05:40:27:WU00:FS01:Server responded WORK_ACK (400)
05:40:27:WU00:FS01:Cleaning up
*** cut to fit ***
Re: FAULTY project:18450 (Multiple errors)
Posted: Sat Nov 19, 2022 4:13 pm
by Joe_H
The errors I see in the logs indicate the client is not able to find the GPUs at the hardware locations first found at setup. This sometimes happens after a driver or OS update, did anything like that happen? For example on Windows auto updates for drivers are recommended to be disabled and done periodically after finishing current WUs. Similar problems have been seen with virtualized connections to GPUs if the setup has changed.
Re: FAULTY project:18450 (Multiple errors)
Posted: Sat Nov 19, 2022 6:32 pm
by n1np
Thanks for the response, Joe.
There were no OS or driver updates. The machine was running, relatively new setup, okay for a few WU then the errors started.
The machine seems to be running stable now with a single GPU in the slot 1.
I had previously run two Quadro K4000 Kepler GPUs in these machines with no issues, then pulled them in spring 2021 because the PPD (60-80k) wasn't worth the electricity cost. Now trying again with Quadro P4000 Pascal GPUs. The points are nice if they will run stable reliably. I prefer to set up a machine and walk away, expecting it to just work, thus my preference for the rack-mount servers.
Ben N1NP
Re: FAULTY project:18450 (Multiple errors)
Posted: Sat Nov 19, 2022 7:02 pm
by Joe_H
The only thing else I can think of is incompatibilities between the Linux kernel and the drivers. There have been a number of cases that I have seen in the past where the drivers needed to be used with specific kernel versions, otherwise some things did not work correctly.
I did notice when I looked up current drivers that 5.15.76 was the recommended release, with 5.20 as a "new feature" release. Not sure if dropping back to the older driver version will help, currently not running my Linux system on recent Ubuntu releases.
Re: FAULTY project:18450 (Multiple errors)
Posted: Wed Nov 23, 2022 8:45 pm
by toTOW
It looks like your GPU at folding slot 01 (Bus:33 Slot:0 Func:0 NVIDIA:5 GP104GL [Quadro P4000]) is unstable. It is probably crashing badly since the driver could no longer find it (Error initializing CUDA: CUDA_ERROR_UNKNOWN (999) and ERROR:125: Failed to create a GPU-enabled OpenMM Context.).
I suggest that you test each GPU individually to confirm which one is bad.
Re: FAULTY project:18450 (Multiple errors)
Posted: Fri Nov 25, 2022 12:04 am
by n1np
Thanks for the tip.
Both GPUs are running with no errors for the past few days in a different computer.