Project: 14719 (Run 802, Clone 4, Gen 0) GROMACS Error
Posted: Sun Jul 26, 2020 4:33 pm
Project: 14719 (Run 802, Clone 4, Gen 0) is stuck on my 48 thread server with this error. Any way I can process this or should I get rid of it? I have not seen this error before. Log extract follows:
Code: Select all
15:03:18:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14719 run:802 clone:4 gen:0 core:0xa7 unit:0x000000022879986c5ea9670ab49b086c
15:03:18:WU01:FS00:Starting
15:03:18:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 706 -lifeline 1979 -checkpoint 15 -np 48
15:03:18:WU01:FS00:Started FahCore on PID 14407
15:03:18:WU01:FS00:Core PID:14411
15:03:18:WU01:FS00:FahCore 0xa7 started
15:03:18:WU01:FS00:0xa7:*********************** Log Started 2020-07-26T15:03:18Z ***********************
15:03:18:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
15:03:18:WU01:FS00:0xa7: Type: 0xa7
15:03:18:WU01:FS00:0xa7: Core: Gromacs
15:03:18:WU01:FS00:0xa7: Args: -dir 01 -suffix 01 -version 706 -lifeline 14407 -checkpoint 15 -np
15:03:18:WU01:FS00:0xa7: 48
15:03:18:WU01:FS00:0xa7:************************************ CBang *************************************
15:03:18:WU01:FS00:0xa7: Date: Nov 27 2019
15:03:18:WU01:FS00:0xa7: Time: 11:26:54
15:03:18:WU01:FS00:0xa7: Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
15:03:18:WU01:FS00:0xa7: Branch: master
15:03:18:WU01:FS00:0xa7: Compiler: GNU 8.3.0
15:03:18:WU01:FS00:0xa7: Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
15:03:18:WU01:FS00:0xa7: -fno-pie -fPIC
15:03:18:WU01:FS00:0xa7: Platform: linux2 4.19.0-5-amd64
15:03:18:WU01:FS00:0xa7: Bits: 64
15:03:18:WU01:FS00:0xa7: Mode: Release
15:03:18:WU01:FS00:0xa7:************************************ System ************************************
15:03:18:WU01:FS00:0xa7: CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
15:03:18:WU01:FS00:0xa7: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
15:03:18:WU01:FS00:0xa7: CPUs: 48
15:03:18:WU01:FS00:0xa7: Memory: 62.80GiB
15:03:18:WU01:FS00:0xa7:Free Memory: 59.87GiB
15:03:18:WU01:FS00:0xa7: Threads: POSIX_THREADS
15:03:18:WU01:FS00:0xa7: OS Version: 5.3
15:03:18:WU01:FS00:0xa7:Has Battery: false
15:03:18:WU01:FS00:0xa7: On Battery: false
15:03:18:WU01:FS00:0xa7: UTC Offset: -5
15:03:18:WU01:FS00:0xa7: PID: 14411
15:03:18:WU01:FS00:0xa7: CWD: /var/lib/fahclient/work
15:03:18:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
15:03:18:WU01:FS00:0xa7: Version: 0.0.19
15:03:18:WU01:FS00:0xa7: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:03:18:WU01:FS00:0xa7: Copyright: 2019 foldingathome.org
15:03:18:WU01:FS00:0xa7: Homepage: https://foldingathome.org/
15:03:18:WU01:FS00:0xa7: Date: Nov 26 2019
15:03:18:WU01:FS00:0xa7: Time: 00:41:42
15:03:18:WU01:FS00:0xa7: Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
15:03:18:WU01:FS00:0xa7: Branch: master
15:03:18:WU01:FS00:0xa7: Compiler: GNU 8.3.0
15:03:18:WU01:FS00:0xa7: Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
15:03:18:WU01:FS00:0xa7: -fno-pie
15:03:18:WU01:FS00:0xa7: Platform: linux2 4.19.0-5-amd64
15:03:18:WU01:FS00:0xa7: Bits: 64
15:03:18:WU01:FS00:0xa7: Mode: Release
15:03:18:WU01:FS00:0xa7:************************************ Build *************************************
15:03:18:WU01:FS00:0xa7: SIMD: avx_256
15:03:18:WU01:FS00:0xa7:********************************************************************************
15:03:18:WU01:FS00:0xa7:Project: 14719 (Run 802, Clone 4, Gen 0)
15:03:18:WU01:FS00:0xa7:Unit: 0x000000022879986c5ea9670ab49b086c
15:03:18:WU01:FS00:0xa7:Reading tar file core.xml
15:03:18:WU01:FS00:0xa7:Reading tar file frame0.tpr
15:03:18:WU01:FS00:0xa7:Digital signatures verified
15:03:18:WU01:FS00:0xa7:Calling: mdrun -s frame0.tpr -o frame0.trr -cpt 15 -nt 48
15:03:18:WU01:FS00:0xa7:Steps: first=0 total=250000
15:03:20:WU01:FS00:0xa7:Completed 1 out of 250000 steps (0%)
15:03:23:WU00:FS00:Upload 79.17%
15:03:23:WU01:FS00:0xa7:ERROR:
15:03:23:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
15:03:24:WU01:FS00:Starting
15:03:24:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 706 -lifeline 1979 -checkpoint 15 -np 48
15:03:24:WU01:FS00:Started FahCore on PID 14463
15:03:24:WU01:FS00:Core PID:14467
15:03:24:WU01:FS00:FahCore 0xa7 started
15:03:24:WU01:FS00:0xa7:*********************** Log Started 2020-07-26T15:03:24Z ***********************
15:03:24:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
15:03:24:WU01:FS00:0xa7: Type: 0xa7
15:03:24:WU01:FS00:0xa7: Core: Gromacs
15:03:24:WU01:FS00:0xa7: Args: -dir 01 -suffix 01 -version 706 -lifeline 14463 -checkpoint 15 -np
15:03:24:WU01:FS00:0xa7: 48
15:03:24:WU01:FS00:0xa7:************************************ CBang *************************************
15:03:24:WU01:FS00:0xa7: Date: Nov 27 2019
15:03:24:WU01:FS00:0xa7: Time: 11:26:54
15:03:24:WU01:FS00:0xa7: Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
15:03:24:WU01:FS00:0xa7: Branch: master
15:03:24:WU01:FS00:0xa7: Compiler: GNU 8.3.0
15:03:24:WU01:FS00:0xa7: Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
15:03:24:WU01:FS00:0xa7: -fno-pie -fPIC
15:03:24:WU01:FS00:0xa7: Platform: linux2 4.19.0-5-amd64
15:03:24:WU01:FS00:0xa7: Bits: 64
15:03:24:WU01:FS00:0xa7: Mode: Release
15:03:24:WU01:FS00:0xa7:************************************ System ************************************
15:03:24:WU01:FS00:0xa7: CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
15:03:24:WU01:FS00:0xa7: CPU ID: GenuineIntel Family 6 Model 63 Stepping 2
15:03:24:WU01:FS00:0xa7: CPUs: 48
15:03:24:WU01:FS00:0xa7: Memory: 62.80GiB
15:03:24:WU01:FS00:0xa7:Free Memory: 59.86GiB
15:03:24:WU01:FS00:0xa7: Threads: POSIX_THREADS
15:03:24:WU01:FS00:0xa7: OS Version: 5.3
15:03:24:WU01:FS00:0xa7:Has Battery: false
15:03:24:WU01:FS00:0xa7: On Battery: false
15:03:24:WU01:FS00:0xa7: UTC Offset: -5
15:03:24:WU01:FS00:0xa7: PID: 14467
15:03:24:WU01:FS00:0xa7: CWD: /var/lib/fahclient/work
15:03:24:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
15:03:24:WU01:FS00:0xa7: Version: 0.0.19
15:03:24:WU01:FS00:0xa7: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:03:24:WU01:FS00:0xa7: Copyright: 2019 foldingathome.org
15:03:24:WU01:FS00:0xa7: Homepage: https://foldingathome.org/
15:03:24:WU01:FS00:0xa7: Date: Nov 26 2019
15:03:24:WU01:FS00:0xa7: Time: 00:41:42
15:03:24:WU01:FS00:0xa7: Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
15:03:24:WU01:FS00:0xa7: Branch: master
15:03:24:WU01:FS00:0xa7: Compiler: GNU 8.3.0
15:03:24:WU01:FS00:0xa7: Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
15:03:24:WU01:FS00:0xa7: -fno-pie
15:03:24:WU01:FS00:0xa7: Platform: linux2 4.19.0-5-amd64
15:03:24:WU01:FS00:0xa7: Bits: 64
15:03:24:WU01:FS00:0xa7: Mode: Release
15:03:24:WU01:FS00:0xa7:************************************ Build *************************************
15:03:24:WU01:FS00:0xa7: SIMD: avx_256
15:03:24:WU01:FS00:0xa7:********************************************************************************
15:03:24:WU01:FS00:0xa7:Project: 14719 (Run 802, Clone 4, Gen 0)
15:03:24:WU01:FS00:0xa7:Unit: 0x000000022879986c5ea9670ab49b086c
15:03:24:WU01:FS00:0xa7:Digital signatures verified
15:03:24:WU01:FS00:0xa7:Calling: mdrun -s frame0.tpr -o frame0.trr -cpt 15 -nt 48
15:03:24:WU01:FS00:0xa7:Steps: first=0 total=250000
15:03:25:WU00:FS00:Upload complete
15:03:25:WU00:FS00:Server responded WORK_ACK (400)
15:03:25:WU00:FS00:Final credit estimate, 9043.00 points
15:03:25:WU00:FS00:Cleaning up
15:03:26:WU01:FS00:0xa7:Completed 1 out of 250000 steps (0%)
15:03:29:WU01:FS00:0xa7:ERROR:
15:03:29:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
15:03:29:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
15:03:29:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/expanded.c, line: 946
15:03:29:WU01:FS00:0xa7:ERROR:
15:03:29:WU01:FS00:0xa7:ERROR:Fatal error:
15:03:29:WU01:FS00:0xa7:ERROR:Something wrong in choosing new lambda state with a Gibbs move -- probably underflow in weight determination.
15:03:29:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
15:04:24:WU01:FS00:Starting
15:04:24:WU01:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7 -dir 01 -suffix 01 -version 706 -lifeline 1979 -checkpoint 15 -np 48
15:04:24:WU01:FS00:Started FahCore on PID 14520
15:04:24:WU01:FS00:Core PID:14524
15:04:24:WU01:FS00:FahCore 0xa7 started
15:04:24:WU01:FS00:0xa7:*********************** Log Started 2020-07-26T15:04:24Z ***********************
15:04:24:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
15:04:24:WU01:FS00:0xa7: Type: 0xa7
15:04:24:WU01:FS00:0xa7: Core: Gromacs
15:04:24:WU01:FS00:0xa7: Args: -dir 01 -suffix 01 -version 706 -lifeline 14520 -checkpoint 15 -np
15:04:24:WU01:FS00:0xa7: 48
15:04:24:WU01:FS00:0xa7:************************************ CBang *********************************
... errors continue ...
16:20:26:WU01:FS00:0xa7:Project: 14719 (Run 802, Clone 4, Gen 0)
16:20:26:WU01:FS00:0xa7:Unit: 0x000000022879986c5ea9670ab49b086c
16:20:26:WU01:FS00:0xa7:Digital signatures verified
16:20:26:WU01:FS00:0xa7:Calling: mdrun -s frame0.tpr -o frame0.trr -cpt 15 -nt 48
16:20:26:WU01:FS00:0xa7:Steps: first=0 total=250000
16:20:28:WU01:FS00:0xa7:Completed 1 out of 250000 steps (0%)
16:20:31:WU01:FS00:0xa7:ERROR:
16:20:31:WU01:FS00:0xa7:ERROR:-------------------------------------------------------
16:20:31:WU01:FS00:0xa7:ERROR:Program GROMACS, VERSION 5.0.4-20191026-456f0d636-unknown
16:20:31:WU01:FS00:0xa7:ERROR:Source code file: /host/debian-stable-64bit-core-a7-avx-release/gromacs-core/build/gromacs/src/gromacs/mdlib/expanded.c, line: 946
16:20:31:WU01:FS00:0xa7:ERROR:
16:20:31:WU01:FS00:0xa7:ERROR:Fatal error:
16:20:31:WU01:FS00:0xa7:ERROR:Something wrong in choosing new lambda state with a Gibbs move -- probably underflow in weight determination.
16:20:31:WU01:FS00:0xa7:ERROR:Denominator is: 0 1.0000000000e+00
16:20:31:WU01:FS00:0xa7:ERROR: i dE numerator weights
16:20:31:WU01:FS00:0xa7:ERROR: 0 0.0000000000e+00 1.0000000000e+00 0.0000000000e+00
16:20:31:WU01:FS00:0xa7:ERROR: 1 -3.5460235596e+01 3.9793794585e-16 1.0000000000e+01
16:20:31:WU01:FS00:0xa7:ERROR: 2 -7.0920471191e+01 1.5835460875e-31 1.0000000000e+01
16:20:31:WU01:FS00:0xa7:ERROR: 3 -1.0638024139e+02 6.3044641435e-47 1.0000000000e+01
16:20:32:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)