FahCore_22 crashes under certain circumstances

[Gennady]

Hi,

By stopping the FAH application, I received information from the operating system that the application crashed.
I saved a dump of the generated by WER just in case. Is it interesting for developers and can they analyze it?
Preliminary analysis done by WinDbg
------
STACK_TEXT:
00000000`0025d640 000007fe`e30c7fb6 OpenMMCUDA!OpenMM::CudaSort::sort+0x9736
00000000`0025d670 000007fe`e30d249e OpenMMCUDA!OpenMM::CudaSort::sort+0x13c1e
00000000`0025d6a0 000007fe`e30c28fb OpenMMCUDA!OpenMM::CudaSort::sort+0x407b
00000000`0025d780 000007fe`e30c352d OpenMMCUDA!OpenMM::CudaSort::sort+0x4cad
00000000`0025d7f0 000007fe`e30c433d OpenMMCUDA!OpenMM::CudaSort::sort+0x5abd
00000000`0025d840 000007fe`e30c157c OpenMMCUDA!OpenMM::CudaSort::sort+0x2cfc
00000000`0025d870 00000000`7771be9d ntdll!_chkstk+0x9d
00000000`0025d8a0 00000000`776f040a ntdll!RtlInitializeResource+0x53a
00000000`0025df80 00000000`776f3c0f ntdll!RtlRaiseException+0x22f
00000000`0025e930 000007fe`fd35b87d KERNELBASE!RaiseException+0x3d
00000000`0025ea00 000007fe`e30c3c8a OpenMMCUDA!OpenMM::CudaSort::sort+0x540a
00000000`0025ea80 000007fe`e2fef68d OpenMMCUDA!OpenMM::CudaArray::~CudaArray+0x22d
00000000`0025ec30 000007fe`e3035bc8 OpenMMCUDA!OpenMM::CudaIntegrationUtilities::setNextStepSize+0x5308
00000000`0025ec60 000007fe`e11d7d32 OpenMM!OpenMM::HarmonicBondForce::usesPeriodicBoundaryConditions+0x72
00000000`0025ec90 000007fe`e1189692 OpenMM!OpenMM::ContextImpl::~ContextImpl+0x62
00000000`0025ece0 000007fe`e1184ad7 OpenMM!OpenMM::Context::~Context+0x27
00000000`0025ed20 00000001`3f123bb5 FahCore_22+0x43bb5
00000000`0025ed60 00000001`3f14e282 FahCore_22+0x6e282
00000000`0025f7b0 00000001`3f58c941 FahCore_22+0x4ac941
00000000`0025f7f0 00000000`774a556d kernel32!BaseThreadInitThunk+0xd
00000000`0025f820 00000000`7770372d ntdll!RtlUserThreadStart+0x1d

[PantherX]

Can you please post the log file? Ensure you include the first 100 lines which will inform us of what the system configuration is and what the client settings are. If you require guidance, please view this topic: viewtopic.php?f=24&t=26036

[Gennady]

Hello again, yep no problem

Code: Select all

*********************** Log Started 2020-10-07T04:14:29Z ***********************
04:14:29:Trying to access database...
04:14:29:Successfully acquired database lock
04:14:29:Read GPUs.txt
04:14:29:Enabled folding slot 00: PAUSED cpu:6 (by user)
04:14:29:Enabled folding slot 01: PAUSED gpu:0:TU116 [GeForce GTX 1660 Ti] (by user)
04:14:29:****************************** FAHClient ******************************
04:14:29:        Version: 7.6.13
04:14:29:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:14:29:      Copyright: 2020 foldingathome.org
04:14:29:       Homepage: https://foldingathome.org/
04:14:29:           Date: Apr 27 2020
04:14:29:           Time: 21:21:01
04:14:29:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
04:14:29:         Branch: master
04:14:29:       Compiler: Visual C++ 2008
04:14:29:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
04:14:29:       Platform: win32 10
04:14:29:           Bits: 32
04:14:29:           Mode: Release
04:14:29:           Args: --open-web-control
04:14:29:         Config: E:\Users\Gennady\AppData\Roaming\FAHClient\config.xml
04:14:29:******************************** CBang ********************************
04:14:29:           Date: Apr 24 2020
04:14:29:           Time: 17:07:55
04:14:29:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
04:14:29:         Branch: master
04:14:29:       Compiler: Visual C++ 2008
04:14:29:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
04:14:29:       Platform: win32 10
04:14:29:           Bits: 32
04:14:29:           Mode: Release
04:14:29:******************************* System ********************************
04:14:29:            CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
04:14:29:         CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
04:14:29:           CPUs: 8
04:14:29:         Memory: 31.87GiB
04:14:29:    Free Memory: 20.24GiB
04:14:29:        Threads: WINDOWS_THREADS
04:14:29:     OS Version: 6.1
04:14:29:    Has Battery: false
04:14:29:     On Battery: false
04:14:29:     UTC Offset: 3
04:14:29:            PID: 11896
04:14:29:            CWD: E:\Users\Gennady\AppData\Roaming\FAHClient
04:14:29:  Win32 Service: false
04:14:29:             OS: Windows 7 Ultimate
04:14:29:        OS Arch: AMD64
04:14:29:           GPUs: 2
04:14:29:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 TU116 [GeForce GTX 1660 Ti]
04:14:29:          GPU 1: Bus:0 Slot:2 Func:0 INTEL:1 Gen9/GT2 [HD Graphics 530]
04:14:29:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:7.5 Driver:11.1
04:14:29:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:456.55
04:14:29:******************************* libFAH ********************************
04:14:29:           Date: Apr 15 2020
04:14:29:           Time: 14:53:14
04:14:29:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
04:14:29:         Branch: master
04:14:29:       Compiler: Visual C++ 2008
04:14:29:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
04:14:29:       Platform: win32 10
04:14:29:           Bits: 32
04:14:29:           Mode: Release
04:14:29:***********************************************************************
04:14:29:<config>
04:14:29:  <!-- Folding Slot Configuration -->
04:14:29:  <cause v='COVID_19'/>
04:14:29:
04:14:29:  <!-- Network -->
04:14:29:  <proxy v=':8080'/>
04:14:29:
04:14:29:  <!-- User Information -->
04:14:29:  <passkey v='*****'/>
04:14:29:  <team v='2164'/>
04:14:29:  <user v='Gennady'/>
04:14:29:
04:14:29:  <!-- Folding Slots -->
04:14:29:  <slot id='0' type='CPU'>
04:14:29:    <paused v='true'/>
04:14:29:  </slot>
04:14:29:  <slot id='1' type='GPU'>
04:14:29:    <paused v='true'/>
04:14:29:  </slot>
04:14:29:</config>
04:14:38:17:127.0.0.1:New Web session
04:16:21:FS00:Unpaused
04:16:21:FS01:Unpaused
04:16:21:WU02:FS01:Starting
04:16:21:WU02:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" E:\Users\Gennady\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 11896 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
04:16:21:WU02:FS01:Started FahCore on PID 11356
04:16:21:WU02:FS01:Core PID:11524
04:16:21:WU02:FS01:FahCore 0x22 started
04:16:21:WU00:FS00:Starting
04:16:21:WU00:FS00:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" E:\Users\Gennady\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 11896 -checkpoint 15 -np 6
04:16:21:WU00:FS00:Started FahCore on PID 9340
04:16:21:WU00:FS00:Core PID:10940
04:16:21:WU00:FS00:FahCore 0xa7 started
04:16:23:WU00:FS00:0xa7:*********************** Log Started 2020-10-07T04:16:23Z ***********************
04:16:23:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
04:16:23:WU00:FS00:0xa7:       Type: 0xa7
04:16:23:WU00:FS00:0xa7:       Core: Gromacs
04:16:23:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 9340 -checkpoint 15 -np 6
04:16:23:WU00:FS00:0xa7:************************************ CBang *************************************
04:16:23:WU00:FS00:0xa7:       Date: Nov 27 2019
04:16:23:WU00:FS00:0xa7:       Time: 03:40:09
04:16:23:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
04:16:23:WU00:FS00:0xa7:     Branch: master
04:16:23:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
04:16:23:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
04:16:23:WU00:FS00:0xa7:   Platform: win32 10
04:16:23:WU00:FS00:0xa7:       Bits: 64
04:16:23:WU00:FS00:0xa7:       Mode: Release
04:16:23:WU00:FS00:0xa7:************************************ System ************************************
04:16:23:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
04:16:23:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
04:16:23:WU00:FS00:0xa7:       CPUs: 8
04:16:23:WU00:FS00:0xa7:     Memory: 31.87GiB
04:16:23:WU00:FS00:0xa7:Free Memory: 19.99GiB
04:16:23:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
04:16:23:WU00:FS00:0xa7: OS Version: 6.1
04:16:23:WU00:FS00:0xa7:Has Battery: false
04:16:23:WU00:FS00:0xa7: On Battery: false
04:16:23:WU00:FS00:0xa7: UTC Offset: 3
04:16:23:WU00:FS00:0xa7:        PID: 10940
04:16:23:WU00:FS00:0xa7:        CWD: E:\Users\Gennady\AppData\Roaming\FAHClient\work
04:16:23:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
04:16:23:WU00:FS00:0xa7:    Version: 0.0.19
04:16:23:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:16:23:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
04:16:23:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
04:16:23:WU00:FS00:0xa7:       Date: Nov 25 2019
04:16:23:WU00:FS00:0xa7:       Time: 17:12:41
04:16:23:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
04:16:23:WU00:FS00:0xa7:     Branch: master
04:16:23:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
04:16:23:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
04:16:23:WU00:FS00:0xa7:   Platform: win32 10
04:16:23:WU00:FS00:0xa7:       Bits: 64
04:16:23:WU00:FS00:0xa7:       Mode: Release
04:16:23:WU00:FS00:0xa7:************************************ Build *************************************
04:16:23:WU00:FS00:0xa7:       SIMD: avx_256
04:16:23:WU00:FS00:0xa7:********************************************************************************
04:16:23:WU00:FS00:0xa7:Project: 16806 (Run 18, Clone 400, Gen 48)
04:16:23:WU00:FS00:0xa7:Unit: 0x0000004582ed0b915eb419782ba2f347
04:16:23:WU00:FS00:0xa7:Digital signatures verified
04:16:23:WU00:FS00:0xa7:Calling: mdrun -s frame48.tpr -o frame48.trr -cpi state.cpt -cpt 15 -nt 6
04:16:24:WU02:FS01:0x22:*********************** Log Started 2020-10-07T04:16:24Z ***********************
04:16:24:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
04:16:24:WU02:FS01:0x22:       Core: Core22
04:16:24:WU02:FS01:0x22:       Type: 0x22
04:16:24:WU02:FS01:0x22:    Version: 0.0.13
04:16:24:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:16:24:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
04:16:24:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
04:16:24:WU02:FS01:0x22:       Date: Sep 19 2020
04:16:24:WU02:FS01:0x22:       Time: 02:35:58
04:16:24:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
04:16:24:WU02:FS01:0x22:     Branch: core22-0.0.13
04:16:24:WU02:FS01:0x22:   Compiler: Visual C++ 2015
04:16:24:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
04:16:24:WU02:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
04:16:24:WU02:FS01:0x22:   Platform: win32 10
04:16:24:WU02:FS01:0x22:       Bits: 64
04:16:24:WU02:FS01:0x22:       Mode: Release
04:16:24:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
04:16:24:WU02:FS01:0x22:             <peastman@stanford.edu>
04:16:24:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 11356 -checkpoint 15
04:16:24:WU02:FS01:0x22:             -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device
04:16:24:WU02:FS01:0x22:             0 -gpu 0
04:16:24:WU02:FS01:0x22:************************************ libFAH ************************************
04:16:24:WU02:FS01:0x22:       Date: Sep 7 2020
04:16:24:WU02:FS01:0x22:       Time: 19:09:56
04:16:24:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
04:16:24:WU02:FS01:0x22:     Branch: HEAD
04:16:24:WU02:FS01:0x22:   Compiler: Visual C++ 2015
04:16:24:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
04:16:24:WU02:FS01:0x22:   Platform: win32 10
04:16:24:WU02:FS01:0x22:       Bits: 64
04:16:24:WU02:FS01:0x22:       Mode: Release
04:16:24:WU02:FS01:0x22:************************************ CBang *************************************
04:16:24:WU02:FS01:0x22:       Date: Sep 7 2020
04:16:24:WU02:FS01:0x22:       Time: 19:08:30
04:16:24:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
04:16:24:WU02:FS01:0x22:     Branch: HEAD
04:16:24:WU02:FS01:0x22:   Compiler: Visual C++ 2015
04:16:24:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
04:16:24:WU02:FS01:0x22:   Platform: win32 10
04:16:24:WU02:FS01:0x22:       Bits: 64
04:16:24:WU02:FS01:0x22:       Mode: Release
04:16:24:WU02:FS01:0x22:************************************ System ************************************
04:16:24:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
04:16:24:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
04:16:24:WU02:FS01:0x22:       CPUs: 8
04:16:24:WU02:FS01:0x22:     Memory: 31.87GiB
04:16:24:WU02:FS01:0x22:Free Memory: 19.98GiB
04:16:24:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
04:16:24:WU02:FS01:0x22: OS Version: 6.1
04:16:24:WU02:FS01:0x22:Has Battery: false
04:16:24:WU02:FS01:0x22: On Battery: false
04:16:24:WU02:FS01:0x22: UTC Offset: 3
04:16:24:WU02:FS01:0x22:        PID: 11524
04:16:24:WU02:FS01:0x22:        CWD: E:\Users\Gennady\AppData\Roaming\FAHClient\work
04:16:24:WU02:FS01:0x22:************************************ OpenMM ************************************
04:16:24:WU02:FS01:0x22:   Revision: 189320d0
04:16:24:WU02:FS01:0x22:********************************************************************************
04:16:24:WU02:FS01:0x22:Project: 14909 (Run 371, Clone 1, Gen 36)
04:16:24:WU02:FS01:0x22:Unit: 0x0000003581d59d695f526020a15c6e4c
04:16:24:WU02:FS01:0x22:Digital signatures verified
04:16:24:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
04:16:24:WU02:FS01:0x22:Version 0.0.13
04:16:24:WU02:FS01:0x22:  Checkpoint write interval: 100000 steps (5%) [20 total]
04:16:24:WU02:FS01:0x22:  JSON viewer frame write interval: 20000 steps (1%) [100 total]
04:16:24:WU02:FS01:0x22:  XTC frame write interval: 50000 steps (2.5%) [40 total]
04:16:24:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
04:16:26:WU00:FS00:0xa7:Steps: first=24000000 total=500000
04:16:28:WU02:FS01:0x22:There are 4 platforms available.
04:16:28:WU02:FS01:0x22:Platform 0: Reference
04:16:28:WU02:FS01:0x22:Platform 1: CPU
04:16:28:WU02:FS01:0x22:Platform 2: OpenCL
04:16:28:WU02:FS01:0x22:  opencl-device 0 specified
04:16:28:WU02:FS01:0x22:Platform 3: CUDA
04:16:28:WU02:FS01:0x22:  cuda-device 0 specified
04:16:29:WU00:FS00:0xa7:Completed 423442 out of 500000 steps (84%)
04:16:31:Removing old file 'configs/config-20200927-234448.xml'
04:16:31:Saving configuration to config.xml
04:16:31:<config>
04:16:31:  <!-- Folding Slot Configuration -->
04:16:31:  <cause v='COVID_19'/>
04:16:31:
04:16:31:  <!-- Network -->
04:16:31:  <proxy v=':8080'/>
04:16:31:
04:16:31:  <!-- User Information -->
04:16:31:  <passkey v='*****'/>
04:16:31:  <team v='2164'/>
04:16:31:  <user v='Gennady'/>
04:16:31:
04:16:31:  <!-- Folding Slots -->
04:16:31:  <slot id='0' type='CPU'/>
04:16:31:  <slot id='1' type='GPU'/>
04:16:31:</config>
04:16:48:WU02:FS01:0x22:Attempting to create CUDA context:
04:16:48:WU02:FS01:0x22:  Configuring platform CUDA
04:16:57:WU02:FS01:0x22:  Using CUDA and gpu 0
04:17:00:WU02:FS01:0x22:Completed 0 out of 2000000 steps (0%)
04:17:01:WU02:FS01:0x22:Checkpoint completed at step 0
04:17:52:WU00:FS00:0xa7:Completed 425000 out of 500000 steps (85%)
04:20:29:WU02:FS01:0x22:Completed 20000 out of 2000000 steps (1%)
04:22:03:WU00:FS00:0xa7:Completed 430000 out of 500000 steps (86%)
04:24:02:WU02:FS01:0x22:Completed 40000 out of 2000000 steps (2%)
04:27:09:WU00:FS00:0xa7:Completed 435000 out of 500000 steps (87%)
04:27:37:WU02:FS01:0x22:Completed 60000 out of 2000000 steps (3%)
04:31:20:WU02:FS01:0x22:Completed 80000 out of 2000000 steps (4%)
04:34:49:WU00:FS00:0xa7:Completed 440000 out of 500000 steps (88%)
04:34:56:WU02:FS01:0x22:Completed 100000 out of 2000000 steps (5%)
04:34:58:WU02:FS01:0x22:Checkpoint completed at step 100000
04:38:35:WU02:FS01:0x22:Completed 120000 out of 2000000 steps (6%)
04:40:13:WU00:FS00:0xa7:Completed 445000 out of 500000 steps (89%)
04:42:09:WU02:FS01:0x22:Completed 140000 out of 2000000 steps (7%)
04:45:04:WU00:FS00:0xa7:Completed 450000 out of 500000 steps (90%)
04:45:43:WU02:FS01:0x22:Completed 160000 out of 2000000 steps (8%)
04:49:17:WU02:FS01:0x22:Completed 180000 out of 2000000 steps (9%)
04:49:23:WU00:FS00:0xa7:Completed 455000 out of 500000 steps (91%)
04:52:48:WU02:FS01:0x22:Completed 200000 out of 2000000 steps (10%)
04:52:49:WU02:FS01:0x22:Checkpoint completed at step 200000
04:53:26:WU00:FS00:0xa7:Completed 460000 out of 500000 steps (92%)
04:56:18:WU02:FS01:0x22:Completed 220000 out of 2000000 steps (11%)
04:57:21:WU00:FS00:0xa7:Completed 465000 out of 500000 steps (93%)
04:59:50:WU02:FS01:0x22:Completed 240000 out of 2000000 steps (12%)
05:01:40:WU00:FS00:0xa7:Completed 470000 out of 500000 steps (94%)
05:03:27:WU02:FS01:0x22:Completed 260000 out of 2000000 steps (13%)
05:05:55:WU00:FS00:0xa7:Completed 475000 out of 500000 steps (95%)
05:06:58:WU02:FS01:0x22:Completed 280000 out of 2000000 steps (14%)
05:09:52:WU00:FS00:0xa7:Completed 480000 out of 500000 steps (96%)
05:10:30:WU02:FS01:0x22:Completed 300000 out of 2000000 steps (15%)
05:10:32:WU02:FS01:0x22:Checkpoint completed at step 300000
05:13:57:WU00:FS00:0xa7:Completed 485000 out of 500000 steps (97%)
05:14:04:WU02:FS01:0x22:Completed 320000 out of 2000000 steps (16%)
05:17:32:WU02:FS01:0x22:Completed 340000 out of 2000000 steps (17%)
05:17:55:WU00:FS00:0xa7:Completed 490000 out of 500000 steps (98%)
05:21:00:WU02:FS01:0x22:Completed 360000 out of 2000000 steps (18%)
05:21:49:WU00:FS00:0xa7:Completed 495000 out of 500000 steps (99%)
05:21:50:WU01:FS00:Connecting to assign1.foldingathome.org:80
05:21:51:WU01:FS00:Assigned to work server 69.94.66.7
05:21:51:WU01:FS00:Requesting new work unit for slot 00: RUNNING cpu:6 from 69.94.66.7
05:21:51:WU01:FS00:Connecting to 69.94.66.7:8080
05:21:51:WU01:FS00:Downloading 2.82MiB
05:21:51:WU01:FS00:Download complete
05:21:52:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:14379 run:670 clone:4 gen:570 core:0xa7 unit:0x00000282455e42075e932ff939c8b583
05:24:28:WU02:FS01:0x22:Completed 380000 out of 2000000 steps (19%)
05:25:41:WU00:FS00:0xa7:Completed 500000 out of 500000 steps (100%)
05:25:44:WU00:FS00:0xa7:Saving result file ..\logfile_01.txt
05:25:44:WU00:FS00:0xa7:Saving result file ener.edr
05:25:44:WU00:FS00:0xa7:Saving result file frame48.trr
05:25:44:WU00:FS00:0xa7:Saving result file md.log
05:25:44:WU00:FS00:0xa7:Saving result file science.log
05:25:44:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:25:44:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16806 run:18 clone:400 gen:48 core:0xa7 unit:0x0000004582ed0b915eb419782ba2f347
05:25:44:WU00:FS00:Uploading 8.51MiB to 130.237.11.145
05:25:44:WU00:FS00:Connecting to 130.237.11.145:8080
05:25:44:WU01:FS00:Starting
05:25:44:WU01:FS00:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" E:\Users\Gennady\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 01 -suffix 01 -version 706 -lifeline 11896 -checkpoint 15 -np 6
05:25:44:WU01:FS00:Started FahCore on PID 4152
05:25:44:WU01:FS00:Core PID:1964
05:25:44:WU01:FS00:FahCore 0xa7 started
05:25:45:WU01:FS00:0xa7:*********************** Log Started 2020-10-07T05:25:44Z ***********************
05:25:45:WU01:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
05:25:45:WU01:FS00:0xa7:       Type: 0xa7
05:25:45:WU01:FS00:0xa7:       Core: Gromacs
05:25:45:WU01:FS00:0xa7:       Args: -dir 01 -suffix 01 -version 706 -lifeline 4152 -checkpoint 15 -np 6
05:25:45:WU01:FS00:0xa7:************************************ CBang *************************************
05:25:45:WU01:FS00:0xa7:       Date: Nov 27 2019
05:25:45:WU01:FS00:0xa7:       Time: 03:40:09
05:25:45:WU01:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
05:25:45:WU01:FS00:0xa7:     Branch: master
05:25:45:WU01:FS00:0xa7:   Compiler: Visual C++ 2008
05:25:45:WU01:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
05:25:45:WU01:FS00:0xa7:   Platform: win32 10
05:25:45:WU01:FS00:0xa7:       Bits: 64
05:25:45:WU01:FS00:0xa7:       Mode: Release
05:25:45:WU01:FS00:0xa7:************************************ System ************************************
05:25:45:WU01:FS00:0xa7:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
05:25:45:WU01:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
05:25:45:WU01:FS00:0xa7:       CPUs: 8
05:25:45:WU01:FS00:0xa7:     Memory: 31.87GiB
05:25:45:WU01:FS00:0xa7:Free Memory: 18.34GiB
05:25:45:WU01:FS00:0xa7:    Threads: WINDOWS_THREADS
05:25:45:WU01:FS00:0xa7: OS Version: 6.1
05:25:45:WU01:FS00:0xa7:Has Battery: false
05:25:45:WU01:FS00:0xa7: On Battery: false
05:25:45:WU01:FS00:0xa7: UTC Offset: 3
05:25:45:WU01:FS00:0xa7:        PID: 1964
05:25:45:WU01:FS00:0xa7:        CWD: E:\Users\Gennady\AppData\Roaming\FAHClient\work
05:25:45:WU01:FS00:0xa7:******************************** Build - libFAH ********************************
05:25:45:WU01:FS00:0xa7:    Version: 0.0.19
05:25:45:WU01:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
05:25:45:WU01:FS00:0xa7:  Copyright: 2019 foldingathome.org
05:25:45:WU01:FS00:0xa7:   Homepage: https://foldingathome.org/
05:25:45:WU01:FS00:0xa7:       Date: Nov 25 2019
05:25:45:WU01:FS00:0xa7:       Time: 17:12:41
05:25:45:WU01:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
05:25:45:WU01:FS00:0xa7:     Branch: master
05:25:45:WU01:FS00:0xa7:   Compiler: Visual C++ 2008
05:25:45:WU01:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
05:25:45:WU01:FS00:0xa7:   Platform: win32 10
05:25:45:WU01:FS00:0xa7:       Bits: 64
05:25:45:WU01:FS00:0xa7:       Mode: Release
05:25:45:WU01:FS00:0xa7:************************************ Build *************************************
05:25:45:WU01:FS00:0xa7:       SIMD: avx_256
05:25:45:WU01:FS00:0xa7:********************************************************************************
05:25:45:WU01:FS00:0xa7:Project: 14379 (Run 670, Clone 4, Gen 570)
05:25:45:WU01:FS00:0xa7:Unit: 0x00000282455e42075e932ff939c8b583
05:25:45:WU01:FS00:0xa7:Reading tar file core.xml
05:25:45:WU01:FS00:0xa7:Reading tar file frame570.tpr
05:25:45:WU01:FS00:0xa7:Digital signatures verified
05:25:45:WU01:FS00:0xa7:Calling: mdrun -s frame570.tpr -o frame570.trr -cpt 15 -nt 6
05:25:45:WU01:FS00:0xa7:Steps: first=0 total=250000
05:25:47:WU01:FS00:0xa7:Completed 1 out of 250000 steps (0%)
05:25:50:WU00:FS00:Upload 10.29%
05:25:56:WU00:FS00:Upload 22.04%
05:26:02:WU00:FS00:Upload 33.80%
05:26:08:WU00:FS00:Upload 45.56%
05:26:14:WU00:FS00:Upload 57.31%
05:26:20:WU00:FS00:Upload 69.07%
05:26:26:WU00:FS00:Upload 80.83%
05:26:32:WU00:FS00:Upload 92.58%
05:26:36:WU00:FS00:Upload complete
05:26:36:WU00:FS00:Server responded WORK_ACK (400)
05:26:36:WU00:FS00:Final credit estimate, 4222.00 points
05:26:36:WU00:FS00:Cleaning up
05:27:16:WU01:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
05:27:55:WU02:FS01:0x22:Completed 400000 out of 2000000 steps (20%)
05:27:57:WU02:FS01:0x22:Checkpoint completed at step 400000
05:28:46:WU01:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
05:30:15:WU01:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
05:31:24:WU02:FS01:0x22:Completed 420000 out of 2000000 steps (21%)
05:31:44:WU01:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
05:33:13:WU01:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
05:34:42:WU01:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
05:34:52:WU02:FS01:0x22:Completed 440000 out of 2000000 steps (22%)
05:36:11:WU01:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
05:37:40:WU01:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
05:38:20:WU02:FS01:0x22:Completed 460000 out of 2000000 steps (23%)
05:39:09:WU01:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
05:40:38:WU01:FS00:0xa7:Completed 25000 out of 250000 steps (10%)
05:41:47:WU02:FS01:0x22:Completed 480000 out of 2000000 steps (24%)
05:42:08:WU01:FS00:0xa7:Completed 27500 out of 250000 steps (11%)
05:43:37:WU01:FS00:0xa7:Completed 30000 out of 250000 steps (12%)
05:45:06:WU01:FS00:0xa7:Completed 32500 out of 250000 steps (13%)
05:45:15:WU02:FS01:0x22:Completed 500000 out of 2000000 steps (25%)
05:45:16:WU02:FS01:0x22:Checkpoint completed at step 500000
05:46:36:WU01:FS00:0xa7:Completed 35000 out of 250000 steps (14%)
05:48:05:WU01:FS00:0xa7:Completed 37500 out of 250000 steps (15%)
05:48:43:WU02:FS01:0x22:Completed 520000 out of 2000000 steps (26%)
05:49:35:WU01:FS00:0xa7:Completed 40000 out of 250000 steps (16%)
05:51:05:WU01:FS00:0xa7:Completed 42500 out of 250000 steps (17%)
05:52:11:WU02:FS01:0x22:Completed 540000 out of 2000000 steps (27%)
05:52:35:WU01:FS00:0xa7:Completed 45000 out of 250000 steps (18%)
05:54:05:WU01:FS00:0xa7:Completed 47500 out of 250000 steps (19%)
05:55:35:WU01:FS00:0xa7:Completed 50000 out of 250000 steps (20%)
05:55:39:WU02:FS01:0x22:Completed 560000 out of 2000000 steps (28%)
05:57:05:WU01:FS00:0xa7:Completed 52500 out of 250000 steps (21%)
05:58:35:WU01:FS00:0xa7:Completed 55000 out of 250000 steps (22%)
05:59:06:WU02:FS01:0x22:Completed 580000 out of 2000000 steps (29%)
06:00:05:WU01:FS00:0xa7:Completed 57500 out of 250000 steps (23%)
06:01:35:WU01:FS00:0xa7:Completed 60000 out of 250000 steps (24%)
06:02:33:WU02:FS01:0x22:Completed 600000 out of 2000000 steps (30%)
06:02:35:WU02:FS01:0x22:Checkpoint completed at step 600000
06:03:05:WU01:FS00:0xa7:Completed 62500 out of 250000 steps (25%)
06:04:36:WU01:FS00:0xa7:Completed 65000 out of 250000 steps (26%)
06:06:02:WU02:FS01:0x22:Completed 620000 out of 2000000 steps (31%)
06:06:06:WU01:FS00:0xa7:Completed 67500 out of 250000 steps (27%)
06:07:36:WU01:FS00:0xa7:Completed 70000 out of 250000 steps (28%)
06:09:06:WU01:FS00:0xa7:Completed 72500 out of 250000 steps (29%)
06:09:30:WU02:FS01:0x22:Completed 640000 out of 2000000 steps (32%)
06:10:35:WU01:FS00:0xa7:Completed 75000 out of 250000 steps (30%)
06:12:06:WU01:FS00:0xa7:Completed 77500 out of 250000 steps (31%)
06:12:57:WU02:FS01:0x22:Completed 660000 out of 2000000 steps (33%)
06:13:36:WU01:FS00:0xa7:Completed 80000 out of 250000 steps (32%)
06:15:06:WU01:FS00:0xa7:Completed 82500 out of 250000 steps (33%)
06:16:25:WU02:FS01:0x22:Completed 680000 out of 2000000 steps (34%)
06:16:36:WU01:FS00:0xa7:Completed 85000 out of 250000 steps (34%)
06:18:07:WU01:FS00:0xa7:Completed 87500 out of 250000 steps (35%)
06:19:38:WU01:FS00:0xa7:Completed 90000 out of 250000 steps (36%)
06:19:52:WU02:FS01:0x22:Completed 700000 out of 2000000 steps (35%)
06:19:54:WU02:FS01:0x22:Checkpoint completed at step 700000
06:21:08:WU01:FS00:0xa7:Completed 92500 out of 250000 steps (37%)
06:22:38:WU01:FS00:0xa7:Completed 95000 out of 250000 steps (38%)
06:23:21:WU02:FS01:0x22:Completed 720000 out of 2000000 steps (36%)
06:24:08:WU01:FS00:0xa7:Completed 97500 out of 250000 steps (39%)
06:25:37:WU01:FS00:0xa7:Completed 100000 out of 250000 steps (40%)
06:26:49:WU02:FS01:0x22:Completed 740000 out of 2000000 steps (37%)
06:27:08:WU01:FS00:0xa7:Completed 102500 out of 250000 steps (41%)
06:28:38:WU01:FS00:0xa7:Completed 105000 out of 250000 steps (42%)
06:30:08:WU01:FS00:0xa7:Completed 107500 out of 250000 steps (43%)
06:30:16:WU02:FS01:0x22:Completed 760000 out of 2000000 steps (38%)
06:31:38:WU01:FS00:0xa7:Completed 110000 out of 250000 steps (44%)
06:33:09:WU01:FS00:0xa7:Completed 112500 out of 250000 steps (45%)
06:33:43:WU02:FS01:0x22:Completed 780000 out of 2000000 steps (39%)
06:34:39:WU01:FS00:0xa7:Completed 115000 out of 250000 steps (46%)
06:36:09:WU01:FS00:0xa7:Completed 117500 out of 250000 steps (47%)
06:37:11:WU02:FS01:0x22:Completed 800000 out of 2000000 steps (40%)
06:37:13:WU02:FS01:0x22:Checkpoint completed at step 800000
06:37:40:WU01:FS00:0xa7:Completed 120000 out of 250000 steps (48%)
06:39:10:WU01:FS00:0xa7:Completed 122500 out of 250000 steps (49%)
06:40:40:WU02:FS01:0x22:Completed 820000 out of 2000000 steps (41%)
06:40:40:WU01:FS00:0xa7:Completed 125000 out of 250000 steps (50%)
06:42:11:WU01:FS00:0xa7:Completed 127500 out of 250000 steps (51%)
06:43:42:WU01:FS00:0xa7:Completed 130000 out of 250000 steps (52%)
06:44:08:WU02:FS01:0x22:Completed 840000 out of 2000000 steps (42%)
06:45:11:WU01:FS00:0xa7:Completed 132500 out of 250000 steps (53%)
06:46:41:WU01:FS00:0xa7:Completed 135000 out of 250000 steps (54%)
06:47:35:WU02:FS01:0x22:Completed 860000 out of 2000000 steps (43%)
06:48:11:WU01:FS00:0xa7:Completed 137500 out of 250000 steps (55%)
06:49:42:WU01:FS00:0xa7:Completed 140000 out of 250000 steps (56%)
06:51:03:WU02:FS01:0x22:Completed 880000 out of 2000000 steps (44%)
06:51:12:WU01:FS00:0xa7:Completed 142500 out of 250000 steps (57%)
06:52:42:WU01:FS00:0xa7:Completed 145000 out of 250000 steps (58%)
06:54:12:WU01:FS00:0xa7:Completed 147500 out of 250000 steps (59%)
06:54:30:WU02:FS01:0x22:Completed 900000 out of 2000000 steps (45%)
06:54:31:WU02:FS01:0x22:Checkpoint completed at step 900000
06:55:43:WU01:FS00:0xa7:Completed 150000 out of 250000 steps (60%)
06:57:14:WU01:FS00:0xa7:Completed 152500 out of 250000 steps (61%)
06:57:59:WU02:FS01:0x22:Completed 920000 out of 2000000 steps (46%)
06:58:44:WU01:FS00:0xa7:Completed 155000 out of 250000 steps (62%)
07:00:15:WU01:FS00:0xa7:Completed 157500 out of 250000 steps (63%)
07:01:27:WU02:FS01:0x22:Completed 940000 out of 2000000 steps (47%)
07:01:45:WU01:FS00:0xa7:Completed 160000 out of 250000 steps (64%)
07:03:15:WU01:FS00:0xa7:Completed 162500 out of 250000 steps (65%)
07:04:45:WU01:FS00:0xa7:Completed 165000 out of 250000 steps (66%)
07:04:54:WU02:FS01:0x22:Completed 960000 out of 2000000 steps (48%)
07:06:14:WU01:FS00:0xa7:Completed 167500 out of 250000 steps (67%)
07:07:45:WU01:FS00:0xa7:Completed 170000 out of 250000 steps (68%)
07:08:22:WU02:FS01:0x22:Completed 980000 out of 2000000 steps (49%)
07:09:15:WU01:FS00:0xa7:Completed 172500 out of 250000 steps (69%)
07:10:45:WU01:FS00:0xa7:Completed 175000 out of 250000 steps (70%)
07:11:50:WU02:FS01:0x22:Completed 1000000 out of 2000000 steps (50%)
07:11:51:WU02:FS01:0x22:Checkpoint completed at step 1000000
07:12:16:WU01:FS00:0xa7:Completed 177500 out of 250000 steps (71%)
07:13:46:WU01:FS00:0xa7:Completed 180000 out of 250000 steps (72%)
07:15:16:WU01:FS00:0xa7:Completed 182500 out of 250000 steps (73%)
07:15:19:WU02:FS01:0x22:Completed 1020000 out of 2000000 steps (51%)
07:16:46:WU01:FS00:0xa7:Completed 185000 out of 250000 steps (74%)
07:18:18:WU01:FS00:0xa7:Completed 187500 out of 250000 steps (75%)
07:18:46:WU02:FS01:0x22:Completed 1040000 out of 2000000 steps (52%)
07:19:49:WU01:FS00:0xa7:Completed 190000 out of 250000 steps (76%)
07:21:20:WU01:FS00:0xa7:Completed 192500 out of 250000 steps (77%)
07:22:14:WU02:FS01:0x22:Completed 1060000 out of 2000000 steps (53%)
07:22:52:WU01:FS00:0xa7:Completed 195000 out of 250000 steps (78%)
07:24:23:WU01:FS00:0xa7:Completed 197500 out of 250000 steps (79%)
07:25:41:WU02:FS01:0x22:Completed 1080000 out of 2000000 steps (54%)
07:25:54:WU01:FS00:0xa7:Completed 200000 out of 250000 steps (80%)
07:27:23:WU01:FS00:0xa7:Completed 202500 out of 250000 steps (81%)
07:28:53:WU01:FS00:0xa7:Completed 205000 out of 250000 steps (82%)
07:29:08:WU02:FS01:0x22:Completed 1100000 out of 2000000 steps (55%)
07:29:10:WU02:FS01:0x22:Checkpoint completed at step 1100000
07:30:24:WU01:FS00:0xa7:Completed 207500 out of 250000 steps (83%)
07:31:54:WU01:FS00:0xa7:Completed 210000 out of 250000 steps (84%)
07:32:37:WU02:FS01:0x22:Completed 1120000 out of 2000000 steps (56%)
07:33:24:WU01:FS00:0xa7:Completed 212500 out of 250000 steps (85%)
07:34:54:WU01:FS00:0xa7:Completed 215000 out of 250000 steps (86%)
07:36:05:WU02:FS01:0x22:Completed 1140000 out of 2000000 steps (57%)
07:36:24:WU01:FS00:0xa7:Completed 217500 out of 250000 steps (87%)
07:37:54:WU01:FS00:0xa7:Completed 220000 out of 250000 steps (88%)
07:39:25:WU01:FS00:0xa7:Completed 222500 out of 250000 steps (89%)
07:39:32:WU02:FS01:0x22:Completed 1160000 out of 2000000 steps (58%)
07:40:56:WU01:FS00:0xa7:Completed 225000 out of 250000 steps (90%)
07:42:26:WU01:FS00:0xa7:Completed 227500 out of 250000 steps (91%)
07:42:59:WU02:FS01:0x22:Completed 1180000 out of 2000000 steps (59%)
07:43:56:WU01:FS00:0xa7:Completed 230000 out of 250000 steps (92%)
07:45:27:WU01:FS00:0xa7:Completed 232500 out of 250000 steps (93%)
07:46:26:WU02:FS01:0x22:Completed 1200000 out of 2000000 steps (60%)
07:46:28:WU02:FS01:0x22:Checkpoint completed at step 1200000
07:46:57:WU01:FS00:0xa7:Completed 235000 out of 250000 steps (94%)
07:48:27:WU01:FS00:0xa7:Completed 237500 out of 250000 steps (95%)
07:49:55:WU02:FS01:0x22:Completed 1220000 out of 2000000 steps (61%)
07:49:57:WU01:FS00:0xa7:Completed 240000 out of 250000 steps (96%)
07:51:27:WU01:FS00:0xa7:Completed 242500 out of 250000 steps (97%)
07:52:57:WU01:FS00:0xa7:Completed 245000 out of 250000 steps (98%)
07:53:23:WU02:FS01:0x22:Completed 1240000 out of 2000000 steps (62%)
07:54:27:WU01:FS00:0xa7:Completed 247500 out of 250000 steps (99%)
07:54:28:WU00:FS00:Connecting to assign1.foldingathome.org:80
07:54:29:WU00:FS00:Assigned to work server 130.237.11.145
07:54:29:WU00:FS00:Requesting new work unit for slot 00: RUNNING cpu:6 from 130.237.11.145
07:54:29:WU00:FS00:Connecting to 130.237.11.145:8080
07:54:31:WU00:FS00:Downloading 3.30MiB
07:54:32:WU00:FS00:Download complete
07:54:32:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16806 run:10 clone:392 gen:57 core:0xa7 unit:0x0000004f82ed0b915eb41b86ecf57142
07:55:58:WU01:FS00:0xa7:Completed 250000 out of 250000 steps (100%)
07:56:01:WU01:FS00:0xa7:Saving result file ..\logfile_01.txt
07:56:01:WU01:FS00:0xa7:Saving result file dhdl.xvg
07:56:01:WU01:FS00:0xa7:Saving result file frame570.trr
07:56:01:WU01:FS00:0xa7:Saving result file md.log
07:56:01:WU01:FS00:0xa7:Saving result file pullf.xvg
07:56:01:WU01:FS00:0xa7:Saving result file pullx.xvg
07:56:01:WU01:FS00:0xa7:Saving result file science.log
07:56:01:WU01:FS00:0xa7:Saving result file traj_comp.xtc
07:56:01:WU01:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
07:56:02:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
07:56:02:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:14379 run:670 clone:4 gen:570 core:0xa7 unit:0x00000282455e42075e932ff939c8b583
07:56:02:WU01:FS00:Uploading 6.44MiB to 69.94.66.7
07:56:02:WU01:FS00:Connecting to 69.94.66.7:8080
07:56:02:WU00:FS00:Starting
07:56:02:WU00:FS00:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" E:\Users\Gennady\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 00 -suffix 01 -version 706 -lifeline 11896 -checkpoint 15 -np 6
07:56:02:WU00:FS00:Started FahCore on PID 7992
07:56:02:WU00:FS00:Core PID:7972
07:56:02:WU00:FS00:FahCore 0xa7 started
07:56:02:WU00:FS00:0xa7:*********************** Log Started 2020-10-07T07:56:02Z ***********************
07:56:02:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:56:02:WU00:FS00:0xa7:       Type: 0xa7
07:56:02:WU00:FS00:0xa7:       Core: Gromacs
07:56:02:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 7992 -checkpoint 15 -np 6
07:56:02:WU00:FS00:0xa7:************************************ CBang *************************************
07:56:02:WU00:FS00:0xa7:       Date: Nov 27 2019
07:56:02:WU00:FS00:0xa7:       Time: 03:40:09
07:56:02:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
07:56:02:WU00:FS00:0xa7:     Branch: master
07:56:02:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
07:56:02:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:56:02:WU00:FS00:0xa7:   Platform: win32 10
07:56:02:WU00:FS00:0xa7:       Bits: 64
07:56:02:WU00:FS00:0xa7:       Mode: Release
07:56:02:WU00:FS00:0xa7:************************************ System ************************************
07:56:02:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
07:56:02:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
07:56:02:WU00:FS00:0xa7:       CPUs: 8
07:56:02:WU00:FS00:0xa7:     Memory: 31.87GiB
07:56:02:WU00:FS00:0xa7:Free Memory: 17.92GiB
07:56:02:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
07:56:02:WU00:FS00:0xa7: OS Version: 6.1
07:56:02:WU00:FS00:0xa7:Has Battery: false
07:56:02:WU00:FS00:0xa7: On Battery: false
07:56:02:WU00:FS00:0xa7: UTC Offset: 3
07:56:02:WU00:FS00:0xa7:        PID: 7972
07:56:02:WU00:FS00:0xa7:        CWD: E:\Users\Gennady\AppData\Roaming\FAHClient\work
07:56:02:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
07:56:02:WU00:FS00:0xa7:    Version: 0.0.19
07:56:02:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:56:02:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:56:02:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
07:56:02:WU00:FS00:0xa7:       Date: Nov 25 2019
07:56:02:WU00:FS00:0xa7:       Time: 17:12:41
07:56:02:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
07:56:02:WU00:FS00:0xa7:     Branch: master
07:56:02:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
07:56:02:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:56:02:WU00:FS00:0xa7:   Platform: win32 10
07:56:02:WU00:FS00:0xa7:       Bits: 64
07:56:02:WU00:FS00:0xa7:       Mode: Release
07:56:02:WU00:FS00:0xa7:************************************ Build *************************************
07:56:02:WU00:FS00:0xa7:       SIMD: avx_256
07:56:02:WU00:FS00:0xa7:********************************************************************************
07:56:02:WU00:FS00:0xa7:Project: 16806 (Run 10, Clone 392, Gen 57)
07:56:02:WU00:FS00:0xa7:Unit: 0x0000004f82ed0b915eb41b86ecf57142
07:56:02:WU00:FS00:0xa7:Reading tar file core.xml
07:56:02:WU00:FS00:0xa7:Reading tar file frame57.tpr
07:56:02:WU00:FS00:0xa7:Digital signatures verified
07:56:02:WU00:FS00:0xa7:Calling: mdrun -s frame57.tpr -o frame57.trr -cpt 15 -nt 6
07:56:02:WU00:FS00:0xa7:Steps: first=28500000 total=500000
07:56:04:WU00:FS00:0xa7:Completed 1 out of 500000 steps (0%)
07:56:08:WU01:FS00:Upload 23.28%
07:56:14:WU01:FS00:Upload 46.57%
07:56:20:WU01:FS00:Upload 70.82%
07:56:26:WU01:FS00:Upload 95.07%
07:56:27:WU01:FS00:Upload complete
07:56:27:WU01:FS00:Server responded WORK_QUIT (404)
07:56:27:WARNING:WU01:FS00:Server did not like results, dumping
07:56:27:WU01:FS00:Cleaning up
07:56:50:WU02:FS01:0x22:Completed 1260000 out of 2000000 steps (63%)
07:59:53:WU00:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
08:00:17:WU02:FS01:0x22:Completed 1280000 out of 2000000 steps (64%)
08:03:43:WU02:FS01:0x22:Completed 1300000 out of 2000000 steps (65%)
08:03:45:WU02:FS01:0x22:Checkpoint completed at step 1300000
08:03:47:WU00:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
08:07:11:WU02:FS01:0x22:Completed 1320000 out of 2000000 steps (66%)
08:07:35:WU00:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
08:10:38:WU02:FS01:0x22:Completed 1340000 out of 2000000 steps (67%)
08:11:24:WU00:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
08:14:05:WU02:FS01:0x22:Completed 1360000 out of 2000000 steps (68%)
08:15:13:WU00:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
08:17:32:WU02:FS01:0x22:Completed 1380000 out of 2000000 steps (69%)
08:19:02:WU00:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
08:20:59:WU02:FS01:0x22:Completed 1400000 out of 2000000 steps (70%)
08:21:00:WU02:FS01:0x22:Checkpoint completed at step 1400000
08:22:51:WU00:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
08:24:27:WU02:FS01:0x22:Completed 1420000 out of 2000000 steps (71%)
08:26:40:WU00:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
08:27:54:WU02:FS01:0x22:Completed 1440000 out of 2000000 steps (72%)
08:30:28:WU00:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
08:31:21:WU02:FS01:0x22:Completed 1460000 out of 2000000 steps (73%)
08:34:18:WU00:FS00:0xa7:Completed 50000 out of 500000 steps (10%)
08:34:49:WU02:FS01:0x22:Completed 1480000 out of 2000000 steps (74%)
08:38:09:WU00:FS00:0xa7:Completed 55000 out of 500000 steps (11%)

I also noticed a line, this is most likely the result of a previous incorrect shutdown.

Code: Select all

07:56:27:WARNING:WU01:FS00:Server did not like results, dumping

[Joe_H]

I also noticed a line, this is most likely the result of a previous incorrect shutdown.

Or in this case possibly a problem on the server end. The WS it was sent back to is currently reporting no space left, I will check to see if that has been reported already.

[Gennady]

At all, will the dump of the crashed process help you to fix the problem I described at the beginning?
If so, please let me know, I will provide it to the development team.

[foldy]

If you are the only one getting this error then most likely it is something on your machine. What was the time of the crash and WER dump? And what are the corresponding logging lines from FAH? The older logs are stored at FAH user folder e.g. C:\ProgramData\FAHClient\logs or C:\Users\oemuser\AppData\Roaming\FAHClient\logs

On the other hand maybe it is worth a github issue at openmm because destructor of CudaArray should never raise exception in CudaSort

[Gennady]

If you are the only one getting this error then most likely it is something on your machine. What was the time of the crash and WER dump? And what are the corresponding logging lines from FAH? The older logs are stored at FAH user folder e.g. C:\ProgramData\FAHClient\logs or C:\Users\oemuser\AppData\Roaming\FAHClient\logs

I found the required log.
according to the time specified in the dump:

Code: Select all

Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Sun Oct  4 19:47:41.000 2020 (UTC + 3:00)
System Uptime: not available
Process Uptime: 0 days 14:18:28.000

We need 16:47 time, since the logs inside are in UTC format.
Here is a log of the required timeline (16:47:42:WARNING:WU02:FS01:FahCore returned an unknown error code which probably indicates that it crashed)

Code: Select all

15:20:07:WU00:FS00:0xa7:*********************** Log Started 2020-10-04T15:20:07Z ***********************
15:20:07:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
15:20:07:WU00:FS00:0xa7:       Type: 0xa7
15:20:07:WU00:FS00:0xa7:       Core: Gromacs
15:20:07:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 24424 -checkpoint 15 -np
15:20:07:WU00:FS00:0xa7:             6
15:20:07:WU00:FS00:0xa7:************************************ CBang *************************************
15:20:07:WU00:FS00:0xa7:       Date: Nov 27 2019
15:20:07:WU00:FS00:0xa7:       Time: 03:40:09
15:20:07:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
15:20:07:WU00:FS00:0xa7:     Branch: master
15:20:07:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
15:20:07:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:20:07:WU00:FS00:0xa7:   Platform: win32 10
15:20:07:WU00:FS00:0xa7:       Bits: 64
15:20:07:WU00:FS00:0xa7:       Mode: Release
15:20:07:WU00:FS00:0xa7:************************************ System ************************************
15:20:07:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
15:20:07:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
15:20:07:WU00:FS00:0xa7:       CPUs: 8
15:20:07:WU00:FS00:0xa7:     Memory: 31.87GiB
15:20:07:WU00:FS00:0xa7:Free Memory: 1.75GiB
15:20:07:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
15:20:07:WU00:FS00:0xa7: OS Version: 6.1
15:20:07:WU00:FS00:0xa7:Has Battery: false
15:20:07:WU00:FS00:0xa7: On Battery: false
15:20:07:WU00:FS00:0xa7: UTC Offset: 3
15:20:07:WU00:FS00:0xa7:        PID: 25656
15:20:07:WU00:FS00:0xa7:        CWD: E:\Users\Gennady\AppData\Roaming\FAHClient\work
15:20:07:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
15:20:07:WU00:FS00:0xa7:    Version: 0.0.19
15:20:07:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:20:07:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
15:20:07:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
15:20:07:WU00:FS00:0xa7:       Date: Nov 25 2019
15:20:07:WU00:FS00:0xa7:       Time: 17:12:41
15:20:07:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
15:20:07:WU00:FS00:0xa7:     Branch: master
15:20:07:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
15:20:07:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
15:20:07:WU00:FS00:0xa7:   Platform: win32 10
15:20:07:WU00:FS00:0xa7:       Bits: 64
15:20:07:WU00:FS00:0xa7:       Mode: Release
15:20:07:WU00:FS00:0xa7:************************************ Build *************************************
15:20:07:WU00:FS00:0xa7:       SIMD: avx_256
15:20:07:WU00:FS00:0xa7:********************************************************************************
15:20:07:WU00:FS00:0xa7:Project: 14828 (Run 540, Clone 3, Gen 211)
15:20:07:WU00:FS00:0xa7:Unit: 0x000000f69bf7a4d65ea6f0070f148ea7
15:20:07:WU00:FS00:0xa7:Reading tar file core.xml
15:20:07:WU00:FS00:0xa7:Reading tar file frame211.tpr
15:20:07:WU00:FS00:0xa7:Digital signatures verified
15:20:07:WU00:FS00:0xa7:Calling: mdrun -s frame211.tpr -o frame211.trr -cpt 15 -nt 6
15:20:08:WU00:FS00:0xa7:Steps: first=0 total=250000
15:20:11:WU00:FS00:0xa7:Completed 1 out of 250000 steps (0%)
15:20:12:WU01:FS00:Upload 14.05%
15:20:18:WU01:FS00:Upload 28.11%
15:20:24:WU01:FS00:Upload 42.16%
15:20:30:WU01:FS00:Upload 57.00%
15:20:36:WU01:FS00:Upload 71.05%
15:20:42:WU01:FS00:Upload 85.10%
15:20:48:WU01:FS00:Upload 99.16%
15:20:48:WU01:FS00:Upload complete
15:20:48:WU01:FS00:Server responded WORK_ACK (400)
15:20:48:WU01:FS00:Final credit estimate, 10763.00 points
15:20:48:WU01:FS00:Cleaning up
15:21:44:WU00:FS00:0xa7:Completed 2500 out of 250000 steps (1%)
15:23:15:WU00:FS00:0xa7:Completed 5000 out of 250000 steps (2%)
15:24:46:WU00:FS00:0xa7:Completed 7500 out of 250000 steps (3%)
15:26:19:WU00:FS00:0xa7:Completed 10000 out of 250000 steps (4%)
15:27:50:WU00:FS00:0xa7:Completed 12500 out of 250000 steps (5%)
15:29:22:WU00:FS00:0xa7:Completed 15000 out of 250000 steps (6%)
15:30:54:WU00:FS00:0xa7:Completed 17500 out of 250000 steps (7%)
15:32:25:WU00:FS00:0xa7:Completed 20000 out of 250000 steps (8%)
15:33:56:WU00:FS00:0xa7:Completed 22500 out of 250000 steps (9%)
15:35:28:WU00:FS00:0xa7:Completed 25000 out of 250000 steps (10%)
15:37:00:WU00:FS00:0xa7:Completed 27500 out of 250000 steps (11%)
15:38:31:WU00:FS00:0xa7:Completed 30000 out of 250000 steps (12%)
15:40:02:WU00:FS00:0xa7:Completed 32500 out of 250000 steps (13%)
15:41:32:WU00:FS00:0xa7:Completed 35000 out of 250000 steps (14%)
15:43:03:WU00:FS00:0xa7:Completed 37500 out of 250000 steps (15%)
15:44:35:WU00:FS00:0xa7:Completed 40000 out of 250000 steps (16%)
15:46:06:WU00:FS00:0xa7:Completed 42500 out of 250000 steps (17%)
15:47:37:WU00:FS00:0xa7:Completed 45000 out of 250000 steps (18%)
15:49:08:WU00:FS00:0xa7:Completed 47500 out of 250000 steps (19%)
15:50:39:WU00:FS00:0xa7:Completed 50000 out of 250000 steps (20%)
15:52:11:WU00:FS00:0xa7:Completed 52500 out of 250000 steps (21%)
15:53:42:WU00:FS00:0xa7:Completed 55000 out of 250000 steps (22%)
15:55:12:WU00:FS00:0xa7:Completed 57500 out of 250000 steps (23%)
15:56:44:WU00:FS00:0xa7:Completed 60000 out of 250000 steps (24%)
15:58:16:WU00:FS00:0xa7:Completed 62500 out of 250000 steps (25%)
15:59:48:WU00:FS00:0xa7:Completed 65000 out of 250000 steps (26%)
16:01:20:WU00:FS00:0xa7:Completed 67500 out of 250000 steps (27%)
16:02:52:WU00:FS00:0xa7:Completed 70000 out of 250000 steps (28%)
16:04:23:WU00:FS00:0xa7:Completed 72500 out of 250000 steps (29%)
16:05:55:WU00:FS00:0xa7:Completed 75000 out of 250000 steps (30%)
16:07:25:WU00:FS00:0xa7:Completed 77500 out of 250000 steps (31%)
16:08:56:WU00:FS00:0xa7:Completed 80000 out of 250000 steps (32%)
16:10:27:WU00:FS00:0xa7:Completed 82500 out of 250000 steps (33%)
16:11:59:WU00:FS00:0xa7:Completed 85000 out of 250000 steps (34%)
16:13:32:WU00:FS00:0xa7:Completed 87500 out of 250000 steps (35%)
16:15:03:WU00:FS00:0xa7:Completed 90000 out of 250000 steps (36%)
16:16:34:WU00:FS00:0xa7:Completed 92500 out of 250000 steps (37%)
16:18:09:WU00:FS00:0xa7:Completed 95000 out of 250000 steps (38%)
16:19:40:WU00:FS00:0xa7:Completed 97500 out of 250000 steps (39%)
16:21:11:WU00:FS00:0xa7:Completed 100000 out of 250000 steps (40%)
16:23:31:WU00:FS00:0xa7:Completed 102500 out of 250000 steps (41%)
16:25:08:WU00:FS00:0xa7:Completed 105000 out of 250000 steps (42%)
16:26:42:WU00:FS00:0xa7:Completed 107500 out of 250000 steps (43%)
16:28:15:WU00:FS00:0xa7:Completed 110000 out of 250000 steps (44%)
16:29:54:WU00:FS00:0xa7:Completed 112500 out of 250000 steps (45%)
16:31:27:WU00:FS00:0xa7:Completed 115000 out of 250000 steps (46%)
16:32:58:WU00:FS00:0xa7:Completed 117500 out of 250000 steps (47%)
16:34:33:WU00:FS00:0xa7:Completed 120000 out of 250000 steps (48%)
16:36:42:WU00:FS00:0xa7:Completed 122500 out of 250000 steps (49%)
16:38:40:WU00:FS00:0xa7:Completed 125000 out of 250000 steps (50%)
16:40:23:WU00:FS00:0xa7:Completed 127500 out of 250000 steps (51%)
16:42:09:WU00:FS00:0xa7:Completed 130000 out of 250000 steps (52%)
16:43:51:WU00:FS00:0xa7:Completed 132500 out of 250000 steps (53%)
16:45:32:WU00:FS00:0xa7:Completed 135000 out of 250000 steps (54%)
16:47:12:WU00:FS00:0xa7:Completed 137500 out of 250000 steps (55%)
16:47:42:WARNING:WU02:FS01:FahCore returned an unknown error code which probably indicates that it crashed
16:47:42:WARNING:WU02:FS01:FahCore returned: UNKNOWN_ENUM (3 = 0x3)
16:47:42:WU02:FS01:Starting
16:47:42:WU02:FS01:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" E:\Users\Gennady\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 11528 -checkpoint 15 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0
16:47:42:WU02:FS01:Started FahCore on PID 27736
16:47:42:WU02:FS01:Core PID:23180
16:47:42:WU02:FS01:FahCore 0x22 started
16:47:43:WU02:FS01:0x22:*********************** Log Started 2020-10-04T16:47:42Z ***********************

On the other hand maybe it is worth a github issue at openmm because destructor of CudaArray should never raise exception in CudaSort

I used the standard source for the installation from the site. Also, the required module was obtained automatically, as it was said for this update for CUDA.
Also contrary, according by the callstack, the array destructor, called the sorting that's caused the crash. Please take a look on crash stack again.

Code: Select all

11 00000000`0025e930 000007fe`e30c3c8a KERNELBASE!RaiseException+0x3d
12 00000000`0025ea00 000007fe`e2fef68d OpenMMCUDA!OpenMM::CudaSort::sort+0x540a                          <---- caused exception
13 00000000`0025ea80 000007fe`e3035bc8 OpenMMCUDA!OpenMM::CudaArray::~CudaArray+0x22d               <--- destructor call CudaSort::sort
14 00000000`0025ec30 000007fe`e11d7d32 OpenMMCUDA!OpenMM::CudaIntegrationUtilities::setNextStepSize+0x5308
15 00000000`0025ec60 000007fe`e1189692 OpenMM!OpenMM::HarmonicBondForce::usesPeriodicBoundaryConditions+0x72
16 00000000`0025ec90 000007fe`e1184ad7 OpenMM!OpenMM::ContextImpl::~ContextImpl+0x62
17 00000000`0025ece0 00000001`3f123bb5 OpenMM!OpenMM::Context::~Context+0x27
18 00000000`0025ed20 00000001`3f14e282 FahCore_22+0x43bb5

[foldy]

The good news is FAHCore_22 just restarted after the error. If this was a one time error then maybe ignore it. If it happens more often then it may be worth to look into it.

[Gennady]

This problem occurs (happened) when the program stops working. Those in fact, extremely rare in my case. After manual launch, most likely that line appeared in the logs.
I suspect that this problem has something to do with a change in the performance of the video card. In fact, we had a problem with electricity and the computer switched to UPS. To prepare the computer for shutdown, I stopping folding client and it gave this error. Some devices have completely different behavior when transitioning from D0 to D1 and higher states (what the system does with them when switching to power from the UPS).
Those, in fact, this is an extremely rare problem. I reported, maybe someone came across it. And maybe someone will help fix the deinitialization logic.