WU 13416 Exception access violation

[Swedis]

I had several runs of project 134xx and are aware of the experimental status of these with several crasches and restarts but now i encountered a "new" error i haven´t had before.
See logs below from this specific WU.

Code: Select all

16:39:48:Trying to access database...
16:39:48:Successfully acquired database lock
16:39:48:Downloading GPUs.txt from assign1.foldingathome.org:80
16:39:48:Connecting to assign1.foldingathome.org:80
16:39:49:Read GPUs.txt
16:39:49:Enabled folding slot 00: READY cpu:4
16:39:49:Enabled folding slot 01: READY gpu:0:Hawaii [Radeon R9 200/300X Series]
16:39:49:****************************** FAHClient ******************************
16:39:49:        Version: 7.6.13
16:39:49:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:39:49:      Copyright: 2020 foldingathome.org
16:39:49:       Homepage: https://foldingathome.org/
16:39:49:           Date: Apr 27 2020
16:39:49:           Time: 21:21:01
16:39:49:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
16:39:49:         Branch: master
16:39:49:       Compiler: Visual C++ 2008
16:39:49:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:39:49:       Platform: win32 10
16:39:49:           Bits: 32
16:39:49:           Mode: Release
16:39:49:           Args: --open-web-control
16:39:49:         Config: C:\Users\Swedis\AppData\Roaming\FAHClient\config.xml
16:39:49:******************************** CBang ********************************
16:39:49:           Date: Apr 24 2020
16:39:49:           Time: 17:07:55
16:39:49:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
16:39:49:         Branch: master
16:39:49:       Compiler: Visual C++ 2008
16:39:49:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:39:49:       Platform: win32 10
16:39:49:           Bits: 32
16:39:49:           Mode: Release
16:39:49:******************************* System ********************************
16:39:49:            CPU: Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz
16:39:49:         CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
16:39:49:           CPUs: 8
16:39:49:         Memory: 15.94GiB
16:39:49:    Free Memory: 11.15GiB
16:39:49:        Threads: WINDOWS_THREADS
16:39:49:     OS Version: 6.2
16:39:49:    Has Battery: false
16:39:49:     On Battery: false
16:39:49:     UTC Offset: 2
16:39:49:            PID: 14784
16:39:49:            CWD: C:\Users\Swedis\AppData\Roaming\FAHClient
16:39:49:  Win32 Service: false
16:39:49:             OS: Windows 10 Enterprise
16:39:49:        OS Arch: AMD64
16:39:49:           GPUs: 1
16:39:49:          GPU 0: Bus:1 Slot:0 Func:0 AMD:5 Hawaii [Radeon R9 200/300X Series]
16:39:49:           CUDA: Not detected: Failed to open dynamic library 'nvcuda.dll': Det
16:39:49:                 gr inte att hitta den angivna modulen.
16:39:49:
16:39:49:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:3004.8
16:39:49:******************************* libFAH ********************************
16:39:49:           Date: Apr 15 2020
16:39:49:           Time: 14:53:14
16:39:49:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
16:39:49:         Branch: master
16:39:49:       Compiler: Visual C++ 2008
16:39:49:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:39:49:       Platform: win32 10
16:39:49:           Bits: 32
16:39:49:           Mode: Release
16:39:49:***********************************************************************
16:39:49:<config>
16:39:49:  <!-- Folding Core -->
16:39:49:  <checkpoint v='5'/>
16:39:49:  <core-priority v='low'/>
16:39:49:
16:39:49:  <!-- Folding Slot Configuration -->
16:39:49:  <cause v='COVID_19'/>
16:39:49:  <client-type v='advanced'/>
16:39:49:
16:39:49:  <!-- HTTP Server -->
16:39:49:  <allow v='127.0.0.1 192.168.1.0/24'/>
16:39:49:
16:39:49:  <!-- Network -->
16:39:49:  <proxy v=':8080'/>
16:39:49:
16:39:49:  <!-- Remote Command Server -->
16:39:49:  <password v='*****'/>
16:39:49:
16:39:49:  <!-- Slot Control -->
16:39:49:  <pause-on-battery v='false'/>
16:39:49:  <power v='full'/>
16:39:49:
16:39:49:  <!-- User Information -->
16:39:49:  <passkey v='*****'/>
16:39:49:  <team v='*****'/>
16:39:49:  <user v='*****'/>
16:39:49:
16:39:49:  <!-- Work Unit Control -->
16:39:49:  <next-unit-percentage v='100'/>
16:39:49:
16:39:49:  <!-- Folding Slots -->
16:39:49:  <slot id='0' type='CPU'>
16:39:49:    <cpus v='4'/>
16:39:49:  </slot>
16:39:49:  <slot id='1' type='GPU'/>
16:39:49:</config>

Code: Select all

04:53:15:WU00:FS01:Starting
04:53:15:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Swedis\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 14784 -checkpoint 5 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
04:53:15:WU00:FS01:Started FahCore on PID 17644
04:53:15:WU00:FS01:Core PID:15568
04:53:15:WU00:FS01:FahCore 0x22 started
04:53:15:WU00:FS01:0x22:*********************** Log Started 2020-07-10T04:53:15Z ***********************
04:53:15:WU00:FS01:0x22:*************************** Core22 Folding@home Core ***************************
04:53:15:WU00:FS01:0x22:       Core: Core22
04:53:15:WU00:FS01:0x22:       Type: 0x22
04:53:15:WU00:FS01:0x22:    Version: 0.0.11
04:53:15:WU00:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
04:53:15:WU00:FS01:0x22:  Copyright: 2020 foldingathome.org
04:53:15:WU00:FS01:0x22:   Homepage: https://foldingathome.org/
04:53:15:WU00:FS01:0x22:       Date: Jun 26 2020
04:53:15:WU00:FS01:0x22:       Time: 19:49:16
04:53:15:WU00:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
04:53:15:WU00:FS01:0x22:     Branch: core22-0.0.11
04:53:15:WU00:FS01:0x22:   Compiler: Visual C++ 2015
04:53:15:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
04:53:15:WU00:FS01:0x22:   Platform: win32 10
04:53:15:WU00:FS01:0x22:       Bits: 64
04:53:15:WU00:FS01:0x22:       Mode: Release
04:53:15:WU00:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
04:53:15:WU00:FS01:0x22:             <peastman@stanford.edu>
04:53:15:WU00:FS01:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 17644 -checkpoint 5
04:53:15:WU00:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
04:53:15:WU00:FS01:0x22:************************************ libFAH ************************************
04:53:15:WU00:FS01:0x22:       Date: Jun 26 2020
04:53:15:WU00:FS01:0x22:       Time: 19:47:12
04:53:15:WU00:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
04:53:15:WU00:FS01:0x22:     Branch: HEAD
04:53:15:WU00:FS01:0x22:   Compiler: Visual C++ 2015
04:53:15:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
04:53:15:WU00:FS01:0x22:   Platform: win32 10
04:53:15:WU00:FS01:0x22:       Bits: 64
04:53:15:WU00:FS01:0x22:       Mode: Release
04:53:15:WU00:FS01:0x22:************************************ CBang *************************************
04:53:15:WU00:FS01:0x22:       Date: Jun 26 2020
04:53:15:WU00:FS01:0x22:       Time: 19:46:11
04:53:15:WU00:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
04:53:15:WU00:FS01:0x22:     Branch: master
04:53:15:WU00:FS01:0x22:   Compiler: Visual C++ 2015
04:53:15:WU00:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
04:53:15:WU00:FS01:0x22:   Platform: win32 10
04:53:15:WU00:FS01:0x22:       Bits: 64
04:53:15:WU00:FS01:0x22:       Mode: Release
04:53:15:WU00:FS01:0x22:************************************ System ************************************
04:53:15:WU00:FS01:0x22:        CPU: Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz
04:53:15:WU00:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
04:53:15:WU00:FS01:0x22:       CPUs: 8
04:53:15:WU00:FS01:0x22:     Memory: 15.94GiB
04:53:15:WU00:FS01:0x22:Free Memory: 11.32GiB
04:53:15:WU00:FS01:0x22:    Threads: WINDOWS_THREADS
04:53:15:WU00:FS01:0x22: OS Version: 6.2
04:53:15:WU00:FS01:0x22:Has Battery: false
04:53:15:WU00:FS01:0x22: On Battery: false
04:53:15:WU00:FS01:0x22: UTC Offset: 2
04:53:15:WU00:FS01:0x22:        PID: 15568
04:53:15:WU00:FS01:0x22:        CWD: C:\Users\Swedis\AppData\Roaming\FAHClient\work
04:53:15:WU00:FS01:0x22:********************************************************************************
04:53:15:WU00:FS01:0x22:Project: 13416 (Run 1110, Clone 187, Gen 1)
04:53:15:WU00:FS01:0x22:Unit: 0x0000000212bc7d9a5f02af7c6ff31c57
04:53:15:WU00:FS01:0x22:Reading tar file core.xml
04:53:15:WU00:FS01:0x22:Reading tar file integrator.xml
04:53:15:WU00:FS01:0x22:Reading tar file state.xml.bz2
04:53:15:WU00:FS01:0x22:Reading tar file system.xml.bz2
04:53:15:WU00:FS01:0x22:Digital signatures verified
04:53:15:WU00:FS01:0x22:Folding@home GPU Core22 Folding@home Core
04:53:15:WU00:FS01:0x22:Version 0.0.11
04:53:15:WU00:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
04:53:15:WU00:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
04:53:16:WU00:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
04:53:16:WU00:FS01:0x22:  Global context and integrator variables write interval: 2500 steps (0.25%) [400 total]
04:53:40:WU00:FS01:0x22:Completed 0 out of 1000000 steps (0%)
04:58:47:WU00:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
.
.
06:35:12:WU00:FS01:0x22:Completed 200000 out of 1000000 steps (20%)
06:40:19:WU00:FS01:0x22:Completed 210000 out of 1000000 steps (21%)
06:45:24:WU00:FS01:0x22:Completed 220000 out of 1000000 steps (22%)
06:50:27:WU00:FS01:0x22:Completed 230000 out of 1000000 steps (23%)
06:55:30:WU00:FS01:0x22:Completed 240000 out of 1000000 steps (24%)
07:00:36:WU00:FS01:0x22:Completed 250000 out of 1000000 steps (25%)
07:05:43:WU00:FS01:0x22:Completed 260000 out of 1000000 steps (26%)
07:10:51:WU00:FS01:0x22:Completed 270000 out of 1000000 steps (27%)
07:16:01:WU00:FS01:0x22:Completed 280000 out of 1000000 steps (28%)
07:21:12:WU00:FS01:0x22:Completed 290000 out of 1000000 steps (29%)
07:26:21:WU00:FS01:0x22:Completed 300000 out of 1000000 steps (30%)
07:26:22:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
07:31:34:WU00:FS01:0x22:Completed 310000 out of 1000000 steps (31%)
07:36:44:WU00:FS01:0x22:Completed 320000 out of 1000000 steps (32%)
07:41:55:WU00:FS01:0x22:Completed 330000 out of 1000000 steps (33%)
07:47:04:WU00:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
07:52:12:WU00:FS01:0x22:Completed 350000 out of 1000000 steps (35%)
07:52:13:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
07:57:21:WU00:FS01:0x22:Completed 360000 out of 1000000 steps (36%)
08:02:30:WU00:FS01:0x22:Completed 370000 out of 1000000 steps (37%)
08:07:41:WU00:FS01:0x22:Completed 380000 out of 1000000 steps (38%)
08:12:53:WU00:FS01:0x22:Completed 390000 out of 1000000 steps (39%)
08:18:06:WU00:FS01:0x22:Completed 400000 out of 1000000 steps (40%)
08:18:07:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
08:23:18:WU00:FS01:0x22:Completed 410000 out of 1000000 steps (41%)
08:28:28:WU00:FS01:0x22:Completed 420000 out of 1000000 steps (42%)
08:33:41:WU00:FS01:0x22:Completed 430000 out of 1000000 steps (43%)
08:38:54:WU00:FS01:0x22:Completed 440000 out of 1000000 steps (44%)
08:44:08:WU00:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
08:44:09:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
08:49:23:WU00:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
08:54:38:WU00:FS01:0x22:Completed 470000 out of 1000000 steps (47%)
08:59:52:WU00:FS01:0x22:Completed 480000 out of 1000000 steps (48%)
09:05:07:WU00:FS01:0x22:Completed 490000 out of 1000000 steps (49%)
09:10:24:WU00:FS01:0x22:Completed 500000 out of 1000000 steps (50%)
09:10:25:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
09:15:40:WU00:FS01:0x22:Completed 510000 out of 1000000 steps (51%)
09:20:55:WU00:FS01:0x22:Completed 520000 out of 1000000 steps (52%)
09:26:11:WU00:FS01:0x22:Completed 530000 out of 1000000 steps (53%)
09:31:26:WU00:FS01:0x22:Completed 540000 out of 1000000 steps (54%)
09:36:41:WU00:FS01:0x22:Completed 550000 out of 1000000 steps (55%)
09:36:42:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
09:41:57:WU00:FS01:0x22:Completed 560000 out of 1000000 steps (56%)
09:47:12:WU00:FS01:0x22:Completed 570000 out of 1000000 steps (57%)
09:52:27:WU00:FS01:0x22:Completed 580000 out of 1000000 steps (58%)
09:57:41:WU00:FS01:0x22:Completed 590000 out of 1000000 steps (59%)
10:02:58:WU00:FS01:0x22:Completed 600000 out of 1000000 steps (60%)
10:02:59:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
10:08:15:WU00:FS01:0x22:Completed 610000 out of 1000000 steps (61%)
10:13:32:WU00:FS01:0x22:Completed 620000 out of 1000000 steps (62%)
10:18:48:WU00:FS01:0x22:Completed 630000 out of 1000000 steps (63%)
10:24:04:WU00:FS01:0x22:Completed 640000 out of 1000000 steps (64%)
10:29:21:WU00:FS01:0x22:Completed 650000 out of 1000000 steps (65%)
10:29:22:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
10:34:38:WU00:FS01:0x22:Completed 660000 out of 1000000 steps (66%)
10:39:55:WU00:FS01:0x22:Completed 670000 out of 1000000 steps (67%)
10:45:11:WU00:FS01:0x22:Completed 680000 out of 1000000 steps (68%)
10:50:27:WU00:FS01:0x22:Completed 690000 out of 1000000 steps (69%)
10:55:43:WU00:FS01:0x22:Completed 700000 out of 1000000 steps (70%)
10:55:44:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
11:00:59:WU00:FS01:0x22:Completed 710000 out of 1000000 steps (71%)
11:06:14:WU00:FS01:0x22:Completed 720000 out of 1000000 steps (72%)
11:11:31:WU00:FS01:0x22:Completed 730000 out of 1000000 steps (73%)
******************************* Date: 2020-07-10 *******************************
11:16:48:WU00:FS01:0x22:Completed 740000 out of 1000000 steps (74%)
11:22:04:WU00:FS01:0x22:Completed 750000 out of 1000000 steps (75%)
11:22:05:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
11:27:23:WU00:FS01:0x22:Completed 760000 out of 1000000 steps (76%)
11:32:41:WU00:FS01:0x22:Completed 770000 out of 1000000 steps (77%)
11:37:58:WU00:FS01:0x22:Completed 780000 out of 1000000 steps (78%)
11:43:15:WU00:FS01:0x22:Completed 790000 out of 1000000 steps (79%)
11:48:32:WU00:FS01:0x22:Completed 800000 out of 1000000 steps (80%)
11:48:33:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
11:53:49:WU00:FS01:0x22:Completed 810000 out of 1000000 steps (81%)
11:59:03:WU00:FS01:0x22:Completed 820000 out of 1000000 steps (82%)
12:04:19:WU00:FS01:0x22:Completed 830000 out of 1000000 steps (83%)
12:09:34:WU00:FS01:0x22:Completed 840000 out of 1000000 steps (84%)
12:14:50:WU00:FS01:0x22:Completed 850000 out of 1000000 steps (85%)
12:14:52:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
12:20:08:WU00:FS01:0x22:Completed 860000 out of 1000000 steps (86%)
12:25:25:WU00:FS01:0x22:Completed 870000 out of 1000000 steps (87%)
12:30:40:WU00:FS01:0x22:Completed 880000 out of 1000000 steps (88%)
12:35:56:WU00:FS01:0x22:Completed 890000 out of 1000000 steps (89%)
12:41:12:WU00:FS01:0x22:Completed 900000 out of 1000000 steps (90%)
12:41:13:WU00:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
12:41:13:WU00:FS01:0x22:An exception occurred at step 900000: Win32: 0xc0000005: Exception access violation
12:41:13:WU00:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
12:41:13:WU00:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
12:41:15:WU01:FS01:Starting
12:41:15:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Swedis\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 14784 -checkpoint 5 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
12:41:15:WU01:FS01:Started FahCore on PID 3224
12:41:15:WU01:FS01:Core PID:19092
12:41:15:WU01:FS01:FahCore 0x22 started
12:41:15:WARNING:WU00:FS01:FahCore returned an unknown error code which probably indicates that it crashed
12:41:15:WARNING:WU00:FS01:FahCore returned: UNKNOWN_ENUM (-1073740940 = 0xc0000374)

[_r2w_ben]

Exception access violation appears to happen at 5% intervals, which would be around when a checkpoint is written. Did you try to view any of the files in the work folders using another application?
Perhaps the core didn't close a file properly when the 25% checkpoint was written so subsequent checkpoints encountered an error. It's interesting that the exception was printed twice at 90% and that time it restarted the core.

[Swedis]

Did you try to view any of the files in the work folders using another application?

No i did not, this computer stays in the basement so i hardly never uses it. I use FAHcontrol to remotely control the folding from another computer upstairs and all other stats from that rig aswell. I had this same event with another WU from the same project 2 weeks ago but i noticed it too late and for that reason never pulled the log from it.

[HendricksSA]

Swedis, just curious. Did that work unit restart from its checkpoint and did it finish successfully? Could you post that part of the log?

[Swedis]

Swedis, just curious. Did that work unit restart from its checkpoint and did it finish successfully? Could you post that part of the log?

It did restart from checkpoint and finished but not successfully, i restarted FAHcontrol to get the first 200 lines of log so i can´t print the log for you.

https://apps.foldingathome.org/wu#proje ... =187&gen=1

[Joe_H]

The previous log should still be there, the client keeps the previous 16 by default.

[HendricksSA]

My quick look on the web shows error 0xc0000005 is associated with file corruption (among many other things). _r2w_ben may be on the right trail with the idea that a checkpoint got corrupted and subsequent attempts failed. I do not know if the client will continue processing with a corrupt checkpoint ... your log would indicate it does. I guess after a number of failed attempts the client saw the need to restart the core and reused the corrupt checkpoint. Previous question was to see at what point the core restarted from. I know how you feel having processed the work unit in vain.

[Swedis]

Swedis, just curious. Did that work unit restart from its checkpoint and did it finish successfully? Could you post that part of the log?

Sorry i mixed two different WU:s with same project together, it did not complete, see log below, it is the second try to start the WU.

The previous log should still be there, the client keeps the previous 16 by default.

True!

Code: Select all

21:23:29:WU00:FS01:Starting
21:23:29:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Swedis\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 14784 -checkpoint 5 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
21:23:29:WU00:FS01:Started FahCore on PID 18452
21:23:29:WU00:FS01:Core PID:5024
21:23:29:WU00:FS01:FahCore 0x22 started
21:23:30:WARNING:WU00:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
21:23:30:WU00:FS01:Sending unit results: id:00 state:SEND error:FAULTY project:13416 run:1110 clone:187 gen:1 core:0x22 unit:0x0000000212bc7d9a5f02af7c6ff31c57
21:23:30:WU00:FS01:Uploading 4.50KiB to 18.188.125.154
21:23:30:WU00:FS01:Connecting to 18.188.125.154:8080
21:23:30:WU00:FS01:Upload complete

[HendricksSA]

That answers my question. I am surprised it took nearly nine hours to restart the core for the second time. Yeah, what got sent was probably a 4kb report of the failure that would prompt a reissue. Hopefully this is a one-off event for you.

[Swedis]

I am surprised it took nearly nine hours to restart the core for the second time. Hopefully this is a one-off event for you.

The reason for that is FAH downloaded and crunched another WU in between.. Too bad it was not, got another one yesterday, see log. But this one got violation first at 85%

Code: Select all

11:53:06:WU02:FS01:0x22:*********************** Log Started 2020-07-12T11:53:06Z ***********************
11:53:06:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
11:53:06:WU02:FS01:0x22:       Core: Core22
11:53:06:WU02:FS01:0x22:       Type: 0x22
11:53:06:WU02:FS01:0x22:    Version: 0.0.11
11:53:06:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:53:06:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
11:53:06:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
11:53:06:WU02:FS01:0x22:       Date: Jun 26 2020
11:53:06:WU02:FS01:0x22:       Time: 19:49:16
11:53:06:WU02:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
11:53:06:WU02:FS01:0x22:     Branch: core22-0.0.11
11:53:06:WU02:FS01:0x22:   Compiler: Visual C++ 2015
11:53:06:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:53:06:WU02:FS01:0x22:   Platform: win32 10
11:53:06:WU02:FS01:0x22:       Bits: 64
11:53:06:WU02:FS01:0x22:       Mode: Release
11:53:06:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
11:53:06:WU02:FS01:0x22:             <peastman@stanford.edu>
11:53:06:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 16184 -checkpoint 5
11:53:06:WU02:FS01:0x22:             -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
11:53:06:WU02:FS01:0x22:************************************ libFAH ************************************
11:53:06:WU02:FS01:0x22:       Date: Jun 26 2020
11:53:06:WU02:FS01:0x22:       Time: 19:47:12
11:53:06:WU02:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
11:53:06:WU02:FS01:0x22:     Branch: HEAD
11:53:06:WU02:FS01:0x22:   Compiler: Visual C++ 2015
11:53:06:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:53:06:WU02:FS01:0x22:   Platform: win32 10
11:53:06:WU02:FS01:0x22:       Bits: 64
11:53:06:WU02:FS01:0x22:       Mode: Release
11:53:06:WU02:FS01:0x22:************************************ CBang *************************************
11:53:06:WU02:FS01:0x22:       Date: Jun 26 2020
11:53:06:WU02:FS01:0x22:       Time: 19:46:11
11:53:06:WU02:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
11:53:06:WU02:FS01:0x22:     Branch: master
11:53:06:WU02:FS01:0x22:   Compiler: Visual C++ 2015
11:53:06:WU02:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:53:07:WU02:FS01:0x22:   Platform: win32 10
11:53:07:WU02:FS01:0x22:       Bits: 64
11:53:07:WU02:FS01:0x22:       Mode: Release
11:53:07:WU02:FS01:0x22:************************************ System ************************************
11:53:07:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz
11:53:07:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 62 Stepping 4
11:53:07:WU02:FS01:0x22:       CPUs: 8
11:53:07:WU02:FS01:0x22:     Memory: 15.94GiB
11:53:07:WU02:FS01:0x22:Free Memory: 10.69GiB
11:53:07:WU02:FS01:0x22:    Threads: WINDOWS_THREADS
11:53:07:WU02:FS01:0x22: OS Version: 6.2
11:53:07:WU02:FS01:0x22:Has Battery: false
11:53:07:WU02:FS01:0x22: On Battery: false
11:53:07:WU02:FS01:0x22: UTC Offset: 2
11:53:07:WU02:FS01:0x22:        PID: 15000
11:53:07:WU02:FS01:0x22:        CWD: C:\Users\Swedis\AppData\Roaming\FAHClient\work
11:53:07:WU02:FS01:0x22:********************************************************************************
11:53:07:WU02:FS01:0x22:Project: 13416 (Run 1072, Clone 57, Gen 2)
11:53:07:WU02:FS01:0x22:Unit: 0x0000000512bc7d9a5f02af7fc8cd31af
11:53:07:WU02:FS01:0x22:Reading tar file core.xml
11:53:07:WU02:FS01:0x22:Reading tar file integrator.xml
11:53:07:WU02:FS01:0x22:Reading tar file state.xml.bz2
11:53:07:WU02:FS01:0x22:Reading tar file system.xml.bz2
11:53:07:WU02:FS01:0x22:Digital signatures verified
11:53:07:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
11:53:07:WU02:FS01:0x22:Version 0.0.11
11:53:07:WU02:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
11:53:07:WU02:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
11:53:07:WU02:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
11:53:07:WU02:FS01:0x22:  Global context and integrator variables write interval: 2500 steps (0.25%) [400 total]
11:53:30:WU02:FS01:0x22:Completed 0 out of 1000000 steps (0%)
11:59:15:WU02:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
12:04:59:WU02:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
12:10:40:WU02:FS01:0x22:Completed 30000 out of 1000000 steps (3%)
12:16:21:WU02:FS01:0x22:Completed 40000 out of 1000000 steps (4%)
12:22:03:WU02:FS01:0x22:Completed 50000 out of 1000000 steps (5%)
.
.
19:03:03:WU02:FS01:0x22:Completed 750000 out of 1000000 steps (75%)
19:08:48:WU02:FS01:0x22:Completed 760000 out of 1000000 steps (76%)
19:14:31:WU02:FS01:0x22:Completed 770000 out of 1000000 steps (77%)
19:20:15:WU02:FS01:0x22:Completed 780000 out of 1000000 steps (78%)
19:25:58:WU02:FS01:0x22:Completed 790000 out of 1000000 steps (79%)
19:31:42:WU02:FS01:0x22:Completed 800000 out of 1000000 steps (80%)
19:37:32:WU02:FS01:0x22:Completed 810000 out of 1000000 steps (81%)
19:43:21:WU02:FS01:0x22:Completed 820000 out of 1000000 steps (82%)
19:49:11:WU02:FS01:0x22:Completed 830000 out of 1000000 steps (83%)
19:54:59:WU02:FS01:0x22:Completed 840000 out of 1000000 steps (84%)
20:00:46:WU02:FS01:0x22:Completed 850000 out of 1000000 steps (85%)
20:00:47:WU02:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
20:06:28:WU02:FS01:0x22:Completed 860000 out of 1000000 steps (86%)
20:12:09:WU02:FS01:0x22:Completed 870000 out of 1000000 steps (87%)
20:17:50:WU02:FS01:0x22:Completed 880000 out of 1000000 steps (88%)
20:23:31:WU02:FS01:0x22:Completed 890000 out of 1000000 steps (89%)
20:29:11:WU02:FS01:0x22:Completed 900000 out of 1000000 steps (90%)
20:29:12:WU02:FS01:0x22:WARNING:Win32: 0xc0000005: Exception access violation
20:29:12:WU02:FS01:0x22:An exception occurred at step 900000: Win32: 0xc0000005: Exception access violation
20:29:12:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
20:29:12:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
.
.
.
05:08:36:WU02:FS01:Starting
05:08:36:WU02:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Swedis\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/v7/win/64bit/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 14784 -checkpoint 5 -gpu-vendor amd -opencl-platform 0 -opencl-device 0 -gpu 0
05:08:36:WU02:FS01:Started FahCore on PID 8792
05:08:36:WU02:FS01:Core PID:2980
05:08:36:WU02:FS01:FahCore 0x22 started
05:08:37:WARNING:WU02:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
05:08:37:WU02:FS01:Sending unit results: id:02 state:SEND error:FAULTY project:13416 run:1072 clone:57 gen:2 core:0x22 unit:0x0000000512bc7d9a5f02af7fc8cd31af
05:08:37:WU02:FS01:Uploading 4.50KiB to 18.188.125.154
05:08:37:WU02:FS01:Connecting to 18.188.125.154:8080
05:08:37:WU02:FS01:Upload complete

[HendricksSA]

Hopefully not a continuing trend. If so, perhaps stopping for a while and running memory tests and check disk/surface scans would tell you something more. Good luck!