Problems with OpenMM Core 23

[ben.ketteridge]

I'm seeing a lot of 'bad WUs' with Open MM Core 23:

Running FahCore: C:\ProgramData\FAHClient\cores/openmm-core-23/windows-10-64bit/release/fahcore-23-windows-10-64bit-release-8.0.3/FahCore_23.exe -dir KMRy6wOZAaA3Sb2LlhG3sleOEzxKtyWLwVH3vQugIFQ -suffix 01 -version 8.3.5 -lifeline 19800 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-platform 0 -cuda-device 0 -gpu 0
10:27:14:I3:WU1179:Started FahCore on PID 24200
10:27:15:W :WU1179:Core returned UNKNOWN_ENUM (3221225478)
10:27:15:W :WU1179:Core exited with Windows unhandled exception code 0xc0000006. See https://bit.ly/2CXgWkZ for more information.

Interspersed with core 23 failures, I'm still seeing core 22 WUs completeing correctly.

I'm running the 8.3.5 beta - is this the reason for the failure? (Windows 11)

[toTOW]

I have no idea what would cause this particular error and v8 client isn't helping ...

We need more details about your system. Posting FAH's log:
How to provide enough info to get helpful support.

[ben.ketteridge]

It's clear from the logging that open core 22 is fine, 23 is the problem.

I've attached a screen shot of the config of my machine.


Here's an example of a WU that works:

Code: Select all

01:43:21:I1:WU1261:Requesting WU assignment for user Ben_Ketteridge team 223518
01:43:21:I3:Connecting to ds01.scs.illinois.edu:443
01:43:21:I3:Connecting to assign3.foldingathome.org:443
01:43:21:I1:OUT353:> POST https://ds01.scs.illinois.edu/api/results HTTP/1.1
01:43:21:I1:OUT354:> POST https://assign3.foldingathome.org/api/assign HTTP/1.1
01:43:21:I1:OUT354:< HTTP/1.1 200 HTTP_OK
01:43:21:I1:WU1261:Received WU assignment Vbk-jGmlxGaJFSyaLp9iXM9J7nIbxbsMAL_O5Pjw_aQ
01:43:21:I1:WU1261:Downloading WU
01:43:21:I3:Connecting to ds01.scs.illinois.edu:443
01:43:22:I1:OUT355:> POST https://ds01.scs.illinois.edu/api/assign HTTP/1.1
01:43:34:I1:OUT355:< HTTP/1.1 200 HTTP_OK
01:43:34:I1:WU1261:Received WU
01:43:34:I3:WU1261:Running FahCore: C:\ProgramData\FAHClient\cores/openmm-core-22/fahcore-22-windows-64bit-release-0.0.20/FahCore_22.exe -dir Vbk-jGmlxGaJFSyaLp9iXM9J7nIbxbsMAL_O5Pjw_aQ -suffix 01 -version 8.3.5 -lifeline 19800 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-platform 0 -cuda-device 0 -gpu 0
01:43:34:I3:WU1261:Started FahCore on PID 1864
01:43:35:I1:WU1261:*********************** Log Started 2024-04-09T01:43:34Z ***********************
01:43:35:I1:WU1261:*************************** Core22 Folding@home Core ***************************
01:43:35:I1:WU1261: Core: Core22
01:43:35:I1:WU1261: Type: 0x22
01:43:35:I1:WU1261: Version: 0.0.20
01:43:35:I1:WU1261: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
01:43:35:I1:WU1261: Copyright: 2020 foldingathome.org
01:43:35:I1:WU1261: Homepage: https://foldingathome.org/
01:43:35:I1:WU1261: Date: Jan 20 2022
01:43:35:I1:WU1261: Time: 01:15:36
01:43:35:I1:WU1261: Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
01:43:35:I1:WU1261: Branch: HEAD
01:43:35:I1:WU1261: Compiler: Visual C++
01:43:35:I1:WU1261: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
01:43:35:I1:WU1261: -DOPENMM_VERSION="\"7.7.0\""
01:43:35:I1:WU1261: Platform: win32 10
01:43:35:I1:WU1261: Bits: 64
01:43:35:I1:WU1261: Mode: Release
01:43:35:I1:WU1261:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
01:43:35:I1:WU1261: <peastman@stanford.edu>
01:43:35:I1:WU1261: Args: -dir Vbk-jGmlxGaJFSyaLp9iXM9J7nIbxbsMAL_O5Pjw_aQ -suffix 01
01:43:35:I1:WU1261: -version 8.3.5 -lifeline 19800 -gpu-vendor nvidia -opencl-platform
01:43:35:I1:WU1261: 0 -opencl-device 0 -cuda-platform 0 -cuda-device 0 -gpu 0
01:43:35:I1:WU1261:************************************ libFAH ************************************
01:43:35:I1:WU1261: Date: Jan 20 2022
01:43:35:I1:WU1261: Time: 01:14:17
01:43:35:I1:WU1261: Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
01:43:35:I1:WU1261: Branch: HEAD
01:43:35:I1:WU1261: Compiler: Visual C++
01:43:35:I1:WU1261: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
01:43:35:I1:WU1261: Platform: win32 10
01:43:35:I1:WU1261: Bits: 64
01:43:35:I1:WU1261: Mode: Release
01:43:35:I1:WU1261:************************************ CBang *************************************
01:43:35:I1:WU1261: Date: Jan 20 2022
01:43:35:I1:WU1261: Time: 01:13:20
01:43:35:I1:WU1261: Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
01:43:35:I1:WU1261: Branch: HEAD
01:43:35:I1:WU1261: Compiler: Visual C++
01:43:35:I1:WU1261: Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
01:43:35:I1:WU1261: Platform: win32 10
01:43:35:I1:WU1261: Bits: 64
01:43:35:I1:WU1261: Mode: Release
01:43:35:I1:WU1261:************************************ System ************************************
01:43:35:I1:WU1261: CPU: Intel(R) Core(TM) i7-9700F CPU @ 3.00GHz
01:43:35:I1:WU1261: CPU ID: GenuineIntel Family 6 Model 158 Stepping 13
01:43:35:I1:WU1261: CPUs: 8
01:43:35:I1:WU1261: Memory: 15.95GiB
01:43:35:I1:WU1261:Free Memory: 7.20GiB
01:43:35:I1:WU1261: Threads: WINDOWS_THREADS
01:43:35:I1:WU1261: OS Version: 6.2
01:43:35:I1:WU1261:Has Battery: false
01:43:35:I1:WU1261: On Battery: false
01:43:35:I1:WU1261: UTC Offset: 1
01:43:35:I1:WU1261: PID: 1864
01:43:35:I1:WU1261: CWD: C:\ProgramData\FAHClient\work
01:43:35:I1:WU1261:************************************ OpenMM ************************************
01:43:35:I1:WU1261: Version: 7.7.0
01:43:35:I1:WU1261:********************************************************************************
01:43:35:I1:WU1261:Project: 19234 (Run 12, Clone 6, Gen 17)
01:43:35:I1:WU1261:Reading tar file core.xml
01:43:35:I1:WU1261:Reading tar file integrator.xml
01:43:35:I1:WU1261:Reading tar file state.xml
01:43:35:I1:WU1261:Reading tar file system.xml
01:43:35:I1:WU1261:Digital signatures verified
01:43:35:I1:WU1261:Folding@home GPU Core22 Folding@home Core
01:43:35:I1:WU1261:Version 0.0.20
01:43:36:I1:WU1261: Checkpoint write interval: 62500 steps (5%) [20 total]
01:43:36:I1:WU1261: JSON viewer frame write interval: 12500 steps (1%) [100 total]
01:43:36:I1:WU1261: XTC frame write interval: 25000 steps (2%) [50 total]
01:43:36:I1:WU1261: Global context and integrator variables write interval: disabled
01:43:36:I1:WU1261:There are 4 platforms available.
01:43:36:I1:WU1261:Platform 0: Reference
01:43:36:I1:WU1261:Platform 1: CPU
01:43:36:I1:WU1261:Platform 2: OpenCL
01:43:36:I1:WU1261: opencl-device 0 specified
01:43:36:I1:WU1261:Platform 3: CUDA
01:43:36:I1:WU1261: cuda-device 0 specified
01:43:39:I1:OUT353:< HTTP/1.1 200 HTTP_OK
01:43:39:I1:WU1260:Credited
01:43:52:I1:WU1261:Attempting to create CUDA context:
01:43:52:I1:WU1261: Configuring platform CUDA
01:43:59:I1:WU1261: Using CUDA and gpu 0
01:43:59:I1:WU1261:Completed 0 out of 1250000 steps (0%)
01:44:01:I1:WU1261:Checkpoint completed at step 0
01:45:06:I1:WU1261:Completed 12500 out of 1250000 steps (1%)
01:46:13:I1:WU1261:Completed 25000 out of 1250000 steps (2%)
01:47:20:I1:WU1261:Completed 37500 out of 1250000 steps (3%)
01:48:27:I1:WU1261:Completed 50000 out of 1250000 steps (4%)
01:49:33:I1:WU1261:Completed 62500 out of 1250000 steps (5%)
01:49:35:I1:WU1261:Checkpoint completed at step 62500
01:50:42:I1:WU1261:Completed 75000 out of 1250000 steps (6%)
01:51:48:I1:WU1261:Completed 87500 out of 1250000 steps (7%)
01:52:55:I1:WU1261:Completed 100000 out of 1250000 steps (8%)
01:54:01:I1:WU1261:Completed 112500 out of 1250000 steps (9%)
01:55:07:I1:WU1261:Completed 125000 out of 1250000 steps (10%)
01:55:09:I1:WU1261:Checkpoint completed at step 125000
01:56:15:I1:WU1261:Completed 137500 out of 1250000 steps (11%)
01:57:22:I1:WU1261:Completed 150000 out of 1250000 steps (12%)
01:58:28:I1:WU1261:Completed 162500 out of 1250000 steps (13%)
01:59:34:I1:WU1261:Completed 175000 out of 1250000 steps (14%)
02:00:40:I1:WU1261:Completed 187500 out of 1250000 steps (15%)
02:00:42:I1:WU1261:Checkpoint completed at step 187500
02:01:48:I1:WU1261:Completed 200000 out of 1250000 steps (16%)
02:02:54:I1:WU1261:Completed 212500 out of 1250000 steps (17%)
02:04:00:I1:WU1261:Completed 225000 out of 1250000 steps (18%)
02:05:06:I1:WU1261:Completed 237500 out of 1250000 steps (19%)
02:06:13:I1:WU1261:Completed 250000 out of 1250000 steps (20%)
02:06:14:I1:WU1261:Checkpoint completed at step 250000
02:07:21:I1:WU1261:Completed 262500 out of 1250000 steps (21%)
02:08:27:I1:WU1261:Completed 275000 out of 1250000 steps (22%)
02:09:33:I1:WU1261:Completed 287500 out of 1250000 steps (23%)
02:10:39:I1:WU1261:Completed 300000 out of 1250000 steps (24%)
02:11:45:I1:WU1261:Completed 312500 out of 1250000 steps (25%)
02:11:47:I1:WU1261:Checkpoint completed at step 312500
02:12:53:I1:WU1261:Completed 325000 out of 1250000 steps (26%)
02:13:59:I1:WU1261:Completed 337500 out of 1250000 steps (27%)
02:15:05:I1:WU1261:Completed 350000 out of 1250000 steps (28%)
02:16:11:I1:WU1261:Completed 362500 out of 1250000 steps (29%)
02:17:17:I1:WU1261:Completed 375000 out of 1250000 steps (30%)
02:17:19:I1:WU1261:Checkpoint completed at step 375000
02:18:25:I1:WU1261:Completed 387500 out of 1250000 steps (31%)
02:19:31:I1:WU1261:Completed 400000 out of 1250000 steps (32%)
02:20:37:I1:WU1261:Completed 412500 out of 1250000 steps (33%)
02:21:43:I1:WU1261:Completed 425000 out of 1250000 steps (34%)
02:22:50:I1:WU1261:Completed 437500 out of 1250000 steps (35%)
02:22:51:I1:WU1261:Checkpoint completed at step 437500
02:23:58:I1:WU1261:Completed 450000 out of 1250000 steps (36%)
02:25:04:I1:WU1261:Completed 462500 out of 1250000 steps (37%)
02:26:10:I1:WU1261:Completed 475000 out of 1250000 steps (38%)
02:27:16:I1:WU1261:Completed 487500 out of 1250000 steps (39%)
02:28:22:I1:WU1261:Completed 500000 out of 1250000 steps (40%)
02:28:23:I1:WU1261:Checkpoint completed at step 500000
02:29:29:I1:WU1261:Completed 512500 out of 1250000 steps (41%)
02:30:36:I1:WU1261:Completed 525000 out of 1250000 steps (42%)
02:31:42:I1:WU1261:Completed 537500 out of 1250000 steps (43%)
02:32:48:I1:WU1261:Completed 550000 out of 1250000 steps (44%)
02:33:54:I1:WU1261:Completed 562500 out of 1250000 steps (45%)
02:33:56:I1:WU1261:Checkpoint completed at step 562500
02:35:02:I1:WU1261:Completed 575000 out of 1250000 steps (46%)
02:36:08:I1:WU1261:Completed 587500 out of 1250000 steps (47%)
02:37:14:I1:WU1261:Completed 600000 out of 1250000 steps (48%)
02:38:19:I1:WU1261:Completed 612500 out of 1250000 steps (49%)
02:39:26:I1:WU1261:Completed 625000 out of 1250000 steps (50%)
02:39:28:I1:WU1261:Checkpoint completed at step 625000
02:40:34:I1:WU1261:Completed 637500 out of 1250000 steps (51%)
02:41:40:I1:WU1261:Completed 650000 out of 1250000 steps (52%)
02:42:46:I1:WU1261:Completed 662500 out of 1250000 steps (53%)
02:43:52:I1:WU1261:Completed 675000 out of 1250000 steps (54%)
02:44:58:I1:WU1261:Completed 687500 out of 1250000 steps (55%)
02:45:00:I1:WU1261:Checkpoint completed at step 687500
02:46:06:I1:WU1261:Completed 700000 out of 1250000 steps (56%)
02:47:12:I1:WU1261:Completed 712500 out of 1250000 steps (57%)
02:48:17:I1:WU1261:Completed 725000 out of 1250000 steps (58%)
02:49:23:I1:WU1261:Completed 737500 out of 1250000 steps (59%)
02:50:29:I1:WU1261:Completed 750000 out of 1250000 steps (60%)
02:50:31:I1:WU1261:Checkpoint completed at step 750000
02:51:37:I1:WU1261:Completed 762500 out of 1250000 steps (61%)
02:52:43:I1:WU1261:Completed 775000 out of 1250000 steps (62%)
02:53:49:I1:WU1261:Completed 787500 out of 1250000 steps (63%)
02:54:55:I1:WU1261:Completed 800000 out of 1250000 steps (64%)
02:56:01:I1:WU1261:Completed 812500 out of 1250000 steps (65%)
02:56:03:I1:WU1261:Checkpoint completed at step 812500
02:57:09:I1:WU1261:Completed 825000 out of 1250000 steps (66%)
02:58:15:I1:WU1261:Completed 837500 out of 1250000 steps (67%)
02:59:21:I1:WU1261:Completed 850000 out of 1250000 steps (68%)
03:00:26:I1:WU1261:Completed 862500 out of 1250000 steps (69%)
03:01:32:I1:WU1261:Completed 875000 out of 1250000 steps (70%)
03:01:34:I1:WU1261:Checkpoint completed at step 875000
03:02:40:I1:WU1261:Completed 887500 out of 1250000 steps (71%)
03:03:46:I1:WU1261:Completed 900000 out of 1250000 steps (72%)
03:04:52:I1:WU1261:Completed 912500 out of 1250000 steps (73%)
03:05:58:I1:WU1261:Completed 925000 out of 1250000 steps (74%)
03:07:04:I1:WU1261:Completed 937500 out of 1250000 steps (75%)
03:07:06:I1:WU1261:Checkpoint completed at step 937500
03:08:12:I1:WU1261:Completed 950000 out of 1250000 steps (76%)
03:09:18:I1:WU1261:Completed 962500 out of 1250000 steps (77%)
03:10:24:I1:WU1261:Completed 975000 out of 1250000 steps (78%)
03:11:30:I1:WU1261:Completed 987500 out of 1250000 steps (79%)
03:12:36:I1:WU1261:Completed 1000000 out of 1250000 steps (80%)
03:12:38:I1:WU1261:Checkpoint completed at step 1000000
03:13:44:I1:WU1261:Completed 1012500 out of 1250000 steps (81%)
03:14:50:I1:WU1261:Completed 1025000 out of 1250000 steps (82%)
03:15:56:I1:WU1261:Completed 1037500 out of 1250000 steps (83%)
03:17:02:I1:WU1261:Completed 1050000 out of 1250000 steps (84%)
03:18:08:I1:WU1261:Completed 1062500 out of 1250000 steps (85%)
03:18:10:I1:WU1261:Checkpoint completed at step 1062500
03:19:16:I1:WU1261:Completed 1075000 out of 1250000 steps (86%)
03:20:22:I1:WU1261:Completed 1087500 out of 1250000 steps (87%)
03:21:28:I1:WU1261:Completed 1100000 out of 1250000 steps (88%)
03:22:34:I1:WU1261:Completed 1112500 out of 1250000 steps (89%)
03:23:40:I1:WU1261:Completed 1125000 out of 1250000 steps (90%)
03:23:42:I1:WU1261:Checkpoint completed at step 1125000
03:24:48:I1:WU1261:Completed 1137500 out of 1250000 steps (91%)
03:25:54:I1:WU1261:Completed 1150000 out of 1250000 steps (92%)
03:27:00:I1:WU1261:Completed 1162500 out of 1250000 steps (93%)
03:28:06:I1:WU1261:Completed 1175000 out of 1250000 steps (94%)
03:29:12:I1:WU1261:Completed 1187500 out of 1250000 steps (95%)
03:29:14:I1:WU1261:Checkpoint completed at step 1187500
03:30:20:I1:WU1261:Completed 1200000 out of 1250000 steps (96%)
03:31:25:I1:WU1261:Completed 1212500 out of 1250000 steps (97%)
03:32:31:I1:WU1261:Completed 1225000 out of 1250000 steps (98%)
03:33:37:I1:WU1261:Completed 1237500 out of 1250000 steps (99%)
03:34:43:I1:WU1261:Completed 1250000 out of 1250000 steps (100%)
03:34:43:I1:WU1261:Average performance: 65.6535 ns/day
03:34:45:I1:WU1261:Checkpoint completed at step 1250000
03:34:49:I1:WU1261:Saving result file ..\logfile_01.txt
03:34:49:I1:WU1261:Saving result file checkpointIntegrator.xml
03:34:49:I1:WU1261:Saving result file checkpointState.xml
03:34:51:I1:WU1261:Saving result file positions.xtc
03:34:52:I1:WU1261:Saving result file science.log
03:34:52:I1:WU1261:Folding@home Core Shutdown: FINISHED_UNIT
03:34:54:I1:WU1261:Core returned FINISHED_UNIT (100)
03:34:54:I1:Default:Added new work unit: cpus:0 gpus:gpu:01:00:00
03:34:54:I1:WU1261:Uploading WU results

and one that does not:

Code: Select all

14:45:57:I1:WU1292:Requesting WU assignment for user Ben_Ketteridge team 223518
14:45:57:I3:Connecting to ds01.scs.illinois.edu:443
14:45:57:I3:Connecting to assign4.foldingathome.org:443
14:45:58:I1:OUT454:> POST https://ds01.scs.illinois.edu/api/results HTTP/1.1
14:45:58:I1:OUT455:> POST https://assign4.foldingathome.org/api/assign HTTP/1.1
14:45:58:I1:OUT455:< HTTP/1.1 200 HTTP_OK
14:45:58:I1:WU1292:Received WU assignment cww160_MSuVaicSSEKaIYlyHOh6F-CsdOLEds-maLk8
14:45:58:I1:WU1292:Downloading WU
14:45:58:I3:Connecting to highland4.seas.upenn.edu:443
14:45:58:I1:OUT456:> POST https://highland4.seas.upenn.edu/api/assign HTTP/1.1
14:46:06:I1:OUT456:< HTTP/1.1 200 HTTP_OK
14:46:07:I1:WU1292:Received WU
14:46:07:I3:WU1292:Running FahCore: C:\ProgramData\FAHClient\cores/openmm-core-23/windows-10-64bit/release/fahcore-23-windows-10-64bit-release-8.0.3/FahCore_23.exe -dir cww160_MSuVaicSSEKaIYlyHOh6F-CsdOLEds-maLk8 -suffix 01 -version 8.3.5 -lifeline 19800 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-platform 0 -cuda-device 0 -gpu 0
14:46:07:E :WU1292:Exception: Failed to create process with: C:\ProgramData\FAHClient\cores/openmm-core-23/windows-10-64bit/release/fahcore-23-windows-10-64bit-release-8.0.3/FahCore_23.exe -dir cww160_MSuVaicSSEKaIYlyHOh6F-CsdOLEds-maLk8 -suffix 01 -version 8.3.5 -lifeline 19800 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-platform 0 -cuda-device 0 -gpu 0: The system cannot find the file specified.
14:46:07:I1:WU1292:Retry #1 in 2 secs
14:46:07:W :WU1292:Core returned FAILED_1 (0)
14:46:07:I1:Default:Added new work unit: cpus:0 gpus:gpu:01:00:00
14:46:07:I1:WU1293:Requesting WU assignment for user Ben_Ketteridge team 223518
14:46:07:I3:Connecting to assign5.foldingathome.org:443
14:46:07:I1:OUT457:> POST https://assign5.foldingathome.org/api/assign HTTP/1.1
14:46:08:I1:OUT457:< HTTP/1.1 200 HTTP_OK
14:46:08:I1:WU1293:Received WU assignment pN3EZm2A-UgQoVnObbj_rrJvtVKINdF1fg1IvpZk7eg
14:46:08:I1:WU1293:Downloading WU
14:46:08:I3:Connecting to highland4.seas.upenn.edu:443
14:46:08:I1:OUT458:> POST https://highland4.seas.upenn.edu/api/assign HTTP/1.1
14:46:09:I1:WU1292:Sending dump report
14:46:09:I3:Connecting to highland4.seas.upenn.edu:443
14:46:09:I1:OUT459:> POST https://highland4.seas.upenn.edu/api/results HTTP/1.1
14:46:09:I1:OUT459:< HTTP/1.1 200 HTTP_OK
14:46:09:I1:WU1292:Dumped

[toTOW]

Do not blame a perfectly working core when using a (early) beta client. The only fix I can suggest here is to use v7 client.

You can try to post here about your v8 client issues : viewforum.php?f=115 but since the developer doesn't read the forums, it might be better to post here : https://github.com/FoldingAtHome/fah-cl ... tet/issues