Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Moderators: Site Moderators, FAHC Science Team

Post Reply
parkut
Posts: 363
Joined: Tue Feb 12, 2008 7:33 am
Hardware configuration: Running exclusively Linux headless blades. All are dedicated crunching machines.
Location: SE Michigan, USA

Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by parkut »

Same issue as reported on 18201 here: viewtopic.php?f=19&t=37275

This WU also processed normally, but on completion, the upload to 128.252.203.11 failed to send, retried 128.252.203.2, got a WORK_QUIT (404) and the work unit was dumped

Code: Select all

17:49:07:WU02:FS01:Connecting to assign1.foldingathome.org:80
17:49:07:WU02:FS01:Assigned to work server 128.252.203.11
17:49:07:WU02:FS01:Requesting new work unit for slot 01: gpu:1:0 TU116 [GeForce GTX 1660 Ti] from 128.252.203.11
17:49:07:WU02:FS01:Connecting to 128.252.203.11:8080
17:49:37:ERROR:WU02:FS01:Exception: 10002: Received short response, expected 512 bytes, got 0
17:49:38:WU02:FS01:Connecting to assign1.foldingathome.org:80
17:49:38:WU02:FS01:Assigned to work server 128.252.203.11
17:49:38:WU02:FS01:Requesting new work unit for slot 01: gpu:1:0 TU116 [GeForce GTX 1660 Ti] from 128.252.203.11
17:49:38:WU02:FS01:Connecting to 128.252.203.11:8080
17:49:53:WU02:FS01:Downloading 26.59MiB
17:49:59:WU02:FS01:Download 63.47%
17:50:07:WU02:FS01:Download 100.00%
17:50:07:WU02:FS01:Download complete
17:50:07:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:18202 run:1345 clone:3 gen:9 core:0x22 unit:0x00000003000000090000471a00000541
17:51:09:WU02:FS01:Starting
17:51:09:WU02:FS01:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 1286 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
17:51:09:WU02:FS01:Started FahCore on PID 24710
17:51:09:WU02:FS01:Core PID:24714
17:51:09:WU02:FS01:FahCore 0x22 started
17:51:09:WU02:FS01:0x22:*********************** Log Started 2021-06-23T17:51:09Z ***********************
17:51:09:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
17:51:09:WU02:FS01:0x22:       Core: Core22
17:51:09:WU02:FS01:0x22:       Type: 0x22
17:51:09:WU02:FS01:0x22:    Version: 0.0.13
17:51:09:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
17:51:09:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
17:51:09:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
17:51:09:WU02:FS01:0x22:       Date: Sep 19 2020
17:51:09:WU02:FS01:0x22:       Time: 01:10:35
17:51:09:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
17:51:09:WU02:FS01:0x22:     Branch: core22-0.0.13
17:51:09:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
17:51:09:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
17:51:09:WU02:FS01:0x22:             -funroll-loops -DOPENMM_GIT_HASH="\"189320d0\""
17:51:09:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
17:51:09:WU02:FS01:0x22:       Bits: 64
17:51:09:WU02:FS01:0x22:       Mode: Release
17:51:09:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
17:51:09:WU02:FS01:0x22:             <peastman@stanford.edu>
17:51:09:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 24710 -checkpoint 15
17:51:09:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
17:51:09:WU02:FS01:0x22:             nvidia -gpu 0 -gpu-usage 100
17:51:09:WU02:FS01:0x22:************************************ libFAH ************************************
17:51:09:WU02:FS01:0x22:       Date: Sep 15 2020
17:51:09:WU02:FS01:0x22:       Time: 05:14:43
17:51:09:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
17:51:09:WU02:FS01:0x22:     Branch: HEAD
17:51:09:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
17:51:09:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
17:51:09:WU02:FS01:0x22:             -funroll-loops
17:51:09:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
17:51:09:WU02:FS01:0x22:       Bits: 64
17:51:09:WU02:FS01:0x22:       Mode: Release
17:51:09:WU02:FS01:0x22:************************************ CBang *************************************
17:51:09:WU02:FS01:0x22:       Date: Sep 15 2020
17:51:09:WU02:FS01:0x22:       Time: 05:11:04
17:51:09:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
17:51:09:WU02:FS01:0x22:     Branch: HEAD
17:51:09:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
17:51:09:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
17:51:09:WU02:FS01:0x22:             -funroll-loops -fPIC
17:51:09:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
17:51:09:WU02:FS01:0x22:       Bits: 64
17:51:09:WU02:FS01:0x22:       Mode: Release
17:51:09:WU02:FS01:0x22:************************************ System ************************************
17:51:09:WU02:FS01:0x22:        CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
17:51:09:WU02:FS01:0x22:     CPU ID: GenuineIntel Family 6 Model 58 Stepping 9
17:51:09:WU02:FS01:0x22:       CPUs: 8
17:51:09:WU02:FS01:0x22:     Memory: 3.56GiB
17:51:09:WU02:FS01:0x22:Free Memory: 1.22GiB
17:51:09:WU02:FS01:0x22:    Threads: POSIX_THREADS
17:51:09:WU02:FS01:0x22: OS Version: 4.15
17:51:09:WU02:FS01:0x22:Has Battery: false
17:51:09:WU02:FS01:0x22: On Battery: false
17:51:09:WU02:FS01:0x22: UTC Offset: -4
17:51:09:WU02:FS01:0x22:        PID: 24714
17:51:09:WU02:FS01:0x22:        CWD: /var/lib/fahclient/work
17:51:09:WU02:FS01:0x22:************************************ OpenMM ************************************
17:51:09:WU02:FS01:0x22:   Revision: 189320d0
17:51:09:WU02:FS01:0x22:********************************************************************************
17:51:09:WU02:FS01:0x22:Project: 18202 (Run 1345, Clone 3, Gen 9)
17:51:09:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
17:51:09:WU02:FS01:0x22:Reading tar file core.xml
17:51:09:WU02:FS01:0x22:Reading tar file integrator.xml
17:51:09:WU02:FS01:0x22:Reading tar file state.xml
17:51:10:WU02:FS01:0x22:Reading tar file system.xml
17:51:11:WU02:FS01:0x22:Digital signatures verified
17:51:11:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
17:51:11:WU02:FS01:0x22:Version 0.0.13
17:51:11:WU02:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
17:51:11:WU02:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
17:51:11:WU02:FS01:0x22:  XTC frame write interval: 20000 steps (1.6%) [62 total]
17:51:11:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
17:51:11:WU02:FS01:0x22:There are 4 platforms available.
17:51:11:WU02:FS01:0x22:Platform 0: Reference
17:51:11:WU02:FS01:0x22:Platform 1: CPU
17:51:11:WU02:FS01:0x22:Platform 2: OpenCL
17:51:11:WU02:FS01:0x22:  opencl-device 0 specified
17:51:11:WU02:FS01:0x22:Platform 3: CUDA
17:51:11:WU02:FS01:0x22:  cuda-device 0 specified
17:51:25:WU02:FS01:0x22:Attempting to create CUDA context:
17:51:25:WU02:FS01:0x22:  Configuring platform CUDA
17:51:40:WU02:FS01:0x22:  Using CUDA and gpu 0
17:51:40:WU02:FS01:0x22:Completed 0 out of 1250000 steps (0%)
17:51:41:WU02:FS01:0x22:Checkpoint completed at step 0
...omitted normal processing steps...
22:55:04:WU02:FS01:0x22:Completed 1250000 out of 1250000 steps (100%)
22:55:04:WU02:FS01:0x22:Average performance: 9.52066 ns/day
22:55:06:WU02:FS01:0x22:Checkpoint completed at step 1250000
22:55:13:WU02:FS01:0x22:Saving result file ../logfile_01.txt
22:55:13:WU02:FS01:0x22:Saving result file checkpointIntegrator.xml
22:55:13:WU02:FS01:0x22:Saving result file checkpointState.xml
22:55:18:WU02:FS01:0x22:Saving result file positions.xtc
22:55:18:WU02:FS01:0x22:Saving result file science.log
22:55:18:WU02:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
22:55:19:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:55:19:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18202 run:1345 clone:3 gen:9 core:0x22 unit:0x00000003000000090000471a00000541
22:55:19:WU02:FS01:Uploading 27.51MiB to 128.252.203.11
22:55:19:WU02:FS01:Connecting to 128.252.203.11:8080
22:55:52:WU02:FS01:Upload 0.91%
22:55:52:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
22:55:52:WU02:FS01:Trying to send results to collection server
22:55:52:WU02:FS01:Uploading 27.51MiB to 128.252.203.2
22:55:52:WU02:FS01:Connecting to 128.252.203.2:8080
22:55:58:WU02:FS01:Upload 0.91%
22:56:04:WU02:FS01:Upload 11.36%
22:56:10:WU02:FS01:Upload 23.86%
22:56:16:WU02:FS01:Upload 35.90%
22:56:22:WU02:FS01:Upload 47.72%
22:56:28:WU02:FS01:Upload 59.76%
22:56:34:WU02:FS01:Upload 72.03%
22:56:40:WU02:FS01:Upload 84.07%
22:56:46:WU02:FS01:Upload 96.12%
22:56:48:WU02:FS01:Upload complete
22:56:48:WU02:FS01:Server responded WORK_QUIT (404)
22:56:48:WARNING:WU02:FS01:Server did not like results, dumping
22:56:48:WU02:FS01:Cleaning up
Top
jjmiller
Scientist
Posts: 136
Joined: Fri Apr 09, 2021 4:43 pm

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by jjmiller »

Hi parkut, Noted and thanks for the report. This is the same server as the 18201 project- I'll see if I can get to the bottom of these.
Top
mgetz
Posts: 57
Joined: Tue Aug 11, 2020 6:23 pm

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by mgetz »

And another 18202 dumped

Code: Select all

14:04:22:WU02:FS01:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:18202 run:9152 clone:0 gen:4 core:0x22 unit:0x00000000000000040000471a000023c0
14:04:22:WU02:FS01:Starting
14:04:22:WU02:FS01:Running FahCore: /snap/folding-at-home-fcole90/168/usr/bin/FAHCoreWrapper /home/<redacted>/snap/folding-at-home-fcole90/common/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 2235 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
14:04:22:WU02:FS01:Started FahCore on PID 1233704
14:04:22:WU02:FS01:Core PID:1233708
14:04:22:WU02:FS01:FahCore 0x22 started
14:04:22:WU02:FS01:0x22:*********************** Log Started 2021-06-24T14:04:22Z ***********************
14:04:22:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
14:04:22:WU02:FS01:0x22:       Core: Core22
14:04:22:WU02:FS01:0x22:       Type: 0x22
14:04:22:WU02:FS01:0x22:    Version: 0.0.13
14:04:22:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
14:04:22:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
14:04:22:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
14:04:22:WU02:FS01:0x22:       Date: Sep 19 2020
14:04:22:WU02:FS01:0x22:       Time: 01:10:35
14:04:22:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
14:04:22:WU02:FS01:0x22:     Branch: core22-0.0.13
14:04:22:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
14:04:22:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
14:04:22:WU02:FS01:0x22:             -funroll-loops -DOPENMM_GIT_HASH="\"189320d0\""
14:04:22:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
14:04:22:WU02:FS01:0x22:       Bits: 64
14:04:22:WU02:FS01:0x22:       Mode: Release
14:04:22:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
14:04:22:WU02:FS01:0x22:             <peastman@stanford.edu>
14:04:22:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 1233704 -checkpoint 15
14:04:22:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
14:04:22:WU02:FS01:0x22:             nvidia -gpu 0 -gpu-usage 100
14:04:22:WU02:FS01:0x22:************************************ libFAH ************************************
14:04:22:WU02:FS01:0x22:       Date: Sep 15 2020
14:04:22:WU02:FS01:0x22:       Time: 05:14:43
14:04:22:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
14:04:22:WU02:FS01:0x22:     Branch: HEAD
14:04:22:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
14:04:22:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
14:04:22:WU02:FS01:0x22:             -funroll-loops
14:04:22:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
14:04:22:WU02:FS01:0x22:       Bits: 64
14:04:22:WU02:FS01:0x22:       Mode: Release
14:04:22:WU02:FS01:0x22:************************************ CBang *************************************
14:04:22:WU02:FS01:0x22:       Date: Sep 15 2020
14:04:22:WU02:FS01:0x22:       Time: 05:11:04
14:04:22:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
14:04:22:WU02:FS01:0x22:     Branch: HEAD
14:04:22:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
14:04:22:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
14:04:22:WU02:FS01:0x22:             -funroll-loops -fPIC
14:04:22:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
14:04:22:WU02:FS01:0x22:       Bits: 64
14:04:22:WU02:FS01:0x22:       Mode: Release
14:04:22:WU02:FS01:0x22:************************************ System ************************************
14:04:22:WU02:FS01:0x22:        CPU: AMD Ryzen 7 3800X 8-Core Processor
14:04:22:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
14:04:22:WU02:FS01:0x22:       CPUs: 8
14:04:22:WU02:FS01:0x22:     Memory: 15.60GiB
14:04:22:WU02:FS01:0x22:Free Memory: 13.01GiB
14:04:22:WU02:FS01:0x22:    Threads: POSIX_THREADS
14:04:22:WU02:FS01:0x22: OS Version: 5.11
14:04:22:WU02:FS01:0x22:Has Battery: false
14:04:22:WU02:FS01:0x22: On Battery: false
14:04:22:WU02:FS01:0x22: UTC Offset: -6
14:04:22:WU02:FS01:0x22:        PID: 1233708
14:04:22:WU02:FS01:0x22:        CWD: /home/<redacted>/snap/folding-at-home-fcole90/common/work
14:04:22:WU02:FS01:0x22:************************************ OpenMM ************************************
14:04:22:WU02:FS01:0x22:   Revision: 189320d0
14:04:22:WU02:FS01:0x22:********************************************************************************
14:04:22:WU02:FS01:0x22:Project: 18202 (Run 9152, Clone 0, Gen 4)
14:04:22:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
14:04:22:WU02:FS01:0x22:Reading tar file core.xml
14:04:22:WU02:FS01:0x22:Reading tar file integrator.xml
14:04:22:WU02:FS01:0x22:Reading tar file state.xml
14:04:23:WU02:FS01:0x22:Reading tar file system.xml
14:04:23:WU02:FS01:0x22:Digital signatures verified
14:04:23:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
14:04:23:WU02:FS01:0x22:Version 0.0.13
14:04:23:WU02:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
14:04:23:WU02:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
14:04:23:WU02:FS01:0x22:  XTC frame write interval: 20000 steps (1.6%) [62 total]
14:04:23:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
14:04:23:WU02:FS01:0x22:There are 4 platforms available.
14:04:23:WU02:FS01:0x22:Platform 0: Reference
14:04:23:WU02:FS01:0x22:Platform 1: CPU
14:04:23:WU02:FS01:0x22:Platform 2: OpenCL
14:04:23:WU02:FS01:0x22:  opencl-device 0 specified
14:04:23:WU02:FS01:0x22:Platform 3: CUDA
14:04:23:WU02:FS01:0x22:  cuda-device 0 specified
14:04:30:WU02:FS01:0x22:Attempting to create CUDA context:
14:04:30:WU02:FS01:0x22:  Configuring platform CUDA
14:04:37:WU02:FS01:0x22:  Using CUDA and gpu 0
14:04:37:WU02:FS01:0x22:Completed 0 out of 1250000 steps (0%)
14:04:37:WU02:FS01:0x22:Checkpoint completed at step 0
14:06:16:WU02:FS01:0x22:Completed 12500 out of 1250000 steps (1%)
14:07:56:WU02:FS01:0x22:Completed 25000 out of 1250000 steps (2%)
14:07:57:WU02:FS01:0x22:Checkpoint completed at step 25000
14:09:37:WU02:FS01:0x22:Completed 37500 out of 1250000 steps (3%)
14:11:16:WU02:FS01:0x22:Completed 50000 out of 1250000 steps (4%)
14:11:17:WU02:FS01:0x22:Checkpoint completed at step 50000
14:12:57:WU02:FS01:0x22:Completed 62500 out of 1250000 steps (5%)
14:14:37:WU02:FS01:0x22:Completed 75000 out of 1250000 steps (6%)
14:14:38:WU02:FS01:0x22:Checkpoint completed at step 75000
14:16:18:WU02:FS01:0x22:Completed 87500 out of 1250000 steps (7%)
14:17:58:WU02:FS01:0x22:Completed 100000 out of 1250000 steps (8%)
14:17:59:WU02:FS01:0x22:Checkpoint completed at step 100000
14:19:39:WU02:FS01:0x22:Completed 112500 out of 1250000 steps (9%)
14:21:18:WU02:FS01:0x22:Completed 125000 out of 1250000 steps (10%)
14:21:20:WU02:FS01:0x22:Checkpoint completed at step 125000
14:22:59:WU02:FS01:0x22:Completed 137500 out of 1250000 steps (11%)
14:24:39:WU02:FS01:0x22:Completed 150000 out of 1250000 steps (12%)
14:24:40:WU02:FS01:0x22:Checkpoint completed at step 150000
14:26:20:WU02:FS01:0x22:Completed 162500 out of 1250000 steps (13%)
14:28:00:WU02:FS01:0x22:Completed 175000 out of 1250000 steps (14%)
14:28:01:WU02:FS01:0x22:Checkpoint completed at step 175000
14:29:41:WU02:FS01:0x22:Completed 187500 out of 1250000 steps (15%)
14:31:21:WU02:FS01:0x22:Completed 200000 out of 1250000 steps (16%)
14:31:22:WU02:FS01:0x22:Checkpoint completed at step 200000
14:33:01:WU02:FS01:0x22:Completed 212500 out of 1250000 steps (17%)
14:34:41:WU02:FS01:0x22:Completed 225000 out of 1250000 steps (18%)
14:34:42:WU02:FS01:0x22:Checkpoint completed at step 225000
14:36:22:WU02:FS01:0x22:Completed 237500 out of 1250000 steps (19%)
14:38:02:WU02:FS01:0x22:Completed 250000 out of 1250000 steps (20%)
14:38:03:WU02:FS01:0x22:Checkpoint completed at step 250000
14:39:42:WU02:FS01:0x22:Completed 262500 out of 1250000 steps (21%)
14:41:22:WU02:FS01:0x22:Completed 275000 out of 1250000 steps (22%)
14:41:23:WU02:FS01:0x22:Checkpoint completed at step 275000
14:43:03:WU02:FS01:0x22:Completed 287500 out of 1250000 steps (23%)
14:44:43:WU02:FS01:0x22:Completed 300000 out of 1250000 steps (24%)
14:44:44:WU02:FS01:0x22:Checkpoint completed at step 300000
14:46:24:WU02:FS01:0x22:Completed 312500 out of 1250000 steps (25%)
14:48:03:WU02:FS01:0x22:Completed 325000 out of 1250000 steps (26%)
14:48:04:WU02:FS01:0x22:Checkpoint completed at step 325000
14:49:44:WU02:FS01:0x22:Completed 337500 out of 1250000 steps (27%)
14:51:24:WU02:FS01:0x22:Completed 350000 out of 1250000 steps (28%)
14:51:25:WU02:FS01:0x22:Checkpoint completed at step 350000
14:53:04:WU02:FS01:0x22:Completed 362500 out of 1250000 steps (29%)
14:54:44:WU02:FS01:0x22:Completed 375000 out of 1250000 steps (30%)
14:54:45:WU02:FS01:0x22:Checkpoint completed at step 375000
14:56:25:WU02:FS01:0x22:Completed 387500 out of 1250000 steps (31%)
14:58:05:WU02:FS01:0x22:Completed 400000 out of 1250000 steps (32%)
14:58:06:WU02:FS01:0x22:Checkpoint completed at step 400000
14:59:45:WU02:FS01:0x22:Completed 412500 out of 1250000 steps (33%)
15:01:25:WU02:FS01:0x22:Completed 425000 out of 1250000 steps (34%)
15:01:26:WU02:FS01:0x22:Checkpoint completed at step 425000
15:03:06:WU02:FS01:0x22:Completed 437500 out of 1250000 steps (35%)
15:04:46:WU02:FS01:0x22:Completed 450000 out of 1250000 steps (36%)
15:04:47:WU02:FS01:0x22:Checkpoint completed at step 450000
15:06:26:WU02:FS01:0x22:Completed 462500 out of 1250000 steps (37%)
15:08:06:WU02:FS01:0x22:Completed 475000 out of 1250000 steps (38%)
15:08:07:WU02:FS01:0x22:Checkpoint completed at step 475000
15:09:47:WU02:FS01:0x22:Completed 487500 out of 1250000 steps (39%)
15:11:27:WU02:FS01:0x22:Completed 500000 out of 1250000 steps (40%)
15:11:28:WU02:FS01:0x22:Checkpoint completed at step 500000
15:13:08:WU02:FS01:0x22:Completed 512500 out of 1250000 steps (41%)
15:14:48:WU02:FS01:0x22:Completed 525000 out of 1250000 steps (42%)
15:14:49:WU02:FS01:0x22:Checkpoint completed at step 525000
15:16:29:WU02:FS01:0x22:Completed 537500 out of 1250000 steps (43%)
15:18:08:WU02:FS01:0x22:Completed 550000 out of 1250000 steps (44%)
15:18:10:WU02:FS01:0x22:Checkpoint completed at step 550000
15:19:49:WU02:FS01:0x22:Completed 562500 out of 1250000 steps (45%)
15:21:29:WU02:FS01:0x22:Completed 575000 out of 1250000 steps (46%)
15:21:30:WU02:FS01:0x22:Checkpoint completed at step 575000
15:23:10:WU02:FS01:0x22:Completed 587500 out of 1250000 steps (47%)
15:24:50:WU02:FS01:0x22:Completed 600000 out of 1250000 steps (48%)
15:24:51:WU02:FS01:0x22:Checkpoint completed at step 600000
15:26:30:WU02:FS01:0x22:Completed 612500 out of 1250000 steps (49%)
15:28:10:WU02:FS01:0x22:Completed 625000 out of 1250000 steps (50%)
15:28:11:WU02:FS01:0x22:Checkpoint completed at step 625000
15:29:51:WU02:FS01:0x22:Completed 637500 out of 1250000 steps (51%)
15:31:30:WU02:FS01:0x22:Completed 650000 out of 1250000 steps (52%)
15:31:31:WU02:FS01:0x22:Checkpoint completed at step 650000
15:33:11:WU02:FS01:0x22:Completed 662500 out of 1250000 steps (53%)
15:34:51:WU02:FS01:0x22:Completed 675000 out of 1250000 steps (54%)
15:34:52:WU02:FS01:0x22:Checkpoint completed at step 675000
15:36:32:WU02:FS01:0x22:Completed 687500 out of 1250000 steps (55%)
15:38:12:WU02:FS01:0x22:Completed 700000 out of 1250000 steps (56%)
15:38:13:WU02:FS01:0x22:Checkpoint completed at step 700000
15:39:52:WU02:FS01:0x22:Completed 712500 out of 1250000 steps (57%)
15:41:32:WU02:FS01:0x22:Completed 725000 out of 1250000 steps (58%)
15:41:33:WU02:FS01:0x22:Checkpoint completed at step 725000
15:43:13:WU02:FS01:0x22:Completed 737500 out of 1250000 steps (59%)
15:44:53:WU02:FS01:0x22:Completed 750000 out of 1250000 steps (60%)
15:44:54:WU02:FS01:0x22:Checkpoint completed at step 750000
15:46:33:WU02:FS01:0x22:Completed 762500 out of 1250000 steps (61%)
15:48:13:WU02:FS01:0x22:Completed 775000 out of 1250000 steps (62%)
15:48:14:WU02:FS01:0x22:Checkpoint completed at step 775000
15:49:54:WU02:FS01:0x22:Completed 787500 out of 1250000 steps (63%)
15:51:34:WU02:FS01:0x22:Completed 800000 out of 1250000 steps (64%)
15:51:35:WU02:FS01:0x22:An exception occurred at step 800000: Force RMSE error of 5.28262 with threshold of 5
15:51:35:WU02:FS01:0x22:ERROR:98: Attempting to restart from last good checkpoint by restarting core.
15:51:35:WU02:FS01:0x22:Folding@home Core Shutdown: CORE_RESTART
15:51:35:WARNING:WU02:FS01:FahCore returned: CORE_RESTART (98 = 0x62)
15:51:35:WU02:FS01:Starting
15:51:35:WU02:FS01:Running FahCore: /snap/folding-at-home-fcole90/168/usr/bin/FAHCoreWrapper /home/<redacted>/snap/folding-at-home-fcole90/common/cores/cores.foldingathome.org/lin/64bit/22-0.0.13/Core_22.fah/FahCore_22 -dir 02 -suffix 01 -version 706 -lifeline 2235 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
15:51:35:WU02:FS01:Started FahCore on PID 1283985
15:51:35:WU02:FS01:Core PID:1283989
15:51:35:WU02:FS01:FahCore 0x22 started
15:51:36:WU02:FS01:0x22:*********************** Log Started 2021-06-24T15:51:35Z ***********************
15:51:36:WU02:FS01:0x22:*************************** Core22 Folding@home Core ***************************
15:51:36:WU02:FS01:0x22:       Core: Core22
15:51:36:WU02:FS01:0x22:       Type: 0x22
15:51:36:WU02:FS01:0x22:    Version: 0.0.13
15:51:36:WU02:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
15:51:36:WU02:FS01:0x22:  Copyright: 2020 foldingathome.org
15:51:36:WU02:FS01:0x22:   Homepage: https://foldingathome.org/
15:51:36:WU02:FS01:0x22:       Date: Sep 19 2020
15:51:36:WU02:FS01:0x22:       Time: 01:10:35
15:51:36:WU02:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
15:51:36:WU02:FS01:0x22:     Branch: core22-0.0.13
15:51:36:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
15:51:36:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
15:51:36:WU02:FS01:0x22:             -funroll-loops -DOPENMM_GIT_HASH="\"189320d0\""
15:51:36:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
15:51:36:WU02:FS01:0x22:       Bits: 64
15:51:36:WU02:FS01:0x22:       Mode: Release
15:51:36:WU02:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
15:51:36:WU02:FS01:0x22:             <peastman@stanford.edu>
15:51:36:WU02:FS01:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 1283985 -checkpoint 15
15:51:36:WU02:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
15:51:36:WU02:FS01:0x22:             nvidia -gpu 0 -gpu-usage 100
15:51:36:WU02:FS01:0x22:************************************ libFAH ************************************
15:51:36:WU02:FS01:0x22:       Date: Sep 15 2020
15:51:36:WU02:FS01:0x22:       Time: 05:14:43
15:51:36:WU02:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
15:51:36:WU02:FS01:0x22:     Branch: HEAD
15:51:36:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
15:51:36:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
15:51:36:WU02:FS01:0x22:             -funroll-loops
15:51:36:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
15:51:36:WU02:FS01:0x22:       Bits: 64
15:51:36:WU02:FS01:0x22:       Mode: Release
15:51:36:WU02:FS01:0x22:************************************ CBang *************************************
15:51:36:WU02:FS01:0x22:       Date: Sep 15 2020
15:51:36:WU02:FS01:0x22:       Time: 05:11:04
15:51:36:WU02:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
15:51:36:WU02:FS01:0x22:     Branch: HEAD
15:51:36:WU02:FS01:0x22:   Compiler: GNU 4.8.2 20140120 (Red Hat 4.8.2-15)
15:51:36:WU02:FS01:0x22:    Options: -std=c++11 -fsigned-char -ffunction-sections -fdata-sections -O3
15:51:36:WU02:FS01:0x22:             -funroll-loops -fPIC
15:51:36:WU02:FS01:0x22:   Platform: linux2 4.19.76-linuxkit
15:51:36:WU02:FS01:0x22:       Bits: 64
15:51:36:WU02:FS01:0x22:       Mode: Release
15:51:36:WU02:FS01:0x22:************************************ System ************************************
15:51:36:WU02:FS01:0x22:        CPU: AMD Ryzen 7 3800X 8-Core Processor
15:51:36:WU02:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
15:51:36:WU02:FS01:0x22:       CPUs: 8
15:51:36:WU02:FS01:0x22:     Memory: 15.60GiB
15:51:36:WU02:FS01:0x22:Free Memory: 12.81GiB
15:51:36:WU02:FS01:0x22:    Threads: POSIX_THREADS
15:51:36:WU02:FS01:0x22: OS Version: 5.11
15:51:36:WU02:FS01:0x22:Has Battery: false
15:51:36:WU02:FS01:0x22: On Battery: false
15:51:36:WU02:FS01:0x22: UTC Offset: -6
15:51:36:WU02:FS01:0x22:        PID: 1283989
15:51:36:WU02:FS01:0x22:        CWD: /home/<redacted>/snap/folding-at-home-fcole90/common/work
15:51:36:WU02:FS01:0x22:************************************ OpenMM ************************************
15:51:36:WU02:FS01:0x22:   Revision: 189320d0
15:51:36:WU02:FS01:0x22:********************************************************************************
15:51:36:WU02:FS01:0x22:Project: 18202 (Run 9152, Clone 0, Gen 4)
15:51:36:WU02:FS01:0x22:Unit: 0x00000000000000000000000000000000
15:51:36:WU02:FS01:0x22:Digital signatures verified
15:51:36:WU02:FS01:0x22:Folding@home GPU Core22 Folding@home Core
15:51:36:WU02:FS01:0x22:Version 0.0.13
15:51:36:WU02:FS01:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
15:51:36:WU02:FS01:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
15:51:36:WU02:FS01:0x22:  XTC frame write interval: 20000 steps (1.6%) [62 total]
15:51:36:WU02:FS01:0x22:  Global context and integrator variables write interval: disabled
15:51:36:WU02:FS01:0x22:There are 4 platforms available.
15:51:36:WU02:FS01:0x22:Platform 0: Reference
15:51:36:WU02:FS01:0x22:Platform 1: CPU
15:51:36:WU02:FS01:0x22:Platform 2: OpenCL
15:51:36:WU02:FS01:0x22:  opencl-device 0 specified
15:51:36:WU02:FS01:0x22:Platform 3: CUDA
15:51:36:WU02:FS01:0x22:  cuda-device 0 specified
15:51:43:WU02:FS01:0x22:Attempting to create CUDA context:
15:51:43:WU02:FS01:0x22:  Configuring platform CUDA
15:51:47:WU02:FS01:0x22:  Using CUDA and gpu 0
15:51:47:WU02:FS01:0x22:Completed 775000 out of 1250000 steps (62%)
15:53:27:WU02:FS01:0x22:Completed 787500 out of 1250000 steps (63%)
15:55:07:WU02:FS01:0x22:Completed 800000 out of 1250000 steps (64%)
15:55:08:WU02:FS01:0x22:Checkpoint completed at step 800000
15:56:48:WU02:FS01:0x22:Completed 812500 out of 1250000 steps (65%)
15:58:27:WU02:FS01:0x22:Completed 825000 out of 1250000 steps (66%)
15:58:29:WU02:FS01:0x22:Checkpoint completed at step 825000
16:00:08:WU02:FS01:0x22:Completed 837500 out of 1250000 steps (67%)
16:01:48:WU02:FS01:0x22:Completed 850000 out of 1250000 steps (68%)
16:01:49:WU02:FS01:0x22:Checkpoint completed at step 850000
16:03:29:WU02:FS01:0x22:Completed 862500 out of 1250000 steps (69%)
16:05:09:WU02:FS01:0x22:Completed 875000 out of 1250000 steps (70%)
16:05:10:WU02:FS01:0x22:Checkpoint completed at step 875000
16:06:50:WU02:FS01:0x22:Completed 887500 out of 1250000 steps (71%)
16:08:30:WU02:FS01:0x22:Completed 900000 out of 1250000 steps (72%)
16:08:31:WU02:FS01:0x22:Checkpoint completed at step 900000
16:10:11:WU02:FS01:0x22:Completed 912500 out of 1250000 steps (73%)
16:11:51:WU02:FS01:0x22:Completed 925000 out of 1250000 steps (74%)
16:11:52:WU02:FS01:0x22:Checkpoint completed at step 925000
16:13:31:WU02:FS01:0x22:Completed 937500 out of 1250000 steps (75%)
16:15:11:WU02:FS01:0x22:Completed 950000 out of 1250000 steps (76%)
16:15:12:WU02:FS01:0x22:Checkpoint completed at step 950000
16:16:52:WU02:FS01:0x22:Completed 962500 out of 1250000 steps (77%)
16:18:31:WU02:FS01:0x22:Completed 975000 out of 1250000 steps (78%)
16:18:33:WU02:FS01:0x22:Checkpoint completed at step 975000
16:20:12:WU02:FS01:0x22:Completed 987500 out of 1250000 steps (79%)
16:21:52:WU02:FS01:0x22:Completed 1000000 out of 1250000 steps (80%)
16:21:53:WU02:FS01:0x22:Checkpoint completed at step 1000000
16:23:33:WU02:FS01:0x22:Completed 1012500 out of 1250000 steps (81%)
16:25:12:WU02:FS01:0x22:Completed 1025000 out of 1250000 steps (82%)
16:25:13:WU02:FS01:0x22:Checkpoint completed at step 1025000
16:26:53:WU02:FS01:0x22:Completed 1037500 out of 1250000 steps (83%)
16:28:33:WU02:FS01:0x22:Completed 1050000 out of 1250000 steps (84%)
16:28:34:WU02:FS01:0x22:Checkpoint completed at step 1050000
16:30:14:WU02:FS01:0x22:Completed 1062500 out of 1250000 steps (85%)
16:31:53:WU02:FS01:0x22:Completed 1075000 out of 1250000 steps (86%)
16:31:54:WU02:FS01:0x22:Checkpoint completed at step 1075000
16:33:34:WU02:FS01:0x22:Completed 1087500 out of 1250000 steps (87%)
16:35:14:WU02:FS01:0x22:Completed 1100000 out of 1250000 steps (88%)
16:35:15:WU02:FS01:0x22:Checkpoint completed at step 1100000
16:36:55:WU02:FS01:0x22:Completed 1112500 out of 1250000 steps (89%)
16:38:34:WU02:FS01:0x22:Completed 1125000 out of 1250000 steps (90%)
16:38:35:WU02:FS01:0x22:Checkpoint completed at step 1125000
16:40:15:WU02:FS01:0x22:Completed 1137500 out of 1250000 steps (91%)
16:41:55:WU02:FS01:0x22:Completed 1150000 out of 1250000 steps (92%)
16:41:56:WU02:FS01:0x22:Checkpoint completed at step 1150000
16:43:35:WU02:FS01:0x22:Completed 1162500 out of 1250000 steps (93%)
16:45:15:WU02:FS01:0x22:Completed 1175000 out of 1250000 steps (94%)
16:45:16:WU02:FS01:0x22:Checkpoint completed at step 1175000
16:46:56:WU02:FS01:0x22:Completed 1187500 out of 1250000 steps (95%)
16:48:35:WU02:FS01:0x22:Completed 1200000 out of 1250000 steps (96%)
16:48:36:WU02:FS01:0x22:Checkpoint completed at step 1200000
16:50:16:WU02:FS01:0x22:Completed 1212500 out of 1250000 steps (97%)
16:51:56:WU02:FS01:0x22:Completed 1225000 out of 1250000 steps (98%)
16:51:57:WU02:FS01:0x22:Checkpoint completed at step 1225000
16:53:36:WU02:FS01:0x22:Completed 1237500 out of 1250000 steps (99%)
16:55:16:WU02:FS01:0x22:Completed 1250000 out of 1250000 steps (100%)
16:55:16:WU02:FS01:0x22:Average performance: 17.3668 ns/day
16:55:17:WU02:FS01:0x22:Checkpoint completed at step 1250000
16:55:21:WU02:FS01:0x22:Saving result file ../logfile_01.txt
16:55:21:WU02:FS01:0x22:Saving result file checkpointIntegrator.xml
16:55:21:WU02:FS01:0x22:Saving result file checkpointState.xml
16:55:24:WU02:FS01:0x22:Saving result file positions.xtc
16:55:24:WU02:FS01:0x22:Saving result file science.log
16:55:24:WU02:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
16:55:25:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
16:55:25:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18202 run:9152 clone:0 gen:4 core:0x22 unit:0x00000000000000040000471a000023c0
16:55:25:WU02:FS01:Uploading 27.51MiB to 128.252.203.11
16:55:25:WU02:FS01:Connecting to 128.252.203.11:8080
16:55:56:WARNING:WU02:FS01:Exception: Failed to send results to work server: Not connected
16:55:56:WU02:FS01:Trying to send results to collection server
16:55:56:WU02:FS01:Uploading 27.51MiB to 128.252.203.2
16:55:56:WU02:FS01:Connecting to 128.252.203.2:8080
16:56:02:WU02:FS01:Upload 41.35%
16:56:08:WU02:FS01:Upload 99.74%
16:56:08:WU02:FS01:Upload complete
16:56:08:WU02:FS01:Server responded WORK_QUIT (404)
16:56:08:WARNING:WU02:FS01:Server did not like results, dumping
16:56:08:WU02:FS01:Cleaning up
Did 18202 even go through Beta? It doesn't seem stable enough for full public.

I should note that as far as I can tell even the 18202 WUs that I did show as uploaded have not been credited. So as far as I'm concerned 18202 is a write off and is just wasting electricity.
Image
Top
jjmiller
Scientist
Posts: 136
Joined: Fri Apr 09, 2021 4:43 pm

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by jjmiller »

Hi mgetz,

Yes, 18202 went through internal and beta testing and performed well in both. One issue that it seems like we see (especially for bigger systems as we have in 18201/18202) is that differences in the GPU being used can severely change the stability of a project. Based on the current stats, it may be that we did not adequately capture enough GPU diversity during beta testing. Early on instabilities were very rare in this project and have only reached a higher % of late. As instabilities don't impact our end calculations, I had left the project running. We're reevaluating whether to keep running this project now or whether, whether to restrict the project to a narrower range of GPUs (thus limiting the diversity and allowing us to optimize stability for a smaller range), or whether tinker more with our initial system in order to make the project more stable for a diverse set of GPUs. Running larger projects stably on diverse GPU species is an active area of work for several of the labs involved in folding at home.

Regarding dumping- to my knowledge this is a new issue that has arisen since 128.252.203.11's load and points issues at the end of last week through the beginning of this week. We have not observed this error before and are actively working to understand what's caused the issue and whether it is a ripple effect from the high load incident. I have halted all assignments from 128.252.203.11 again until we can resolve this issue. Our intention is never to assign WUs that are meaningless work and I'm similarly unhappy that we're seeing this.
Top
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by Joe_H »

Check the WS connection to the CS, the dumping is occurring when the upload fails to the WS and goes to the CS instead. If the CS does not have the WU data to recognize a valid one, it gets dumped.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Top
jjmiller
Scientist
Posts: 136
Joined: Fri Apr 09, 2021 4:43 pm

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by jjmiller »

The configuration of 128.252.203.11 is really strange in that it functions as both a WS and a CS. One thing that we've noticed so far is that in the WUs being dumped our trajectory file hasn't been updated with the newest generations that have come back. For example, in R1245:C3:G9 the trajectory file (state.xml) was written last on Jun 15 at 04:07. Results8 (results from gen8) was written on Jun 17 12:28. State,xml should have been updated after gen8 returned, but this has not happened.
Top
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by Joe_H »

Yes, so it is acting as both, but in this case the WUs were dumped after an upload to a different server acting as CS, 128.252.203.2:

Code: Select all

16:55:56:WARNING:WU02:FS01:Exception: Failed to send results to work server: Not connected
16:55:56:WU02:FS01:Trying to send results to collection server
16:55:56:WU02:FS01:Uploading 27.51MiB to 128.252.203.2
16:55:56:WU02:FS01:Connecting to 128.252.203.2:8080
16:56:02:WU02:FS01:Upload 41.35%
16:56:08:WU02:FS01:Upload 99.74%
16:56:08:WU02:FS01:Upload complete
16:56:08:WU02:FS01:Server responded WORK_QUIT (404)
16:56:08:WARNING:WU02:FS01:Server did not like results, dumping
16:56:08:WU02:FS01:Cleaning up
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Top
parkut
Posts: 363
Joined: Tue Feb 12, 2008 7:33 am
Hardware configuration: Running exclusively Linux headless blades. All are dedicated crunching machines.
Location: SE Michigan, USA

Project: 18202 (Run 15116, Clone 2, Gen 6) dumped

Post by parkut »

Another 18202 dumped, just like the WU above: viewtopic.php?f=19&t=37280#p352083
This is project:18202 run:15116 clone:2 gen:6

Code: Select all

19:07:59:WU02:FS01:0x22:Completed 1225000 out of 1250000 steps (98%)
19:08:00:WU02:FS01:0x22:Checkpoint completed at step 1225000
19:11:02:WU02:FS01:0x22:Completed 1237500 out of 1250000 steps (99%)
19:14:03:WU02:FS01:0x22:Completed 1250000 out of 1250000 steps (100%)
19:14:03:WU02:FS01:0x22:Average performance: 9.52066 ns/day
19:14:05:WU02:FS01:0x22:Checkpoint completed at step 1250000
19:14:12:WU02:FS01:0x22:Saving result file ../logfile_01.txt
19:14:12:WU02:FS01:0x22:Saving result file checkpointIntegrator.xml
19:14:12:WU02:FS01:0x22:Saving result file checkpointState.xml
19:14:18:WU02:FS01:0x22:Saving result file positions.xtc
19:14:18:WU02:FS01:0x22:Saving result file science.log
19:14:18:WU02:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
19:14:18:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
19:14:19:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18202 run:15116 clone:2 gen:6 core:0x22 unit:0x00000002000000060000471a00003b0c
19:14:19:WU02:FS01:Uploading 27.50MiB to 128.252.203.11
19:14:19:WU02:FS01:Connecting to 128.252.203.11:8080
19:14:50:WARNING:WU02:FS01:Exception: Failed to send results to work server: Not connected
19:14:50:WU02:FS01:Trying to send results to collection server
19:14:51:WU02:FS01:Uploading 27.50MiB to 128.252.203.2
19:14:51:WU02:FS01:Connecting to 128.252.203.2:8080
19:14:57:WU02:FS01:Upload 12.50%
19:15:03:WU02:FS01:Upload 24.77%
19:15:09:WU02:FS01:Upload 37.05%
19:15:15:WU02:FS01:Upload 49.10%
19:15:21:WU02:FS01:Upload 61.37%
19:15:27:WU02:FS01:Upload 73.64%
19:15:33:WU02:FS01:Upload 85.69%
19:15:39:WU02:FS01:Upload 97.96%
19:15:40:WU02:FS01:Upload complete
19:15:40:WU02:FS01:Server responded WORK_QUIT (404)
19:15:40:WARNING:WU02:FS01:Server did not like results, dumping
19:15:40:WU02:FS01:Cleaning up
Top
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by bruce »

Work Quit generally means that you've already uploaded that WU and the server is refusing to give you credit for it again.
NOTE: This server is apparently accepting WUs (once) but the credit reports are not reaching the stats server so you can't tell what's actually happening.
Top
aetch
Posts: 436
Joined: Thu Jun 25, 2020 3:04 pm
Location: Between chair and keyboard

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by aetch »

I've had similar for one of my units today (project:18202 run:1927 clone:2 gen:5), tried to upload to Work Server 128.252.203.11, failed, uploaded to Collection Server 128.252.203.2, which then dumped the results.

I checked the Work unit status to find that it had been returned "Faulty 2" by 2 previous donors.
The last was recorded 10 days ago.
Considering the timeout is 2 days and many donors will able to churn through these units in a few hours I can't help but wonder if the 2 recorded returns, of my unit, were actually Faulty and how many times the units have been assigned to donors and then had their results dumped.
https://apps.foldingathome.org/wu#proje ... ne=2&gen=5

Code: Select all

11:43:09:WU01:FS01:Assigned to work server 207.53.233.146
11:43:09:WU01:FS01:Requesting new work unit for slot 01: gpu:4:0 TU104 [GeForce RTX 2070 SUPER] 8218 from 207.53.233.146
11:43:09:WU01:FS01:Connecting to 207.53.233.146:8080
11:43:10:WU01:FS01:Downloading 6.10MiB
11:43:12:WU01:FS01:Download complete
11:43:12:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:17804 run:83 clone:289 gen:77 core:0x22 unit:0x000001210000004d0000458c00000053
11:43:12:WU02:FS01:0x22:Checkpoint completed at step 1250000
11:43:18:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
11:43:18:WU02:FS01:0x22:Saving result file checkpointIntegrator.xml
11:43:18:WU02:FS01:0x22:Saving result file checkpointState.xml
11:43:22:WU02:FS01:0x22:Saving result file positions.xtc
11:43:22:WU02:FS01:0x22:Saving result file science.log
11:43:22:WU02:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
11:43:23:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:43:23:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18202 run:1927 clone:2 gen:5 core:0x22 unit:0x00000002000000050000471a00000787
11:43:23:WU02:FS01:Uploading 27.50MiB to 128.252.203.11
11:43:23:WU02:FS01:Connecting to 128.252.203.11:8080
11:43:23:WU01:FS01:Starting
11:43:23:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 9352 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
11:43:23:WU01:FS01:Started FahCore on PID 5552
11:43:23:WU01:FS01:Core PID:11248
11:43:23:WU01:FS01:FahCore 0x22 started
11:43:24:WU01:FS01:0x22:*********************** Log Started 2021-06-27T11:43:23Z ***********************
11:43:24:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
11:43:24:WU01:FS01:0x22:       Core: Core22
11:43:24:WU01:FS01:0x22:       Type: 0x22
11:43:24:WU01:FS01:0x22:    Version: 0.0.13
11:43:24:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
11:43:24:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
11:43:24:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
11:43:24:WU01:FS01:0x22:       Date: Sep 19 2020
11:43:24:WU01:FS01:0x22:       Time: 02:35:58
11:43:24:WU01:FS01:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
11:43:24:WU01:FS01:0x22:     Branch: core22-0.0.13
11:43:24:WU01:FS01:0x22:   Compiler: Visual C++ 2015
11:43:24:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:43:24:WU01:FS01:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
11:43:24:WU01:FS01:0x22:   Platform: win32 10
11:43:24:WU01:FS01:0x22:       Bits: 64
11:43:24:WU01:FS01:0x22:       Mode: Release
11:43:24:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
11:43:24:WU01:FS01:0x22:             <peastman@stanford.edu>
11:43:24:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 5552 -checkpoint 15
11:43:24:WU01:FS01:0x22:             -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
11:43:24:WU01:FS01:0x22:             nvidia -gpu 0 -gpu-usage 100
11:43:24:WU01:FS01:0x22:************************************ libFAH ************************************
11:43:24:WU01:FS01:0x22:       Date: Sep 7 2020
11:43:24:WU01:FS01:0x22:       Time: 19:09:56
11:43:24:WU01:FS01:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
11:43:24:WU01:FS01:0x22:     Branch: HEAD
11:43:24:WU01:FS01:0x22:   Compiler: Visual C++ 2015
11:43:24:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:43:24:WU01:FS01:0x22:   Platform: win32 10
11:43:24:WU01:FS01:0x22:       Bits: 64
11:43:24:WU01:FS01:0x22:       Mode: Release
11:43:24:WU01:FS01:0x22:************************************ CBang *************************************
11:43:24:WU01:FS01:0x22:       Date: Sep 7 2020
11:43:24:WU01:FS01:0x22:       Time: 19:08:30
11:43:24:WU01:FS01:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
11:43:24:WU01:FS01:0x22:     Branch: HEAD
11:43:24:WU01:FS01:0x22:   Compiler: Visual C++ 2015
11:43:24:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
11:43:24:WU01:FS01:0x22:   Platform: win32 10
11:43:24:WU01:FS01:0x22:       Bits: 64
11:43:24:WU01:FS01:0x22:       Mode: Release
11:43:24:WU01:FS01:0x22:************************************ System ************************************
11:43:24:WU01:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
11:43:24:WU01:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
11:43:24:WU01:FS01:0x22:       CPUs: 24
11:43:24:WU01:FS01:0x22:     Memory: 15.95GiB
11:43:24:WU01:FS01:0x22:Free Memory: 10.89GiB
11:43:24:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
11:43:24:WU01:FS01:0x22: OS Version: 6.2
11:43:24:WU01:FS01:0x22:Has Battery: false
11:43:24:WU01:FS01:0x22: On Battery: false
11:43:24:WU01:FS01:0x22: UTC Offset: 1
11:43:24:WU01:FS01:0x22:        PID: 11248
11:43:24:WU01:FS01:0x22:        CWD: C:\ProgramData\FAHClient\work
11:43:24:WU01:FS01:0x22:************************************ OpenMM ************************************
11:43:24:WU01:FS01:0x22:   Revision: 189320d0
11:43:24:WU01:FS01:0x22:********************************************************************************
11:43:24:WU01:FS01:0x22:Project: 17804 (Run 83, Clone 289, Gen 77)
11:43:24:WU01:FS01:0x22:Unit: 0x00000000000000000000000000000000
11:43:24:WU01:FS01:0x22:Reading tar file core.xml
11:43:24:WU01:FS01:0x22:Reading tar file integrator.xml.bz2
11:43:24:WU01:FS01:0x22:Reading tar file state.xml.bz2
11:43:24:WU01:FS01:0x22:Reading tar file system.xml.bz2
11:43:24:WU01:FS01:0x22:Digital signatures verified
11:43:24:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
11:43:24:WU01:FS01:0x22:Version 0.0.13
11:43:24:WU01:FS01:0x22:  Checkpoint write interval: 125000 steps (5%) [20 total]
11:43:24:WU01:FS01:0x22:  JSON viewer frame write interval: 25000 steps (1%) [100 total]
11:43:24:WU01:FS01:0x22:  XTC frame write interval: 25000 steps (1%) [100 total]
11:43:24:WU01:FS01:0x22:  Global context and integrator variables write interval: disabled
11:43:24:WU01:FS01:0x22:There are 4 platforms available.
11:43:24:WU01:FS01:0x22:Platform 0: Reference
11:43:24:WU01:FS01:0x22:Platform 1: CPU
11:43:24:WU01:FS01:0x22:Platform 2: OpenCL
11:43:24:WU01:FS01:0x22:  opencl-device 0 specified
11:43:24:WU01:FS01:0x22:Platform 3: CUDA
11:43:24:WU01:FS01:0x22:  cuda-device 0 specified
11:43:29:WU01:FS01:0x22:Attempting to create CUDA context:
11:43:29:WU01:FS01:0x22:  Configuring platform CUDA
11:43:31:WU01:FS01:0x22:  Using CUDA and gpu 0
11:43:31:WU01:FS01:0x22:Completed 0 out of 2500000 steps (0%)
11:43:31:WU01:FS01:0x22:Checkpoint completed at step 0
11:43:54:WU02:FS01:Upload 0.68%
11:43:54:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
11:43:54:WU02:FS01:Trying to send results to collection server
11:43:54:WU02:FS01:Uploading 27.50MiB to 128.252.203.2
11:43:54:WU02:FS01:Connecting to 128.252.203.2:8080
11:44:00:WU02:FS01:Upload 9.77%
11:44:06:WU02:FS01:Upload 20.91%
11:44:12:WU02:FS01:Upload 32.27%
11:44:18:WU02:FS01:Upload 43.18%
11:44:24:WU02:FS01:Upload 54.54%
11:44:30:WU02:FS01:Upload 65.90%
11:44:36:WU02:FS01:Upload 77.03%
11:44:42:WU02:FS01:Upload 88.40%
11:44:43:WU01:FS01:0x22:Completed 25000 out of 2500000 steps (1%)
11:44:48:WU02:FS01:Upload 99.30%
11:44:48:WU02:FS01:Upload complete
11:44:48:WU02:FS01:Server responded WORK_QUIT (404)
11:44:48:WARNING:WU02:FS01:Server did not like results, dumping
11:44:48:WU02:FS01:Cleaning up
Folding Rigs - None (25-Jun-2022)

ImageImage
Top
jjmiller
Scientist
Posts: 136
Joined: Fri Apr 09, 2021 4:43 pm

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by jjmiller »

Hi Aetch, The log you posted is for Project 17804, Run 83, Clone 289, Gen 77. Would you be willing to post the log for project:18202 run:1927 clone:2 gen:5?

Some of the gap that you're seeing on the WU status app is because we actually took this project down for ~8 days to resolve some server issues we were having. During that time, no new WUs were being assigned. We turned this project back on yesterday ~mid afternoon CST thus you are likely the first one to get this specific WU again.

Several things can result in "faulty WUs". One of them is that the WU has to restart in the middle because the protein goes out of the "tolerated" forces expected on the system. While this isn't ultimately bad for the project (it happens more for larger systems such as project 18202), it does give the client some issues. With the server reset a few days ago we slightly stepped back the tolerances which should reduce the likelihood of these instances happening.

Thanks,
Top
aetch
Posts: 436
Joined: Thu Jun 25, 2020 3:04 pm
Location: Between chair and keyboard

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by aetch »

I did post project 18202, give me a moment or two while I edit out 17804 that was intermingled with it during the transition from 18202 to 17804.

Code: Select all

11:43:12:WU02:FS01:0x22:Checkpoint completed at step 1250000
11:43:18:WU02:FS01:0x22:Saving result file ..\logfile_01.txt
11:43:18:WU02:FS01:0x22:Saving result file checkpointIntegrator.xml
11:43:18:WU02:FS01:0x22:Saving result file checkpointState.xml
11:43:22:WU02:FS01:0x22:Saving result file positions.xtc
11:43:22:WU02:FS01:0x22:Saving result file science.log
11:43:22:WU02:FS01:0x22:Folding@home Core Shutdown: FINISHED_UNIT
11:43:23:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
11:43:23:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:18202 run:1927 clone:2 gen:5 core:0x22 unit:0x00000002000000050000471a00000787
11:43:23:WU02:FS01:Uploading 27.50MiB to 128.252.203.11
11:43:23:WU02:FS01:Connecting to 128.252.203.11:8080
11:43:54:WU02:FS01:Upload 0.68%
11:43:54:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
11:43:54:WU02:FS01:Trying to send results to collection server
11:43:54:WU02:FS01:Uploading 27.50MiB to 128.252.203.2
11:43:54:WU02:FS01:Connecting to 128.252.203.2:8080
11:44:00:WU02:FS01:Upload 9.77%
11:44:06:WU02:FS01:Upload 20.91%
11:44:12:WU02:FS01:Upload 32.27%
11:44:18:WU02:FS01:Upload 43.18%
11:44:24:WU02:FS01:Upload 54.54%
11:44:30:WU02:FS01:Upload 65.90%
11:44:36:WU02:FS01:Upload 77.03%
11:44:42:WU02:FS01:Upload 88.40%
11:44:48:WU02:FS01:Upload 99.30%
11:44:48:WU02:FS01:Upload complete
11:44:48:WU02:FS01:Server responded WORK_QUIT (404)
11:44:48:WARNING:WU02:FS01:Server did not like results, dumping
11:44:48:WU02:FS01:Cleaning up
I'm cool with the project being withdrawn while the bugs are investigated and sorted, my concern had been that precious folding resources had been wasted.
Folding Rigs - None (25-Jun-2022)

ImageImage
Top
jjmiller
Scientist
Posts: 136
Joined: Fri Apr 09, 2021 4:43 pm

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by jjmiller »

Ah, I see- thanks for the clarification. I'll check in with our collection server again and will report back!

EDIT- Since we have many WUs coming back successfully when uploaded to the work server and all of these reports seem to follow a consistent trend (WS busy -> upload to CS -> CS dumps) we've swapped our collection server out for a new one. It may take a little while for this change to permeate to all WUs on highland1 (majority of WUs out are: 18201, 18202, 18206), but we're starting to see a lot of successful WUs returned to the new CS as opposed to our old one giving us hope that this fixes the problem. We're continuing to assess our new CS. Thanks again for raising this issue.
Top
Kebast
Posts: 386
Joined: Thu Aug 06, 2015 5:21 pm

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by Kebast »

I just got this again today after rebooting machine and grabbing new work:

Code: Select all

16:00:52:WU02:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:00:52:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:00:52:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:18202 run:9669 clone:3 gen:15 core:0x22 unit:0x000000030000000f0000471a000025c5
16:00:52:WU02:FS00:Uploading 2.64KiB to 128.252.203.11
16:00:52:WU02:FS00:Connecting to 128.252.203.11:8080
16:00:52:WU02:FS00:Upload complete
16:00:52:WU02:FS00:Server responded WORK_ACK (400)
16:00:52:WU02:FS00:Cleaning up
16:00:52:WU01:FS00:Connecting to assign1.foldingathome.org:80
16:00:52:WU01:FS00:Assigned to work server 34.72.228.44
16:00:52:WU01:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 34.72.228.44
16:00:52:WU01:FS00:Connecting to 34.72.228.44:8080
16:00:53:WU01:FS00:Downloading 1.69MiB
16:00:53:WU01:FS00:Download complete
16:00:53:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:18105 run:0 clone:72 gen:83 core:0x22 unit:0x0000004800000053000046b900000000
16:00:53:WU01:FS00:Starting
16:00:53:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:00:53:WU01:FS00:Started FahCore on PID 12844
16:00:53:WU01:FS00:Core PID:5512
16:00:53:WU01:FS00:FahCore 0x22 started
16:00:54:WU01:FS00:0x22:*********************** Log Started 2021-07-15T16:00:53Z ***********************
16:00:54:WU01:FS00:0x22:*************************** Core22 Folding@home Core ***************************
16:00:54:WU01:FS00:0x22:       Core: Core22
16:00:54:WU01:FS00:0x22:       Type: 0x22
16:00:54:WU01:FS00:0x22:    Version: 0.0.13
16:00:54:WU01:FS00:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:00:54:WU01:FS00:0x22:  Copyright: 2020 foldingathome.org
16:00:54:WU01:FS00:0x22:   Homepage: https://foldingathome.org/
16:00:54:WU01:FS00:0x22:       Date: Sep 19 2020
16:00:54:WU01:FS00:0x22:       Time: 02:35:58
16:00:54:WU01:FS00:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:00:54:WU01:FS00:0x22:     Branch: core22-0.0.13
16:00:54:WU01:FS00:0x22:   Compiler: Visual C++ 2015
16:00:54:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:00:54:WU01:FS00:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:00:54:WU01:FS00:0x22:   Platform: win32 10
16:00:54:WU01:FS00:0x22:       Bits: 64
16:00:54:WU01:FS00:0x22:       Mode: Release
16:00:54:WU01:FS00:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:00:54:WU01:FS00:0x22:             <peastman@stanford.edu>
16:00:54:WU01:FS00:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 12844 -checkpoint 15
16:00:54:WU01:FS00:0x22:             -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
16:00:54:WU01:FS00:0x22:             nvidia -gpu 1 -gpu-usage 100
16:00:54:WU01:FS00:0x22:************************************ libFAH ************************************
16:00:54:WU01:FS00:0x22:       Date: Sep 7 2020
16:00:54:WU01:FS00:0x22:       Time: 19:09:56
16:00:54:WU01:FS00:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:00:54:WU01:FS00:0x22:     Branch: HEAD
16:00:54:WU01:FS00:0x22:   Compiler: Visual C++ 2015
16:00:54:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:00:54:WU01:FS00:0x22:   Platform: win32 10
16:00:54:WU01:FS00:0x22:       Bits: 64
16:00:54:WU01:FS00:0x22:       Mode: Release
16:00:54:WU01:FS00:0x22:************************************ CBang *************************************
16:00:54:WU01:FS00:0x22:       Date: Sep 7 2020
16:00:54:WU01:FS00:0x22:       Time: 19:08:30
16:00:54:WU01:FS00:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:00:54:WU01:FS00:0x22:     Branch: HEAD
16:00:54:WU01:FS00:0x22:   Compiler: Visual C++ 2015
16:00:54:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:00:54:WU01:FS00:0x22:   Platform: win32 10
16:00:54:WU01:FS00:0x22:       Bits: 64
16:00:54:WU01:FS00:0x22:       Mode: Release
16:00:54:WU01:FS00:0x22:************************************ System ************************************
16:00:54:WU01:FS00:0x22:        CPU: AMD FX(tm)-6300 Six-Core Processor
16:00:54:WU01:FS00:0x22:     CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
16:00:54:WU01:FS00:0x22:       CPUs: 6
16:00:54:WU01:FS00:0x22:     Memory: 15.97GiB
16:00:54:WU01:FS00:0x22:Free Memory: 11.71GiB
16:00:54:WU01:FS00:0x22:    Threads: WINDOWS_THREADS
16:00:54:WU01:FS00:0x22: OS Version: 6.2
16:00:54:WU01:FS00:0x22:Has Battery: true
16:00:54:WU01:FS00:0x22: On Battery: false
16:00:54:WU01:FS00:0x22: UTC Offset: -4
16:00:54:WU01:FS00:0x22:        PID: 5512
16:00:54:WU01:FS00:0x22:        CWD: C:\ProgramData\FAHClient\work
16:00:54:WU01:FS00:0x22:************************************ OpenMM ************************************
16:00:54:WU01:FS00:0x22:   Revision: 189320d0
16:00:54:WU01:FS00:0x22:********************************************************************************
16:00:54:WU01:FS00:0x22:Project: 18105 (Run 0, Clone 72, Gen 83)
16:00:54:WU01:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:00:54:WU01:FS00:0x22:Reading tar file core.xml
16:00:54:WU01:FS00:0x22:Reading tar file integrator.xml.bz2
16:00:54:WU01:FS00:0x22:Reading tar file state.xml.bz2
16:00:54:WU01:FS00:0x22:Reading tar file system.xml.bz2
16:00:54:WU01:FS00:0x22:Digital signatures verified
16:00:54:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:00:54:WU01:FS00:0x22:Version 0.0.13
16:00:54:WU01:FS00:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
16:00:54:WU01:FS00:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
16:00:54:WU01:FS00:0x22:  XTC frame write interval: 50000 steps (1%) [100 total]
16:00:54:WU01:FS00:0x22:  Global context and integrator variables write interval: disabled
16:00:54:WU01:FS00:0x22:There are 4 platforms available.
16:00:54:WU01:FS00:0x22:Platform 0: Reference
16:00:54:WU01:FS00:0x22:Platform 1: CPU
16:00:54:WU01:FS00:0x22:Platform 2: OpenCL
16:00:54:WU01:FS00:0x22:  opencl-device 1 specified
16:00:54:WU01:FS00:0x22:Platform 3: CUDA
16:00:54:WU01:FS00:0x22:  cuda-device 1 specified
16:00:56:WU01:FS00:0x22:Attempting to create CUDA context:
16:00:56:WU01:FS00:0x22:  Configuring platform CUDA
16:00:56:WU01:FS00:0x22:Failed to create CUDA context:
16:00:56:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:00:56:WU01:FS00:0x22:Attempting to create OpenCL context:
16:00:56:WU01:FS00:0x22:  Configuring platform OpenCL
16:00:56:WU01:FS00:0x22:Failed to create OpenCL context:
16:00:56:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:00:56:WU01:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:00:56:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
16:00:56:WU01:FS00:0x22:Saving result file science.log
16:00:56:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:00:57:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:00:57:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:18105 run:0 clone:72 gen:83 core:0x22 unit:0x0000004800000053000046b900000000
16:00:57:WU01:FS00:Uploading 11.50KiB to 34.72.228.44
16:00:57:WU01:FS00:Connecting to 34.72.228.44:8080
16:00:57:WU01:FS00:Upload complete
16:00:57:WU01:FS00:Server responded WORK_ACK (400)
16:00:57:WU01:FS00:Cleaning up
16:00:57:WU02:FS00:Connecting to assign1.foldingathome.org:80
16:00:57:WU02:FS00:Assigned to work server 207.53.233.146
16:00:57:WU02:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 207.53.233.146
16:00:57:WU02:FS00:Connecting to 207.53.233.146:8080
16:00:57:WU02:FS00:Downloading 2.47MiB
16:00:58:WU02:FS00:Download complete
16:00:58:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:17806 run:49 clone:67 gen:40 core:0x22 unit:0x00000043000000280000458e00000031
16:00:58:WU02:FS00:Starting
16:00:58:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:00:58:WU02:FS00:Started FahCore on PID 10876
16:00:58:WU02:FS00:Core PID:9956
16:00:58:WU02:FS00:FahCore 0x22 started
16:00:58:WU02:FS00:0x22:*********************** Log Started 2021-07-15T16:00:58Z ***********************
16:00:58:WU02:FS00:0x22:*************************** Core22 Folding@home Core ***************************
16:00:58:WU02:FS00:0x22:       Core: Core22
16:00:58:WU02:FS00:0x22:       Type: 0x22
16:00:58:WU02:FS00:0x22:    Version: 0.0.13
16:00:58:WU02:FS00:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:00:58:WU02:FS00:0x22:  Copyright: 2020 foldingathome.org
16:00:58:WU02:FS00:0x22:   Homepage: https://foldingathome.org/
16:00:58:WU02:FS00:0x22:       Date: Sep 19 2020
16:00:58:WU02:FS00:0x22:       Time: 02:35:58
16:00:58:WU02:FS00:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:00:58:WU02:FS00:0x22:     Branch: core22-0.0.13
16:00:58:WU02:FS00:0x22:   Compiler: Visual C++ 2015
16:00:58:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:00:58:WU02:FS00:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:00:58:WU02:FS00:0x22:   Platform: win32 10
16:00:58:WU02:FS00:0x22:       Bits: 64
16:00:58:WU02:FS00:0x22:       Mode: Release
16:00:58:WU02:FS00:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:00:58:WU02:FS00:0x22:             <peastman@stanford.edu>
16:00:58:WU02:FS00:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 10876 -checkpoint 15
16:00:58:WU02:FS00:0x22:             -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
16:00:58:WU02:FS00:0x22:             nvidia -gpu 1 -gpu-usage 100
16:00:58:WU02:FS00:0x22:************************************ libFAH ************************************
16:00:58:WU02:FS00:0x22:       Date: Sep 7 2020
16:00:58:WU02:FS00:0x22:       Time: 19:09:56
16:00:58:WU02:FS00:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:00:58:WU02:FS00:0x22:     Branch: HEAD
16:00:58:WU02:FS00:0x22:   Compiler: Visual C++ 2015
16:00:58:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:00:58:WU02:FS00:0x22:   Platform: win32 10
16:00:58:WU02:FS00:0x22:       Bits: 64
16:00:58:WU02:FS00:0x22:       Mode: Release
16:00:58:WU02:FS00:0x22:************************************ CBang *************************************
16:00:58:WU02:FS00:0x22:       Date: Sep 7 2020
16:00:58:WU02:FS00:0x22:       Time: 19:08:30
16:00:58:WU02:FS00:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:00:58:WU02:FS00:0x22:     Branch: HEAD
16:00:58:WU02:FS00:0x22:   Compiler: Visual C++ 2015
16:00:58:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:00:58:WU02:FS00:0x22:   Platform: win32 10
16:00:58:WU02:FS00:0x22:       Bits: 64
16:00:58:WU02:FS00:0x22:       Mode: Release
16:00:58:WU02:FS00:0x22:************************************ System ************************************
16:00:58:WU02:FS00:0x22:        CPU: AMD FX(tm)-6300 Six-Core Processor
16:00:58:WU02:FS00:0x22:     CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
16:00:58:WU02:FS00:0x22:       CPUs: 6
16:00:58:WU02:FS00:0x22:     Memory: 15.97GiB
16:00:58:WU02:FS00:0x22:Free Memory: 11.73GiB
16:00:58:WU02:FS00:0x22:    Threads: WINDOWS_THREADS
16:00:58:WU02:FS00:0x22: OS Version: 6.2
16:00:58:WU02:FS00:0x22:Has Battery: true
16:00:58:WU02:FS00:0x22: On Battery: false
16:00:58:WU02:FS00:0x22: UTC Offset: -4
16:00:58:WU02:FS00:0x22:        PID: 9956
16:00:58:WU02:FS00:0x22:        CWD: C:\ProgramData\FAHClient\work
16:00:58:WU02:FS00:0x22:************************************ OpenMM ************************************
16:00:58:WU02:FS00:0x22:   Revision: 189320d0
16:00:58:WU02:FS00:0x22:********************************************************************************
16:00:58:WU02:FS00:0x22:Project: 17806 (Run 49, Clone 67, Gen 40)
16:00:58:WU02:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:00:58:WU02:FS00:0x22:Reading tar file core.xml
16:00:58:WU02:FS00:0x22:Reading tar file integrator.xml.bz2
16:00:58:WU02:FS00:0x22:Reading tar file state.xml.bz2
16:00:58:WU02:FS00:0x22:Reading tar file system.xml.bz2
16:00:58:WU02:FS00:0x22:Digital signatures verified
16:00:58:WU02:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:00:58:WU02:FS00:0x22:Version 0.0.13
16:00:58:WU02:FS00:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
16:00:58:WU02:FS00:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
16:00:58:WU02:FS00:0x22:  XTC frame write interval: 25000 steps (0.5%) [200 total]
16:00:58:WU02:FS00:0x22:  Global context and integrator variables write interval: disabled
16:00:58:WU02:FS00:0x22:There are 4 platforms available.
16:00:58:WU02:FS00:0x22:Platform 0: Reference
16:00:58:WU02:FS00:0x22:Platform 1: CPU
16:00:58:WU02:FS00:0x22:Platform 2: OpenCL
16:00:58:WU02:FS00:0x22:  opencl-device 1 specified
16:00:58:WU02:FS00:0x22:Platform 3: CUDA
16:00:58:WU02:FS00:0x22:  cuda-device 1 specified
16:01:02:WU02:FS00:0x22:Attempting to create CUDA context:
16:01:02:WU02:FS00:0x22:  Configuring platform CUDA
16:01:02:WU02:FS00:0x22:Failed to create CUDA context:
16:01:02:WU02:FS00:0x22:Illegal value for DeviceIndex: 1
16:01:02:WU02:FS00:0x22:Attempting to create OpenCL context:
16:01:02:WU02:FS00:0x22:  Configuring platform OpenCL
16:01:02:WU02:FS00:0x22:Failed to create OpenCL context:
16:01:02:WU02:FS00:0x22:Illegal value for DeviceIndex: 1
16:01:02:WU02:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:01:02:WU02:FS00:0x22:Saving result file ..\logfile_01.txt
16:01:02:WU02:FS00:0x22:Saving result file science.log
16:01:02:WU02:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:01:02:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:01:02:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:17806 run:49 clone:67 gen:40 core:0x22 unit:0x00000043000000280000458e00000031
16:01:02:WU02:FS00:Uploading 11.50KiB to 207.53.233.146
16:01:02:WU02:FS00:Connecting to 207.53.233.146:8080
16:01:02:WU02:FS00:Upload complete
16:01:03:WU02:FS00:Server responded WORK_ACK (400)
16:01:03:WU02:FS00:Cleaning up
16:01:03:WU01:FS00:Connecting to assign1.foldingathome.org:80
16:01:03:WU01:FS00:Assigned to work server 207.53.233.146
16:01:03:WU01:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 207.53.233.146
16:01:03:WU01:FS00:Connecting to 207.53.233.146:8080
16:01:03:WU01:FS00:Downloading 5.48MiB
16:01:04:WU01:FS00:Download complete
16:01:04:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:17804 run:52 clone:269 gen:142 core:0x22 unit:0x0000010d0000008e0000458c00000034
16:01:04:WU01:FS00:Starting
16:01:04:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:01:04:WU01:FS00:Started FahCore on PID 12980
16:01:04:WU01:FS00:Core PID:9668
16:01:04:WU01:FS00:FahCore 0x22 started


16:01:04:WU01:FS00:0x22:Project: 17804 (Run 52, Clone 269, Gen 142)
16:01:04:WU01:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:01:04:WU01:FS00:0x22:Reading tar file core.xml
16:01:04:WU01:FS00:0x22:Reading tar file integrator.xml.bz2
16:01:04:WU01:FS00:0x22:Reading tar file state.xml.bz2
16:01:04:WU01:FS00:0x22:Reading tar file system.xml.bz2
16:01:04:WU01:FS00:0x22:Digital signatures verified
16:01:04:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:01:04:WU01:FS00:0x22:Version 0.0.13
16:01:04:WU01:FS00:0x22:  Checkpoint write interval: 125000 steps (5%) [20 total]
16:01:04:WU01:FS00:0x22:  JSON viewer frame write interval: 25000 steps (1%) [100 total]
16:01:04:WU01:FS00:0x22:  XTC frame write interval: 25000 steps (1%) [100 total]
16:01:04:WU01:FS00:0x22:  Global context and integrator variables write interval: disabled
16:01:04:WU01:FS00:0x22:There are 4 platforms available.
16:01:04:WU01:FS00:0x22:Platform 0: Reference
16:01:04:WU01:FS00:0x22:Platform 1: CPU
16:01:04:WU01:FS00:0x22:Platform 2: OpenCL
16:01:04:WU01:FS00:0x22:  opencl-device 1 specified
16:01:04:WU01:FS00:0x22:Platform 3: CUDA
16:01:04:WU01:FS00:0x22:  cuda-device 1 specified
16:01:12:WU01:FS00:0x22:Attempting to create CUDA context:
16:01:12:WU01:FS00:0x22:  Configuring platform CUDA
16:01:12:WU01:FS00:0x22:Failed to create CUDA context:
16:01:12:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:01:12:WU01:FS00:0x22:Attempting to create OpenCL context:
16:01:12:WU01:FS00:0x22:  Configuring platform OpenCL
16:01:13:WU01:FS00:0x22:Failed to create OpenCL context:
16:01:13:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:01:13:WU01:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:01:13:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
16:01:13:WU01:FS00:0x22:Saving result file science.log
16:01:13:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:01:13:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:01:13:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:17804 run:52 clone:269 gen:142 core:0x22 unit:0x0000010d0000008e0000458c00000034
16:01:13:WU01:FS00:Uploading 11.50KiB to 207.53.233.146
16:01:13:WU01:FS00:Connecting to 207.53.233.146:8080
16:01:13:WU01:FS00:Upload complete
16:01:13:WU01:FS00:Server responded WORK_ACK (400)
16:01:13:WU01:FS00:Cleaning up
16:01:13:WU02:FS00:Connecting to assign1.foldingathome.org:80
16:01:13:WU02:FS00:Assigned to work server 206.223.170.146
16:01:13:WU02:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 206.223.170.146
16:01:13:WU02:FS00:Connecting to 206.223.170.146:8080
16:01:14:WU02:FS00:Downloading 22.92MiB
16:01:17:WU02:FS00:Download complete
16:01:17:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:17435 run:0 clone:728 gen:240 core:0x22 unit:0x000002d8000000f00000441b00000000
16:01:17:WU02:FS00:Starting
16:01:17:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:01:17:WU02:FS00:Started FahCore on PID 12588
16:01:17:WU02:FS00:Core PID:676
16:01:17:WU02:FS00:FahCore 0x22 started
16:01:18:WU02:FS00:0x22:*********************** Log Started 2021-07-15T16:01:17Z ***********************
16:01:18:WU02:FS00:0x22:*************************** Core22 Folding@home Core ***************************
16:01:18:WU02:FS00:0x22:       Core: Core22
16:01:18:WU02:FS00:0x22:       Type: 0x22
16:01:18:WU02:FS00:0x22:    Version: 0.0.13
16:01:18:WU02:FS00:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:01:18:WU02:FS00:0x22:  Copyright: 2020 foldingathome.org
16:01:18:WU02:FS00:0x22:   Homepage: https://foldingathome.org/
16:01:18:WU02:FS00:0x22:       Date: Sep 19 2020
16:01:18:WU02:FS00:0x22:       Time: 02:35:58
16:01:18:WU02:FS00:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:01:18:WU02:FS00:0x22:     Branch: core22-0.0.13
16:01:18:WU02:FS00:0x22:   Compiler: Visual C++ 2015
16:01:18:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:18:WU02:FS00:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:01:18:WU02:FS00:0x22:   Platform: win32 10
16:01:18:WU02:FS00:0x22:       Bits: 64
16:01:18:WU02:FS00:0x22:       Mode: Release
16:01:18:WU02:FS00:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:01:18:WU02:FS00:0x22:             <peastman@stanford.edu>
16:01:18:WU02:FS00:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 12588 -checkpoint 15
16:01:18:WU02:FS00:0x22:             -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
16:01:18:WU02:FS00:0x22:             nvidia -gpu 1 -gpu-usage 100
16:01:18:WU02:FS00:0x22:************************************ libFAH ************************************
16:01:18:WU02:FS00:0x22:       Date: Sep 7 2020
16:01:18:WU02:FS00:0x22:       Time: 19:09:56
16:01:18:WU02:FS00:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:01:18:WU02:FS00:0x22:     Branch: HEAD
16:01:18:WU02:FS00:0x22:   Compiler: Visual C++ 2015
16:01:18:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:18:WU02:FS00:0x22:   Platform: win32 10
16:01:18:WU02:FS00:0x22:       Bits: 64
16:01:18:WU02:FS00:0x22:       Mode: Release
16:01:18:WU02:FS00:0x22:************************************ CBang *************************************
16:01:18:WU02:FS00:0x22:       Date: Sep 7 2020
16:01:18:WU02:FS00:0x22:       Time: 19:08:30
16:01:18:WU02:FS00:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:01:18:WU02:FS00:0x22:     Branch: HEAD
16:01:18:WU02:FS00:0x22:   Compiler: Visual C++ 2015
16:01:18:WU02:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:18:WU02:FS00:0x22:   Platform: win32 10
16:01:18:WU02:FS00:0x22:       Bits: 64
16:01:18:WU02:FS00:0x22:       Mode: Release
16:01:18:WU02:FS00:0x22:************************************ System ************************************
16:01:18:WU02:FS00:0x22:        CPU: AMD FX(tm)-6300 Six-Core Processor
16:01:18:WU02:FS00:0x22:     CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
16:01:18:WU02:FS00:0x22:       CPUs: 6
16:01:18:WU02:FS00:0x22:     Memory: 15.97GiB
16:01:18:WU02:FS00:0x22:Free Memory: 11.71GiB
16:01:18:WU02:FS00:0x22:    Threads: WINDOWS_THREADS
16:01:18:WU02:FS00:0x22: OS Version: 6.2
16:01:18:WU02:FS00:0x22:Has Battery: true
16:01:18:WU02:FS00:0x22: On Battery: false
16:01:18:WU02:FS00:0x22: UTC Offset: -4
16:01:18:WU02:FS00:0x22:        PID: 676
16:01:18:WU02:FS00:0x22:        CWD: C:\ProgramData\FAHClient\work
16:01:18:WU02:FS00:0x22:************************************ OpenMM ************************************
16:01:18:WU02:FS00:0x22:   Revision: 189320d0
16:01:18:WU02:FS00:0x22:********************************************************************************
16:01:18:WU02:FS00:0x22:Project: 17435 (Run 0, Clone 728, Gen 240)
16:01:18:WU02:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:01:18:WU02:FS00:0x22:Reading tar file core.xml
16:01:18:WU02:FS00:0x22:Reading tar file integrator.xml.bz2
16:01:18:WU02:FS00:0x22:Reading tar file state.xml.bz2
16:01:18:WU02:FS00:0x22:Reading tar file system.xml.bz2
16:01:18:WU02:FS00:0x22:Digital signatures verified
16:01:18:WU02:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:01:18:WU02:FS00:0x22:Version 0.0.13
16:01:18:WU02:FS00:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
16:01:18:WU02:FS00:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
16:01:18:WU02:FS00:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
16:01:18:WU02:FS00:0x22:  Global context and integrator variables write interval: disabled
16:01:18:WU02:FS00:0x22:There are 4 platforms available.
16:01:18:WU02:FS00:0x22:Platform 0: Reference
16:01:18:WU02:FS00:0x22:Platform 1: CPU
16:01:18:WU02:FS00:0x22:Platform 2: OpenCL
16:01:18:WU02:FS00:0x22:  opencl-device 1 specified
16:01:18:WU02:FS00:0x22:Platform 3: CUDA
16:01:18:WU02:FS00:0x22:  cuda-device 1 specified
16:01:49:WU02:FS00:0x22:Attempting to create CUDA context:
16:01:49:WU02:FS00:0x22:  Configuring platform CUDA
16:01:49:WU02:FS00:0x22:Failed to create CUDA context:
16:01:49:WU02:FS00:0x22:Illegal value for DeviceIndex: 1
16:01:49:WU02:FS00:0x22:Attempting to create OpenCL context:
16:01:49:WU02:FS00:0x22:  Configuring platform OpenCL
16:01:49:WU02:FS00:0x22:Failed to create OpenCL context:
16:01:49:WU02:FS00:0x22:Illegal value for DeviceIndex: 1
16:01:49:WU02:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:01:49:WU02:FS00:0x22:Saving result file ..\logfile_01.txt
16:01:49:WU02:FS00:0x22:Saving result file science.log
16:01:49:WU02:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:01:49:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:01:49:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:17435 run:0 clone:728 gen:240 core:0x22 unit:0x000002d8000000f00000441b00000000
16:01:49:WU02:FS00:Uploading 11.50KiB to 206.223.170.146
16:01:49:WU02:FS00:Connecting to 206.223.170.146:8080
16:01:49:WU02:FS00:Upload complete
16:01:49:WU02:FS00:Server responded WORK_ACK (400)
16:01:49:WU02:FS00:Cleaning up
16:01:49:WU01:FS00:Connecting to assign1.foldingathome.org:80
16:01:50:WU01:FS00:Assigned to work server 34.72.228.44
16:01:50:WU01:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 34.72.228.44
16:01:50:WU01:FS00:Connecting to 34.72.228.44:8080
16:01:50:WU01:FS00:Downloading 5.07MiB
16:01:50:WU01:FS00:Download complete
16:01:50:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:18003 run:0 clone:1 gen:46 core:0x22 unit:0x000000010000002e0000465300000000
16:01:50:WU01:FS00:Starting
16:01:50:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:01:50:WU01:FS00:Started FahCore on PID 10560
16:01:50:WU01:FS00:Core PID:5788
16:01:50:WU01:FS00:FahCore 0x22 started
16:01:51:WU01:FS00:0x22:*********************** Log Started 2021-07-15T16:01:51Z ***********************
16:01:51:WU01:FS00:0x22:*************************** Core22 Folding@home Core ***************************
16:01:51:WU01:FS00:0x22:       Core: Core22
16:01:51:WU01:FS00:0x22:       Type: 0x22
16:01:51:WU01:FS00:0x22:    Version: 0.0.13
16:01:51:WU01:FS00:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:01:51:WU01:FS00:0x22:  Copyright: 2020 foldingathome.org
16:01:51:WU01:FS00:0x22:   Homepage: https://foldingathome.org/
16:01:51:WU01:FS00:0x22:       Date: Sep 19 2020
16:01:51:WU01:FS00:0x22:       Time: 02:35:58
16:01:51:WU01:FS00:0x22:   Revision: 571cf95de6de2c592c7c3ed48fcfb2e33e9ea7d3
16:01:51:WU01:FS00:0x22:     Branch: core22-0.0.13
16:01:51:WU01:FS00:0x22:   Compiler: Visual C++ 2015
16:01:51:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:51:WU01:FS00:0x22:             -DOPENMM_GIT_HASH="\"189320d0\""
16:01:51:WU01:FS00:0x22:   Platform: win32 10
16:01:51:WU01:FS00:0x22:       Bits: 64
16:01:51:WU01:FS00:0x22:       Mode: Release
16:01:51:WU01:FS00:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
16:01:51:WU01:FS00:0x22:             <peastman@stanford.edu>
16:01:51:WU01:FS00:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 10560 -checkpoint 15
16:01:51:WU01:FS00:0x22:             -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor
16:01:51:WU01:FS00:0x22:             nvidia -gpu 1 -gpu-usage 100
16:01:51:WU01:FS00:0x22:************************************ libFAH ************************************
16:01:51:WU01:FS00:0x22:       Date: Sep 7 2020
16:01:51:WU01:FS00:0x22:       Time: 19:09:56
16:01:51:WU01:FS00:0x22:   Revision: 44301ed97b996b63fe736bb8073f22209cb2b603
16:01:51:WU01:FS00:0x22:     Branch: HEAD
16:01:51:WU01:FS00:0x22:   Compiler: Visual C++ 2015
16:01:51:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:51:WU01:FS00:0x22:   Platform: win32 10
16:01:51:WU01:FS00:0x22:       Bits: 64
16:01:51:WU01:FS00:0x22:       Mode: Release
16:01:51:WU01:FS00:0x22:************************************ CBang *************************************
16:01:51:WU01:FS00:0x22:       Date: Sep 7 2020
16:01:51:WU01:FS00:0x22:       Time: 19:08:30
16:01:51:WU01:FS00:0x22:   Revision: 33fcfc2b3ed2195a423606a264718e31e6b3903f
16:01:51:WU01:FS00:0x22:     Branch: HEAD
16:01:51:WU01:FS00:0x22:   Compiler: Visual C++ 2015
16:01:51:WU01:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
16:01:51:WU01:FS00:0x22:   Platform: win32 10
16:01:51:WU01:FS00:0x22:       Bits: 64
16:01:51:WU01:FS00:0x22:       Mode: Release
16:01:51:WU01:FS00:0x22:************************************ System ************************************
16:01:51:WU01:FS00:0x22:        CPU: AMD FX(tm)-6300 Six-Core Processor
16:01:51:WU01:FS00:0x22:     CPU ID: AuthenticAMD Family 21 Model 2 Stepping 0
16:01:51:WU01:FS00:0x22:       CPUs: 6
16:01:51:WU01:FS00:0x22:     Memory: 15.97GiB
16:01:51:WU01:FS00:0x22:Free Memory: 11.74GiB
16:01:51:WU01:FS00:0x22:    Threads: WINDOWS_THREADS
16:01:51:WU01:FS00:0x22: OS Version: 6.2
16:01:51:WU01:FS00:0x22:Has Battery: true
16:01:51:WU01:FS00:0x22: On Battery: false
16:01:51:WU01:FS00:0x22: UTC Offset: -4
16:01:51:WU01:FS00:0x22:        PID: 5788
16:01:51:WU01:FS00:0x22:        CWD: C:\ProgramData\FAHClient\work
16:01:51:WU01:FS00:0x22:************************************ OpenMM ************************************
16:01:51:WU01:FS00:0x22:   Revision: 189320d0
16:01:51:WU01:FS00:0x22:********************************************************************************
16:01:51:WU01:FS00:0x22:Project: 18003 (Run 0, Clone 1, Gen 46)
16:01:51:WU01:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:01:51:WU01:FS00:0x22:Reading tar file core.xml
16:01:51:WU01:FS00:0x22:Reading tar file integrator.xml.bz2
16:01:51:WU01:FS00:0x22:Reading tar file state.xml.bz2
16:01:51:WU01:FS00:0x22:Reading tar file system.xml.bz2
16:01:51:WU01:FS00:0x22:Digital signatures verified
16:01:51:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:01:51:WU01:FS00:0x22:Version 0.0.13
16:01:51:WU01:FS00:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
16:01:51:WU01:FS00:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
16:01:51:WU01:FS00:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
16:01:51:WU01:FS00:0x22:  Global context and integrator variables write interval: disabled
16:01:51:WU01:FS00:0x22:There are 4 platforms available.
16:01:51:WU01:FS00:0x22:Platform 0: Reference
16:01:51:WU01:FS00:0x22:Platform 1: CPU
16:01:51:WU01:FS00:0x22:Platform 2: OpenCL
16:01:51:WU01:FS00:0x22:  opencl-device 1 specified
16:01:51:WU01:FS00:0x22:Platform 3: CUDA
16:01:51:WU01:FS00:0x22:  cuda-device 1 specified
16:01:58:WU01:FS00:0x22:Attempting to create CUDA context:
16:01:58:WU01:FS00:0x22:  Configuring platform CUDA
16:01:58:WU01:FS00:0x22:Failed to create CUDA context:
16:01:58:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:01:58:WU01:FS00:0x22:Attempting to create OpenCL context:
16:01:58:WU01:FS00:0x22:  Configuring platform OpenCL
16:01:58:WU01:FS00:0x22:Failed to create OpenCL context:
16:01:58:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:01:58:WU01:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:01:58:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
16:01:58:WU01:FS00:0x22:Saving result file science.log
16:01:58:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:01:59:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:01:59:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:18003 run:0 clone:1 gen:46 core:0x22 unit:0x000000010000002e0000465300000000
16:01:59:WU01:FS00:Uploading 11.50KiB to 34.72.228.44
16:01:59:WU01:FS00:Connecting to 34.72.228.44:8080
16:01:59:WU01:FS00:Upload complete
16:01:59:WU01:FS00:Server responded WORK_ACK (400)
16:01:59:WU01:FS00:Cleaning up
16:01:59:WU02:FS00:Connecting to assign1.foldingathome.org:80
16:01:59:WU02:FS00:Assigned to work server 128.252.203.11
16:01:59:WU02:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 128.252.203.11
16:01:59:WU02:FS00:Connecting to 128.252.203.11:8080
16:02:09:WU02:FS00:Downloading 26.52MiB
16:02:11:WU02:FS00:Download complete
16:02:11:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:18202 run:10062 clone:3 gen:15 core:0x22 unit:0x000000030000000f0000471a0000274e
16:02:11:WU02:FS00:Starting
16:02:11:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:02:11:WU02:FS00:Started FahCore on PID 9476
16:02:11:WU02:FS00:Core PID:10248
16:02:11:WU02:FS00:FahCore 0x22 started


16:02:12:WU02:FS00:0x22:Project: 18202 (Run 10062, Clone 3, Gen 15)
16:02:12:WU02:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:02:12:WU02:FS00:0x22:Reading tar file core.xml
16:02:12:WU02:FS00:0x22:Reading tar file integrator.xml
16:02:12:WU02:FS00:0x22:Reading tar file state.xml
16:02:13:WU02:FS00:0x22:Reading tar file system.xml
16:02:14:WU02:FS00:0x22:Digital signatures verified
16:02:14:WU02:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:02:14:WU02:FS00:0x22:Version 0.0.13
16:02:14:WU02:FS00:0x22:  Checkpoint write interval: 25000 steps (2%) [50 total]
16:02:14:WU02:FS00:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
16:02:14:WU02:FS00:0x22:  XTC frame write interval: 20000 steps (1.6%) [62 total]
16:02:14:WU02:FS00:0x22:  Global context and integrator variables write interval: disabled
16:02:14:WU02:FS00:0x22:There are 4 platforms available.
16:02:14:WU02:FS00:0x22:Platform 0: Reference
16:02:14:WU02:FS00:0x22:Platform 1: CPU
16:02:14:WU02:FS00:0x22:Platform 2: OpenCL
16:02:14:WU02:FS00:0x22:  opencl-device 1 specified
16:02:14:WU02:FS00:0x22:Platform 3: CUDA
16:02:14:WU02:FS00:0x22:  cuda-device 1 specified
16:02:41:WU02:FS00:0x22:Attempting to create CUDA context:
16:02:41:WU02:FS00:0x22:  Configuring platform CUDA
16:02:42:WU02:FS00:0x22:Failed to create CUDA context:
16:02:42:WU02:FS00:0x22:Illegal value for DeviceIndex: 1
16:02:42:WU02:FS00:0x22:Attempting to create OpenCL context:
16:02:42:WU02:FS00:0x22:  Configuring platform OpenCL
16:02:42:WU02:FS00:0x22:Failed to create OpenCL context:
16:02:42:WU02:FS00:0x22:Illegal value for DeviceIndex: 1
16:02:42:WU02:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:02:42:WU02:FS00:0x22:Saving result file ..\logfile_01.txt
16:02:42:WU02:FS00:0x22:Saving result file science.log
16:02:42:WU02:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:02:42:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:02:42:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:18202 run:10062 clone:3 gen:15 core:0x22 unit:0x000000030000000f0000471a0000274e
16:02:42:WU02:FS00:Uploading 2.64KiB to 128.252.203.11
16:02:42:WU02:FS00:Connecting to 128.252.203.11:8080
16:02:42:WU02:FS00:Upload complete
16:02:42:WU02:FS00:Server responded WORK_ACK (400)
16:02:42:WU02:FS00:Cleaning up
16:02:42:WU01:FS00:Connecting to assign1.foldingathome.org:80
16:02:43:WU01:FS00:Assigned to work server 34.72.228.44
16:02:43:WU01:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 34.72.228.44
16:02:43:WU01:FS00:Connecting to 34.72.228.44:8080
16:02:43:WU01:FS00:Downloading 5.07MiB
16:02:43:WU01:FS00:Download complete
16:02:43:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:18003 run:0 clone:425 gen:58 core:0x22 unit:0x000001a90000003a0000465300000000
16:02:43:WU01:FS00:Starting
16:02:43:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:02:43:WU01:FS00:Started FahCore on PID 692
16:02:44:WU01:FS00:Core PID:13216
16:02:44:WU01:FS00:FahCore 0x22 started


16:02:44:WU01:FS00:0x22:Project: 18003 (Run 0, Clone 425, Gen 58)
16:02:44:WU01:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:02:44:WU01:FS00:0x22:Reading tar file core.xml
16:02:44:WU01:FS00:0x22:Reading tar file integrator.xml.bz2
16:02:44:WU01:FS00:0x22:Reading tar file state.xml.bz2
16:02:44:WU01:FS00:0x22:Reading tar file system.xml.bz2
16:02:44:WU01:FS00:0x22:Digital signatures verified
16:02:44:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:02:44:WU01:FS00:0x22:Version 0.0.13
16:02:44:WU01:FS00:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
16:02:44:WU01:FS00:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
16:02:44:WU01:FS00:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
16:02:44:WU01:FS00:0x22:  Global context and integrator variables write interval: disabled
16:02:44:WU01:FS00:0x22:There are 4 platforms available.
16:02:44:WU01:FS00:0x22:Platform 0: Reference
16:02:44:WU01:FS00:0x22:Platform 1: CPU
16:02:44:WU01:FS00:0x22:Platform 2: OpenCL
16:02:44:WU01:FS00:0x22:  opencl-device 1 specified
16:02:44:WU01:FS00:0x22:Platform 3: CUDA
16:02:44:WU01:FS00:0x22:  cuda-device 1 specified
16:02:51:WU01:FS00:0x22:Attempting to create CUDA context:
16:02:51:WU01:FS00:0x22:  Configuring platform CUDA
16:02:51:WU01:FS00:0x22:Failed to create CUDA context:
16:02:51:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:02:51:WU01:FS00:0x22:Attempting to create OpenCL context:
16:02:51:WU01:FS00:0x22:  Configuring platform OpenCL
16:02:51:WU01:FS00:0x22:Failed to create OpenCL context:
16:02:51:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:02:51:WU01:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:02:51:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
16:02:51:WU01:FS00:0x22:Saving result file science.log
16:02:51:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:02:52:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:02:52:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:18003 run:0 clone:425 gen:58 core:0x22 unit:0x000001a90000003a0000465300000000
16:02:52:WU01:FS00:Uploading 11.50KiB to 34.72.228.44
16:02:52:WU01:FS00:Connecting to 34.72.228.44:8080
16:02:52:WU01:FS00:Upload complete
16:02:52:WU01:FS00:Server responded WORK_ACK (400)
16:02:52:WU01:FS00:Cleaning up
16:02:52:WU02:FS00:Connecting to assign1.foldingathome.org:80
16:02:53:WU02:FS00:Assigned to work server 34.72.228.44
16:02:53:WU02:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 34.72.228.44
16:02:53:WU02:FS00:Connecting to 34.72.228.44:8080
16:02:53:WU02:FS00:Downloading 3.92MiB
16:02:53:WU02:FS00:Download complete
16:02:53:WU02:FS00:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:18014 run:0 clone:190 gen:48 core:0x22 unit:0x000000be000000300000465e00000000
16:02:53:WU02:FS00:Starting
16:02:53:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 02 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:02:53:WU02:FS00:Started FahCore on PID 5240
16:02:53:WU02:FS00:Core PID:13168
16:02:53:WU02:FS00:FahCore 0x22 started


16:02:54:WU02:FS00:0x22:Project: 18014 (Run 0, Clone 190, Gen 48)
16:02:54:WU02:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:02:54:WU02:FS00:0x22:Reading tar file core.xml
16:02:54:WU02:FS00:0x22:Reading tar file integrator.xml.bz2
16:02:54:WU02:FS00:0x22:Reading tar file state.xml.bz2
16:02:54:WU02:FS00:0x22:Reading tar file system.xml.bz2
16:02:54:WU02:FS00:0x22:Digital signatures verified
16:02:54:WU02:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:02:54:WU02:FS00:0x22:Version 0.0.13
16:02:54:WU02:FS00:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
16:02:54:WU02:FS00:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
16:02:54:WU02:FS00:0x22:  XTC frame write interval: 250000 steps (5%) [20 total]
16:02:54:WU02:FS00:0x22:  Global context and integrator variables write interval: disabled
16:02:54:WU02:FS00:0x22:There are 4 platforms available.
16:02:54:WU02:FS00:0x22:Platform 0: Reference
16:02:54:WU02:FS00:0x22:Platform 1: CPU
16:02:54:WU02:FS00:0x22:Platform 2: OpenCL
16:02:54:WU02:FS00:0x22:  opencl-device 1 specified
16:02:54:WU02:FS00:0x22:Platform 3: CUDA
16:02:54:WU02:FS00:0x22:  cuda-device 1 specified
16:02:59:WU02:FS00:0x22:Attempting to create CUDA context:
16:02:59:WU02:FS00:0x22:  Configuring platform CUDA
16:02:59:WU02:FS00:0x22:Failed to create CUDA context:
16:02:59:WU02:FS00:0x22:Illegal value for DeviceIndex: 1
16:02:59:WU02:FS00:0x22:Attempting to create OpenCL context:
16:02:59:WU02:FS00:0x22:  Configuring platform OpenCL
16:02:59:WU02:FS00:0x22:Failed to create OpenCL context:
16:02:59:WU02:FS00:0x22:Illegal value for DeviceIndex: 1
16:02:59:WU02:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:02:59:WU02:FS00:0x22:Saving result file ..\logfile_01.txt
16:02:59:WU02:FS00:0x22:Saving result file science.log
16:02:59:WU02:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:03:00:WARNING:WU02:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:03:00:WU02:FS00:Sending unit results: id:02 state:SEND error:FAULTY project:18014 run:0 clone:190 gen:48 core:0x22 unit:0x000000be000000300000465e00000000
16:03:00:WU02:FS00:Uploading 11.50KiB to 34.72.228.44
16:03:00:WU02:FS00:Connecting to 34.72.228.44:8080
16:03:00:WU02:FS00:Upload complete
16:03:00:WU02:FS00:Server responded WORK_ACK (400)
16:03:00:WU02:FS00:Cleaning up
16:03:00:WU01:FS00:Connecting to assign1.foldingathome.org:80
16:03:00:WU01:FS00:Assigned to work server 207.53.233.146
16:03:00:WU01:FS00:Requesting new work unit for slot 00: gpu:4:0 GM204 [GeForce GTX 970] 3494 from 207.53.233.146
16:03:00:WU01:FS00:Connecting to 207.53.233.146:8080
16:03:00:WU01:FS00:Downloading 1.87MiB
16:03:01:WU01:FS00:Download complete
16:03:01:WU01:FS00:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:17803 run:0 clone:564 gen:280 core:0x22 unit:0x00000234000001180000458b00000000
16:03:01:WU01:FS00:Starting
16:03:01:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.13/Core_22.fah/FahCore_22.exe -dir 01 -suffix 01 -version 706 -lifeline 6900 -checkpoint 15 -opencl-platform 0 -opencl-device 1 -cuda-device 1 -gpu-vendor nvidia -gpu 1 -gpu-usage 100
16:03:01:WU01:FS00:Started FahCore on PID 12868
16:03:01:WU01:FS00:Core PID:5536
16:03:01:WU01:FS00:FahCore 0x22 started

16:03:01:WU01:FS00:0x22:   Revision: 189320d0
16:03:01:WU01:FS00:0x22:********************************************************************************
16:03:01:WU01:FS00:0x22:Project: 17803 (Run 0, Clone 564, Gen 280)
16:03:01:WU01:FS00:0x22:Unit: 0x00000000000000000000000000000000
16:03:01:WU01:FS00:0x22:Reading tar file core.xml
16:03:01:WU01:FS00:0x22:Reading tar file integrator.xml.bz2
16:03:01:WU01:FS00:0x22:Reading tar file state.xml.bz2
16:03:01:WU01:FS00:0x22:Reading tar file system.xml.bz2
16:03:01:WU01:FS00:0x22:Digital signatures verified
16:03:01:WU01:FS00:0x22:Folding@home GPU Core22 Folding@home Core
16:03:01:WU01:FS00:0x22:Version 0.0.13
16:03:01:WU01:FS00:0x22:  Checkpoint write interval: 250000 steps (5%) [20 total]
16:03:01:WU01:FS00:0x22:  JSON viewer frame write interval: 50000 steps (1%) [100 total]
16:03:01:WU01:FS00:0x22:  XTC frame write interval: 25000 steps (0.5%) [200 total]
16:03:01:WU01:FS00:0x22:  Global context and integrator variables write interval: disabled
16:03:01:WU01:FS00:0x22:There are 4 platforms available.
16:03:01:WU01:FS00:0x22:Platform 0: Reference
16:03:01:WU01:FS00:0x22:Platform 1: CPU
16:03:01:WU01:FS00:0x22:Platform 2: OpenCL
16:03:01:WU01:FS00:0x22:  opencl-device 1 specified
16:03:01:WU01:FS00:0x22:Platform 3: CUDA
16:03:01:WU01:FS00:0x22:  cuda-device 1 specified
16:03:04:WU01:FS00:0x22:Attempting to create CUDA context:
16:03:04:WU01:FS00:0x22:  Configuring platform CUDA
16:03:04:WU01:FS00:0x22:Failed to create CUDA context:
16:03:04:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:03:04:WU01:FS00:0x22:Attempting to create OpenCL context:
16:03:04:WU01:FS00:0x22:  Configuring platform OpenCL
16:03:04:WU01:FS00:0x22:Failed to create OpenCL context:
16:03:04:WU01:FS00:0x22:Illegal value for DeviceIndex: 1
16:03:04:WU01:FS00:0x22:ERROR:125: Failed to create a GPU-enabled OpenMM Context.
16:03:04:WU01:FS00:0x22:Saving result file ..\logfile_01.txt
16:03:04:WU01:FS00:0x22:Saving result file science.log
16:03:04:WU01:FS00:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
16:03:04:WARNING:WU01:FS00:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
16:03:04:WU01:FS00:Sending unit results: id:01 state:SEND error:FAULTY project:17803 run:0 clone:564 gen:280 core:0x22 unit:0x00000234000001180000458b00000000
16:03:04:WU01:FS00:Uploading 11.50KiB to 207.53.233.146
16:03:04:WU01:FS00:Connecting to 207.53.233.146:8080
16:03:05:WU01:FS00:Upload complete
16:03:05:WU01:FS00:Server responded WORK_ACK (400)
16:03:05:WU01:FS00:Cleaning up
Image
Ryzen 5900x 12T - RTX 4070 TI
Top
Kebast
Posts: 386
Joined: Thu Aug 06, 2015 5:21 pm

Re: Project: 18202 (Run 1345, Clone 3, Gen 9) dumped

Post by Kebast »

Hmm, mine might be a configuration issue. Somehow I have more GPU slots now than I have GPUs. I've removed the offending ones and will try again.
Image
Ryzen 5900x 12T - RTX 4070 TI
Top
Post Reply

Return to “Issues with a specific WU”