Folding Forum

Posted: **Mon Aug 03, 2020 4:51 pm**

Your first main question is something for Bruce and the dev team. I prefer to focus on short-term solutions. And the point still is a bit moot: there always have been problems with OpenCL, so we don't know which present issues really are punctual, linked to some hypothetical event a few days ago, and which ones are just business as usual. In order to find that out, all concerned users should publish their logs, as some have done (thank you!). And the dev team can try and find some clues, some commun elements. Good luck!

The problem does seem to revolve around OpenCL, so it is not really surprising that it manifests itself on several operating systems. It takes different forms as Windows and Linux (or Linux distros) are handling this installation differently.

Posted: **Mon Aug 03, 2020 5:43 pm**

goodyca wrote:
Endgame124 wrote: Softlink was already there, so I guess mesa wasn't the problem. I had previously set openCL index (it wouldn't detect automatically), and tried again, but that didn't work either. Worst part is, this was working fine for quite some time, so I guess I go without folding for now.
I have gpu-index set to -1 and opencl-index set to 0

Be sure to restart the FAHClient.service after making changes.

To further clarify. I am using Nvidia GPU's with the latest drivers from RPM Fusion for Nvidia and Nvidia-cuda. When I removed mesa-libOpenCL, I used dnf, and it also removed the dependency for mesa-libOpenCL , libclc

Posted: **Mon Aug 03, 2020 5:48 pm**

Jupis_folder wrote:@ajm: These threads of OpenCL dev package do not cover the main questions:
1. What is the difference in folding on GPU after 1. Aug 2:00 UTC, which was the time when the problem aroused, after it ran fine for months.
2. Why does the problem occur on Linux and Windows at the same time, while we cannot install the OpenCL dev package on Windows.

100% agreement. The issue is fixed by fixing the local OpenCL install. However this is not the root cause. Having Linux and Windows machines with AMD and Nvidia GPUs fail at the same point in time is not coincidence (and not any auto updates etc.).

Let your dev team figure this out, my best guess is still they started using other OpenCL functions than they used before, from this point in time.

Best regards,
TurboAsterix

Posted: **Mon Aug 03, 2020 6:47 pm**

OK, if we are guessing then my best guess might be that the AS logic changed to ensure future assignments fail less - possible that past failures linked with incomplete/incompatible driver installations that were not being caught by AS to date - but hey my guesses are normally way off

Posted: **Mon Aug 03, 2020 9:00 pm**

Windows 7 Pro 64 bits here, running NVIDIA RTX2070S (uploaded a log earlier in this thread).

@bruce: My setup was running at 2000 k PPD, so no doubt the RTX2070S was running (not mentioning heat generated and electric power ...). Here's my profile:
https://folding.extremeoverclocking.com ... &u=1017089
At the moment, only an AMD RX580 8GB is running on another Windows 7 Pro 64 setup (about 430 k PPD).

FAHCore_22 does not currently use CUDA

CUDA is detected by FAH when it starts, and a parameter in the GUI seems to deal with CUDA => cuda-index.
If FAH doesn't use CUDA then it should be properly documented to avoid headaches; Or did I miss something ?

I've checked again the presence and availability of Open CL 1.2, running:

- GPUPI 3.3.3

- GPU Z 2.33.0

- AIDA 64 GPU benchmark (requires Open CL)

All running without issues.

Also checked the presence of required DLLs in Windows\system32: OpenCL.dll, nvopencl64.dll ...

Despite this, FAH still shows: in the log:

Code: Select all

20:44:53:           OS: Windows 7 Professional
20:44:53:      OS Arch: AMD64
20:44:53:         GPUs: 1
20:44:53:        GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:8 TU104 [GeForce RTX 2070 SUPER] 8218
20:44:53:CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:7.5 Driver:11.0
20:44:53:       [b]OpenCL: Not detected: clGetDeviceIDs() returned -1[/b]

Reinstalled a fresh FAH, clearing FAH data, tried default setup, tweaked the gpu-index and opencl-index (and cuda-index for testing), as this is what I had to do the first time to have the RTX running; this worked for months; this time, no luck.
Quite sorry to have an RTX idling ...

Posted: **Mon Aug 03, 2020 10:27 pm**

Re: 200GE That's what I understood. I just added superfluous information in case you were curious..

Posted: **Mon Aug 03, 2020 10:38 pm**

foldingfanmucde wrote: Addtional info, that now comes to mind and may be helpful:
When provisioning GPU slots (back when things were still working) I was never able to use the default "-1" value for the parameter opencl-index,
instead I would have to set opencl-index = 0 for the first GPU and opencl-index = 1 for the second.
If I did not do it that way, then I would always get an error message stating that openCL is not installed.

Any other ideas?

Many thanks.

Update: to my knowledge there is not more than one openCL version on my Win10 system - not sure that is even possible within windows. Ist there a way for me to verify that?
Update 2: going by the low scores in the ranking lists in the last 12 hours, this issue (or maybe related issue) seems to be affecting quite some folders.

Technically speaking, you can install more than one version of OpenCL but apparently FAH has never been able to support that configuration. There have been lots of errors reported to the WS even though you were not clearly notified.

As of Friday night, systems with more than one were blacklisted, forcing you to fix the problem, one way or another.

Posted: **Mon Aug 03, 2020 11:43 pm**

> Technically speaking, you can install more than one version of OpenCL but apparently FAH has never been able to support that configuration. There have been lots of errors reported to the WS even though you were not clearly notified.

To clarify: We *should* be able to support this, but due to some client programming errors in detection, the client may not end up reporting a valid OpenCL platform is available.

Posted: **Tue Aug 04, 2020 12:00 am**

Hi all,
I've found a solution that worked for me (affected system configuration: AMD Ryzen 9 3900X / 64GB RAM / 2x NVIDIA RTX 2080 Ti / Win10 64 Pro / F@H 7.6.13 )
The steps taken to restore folding functionality, which ultimately worked for me - but I'm still keeping my fingers crossed at this stage - were:

0. All software packages and tools used in later steps were downloaded as preparation tasks. All steps were done with administrator priviledges.
1. Go offline (disconnect from Internet)
2. Stop folding, remove all configured folding slots, Exit the Folding@Home Client Advanced Control Panel (or web client), Quit the FAHClient (right click in task bar)
3. Reboot
4. Uninstall F@H 7.6.13 using Windows Control Panel standard software uninstall. Select the check box to remove "data".
5. Reboot
6. Uninstall NVIDIA Drivers using Windows Control Panel standard software uninstall
7. [this may very well be optional] Uninstall other related NVIDIA components including drivers for PhysX, HD-Audio, USB-C, also rebooting each time as required by uninstaller
(the driver version that was uninstalled was = Game Ready Driver v451.67 DCH version)
8. Reboot
9. Execute the DDU cleaner tool (=Display Driver Uninstall) a FIRST time, selecting the NVIDIA GPU
Use the recommended settings (I did not go into safe/debug mode). Activate the option within the settings to disable automatic driver updates by Windows.
(Version used = v18.0.2.8.exe). Downloaded from here: https://www.guru3d.com/files-details/di ... nload.html
The tool will force a reboot.
10. Execute the DDU cleaner tool a SECOND time (as recommended on website), with the same settings as in the previous step above
The tool will force a reboot.
11. Reinstall NVIDIA Drivers and the related NVIDIA components including drivers for PhysX, HD-Audio, USB-C, also rebooting each time as required by uninstaller.
(the driver version installed was = Game Ready Driver v451.67 DCH version, downloaded from here: https://www.nvidia.com/Download/Find.aspx?lang=en-us#)
12. Reboot
13. Reinstall F@H 7.6.13 (I used the standard install paths)
14. Start the the FAHClient (admin rights). Check the log file. I still got the message " OpenCL: Not detected: clGetDeviceIDs() returned -1 " (refer to attached logfile)
15. The CPU and the 2 GPU slots (using standard self discovered slot configuration after fresh reinstall) immediately requested work units from valid work servers and were very quickly assigned new work units (no longer being requested from the not reachable IP address "192.0.2.1" (dummy work server IP in case of error) )
16. The CPU started working on the WUs immediately.
17. The GPUs did not start working on the assigned WUs. Those slots stayed in state "ready", even though work was clearly assigned and folding was not paused.
18. Paused the 2 GPU slots.
19. Unconfigured (removed) the 2 GPU slots (the assigned work units were dumped).
20. Reconfigured the 2 GPU slots with opencl-index = 0 for the first GPU and opencl-index = 1 for the second GPU (the default value of -1 does not work for reasons unknown to me).
21. Unpaused the 2 GPUs.
22. The 2 GPUs requested new work units, quickly received WUs and immediately started working on them.
23.

- Fingers crossed this does not happen again.

Full logfile from the start after fresh reinstall of F@H:

Code: Select all

22:24:23:Trying to access database...
22:24:23:Successfully acquired database lock
22:24:23:Downloading GPUs.txt from assign1.foldingathome.org:80
22:24:23:Connecting to assign1.foldingathome.org:80
22:24:24:Read GPUs.txt
22:24:24:Enabled folding slot 00: PAUSED cpu:21 (not configured)
22:24:24:Enabled folding slot 01: PAUSED gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 (not configured)
22:24:24:Enabled folding slot 02: PAUSED gpu:1:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 (not configured)
22:24:24:****************************** FAHClient ******************************
22:24:24:      Version: 7.6.13
22:24:24:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:24:24:    Copyright: 2020 foldingathome.org
22:24:24:     Homepage: https://foldingathome.org/
22:24:24:         Date: Apr 27 2020
22:24:24:         Time: 21:21:01
22:24:24:     Revision: 5a652817f46116b6e135503af97f18e094414e3b
22:24:24:       Branch: master
22:24:24:     Compiler: Visual C++ 2008
22:24:24:      Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:24:24:     Platform: win32 10
22:24:24:         Bits: 32
22:24:24:         Mode: Release
22:24:24:******************************** CBang ********************************
22:24:24:         Date: Apr 24 2020
22:24:24:         Time: 17:07:55
22:24:24:     Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
22:24:24:       Branch: master
22:24:24:     Compiler: Visual C++ 2008
22:24:24:      Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:24:24:     Platform: win32 10
22:24:24:         Bits: 32
22:24:24:         Mode: Release
22:24:24:******************************* System ********************************
22:24:24:          CPU: AMD Ryzen 9 3900X 12-Core Processor
22:24:24:       CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
22:24:24:         CPUs: 24
22:24:24:       Memory: 63.92GiB
22:24:24:  Free Memory: 60.12GiB
22:24:24:      Threads: WINDOWS_THREADS
22:24:24:   OS Version: 6.2
22:24:24:  Has Battery: false
22:24:24:   On Battery: false
22:24:24:   UTC Offset: 2
22:24:24:          PID: 6892
22:24:24:          CWD: C:\Program Files (x86)\FAHClient
22:24:24:Win32 Service: false
22:24:24:           OS: Windows 10 Enterprise
22:24:24:      OS Arch: AMD64
22:24:24:         GPUs: 2
22:24:24:        GPU 0: Bus:12 Slot:0 Func:0 NVIDIA:8 TU102 [GeForce RTX 2080 Ti Rev. A]
22:24:24:               M 13448
22:24:24:        GPU 1: Bus:11 Slot:0 Func:0 NVIDIA:8 TU102 [GeForce RTX 2080 Ti Rev. A]
22:24:24:               M 13448
22:24:24:CUDA Device 0: Platform:0 Device:0 Bus:11 Slot:0 Compute:7.5 Driver:11.0
22:24:24:CUDA Device 1: Platform:0 Device:1 Bus:12 Slot:0 Compute:7.5 Driver:11.0
22:24:24:       OpenCL: Not detected: clGetDeviceIDs() returned -1
22:24:24:******************************* libFAH ********************************
22:24:24:         Date: Apr 15 2020
22:24:24:         Time: 14:53:14
22:24:24:     Revision: 216968bc7025029c841ed6e36e81a03a316890d3
22:24:24:       Branch: master
22:24:24:     Compiler: Visual C++ 2008
22:24:24:      Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:24:24:     Platform: win32 10
22:24:24:         Bits: 32
22:24:24:         Mode: Release
22:24:24:***********************************************************************
22:24:24:<config>
22:24:24:  <!-- Folding Slots -->
22:24:24:  <slot id='0' type='CPU'/>
22:24:24:  <slot id='1' type='GPU'/>
22:24:24:  <slot id='2' type='GPU'/>
22:24:24:</config>
22:25:24:Saving configuration to config.xml
22:25:24:<config>
22:25:24:  <!-- Folding Slots -->
22:25:24:  <slot id='0' type='CPU'/>
22:25:24:  <slot id='1' type='GPU'/>
22:25:24:  <slot id='2' type='GPU'/>
22:25:24:</config>
22:25:24:Set client configured
22:25:24:WU00:FS00:Connecting to assign1.foldingathome.org:80
22:25:24:WU01:FS01:Connecting to assign1.foldingathome.org:80
22:25:24:WU00:FS00:Connecting to assign1.foldingathome.org:80
22:25:24:WU02:FS02:Connecting to assign1.foldingathome.org:80
22:25:24:WU01:FS01:Connecting to assign1.foldingathome.org:80
22:25:24:WARNING:WU00:FS00:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration
22:25:24:WU00:FS00:Connecting to assign2.foldingathome.org:80
22:25:25:WU02:FS02:Assigned to work server 18.188.125.154
22:25:25:WU02:FS02:Requesting new work unit for slot 02: READY gpu:1:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 18.188.125.154
22:25:25:WU01:FS01:Assigned to work server 18.188.125.154
22:25:25:WU02:FS02:Connecting to 18.188.125.154:8080
22:25:25:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 18.188.125.154
22:25:25:WU00:FS00:Assigned to work server 206.223.170.146
22:25:25:WU01:FS01:Connecting to 18.188.125.154:8080
22:25:25:WU00:FS00:Requesting new work unit for slot 00: READY cpu:21 from 206.223.170.146
22:25:25:WU00:FS00:Connecting to 206.223.170.146:8080
22:25:28:WU02:FS02:Downloading 7.03MiB
22:25:28:WU01:FS01:Downloading 7.03MiB
22:25:28:WU02:FS02:Download complete
22:25:28:WU01:FS01:Download complete
22:25:28:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13420 run:8276 clone:66 gen:0 core:0x22 unit:0x0000000012bc7d9a5f284d740a2d9c11
22:25:28:WU02:FS02:Downloading core from http://cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah
22:25:28:WU02:FS02:Connecting to cores.foldingathome.org:80
22:25:28:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13420 run:8277 clone:66 gen:0 core:0x22 unit:0x0000000012bc7d9a5f284d74a64a36a2
22:25:30:WU02:FS02:FahCore 22: Downloading 4.39MiB
22:25:30:WU02:FS02:FahCore 22: Download complete
22:25:31:WU02:FS02:Valid core signature
22:25:31:WU02:FS02:Unpacked 14.46MiB to cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe
22:25:31:WU02:FS02:Starting
22:25:31:ERROR:WU02:FS02:Failed to start core: OpenCL device matching slot 2 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:25:31:WU01:FS01:Starting
22:25:31:ERROR:WU01:FS01:Failed to start core: OpenCL device matching slot 1 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:25:31:WU02:FS02:Starting
22:25:31:ERROR:WU02:FS02:Failed to start core: OpenCL device matching slot 2 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:25:32:WU01:FS01:Starting
22:25:32:ERROR:WU01:FS01:Failed to start core: OpenCL device matching slot 1 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:25:51:FS00:Paused
22:25:51:FS01:Paused
22:25:51:FS02:Paused
22:25:56:WU00:FS00:Downloading 21.90MiB
22:25:56:WU00:FS00:Download complete
22:25:56:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:14216 run:1119 clone:1 gen:30 core:0xa7 unit:0x00000024cedfaa925ea342b83d028183
22:26:17:Saving configuration to config.xml
22:26:17:<config>
22:26:17:  <!-- Network -->
22:26:17:  <proxy v=':8080'/>
22:26:17:
22:26:17:  <!-- Slot Control -->
22:26:17:  <pause-on-start v='true'/>
22:26:17:
22:26:17:  <!-- User Information -->
22:26:17:  <passkey v='*****'/>
22:26:17:  <team v='TeamRemoved'/>
22:26:17:  <user v='UserRemoved'/>
22:26:17:
22:26:17:  <!-- Folding Slots -->
22:26:17:  <slot id='0' type='CPU'>
22:26:17:    <paused v='true'/>
22:26:17:  </slot>
22:26:17:  <slot id='1' type='GPU'>
22:26:17:    <paused v='true'/>
22:26:17:  </slot>
22:26:17:  <slot id='2' type='GPU'>
22:26:17:    <paused v='true'/>
22:26:17:  </slot>
22:26:17:</config>
22:26:25:Saving configuration to config.xml
22:26:25:<config>
22:26:25:  <!-- Network -->
22:26:25:  <proxy v=':8080'/>
22:26:25:
22:26:25:  <!-- Slot Control -->
22:26:25:  <pause-on-start v='true'/>
22:26:25:
22:26:25:  <!-- User Information -->
22:26:25:  <passkey v='*****'/>
22:26:25:  <team v='TeamRemoved'/>
22:26:25:  <user v='UserRemoved'/>
22:26:25:
22:26:25:  <!-- Folding Slots -->
22:26:25:  <slot id='0' type='CPU'>
22:26:25:    <paused v='true'/>
22:26:25:  </slot>
22:26:25:  <slot id='1' type='GPU'>
22:26:25:    <paused v='true'/>
22:26:25:  </slot>
22:26:25:  <slot id='2' type='GPU'>
22:26:25:    <paused v='true'/>
22:26:25:  </slot>
22:26:25:</config>
22:27:12:FS00:Unpaused
22:27:12:FS01:Unpaused
22:27:12:FS02:Unpaused
22:27:12:WU00:FS00:Downloading core from http://cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah
22:27:12:WU00:FS00:Connecting to cores.foldingathome.org:80
22:27:14:WU00:FS00:FahCore a7: Downloading 6.71MiB
22:27:14:WU01:FS01:Starting
22:27:14:ERROR:WU01:FS01:Failed to start core: OpenCL device matching slot 1 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:27:14:WU02:FS02:Starting
22:27:14:ERROR:WU02:FS02:Failed to start core: OpenCL device matching slot 2 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:27:15:WU00:FS00:FahCore a7: Download complete
22:27:16:WU00:FS00:Valid core signature
22:27:16:WU00:FS00:Unpacked 19.85MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe
22:27:16:WU00:FS00:Unpacked 2.64MiB to cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/libfftw3f-3.dll
22:27:16:WU00:FS00:Starting
22:27:16:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Program Files (x86)\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe" -dir 00 -suffix 01 -version 706 -lifeline 6892 -checkpoint 10 -np 21
22:27:16:WU00:FS00:Started FahCore on PID 904
22:27:16:WU00:FS00:Core PID:1080
22:27:16:WU00:FS00:FahCore 0xa7 started
22:27:16:WU00:FS00:0xa7:*********************** Log Started 2020-08-03T22:27:16Z ***********************
22:27:16:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
22:27:16:WU00:FS00:0xa7:       Type: 0xa7
22:27:16:WU00:FS00:0xa7:       Core: Gromacs
22:27:16:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 904 -checkpoint 10 -np 21
22:27:16:WU00:FS00:0xa7:************************************ CBang *************************************
22:27:16:WU00:FS00:0xa7:       Date: Nov 27 2019
22:27:16:WU00:FS00:0xa7:       Time: 03:40:09
22:27:16:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
22:27:16:WU00:FS00:0xa7:     Branch: master
22:27:16:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
22:27:16:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:27:16:WU00:FS00:0xa7:   Platform: win32 10
22:27:16:WU00:FS00:0xa7:       Bits: 64
22:27:16:WU00:FS00:0xa7:       Mode: Release
22:27:16:WU00:FS00:0xa7:************************************ System ************************************
22:27:16:WU00:FS00:0xa7:        CPU: AMD Ryzen 9 3900X 12-Core Processor
22:27:16:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
22:27:16:WU00:FS00:0xa7:       CPUs: 24
22:27:16:WU00:FS00:0xa7:     Memory: 63.92GiB
22:27:16:WU00:FS00:0xa7:Free Memory: 59.97GiB
22:27:16:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
22:27:16:WU00:FS00:0xa7: OS Version: 6.2
22:27:16:WU00:FS00:0xa7:Has Battery: false
22:27:16:WU00:FS00:0xa7: On Battery: false
22:27:16:WU00:FS00:0xa7: UTC Offset: 2
22:27:16:WU00:FS00:0xa7:        PID: 1080
22:27:16:WU00:FS00:0xa7:        CWD: C:\Program Files (x86)\FAHClient\work
22:27:16:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
22:27:16:WU00:FS00:0xa7:    Version: 0.0.19
22:27:16:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:27:16:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
22:27:16:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
22:27:16:WU00:FS00:0xa7:       Date: Nov 25 2019
22:27:16:WU00:FS00:0xa7:       Time: 17:12:41
22:27:16:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
22:27:16:WU00:FS00:0xa7:     Branch: master
22:27:16:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
22:27:16:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:27:16:WU00:FS00:0xa7:   Platform: win32 10
22:27:16:WU00:FS00:0xa7:       Bits: 64
22:27:16:WU00:FS00:0xa7:       Mode: Release
22:27:16:WU00:FS00:0xa7:************************************ Build *************************************
22:27:16:WU00:FS00:0xa7:       SIMD: avx_256
22:27:16:WU00:FS00:0xa7:********************************************************************************
22:27:16:WU00:FS00:0xa7:Project: 14216 (Run 1119, Clone 1, Gen 30)
22:27:16:WU00:FS00:0xa7:Unit: 0x00000024cedfaa925ea342b83d028183
22:27:16:WU00:FS00:0xa7:Reading tar file core.xml
22:27:16:WU00:FS00:0xa7:Reading tar file frame30.tpr
22:27:16:WU00:FS00:0xa7:Digital signatures verified
22:27:16:WU00:FS00:0xa7:Calling: mdrun -s frame30.tpr -o frame30.trr -x frame30.xtc -cpt 10 -nt 21
22:27:16:WU00:FS00:0xa7:Steps: first=1875000 total=62500
22:27:19:WU00:FS00:0xa7:Completed 1 out of 62500 steps (0%)
22:27:26:Saving configuration to config.xml
22:27:26:<config>
22:27:26:  <!-- Folding Core -->
22:27:26:  <checkpoint v='10'/>
22:27:26:  <core-priority v='low'/>
22:27:26:
22:27:26:  <!-- Folding Slot Configuration -->
22:27:26:  <cause v='COVID_19'/>
22:27:26:
22:27:26:  <!-- Network -->
22:27:26:  <proxy v=':8080'/>
22:27:26:
22:27:26:  <!-- Slot Control -->
22:27:26:  <pause-on-start v='true'/>
22:27:26:
22:27:26:  <!-- User Information -->
22:27:26:  <passkey v='*****'/>
22:27:26:  <team v='TeamRemoved'/>
22:27:26:  <user v='UserRemoved'/>
22:27:26:
22:27:26:  <!-- Folding Slots -->
22:27:26:  <slot id='0' type='CPU'/>
22:27:26:  <slot id='1' type='GPU'/>
22:27:26:  <slot id='2' type='GPU'/>
22:27:26:</config>
22:27:59:WU00:FS00:0xa7:Completed 625 out of 62500 steps (1%)
22:28:32:WU00:FS00:0xa7:Completed 1250 out of 62500 steps (2%)
22:28:51:WU01:FS01:Starting
22:28:51:ERROR:WU01:FS01:Failed to start core: OpenCL device matching slot 1 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:28:51:WU02:FS02:Starting
22:28:51:ERROR:WU02:FS02:Failed to start core: OpenCL device matching slot 2 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:29:04:WU00:FS00:0xa7:Completed 1875 out of 62500 steps (3%)
22:29:36:WU00:FS00:0xa7:Completed 2500 out of 62500 steps (4%)
22:30:09:WU00:FS00:0xa7:Completed 3125 out of 62500 steps (5%)
22:30:42:WU00:FS00:0xa7:Completed 3750 out of 62500 steps (6%)
22:31:15:WU00:FS00:0xa7:Completed 4375 out of 62500 steps (7%)
22:31:29:WU01:FS01:Starting
22:31:29:ERROR:WU01:FS01:Failed to start core: OpenCL device matching slot 1 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:31:29:WU02:FS02:Starting
22:31:29:ERROR:WU02:FS02:Failed to start core: OpenCL device matching slot 2 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually
22:31:47:WU00:FS00:0xa7:Completed 5000 out of 62500 steps (8%)
22:31:57:FS01:Paused
22:31:58:FS02:Paused
22:32:19:WU00:FS00:0xa7:Completed 5625 out of 62500 steps (9%)
22:32:31:Saving configuration to config.xml
22:32:31:<config>
22:32:31:  <!-- Folding Core -->
22:32:31:  <checkpoint v='10'/>
22:32:31:  <core-priority v='low'/>
22:32:31:
22:32:31:  <!-- Folding Slot Configuration -->
22:32:31:  <cause v='COVID_19'/>
22:32:31:
22:32:31:  <!-- Network -->
22:32:31:  <proxy v=':8080'/>
22:32:31:
22:32:31:  <!-- Slot Control -->
22:32:31:  <pause-on-start v='true'/>
22:32:31:
22:32:31:  <!-- User Information -->
22:32:31:  <passkey v='*****'/>
22:32:31:  <team v='TeamRemoved'/>
22:32:31:  <user v='UserRemoved'/>
22:32:31:
22:32:31:  <!-- Folding Slots -->
22:32:31:  <slot id='0' type='CPU'/>
22:32:31:  <slot id='1' type='GPU'>
22:32:31:    <paused v='true'/>
22:32:31:  </slot>
22:32:31:  <slot id='2' type='GPU'>
22:32:31:    <paused v='true'/>
22:32:31:  </slot>
22:32:31:</config>
22:32:51:WU00:FS00:0xa7:Completed 6250 out of 62500 steps (10%)
22:32:58:Saving configuration to config.xml
22:32:58:<config>
22:32:58:  <!-- Folding Core -->
22:32:58:  <checkpoint v='10'/>
22:32:58:  <core-priority v='low'/>
22:32:58:
22:32:58:  <!-- Folding Slot Configuration -->
22:32:58:  <cause v='COVID_19'/>
22:32:58:
22:32:58:  <!-- Network -->
22:32:58:  <proxy v=':8080'/>
22:32:58:
22:32:58:  <!-- Slot Control -->
22:32:58:  <pause-on-start v='true'/>
22:32:58:
22:32:58:  <!-- User Information -->
22:32:58:  <passkey v='*****'/>
22:32:58:  <team v='TeamRemoved'/>
22:32:58:  <user v='UserRemoved'/>
22:32:58:
22:32:58:  <!-- Folding Slots -->
22:32:58:  <slot id='0' type='CPU'/>
22:32:58:</config>
22:32:58:FS00:Shutting core down
22:32:58:WARNING:WU01:Slot ID 1 no longer exists and there are no other matching slots, dumping
22:32:58:WU01:Sending unit results: id:01 state:SEND error:DUMPED project:13420 run:8277 clone:66 gen:0 core:0x22 unit:0x0000000012bc7d9a5f284d74a64a36a2
22:32:58:WU01:Connecting to 18.188.125.154:8080
22:32:58:WARNING:WU02:Slot ID 2 no longer exists and there are no other matching slots, dumping
22:32:58:WU02:Sending unit results: id:02 state:SEND error:DUMPED project:13420 run:8276 clone:66 gen:0 core:0x22 unit:0x0000000012bc7d9a5f284d740a2d9c11
22:32:58:WU02:Connecting to 18.188.125.154:8080
22:32:58:WU00:FS00:0xa7:WARNING:Console control signal 1 on PID 1080
22:32:58:WU00:FS00:0xa7:Exiting, please wait. . .
22:32:58:WU01:Server responded WORK_ACK (400)
22:32:58:WU01:Cleaning up
22:32:59:WU02:Server responded WORK_ACK (400)
22:32:59:WU02:Cleaning up
22:33:02:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
22:33:02:WU00:FS00:Starting
22:33:02:WARNING:WU00:FS00:Changed SMP threads from 21 to 23 this can cause some work units to fail
22:33:02:WARNING:WU00:FS00:AS lowered CPUs from 23 to 21
22:33:02:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Program Files (x86)\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe" -dir 00 -suffix 01 -version 706 -lifeline 6892 -checkpoint 10 -np 21
22:33:02:WU00:FS00:Started FahCore on PID 6388
22:33:02:WU00:FS00:Core PID:2704
22:33:02:WU00:FS00:FahCore 0xa7 started
22:33:03:WU00:FS00:0xa7:*********************** Log Started 2020-08-03T22:33:02Z ***********************
22:33:03:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
22:33:03:WU00:FS00:0xa7:       Type: 0xa7
22:33:03:WU00:FS00:0xa7:       Core: Gromacs
22:33:03:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 6388 -checkpoint 10 -np
22:33:03:WU00:FS00:0xa7:             21
22:33:03:WU00:FS00:0xa7:************************************ CBang *************************************
22:33:03:WU00:FS00:0xa7:       Date: Nov 27 2019
22:33:03:WU00:FS00:0xa7:       Time: 03:40:09
22:33:03:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
22:33:03:WU00:FS00:0xa7:     Branch: master
22:33:03:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
22:33:03:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:33:03:WU00:FS00:0xa7:   Platform: win32 10
22:33:03:WU00:FS00:0xa7:       Bits: 64
22:33:03:WU00:FS00:0xa7:       Mode: Release
22:33:03:WU00:FS00:0xa7:************************************ System ************************************
22:33:03:WU00:FS00:0xa7:        CPU: AMD Ryzen 9 3900X 12-Core Processor
22:33:03:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
22:33:03:WU00:FS00:0xa7:       CPUs: 24
22:33:03:WU00:FS00:0xa7:     Memory: 63.92GiB
22:33:03:WU00:FS00:0xa7:Free Memory: 59.16GiB
22:33:03:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
22:33:03:WU00:FS00:0xa7: OS Version: 6.2
22:33:03:WU00:FS00:0xa7:Has Battery: false
22:33:03:WU00:FS00:0xa7: On Battery: false
22:33:03:WU00:FS00:0xa7: UTC Offset: 2
22:33:03:WU00:FS00:0xa7:        PID: 2704
22:33:03:WU00:FS00:0xa7:        CWD: C:\Program Files (x86)\FAHClient\work
22:33:03:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
22:33:03:WU00:FS00:0xa7:    Version: 0.0.19
22:33:03:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:33:03:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
22:33:03:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
22:33:03:WU00:FS00:0xa7:       Date: Nov 25 2019
22:33:03:WU00:FS00:0xa7:       Time: 17:12:41
22:33:03:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
22:33:03:WU00:FS00:0xa7:     Branch: master
22:33:03:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
22:33:03:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:33:03:WU00:FS00:0xa7:   Platform: win32 10
22:33:03:WU00:FS00:0xa7:       Bits: 64
22:33:03:WU00:FS00:0xa7:       Mode: Release
22:33:03:WU00:FS00:0xa7:************************************ Build *************************************
22:33:03:WU00:FS00:0xa7:       SIMD: avx_256
22:33:03:WU00:FS00:0xa7:********************************************************************************
22:33:03:WU00:FS00:0xa7:Project: 14216 (Run 1119, Clone 1, Gen 30)
22:33:03:WU00:FS00:0xa7:Unit: 0x00000024cedfaa925ea342b83d028183
22:33:03:WU00:FS00:0xa7:Digital signatures verified
22:33:03:WU00:FS00:0xa7:Calling: mdrun -s frame30.tpr -o frame30.trr -x frame30.xtc -cpi state.cpt -cpt 10 -nt 21
22:33:03:WU00:FS00:0xa7:Steps: first=1875000 total=62500
22:33:06:WU00:FS00:0xa7:Completed 6392 out of 62500 steps (10%)
22:33:28:Adding folding slot 01: PAUSED gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 (by user)
22:33:28:Adding folding slot 02: PAUSED gpu:1:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 (by user)
22:33:28:Saving configuration to config.xml
22:33:28:<config>
22:33:28:  <!-- Folding Core -->
22:33:28:  <checkpoint v='10'/>
22:33:28:  <core-priority v='low'/>
22:33:28:
22:33:28:  <!-- Folding Slot Configuration -->
22:33:28:  <cause v='COVID_19'/>
22:33:28:
22:33:28:  <!-- Network -->
22:33:28:  <proxy v=':8080'/>
22:33:28:
22:33:28:  <!-- Slot Control -->
22:33:28:  <pause-on-start v='true'/>
22:33:28:
22:33:28:  <!-- User Information -->
22:33:28:  <passkey v='*****'/>
22:33:28:  <team v='TeamRemoved'/>
22:33:28:  <user v='UserRemoved'/>
22:33:28:
22:33:28:  <!-- Folding Slots -->
22:33:28:  <slot id='0' type='CPU'/>
22:33:28:  <slot id='1' type='GPU'>
22:33:28:    <opencl-index v='0'/>
22:33:28:  </slot>
22:33:28:  <slot id='2' type='GPU'>
22:33:28:    <opencl-index v='1'/>
22:33:28:  </slot>
22:33:28:</config>
22:33:28:FS00:Shutting core down
22:33:28:WU00:FS00:0xa7:WARNING:Console control signal 1 on PID 2704
22:33:28:WU00:FS00:0xa7:Exiting, please wait. . .
22:33:30:WU00:FS00:0xa7:Folding@home Core Shutdown: INTERRUPTED
22:33:31:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
22:33:32:Saving configuration to config.xml
22:33:32:<config>
22:33:32:  <!-- Folding Core -->
22:33:32:  <checkpoint v='10'/>
22:33:32:  <core-priority v='low'/>
22:33:32:
22:33:32:  <!-- Folding Slot Configuration -->
22:33:32:  <cause v='COVID_19'/>
22:33:32:
22:33:32:  <!-- Network -->
22:33:32:  <proxy v=':8080'/>
22:33:32:
22:33:32:  <!-- Slot Control -->
22:33:32:  <pause-on-start v='true'/>
22:33:32:
22:33:32:  <!-- User Information -->
22:33:32:  <passkey v='*****'/>
22:33:32:  <team v='TeamRemoved'/>
22:33:32:  <user v='UserRemoved'/>
22:33:32:
22:33:32:  <!-- Folding Slots -->
22:33:32:  <slot id='0' type='CPU'/>
22:33:32:  <slot id='1' type='GPU'>
22:33:32:    <opencl-index v='0'/>
22:33:32:  </slot>
22:33:32:  <slot id='2' type='GPU'>
22:33:32:    <opencl-index v='1'/>
22:33:32:  </slot>
22:33:32:</config>
22:33:39:FS01:Unpaused
22:33:39:WU01:FS01:Connecting to assign1.foldingathome.org:80
22:33:40:WU01:FS01:Assigned to work server 18.188.125.154
22:33:40:WU01:FS01:Requesting new work unit for slot 01: READY gpu:0:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 18.188.125.154
22:33:40:WU01:FS01:Connecting to 18.188.125.154:8080
22:33:43:WU01:FS01:Downloading 7.02MiB
22:33:43:WU01:FS01:Download complete
22:33:43:WU01:FS01:Received Unit: id:01 state:DOWNLOAD error:NO_ERROR project:13420 run:8219 clone:68 gen:0 core:0x22 unit:0x0000000012bc7d9a5f284d7659ebc0dd
22:33:43:WU01:FS01:Starting
22:33:43:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Program Files (x86)\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe" -dir 01 -suffix 01 -version 706 -lifeline 6892 -checkpoint 10 -gpu-vendor nvidia -opencl-device 0 -cuda-device 1 -gpu 0
22:33:43:WU01:FS01:Started FahCore on PID 10624
22:33:43:WU01:FS01:Core PID:1920
22:33:43:WU01:FS01:FahCore 0x22 started
22:33:44:WU01:FS01:0x22:*********************** Log Started 2020-08-03T22:33:43Z ***********************
22:33:44:WU01:FS01:0x22:*************************** Core22 Folding@home Core ***************************
22:33:44:WU01:FS01:0x22:       Core: Core22
22:33:44:WU01:FS01:0x22:       Type: 0x22
22:33:44:WU01:FS01:0x22:    Version: 0.0.11
22:33:44:WU01:FS01:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:33:44:WU01:FS01:0x22:  Copyright: 2020 foldingathome.org
22:33:44:WU01:FS01:0x22:   Homepage: https://foldingathome.org/
22:33:44:WU01:FS01:0x22:       Date: Jun 26 2020
22:33:44:WU01:FS01:0x22:       Time: 19:49:16
22:33:44:WU01:FS01:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
22:33:44:WU01:FS01:0x22:     Branch: core22-0.0.11
22:33:44:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:33:44:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:33:44:WU01:FS01:0x22:   Platform: win32 10
22:33:44:WU01:FS01:0x22:       Bits: 64
22:33:44:WU01:FS01:0x22:       Mode: Release
22:33:44:WU01:FS01:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
22:33:44:WU01:FS01:0x22:             <peastman@stanford.edu>
22:33:44:WU01:FS01:0x22:       Args: -dir 01 -suffix 01 -version 706 -lifeline 10624 -checkpoint 10
22:33:44:WU01:FS01:0x22:             -gpu-vendor nvidia -opencl-device 0 -cuda-device 1 -gpu 0
22:33:44:WU01:FS01:0x22:************************************ libFAH ************************************
22:33:44:WU01:FS01:0x22:       Date: Jun 26 2020
22:33:44:WU01:FS01:0x22:       Time: 19:47:12
22:33:44:WU01:FS01:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
22:33:44:WU01:FS01:0x22:     Branch: HEAD
22:33:44:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:33:44:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:33:44:WU01:FS01:0x22:   Platform: win32 10
22:33:44:WU01:FS01:0x22:       Bits: 64
22:33:44:WU01:FS01:0x22:       Mode: Release
22:33:44:WU01:FS01:0x22:************************************ CBang *************************************
22:33:44:WU01:FS01:0x22:       Date: Jun 26 2020
22:33:44:WU01:FS01:0x22:       Time: 19:46:11
22:33:44:WU01:FS01:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
22:33:44:WU01:FS01:0x22:     Branch: master
22:33:44:WU01:FS01:0x22:   Compiler: Visual C++ 2015
22:33:44:WU01:FS01:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:33:44:WU01:FS01:0x22:   Platform: win32 10
22:33:44:WU01:FS01:0x22:       Bits: 64
22:33:44:WU01:FS01:0x22:       Mode: Release
22:33:44:WU01:FS01:0x22:************************************ System ************************************
22:33:44:WU01:FS01:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
22:33:44:WU01:FS01:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
22:33:44:WU01:FS01:0x22:       CPUs: 24
22:33:44:WU01:FS01:0x22:     Memory: 63.92GiB
22:33:44:WU01:FS01:0x22:Free Memory: 59.16GiB
22:33:44:WU01:FS01:0x22:    Threads: WINDOWS_THREADS
22:33:44:WU01:FS01:0x22: OS Version: 6.2
22:33:44:WU01:FS01:0x22:Has Battery: false
22:33:44:WU01:FS01:0x22: On Battery: false
22:33:44:WU01:FS01:0x22: UTC Offset: 2
22:33:44:WU01:FS01:0x22:        PID: 1920
22:33:44:WU01:FS01:0x22:        CWD: C:\Program Files (x86)\FAHClient\work
22:33:44:WU01:FS01:0x22:********************************************************************************
22:33:44:WU01:FS01:0x22:Project: 13420 (Run 8219, Clone 68, Gen 0)
22:33:44:WU01:FS01:0x22:Unit: 0x0000000012bc7d9a5f284d7659ebc0dd
22:33:44:WU01:FS01:0x22:Reading tar file core.xml
22:33:44:WU01:FS01:0x22:Reading tar file integrator.xml
22:33:44:WU01:FS01:0x22:Reading tar file state.xml.bz2
22:33:44:WU01:FS01:0x22:Reading tar file system.xml.bz2
22:33:44:WU01:FS01:0x22:Digital signatures verified
22:33:44:WU01:FS01:0x22:Folding@home GPU Core22 Folding@home Core
22:33:44:WU01:FS01:0x22:Version 0.0.11
22:33:44:WU01:FS01:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
22:33:44:WU01:FS01:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
22:33:44:WU01:FS01:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
22:33:44:WU01:FS01:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
22:33:48:FS02:Unpaused
22:33:49:WU02:FS02:Connecting to assign1.foldingathome.org:80
22:33:49:WU02:FS02:Assigned to work server 18.188.125.154
22:33:49:WU02:FS02:Requesting new work unit for slot 02: READY gpu:1:TU102 [GeForce RTX 2080 Ti Rev. A] M 13448 from 18.188.125.154
22:33:49:WU02:FS02:Connecting to 18.188.125.154:8080
22:33:53:WU02:FS02:Downloading 7.03MiB
22:33:53:WU02:FS02:Download complete
22:33:53:WU02:FS02:Received Unit: id:02 state:DOWNLOAD error:NO_ERROR project:13420 run:8236 clone:68 gen:0 core:0x22 unit:0x0000000012bc7d9a5f284d751639643e
22:33:53:WU02:FS02:Starting
22:33:53:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Program Files (x86)\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.11/Core_22.fah/FahCore_22.exe" -dir 02 -suffix 01 -version 706 -lifeline 6892 -checkpoint 10 -gpu-vendor nvidia -opencl-device 1 -cuda-device 0 -gpu 1
22:33:53:WU02:FS02:Started FahCore on PID 8528
22:33:53:WU02:FS02:Core PID:2900
22:33:53:WU02:FS02:FahCore 0x22 started
22:33:53:WU02:FS02:0x22:*********************** Log Started 2020-08-03T22:33:53Z ***********************
22:33:53:WU02:FS02:0x22:*************************** Core22 Folding@home Core ***************************
22:33:53:WU02:FS02:0x22:       Core: Core22
22:33:53:WU02:FS02:0x22:       Type: 0x22
22:33:53:WU02:FS02:0x22:    Version: 0.0.11
22:33:53:WU02:FS02:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:33:53:WU02:FS02:0x22:  Copyright: 2020 foldingathome.org
22:33:53:WU02:FS02:0x22:   Homepage: https://foldingathome.org/
22:33:53:WU02:FS02:0x22:       Date: Jun 26 2020
22:33:53:WU02:FS02:0x22:       Time: 19:49:16
22:33:53:WU02:FS02:0x22:   Revision: 22010df8a4db48db1b35d33e666b64d8ce48689d
22:33:53:WU02:FS02:0x22:     Branch: core22-0.0.11
22:33:53:WU02:FS02:0x22:   Compiler: Visual C++ 2015
22:33:53:WU02:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:33:53:WU02:FS02:0x22:   Platform: win32 10
22:33:53:WU02:FS02:0x22:       Bits: 64
22:33:53:WU02:FS02:0x22:       Mode: Release
22:33:53:WU02:FS02:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
22:33:53:WU02:FS02:0x22:             <peastman@stanford.edu>
22:33:53:WU02:FS02:0x22:       Args: -dir 02 -suffix 01 -version 706 -lifeline 8528 -checkpoint 10
22:33:53:WU02:FS02:0x22:             -gpu-vendor nvidia -opencl-device 1 -cuda-device 0 -gpu 1
22:33:53:WU02:FS02:0x22:************************************ libFAH ************************************
22:33:53:WU02:FS02:0x22:       Date: Jun 26 2020
22:33:53:WU02:FS02:0x22:       Time: 19:47:12
22:33:53:WU02:FS02:0x22:   Revision: 2b383f4f04f38511dff592885d7c0400e72bdf43
22:33:53:WU02:FS02:0x22:     Branch: HEAD
22:33:53:WU02:FS02:0x22:   Compiler: Visual C++ 2015
22:33:53:WU02:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:33:53:WU02:FS02:0x22:   Platform: win32 10
22:33:53:WU02:FS02:0x22:       Bits: 64
22:33:53:WU02:FS02:0x22:       Mode: Release
22:33:53:WU02:FS02:0x22:************************************ CBang *************************************
22:33:53:WU02:FS02:0x22:       Date: Jun 26 2020
22:33:53:WU02:FS02:0x22:       Time: 19:46:11
22:33:53:WU02:FS02:0x22:   Revision: f8529962055b0e7bde23e429f5072ff758089dee
22:33:53:WU02:FS02:0x22:     Branch: master
22:33:53:WU02:FS02:0x22:   Compiler: Visual C++ 2015
22:33:53:WU02:FS02:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Ob3 /Zc:throwingNew /MT
22:33:53:WU02:FS02:0x22:   Platform: win32 10
22:33:53:WU02:FS02:0x22:       Bits: 64
22:33:53:WU02:FS02:0x22:       Mode: Release
22:33:53:WU02:FS02:0x22:************************************ System ************************************
22:33:53:WU02:FS02:0x22:        CPU: AMD Ryzen 9 3900X 12-Core Processor
22:33:53:WU02:FS02:0x22:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
22:33:53:WU02:FS02:0x22:       CPUs: 24
22:33:53:WU02:FS02:0x22:     Memory: 63.92GiB
22:33:53:WU02:FS02:0x22:Free Memory: 59.01GiB
22:33:53:WU02:FS02:0x22:    Threads: WINDOWS_THREADS
22:33:53:WU02:FS02:0x22: OS Version: 6.2
22:33:53:WU02:FS02:0x22:Has Battery: false
22:33:53:WU02:FS02:0x22: On Battery: false
22:33:53:WU02:FS02:0x22: UTC Offset: 2
22:33:53:WU02:FS02:0x22:        PID: 2900
22:33:53:WU02:FS02:0x22:        CWD: C:\Program Files (x86)\FAHClient\work
22:33:53:WU02:FS02:0x22:********************************************************************************
22:33:53:WU02:FS02:0x22:Project: 13420 (Run 8236, Clone 68, Gen 0)
22:33:53:WU02:FS02:0x22:Unit: 0x0000000012bc7d9a5f284d751639643e
22:33:53:WU02:FS02:0x22:Reading tar file core.xml
22:33:53:WU02:FS02:0x22:Reading tar file integrator.xml
22:33:53:WU02:FS02:0x22:Reading tar file state.xml.bz2
22:33:53:WU02:FS02:0x22:Reading tar file system.xml.bz2
22:33:53:WU02:FS02:0x22:Digital signatures verified
22:33:53:WU02:FS02:0x22:Folding@home GPU Core22 Folding@home Core
22:33:53:WU02:FS02:0x22:Version 0.0.11
22:33:53:WU02:FS02:0x22:  Checkpoint write interval: 50000 steps (5%) [20 total]
22:33:53:WU02:FS02:0x22:  JSON viewer frame write interval: 10000 steps (1%) [100 total]
22:33:53:WU02:FS02:0x22:  XTC frame write interval: 250000 steps (25%) [4 total]
22:33:53:WU02:FS02:0x22:  Global context and integrator variables write interval: 25000 steps (2.5%) [40 total]
22:34:00:WU01:FS01:0x22:Completed 0 out of 1000000 steps (0%)
22:34:02:WU00:FS00:Starting
22:34:02:WARNING:WU00:FS00:Changed SMP threads from 23 to 21 this can cause some work units to fail
22:34:02:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Program Files (x86)\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe" -dir 00 -suffix 01 -version 706 -lifeline 6892 -checkpoint 10 -np 21
22:34:02:WU00:FS00:Started FahCore on PID 3264
22:34:02:WU00:FS00:Core PID:9780
22:34:02:WU00:FS00:FahCore 0xa7 started
22:34:03:WU00:FS00:0xa7:*********************** Log Started 2020-08-03T22:34:02Z ***********************
22:34:03:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
22:34:03:WU00:FS00:0xa7:       Type: 0xa7
22:34:03:WU00:FS00:0xa7:       Core: Gromacs
22:34:03:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 3264 -checkpoint 10 -np
22:34:03:WU00:FS00:0xa7:             21
22:34:03:WU00:FS00:0xa7:************************************ CBang *************************************
22:34:03:WU00:FS00:0xa7:       Date: Nov 27 2019
22:34:03:WU00:FS00:0xa7:       Time: 03:40:09
22:34:03:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
22:34:03:WU00:FS00:0xa7:     Branch: master
22:34:03:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
22:34:03:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:34:03:WU00:FS00:0xa7:   Platform: win32 10
22:34:03:WU00:FS00:0xa7:       Bits: 64
22:34:03:WU00:FS00:0xa7:       Mode: Release
22:34:03:WU00:FS00:0xa7:************************************ System ************************************
22:34:03:WU00:FS00:0xa7:        CPU: AMD Ryzen 9 3900X 12-Core Processor
22:34:03:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
22:34:03:WU00:FS00:0xa7:       CPUs: 24
22:34:03:WU00:FS00:0xa7:     Memory: 63.92GiB
22:34:03:WU00:FS00:0xa7:Free Memory: 58.33GiB
22:34:03:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
22:34:03:WU00:FS00:0xa7: OS Version: 6.2
22:34:03:WU00:FS00:0xa7:Has Battery: false
22:34:03:WU00:FS00:0xa7: On Battery: false
22:34:03:WU00:FS00:0xa7: UTC Offset: 2
22:34:03:WU00:FS00:0xa7:        PID: 9780
22:34:03:WU00:FS00:0xa7:        CWD: C:\Program Files (x86)\FAHClient\work
22:34:03:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
22:34:03:WU00:FS00:0xa7:    Version: 0.0.19
22:34:03:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
22:34:03:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
22:34:03:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
22:34:03:WU00:FS00:0xa7:       Date: Nov 25 2019
22:34:03:WU00:FS00:0xa7:       Time: 17:12:41
22:34:03:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
22:34:03:WU00:FS00:0xa7:     Branch: master
22:34:03:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
22:34:03:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
22:34:03:WU00:FS00:0xa7:   Platform: win32 10
22:34:03:WU00:FS00:0xa7:       Bits: 64
22:34:03:WU00:FS00:0xa7:       Mode: Release
22:34:03:WU00:FS00:0xa7:************************************ Build *************************************
22:34:03:WU00:FS00:0xa7:       SIMD: avx_256
22:34:03:WU00:FS00:0xa7:********************************************************************************
22:34:03:WU00:FS00:0xa7:Project: 14216 (Run 1119, Clone 1, Gen 30)
22:34:03:WU00:FS00:0xa7:Unit: 0x00000024cedfaa925ea342b83d028183
22:34:03:WU00:FS00:0xa7:Digital signatures verified
22:34:03:WU00:FS00:0xa7:Calling: mdrun -s frame30.tpr -o frame30.trr -x frame30.xtc -cpi state.cpt -cpt 10 -nt 21
22:34:03:WU00:FS00:0xa7:Steps: first=1875000 total=62500
22:34:07:WU00:FS00:0xa7:Completed 6722 out of 62500 steps (10%)
22:34:08:WU02:FS02:0x22:Completed 0 out of 1000000 steps (0%)
22:34:23:WU00:FS00:0xa7:Completed 6875 out of 62500 steps (11%)
22:35:00:WU00:FS00:0xa7:Completed 7500 out of 62500 steps (12%)
22:35:05:WU01:FS01:0x22:Completed 10000 out of 1000000 steps (1%)
22:35:13:WU02:FS02:0x22:Completed 10000 out of 1000000 steps (1%)
22:35:37:WU00:FS00:0xa7:Completed 8125 out of 62500 steps (13%)
22:36:11:WU01:FS01:0x22:Completed 20000 out of 1000000 steps (2%)
22:36:14:WU00:FS00:0xa7:Completed 8750 out of 62500 steps (14%)
22:36:18:WU02:FS02:0x22:Completed 20000 out of 1000000 steps (2%)
22:36:52:WU00:FS00:0xa7:Completed 9375 out of 62500 steps (15%)

... (chopped a part here to save space, there were only additional lines here showing progress) ...

23:24:02:WU01:FS01:0x22:Completed 440000 out of 1000000 steps (44%)
23:24:12:WU02:FS02:0x22:Completed 460000 out of 1000000 steps (46%)
23:24:14:WU00:FS00:0xa7:Completed 56250 out of 62500 steps (90%)
23:24:52:WU00:FS00:0xa7:Completed 56875 out of 62500 steps (91%)
23:25:12:WU01:FS01:0x22:Completed 450000 out of 1000000 steps (45%)
23:25:20:WU02:FS02:0x22:Completed 470000 out of 1000000 steps (47%)
23:25:29:WU00:FS00:0xa7:Completed 57500 out of 62500 steps (92%)
23:26:07:WU00:FS00:0xa7:Completed 58125 out of 62500 steps (93%)
23:26:24:WU01:FS01:0x22:Completed 460000 out of 1000000 steps (46%)
23:26:27:WU02:FS02:0x22:Completed 480000 out of 1000000 steps (48%)
23:26:44:WU00:FS00:0xa7:Completed 58750 out of 62500 steps (94%)
23:27:07:FS02:Finishing
23:27:22:WU00:FS00:0xa7:Completed 59375 out of 62500 steps (95%)
23:27:24:FS00:Shutting core down
23:27:25:WU00:FS00:0xa7:WARNING:Console control signal 1 on PID 9780
23:27:25:WU00:FS00:0xa7:Exiting, please wait. . .
23:27:25:Saving configuration to config.xml
23:27:25:<config>
23:27:25:  <!-- Folding Core -->
23:27:25:  <checkpoint v='10'/>
23:27:25:  <core-priority v='low'/>
23:27:25:
23:27:25:  <!-- Folding Slot Configuration -->
23:27:25:  <cause v='COVID_19'/>
23:27:25:
23:27:25:  <!-- Network -->
23:27:25:  <proxy v=':8080'/>
23:27:25:
23:27:25:  <!-- Slot Control -->
23:27:25:  <pause-on-start v='true'/>
23:27:25:  <power v='full'/>
23:27:25:
23:27:25:  <!-- User Information -->
23:27:25:  <passkey v='*****'/>
23:27:25:  <team v='TeamRemoved'/>
23:27:25:  <user v='UserRemoved'/>
23:27:25:
23:27:25:  <!-- Folding Slots -->
23:27:25:  <slot id='0' type='CPU'/>
23:27:25:  <slot id='1' type='GPU'>
23:27:25:    <opencl-index v='0'/>
23:27:25:  </slot>
23:27:25:  <slot id='2' type='GPU'>
23:27:25:    <opencl-index v='1'/>
23:27:25:  </slot>
23:27:25:</config>
23:27:28:WU00:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
23:27:28:WU00:FS00:Starting
23:27:28:WARNING:WU00:FS00:Changed SMP threads from 21 to 22 this can cause some work units to fail
23:27:28:WARNING:WU00:FS00:AS lowered CPUs from 22 to 21
23:27:28:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" "C:\Program Files (x86)\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe" -dir 00 -suffix 01 -version 706 -lifeline 6892 -checkpoint 10 -np 21
23:27:28:WU00:FS00:Started FahCore on PID 4388
23:27:28:WU00:FS00:Core PID:5264
23:27:28:WU00:FS00:FahCore 0xa7 started
23:27:29:WU00:FS00:0xa7:*********************** Log Started 2020-08-03T23:27:28Z ***********************
23:27:29:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
23:27:29:WU00:FS00:0xa7:       Type: 0xa7
23:27:29:WU00:FS00:0xa7:       Core: Gromacs
23:27:29:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 4388 -checkpoint 10 -np
23:27:29:WU00:FS00:0xa7:             21
23:27:29:WU00:FS00:0xa7:************************************ CBang *************************************
23:27:29:WU00:FS00:0xa7:       Date: Nov 27 2019
23:27:29:WU00:FS00:0xa7:       Time: 03:40:09
23:27:29:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
23:27:29:WU00:FS00:0xa7:     Branch: master
23:27:29:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
23:27:29:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
23:27:29:WU00:FS00:0xa7:   Platform: win32 10
23:27:29:WU00:FS00:0xa7:       Bits: 64
23:27:29:WU00:FS00:0xa7:       Mode: Release
23:27:29:WU00:FS00:0xa7:************************************ System ************************************
23:27:29:WU00:FS00:0xa7:        CPU: AMD Ryzen 9 3900X 12-Core Processor
23:27:29:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
23:27:29:WU00:FS00:0xa7:       CPUs: 24
23:27:29:WU00:FS00:0xa7:     Memory: 63.92GiB
23:27:29:WU00:FS00:0xa7:Free Memory: 57.66GiB
23:27:29:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
23:27:29:WU00:FS00:0xa7: OS Version: 6.2
23:27:29:WU00:FS00:0xa7:Has Battery: false
23:27:29:WU00:FS00:0xa7: On Battery: false
23:27:29:WU00:FS00:0xa7: UTC Offset: 2
23:27:29:WU00:FS00:0xa7:        PID: 5264
23:27:29:WU00:FS00:0xa7:        CWD: C:\Program Files (x86)\FAHClient\work
23:27:29:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
23:27:29:WU00:FS00:0xa7:    Version: 0.0.19
23:27:29:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
23:27:29:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
23:27:29:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
23:27:29:WU00:FS00:0xa7:       Date: Nov 25 2019
23:27:29:WU00:FS00:0xa7:       Time: 17:12:41
23:27:29:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
23:27:29:WU00:FS00:0xa7:     Branch: master
23:27:29:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
23:27:29:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
23:27:29:WU00:FS00:0xa7:   Platform: win32 10
23:27:29:WU00:FS00:0xa7:       Bits: 64
23:27:29:WU00:FS00:0xa7:       Mode: Release
23:27:29:WU00:FS00:0xa7:************************************ Build *************************************
23:27:29:WU00:FS00:0xa7:       SIMD: avx_256
23:27:29:WU00:FS00:0xa7:********************************************************************************
23:27:29:WU00:FS00:0xa7:Project: 14216 (Run 1119, Clone 1, Gen 30)
23:27:29:WU00:FS00:0xa7:Unit: 0x00000024cedfaa925ea342b83d028183
23:27:29:WU00:FS00:0xa7:Digital signatures verified
23:27:29:WU00:FS00:0xa7:Calling: mdrun -s frame30.tpr -o frame30.trr -x frame30.xtc -cpi state.cpt -cpt 10 -nt 21
23:27:29:WU00:FS00:0xa7:Steps: first=1875000 total=62500
23:27:33:WU00:FS00:0xa7:Completed 59422 out of 62500 steps (95%)
23:27:34:WU01:FS01:0x22:Completed 470000 out of 1000000 steps (47%)

The content of the GPUs.txt file is also available, but the text is too large to post here (if that has any relevance, then I can provide that too).
I hope I covered everything fully and comprehensibly. If questions arise, please let me know.

Posted: **Tue Aug 04, 2020 1:07 am**

dnf remove mesa-libOpenCL
(as root) worked on my Fedora 31 AMD box with GeForce GTX 1660 GPU.
(Still says it doesn't find OpenCL)

Posted: **Tue Aug 04, 2020 6:16 am**

bruce wrote: As of Friday night, systems with more than one were blacklisted, forcing you to fix the problem, one way or another.

This was the root cause. Thanks for clarifying this.

Posted: **Tue Aug 04, 2020 9:33 am**

I confirm this is related to the presence of two Open CL DLLs ("OpenCL.dll"); my Windows 7 pro setup with NVIDIA RTX2070S is now running.

Here's how I just proceeded:

- searched through Windows directory all occurrences of "opencl*", using eg. Ultra Search

There are two directories including OpenCL.dll: System32 and SysWOW64:
System32 holds 64 bit DLLs, SysWOW64 holds 32 bit DLLs ... seems weird, but required for compatibility/historical reasons.

So, I renamed "SysWOW64\OpenCL.dll" to "SysWOW64\OpenCL.dll.old" to disable the 32 bit version of the Open CL DLL.

Had this "new" error in FAH:

Code: Select all

09:17:19:ERROR:WU01:FS01:Failed to start core: OpenCL device matching slot 1 not found, make sure the OpenCL driver is installed or try setting 'opencl-index' manually

So, I set the indexes as follow:
- gpu-index = 1
- opencl-index = 1

Getting this in the log:

Code: Select all

09:19:21:  <!-- Folding Slots -->
09:19:21:  <slot id='0' type='CPU'/>
09:19:21:  <slot id='1' type='GPU'>
09:19:21:    <gpu-index v='1'/>
09:19:21:    <opencl-index v='1'/>
09:19:21:  </slot>

The RTX2070S is now running fully under Windows 7 pro 64 bits ... good, but disabling the 32 bit version of Open CL DLL cannot be considered as a "standard" operation, this should be avoided as this may cause issues for some other applications.

Posted: **Tue Aug 04, 2020 11:29 am**

bruce wrote:As of Friday night, systems with more than one were blacklisted, forcing you to fix the problem, one way or another.

Thanks for the info. Shame the community were not informed beforehand, as presumably someone would have been aware of the potential consequences of the change when they made it.
A quick post could have saved a lot of hassle for folk ... hopefully, lesson learnt.

Posted: **Wed Aug 05, 2020 12:18 am**

> Thanks for the info. Shame the community were not informed beforehand, as presumably someone would have been aware of the potential consequences of the change when they made it.
A quick post could have saved a lot of hassle for folk ... hopefully, lesson learnt.

I totally hear you on this! Unfortunately, this was one of those times where (1) the symptom appeared to indicate a different problem from what ultimately caused it, and (2) the consequences of the action taken to minimize damage to the network were not realized until Bruce helped us put the pieces together ~48 hours later. The change went into effect Fri night and we were able to get everyone together as soon as possible on Monday to meet and work through the issue and true causes, but I fully admit our response (and the posting of useful information) could have been better coordinated. We're working on ways to do that (as well as hiring someone to help us coordinate rapid responses to issues like this!).

Thanks again for bearing with us even through issues like this---we really appreciate it!

~ John Chodera // MSKCC

Posted: **Wed Aug 05, 2020 2:23 am**

Dear John and all others in the F@H team,
Thanks for your statement and the support.
Please keep up the great work and ongoing improvements coming and we'll keep the folding machinery up and running out there in the other dimension of the realm.

Regards from Munich.
foldingfanmucde

1. AMD Ryzen 9 3900X / 64GB RAM / 2x NVIDIA RTX 2080 Ti / Win10 64 Pro / F@H 7.6.13 / ~6-7M PPD /
2. Intel Core i7 4790K / 32GB RAM / 1x AMD RX 5700 XT / Win10 64 Pro / F@H 7.6.13 / ~1-1.5M PPD /

Folding Forum

Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?

Re: Waiting for work on 192.0.2.1 ?