Error 997: Overlapped I/O operation is in progress

[Rune]

Hi there, long time viewer first time caller. I've got a few dozen machines up and running, but one of them is having an issue I have to say I've never seen before. System config is as follows:

AMD Threadripper 3990x
Crucial Ballistix 32Gb 3600c16
Gigabyte TRX40 Aorus Xtreme
EVGA 2080ti Black x3
EVGA 1600t2
Windows 10 pro 1909

So, the problem is actually two-fold, though I am thinking they are probably somehow related. Problem one being... All three GPUs in this machine are folding at roughly a quarter what they should be PPDwise. (Picture 1) I've isolated heat and power limits being an issue, yet the cards are still greatly underperforming. Estimates are showing 500kPPD each whereas they should be in the neighborhood of 2mPPD+



Problem two, I think being the more unusual and fun to diagnose one, is the error I am getting in the title. (Picture 2) It doesn't appear to be tied to any particular slot. Reducing the number of cores FAH uses slightly reduces the errors, but they do not go away. When the error occurs, all of FAHControl freezes and resets. It also appears to temporarily pause folding for a few seconds. I've tried disabling SMT, setting core affinities to one NUMA node, and even underclocking the chip. None of these are the solution.



Any ideas out there for either issue? Thanks.

[Joe_H]

Posted by another person here in the last couple weeks, part of the solution is updating your motherboard BIOS. The 997 error is not specific to folding, it comes from the Threadripper being used.

[Rune]

Posted by another person here in the last couple weeks, part of the solution is updating your motherboard BIOS. The 997 error is not specific to folding, it comes from the Threadripper being used.

Interesting. I am currently already running the latest f4d bios though.

[PantherX]

Welcome to the F@H Forum Rune,

Could you please post the first 100 lines of the log file so we can see what the client has detected and how it's setup? Guidance is provided here if needed: viewtopic.php?f=24&t=26036

Regarding the error message, I believe that AGESA 1.0.0.5 might have the fix. Your BIOS is running AGESA 1.0.0.3 B which is why the error hasn't gone.

I would suggest that you run a single CPU Slot of 32 CPUs instead of 2 CPU Slots of 16 CPUs.

When it comes to GPU performance, what are the clock frequencies, what PCIe lanes are they running at? You can use GPU-Z to get those details. Also, what happens if you pause everything except 1 GPU slot, including any background tasks? Wait for about 15 to 30 minutes to ensure that the TFP has been updated. Repeat it across the remaining GPUs. Then, test with 2 GPUs folding, and then all three.

[ajm]

I have had that problem since I use a Threadripper (3970X, Win 10 Pro 1909). After a BIOS update, it went better, then worse. In the meantime, my Threadripper died on me (error CPU 07, still in RMA) and I replaced it yesterday. I've become careful... I now run only the GPU on that machine and I carefully follow the log, tailing it in a separate software. I can say that the problem only occurs when Advanced Control (FAHControl, 7.6.13) is running. Absolutely no occurrence of the error 997 when the machine simply is folding. But as soon as I launch FAHControl, all hell break loose, and FAHControl can become totally unserviceable, that is, you haven't enough time between two errors 997 for pausing or configuring a slot. Would be worth investigating, maybe.

EDIT: I also roamed the Event Viewer: no occurrence of that error anywhere.

EDIT: My dealer had sent the 3970X to AMD. AMD now decided to replace it as DOA.

EDIT: The error 997 persists even when no slot is configured (also with the latest bios, from 5/21/20). It really is the mere presence of FAHControl that triggers it. Uninstalled the whole thing on the TRX40 machine.

EDIT: Today, I again ran some tests. I crunched 5 CPU WUs at 32 threads (=half). While drawing some 150 W (peaks at 180) the temperature reached over 60C on the CPU (70C on the package), which is extremely high with only half of the cores involved and considering the cooling at work here. To get an idea, I then ran the Blender Benchmarks, the whole series. With all 64 cores solidly stuck at 100%, between 4 and 4.4 GHz, pulling over 200 W, I never exceeded 51°C, mostly rather 45-46.

[ajm]

An update. The errors also appear when FAHControl is running, but NOT open, if the FAH screensaver is on. I happened to forget to quit FAHControl last night, left it in paused state, and I had installed the Screensaver that day for some other tests. In the morning:

Code: Select all

22:32:36:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
22:33:23:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
22:34:25:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
22:35:31:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
22:35:33:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
22:36:09:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
22:36:19:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
22:36:36:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
22:46:46:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:02:43:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:14:31:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:18:03:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:20:07:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:20:51:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:21:13:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:21:58:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:21:58:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:23:07:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:24:35:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:27:23:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:30:05:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:30:53:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:30:54:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:34:15:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:36:02:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:36:34:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:40:28:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:42:11:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:42:46:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:53:29:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:54:27:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:55:23:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:58:42:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:58:58:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:00:50:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:05:31:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:05:34:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:07:07:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:08:02:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:11:01:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:13:23:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:20:43:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:23:05:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:24:21:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:28:11:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:28:15:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:33:50:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:34:05:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:37:09:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:37:21:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:38:05:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:46:44:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:48:34:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:49:04:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:49:38:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
00:55:25:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:18:49:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:19:59:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:25:46:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:26:01:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:28:00:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:29:54:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:30:41:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:31:25:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:31:32:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:32:29:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:35:56:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:37:06:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:37:20:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:38:13:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:38:17:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:38:43:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:39:30:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:42:01:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:42:40:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:42:43:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:43:48:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:49:41:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:49:45:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:50:27:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:50:28:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:51:08:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:51:37:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:54:58:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
01:57:09:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:02:27:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:03:41:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:04:50:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:05:03:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:05:55:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:06:18:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:06:45:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:08:44:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:10:08:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:14:00:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:14:02:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:26:20:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:27:05:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:27:20:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:29:38:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:32:56:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:38:42:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:40:36:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:40:58:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:46:31:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:46:59:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:47:06:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:47:47:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:48:26:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:49:29:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:49:41:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:50:24:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:51:42:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:57:13:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:58:23:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:58:39:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
02:59:07:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:02:32:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:05:11:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:06:06:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:06:51:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:07:59:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:08:14:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:08:36:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:08:39:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:09:59:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:11:05:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:11:26:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:11:33:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:12:49:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:17:45:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:17:46:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:19:42:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:20:19:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:20:42:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:21:34:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:23:14:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:23:35:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:28:25:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:29:48:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:31:53:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:32:00:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:34:32:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:40:57:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:41:43:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:45:25:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:46:37:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:47:10:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:49:47:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:50:17:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:52:49:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:53:34:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:53:55:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:56:09:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:56:54:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:57:30:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:58:03:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:58:31:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:58:59:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
03:59:20:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
04:01:31:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
04:02:35:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
04:04:54:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
04:06:48:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
04:08:31:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
04:10:36:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
04:11:14:ERROR:Receive error: 997: Overlapped I/O operation is in progress.

[ajm]

The 997 error closes the telnet connection, and this even while staying idle and while FAHControl is not open. Tried it like this:

While tailing log.txt in a text editor
Launched FAH with a double-click on the desktop icon -> FAHControl launches in paused state and stays in the taskbar - it this state, it doesn't generate 997 errors.
Launched a CLI with telnet
open localhost 36330 -> Welcome to the FAHClient command server. >
then -> Connection to host lost. Press any key to continue...
Text editor: 16:23:04:ERROR:Receive error: 997: Overlapped I/O operation is in progress.

Tried 5 times in a row. Sometimes it takes a little while (but less than one minute) until the error closes the connection, sometimes a second or two.

Code: Select all

*********************** Log Started 2020-07-05T16:13:10Z ***********************
16:13:10:Trying to access database...
16:13:10:Successfully acquired database lock
16:13:10:Read GPUs.txt
16:13:10:Enabled folding slot 01: PAUSED gpu:0:TU104 [GeForce RTX 2080 SUPER] (by user)
16:13:10:****************************** FAHClient ******************************
16:13:10:        Version: 7.6.13
16:13:10:         Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:13:10:      Copyright: 2020 foldingathome.org
16:13:10:       Homepage: https://foldingathome.org/
16:13:10:           Date: Apr 27 2020
16:13:10:           Time: 21:21:01
16:13:10:       Revision: 5a652817f46116b6e135503af97f18e094414e3b
16:13:10:         Branch: master
16:13:10:       Compiler: Visual C++ 2008
16:13:10:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:13:10:       Platform: win32 10
16:13:10:           Bits: 32
16:13:10:           Mode: Release
16:13:10:           Args: --open-web-control
16:13:10:         Config: C:\Users\ajmch\AppData\Roaming\FAHClient\config.xml
16:13:10:******************************** CBang ********************************
16:13:10:           Date: Apr 24 2020
16:13:10:           Time: 17:07:55
16:13:10:       Revision: ea081a3b3b0f4a37c4d0440b4f1bc184197c7797
16:13:10:         Branch: master
16:13:10:       Compiler: Visual C++ 2008
16:13:10:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:13:10:       Platform: win32 10
16:13:10:           Bits: 32
16:13:10:           Mode: Release
16:13:10:******************************* System ********************************
16:13:10:            CPU: AMD Ryzen Threadripper 3970X 32-Core Processor
16:13:10:         CPU ID: AuthenticAMD Family 23 Model 49 Stepping 0
16:13:10:           CPUs: 32
16:13:10:         Memory: 127.88GiB
16:13:10:    Free Memory: 49.78GiB
16:13:10:        Threads: WINDOWS_THREADS
16:13:10:     OS Version: 6.2
16:13:10:    Has Battery: true
16:13:10:     On Battery: false
16:13:10:     UTC Offset: 2
16:13:10:            PID: 39604
16:13:10:            CWD: C:\Users\ajmch\AppData\Roaming\FAHClient
16:13:10:  Win32 Service: false
16:13:10:             OS: Windows 10 Enterprise
16:13:10:        OS Arch: AMD64
16:13:10:           GPUs: 1
16:13:10:          GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:8 TU104 [GeForce RTX 2080 SUPER]
16:13:10:  CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:7.5 Driver:10.2
16:13:10:OpenCL Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:442.92
16:13:10:******************************* libFAH ********************************
16:13:10:           Date: Apr 15 2020
16:13:10:           Time: 14:53:14
16:13:10:       Revision: 216968bc7025029c841ed6e36e81a03a316890d3
16:13:10:         Branch: master
16:13:10:       Compiler: Visual C++ 2008
16:13:10:        Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
16:13:10:       Platform: win32 10
16:13:10:           Bits: 32
16:13:10:           Mode: Release
16:13:10:***********************************************************************
16:13:11:<config>
16:13:11:  <!-- Network -->
16:13:11:  <proxy v=':8080'/>
16:13:11:
16:13:11:  <!-- Slot Control -->
16:13:11:  <power v='FULL'/>
16:13:11:
16:13:11:  <!-- User Information -->
16:13:11:  <passkey v='*****'/>
16:13:11:  <team v='182116'/>
16:13:11:  <user v='ajm'/>
16:13:11:
16:13:11:  <!-- Folding Slots -->
16:13:11:  <slot id='1' type='GPU'>
16:13:11:    <paused v='true'/>
16:13:11:  </slot>
16:13:11:</config>
16:13:12:3:127.0.0.1:New Web session
16:14:18:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
16:16:43:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
16:18:51:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
16:22:22:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
16:23:04:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
16:30:21:Clean exit

EDIT: Done the same almost in the same time on another machine (X299). There, the telnet connection is still open and functional.

[ajm]

Another small test today.
These 997 errors don't appear while just tailing the log with a text editor, as FAHControl is not open, but they do if the machine is monitored with FAH Remote Access function from another computer on the network, albeit not as frequently as in the cases above. Here, I've observed up to almost 8 minutes between two occurrences.

Code: Select all

12:32:22:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
12:32:25:WU00:FS01:0x22:Completed 540000 out of 1000000 steps (54%)
12:32:38:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
12:33:23:WU00:FS01:0x22:Completed 550000 out of 1000000 steps (55%)
12:34:22:WU00:FS01:0x22:Completed 560000 out of 1000000 steps (56%)
12:35:23:WU00:FS01:0x22:Completed 570000 out of 1000000 steps (57%)
12:36:27:WU00:FS01:0x22:Completed 580000 out of 1000000 steps (58%)
12:37:30:WU00:FS01:0x22:Completed 590000 out of 1000000 steps (59%)
12:38:30:WU00:FS01:0x22:Completed 600000 out of 1000000 steps (60%)
12:39:32:WU00:FS01:0x22:Completed 610000 out of 1000000 steps (61%)
12:40:23:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
12:40:32:WU00:FS01:0x22:Completed 620000 out of 1000000 steps (62%)
12:41:33:WU00:FS01:0x22:Completed 630000 out of 1000000 steps (63%)
12:42:35:WU00:FS01:0x22:Completed 640000 out of 1000000 steps (64%)
12:43:41:WU00:FS01:0x22:Completed 650000 out of 1000000 steps (65%)
12:44:36:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
12:44:52:WU00:FS01:0x22:Completed 660000 out of 1000000 steps (66%)
12:45:26:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
12:45:54:FS01:Finishing
12:45:57:WU00:FS01:0x22:Completed 670000 out of 1000000 steps (67%)

[ajm]

As I wrote elsewhere, it did only for a few minutes. I have updated the bios a few times since then, with the same results. And there's no new update right now.

EDIT: For the record, my bios is now 1101 (https://www.asus.com/Motherboards/ROG-Z ... _Download/)

[ajm]

Installed 7.6.15 beta on this machine. Simple update with paused slots, no problem.
After that, the system was able to fold with FAH open for 6 whole minutes without a 997 error.
But now back to "normal," with 997 errors popping at shorter and shorter intervals.

[foldy]

Really don't know if it is related but this forum tells about a solution on installation usecase.
https://community.spiceworks.com/topic/ ... t-progress

Another user has this issue with 3900x and bios update is only recommendation.
https://forums.tomshardware.com/threads ... s.3593243/

Nevertheless the error seems only cosmetic since the FAHcores continue folding without an issue.

[ajm]

There are countless occurrences of this error, in several different contexts. But no solution yet for FAH - a new Bios makes only a difference for a while.
It may be only cosmetic, but if I let Advanced Control open, the errors stack up to the point where it becomes unserviceable and I have to close it anyway.
And, there's an admittedly weak but not non-existant possibility that it's not just cosmetic - after all, I lost a processor while folding on it with this error.

[bruce]

Does this error happen when a WU is being uploaded to a server and FAHClient decides it's time to retry a (supposedly failed) upload? I've never seen the error, myself.

[ajm]

No, it happens all the time, seemingly at random. For example, just a few minutes back, as I forgot to close Advanced Control immediately after finishing the slots:

Code: Select all

23:23:10:FS01:Finishing
23:23:10:FS00:Finishing
23:23:10:FS02:Unpaused
23:23:10:FS02:Finishing
23:23:30:Removing old file 'configs/config-20200805-125948.xml'
23:23:30:Saving configuration to config.xml
23:23:30:<config>
23:23:30:  <!-- Folding Slot Configuration -->
23:23:30:  <client-type v='advanced'/>
23:23:30:
23:23:30:  <!-- HTTP Server -->
23:23:30:  <allow v='127.0.0.1 192.168.1.126-192.168.1.127'/>
23:23:30:
23:23:30:  <!-- Network -->
23:23:30:  <proxy v=':8080'/>
23:23:30:
23:23:30:  <!-- Remote Command Server -->
23:23:30:  <command-allow-no-pass v='127.0.0.1 192.168.1.126-192.168.1.127'/>
23:23:30:
23:23:30:  <!-- Slot Control -->
23:23:30:  <power v='FULL'/>
23:23:30:
23:23:30:  <!-- User Information -->
23:23:30:  <passkey v='*****'/>
23:23:30:  <team v='182116'/>
23:23:30:  <user v='ajm'/>
23:23:30:
23:23:30:  <!-- Folding Slots -->
23:23:30:  <slot id='1' type='GPU'>
23:23:30:    <pause-on-start v='true'/>
23:23:30:    <pci-bus v='78'/>
23:23:30:    <pci-slot v='0'/>
23:23:30:  </slot>
23:23:30:  <slot id='0' type='GPU'>
23:23:30:    <pci-bus v='1'/>
23:23:30:    <pci-slot v='0'/>
23:23:30:  </slot>
23:23:30:  <slot id='2' type='CPU'>
23:23:30:    <cpus v='32'/>
23:23:30:  </slot>
23:23:30:</config>
23:23:42:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:23:43:WU00:FS01:0x22:Completed 115000 out of 500000 steps (23%)
23:24:17:WU00:FS01:0x22:Completed 120000 out of 500000 steps (24%)
23:24:32:WU01:FS00:0x22:Completed 250000 out of 1000000 steps (25%)
23:24:51:WU00:FS01:0x22:Completed 125000 out of 500000 steps (25%)
23:25:26:WU00:FS01:0x22:Completed 130000 out of 500000 steps (26%)
23:25:59:WU01:FS00:0x22:Completed 260000 out of 1000000 steps (26%)
23:26:00:WU00:FS01:0x22:Completed 135000 out of 500000 steps (27%)
23:26:34:WU00:FS01:0x22:Completed 140000 out of 500000 steps (28%)
23:27:08:WU00:FS01:0x22:Completed 145000 out of 500000 steps (29%)
23:27:22:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:27:25:WU01:FS00:0x22:Completed 270000 out of 1000000 steps (27%)
23:27:28:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:27:42:WU00:FS01:0x22:Completed 150000 out of 500000 steps (30%)
23:28:17:WU00:FS01:0x22:Completed 155000 out of 500000 steps (31%)
23:28:42:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:28:50:WU00:FS01:0x22:Completed 160000 out of 500000 steps (32%)
23:28:51:WU01:FS00:0x22:Completed 280000 out of 1000000 steps (28%)
23:29:05:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:29:24:WU00:FS01:0x22:Completed 165000 out of 500000 steps (33%)
23:29:52:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:29:55:ERROR:Receive error: 997: Overlapped I/O operation is in progress.
23:29:58:WU00:FS01:0x22:Completed 170000 out of 500000 steps (34%)
23:30:18:WU01:FS00:0x22:Completed 290000 out of 1000000 steps (29%)

[ajm]

It started as soon as I built the TRX40 system. At the time, I wanted to have Advanced Control open on that machine, in order to keep a look at the other rigs via Remote Access. At the beginning, the first hours, it's just a few glitches here and there, and the errors in the logs. Albeit you don't even take a look at the log at first, you just see that Advanced Control is sometimes unresponsive or greyed out. After a few days of 24/7 folding, it becomes almost constant and Advanced Control is totally blocked, like dead. If you make some important change, like reinstalling FAH or updating the bios, or simply rebooting the machine after a Windows update, actually, the problem starts from scratch: a few minutes OK, then a glitch, then two, and so on.

Then my 3970X died on me. I wasn't sure at first - could have been the MB, or something else. My dealer sent the CPU to AMD and they replaced it, which should mean something, but they refused to make any comment. So in order to be prepared for the worst, I replaced everything: I bought a second 3970X, a second MB, new RAM, a new PSU, new cables, new GPUs. I only kept the case and the loop (external, with its own power adapter, so not powered via the PSU). No change.

I now use my main machine as an isolated rig and I check the other ones with an RDP connection to one of the other machines, and from there with Advanced Control and Remote Access. On my main machine, I use Advanced Control only when necessary, tailing the log in a text editor, and I fold only on GPUs.