155.247.166.219 responding "PLEASE_WAIT"

[itskieran]

Couldn't find a recent post on this, so thought I'd make a report, as it is showing fine on the server stats page.

I've got CPU project 14188 stuck sending, (PRCG 6, 161, 3) with the log showing "PLEASE_WAIT" after uploading.

Here's the log filtered to just the problematic WU in case it's a config issue on my end:

Code: Select all

*********************** Log Started 2021-04-02T07:26:55Z ***********************
07:26:55:WU02:FS00:Starting
07:26:55:WU02:FS00:Running FahCore: "D:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\CENSORED\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 02 -suffix 01 -version 706 -lifeline 1884 -checkpoint 15 -np 6
07:26:57:WU02:FS00:Started FahCore on PID 22852
07:26:57:WU02:FS00:Core PID:22876
07:26:57:WU02:FS00:FahCore 0xa7 started
07:26:58:WU02:FS00:0xa7:*********************** Log Started 2021-04-02T07:26:57Z ***********************
07:26:58:WU02:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
07:26:58:WU02:FS00:0xa7:       Type: 0xa7
07:26:58:WU02:FS00:0xa7:       Core: Gromacs
07:26:58:WU02:FS00:0xa7:       Args: -dir 02 -suffix 01 -version 706 -lifeline 22852 -checkpoint 15 -np
07:26:58:WU02:FS00:0xa7:             6
07:26:58:WU02:FS00:0xa7:************************************ CBang *************************************
07:26:58:WU02:FS00:0xa7:       Date: Nov 27 2019
07:26:58:WU02:FS00:0xa7:       Time: 03:40:09
07:26:58:WU02:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
07:26:58:WU02:FS00:0xa7:     Branch: master
07:26:58:WU02:FS00:0xa7:   Compiler: Visual C++ 2008
07:26:58:WU02:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:26:58:WU02:FS00:0xa7:   Platform: win32 10
07:26:58:WU02:FS00:0xa7:       Bits: 64
07:26:58:WU02:FS00:0xa7:       Mode: Release
07:26:58:WU02:FS00:0xa7:************************************ System ************************************
07:26:58:WU02:FS00:0xa7:        CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
07:26:58:WU02:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 158 Stepping 10
07:26:58:WU02:FS00:0xa7:       CPUs: 12
07:26:58:WU02:FS00:0xa7:     Memory: 15.89GiB
07:26:58:WU02:FS00:0xa7:Free Memory: 10.44GiB
07:26:58:WU02:FS00:0xa7:    Threads: WINDOWS_THREADS
07:26:58:WU02:FS00:0xa7: OS Version: 6.2
07:26:58:WU02:FS00:0xa7:Has Battery: false
07:26:58:WU02:FS00:0xa7: On Battery: false
07:26:58:WU02:FS00:0xa7: UTC Offset: 1
07:26:58:WU02:FS00:0xa7:        PID: 22876
07:26:58:WU02:FS00:0xa7:        CWD: C:\Users\CENSORED\AppData\Roaming\FAHClient\work
07:26:58:WU02:FS00:0xa7:******************************** Build - libFAH ********************************
07:26:58:WU02:FS00:0xa7:    Version: 0.0.19
07:26:58:WU02:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
07:26:58:WU02:FS00:0xa7:  Copyright: 2019 foldingathome.org
07:26:58:WU02:FS00:0xa7:   Homepage: https://foldingathome.org/
07:26:58:WU02:FS00:0xa7:       Date: Nov 25 2019
07:26:58:WU02:FS00:0xa7:       Time: 17:12:41
07:26:58:WU02:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
07:26:58:WU02:FS00:0xa7:     Branch: master
07:26:58:WU02:FS00:0xa7:   Compiler: Visual C++ 2008
07:26:58:WU02:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
07:26:58:WU02:FS00:0xa7:   Platform: win32 10
07:26:58:WU02:FS00:0xa7:       Bits: 64
07:26:58:WU02:FS00:0xa7:       Mode: Release
07:26:58:WU02:FS00:0xa7:************************************ Build *************************************
07:26:58:WU02:FS00:0xa7:       SIMD: avx_256
07:26:58:WU02:FS00:0xa7:********************************************************************************
07:26:58:WU02:FS00:0xa7:Project: 14188 (Run 6, Clone 161, Gen 3)
07:26:58:WU02:FS00:0xa7:Unit: 0x00000000000000000000000000000000
07:26:58:WU02:FS00:0xa7:Digital signatures verified
07:26:58:WU02:FS00:0xa7:Calling: mdrun -s frame3.tpr -o frame3.trr -cpi state.cpt -cpt 15 -nt 6
07:26:58:WU02:FS00:0xa7:Steps: first=3750000 total=1250000
07:27:00:WU02:FS00:0xa7:Completed 1049701 out of 1250000 steps (83%)
07:27:14:WU02:FS00:0xa7:Completed 1050000 out of 1250000 steps (84%)
07:39:37:WU02:FS00:0xa7:Completed 1062500 out of 1250000 steps (85%)
07:49:51:WU02:FS00:0xa7:Completed 1075000 out of 1250000 steps (86%)
07:59:12:WU02:FS00:0xa7:Completed 1087500 out of 1250000 steps (87%)
08:08:24:WU02:FS00:0xa7:Completed 1100000 out of 1250000 steps (88%)
08:17:35:WU02:FS00:0xa7:Completed 1112500 out of 1250000 steps (89%)
08:26:50:WU02:FS00:0xa7:Completed 1125000 out of 1250000 steps (90%)
08:36:20:WU02:FS00:0xa7:Completed 1137500 out of 1250000 steps (91%)
08:45:35:WU02:FS00:0xa7:Completed 1150000 out of 1250000 steps (92%)
08:54:42:WU02:FS00:0xa7:Completed 1162500 out of 1250000 steps (93%)
09:03:51:WU02:FS00:0xa7:Completed 1175000 out of 1250000 steps (94%)
09:13:04:WU02:FS00:0xa7:Completed 1187500 out of 1250000 steps (95%)
09:22:11:WU02:FS00:0xa7:Completed 1200000 out of 1250000 steps (96%)
09:31:21:WU02:FS00:0xa7:Completed 1212500 out of 1250000 steps (97%)
09:40:29:WU02:FS00:0xa7:Completed 1225000 out of 1250000 steps (98%)
09:49:39:WU02:FS00:0xa7:Completed 1237500 out of 1250000 steps (99%)
09:58:48:WU02:FS00:0xa7:Completed 1250000 out of 1250000 steps (100%)
09:58:52:WU02:FS00:0xa7:Saving result file ..\logfile_01.txt
09:58:52:WU02:FS00:0xa7:Saving result file frame3.trr
09:58:53:WU02:FS00:0xa7:Saving result file md.log
09:58:53:WU02:FS00:0xa7:Saving result file science.log
09:58:53:WU02:FS00:0xa7:Saving result file traj_comp.xtc
09:58:53:WU02:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
09:58:53:WU02:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
09:58:53:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
09:58:53:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
09:58:53:WU02:FS00:Connecting to 155.247.166.219:8080
09:58:59:WU02:FS00:Upload 38.04%
09:59:05:WU02:FS00:Upload 81.52%
09:59:07:WU02:FS00:Upload complete
09:59:07:WU02:FS00:Server responded PLEASE_WAIT (464)
09:59:07:WARNING:WU02:FS00:Failed to send results, will try again later
09:59:08:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
09:59:08:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
09:59:08:WU02:FS00:Connecting to 155.247.166.219:8080
09:59:14:WU02:FS00:Upload 41.44%
09:59:20:WU02:FS00:Upload 84.91%
09:59:22:WU02:FS00:Upload complete
09:59:22:WU02:FS00:Server responded PLEASE_WAIT (464)
09:59:22:WARNING:WU02:FS00:Failed to send results, will try again later
10:00:08:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
10:00:08:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
10:00:08:WU02:FS00:Connecting to 155.247.166.219:8080
10:00:14:WU02:FS00:Upload 40.76%
10:00:20:WU02:FS00:Upload 84.23%
10:00:22:WU02:FS00:Upload complete
10:00:22:WU02:FS00:Server responded PLEASE_WAIT (464)
10:00:22:WARNING:WU02:FS00:Failed to send results, will try again later
10:01:45:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
10:01:45:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
10:01:45:WU02:FS00:Connecting to 155.247.166.219:8080
10:01:51:WU02:FS00:Upload 38.72%
10:01:57:WU02:FS00:Upload 82.20%
10:01:59:WU02:FS00:Upload complete
10:01:59:WU02:FS00:Server responded PLEASE_WAIT (464)
10:01:59:WARNING:WU02:FS00:Failed to send results, will try again later
10:04:22:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
10:04:22:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
10:04:22:WU02:FS00:Connecting to 155.247.166.219:8080
10:04:28:WU02:FS00:Upload 38.04%
10:04:34:WU02:FS00:Upload 81.52%
10:04:36:WU02:FS00:Upload complete
10:04:36:WU02:FS00:Server responded PLEASE_WAIT (464)
10:04:36:WARNING:WU02:FS00:Failed to send results, will try again later
10:08:36:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
10:08:36:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
10:08:36:WU02:FS00:Connecting to 155.247.166.219:8080
10:08:42:WU02:FS00:Upload 27.85%
10:08:48:WU02:FS00:Upload 71.33%
10:08:52:WU02:FS00:Upload complete
10:08:52:WU02:FS00:Server responded PLEASE_WAIT (464)
10:08:52:WARNING:WU02:FS00:Failed to send results, will try again later
10:15:28:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
10:15:28:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
10:15:28:WU02:FS00:Connecting to 155.247.166.219:8080
10:15:34:WU02:FS00:Upload 38.72%
10:15:40:WU02:FS00:Upload 82.20%
10:15:42:WU02:FS00:Upload complete
10:15:42:WU02:FS00:Server responded PLEASE_WAIT (464)
10:15:42:WARNING:WU02:FS00:Failed to send results, will try again later
10:26:33:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
10:26:33:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
10:26:33:WU02:FS00:Connecting to 155.247.166.219:8080
10:26:39:WU02:FS00:Upload 37.36%
10:26:45:WU02:FS00:Upload 80.84%
10:26:47:WU02:FS00:Upload complete
10:26:47:WU02:FS00:Server responded PLEASE_WAIT (464)
10:26:47:WARNING:WU02:FS00:Failed to send results, will try again later
10:44:30:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
10:44:30:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
10:44:30:WU02:FS00:Connecting to 155.247.166.219:8080
10:44:36:WU02:FS00:Upload 39.40%
10:44:42:WU02:FS00:Upload 82.88%
10:44:44:WU02:FS00:Upload complete
10:44:44:WU02:FS00:Server responded PLEASE_WAIT (464)
10:44:44:WARNING:WU02:FS00:Failed to send results, will try again later
11:13:32:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
11:13:32:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
11:13:32:WU02:FS00:Connecting to 155.247.166.219:8080
11:13:38:WU02:FS00:Upload 31.93%
11:13:44:WU02:FS00:Upload 75.40%
11:13:47:WU02:FS00:Upload complete
11:13:47:WU02:FS00:Server responded PLEASE_WAIT (464)
11:13:47:WARNING:WU02:FS00:Failed to send results, will try again later
12:00:31:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
12:00:31:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
12:00:31:WU02:FS00:Connecting to 155.247.166.219:8080
12:00:37:WU02:FS00:Upload 38.72%
12:00:43:WU02:FS00:Upload 82.20%
12:00:45:WU02:FS00:Upload complete
12:00:45:WU02:FS00:Server responded PLEASE_WAIT (464)
12:00:45:WARNING:WU02:FS00:Failed to send results, will try again later
13:00:31:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
13:00:31:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
13:00:31:WU02:FS00:Connecting to 155.247.166.219:8080
13:00:37:WU02:FS00:Upload 34.64%
13:00:43:WU02:FS00:Upload 74.72%
13:00:48:WU02:FS00:Upload complete
13:00:48:WU02:FS00:Server responded PLEASE_WAIT (464)
13:00:48:WARNING:WU02:FS00:Failed to send results, will try again later
******************************* Date: 2021-04-02 *******************************
14:00:31:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
14:00:31:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
14:00:31:WU02:FS00:Connecting to 155.247.166.219:8080
14:00:37:WU02:FS00:Upload 36.00%
14:00:43:WU02:FS00:Upload 78.12%
14:00:46:WU02:FS00:Upload complete
14:00:46:WU02:FS00:Server responded PLEASE_WAIT (464)
14:00:46:WARNING:WU02:FS00:Failed to send results, will try again later
14:31:44:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
14:31:44:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
14:31:44:WU02:FS00:Connecting to 155.247.166.219:8080
14:31:50:WU02:FS00:Upload 38.04%
14:31:56:WU02:FS00:Upload 78.80%
14:31:59:WU02:FS00:Upload complete
14:31:59:WU02:FS00:Server responded PLEASE_WAIT (464)
14:31:59:WARNING:WU02:FS00:Failed to send results, will try again later
14:33:21:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
14:33:21:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
14:33:21:WU02:FS00:Connecting to 155.247.166.219:8080
14:33:27:WU02:FS00:Upload 35.32%
14:33:33:WU02:FS00:Upload 76.08%
14:33:36:WU02:FS00:Upload complete
14:33:36:WU02:FS00:Server responded PLEASE_WAIT (464)
14:33:36:WARNING:WU02:FS00:Failed to send results, will try again later
14:35:58:WU02:FS00:Sending unit results: id:02 state:SEND error:NO_ERROR project:14188 run:6 clone:161 gen:3 core:0xa7 unit:0x000000a1000000030000376c00000006
14:35:58:WU02:FS00:Uploading 9.20MiB to 155.247.166.219
14:35:58:WU02:FS00:Connecting to 155.247.166.219:8080
14:36:04:WU02:FS00:Upload 34.64%
14:36:10:WU02:FS00:Upload 76.76%
14:36:13:WU02:FS00:Upload complete
14:36:13:WU02:FS00:Server responded PLEASE_WAIT (464)
14:36:13:WARNING:WU02:FS00:Failed to send results, will try again later

[itskieran]

It's been 24 hours, and I've decided to take another look at my logs.

It seems that F@H is trying to upload to the Work Server, 155.247.166.219, and hasn't even tried the collection server, 155.247.164.213. Is there a way I can force F@H to try uploading to the CS? (Though both are showing as WSes on the server stats page...?)

[bruce]

If there is a CS associated with that particular WU, it will be trying automatically. The choice of Assignment Server happens when the WU is initially downloaded and unfortunately is bonded to that particular WU until it is returned. If none were functional at the moment you got the WU, you won't have one. If that CS goes offline while you have that WU, you'll be limited to the original Work Server. There's an enhancement request to improve that function but no assurance when it might happen. Both FAHClient and the Server code would need to change.

[bruce]

I don't know about "Please Wait" but it's rare. It connects but then fails. FAHClient only tries to send to a Collection Server if the Work Server FAILS to connect.

If there is a CS associated with that particular WU, it will be trying automatically but only after the initial connection fails. Temporarily blacklisting that IP might force it to try the CS.

The choice of Assignment Server happens when the WU is initially downloaded and unfortunately is bonded to that particular WU until it is returned. If none were functional at the moment you got the WU, you won't have one. Once it is chosen, if that CS goes offline while you have that WU, you'll be limited to the original Work Server. There's an enhancement request to improve that function but no assurance when it might happen. Both FAHClient and the Server code would need to change.

[bruce]

The title mentions 155.247.164.213 but the log shows PLEASE_WAIT is the response from 155.247.166.219

I'm going to change the title of the first post to 155.247.166.219

[Maddog]

I also have this exact same poblem. Server 155.247.166.219. The server connects and the WU uploads to 95% before the Please wait message.

[itskieran]

The title mentions 155.247.164.213 but the log shows PLEASE_WAIT is the response from 155.247.166.219

I'm going to change the title of the first post to 155.247.166.219

Thanks - only put that in the subject as I took that from the CS in the Status tab, assuming it would be trying there only, not knowing that a CS was only a backup if the WS was down.

I'll look up how I can blacklist an IP on my machine, thanks for the advice!

Edit: blocked all connections to the WS in Windows Firewall, it successfully sent to the CS with WORK_ACK. I'll leave it blocked for now in case it's a server issue to stop getting given WUs for it, and make sure e-mail notifications are on for the forum to hopefully get an update when it gets fixed so I can start receiving WUs from it again.