Page 1 of 1

13850 (0, 6051, 32) - Server responded WORK_QUIT

Posted: Sun Apr 19, 2020 2:56 pm
by simon-mj-carter

Code: Select all

12:18:08:WU00:FS00:0xa7:*********************** Log Started 2020-04-11T12:18:08Z ***********************
12:18:08:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
12:18:08:WU00:FS00:0xa7:       Type: 0xa7
12:18:08:WU00:FS00:0xa7:       Core: Gromacs
12:18:08:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 705 -lifeline 1540 -checkpoint 15 -np 4
12:18:08:WU00:FS00:0xa7:************************************ CBang *************************************
12:18:08:WU00:FS00:0xa7:       Date: Oct 26 2019
12:18:08:WU00:FS00:0xa7:       Time: 01:38:25
12:18:08:WU00:FS00:0xa7:   Revision: c46a1a011a24143739ac7218c5a435f66777f62f
12:18:08:WU00:FS00:0xa7:     Branch: master
12:18:08:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
12:18:08:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
12:18:08:WU00:FS00:0xa7:   Platform: win32 10
12:18:08:WU00:FS00:0xa7:       Bits: 64
12:18:08:WU00:FS00:0xa7:       Mode: Release
12:18:08:WU00:FS00:0xa7:************************************ System ************************************
12:18:08:WU00:FS00:0xa7:        CPU: Intel(R) Core(TM) i3-8145U CPU @ 2.10GHz
12:18:08:WU00:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 142 Stepping 12
12:18:08:WU00:FS00:0xa7:       CPUs: 4
12:18:08:WU00:FS00:0xa7:     Memory: 7.82GiB
12:18:08:WU00:FS00:0xa7:Free Memory: 1.60GiB
12:18:08:WU00:FS00:0xa7:    Threads: WINDOWS_THREADS
12:18:08:WU00:FS00:0xa7: OS Version: 6.2
12:18:08:WU00:FS00:0xa7:Has Battery: true
12:18:08:WU00:FS00:0xa7: On Battery: false
12:18:08:WU00:FS00:0xa7: UTC Offset: 10
12:18:08:WU00:FS00:0xa7:        PID: 17864
12:18:08:WU00:FS00:0xa7:        CWD: C:\Users\simon\AppData\Roaming\FAHClient\work
12:18:08:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
12:18:08:WU00:FS00:0xa7:    Version: 0.0.18
12:18:08:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
12:18:08:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
12:18:08:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
12:18:08:WU00:FS00:0xa7:       Date: Oct 26 2019
12:18:08:WU00:FS00:0xa7:       Time: 01:52:30
12:18:08:WU00:FS00:0xa7:   Revision: c1e3513b1bc0c16013668f2173ee969e5995b38e
12:18:08:WU00:FS00:0xa7:     Branch: master
12:18:08:WU00:FS00:0xa7:   Compiler: Visual C++ 2008
12:18:08:WU00:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
12:18:08:WU00:FS00:0xa7:   Platform: win32 10
12:18:08:WU00:FS00:0xa7:       Bits: 64
12:18:08:WU00:FS00:0xa7:       Mode: Release
12:18:08:WU00:FS00:0xa7:************************************ Build *************************************
12:18:08:WU00:FS00:0xa7:       SIMD: avx_256
12:18:08:WU00:FS00:0xa7:********************************************************************************
12:18:08:WU00:FS00:0xa7:Project: 13850 (Run 0, Clone 6051, Gen 32)
12:18:08:WU00:FS00:0xa7:Unit: 0x0000002c287234c95e72ebd2c86103df
12:18:08:WU00:FS00:0xa7:Reading tar file core.xml
12:18:08:WU00:FS00:0xa7:Reading tar file frame32.tpr
12:18:08:WU00:FS00:0xa7:Digital signatures verified
12:18:08:WU00:FS00:0xa7:Calling: mdrun -s frame32.tpr -o frame32.trr -x frame32.xtc -e frame32.edr -cpt 15 -nt 4
12:18:08:WU00:FS00:0xa7:Steps: first=16000000 total=500000
12:18:10:WU00:FS00:0xa7:Completed 1 out of 500000 steps (0%)
12:22:34:WU00:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
12:26:51:WU00:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
12:31:01:WU00:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
12:35:08:WU00:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
12:39:22:WU00:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
12:43:38:WU00:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
12:48:16:WU00:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
12:52:27:WU00:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
12:56:46:WU00:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
13:01:05:WU00:FS00:0xa7:Completed 50000 out of 500000 steps (10%)
13:05:16:WU00:FS00:0xa7:Completed 55000 out of 500000 steps (11%)
13:09:12:WU00:FS00:0xa7:Completed 60000 out of 500000 steps (12%)
13:13:08:WU00:FS00:0xa7:Completed 65000 out of 500000 steps (13%)
13:17:03:WU00:FS00:0xa7:Completed 70000 out of 500000 steps (14%)
13:20:58:WU00:FS00:0xa7:Completed 75000 out of 500000 steps (15%)
13:24:50:WU00:FS00:0xa7:Completed 80000 out of 500000 steps (16%)
13:28:43:WU00:FS00:0xa7:Completed 85000 out of 500000 steps (17%)
******************************* Date: 2020-04-11 *******************************
13:32:34:WU00:FS00:0xa7:Completed 90000 out of 500000 steps (18%)
13:36:30:WU00:FS00:0xa7:Completed 95000 out of 500000 steps (19%)
13:40:22:WU00:FS00:0xa7:Completed 100000 out of 500000 steps (20%)
13:44:20:WU00:FS00:0xa7:Completed 105000 out of 500000 steps (21%)
13:48:14:WU00:FS00:0xa7:Completed 110000 out of 500000 steps (22%)
13:52:05:WU00:FS00:0xa7:Completed 115000 out of 500000 steps (23%)
13:55:59:WU00:FS00:0xa7:Completed 120000 out of 500000 steps (24%)
13:59:52:WU00:FS00:0xa7:Completed 125000 out of 500000 steps (25%)
14:03:47:WU00:FS00:0xa7:Completed 130000 out of 500000 steps (26%)
14:07:42:WU00:FS00:0xa7:Completed 135000 out of 500000 steps (27%)
14:11:37:WU00:FS00:0xa7:Completed 140000 out of 500000 steps (28%)
14:15:35:WU00:FS00:0xa7:Completed 145000 out of 500000 steps (29%)
14:19:32:WU00:FS00:0xa7:Completed 150000 out of 500000 steps (30%)
14:23:58:WU00:FS00:0xa7:Completed 155000 out of 500000 steps (31%)
14:27:58:WU00:FS00:0xa7:Completed 160000 out of 500000 steps (32%)
14:31:53:WU00:FS00:0xa7:Completed 165000 out of 500000 steps (33%)
14:35:51:WU00:FS00:0xa7:Completed 170000 out of 500000 steps (34%)
14:39:48:WU00:FS00:0xa7:Completed 175000 out of 500000 steps (35%)
14:43:45:WU00:FS00:0xa7:Completed 180000 out of 500000 steps (36%)
14:47:43:WU00:FS00:0xa7:Completed 185000 out of 500000 steps (37%)
14:51:38:WU00:FS00:0xa7:Completed 190000 out of 500000 steps (38%)
14:55:31:WU00:FS00:0xa7:Completed 195000 out of 500000 steps (39%)
14:59:24:WU00:FS00:0xa7:Completed 200000 out of 500000 steps (40%)
15:03:18:WU00:FS00:0xa7:Completed 205000 out of 500000 steps (41%)
15:07:15:WU00:FS00:0xa7:Completed 210000 out of 500000 steps (42%)
15:11:14:WU00:FS00:0xa7:Completed 215000 out of 500000 steps (43%)
15:15:10:WU00:FS00:0xa7:Completed 220000 out of 500000 steps (44%)
15:19:10:WU00:FS00:0xa7:Completed 225000 out of 500000 steps (45%)
15:23:14:WU00:FS00:0xa7:Completed 230000 out of 500000 steps (46%)
15:27:10:WU00:FS00:0xa7:Completed 235000 out of 500000 steps (47%)
15:31:09:WU00:FS00:0xa7:Completed 240000 out of 500000 steps (48%)
15:35:03:WU00:FS00:0xa7:Completed 245000 out of 500000 steps (49%)
15:38:57:WU00:FS00:0xa7:Completed 250000 out of 500000 steps (50%)
15:42:56:WU00:FS00:0xa7:Completed 255000 out of 500000 steps (51%)
15:46:57:WU00:FS00:0xa7:Completed 260000 out of 500000 steps (52%)
15:50:51:WU00:FS00:0xa7:Completed 265000 out of 500000 steps (53%)
15:54:47:WU00:FS00:0xa7:Completed 270000 out of 500000 steps (54%)
15:58:42:WU00:FS00:0xa7:Completed 275000 out of 500000 steps (55%)
16:02:37:WU00:FS00:0xa7:Completed 280000 out of 500000 steps (56%)
16:06:34:WU00:FS00:0xa7:Completed 285000 out of 500000 steps (57%)
16:10:40:WU00:FS00:0xa7:Completed 290000 out of 500000 steps (58%)
16:14:39:WU00:FS00:0xa7:Completed 295000 out of 500000 steps (59%)
16:18:41:WU00:FS00:0xa7:Completed 300000 out of 500000 steps (60%)
16:23:09:WU00:FS00:0xa7:Completed 305000 out of 500000 steps (61%)
16:27:29:WU00:FS00:0xa7:Completed 310000 out of 500000 steps (62%)
16:31:39:WU00:FS00:0xa7:Completed 315000 out of 500000 steps (63%)
16:35:49:WU00:FS00:0xa7:Completed 320000 out of 500000 steps (64%)
16:40:09:WU00:FS00:0xa7:Completed 325000 out of 500000 steps (65%)
16:44:34:WU00:FS00:0xa7:Completed 330000 out of 500000 steps (66%)
16:48:54:WU00:FS00:0xa7:Completed 335000 out of 500000 steps (67%)
16:53:26:WU00:FS00:0xa7:Completed 340000 out of 500000 steps (68%)
16:57:55:WU00:FS00:0xa7:Completed 345000 out of 500000 steps (69%)
17:02:33:WU00:FS00:0xa7:Completed 350000 out of 500000 steps (70%)
17:07:03:WU00:FS00:0xa7:Completed 355000 out of 500000 steps (71%)
17:11:40:WU00:FS00:0xa7:Completed 360000 out of 500000 steps (72%)
17:16:04:WU00:FS00:0xa7:Completed 365000 out of 500000 steps (73%)
17:20:42:WU00:FS00:0xa7:Completed 370000 out of 500000 steps (74%)
17:25:16:WU00:FS00:0xa7:Completed 375000 out of 500000 steps (75%)
17:30:52:WU00:FS00:0xa7:Completed 380000 out of 500000 steps (76%)
17:36:21:WU00:FS00:0xa7:Completed 385000 out of 500000 steps (77%)
17:41:55:WU00:FS00:0xa7:Completed 390000 out of 500000 steps (78%)
17:47:17:WU00:FS00:0xa7:Completed 395000 out of 500000 steps (79%)
17:52:42:WU00:FS00:0xa7:Completed 400000 out of 500000 steps (80%)
17:58:11:WU00:FS00:0xa7:Completed 405000 out of 500000 steps (81%)
18:03:25:WU00:FS00:0xa7:Completed 410000 out of 500000 steps (82%)
18:08:47:WU00:FS00:0xa7:Completed 415000 out of 500000 steps (83%)
18:14:25:WU00:FS00:0xa7:Completed 420000 out of 500000 steps (84%)
18:19:56:WU00:FS00:0xa7:Completed 425000 out of 500000 steps (85%)
18:25:20:WU00:FS00:0xa7:Completed 430000 out of 500000 steps (86%)
18:30:28:WU00:FS00:0xa7:Completed 435000 out of 500000 steps (87%)
18:35:32:WU00:FS00:0xa7:Completed 440000 out of 500000 steps (88%)
18:39:56:WU00:FS00:0xa7:Completed 445000 out of 500000 steps (89%)
18:44:25:WU00:FS00:0xa7:Completed 450000 out of 500000 steps (90%)
18:48:49:WU00:FS00:0xa7:Completed 455000 out of 500000 steps (91%)
18:53:03:WU00:FS00:0xa7:Completed 460000 out of 500000 steps (92%)
18:57:23:WU00:FS00:0xa7:Completed 465000 out of 500000 steps (93%)
19:01:50:WU00:FS00:0xa7:Completed 470000 out of 500000 steps (94%)
19:05:41:WU00:FS00:0xa7:Completed 475000 out of 500000 steps (95%)
19:09:36:WU00:FS00:0xa7:Completed 480000 out of 500000 steps (96%)
19:13:23:WU00:FS00:0xa7:Completed 485000 out of 500000 steps (97%)
19:17:09:WU00:FS00:0xa7:Completed 490000 out of 500000 steps (98%)
19:20:56:WU00:FS00:0xa7:Completed 495000 out of 500000 steps (99%)
19:20:56:ERROR:WU01:FS00:Exception: Could not get IP address for assign1.foldingathome.org: No such host is known. 
19:20:56:ERROR:WU01:FS00:Exception: Could not get IP address for assign2.foldingathome.org: No such host is known. 
19:20:56:WARNING:WU01:FS00:Exception: Failed to find any IP addresses for assignment servers
19:20:56:ERROR:WU01:FS00:Exception: Could not get an assignment
19:20:56:ERROR:WU01:FS00:Exception: Could not get IP address for assign1.foldingathome.org: No such host is known. 
19:20:56:ERROR:WU01:FS00:Exception: Could not get IP address for assign2.foldingathome.org: No such host is known. 
19:20:56:WARNING:WU01:FS00:Exception: Failed to find any IP addresses for assignment servers
19:20:56:ERROR:WU01:FS00:Exception: Could not get an assignment
19:21:56:ERROR:WU01:FS00:Exception: Could not get IP address for assign1.foldingathome.org: No such host is known. 
19:21:56:ERROR:WU01:FS00:Exception: Could not get IP address for assign2.foldingathome.org: No such host is known. 
19:21:56:WARNING:WU01:FS00:Exception: Failed to find any IP addresses for assignment servers
19:21:56:ERROR:WU01:FS00:Exception: Could not get an assignment
19:23:33:ERROR:WU01:FS00:Exception: Could not get IP address for assign1.foldingathome.org: No such host is known. 
19:23:33:ERROR:WU01:FS00:Exception: Could not get IP address for assign2.foldingathome.org: No such host is known. 
19:23:33:WARNING:WU01:FS00:Exception: Failed to find any IP addresses for assignment servers
19:23:33:ERROR:WU01:FS00:Exception: Could not get an assignment
19:24:42:WU00:FS00:0xa7:Completed 500000 out of 500000 steps (100%)
19:24:44:WU00:FS00:0xa7:Saving result file ..\logfile_01.txt
19:24:44:WU00:FS00:0xa7:Saving result file frame32.edr
19:24:44:WU00:FS00:0xa7:Saving result file frame32.trr
19:24:44:WU00:FS00:0xa7:Saving result file frame32.xtc
19:24:44:WU00:FS00:0xa7:Saving result file md.log
19:24:44:WU00:FS00:0xa7:Saving result file science.log
19:24:44:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
19:24:44:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
19:24:44:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:13850 run:0 clone:6051 gen:32 core:0xa7 unit:0x0000002c287234c95e72ebd2c86103df
19:24:44:WU00:FS00:Uploading 2.48MiB to 40.114.52.201
19:24:44:WU00:FS00:Connecting to 40.114.52.201:8080
19:24:44:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
19:24:44:WU00:FS00:Connecting to 40.114.52.201:80
19:24:44:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: A socket operation was attempted to an unreachable network.
...
04:04:09:WU00:FS00:Trying to send results to collection server
04:04:09:WU00:FS00:Uploading 2.48MiB to 52.224.109.74
04:04:09:WU00:FS00:Connecting to 52.224.109.74:8080
04:04:09:WARNING:WU00:FS00:WorkServer connection failed on port 8080 trying 80
04:04:09:WU00:FS00:Connecting to 52.224.109.74:80
04:04:09:ERROR:WU00:FS00:Exception: Failed to connect to 52.224.109.74:80: A socket operation was attempted to an unreachable network.
04:09:51:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:13850 run:0 clone:6051 gen:32 core:0xa7 unit:0x0000002c287234c95e72ebd2c86103df
04:09:51:WU00:FS00:Uploading 2.48MiB to 40.114.52.201
04:09:51:WU00:FS00:Connecting to 40.114.52.201:8080
04:09:53:WU01:FS00:Connecting to 65.254.110.245:8080
04:09:57:WARNING:WU01:FS00:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
04:09:57:WU01:FS00:Connecting to 18.218.241.186:80
04:10:02:WU01:FS00:Assigned to work server 168.245.198.125
04:10:02:WU01:FS00:Requesting new work unit for slot 00: READY cpu:4 from 168.245.198.125
04:10:02:WU01:FS00:Connecting to 168.245.198.125:8080
04:10:03:WU00:FS00:Upload 7.56%
04:10:09:ERROR:WU01:FS00:Exception: Server did not assign work unit
04:10:12:WU00:FS00:Upload 12.59%
04:10:18:WU00:FS00:Upload 17.63%
04:10:24:WU00:FS00:Upload 27.71%
04:10:30:WU00:FS00:Upload 37.78%
04:10:37:WU00:FS00:Upload 50.37%
04:10:43:WU00:FS00:Upload 62.97%
04:10:49:WU00:FS00:Upload 78.08%
04:10:55:WU00:FS00:Upload 85.64%
04:11:01:WU00:FS00:Upload 90.67%
04:11:07:WU00:FS00:Upload 95.71%
04:11:14:WU00:FS00:Upload complete
04:11:14:WU00:FS00:Server responded WORK_QUIT (404)
04:11:14:WARNING:WU00:FS00:Server did not like results, dumping
04:11:14:WU00:FS00:Cleaning up

Re: 13850 (0, 6051, 32) - Server responded WORK_QUIT

Posted: Mon Apr 20, 2020 3:14 am
by PantherX
Welcome to the F@H Forum simon-mj-carter,

Please note that the error means that the WU failed a validation check on the server. It could be due to data corruption during transit, disk issues on your system, a faulty hardware, or rarely, a bad WU.

I would suggest that you keep an eye and if you encounter more of those issues, then we can investigate it further :)

Re: 13850 (0, 6051, 32) - Server responded WORK_QUIT

Posted: Thu Apr 23, 2020 9:33 am
by simon-mj-carter
PantherX wrote:Welcome to the F@H Forum simon-mj-carter,

Please note that the error means that the WU failed a validation check on the server. It could be due to data corruption during transit, disk issues on your system, a faulty hardware, or rarely, a bad WU.

I would suggest that you keep an eye and if you encounter more of those issues, then we can investigate it further :)
Thanks, I raised the issue as I'd had the same error when returning two WUs 6 weeks ago, don't recall coming across this problem previously. FYI, topic '14197 (3, 790, 3) & (1, 763, 0) - Server responded WORK_QUIT'
tug27224 wrote:Thanks for reporting these. Just saw the increased error rates on this projects. Looking into it now.

I've paused this project for now while I investigate. Let me know if you see this error on any others in the mean time.

Re: 13850 (0, 6051, 32) - Server responded WORK_QUIT

Posted: Thu Apr 23, 2020 10:12 am
by Neil-B
As a matter of interest do you usually see these type of errors:

19:20:56:ERROR:WU01:FS00:Exception: Could not get IP address for assign1.foldingathome.org: No such host is known.

19:24:44:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: A socket operation was attempted to an unreachable network.

04:04:09:ERROR:WU00:FS00:Exception: Failed to connect to 52.224.109.74:80: A socket operation was attempted to an unreachable network.

... or do they only happen around the times you have had these WORK_QUIT errors?

To me they have the feel of some sort network/router issue - which might also explain some potential data corruption that could cause this type of error - Not sure but putting the thought out there for consideration.

Re: 13850 (0, 6051, 32) - Server responded WORK_QUIT

Posted: Wed Apr 29, 2020 3:37 am
by simon-mj-carter
I get these errors frequently and assumed they had something to do with the servers being overloaded and/or WUs not being available. Would be interested in any feedback. These errors don't only happen around the times I've had WORK_QUIT errors. I've only had 3 WORK_QUIT errors in the past 6 or so months.
Neil-B wrote:As a matter of interest do you usually see these type of errors:

19:20:56:ERROR:WU01:FS00:Exception: Could not get IP address for assign1.foldingathome.org: No such host is known.

19:24:44:WARNING:WU00:FS00:Exception: Failed to send results to work server: Failed to connect to 40.114.52.201:80: A socket operation was attempted to an unreachable network.

04:04:09:ERROR:WU00:FS00:Exception: Failed to connect to 52.224.109.74:80: A socket operation was attempted to an unreachable network.

... or do they only happen around the times you have had these WORK_QUIT errors?

To me they have the feel of some sort network/router issue - which might also explain some potential data corruption that could cause this type of error - Not sure but putting the thought out there for consideration.

Re: 13850 (0, 6051, 32) - Server responded WORK_QUIT

Posted: Wed Apr 29, 2020 7:42 am
by Neil-B
Hopefully one of the technically minded folders will spot this thread ... those errors aren't the usual ones afaik for server overload ... they seem to imply some sort of routing/comms issue that just might if the timing were bad impact data transit on upload and possibly be the cause of the Work Quits issues.