Uploads 17411 'WAIT'(464) fails for a day

[new08]

Not seen this persist before. Over a day now..
project:17411 run:0 clone:653 gen:530
Other CPU units are doing ok..
Is there any way to trigger it my end?
Soon to expire, though the project doesn't seem to be current on summary page!

Code: Select all

00:34:04:WU00:FS00:0xa8:*********************** Log Started 2021-02-15T00:34:04Z ***********************
00:34:04:WU00:FS00:0xa8:************************** Gromacs Folding@home Core ***************************
00:34:04:WU00:FS00:0xa8:       Core: Gromacs
00:34:04:WU00:FS00:0xa8:       Type: 0xa8
00:34:04:WU00:FS00:0xa8:    Version: 0.0.12
00:34:04:WU00:FS00:0xa8:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
00:34:04:WU00:FS00:0xa8:  Copyright: 2020 foldingathome.org
00:34:04:WU00:FS00:0xa8:   Homepage: https://foldingathome.org/
00:34:04:WU00:FS00:0xa8:       Date: Jan 16 2021
00:34:04:WU00:FS00:0xa8:       Time: 11:56:07
00:34:04:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
00:34:04:WU00:FS00:0xa8:     Branch: master
00:34:04:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
00:34:04:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
00:34:04:WU00:FS00:0xa8:   Platform: win32 10
00:34:04:WU00:FS00:0xa8:       Bits: 64
00:34:04:WU00:FS00:0xa8:       Mode: Release
00:34:04:WU00:FS00:0xa8:       SIMD: avx_256
00:34:04:WU00:FS00:0xa8:     OpenMP: ON
00:34:04:WU00:FS00:0xa8:       CUDA: OFF
00:34:04:WU00:FS00:0xa8:       Args: -dir 00 -suffix 01 -version 706 -lifeline 12008 -checkpoint 20 -np
00:34:04:WU00:FS00:0xa8:             2
00:34:04:WU00:FS00:0xa8:************************************ libFAH ************************************
00:34:04:WU00:FS00:0xa8:       Date: Jan 16 2021
00:34:04:WU00:FS00:0xa8:       Time: 11:24:13
00:34:04:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
00:34:04:WU00:FS00:0xa8:     Branch: master
00:34:04:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
00:34:04:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
00:34:04:WU00:FS00:0xa8:   Platform: win32 10
00:34:04:WU00:FS00:0xa8:       Bits: 64
00:34:04:WU00:FS00:0xa8:       Mode: Release
00:34:04:WU00:FS00:0xa8:************************************ CBang *************************************
00:34:04:WU00:FS00:0xa8:       Date: Jan 16 2021
00:34:04:WU00:FS00:0xa8:       Time: 11:23:53
00:34:04:WU00:FS00:0xa8:   Revision: c5816759c404e4b65f9f364c3d1ef554a67c4225
00:34:04:WU00:FS00:0xa8:     Branch: master
00:34:04:WU00:FS00:0xa8:   Compiler: Visual C++ 2019 16.7
00:34:04:WU00:FS00:0xa8:    Options: /TP /std:c++14 /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
00:34:04:WU00:FS00:0xa8:   Platform: win32 10
00:34:04:WU00:FS00:0xa8:       Bits: 64
00:34:04:WU00:FS00:0xa8:       Mode: Release
00:34:04:WU00:FS00:0xa8:************************************ System ************************************
00:34:04:WU00:FS00:0xa8:        CPU: Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz
00:34:04:WU00:FS00:0xa8:     CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
00:34:04:WU00:FS00:0xa8:       CPUs: 4
00:34:04:WU00:FS00:0xa8:     Memory: 3.96GiB
00:34:04:WU00:FS00:0xa8:Free Memory: 1.22GiB
00:34:04:WU00:FS00:0xa8:    Threads: WINDOWS_THREADS
00:34:04:WU00:FS00:0xa8: OS Version: 6.2
00:34:04:WU00:FS00:0xa8:Has Battery: false
00:34:04:WU00:FS00:0xa8: On Battery: false
00:34:04:WU00:FS00:0xa8: UTC Offset: 0
00:34:04:WU00:FS00:0xa8:        PID: 14248
00:34:04:WU00:FS00:0xa8:        CWD: C:\Users\Admin\AppData\Roaming\FAHClient\work
00:34:04:WU00:FS00:0xa8:********************************************************************************
00:34:04:WU00:FS00:0xa8:Project: 16931 (Run 3, Clone 44, Gen 167)
00:34:04:WU00:FS00:0xa8:Unit: 0x00000000000000000000000000000000
00:34:04:WU00:FS00:0xa8:Reading tar file core.xml
00:34:04:WU00:FS00:0xa8:Reading tar file frame167.tpr
00:34:04:WU00:FS00:0xa8:Digital signatures verified
00:34:04:WU00:FS00:0xa8:Calling: mdrun -c frame167.gro -s frame167.tpr -x frame167.xtc -cpt 20 -nt 2 -ntmpi 1
00:34:04:WU00:FS00:0xa8:Steps: first=83500000 total=84000000
00:34:07:WU00:FS00:0xa8:Completed 1 out of 500000 steps (0%)
00:34:09:WU02:FS00:Upload 73.99%
00:34:10:WU02:FS00:Upload complete
00:34:10:WU02:FS00:Server responded WORK_ACK (400)
00:34:10:WU02:FS00:Final credit estimate, 2043.00 points
00:34:10:WU02:FS00:Cleaning up
00:37:21:WU00:FS00:0xa8:Completed 5000 out of 500000 steps (1%)
00:40:36:WU00:FS00:0xa8:Completed 10000 out of 500000 steps (2%)
00:43:49:WU00:FS00:0xa8:Completed 15000 out of 500000 steps (3%)
00:47:06:WU00:FS00:0xa8:Completed 20000 out of 500000 steps (4%)
00:50:22:WU00:FS00:0xa8:Completed 25000 out of 500000 steps (5%)
00:53:39:WU00:FS00:0xa8:Completed 30000 out of 500000 steps (6%)
00:55:11:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
00:55:11:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
00:55:11:WU01:FS00:Connecting to 66.170.111.50:8080
00:55:17:WU01:FS00:Upload 8.81%
00:55:23:WU01:FS00:Upload 28.33%
00:55:29:WU01:FS00:Upload 50.36%
00:55:35:WU01:FS00:Upload 72.40%
00:55:41:WU01:FS00:Upload 94.43%
00:55:42:WU01:FS00:Upload complete
00:55:42:WU01:FS00:Server responded PLEASE_WAIT (464)
00:55:42:WARNING:WU01:FS00:Failed to send results, will try again later
00:56:57:WU00:FS00:0xa8:Completed 35000 out of 500000 steps (7%)
01:00:12:WU00:FS00:0xa8:Completed 40000 out of 500000 steps (8%)
01:03:28:WU00:FS00:0xa8:Completed 45000 out of 500000 steps (9%)
01:06:43:WU00:FS00:0xa8:Completed 50000 out of 500000 steps (10%)
01:09:57:WU00:FS00:0xa8:Completed 55000 out of 500000 steps (11%)
01:13:11:WU00:FS00:0xa8:Completed 60000 out of 500000 steps (12%)
01:16:23:WU00:FS00:0xa8:Completed 65000 out of 500000 steps (13%)
01:19:34:WU00:FS00:0xa8:Completed 70000 out of 500000 steps (14%)
01:22:47:WU00:FS00:0xa8:Completed 75000 out of 500000 steps (15%)
01:26:01:WU00:FS00:0xa8:Completed 80000 out of 500000 steps (16%)
01:29:15:WU00:FS00:0xa8:Completed 85000 out of 500000 steps (17%)
01:32:28:WU00:FS00:0xa8:Completed 90000 out of 500000 steps (18%)
01:35:42:WU00:FS00:0xa8:Completed 95000 out of 500000 steps (19%)
01:38:56:WU00:FS00:0xa8:Completed 100000 out of 500000 steps (20%)
01:42:10:WU00:FS00:0xa8:Completed 105000 out of 500000 steps (21%)
01:45:24:WU00:FS00:0xa8:Completed 110000 out of 500000 steps (22%)
01:48:38:WU00:FS00:0xa8:Completed 115000 out of 500000 steps (23%)
01:51:52:WU00:FS00:0xa8:Completed 120000 out of 500000 steps (24%)
01:55:05:WU00:FS00:0xa8:Completed 125000 out of 500000 steps (25%)
01:55:11:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
01:55:11:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
01:55:11:WU01:FS00:Connecting to 66.170.111.50:8080
01:55:17:WU01:FS00:Upload 17.63%
01:55:23:WU01:FS00:Upload 39.66%
01:55:29:WU01:FS00:Upload 61.70%
01:55:35:WU01:FS00:Upload 83.10%
01:55:39:WU01:FS00:Upload complete
01:55:39:WU01:FS00:Server responded PLEASE_WAIT (464)
01:55:39:WARNING:WU01:FS00:Failed to send results, will try again later
01:58:19:WU00:FS00:0xa8:Completed 130000 out of 500000 steps (26%)
02:01:32:WU00:FS00:0xa8:Completed 135000 out of 500000 steps (27%)
02:04:46:WU00:FS00:0xa8:Completed 140000 out of 500000 steps (28%)
02:08:02:WU00:FS00:0xa8:Completed 145000 out of 500000 steps (29%)
02:11:18:WU00:FS00:0xa8:Completed 150000 out of 500000 steps (30%)
02:14:37:WU00:FS00:0xa8:Completed 155000 out of 500000 steps (31%)
02:17:55:WU00:FS00:0xa8:Completed 160000 out of 500000 steps (32%)
02:21:13:WU00:FS00:0xa8:Completed 165000 out of 500000 steps (33%)
02:24:29:WU00:FS00:0xa8:Completed 170000 out of 500000 steps (34%)
02:27:44:WU00:FS00:0xa8:Completed 175000 out of 500000 steps (35%)
02:30:59:WU00:FS00:0xa8:Completed 180000 out of 500000 steps (36%)
02:34:13:WU00:FS00:0xa8:Completed 185000 out of 500000 steps (37%)
02:37:28:WU00:FS00:0xa8:Completed 190000 out of 500000 steps (38%)
02:40:42:WU00:FS00:0xa8:Completed 195000 out of 500000 steps (39%)
02:43:56:WU00:FS00:0xa8:Completed 200000 out of 500000 steps (40%)
02:47:10:WU00:FS00:0xa8:Completed 205000 out of 500000 steps (41%)
02:50:25:WU00:FS00:0xa8:Completed 210000 out of 500000 steps (42%)
02:53:39:WU00:FS00:0xa8:Completed 215000 out of 500000 steps (43%)
02:55:11:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
02:55:11:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
02:55:11:WU01:FS00:Connecting to 66.170.111.50:8080
02:55:17:WU01:FS00:Upload 17.63%
02:55:23:WU01:FS00:Upload 39.03%
02:55:29:WU01:FS00:Upload 61.07%
02:55:35:WU01:FS00:Upload 83.10%
02:55:39:WU01:FS00:Upload complete
02:55:40:WU01:FS00:Server responded PLEASE_WAIT (464)
02:55:40:WARNING:WU01:FS00:Failed to send results, will try again later
02:56:53:WU00:FS00:0xa8:Completed 220000 out of 500000 steps (44%)
03:00:08:WU00:FS00:0xa8:Completed 225000 out of 500000 steps (45%)
03:03:21:WU00:FS00:0xa8:Completed 230000 out of 500000 steps (46%)
03:06:34:WU00:FS00:0xa8:Completed 235000 out of 500000 steps (47%)
03:09:48:WU00:FS00:0xa8:Completed 240000 out of 500000 steps (48%)
03:13:00:WU00:FS00:0xa8:Completed 245000 out of 500000 steps (49%)
03:16:14:WU00:FS00:0xa8:Completed 250000 out of 500000 steps (50%)
03:19:29:WU00:FS00:0xa8:Completed 255000 out of 500000 steps (51%)
03:22:43:WU00:FS00:0xa8:Completed 260000 out of 500000 steps (52%)
03:25:56:WU00:FS00:0xa8:Completed 265000 out of 500000 steps (53%)
03:29:11:WU00:FS00:0xa8:Completed 270000 out of 500000 steps (54%)
03:32:28:WU00:FS00:0xa8:Completed 275000 out of 500000 steps (55%)
03:35:46:WU00:FS00:0xa8:Completed 280000 out of 500000 steps (56%)
03:39:05:WU00:FS00:0xa8:Completed 285000 out of 500000 steps (57%)
03:42:21:WU00:FS00:0xa8:Completed 290000 out of 500000 steps (58%)
03:45:39:WU00:FS00:0xa8:Completed 295000 out of 500000 steps (59%)
03:48:54:WU00:FS00:0xa8:Completed 300000 out of 500000 steps (60%)
03:52:08:WU00:FS00:0xa8:Completed 305000 out of 500000 steps (61%)
03:55:11:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
03:55:11:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
03:55:11:WU01:FS00:Connecting to 66.170.111.50:8080
03:55:17:WU01:FS00:Upload 8.81%
03:55:23:WU01:FS00:Upload 27.70%
03:55:29:WU01:FS00:Upload 49.73%
03:55:35:WU01:FS00:Upload 71.77%
03:55:40:WU00:FS00:0xa8:Completed 310000 out of 500000 steps (62%)
03:55:41:WU01:FS00:Upload 93.80%
03:55:43:WU01:FS00:Upload complete
03:55:43:WU01:FS00:Server responded PLEASE_WAIT (464)
03:55:43:WARNING:WU01:FS00:Failed to send results, will try again later
03:58:54:WU00:FS00:0xa8:Completed 315000 out of 500000 steps (63%)
04:02:09:WU00:FS00:0xa8:Completed 320000 out of 500000 steps (64%)
04:05:23:WU00:FS00:0xa8:Completed 325000 out of 500000 steps (65%)
04:08:37:WU00:FS00:0xa8:Completed 330000 out of 500000 steps (66%)
04:11:51:WU00:FS00:0xa8:Completed 335000 out of 500000 steps (67%)
04:15:06:WU00:FS00:0xa8:Completed 340000 out of 500000 steps (68%)
04:18:23:WU00:FS00:0xa8:Completed 345000 out of 500000 steps (69%)
04:21:37:WU00:FS00:0xa8:Completed 350000 out of 500000 steps (70%)
04:24:52:WU00:FS00:0xa8:Completed 355000 out of 500000 steps (71%)
04:28:06:WU00:FS00:0xa8:Completed 360000 out of 500000 steps (72%)
04:31:20:WU00:FS00:0xa8:Completed 365000 out of 500000 steps (73%)
04:34:35:WU00:FS00:0xa8:Completed 370000 out of 500000 steps (74%)
04:37:49:WU00:FS00:0xa8:Completed 375000 out of 500000 steps (75%)
04:41:03:WU00:FS00:0xa8:Completed 380000 out of 500000 steps (76%)
04:44:16:WU00:FS00:0xa8:Completed 385000 out of 500000 steps (77%)
04:47:29:WU00:FS00:0xa8:Completed 390000 out of 500000 steps (78%)
04:50:24:WU00:FS00:0xa8:Completed 395000 out of 500000 steps (79%)
04:53:19:WU00:FS00:0xa8:Completed 400000 out of 500000 steps (80%)
04:55:11:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
04:55:11:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
04:55:11:WU01:FS00:Connecting to 66.170.111.50:8080
04:55:17:WU01:FS00:Upload 8.18%
04:55:23:WU01:FS00:Upload 27.07%
04:55:29:WU01:FS00:Upload 49.10%
04:55:35:WU01:FS00:Upload 70.51%
04:55:41:WU01:FS00:Upload 92.54%
04:55:43:WU01:FS00:Upload complete
04:55:43:WU01:FS00:Server responded PLEASE_WAIT (464)
04:55:43:WARNING:WU01:FS00:Failed to send results, will try again later
04:56:16:WU00:FS00:0xa8:Completed 405000 out of 500000 steps (81%)
04:59:11:WU00:FS00:0xa8:Completed 410000 out of 500000 steps (82%)
05:02:06:WU00:FS00:0xa8:Completed 415000 out of 500000 steps (83%)
05:05:01:WU00:FS00:0xa8:Completed 420000 out of 500000 steps (84%)
05:07:56:WU00:FS00:0xa8:Completed 425000 out of 500000 steps (85%)
05:10:50:WU00:FS00:0xa8:Completed 430000 out of 500000 steps (86%)
05:13:44:WU00:FS00:0xa8:Completed 435000 out of 500000 steps (87%)
05:16:38:WU00:FS00:0xa8:Completed 440000 out of 500000 steps (88%)
05:19:32:WU00:FS00:0xa8:Completed 445000 out of 500000 steps (89%)
05:22:25:WU00:FS00:0xa8:Completed 450000 out of 500000 steps (90%)
05:25:19:WU00:FS00:0xa8:Completed 455000 out of 500000 steps (91%)
05:28:14:WU00:FS00:0xa8:Completed 460000 out of 500000 steps (92%)
05:31:08:WU00:FS00:0xa8:Completed 465000 out of 500000 steps (93%)
05:34:02:WU00:FS00:0xa8:Completed 470000 out of 500000 steps (94%)
05:36:59:WU00:FS00:0xa8:Completed 475000 out of 500000 steps (95%)
05:39:53:WU00:FS00:0xa8:Completed 480000 out of 500000 steps (96%)
05:42:47:WU00:FS00:0xa8:Completed 485000 out of 500000 steps (97%)
05:45:41:WU00:FS00:0xa8:Completed 490000 out of 500000 steps (98%)
05:45:42:WU03:FS00:Connecting to assign1.foldingathome.org:80
05:45:43:WU03:FS00:Assigned to work server 129.32.209.201
05:45:43:WU03:FS00:Requesting new work unit for slot 00: cpu:2 from 129.32.209.201
05:45:43:WU03:FS00:Connecting to 129.32.209.201:8080
05:45:43:WU03:FS00:Downloading 6.65MiB
05:45:47:WU03:FS00:Download complete
05:45:47:WU03:FS00:Received Unit: id:03 state:DOWNLOAD error:NO_ERROR project:16927 run:12 clone:383 gen:114 core:0xa7 unit:0x0000017f000000720000421f0000000c
05:48:35:WU00:FS00:0xa8:Completed 495000 out of 500000 steps (99%)
05:51:29:WU00:FS00:0xa8:Completed 500000 out of 500000 steps (100%)
05:51:30:WU00:FS00:0xa8:Saving result file ..\logfile_01.txt
05:51:30:WU00:FS00:0xa8:Saving result file dhdl.xvg
05:51:30:WU00:FS00:0xa8:Saving result file frame167.gro
05:51:30:WU00:FS00:0xa8:Saving result file frame167.xtc
05:51:30:WU00:FS00:0xa8:Saving result file md.log
05:51:30:WU00:FS00:0xa8:Saving result file science.log
05:51:30:WU00:FS00:0xa8:Saving result file state.cpt
05:51:30:WU00:FS00:0xa8:Folding@home Core Shutdown: FINISHED_UNIT
05:51:31:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
05:51:31:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16931 run:3 clone:44 gen:167 core:0xa8 unit:0x0000002c000000a70000422300000003
05:51:31:WU00:FS00:Uploading 3.47MiB to 129.32.209.203
05:51:31:WU00:FS00:Connecting to 129.32.209.203:8080
05:51:31:WU03:FS00:Starting
05:51:31:WU03:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\Users\Admin\AppData\Roaming\FAHClient\cores/cores.foldingathome.org/win/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7.exe -dir 03 -suffix 01 -version 706 -lifeline 9568 -checkpoint 20 -np 2
05:51:31:WU03:FS00:Started FahCore on PID 13768
05:51:31:WU03:FS00:Core PID:12124
05:51:31:WU03:FS00:FahCore 0xa7 started
05:51:32:WU03:FS00:0xa7:*********************** Log Started 2021-02-15T05:51:31Z ***********************
05:51:32:WU03:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
05:51:32:WU03:FS00:0xa7:       Type: 0xa7
05:51:32:WU03:FS00:0xa7:       Core: Gromacs
05:51:32:WU03:FS00:0xa7:       Args: -dir 03 -suffix 01 -version 706 -lifeline 13768 -checkpoint 20 -np
05:51:32:WU03:FS00:0xa7:             2
05:51:32:WU03:FS00:0xa7:************************************ CBang *************************************
05:51:32:WU03:FS00:0xa7:       Date: Nov 27 2019
05:51:32:WU03:FS00:0xa7:       Time: 03:40:09
05:51:32:WU03:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
05:51:32:WU03:FS00:0xa7:     Branch: master
05:51:32:WU03:FS00:0xa7:   Compiler: Visual C++ 2008
05:51:32:WU03:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
05:51:32:WU03:FS00:0xa7:   Platform: win32 10
05:51:32:WU03:FS00:0xa7:       Bits: 64
05:51:32:WU03:FS00:0xa7:       Mode: Release
05:51:32:WU03:FS00:0xa7:************************************ System ************************************
05:51:32:WU03:FS00:0xa7:        CPU: Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz
05:51:32:WU03:FS00:0xa7:     CPU ID: GenuineIntel Family 6 Model 42 Stepping 7
05:51:32:WU03:FS00:0xa7:       CPUs: 4
05:51:32:WU03:FS00:0xa7:     Memory: 3.96GiB
05:51:32:WU03:FS00:0xa7:Free Memory: 1.30GiB
05:51:32:WU03:FS00:0xa7:    Threads: WINDOWS_THREADS
05:51:32:WU03:FS00:0xa7: OS Version: 6.2
05:51:32:WU03:FS00:0xa7:Has Battery: false
05:51:32:WU03:FS00:0xa7: On Battery: false
05:51:32:WU03:FS00:0xa7: UTC Offset: 0
05:51:32:WU03:FS00:0xa7:        PID: 12124
05:51:32:WU03:FS00:0xa7:        CWD: C:\Users\Admin\AppData\Roaming\FAHClient\work
05:51:32:WU03:FS00:0xa7:******************************** Build - libFAH ********************************
05:51:32:WU03:FS00:0xa7:    Version: 0.0.19
05:51:32:WU03:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
05:51:32:WU03:FS00:0xa7:  Copyright: 2019 foldingathome.org
05:51:32:WU03:FS00:0xa7:   Homepage: https://foldingathome.org/
05:51:32:WU03:FS00:0xa7:       Date: Nov 25 2019
05:51:32:WU03:FS00:0xa7:       Time: 17:12:41
05:51:32:WU03:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
05:51:32:WU03:FS00:0xa7:     Branch: master
05:51:32:WU03:FS00:0xa7:   Compiler: Visual C++ 2008
05:51:32:WU03:FS00:0xa7:    Options: /TP /nologo /EHa /wd4297 /wd4103 /Ox /MT
05:51:32:WU03:FS00:0xa7:   Platform: win32 10
05:51:32:WU03:FS00:0xa7:       Bits: 64
05:51:32:WU03:FS00:0xa7:       Mode: Release
05:51:32:WU03:FS00:0xa7:************************************ Build *************************************
05:51:32:WU03:FS00:0xa7:       SIMD: avx_256
05:51:32:WU03:FS00:0xa7:********************************************************************************
05:51:32:WU03:FS00:0xa7:Project: 16927 (Run 12, Clone 383, Gen 114)
05:51:32:WU03:FS00:0xa7:Unit: 0x00000000000000000000000000000000
05:51:32:WU03:FS00:0xa7:Reading tar file core.xml
05:51:32:WU03:FS00:0xa7:Reading tar file frame114.tpr
05:51:32:WU03:FS00:0xa7:Digital signatures verified
05:51:32:WU03:FS00:0xa7:Calling: mdrun -s frame114.tpr -o frame114.trr -cpt 20 -nt 2
05:51:32:WU03:FS00:0xa7:Steps: first=57000000 total=500000
05:51:35:WU03:FS00:0xa7:Completed 1 out of 500000 steps (0%)
05:51:37:WU00:FS00:Upload 79.36%
05:51:38:WU00:FS00:Upload complete
05:51:39:WU00:FS00:Server responded WORK_ACK (400)
05:51:39:WU00:FS00:Final credit estimate, 1821.00 points
05:51:39:WU00:FS00:Cleaning up
05:55:11:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
05:55:12:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
05:55:12:WU01:FS00:Connecting to 66.170.111.50:8080
05:55:18:WU01:FS00:Upload 20.15%
05:55:24:WU01:FS00:Upload 42.18%
05:55:30:WU01:FS00:Upload 64.21%
05:55:36:WU01:FS00:Upload 85.62%
05:55:40:WU01:FS00:Upload complete
05:55:40:WU01:FS00:Server responded PLEASE_WAIT (464)
05:55:40:WARNING:WU01:FS00:Failed to send results, will try again later
06:03:31:WU03:FS00:0xa7:Completed 5000 out of 500000 steps (1%)
06:14:43:WU03:FS00:0xa7:Completed 10000 out of 500000 steps (2%)
******************************* Date: 2021-02-15 *******************************
06:25:39:WU03:FS00:0xa7:Completed 15000 out of 500000 steps (3%)
06:36:37:WU03:FS00:0xa7:Completed 20000 out of 500000 steps (4%)
06:47:34:WU03:FS00:0xa7:Completed 25000 out of 500000 steps (5%)
06:55:11:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
06:55:12:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
06:55:12:WU01:FS00:Connecting to 66.170.111.50:8080
06:55:18:WU01:FS00:Upload 10.70%
06:55:24:WU01:FS00:Upload 30.85%
06:55:30:WU01:FS00:Upload 52.88%
06:55:36:WU01:FS00:Upload 74.92%
06:55:42:WU01:FS00:Upload 96.32%
06:55:43:WU01:FS00:Upload complete
06:55:43:WU01:FS00:Server responded PLEASE_WAIT (464)
06:55:43:WARNING:WU01:FS00:Failed to send results, will try again later
06:58:31:WU03:FS00:0xa7:Completed 30000 out of 500000 steps (6%)
07:09:25:WU03:FS00:0xa7:Completed 35000 out of 500000 steps (7%)
07:20:22:WU03:FS00:0xa7:Completed 40000 out of 500000 steps (8%)
07:31:16:WU03:FS00:0xa7:Completed 45000 out of 500000 steps (9%)
07:42:13:WU03:FS00:0xa7:Completed 50000 out of 500000 steps (10%)
07:53:08:WU03:FS00:0xa7:Completed 55000 out of 500000 steps (11%)
07:55:12:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
07:55:12:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
07:55:12:WU01:FS00:Connecting to 66.170.111.50:8080
07:55:18:WU01:FS00:Upload 10.07%
07:55:24:WU01:FS00:Upload 30.22%
07:55:30:WU01:FS00:Upload 51.62%
07:55:36:WU01:FS00:Upload 73.03%
07:55:42:WU01:FS00:Upload 94.43%
07:55:43:WU01:FS00:Upload complete
07:55:43:WU01:FS00:Server responded PLEASE_WAIT (464)
07:55:43:WARNING:WU01:FS00:Failed to send results, will try again later
08:04:03:WU03:FS00:0xa7:Completed 60000 out of 500000 steps (12%)
08:14:59:WU03:FS00:0xa7:Completed 65000 out of 500000 steps (13%)
08:25:54:WU03:FS00:0xa7:Completed 70000 out of 500000 steps (14%)
08:36:53:WU03:FS00:0xa7:Completed 75000 out of 500000 steps (15%)
08:47:48:WU03:FS00:0xa7:Completed 80000 out of 500000 steps (16%)
08:55:12:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
08:55:12:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
08:55:12:WU01:FS00:Connecting to 66.170.111.50:8080
08:55:18:WU01:FS00:Upload 9.44%
08:55:24:WU01:FS00:Upload 28.96%
08:55:30:WU01:FS00:Upload 50.36%
08:55:36:WU01:FS00:Upload 72.40%
08:55:42:WU01:FS00:Upload 93.17%
08:55:44:WU01:FS00:Upload complete
08:55:44:WU01:FS00:Server responded PLEASE_WAIT (464)
08:55:44:WARNING:WU01:FS00:Failed to send results, will try again later
08:58:45:WU03:FS00:0xa7:Completed 85000 out of 500000 steps (17%)
09:09:40:WU03:FS00:0xa7:Completed 90000 out of 500000 steps (18%)
09:20:36:WU03:FS00:0xa7:Completed 95000 out of 500000 steps (19%)
09:31:31:WU03:FS00:0xa7:Completed 100000 out of 500000 steps (20%)
09:42:27:WU03:FS00:0xa7:Completed 105000 out of 500000 steps (21%)
09:53:24:WU03:FS00:0xa7:Completed 110000 out of 500000 steps (22%)
09:55:12:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17411 run:0 clone:653 gen:530 core:0xa7 unit:0x0000028d000002120000440300000000
09:55:12:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
09:55:12:WU01:FS00:Connecting to 66.170.111.50:8080
09:55:18:WU01:FS00:Upload 9.44%
09:55:24:WU01:FS00:Upload 29.59%
09:55:30:WU01:FS00:Upload 51.62%
09:55:36:WU01:FS00:Upload 73.03%
09:55:42:WU01:FS00:Upload 95.06%
09:55:43:WU01:FS00:Upload complete
09:55:43:WU01:FS00:Server responded PLEASE_WAIT (464)
09:55:43:WARNING:WU01:FS00:Failed to send results, will try again later

[ajm]

The corresponding server 66.170.111.50 doesn't seem to have any disk space left: https://apps.foldingathome.org/serverstats

Code: Select all

66.170.111.50	fah-w1.vmware.com	WS	10.0.7	joseph	0.00/hr	2	1	No	Accept	162	0	GRO_A7	0bytes	a month	2021-02-15T12:33:49Z

It has been in trouble for nearly a month now: viewtopic.php?f=18&t=36581

[new08]

One of the few, less common, true data losses- on a short expiry then..
Thanx for check. The project seems off duty- so that's why I flagged it.

[Callaghan]

Are we wasting our time processing WU data pointed at this server, or will the data be recovered and stored at a later date.
If the data cannot be recovered/saved should WU's pointed at this server not be allocated for processing.
I appreciate i am a newbie at this project so appologies if i am talking out of the back of my neck but it seemed a reasonable question to ask.

[new08]

That was my point -posted on the link, from ajm above..
viewtopic.php?f=18&t=36581

[ajm]

It looks like this server doesn't assign many jobs anymore (0.00/hr, total 162).
I guess the person responsible is trying to fix whatever is bugging it, and is only making a few tests once in a while. Folding with this server may look like a waste of time for a donor right now, but there very well might be a good reason behind it.

[new08]

I haven't got the time annotated logs but that work was 8 hrs long from the start to finish on result file.
If testing needs doing- why not use a short send receive script -rather than use work spaces?
A month plus seems a long time to leave a server limping along part time, tbh.

[ajm]

I don't know. Maybe the problem doesn't lie in the send/receive process.

[new08]

Well, hopefully an admin will pick up the trail of this long runner

[Joe_H]

As mentioned in the topic for this server, they are aware of the space problem and working on it. An initial cleanup process failed before completion and the person looking into this is trying to get it fixed.

Soon to expire, though the project doesn't seem to be current on summary page!

Projects will only appear on the active list shown by the summary page when actively assigned by a WS. Depending on when the update process for the list runs, a project may show for a while after active assignment ends, but eventually doesn't.

It has been in trouble for nearly a month now

Actually it hasn't. The last time there was a problem was back in December, the last post connected with that was on the 21st, so over a month and a half until the post on Saturday, February 14.

Are we wasting our time processing WU data pointed at this server, or will the data be recovered and stored at a later date.

Unfortunately WUs which pass the final deadline time will be dumped by the client, and those which pass the expiration time will not get the bonus. When the server is fixed and back assigning WUs the dumped WUs will be reassigned. The server is on Accept only, so you should not be getting any WUs from this WS since sometime Saturday or early Sunday.

It looks like this server doesn't assign many jobs anymore (0.00/hr, total 162).

The server is on Accept only, the 162 WUs are ones created by the acceptance of previous WUs and the generation of the next Gen of that PRCG trajectory. Some may have been received during the first attempt at freeing up space, and then space filled up when the cleanup process exited before completion.

[new08]

Thanks for that in depth overview, Joe.
I thought cleaning up was at the client end, sorting out redundant files- but it actually refers to making space at the server end?
Unit now dumped anyway..
I didn't want it hanging around ages, reminding me

[Joe_H]

Two sorts of cleanup involved. On the client end once the WU has been accepted the work files are deleted. On the server end the completed WUs are stored initially with all data returned with them. However that data includes redundant information such as the location of all of the solvent molecules (mostly water). That can be removed once the WU data has been verified, and the results take up less space.

Besides this some error logs can be trimmed, completed projects transferred to other permanent storage, and so on to free up space on the server.

[bruce]

https://apps.foldingathome.org/wu#proje ... 53&gen=530 says it is NOT FOUND, which is not surprising if the server doesn't have room to store it.