Upload to 66.170.111.50:8080 failing repeatedly

[bikeaddict]

Never seen this response before...

Code: Select all

00:26:58:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:16701 run:128 clone:3 gen:148 core:0x22 unit:0x00000003000000940000413d00000080
00:26:58:WU01:FS01:Uploading 65.32MiB to 66.170.111.50
00:26:58:WU01:FS01:Connecting to 66.170.111.50:8080
00:27:04:WU01:FS01:Upload 6.51%
00:27:10:WU01:FS01:Upload 13.59%
00:27:16:WU01:FS01:Upload 20.48%
00:27:22:WU01:FS01:Upload 27.56%
00:27:28:WU01:FS01:Upload 34.44%
00:27:34:WU01:FS01:Upload 41.52%
00:27:40:WU01:FS01:Upload 48.32%
00:27:46:WU01:FS01:Upload 55.40%
00:27:52:WU01:FS01:Upload 62.48%
00:27:58:WU01:FS01:Upload 69.37%
00:28:04:WU01:FS01:Upload 76.16%
00:28:10:WU01:FS01:Upload 83.24%
00:28:16:WU01:FS01:Upload 90.13%
00:28:22:WU01:FS01:Upload 97.21%
00:28:25:WU01:FS01:Upload complete
00:28:25:WU01:FS01:Server responded PLEASE_WAIT (464)
00:28:25:WARNING:WU01:FS01:Failed to send results, will try again later

<snip of same error eight more times>

01:14:14:WU01:FS01:Sending unit results: id:01 state:SEND error:NO_ERROR project:16701 run:128 clone:3 gen:148 core:0x22 unit:0x00000003000000940000413d00000080
01:14:14:WU01:FS01:Uploading 65.32MiB to 66.170.111.50
01:14:14:WU01:FS01:Connecting to 66.170.111.50:8080
01:14:20:WU01:FS01:Upload 6.79%
01:14:26:WU01:FS01:Upload 13.87%
01:14:32:WU01:FS01:Upload 20.67%
01:14:38:WU01:FS01:Upload 27.75%
01:14:44:WU01:FS01:Upload 34.83%
01:14:50:WU01:FS01:Upload 41.81%
01:14:56:WU01:FS01:Upload 48.89%
01:15:02:WU01:FS01:Upload 55.68%
01:15:08:WU01:FS01:Upload 62.76%
01:15:14:WU01:FS01:Upload 69.65%
01:15:20:WU01:FS01:Upload 76.73%
01:15:26:WU01:FS01:Upload 83.62%
01:15:32:WU01:FS01:Upload 90.42%
01:15:38:WU01:FS01:Upload 97.59%
01:15:40:WU01:FS01:Upload complete
01:15:40:WU01:FS01:Server responded PLEASE_WAIT (464)
01:15:40:WARNING:WU01:FS01:Failed to send results, will try again later

[bikeaddict]

The server stats page shows the server is out of disk space.

[Joe_H]

I have sent a report to the persons responsible for the project and WS.

[bikeaddict]

About 15 hours later the problem has not been fixed. You'd think all these servers should have monitoring that starts alerting people when the free disk space gets below about 5-10%.

[icunarta]

Thank you for the notice. I am trying to free up some space for the work server.

[Joe_H]

About 15 hours later the problem has not been fixed. You'd think all these servers should have monitoring that starts alerting people when the free disk space gets below about 5-10%.

There is monitoring, it works most of the time. In this case one of the places low disk space is reported last got a report from a different server about 2 weeks ago. Complicating matters is that this WS is a donated server and not at one of the research labs involved in F@h. So network connections are going through a corporate firewall that is maintained by a non-F@h group.

Now your report came in during the evening here in the US, so no response likely overnight. Then one of the persons contacted I know is on the West Coast, so it is early morning there. The other person I don't know where they are located, but I just got a notice they had signed in and read my report of the problem and are going to work on getting it fixed.

[gunnarre]

Thank you.

A good thing is that this WU has pretty generous timeout of 3 days, and expiration 7 days, so as long as it's fixed within the weekend, everyone will get their points.

[gunnarre]

Seems to be fixed:

Code: Select all

16:45:00:WU00:FS01:Server responded WORK_ACK (400)

Space: 7.47TiB

[jrsynow]

Moved post to existing topic and this WS - 66.170.111.50 - j

Just wanted to confirm that I'm seeing similar issues with 66.170.111.50 as well as 158.130.118.25 and have been for a couple days now, so it looks like a bigger issue beyond the Tempe University servers. Not sure if we should now change the post subject. I had the one workstation with the slower ISP doing exclusively Alzheimer's work units (18213) for the past few months and never had connectivity issues.

I did catch one job being hung-up during upload to 66.170.111.50 from my workstation with the faster ISP, similar behavior. Logs are below.

15 attempts to get a 16706 WU uploaded:

Code: Select all

12:25:58:WU00:FS00:Upload 12.36%
12:26:04:WU00:FS00:Upload 27.20%
12:26:10:WU00:FS00:Upload 40.19%
12:26:16:WU00:FS00:Upload 51.31%
12:26:22:WU00:FS00:Upload 63.68%
12:26:28:WU00:FS00:Upload 77.28%
12:26:34:WU00:FS00:Upload 88.41%
12:26:45:WU00:FS00:Upload complete
12:26:45:WU00:FS00:Server responded WORK_ACK (400)
12:26:45:WU00:FS00:Final credit estimate, 599018.00 points
12:26:45:WU00:FS00:Cleaning up
13:13:24:WU00:FS00:Connecting to assign1.foldingathome.org:80
13:13:25:WU00:FS00:Assigned to work server 66.170.111.50
13:13:25:WU00:FS00:Requesting new work unit for slot 00: gpu:9:0 AD103 [GeForce RTX 4080] from 66.170.111.50
13:13:25:WU00:FS00:Connecting to 66.170.111.50:8080
13:13:48:WU00:FS00:Downloading 46.38MiB
13:13:54:WU00:FS00:Download 5.39%
13:14:00:WU00:FS00:Download 13.61%
13:14:06:WU00:FS00:Download 19.41%
13:14:12:WU00:FS00:Download 22.24%
13:14:18:WU00:FS00:Download 24.26%
13:14:24:WU00:FS00:Download 33.83%
13:14:31:WU00:FS00:Download 34.90%
13:14:37:WU00:FS00:Download 35.85%
13:14:43:WU00:FS00:Download 38.14%
13:14:49:WU00:FS00:Download 40.16%
13:14:55:WU00:FS00:Download 43.12%
13:15:01:WU00:FS00:Download 48.78%
13:15:07:WU00:FS00:Download 50.27%
13:15:14:WU00:FS00:Download 51.75%
13:15:21:WU00:FS00:Download 52.56%
13:15:27:WU00:FS00:Download 54.18%
13:15:33:WU00:FS00:Download 55.52%
13:15:39:WU00:FS00:Download 57.27%
13:15:45:WU00:FS00:Download 60.24%
13:15:52:WU00:FS00:Download 61.72%
13:15:58:WU00:FS00:Download 62.53%
13:16:04:WU00:FS00:Download 64.82%
13:16:11:WU00:FS00:Download 65.50%
13:16:17:WU00:FS00:Download 67.11%
13:16:23:WU00:FS00:Download 71.69%
13:16:29:WU00:FS00:Download 77.08%
13:16:35:WU00:FS00:Download 83.82%
13:16:41:WU00:FS00:Download 90.29%
13:16:47:WU00:FS00:Download 96.90%
13:16:49:WU00:FS00:Download complete
13:16:49:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
13:16:49:WU00:FS00:Starting
13:16:49:WU00:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:\ProgramData\FAHClient\cores/cores.foldingathome.org/win/64bit/22-0.0.20/Core_22.fah/FahCore_22.exe -dir 00 -suffix 01 -version 706 -lifeline 14048 -checkpoint 15 -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor nvidia -gpu 0 -gpu-usage 100
13:16:49:WU00:FS00:Started FahCore on PID 1684
13:16:49:WU00:FS00:Core PID:1476
13:16:49:WU00:FS00:FahCore 0x22 started
13:16:50:WU00:FS00:0x22:*********************** Log Started 2023-04-06T13:16:49Z ***********************
13:16:50:WU00:FS00:0x22:*************************** Core22 Folding@home Core ***************************
13:16:50:WU00:FS00:0x22:       Core: Core22
13:16:50:WU00:FS00:0x22:       Type: 0x22
13:16:50:WU00:FS00:0x22:    Version: 0.0.20
13:16:50:WU00:FS00:0x22:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
13:16:50:WU00:FS00:0x22:  Copyright: 2020 foldingathome.org
13:16:50:WU00:FS00:0x22:   Homepage: https://foldingathome.org/
13:16:50:WU00:FS00:0x22:       Date: Jan 20 2022
13:16:50:WU00:FS00:0x22:       Time: 01:15:36
13:16:50:WU00:FS00:0x22:   Revision: 3f211b8a4346514edbff34e3cb1c0e0ec951373c
13:16:50:WU00:FS00:0x22:     Branch: HEAD
13:16:50:WU00:FS00:0x22:   Compiler: Visual C++
13:16:50:WU00:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
13:16:50:WU00:FS00:0x22:             -DOPENMM_VERSION="\"7.7.0\""
13:16:50:WU00:FS00:0x22:   Platform: win32 10
13:16:50:WU00:FS00:0x22:       Bits: 64
13:16:50:WU00:FS00:0x22:       Mode: Release
13:16:50:WU00:FS00:0x22:Maintainers: John Chodera <john.chodera@choderalab.org> and Peter Eastman
13:16:50:WU00:FS00:0x22:             <peastman@stanford.edu>
13:16:50:WU00:FS00:0x22:       Args: -dir 00 -suffix 01 -version 706 -lifeline 1684 -checkpoint 15
13:16:50:WU00:FS00:0x22:             -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu-vendor
13:16:50:WU00:FS00:0x22:             nvidia -gpu 0 -gpu-usage 100
13:16:50:WU00:FS00:0x22:************************************ libFAH ************************************
13:16:50:WU00:FS00:0x22:       Date: Jan 20 2022
13:16:50:WU00:FS00:0x22:       Time: 01:14:17
13:16:50:WU00:FS00:0x22:   Revision: 9f4ad694e75c2350d4bb6b8b5b769ba27e483a2f
13:16:50:WU00:FS00:0x22:     Branch: HEAD
13:16:50:WU00:FS00:0x22:   Compiler: Visual C++
13:16:50:WU00:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
13:16:50:WU00:FS00:0x22:   Platform: win32 10
13:16:50:WU00:FS00:0x22:       Bits: 64
13:16:50:WU00:FS00:0x22:       Mode: Release
13:16:50:WU00:FS00:0x22:************************************ CBang *************************************
13:16:50:WU00:FS00:0x22:       Date: Jan 20 2022
13:16:50:WU00:FS00:0x22:       Time: 01:13:20
13:16:50:WU00:FS00:0x22:   Revision: ab023d155b446906d55b0f6c9a1eedeea04f7a1a
13:16:50:WU00:FS00:0x22:     Branch: HEAD
13:16:50:WU00:FS00:0x22:   Compiler: Visual C++
13:16:50:WU00:FS00:0x22:    Options: /TP /nologo /EHa /wd4297 /wd4103 /O2 /Zc:throwingNew /MT
13:16:50:WU00:FS00:0x22:   Platform: win32 10
13:16:50:WU00:FS00:0x22:       Bits: 64
13:16:50:WU00:FS00:0x22:       Mode: Release
13:16:50:WU00:FS00:0x22:************************************ System ************************************
13:16:50:WU00:FS00:0x22:        CPU: AMD Ryzen 9 5950X 16-Core Processor
13:16:50:WU00:FS00:0x22:     CPU ID: AuthenticAMD Family 25 Model 33 Stepping 2
13:16:50:WU00:FS00:0x22:       CPUs: 32
13:16:50:WU00:FS00:0x22:     Memory: 31.91GiB
13:16:50:WU00:FS00:0x22:Free Memory: 28.22GiB
13:16:50:WU00:FS00:0x22:    Threads: WINDOWS_THREADS
13:16:50:WU00:FS00:0x22: OS Version: 6.2
13:16:50:WU00:FS00:0x22:Has Battery: false
13:16:50:WU00:FS00:0x22: On Battery: false
13:16:50:WU00:FS00:0x22: UTC Offset: -4
13:16:50:WU00:FS00:0x22:        PID: 1476
13:16:50:WU00:FS00:0x22:        CWD: C:\ProgramData\FAHClient\work
13:16:50:WU00:FS00:0x22:************************************ OpenMM ************************************
13:16:50:WU00:FS00:0x22:    Version: 7.7.0
13:16:50:WU00:FS00:0x22:********************************************************************************
13:16:50:WU00:FS00:0x22:Project: 16706 (Run 162, Clone 4, Gen 39)
13:16:50:WU00:FS00:0x22:Reading tar file core.xml
13:16:50:WU00:FS00:0x22:Reading tar file integrator.xml
13:16:50:WU00:FS00:0x22:Reading tar file state.xml
13:16:50:WU00:FS00:0x22:Reading tar file system.xml
13:16:51:WU00:FS00:0x22:Digital signatures verified
13:16:51:WU00:FS00:0x22:Folding@home GPU Core22 Folding@home Core
13:16:51:WU00:FS00:0x22:Version 0.0.20
13:16:51:WU00:FS00:0x22:  Checkpoint write interval: 62500 steps (5%) [20 total]
13:16:51:WU00:FS00:0x22:  JSON viewer frame write interval: 12500 steps (1%) [100 total]
13:16:51:WU00:FS00:0x22:  XTC frame write interval: 10000 steps (0.8%) [125 total]
13:16:51:WU00:FS00:0x22:  Global context and integrator variables write interval: disabled
13:16:51:WU00:FS00:0x22:There are 4 platforms available.
13:16:51:WU00:FS00:0x22:Platform 0: Reference
13:16:51:WU00:FS00:0x22:Platform 1: CPU
13:16:51:WU00:FS00:0x22:Platform 2: OpenCL
13:16:51:WU00:FS00:0x22:  opencl-device 0 specified
13:16:51:WU00:FS00:0x22:Platform 3: CUDA
13:16:51:WU00:FS00:0x22:  cuda-device 0 specified
13:17:24:WU00:FS00:0x22:Attempting to create CUDA context:
13:17:24:WU00:FS00:0x22:  Configuring platform CUDA
13:17:33:WU00:FS00:0x22:  Using CUDA and gpu 0
13:17:34:WU00:FS00:0x22:Completed 0 out of 1250000 steps (0%)
13:17:35:WU00:FS00:0x22:Checkpoint completed at step 0
13:18:22:WU00:FS00:0x22:Completed 12500 out of 1250000 steps (1%)
13:19:09:WU00:FS00:0x22:Completed 25000 out of 1250000 steps (2%)
13:19:57:WU00:FS00:0x22:Completed 37500 out of 1250000 steps (3%)
13:20:44:WU00:FS00:0x22:Completed 50000 out of 1250000 steps (4%)
13:21:31:WU00:FS00:0x22:Completed 62500 out of 1250000 steps (5%)
13:21:33:WU00:FS00:0x22:Checkpoint completed at step 62500
13:22:20:WU00:FS00:0x22:Completed 75000 out of 1250000 steps (6%)
13:23:08:WU00:FS00:0x22:Completed 87500 out of 1250000 steps (7%)
13:23:55:WU00:FS00:0x22:Completed 100000 out of 1250000 steps (8%)
13:24:43:WU00:FS00:0x22:Completed 112500 out of 1250000 steps (9%)
13:25:30:WU00:FS00:0x22:Completed 125000 out of 1250000 steps (10%)
13:25:32:WU00:FS00:0x22:Checkpoint completed at step 125000
13:26:19:WU00:FS00:0x22:Completed 137500 out of 1250000 steps (11%)
13:27:06:WU00:FS00:0x22:Completed 150000 out of 1250000 steps (12%)
13:27:54:WU00:FS00:0x22:Completed 162500 out of 1250000 steps (13%)
13:28:41:WU00:FS00:0x22:Completed 175000 out of 1250000 steps (14%)
13:29:29:WU00:FS00:0x22:Completed 187500 out of 1250000 steps (15%)
13:29:30:WU00:FS00:0x22:Checkpoint completed at step 187500
13:30:18:WU00:FS00:0x22:Completed 200000 out of 1250000 steps (16%)
13:31:05:WU00:FS00:0x22:Completed 212500 out of 1250000 steps (17%)
13:31:52:WU00:FS00:0x22:Completed 225000 out of 1250000 steps (18%)
13:32:40:WU00:FS00:0x22:Completed 237500 out of 1250000 steps (19%)
13:33:27:WU00:FS00:0x22:Completed 250000 out of 1250000 steps (20%)
13:33:29:WU00:FS00:0x22:Checkpoint completed at step 250000
13:34:16:WU00:FS00:0x22:Completed 262500 out of 1250000 steps (21%)
13:35:04:WU00:FS00:0x22:Completed 275000 out of 1250000 steps (22%)
13:35:51:WU00:FS00:0x22:Completed 287500 out of 1250000 steps (23%)
13:36:39:WU00:FS00:0x22:Completed 300000 out of 1250000 steps (24%)
13:37:26:WU00:FS00:0x22:Completed 312500 out of 1250000 steps (25%)
13:37:28:WU00:FS00:0x22:Checkpoint completed at step 312500
13:38:15:WU00:FS00:0x22:Completed 325000 out of 1250000 steps (26%)
13:39:03:WU00:FS00:0x22:Completed 337500 out of 1250000 steps (27%)
13:39:50:WU00:FS00:0x22:Completed 350000 out of 1250000 steps (28%)
13:40:37:WU00:FS00:0x22:Completed 362500 out of 1250000 steps (29%)
13:41:25:WU00:FS00:0x22:Completed 375000 out of 1250000 steps (30%)
13:41:26:WU00:FS00:0x22:Checkpoint completed at step 375000
13:42:14:WU00:FS00:0x22:Completed 387500 out of 1250000 steps (31%)
13:43:01:WU00:FS00:0x22:Completed 400000 out of 1250000 steps (32%)
13:43:49:WU00:FS00:0x22:Completed 412500 out of 1250000 steps (33%)
13:44:36:WU00:FS00:0x22:Completed 425000 out of 1250000 steps (34%)
13:45:24:WU00:FS00:0x22:Completed 437500 out of 1250000 steps (35%)
13:45:25:WU00:FS00:0x22:Checkpoint completed at step 437500
13:46:13:WU00:FS00:0x22:Completed 450000 out of 1250000 steps (36%)
13:47:00:WU00:FS00:0x22:Completed 462500 out of 1250000 steps (37%)
13:47:47:WU00:FS00:0x22:Completed 475000 out of 1250000 steps (38%)
13:48:35:WU00:FS00:0x22:Completed 487500 out of 1250000 steps (39%)
13:49:23:WU00:FS00:0x22:Completed 500000 out of 1250000 steps (40%)
13:49:24:WU00:FS00:0x22:Checkpoint completed at step 500000
13:50:12:WU00:FS00:0x22:Completed 512500 out of 1250000 steps (41%)
13:50:59:WU00:FS00:0x22:Completed 525000 out of 1250000 steps (42%)
13:51:46:WU00:FS00:0x22:Completed 537500 out of 1250000 steps (43%)
13:52:34:WU00:FS00:0x22:Completed 550000 out of 1250000 steps (44%)
13:53:21:WU00:FS00:0x22:Completed 562500 out of 1250000 steps (45%)
13:53:23:WU00:FS00:0x22:Checkpoint completed at step 562500
13:54:10:WU00:FS00:0x22:Completed 575000 out of 1250000 steps (46%)
13:54:58:WU00:FS00:0x22:Completed 587500 out of 1250000 steps (47%)
13:55:45:WU00:FS00:0x22:Completed 600000 out of 1250000 steps (48%)
13:56:33:WU00:FS00:0x22:Completed 612500 out of 1250000 steps (49%)
13:57:20:WU00:FS00:0x22:Completed 625000 out of 1250000 steps (50%)
13:57:22:WU00:FS00:0x22:Checkpoint completed at step 625000
13:58:09:WU00:FS00:0x22:Completed 637500 out of 1250000 steps (51%)
13:58:57:WU00:FS00:0x22:Completed 650000 out of 1250000 steps (52%)
13:59:44:WU00:FS00:0x22:Completed 662500 out of 1250000 steps (53%)
14:00:32:WU00:FS00:0x22:Completed 675000 out of 1250000 steps (54%)
14:01:19:WU00:FS00:0x22:Completed 687500 out of 1250000 steps (55%)
14:01:21:WU00:FS00:0x22:Checkpoint completed at step 687500
14:02:08:WU00:FS00:0x22:Completed 700000 out of 1250000 steps (56%)
14:02:56:WU00:FS00:0x22:Completed 712500 out of 1250000 steps (57%)
14:03:43:WU00:FS00:0x22:Completed 725000 out of 1250000 steps (58%)
14:04:31:WU00:FS00:0x22:Completed 737500 out of 1250000 steps (59%)
14:05:18:WU00:FS00:0x22:Completed 750000 out of 1250000 steps (60%)
14:05:20:WU00:FS00:0x22:Checkpoint completed at step 750000
14:06:07:WU00:FS00:0x22:Completed 762500 out of 1250000 steps (61%)
14:06:55:WU00:FS00:0x22:Completed 775000 out of 1250000 steps (62%)
14:07:43:WU00:FS00:0x22:Completed 787500 out of 1250000 steps (63%)
14:08:30:WU00:FS00:0x22:Completed 800000 out of 1250000 steps (64%)
14:09:18:WU00:FS00:0x22:Completed 812500 out of 1250000 steps (65%)
14:09:19:WU00:FS00:0x22:Checkpoint completed at step 812500
14:10:07:WU00:FS00:0x22:Completed 825000 out of 1250000 steps (66%)
14:10:54:WU00:FS00:0x22:Completed 837500 out of 1250000 steps (67%)
14:11:42:WU00:FS00:0x22:Completed 850000 out of 1250000 steps (68%)
14:12:29:WU00:FS00:0x22:Completed 862500 out of 1250000 steps (69%)
14:13:17:WU00:FS00:0x22:Completed 875000 out of 1250000 steps (70%)
14:13:19:WU00:FS00:0x22:Checkpoint completed at step 875000
14:14:06:WU00:FS00:0x22:Completed 887500 out of 1250000 steps (71%)
14:14:54:WU00:FS00:0x22:Completed 900000 out of 1250000 steps (72%)
14:15:41:WU00:FS00:0x22:Completed 912500 out of 1250000 steps (73%)
14:16:29:WU00:FS00:0x22:Completed 925000 out of 1250000 steps (74%)
14:17:16:WU00:FS00:0x22:Completed 937500 out of 1250000 steps (75%)
14:17:18:WU00:FS00:0x22:Checkpoint completed at step 937500
14:18:06:WU00:FS00:0x22:Completed 950000 out of 1250000 steps (76%)
14:18:53:WU00:FS00:0x22:Completed 962500 out of 1250000 steps (77%)
14:19:41:WU00:FS00:0x22:Completed 975000 out of 1250000 steps (78%)
14:20:28:WU00:FS00:0x22:Completed 987500 out of 1250000 steps (79%)
14:21:16:WU00:FS00:0x22:Completed 1000000 out of 1250000 steps (80%)
14:21:18:WU00:FS00:0x22:Checkpoint completed at step 1000000
14:22:05:WU00:FS00:0x22:Completed 1012500 out of 1250000 steps (81%)
14:22:53:WU00:FS00:0x22:Completed 1025000 out of 1250000 steps (82%)
14:23:40:WU00:FS00:0x22:Completed 1037500 out of 1250000 steps (83%)
14:24:28:WU00:FS00:0x22:Completed 1050000 out of 1250000 steps (84%)
14:25:16:WU00:FS00:0x22:Completed 1062500 out of 1250000 steps (85%)
14:25:17:WU00:FS00:0x22:Checkpoint completed at step 1062500
14:26:05:WU00:FS00:0x22:Completed 1075000 out of 1250000 steps (86%)
14:26:53:WU00:FS00:0x22:Completed 1087500 out of 1250000 steps (87%)
14:27:40:WU00:FS00:0x22:Completed 1100000 out of 1250000 steps (88%)
14:28:28:WU00:FS00:0x22:Completed 1112500 out of 1250000 steps (89%)
14:29:15:WU00:FS00:0x22:Completed 1125000 out of 1250000 steps (90%)
14:29:17:WU00:FS00:0x22:Checkpoint completed at step 1125000
14:30:05:WU00:FS00:0x22:Completed 1137500 out of 1250000 steps (91%)
14:30:52:WU00:FS00:0x22:Completed 1150000 out of 1250000 steps (92%)
14:31:40:WU00:FS00:0x22:Completed 1162500 out of 1250000 steps (93%)
14:32:28:WU00:FS00:0x22:Completed 1175000 out of 1250000 steps (94%)
14:33:15:WU00:FS00:0x22:Completed 1187500 out of 1250000 steps (95%)
14:33:17:WU00:FS00:0x22:Checkpoint completed at step 1187500
14:34:04:WU00:FS00:0x22:Completed 1200000 out of 1250000 steps (96%)
14:34:52:WU00:FS00:0x22:Completed 1212500 out of 1250000 steps (97%)
14:35:39:WU00:FS00:0x22:Completed 1225000 out of 1250000 steps (98%)
14:36:27:WU00:FS00:0x22:Completed 1237500 out of 1250000 steps (99%)
14:37:15:WU00:FS00:0x22:Completed 1250000 out of 1250000 steps (100%)
14:37:15:WU00:FS00:0x22:Average performance: 45.5696 ns/day
14:37:16:WU00:FS00:0x22:Checkpoint completed at step 1250000
14:37:26:WU00:FS00:0x22:Saving result file ..\logfile_01.txt
14:37:26:WU00:FS00:0x22:Saving result file checkpointIntegrator.xml
14:37:26:WU00:FS00:0x22:Saving result file checkpointState.xml
14:37:33:WU00:FS00:0x22:Saving result file positions.xtc
14:37:33:WU00:FS00:0x22:Saving result file science.log
14:37:33:WU00:FS00:0x22:Saving result file xtcAtoms.csv.bz2
14:37:33:WU00:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
14:37:35:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
14:37:35:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
14:37:35:WU00:FS00:Uploading 64.98MiB to 66.170.111.50
14:37:35:WU00:FS00:Connecting to 66.170.111.50:8080
14:37:41:WU00:FS00:Upload 2.21%
14:37:47:WU00:FS00:Upload 4.62%
14:37:53:WU00:FS00:Upload 6.16%
14:38:52:WU00:FS00:Upload 6.54%
14:39:21:WU00:FS00:Upload 6.64%
14:39:27:WU00:FS00:Upload 8.75%
14:39:34:WU00:FS00:Upload 10.48%
14:40:58:WU00:FS00:Upload 11.25%
14:41:04:WU00:FS00:Upload 13.27%
14:41:11:WU00:FS00:Upload 15.10%
14:42:11:WU00:FS00:Upload 15.77%
14:43:02:WU00:FS00:Upload 15.87%
14:43:08:WU00:FS00:Upload 17.99%
14:43:14:WU00:FS00:Upload 19.52%
14:43:38:WU00:FS00:Upload 19.91%
14:44:28:WU00:FS00:Upload 20.01%
14:44:34:WU00:FS00:Upload 22.02%
14:44:40:WU00:FS00:Upload 23.47%
14:44:57:WU00:FS00:Upload 23.85%
14:45:42:WU00:FS00:Upload 23.95%
14:46:05:WU00:FS00:Upload 24.53%
14:46:14:WU00:FS00:Upload 25.10%
14:46:42:WU00:FS00:Upload 25.20%
14:46:48:WU00:FS00:Upload 27.31%
14:46:57:WU00:FS00:Upload 29.14%
14:47:03:WU00:FS00:Upload 30.58%
14:47:09:WU00:FS00:Upload 32.03%
14:47:15:WU00:FS00:Upload 33.95%
14:48:18:WU00:FS00:Upload 35.20%
14:49:10:WU00:FS00:Upload 35.87%
14:49:54:WU00:FS00:Upload 36.55%
14:50:31:WU00:FS00:Upload 37.22%
14:50:47:WU00:FS00:Upload 37.80%
14:50:59:WU00:FS00:Upload 37.89%
14:51:16:WU00:FS00:Upload 38.47%
14:51:55:WU00:FS00:Upload 39.82%
14:52:01:WU00:FS00:Upload 41.74%
14:52:07:WU00:FS00:Upload 43.18%
14:53:04:WU00:FS00:Upload 43.67%
14:53:10:WU00:FS00:Upload 44.53%
14:53:16:WU00:FS00:Upload 46.55%
14:53:22:WU00:FS00:Upload 47.99%
14:54:37:WU00:FS00:Upload 48.38%
14:54:55:WU00:FS00:Upload 48.47%
14:55:03:WU00:FS00:Upload 48.96%
14:56:06:WU00:FS00:Upload 49.05%
14:56:53:WU00:FS00:Upload 49.63%
14:56:59:WU00:FS00:Upload 51.26%
14:57:05:WU00:FS00:Upload 53.09%
14:57:11:WU00:FS00:Upload 55.11%
14:57:18:WU00:FS00:Upload 56.07%
14:57:24:WU00:FS00:Upload 57.71%
14:57:30:WU00:FS00:Upload 59.73%
14:57:37:WU00:FS00:Upload 61.65%
14:58:29:WU00:FS00:Upload 62.32%
14:58:59:WU00:FS00:Upload 62.42%
14:58:59:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
14:58:59:WU00:FS00:Trying to send results to collection server
14:58:59:WU00:FS00:Uploading 64.98MiB to 158.130.118.25
14:58:59:WU00:FS00:Connecting to 158.130.118.25:8080
14:59:05:WU00:FS00:Upload 2.02%
14:59:11:WU00:FS00:Upload 3.56%
14:59:17:WU00:FS00:Upload 5.10%
14:59:23:WU00:FS00:Upload 6.35%
14:59:29:WU00:FS00:Upload 8.66%
14:59:50:WU00:FS00:Upload 9.52%
15:00:22:WU00:FS00:Upload 9.62%
15:00:28:WU00:FS00:Upload 11.54%
15:00:37:WU00:FS00:Upload 12.12%
15:00:58:WU00:FS00:Upload 12.50%
15:02:01:WU00:FS00:Upload 12.60%
15:02:01:ERROR:WU00:FS00:Exception: Transfer failed
15:02:01:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
15:02:01:WU00:FS00:Uploading 64.98MiB to 66.170.111.50
15:02:01:WU00:FS00:Connecting to 66.170.111.50:8080
15:02:07:WU00:FS00:Upload 1.64%
15:02:13:WU00:FS00:Upload 3.08%
15:02:19:WU00:FS00:Upload 4.91%
15:02:25:WU00:FS00:Upload 6.83%
15:03:17:WU00:FS00:Upload 7.41%
15:03:17:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
15:03:17:WU00:FS00:Trying to send results to collection server
15:03:17:WU00:FS00:Uploading 64.98MiB to 158.130.118.25
15:03:17:WU00:FS00:Connecting to 158.130.118.25:8080
15:03:23:WU00:FS00:Upload 1.54%
15:03:29:WU00:FS00:Upload 3.08%
15:03:35:WU00:FS00:Upload 4.62%
15:03:41:WU00:FS00:Upload 5.96%
15:03:47:WU00:FS00:Upload 7.02%
15:03:53:WU00:FS00:Upload 8.27%
15:03:59:WU00:FS00:Upload 9.43%
15:04:05:WU00:FS00:Upload 10.58%
15:04:11:WU00:FS00:Upload 12.50%
15:04:19:WU00:FS00:Upload 13.47%
15:04:25:WU00:FS00:Upload 14.91%
15:04:31:WU00:FS00:Upload 16.35%
15:04:37:WU00:FS00:Upload 18.08%
15:04:43:WU00:FS00:Upload 19.81%
15:04:49:WU00:FS00:Upload 21.45%
15:04:55:WU00:FS00:Upload 23.08%
15:05:01:WU00:FS00:Upload 24.62%
15:05:07:WU00:FS00:Upload 26.45%
15:05:13:WU00:FS00:Upload 27.80%
15:05:19:WU00:FS00:Upload 29.05%
15:05:25:WU00:FS00:Upload 30.10%
15:05:31:WU00:FS00:Upload 31.16%
15:05:37:WU00:FS00:Upload 32.51%
15:05:49:WU00:FS00:Upload 33.66%
15:05:55:WU00:FS00:Upload 34.62%
15:06:01:WU00:FS00:Upload 36.16%
15:06:07:WU00:FS00:Upload 37.41%
15:06:13:WU00:FS00:Upload 38.95%
15:06:19:WU00:FS00:Upload 40.49%
15:06:25:WU00:FS00:Upload 41.65%
15:06:31:WU00:FS00:Upload 43.18%
15:06:37:WU00:FS00:Upload 45.01%
15:06:43:WU00:FS00:Upload 46.55%
15:07:05:WU00:FS00:Upload 47.13%
15:07:34:WU00:FS00:Upload 47.22%
15:07:41:WU00:FS00:Upload 48.86%
15:07:48:WU00:FS00:Upload 49.05%
15:07:54:WU00:FS00:Upload 50.88%
15:08:00:WU00:FS00:Upload 52.61%
15:08:06:WU00:FS00:Upload 53.86%
15:08:12:WU00:FS00:Upload 55.40%
15:08:19:WU00:FS00:Upload 57.32%
15:08:25:WU00:FS00:Upload 58.96%
15:08:33:WU00:FS00:Upload 60.59%
15:09:08:WU00:FS00:Upload 61.65%
15:09:08:ERROR:WU00:FS00:Exception: Transfer failed
15:09:08:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
15:09:08:WU00:FS00:Uploading 64.98MiB to 66.170.111.50
15:09:08:WU00:FS00:Connecting to 66.170.111.50:8080
15:09:14:WU00:FS00:Upload 0.96%
15:09:20:WU00:FS00:Upload 2.40%
15:09:26:WU00:FS00:Upload 4.23%
15:09:33:WU00:FS00:Upload 6.25%
15:10:08:WU00:FS00:Upload 7.60%
15:10:08:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
15:10:08:WU00:FS00:Trying to send results to collection server
15:10:08:WU00:FS00:Uploading 64.98MiB to 158.130.118.25
15:10:08:WU00:FS00:Connecting to 158.130.118.25:8080
15:10:14:WU00:FS00:Upload 1.25%
15:10:20:WU00:FS00:Upload 2.89%
15:10:26:WU00:FS00:Upload 4.62%
15:10:32:WU00:FS00:Upload 6.44%
15:10:38:WU00:FS00:Upload 7.89%
15:10:44:WU00:FS00:Upload 9.43%
15:10:50:WU00:FS00:Upload 10.96%
15:10:57:WU00:FS00:Upload 11.54%
15:11:03:WU00:FS00:Upload 13.37%
15:11:09:WU00:FS00:Upload 15.20%
15:11:15:WU00:FS00:Upload 16.93%
15:11:21:WU00:FS00:Upload 19.04%
15:11:36:WU00:FS00:Upload 20.49%
15:11:42:WU00:FS00:Upload 21.74%
15:11:48:WU00:FS00:Upload 23.56%
15:11:54:WU00:FS00:Upload 24.43%
15:12:00:WU00:FS00:Upload 25.58%
15:12:09:WU00:FS00:Upload 27.22%
15:12:15:WU00:FS00:Upload 28.76%
15:12:21:WU00:FS00:Upload 30.39%
15:12:27:WU00:FS00:Upload 32.22%
15:12:33:WU00:FS00:Upload 34.05%
15:12:39:WU00:FS00:Upload 34.62%
15:12:47:WU00:FS00:Upload 35.59%
15:12:53:WU00:FS00:Upload 37.32%
15:13:01:WU00:FS00:Upload 38.95%
15:13:07:WU00:FS00:Upload 40.40%
15:13:13:WU00:FS00:Upload 42.32%
15:13:19:WU00:FS00:Upload 43.28%
15:13:25:WU00:FS00:Upload 45.20%
15:13:31:WU00:FS00:Upload 46.84%
15:13:45:WU00:FS00:Upload 47.03%
15:13:51:WU00:FS00:Upload 48.19%
15:13:57:WU00:FS00:Upload 49.44%
15:14:03:WU00:FS00:Upload 50.97%
15:14:12:WU00:FS00:Upload 52.61%
15:14:18:WU00:FS00:Upload 54.15%
15:14:24:WU00:FS00:Upload 55.78%
15:14:30:WU00:FS00:Upload 56.75%
15:14:36:WU00:FS00:Upload 58.57%
15:14:42:WU00:FS00:Upload 60.11%
15:14:48:WU00:FS00:Upload 60.69%
15:14:54:WU00:FS00:Upload 62.32%
15:15:00:WU00:FS00:Upload 63.67%
15:15:06:WU00:FS00:Upload 65.31%
15:15:12:WU00:FS00:Upload 66.27%
15:15:18:WU00:FS00:Upload 68.00%
15:15:26:WU00:FS00:Upload 69.54%
15:16:07:WU00:FS00:Upload 70.31%
15:16:07:ERROR:WU00:FS00:Exception: Transfer failed
15:16:07:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
15:16:07:WU00:FS00:Uploading 64.98MiB to 66.170.111.50
15:16:07:WU00:FS00:Connecting to 66.170.111.50:8080
15:16:13:WU00:FS00:Upload 1.06%
15:16:19:WU00:FS00:Upload 2.50%
15:16:25:WU00:FS00:Upload 4.62%
15:16:31:WU00:FS00:Upload 6.06%
15:16:37:WU00:FS00:Upload 6.54%
15:16:43:WU00:FS00:Upload 8.18%
15:16:54:WU00:FS00:Upload 9.62%
15:17:00:WU00:FS00:Upload 11.16%
15:17:09:WU00:FS00:Upload 12.70%
15:17:15:WU00:FS00:Upload 14.52%
15:17:26:WU00:FS00:Upload 15.87%
15:17:32:WU00:FS00:Upload 17.22%
15:17:38:WU00:FS00:Upload 19.04%
15:17:44:WU00:FS00:Upload 20.77%
15:17:50:WU00:FS00:Upload 22.41%
15:17:56:WU00:FS00:Upload 24.14%
15:18:02:WU00:FS00:Upload 25.87%
15:18:08:WU00:FS00:Upload 27.51%
15:18:14:WU00:FS00:Upload 28.95%
15:18:20:WU00:FS00:Upload 30.49%
15:18:26:WU00:FS00:Upload 32.41%
15:18:34:WU00:FS00:Upload 32.70%
15:18:40:WU00:FS00:Upload 34.24%
15:18:46:WU00:FS00:Upload 35.87%
15:18:52:WU00:FS00:Upload 36.55%
15:18:58:WU00:FS00:Upload 36.74%
15:19:15:WU00:FS00:Upload 37.13%
15:20:18:WU00:FS00:Upload 37.22%
15:20:24:WU00:FS00:Upload 39.05%
15:20:32:WU00:FS00:Upload 40.78%
15:20:38:WU00:FS00:Upload 42.51%
15:20:44:WU00:FS00:Upload 43.95%
15:21:00:WU00:FS00:Upload 44.53%
15:22:04:WU00:FS00:Upload 44.63%
15:22:04:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
15:22:04:WU00:FS00:Trying to send results to collection server
15:22:04:WU00:FS00:Uploading 64.98MiB to 158.130.118.25
15:22:04:WU00:FS00:Connecting to 158.130.118.25:8080
15:22:12:WU00:FS00:Upload 0.19%
15:22:18:WU00:FS00:Upload 1.44%
15:22:24:WU00:FS00:Upload 3.17%
15:22:30:WU00:FS00:Upload 4.81%
15:22:36:WU00:FS00:Upload 6.44%
15:22:42:WU00:FS00:Upload 7.02%
15:22:48:WU00:FS00:Upload 8.66%
15:22:54:WU00:FS00:Upload 9.71%
15:23:29:WU00:FS00:Upload 9.81%
15:23:29:ERROR:WU00:FS00:Exception: Transfer failed
15:23:29:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
15:23:29:WU00:FS00:Uploading 64.98MiB to 66.170.111.50
15:23:29:WU00:FS00:Connecting to 66.170.111.50:8080
15:23:35:WU00:FS00:Upload 1.25%
15:24:08:WU00:FS00:Upload 2.98%
15:24:38:WU00:FS00:Upload 3.08%
15:24:38:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
15:24:38:WU00:FS00:Trying to send results to collection server
15:24:38:WU00:FS00:Uploading 64.98MiB to 158.130.118.25
15:24:38:WU00:FS00:Connecting to 158.130.118.25:8080
15:24:44:WU00:FS00:Upload 1.64%
15:24:50:WU00:FS00:Upload 3.37%
15:24:56:WU00:FS00:Upload 4.71%
15:25:02:WU00:FS00:Upload 7.12%
15:25:08:WU00:FS00:Upload 7.89%
15:25:14:WU00:FS00:Upload 9.71%
15:25:22:WU00:FS00:Upload 10.77%
15:26:04:WU00:FS00:Upload 11.35%
15:26:04:ERROR:WU00:FS00:Exception: Transfer failed
15:27:44:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
15:27:44:WU00:FS00:Uploading 64.98MiB to 66.170.111.50
15:27:44:WU00:FS00:Connecting to 66.170.111.50:8080
15:27:50:WU00:FS00:Upload 2.02%
15:27:56:WU00:FS00:Upload 3.27%
15:28:02:WU00:FS00:Upload 5.10%
15:28:08:WU00:FS00:Upload 7.02%
15:28:59:WU00:FS00:Upload 7.69%
15:29:14:WU00:FS00:Upload 8.18%
15:29:44:WU00:FS00:Upload 8.27%
15:29:44:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
15:29:44:WU00:FS00:Trying to send results to collection server
15:29:44:WU00:FS00:Uploading 64.98MiB to 158.130.118.25
15:29:44:WU00:FS00:Connecting to 158.130.118.25:8080
15:29:50:WU00:FS00:Upload 1.44%
15:29:56:WU00:FS00:Upload 3.27%
15:30:05:WU00:FS00:Upload 5.58%
15:30:11:WU00:FS00:Upload 6.54%
15:30:21:WU00:FS00:Upload 6.64%
15:30:27:WU00:FS00:Upload 8.18%
15:30:33:WU00:FS00:Upload 9.71%
15:30:39:WU00:FS00:Upload 11.06%
15:30:45:WU00:FS00:Upload 12.41%
15:30:51:WU00:FS00:Upload 13.56%
15:30:57:WU00:FS00:Upload 14.72%
15:31:03:WU00:FS00:Upload 15.39%
15:31:09:WU00:FS00:Upload 16.83%
15:31:15:WU00:FS00:Upload 17.99%
15:31:21:WU00:FS00:Upload 19.52%
15:31:27:WU00:FS00:Upload 21.45%
15:31:33:WU00:FS00:Upload 23.08%
15:31:39:WU00:FS00:Upload 24.72%
15:31:45:WU00:FS00:Upload 26.35%
15:31:51:WU00:FS00:Upload 27.99%
15:31:57:WU00:FS00:Upload 29.53%
15:32:03:WU00:FS00:Upload 31.16%
15:32:09:WU00:FS00:Upload 32.12%
15:32:15:WU00:FS00:Upload 33.76%
15:32:21:WU00:FS00:Upload 35.30%
15:32:27:WU00:FS00:Upload 37.13%
15:32:52:WU00:FS00:Upload 38.38%
15:33:09:WU00:FS00:Upload 38.47%
15:33:58:WU00:FS00:Upload 39.05%
15:33:58:ERROR:WU00:FS00:Exception: Transfer failed
15:34:35:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
15:34:35:WU00:FS00:Uploading 64.98MiB to 66.170.111.50
15:34:35:WU00:FS00:Connecting to 66.170.111.50:8080
15:34:41:WU00:FS00:Upload 2.02%
15:35:20:WU00:FS00:Upload 2.50%
15:35:20:WARNING:WU00:FS00:Exception: Failed to send results to work server: Transfer failed
15:35:20:WU00:FS00:Trying to send results to collection server
15:35:20:WU00:FS00:Uploading 64.98MiB to 158.130.118.25
15:35:20:WU00:FS00:Connecting to 158.130.118.25:8080
15:35:26:WU00:FS00:Upload 1.83%
15:35:32:WU00:FS00:Upload 3.75%
15:35:38:WU00:FS00:Upload 4.52%
15:35:44:WU00:FS00:Upload 5.96%
15:35:51:WU00:FS00:Upload 7.60%
15:35:57:WU00:FS00:Upload 9.23%
15:36:03:WU00:FS00:Upload 10.96%
15:36:09:WU00:FS00:Upload 12.60%
15:36:15:WU00:FS00:Upload 14.14%
15:36:21:WU00:FS00:Upload 15.87%
15:36:27:WU00:FS00:Upload 17.70%
15:36:36:WU00:FS00:Upload 18.18%
15:36:42:WU00:FS00:Upload 19.72%
15:36:48:WU00:FS00:Upload 20.97%
15:36:54:WU00:FS00:Upload 22.51%
15:37:00:WU00:FS00:Upload 24.14%
15:37:09:WU00:FS00:Upload 25.30%
15:37:15:WU00:FS00:Upload 27.03%
15:37:21:WU00:FS00:Upload 28.57%
15:37:27:WU00:FS00:Upload 30.49%
15:37:44:WU00:FS00:Upload 31.74%
15:38:36:WU00:FS00:Upload 31.84%
15:38:36:ERROR:WU00:FS00:Exception: Transfer failed
15:45:41:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:16706 run:162 clone:4 gen:39 core:0x22 unit:0x000000040000002700004142000000a2
15:45:41:WU00:FS00:Uploading 64.98MiB to 66.170.111.50
15:45:41:WU00:FS00:Connecting to 66.170.111.50:8080
15:45:47:WU00:FS00:Upload 2.40%
15:45:53:WU00:FS00:Upload 4.81%
15:46:32:WU00:FS00:Upload 6.35%
15:47:20:WU00:FS00:Upload 6.44%
15:47:26:WU00:FS00:Upload 8.37%
15:47:32:WU00:FS00:Upload 9.91%
15:48:36:WU00:FS00:Upload 10.29%
15:48:56:WU00:FS00:Upload 10.87%
15:49:13:WU00:FS00:Upload 11.93%
15:49:19:WU00:FS00:Upload 14.04%
15:49:26:WU00:FS00:Upload 15.97%
15:50:18:WU00:FS00:Upload 16.64%
15:51:05:WU00:FS00:Upload 16.74%
15:51:11:WU00:FS00:Upload 17.89%
15:51:17:WU00:FS00:Upload 19.91%
15:51:23:WU00:FS00:Upload 22.22%
15:51:39:WU00:FS00:Upload 22.70%
15:52:46:WU00:FS00:Upload 22.79%
15:53:28:WU00:FS00:Upload 22.89%
15:53:35:WU00:FS00:Upload 24.72%
15:54:02:WU00:FS00:Upload 25.30%
15:54:08:WU00:FS00:Upload 27.12%
15:54:14:WU00:FS00:Upload 28.76%
15:54:38:WU00:FS00:Upload 29.33%
15:54:44:WU00:FS00:Upload 31.26%
15:54:54:WU00:FS00:Upload 33.18%
15:55:00:WU00:FS00:Upload 35.01%
15:55:52:WU00:FS00:Upload 35.97%
15:55:58:WU00:FS00:Upload 37.80%
15:56:04:WU00:FS00:Upload 39.82%
15:56:10:WU00:FS00:Upload 41.84%
15:56:40:WU00:FS00:Upload 42.61%
15:57:45:WU00:FS00:Upload 42.70%
15:58:50:WU00:FS00:Upload 42.80%
15:58:56:WU00:FS00:Upload 44.53%
15:59:06:WU00:FS00:Upload 46.65%
15:59:12:WU00:FS00:Upload 48.67%
15:59:18:WU00:FS00:Upload 50.30%
15:59:45:WU00:FS00:Upload 51.07%
16:00:58:WU00:FS00:Upload 51.17%
16:01:41:WU00:FS00:Upload 51.84%
16:02:08:WU00:FS00:Upload 51.94%
16:02:14:WU00:FS00:Upload 53.76%
16:02:21:WU00:FS00:Upload 55.50%
16:04:31:WU00:FS00:Upload 56.26%
16:04:58:WU00:FS00:Upload 56.36%
16:05:04:WU00:FS00:Upload 57.13%
16:05:10:WU00:FS00:Upload 59.15%
16:06:30:WU00:FS00:Upload 60.40%
16:06:59:WU00:FS00:Upload 60.50%
16:07:09:WU00:FS00:Upload 60.98%
16:08:25:WU00:FS00:Upload 61.07%
16:08:31:WU00:FS00:Upload 63.00%
16:08:41:WU00:FS00:Upload 64.82%
16:08:47:WU00:FS00:Upload 66.65%
16:08:57:WU00:FS00:Upload 68.29%
16:09:03:WU00:FS00:Upload 70.21%
16:09:27:WU00:FS00:Upload 70.88%
16:09:40:WU00:FS00:Upload 71.27%
16:09:46:WU00:FS00:Upload 73.10%
16:09:52:WU00:FS00:Upload 75.12%
16:09:58:WU00:FS00:Upload 75.98%
16:10:05:WU00:FS00:Upload 77.90%
16:10:30:WU00:FS00:Upload 78.48%
16:11:02:WU00:FS00:Upload 78.58%
16:11:08:WU00:FS00:Upload 79.92%
16:11:14:WU00:FS00:Upload 81.94%
16:11:20:WU00:FS00:Upload 83.96%
16:11:50:WU00:FS00:Upload 85.12%
16:13:16:WU00:FS00:Upload 85.21%
16:13:37:WU00:FS00:Upload 85.79%
16:14:51:WU00:FS00:Upload 85.89%
16:14:57:WU00:FS00:Upload 87.72%
16:15:05:WU00:FS00:Upload 89.64%
16:15:11:WU00:FS00:Upload 91.56%
16:15:17:WU00:FS00:Upload 93.29%
16:16:27:WU00:FS00:Upload 94.06%
16:17:03:WU00:FS00:Upload 94.16%
16:17:09:WU00:FS00:Upload 96.28%
16:17:15:WU00:FS00:Upload 97.72%
16:18:40:WU00:FS00:Upload 98.39%

Uploaded at location with the faster ISP:

Code: Select all

10:54:53:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:16705 run:63 clone:1 gen:58 core:0x22 unit:0x000000010000003a000041410000003f
10:54:53:WU00:FS01:Uploading 65.08MiB to 66.170.111.50
10:54:53:WU00:FS01:Connecting to 66.170.111.50:8080
10:54:59:WU00:FS01:Upload 9.99%
10:55:05:WU00:FS01:Upload 40.43%
10:55:17:WU00:FS01:Upload 61.75%
10:55:28:WU00:FS01:Upload 61.84%
10:55:39:WU00:FS01:Upload 61.94%
10:55:48:WU00:FS01:Upload 62.80%
10:56:45:WU00:FS01:Upload 62.90%
10:56:45:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
10:56:45:WU00:FS01:Trying to send results to collection server
10:56:45:WU00:FS01:Uploading 65.08MiB to 128.174.73.74
10:56:45:WU00:FS01:Connecting to 128.174.73.74:8080
10:56:51:WU00:FS01:Upload 62.13%
10:56:59:WU00:FS01:Upload complete
10:56:59:WU00:FS01:Server responded WORK_ACK (400)
10:56:59:WU00:FS01:Final credit estimate, 836594.00 points
10:56:59:WU00:FS01:Cleaning up

[toTOW]

Unfortunately, the time-out values you see on your slower ISP are all over the place, so you can't blame the servers for it ... if the servers were to blame, you should see a fixed value for every time-out on this server (for instance 15 minutes or 30 minutes).

Maybe this ISP is too slow, it is dropping too many packets or it has poor peering links ... did you try to use a VPN to see it helps with the error rate ?

[jrsynow]

I just setup the workstation with the slower ISP to route all traffic through a VPN connection to the faster location, eager to see if this helps. The slower ISP has an upload rate that usually falls between 1.5-2 Mbps, so it should be able to upload the 65MB result info in less than 10 minutes if the connection isn't being throttled. I have run speed tests in parallel when I've caught the slowdowns happening and have seen the bandwidth I'm expecting. I'm not an expert on networking, so if there are some other tests that can be run when I'm experiencing these slowdowns that might shed light on an issue, please let me know.

[BobWilliams757]

I've had a couple of failures with this same server lately. The first two were reassigned after an hour, after spitting out the error message...

11:22:34:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed

I just had a slowdown on the same server today, this time clearing in only a few minutes...

Code: Select all

  21:05:58:WU00:FS01:Upload 17.46%
21:06:09:WU00:FS01:Upload 17.60%
21:06:19:WU00:FS01:Upload 17.75%
21:06:32:WU00:FS01:Upload 17.90%
21:06:45:WU00:FS01:Upload 18.05%
21:06:58:WU00:FS01:Upload 18.20%
21:07:09:WU00:FS01:Upload 18.34%
21:07:20:WU00:FS01:Upload 18.49%
21:07:31:WU00:FS01:Upload 18.64%
21:07:42:WU00:FS01:Upload 18.79%
21:07:52:WU00:FS01:Upload 18.94%
21:08:04:WU00:FS01:Upload 19.08%
21:08:16:WU00:FS01:Upload 19.23%
21:08:29:WU00:FS01:Upload 19.38%
21:08:41:WU00:FS01:Upload 19.53%
21:08:51:WU00:FS01:Upload 19.68%
21:09:00:WU00:FS01:Upload 22.93%
21:09:15:WU00:FS01:Upload 24.41%
21:09:26:WU00:FS01:Upload 24.56%
21:09:37:WU00:FS01:Upload 24.71%
21:09:50:WU00:FS01:Upload 26.18%
21:09:54:WU01:FS01:0x22:Completed 50000 out of 5000000 steps (1%)
21:10:02:WU00:FS01:Upload 26.33%
21:10:15:WU00:FS01:Upload 26.48%
21:10:27:WU00:FS01:Upload 26.63%
21:10:39:WU00:FS01:Upload 26.78%
21:10:50:WU00:FS01:Upload 26.92%
21:11:02:WU00:FS01:Upload 27.07%
21:11:14:WU00:FS01:Upload 27.22%
21:11:26:WU00:FS01:Upload 27.37%
21:11:36:WU00:FS01:Upload 27.52%
21:11:44:WU00:FS01:Upload 27.66%
21:11:55:WU00:FS01:Upload 92.61%
21:12:08:WU00:FS01:Upload 92.76%
21:12:21:WU00:FS01:Upload 92.90%
21:12:27:WU00:FS01:Upload 95.71%
21:12:37:WU00:FS01:Upload complete   

The server has disk space, and I have plenty of bandwidth for the uploads (80+ Mbps), so I'm wondering if something else might be going on with the server. The first time I noticed it after the fact, the second I noticed it in delay mode and checked connectivity, the third time I did a speed test.

[jrsynow]

I'm seeing the connection issues with my faster ISP as well, so I would think this would rule out both ISPs I'm using to upload results. Could there be bandwidth/client connection limits we're running into on the server?

Code: Select all

18:24:37:WU01:FS00:0x22:Folding@home Core Shutdown: FINISHED_UNIT
18:24:39:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
18:24:39:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:16572 run:6 clone:67 gen:82 core:0x22 unit:0x0000004300000052000040bc00000006
18:24:39:WU01:FS00:Uploading 42.25MiB to 66.170.111.50
18:24:39:WU01:FS00:Connecting to 66.170.111.50:8080
18:24:45:WU01:FS00:Upload 16.57%
18:24:57:WU01:FS00:Upload 16.72%
18:25:07:WU01:FS00:Upload 16.86%
18:25:18:WU01:FS00:Upload 17.01%
18:25:28:WU01:FS00:Upload 17.16%
18:25:39:WU01:FS00:Upload 17.31%
18:25:49:WU01:FS00:Upload 17.46%
18:25:59:WU01:FS00:Upload 17.60%
18:26:11:WU01:FS00:Upload 17.75%
18:26:22:WU01:FS00:Upload 20.42%
18:26:33:WU01:FS00:Upload 20.56%
18:26:44:WU01:FS00:Upload 20.71%
18:26:57:WU01:FS00:Upload 20.86%
18:27:08:WU01:FS00:Upload 21.01%
18:27:19:WU01:FS00:Upload 21.15%
18:27:33:WU01:FS00:Upload 21.30%
18:28:30:WU01:FS00:Upload 21.45%
18:28:30:WARNING:WU01:FS00:Exception: Failed to send results to work server: Transfer failed
18:28:30:WU01:FS00:Trying to send results to collection server
18:28:30:WU01:FS00:Uploading 42.25MiB to 128.174.73.74
18:28:30:WU01:FS00:Connecting to 128.174.73.74:8080
18:28:40:WU01:FS00:Upload complete
18:28:40:WU01:FS00:Server responded WORK_ACK (400)
18:28:40:WU01:FS00:Final credit estimate, 880687.00 points
18:28:40:WU01:FS00:Cleaning up

[BobWilliams757]

I've had that server create both a very slow upload as we have shown, to allowing nothing but short periods of upload.

Code: Select all

10:21:37:WU00:FS01:Uploading 38.27MiB to 66.170.111.50
10:21:37:WU00:FS01:Connecting to 66.170.111.50:8080

10:21:48:WU00:FS01:Upload 19.11%
10:22:00:WU00:FS01:Upload 19.27%
10:22:12:WU00:FS01:Upload 19.43%
10:22:23:WU00:FS01:Upload 19.60%
10:22:34:WU00:FS01:Upload 19.76%
10:22:47:WU00:FS01:Upload 19.92%
10:23:00:WU00:FS01:Upload 20.09%
10:23:01:WU01:FS01:0x22:Completed 50000 out of 5000000 steps (1%)
10:23:12:WU00:FS01:Upload 20.25%
10:23:25:WU00:FS01:Upload 20.41%
10:23:38:WU00:FS01:Upload 20.58%
10:23:50:WU00:FS01:Upload 20.74%
10:24:02:WU00:FS01:Upload 20.90%
10:24:15:WU00:FS01:Upload 21.07%
10:24:17:WU01:FS01:0x22:Completed 100000 out of 5000000 steps (2%)
10:24:17:WU01:FS01:0x22:Checkpoint completed at step 100000
10:24:27:WU00:FS01:Upload 21.23%
10:24:40:WU00:FS01:Upload 21.39%
10:24:52:WU00:FS01:Upload 21.56%
10:25:04:WU00:FS01:Upload 21.72%
10:25:16:WU00:FS01:Upload 21.88%
10:25:28:WU00:FS01:Upload 22.05%
10:25:33:WU01:FS01:0x22:Completed 150000 out of 5000000 steps (3%)
10:25:40:WU00:FS01:Upload 25.97%
10:25:52:WU00:FS01:Upload 26.13%
10:26:04:WU00:FS01:Upload 26.29%
10:26:16:WU00:FS01:Upload 26.46%
10:26:27:WU00:FS01:Upload 26.62%
10:26:40:WU00:FS01:Upload 26.78%
10:26:49:WU01:FS01:0x22:Completed 200000 out of 5000000 steps (4%)
10:26:49:WU01:FS01:0x22:Checkpoint completed at step 200000
10:26:51:WU00:FS01:Upload 26.95%
10:27:03:WU00:FS01:Upload 27.11%
10:27:16:WU00:FS01:Upload 27.27%
10:27:28:WU00:FS01:Upload 27.44%
10:27:40:WU00:FS01:Upload 27.60%
10:27:52:WU00:FS01:Upload 27.76%
10:28:04:WU00:FS01:Upload 27.93%
10:28:05:WU01:FS01:0x22:Completed 250000 out of 5000000 steps (5%)
10:28:16:WU00:FS01:Upload 28.09%
10:28:23:WU00:FS01:Upload 28.25%
10:28:32:WU00:FS01:Upload 31.52%
10:28:44:WU00:FS01:Upload 31.68%

10:46:21:WU00:FS01:Upload 31.85%

11:05:27:WU00:FS01:Upload 32.01%

11:22:34:WU00:FS01:Upload 32.17%
11:22:34:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:22:34:WU00:FS01:Trying to send results to collection server
11:22:34:WU00:FS01:Uploading 38.27MiB to 131.239.113.97
11:22:34:WU00:FS01:Connecting to 131.239.113.97:8080
11:22:40:WU00:FS01:Upload 2.78%
11:22:51:WU00:FS01:Upload 4.41%
11:22:57:WU00:FS01:Upload 86.88%
11:22:59:WU00:FS01:Upload complete
11:22:59:WU00:FS01:Server responded WORK_ACK (400)

Everywhere I left a space is simply normal work unit progress and checkpoints, no upload info. IIRC on one other work unit there were quick bursts of connectivity that might allow 15-20 percent to transmit, then it would sit virtually idle for another 10-15 minutes, transmit another burst, etc.... but it still didn't finish in an hour and switched to another server.

[BobWilliams757]

Once again today...

Code: Select all

19:35:05:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:16571 run:1 clone:76 gen:124 core:0x22 unit:0x0000004c0000007c000040bb00000001
19:35:05:WU00:FS01:Uploading 42.29MiB to 66.170.111.50
19:35:05:WU00:FS01:Connecting to 66.170.111.50:8080
19:35:05:WU01:FS01:Starting

19:35:26:WU00:FS01:Upload 19.21%
19:35:37:WU00:FS01:Upload 19.36%
19:35:47:WU00:FS01:Upload 19.51%
19:35:57:WU00:FS01:Upload 19.66%
19:36:13:WU00:FS01:Upload 21.73%
19:36:23:WU00:FS01:Upload 21.88%
19:36:35:WU00:FS01:Upload 22.02%
19:36:46:WU00:FS01:Upload 22.17%

19:40:26:WU00:FS01:Upload 22.32%

19:59:33:WU00:FS01:Upload 22.47%

20:18:40:WU00:FS01:Upload 22.61%

20:35:05:WU00:FS01:Upload 22.76%
20:35:06:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
20:35:06:WU00:FS01:Trying to send results to collection server
20:35:06:WU00:FS01:Uploading 42.29MiB to 128.174.73.74
20:35:06:WU00:FS01:Connecting to 128.174.73.74:8080
20:35:12:WU00:FS01:Upload 76.42%
20:35:17:WU00:FS01:Upload complete
20:35:18:WU00:FS01:Server responded WORK_ACK (400)
20:35:18:WU00:FS01:Final credit estimate, 477323.00 points