Unable to upload to server 66.170.111.50

[comixgoddess]

Here is the section of my log showing the initial upload attempt --

04:25:48:WU00:FS00:0xa7:Completed 125000 out of 125000 steps (100%)
04:25:59:WU00:FS00:0xa7:Saving result file ../logfile_01.txt
04:25:59:WU00:FS00:0xa7:Saving result file frame0.trr
04:25:59:WU00:FS00:0xa7:Saving result file frame0.xtc
04:25:59:WU00:FS00:0xa7:Saving result file md.log
04:25:59:WU00:FS00:0xa7:Saving result file science.log
04:25:59:WU00:FS00:0xa7:Folding@home Core Shutdown: FINISHED_UNIT
04:25:59:WU00:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
04:25:59:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:17411 run:0 clone:220 gen:0 core:0xa7 unit:0x000000dc000000000000440300000000
04:25:59:WU00:FS00:Uploading 9.94MiB to 66.170.111.50
04:25:59:WU00:FS00:Connecting to 66.170.111.50:8080
04:26:05:WU00:FS00:Upload 37.74%
04:26:11:WU00:FS00:Upload 80.52%
04:26:14:WU00:FS00:Upload complete
04:26:14:WU00:FS00:Server responded PLEASE_WAIT (464)
04:26:14:WARNING:WU00:FS00:Failed to send results, will try again later

So far, there have been 9 attempts to upload, all with the same result. According to the server stats page, the server is set to "Accept" and has 959.91GiB of space but no collection server.

[Neil-B]

Would recommend you check the wu status app and see if upload actually completed

[comixgoddess]

Would recommend you check the wu status app and see if upload actually completed

Thanks for the suggestion, Neil-B. I hadn't thought of that. Unfortunately, the WU status page is showing one completion that was returned on 2020-09-22 15:12:13 to user Telefonica_SA. Now up to 24 attempts.

[bruce]

The team that manages that server has been investigating what is (was?) wrong with that server. I don't have a progress report, but they have been aware that something needs (needed?) fixing.

[comixgoddess]

Unfortunately, this WU (project:17411 run:0 clone:220 gen:0) expired. My log now shows --

01:57:54:WARNING:WU00:FS00:Past final deadline 2020-12-20T01:57:53Z, dumping
01:57:54:WU00:FS00:Cleaning up

The time frame for this one seemed quite short --
Assigned 2020-12-18 T01:57:53Z
Timeout 2020-12-19 T01:57:53Z
Expiration 2020-12-20 T01:57:53Z

[bruce]

Sorry to hear that.... but it does happen.

[superpan]

Same here... after 27 hourly attempts similar to...

00:57:08:WU00:FS00:Sending unit results: id:00 state:SEND error:NO_ERROR project:17407 run:0 clone:798 gen:0 core:0xa7 unit:0x0000031e00000000000043ff00000000
00:57:08:WU00:FS00:Uploading 9.94MiB to 66.170.111.50
00:57:08:WU00:FS00:Connecting to 66.170.111.50:8080
00:57:14:WU00:FS00:Upload 19.49%
00:57:20:WU00:FS00:Upload 40.24%
00:57:26:WU00:FS00:Upload 60.37%
00:57:32:WU00:FS00:Upload 80.49%
00:57:39:WU00:FS00:Upload complete
00:57:39:WU00:FS00:Server responded PLEASE_WAIT (464)

finally...

WARNING:WU00:FS00:Past final deadline 2020-12-20T02:07:36Z, dumping

Seems to be working OK now.

[gunnarre]

This server is still not accepting work units. According to the server stats page, it has run out of disk space. https://apps.foldingathome.org/serverstats

Code: Select all

22:01:13:WU01:FS00:FahCore returned: FINISHED_UNIT (100 = 0x64)
22:01:13:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:01:13:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:01:13:WU01:FS00:Connecting to 66.170.111.50:8080
22:01:19:WU01:FS00:Upload 8.19%
22:01:25:WU01:FS00:Upload 16.37%
22:01:31:WU01:FS00:Upload 23.93%
22:01:37:WU01:FS00:Upload 30.85%
22:01:43:WU01:FS00:Upload 39.04%
22:01:49:WU01:FS00:Upload 49.11%
22:01:55:WU01:FS00:Upload 56.67%
22:02:01:WU01:FS00:Upload 64.23%
22:02:07:WU01:FS00:Upload 71.15%
22:02:13:WU01:FS00:Upload 77.45%
22:02:19:WU01:FS00:Upload 90.04%
22:02:24:WU01:FS00:Upload complete
22:02:24:WU01:FS00:Server responded PLEASE_WAIT (464)
22:02:24:WARNING:WU01:FS00:Failed to send results, will try again later
22:02:24:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:02:24:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:02:24:WU01:FS00:Connecting to 66.170.111.50:8080
22:02:30:WU01:FS00:Upload 9.45%
22:02:36:WU01:FS00:Upload 19.52%
22:02:42:WU01:FS00:Upload 32.11%
22:02:48:WU01:FS00:Upload 41.56%
22:02:54:WU01:FS00:Upload 45.97%
22:03:00:WU01:FS00:Upload 51.00%
22:03:06:WU01:FS00:Upload 57.93%
22:03:12:WU01:FS00:Upload 67.38%
22:03:18:WU01:FS00:Upload 77.45%
22:03:24:WU01:FS00:Upload 83.75%
22:03:30:WU01:FS00:Upload 90.67%
22:03:36:WU01:FS00:Upload 98.23%
22:03:37:WU01:FS00:Upload complete
22:03:38:WU01:FS00:Server responded PLEASE_WAIT (464)
22:03:38:WARNING:WU01:FS00:Failed to send results, will try again later
22:03:38:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:03:38:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:03:38:WU01:FS00:Connecting to 66.170.111.50:8080
22:03:44:WU01:FS00:Upload 9.45%
22:03:50:WU01:FS00:Upload 15.11%
22:03:56:WU01:FS00:Upload 21.41%
22:04:02:WU01:FS00:Upload 27.71%
22:04:08:WU01:FS00:Upload 34.63%
22:04:14:WU01:FS00:Upload 44.71%
22:04:20:WU01:FS00:Upload 53.52%
22:04:26:WU01:FS00:Upload 61.71%
22:04:32:WU01:FS00:Upload 69.89%
22:04:38:WU01:FS00:Upload 78.08%
22:04:44:WU01:FS00:Upload 85.01%
22:04:50:WU01:FS00:Upload 90.04%
22:04:56:WU01:FS00:Upload 98.86%
22:04:58:WU01:FS00:Upload complete
22:04:59:WU01:FS00:Server responded PLEASE_WAIT (464)
22:04:59:WARNING:WU01:FS00:Failed to send results, will try again later
22:05:15:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:05:15:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:05:15:WU01:FS00:Connecting to 66.170.111.50:8080
22:05:21:WU01:FS00:Upload 7.56%
22:05:27:WU01:FS00:Upload 13.22%
22:05:33:WU01:FS00:Upload 18.89%
22:05:39:WU01:FS00:Upload 25.19%
22:05:45:WU01:FS00:Upload 32.74%
22:05:51:WU01:FS00:Upload 40.30%
22:05:57:WU01:FS00:Upload 49.74%
22:06:03:WU01:FS00:Upload 59.19%
22:06:09:WU01:FS00:Upload 68.63%
22:06:15:WU01:FS00:Upload 78.08%
22:06:21:WU01:FS00:Upload 88.15%
22:06:27:WU01:FS00:Upload 94.45%
22:06:32:WU01:FS00:Upload complete
22:06:32:WU01:FS00:Server responded PLEASE_WAIT (464)
22:06:32:WARNING:WU01:FS00:Failed to send results, will try again later
22:07:52:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:07:52:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:07:52:WU01:FS00:Connecting to 66.170.111.50:8080
22:07:58:WU01:FS00:Upload 6.93%
22:08:04:WU01:FS00:Upload 12.59%
22:08:10:WU01:FS00:Upload 19.52%
22:08:16:WU01:FS00:Upload 26.45%
22:08:22:WU01:FS00:Upload 37.78%
22:08:28:WU01:FS00:Upload 48.48%
22:08:34:WU01:FS00:Upload 56.67%
22:08:40:WU01:FS00:Upload 67.38%
22:08:46:WU01:FS00:Upload 77.45%
22:08:52:WU01:FS00:Upload 86.27%
22:08:58:WU01:FS00:Upload 93.82%
22:09:03:WU01:FS00:Upload complete
22:09:03:WU01:FS00:Server responded PLEASE_WAIT (464)
22:09:03:WARNING:WU01:FS00:Failed to send results, will try again later
22:12:06:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:12:06:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:12:06:WU01:FS00:Connecting to 66.170.111.50:8080
22:12:12:WU01:FS00:Upload 11.96%
22:12:18:WU01:FS00:Upload 18.26%
22:12:24:WU01:FS00:Upload 24.56%
22:12:30:WU01:FS00:Upload 31.48%
22:12:36:WU01:FS00:Upload 37.78%
22:12:42:WU01:FS00:Upload 42.82%
22:12:48:WU01:FS00:Upload 53.52%
22:12:54:WU01:FS00:Upload 63.60%
22:13:00:WU01:FS00:Upload 72.41%
22:13:06:WU01:FS00:Upload 79.34%
22:13:12:WU01:FS00:Upload 90.04%
22:13:18:WU01:FS00:Upload 98.23%
22:13:20:WU01:FS00:Upload complete
22:13:20:WU01:FS00:Server responded PLEASE_WAIT (464)
22:13:20:WARNING:WU01:FS00:Failed to send results, will try again later
22:18:58:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:18:58:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:18:58:WU01:FS00:Connecting to 66.170.111.50:8080
22:19:04:WU01:FS00:Upload 11.33%
22:19:10:WU01:FS00:Upload 20.15%
22:19:16:WU01:FS00:Upload 27.71%
22:19:22:WU01:FS00:Upload 33.37%
22:19:28:WU01:FS00:Upload 40.30%
22:19:34:WU01:FS00:Upload 49.11%
22:19:40:WU01:FS00:Upload 58.56%
22:19:46:WU01:FS00:Upload 72.41%
22:19:52:WU01:FS00:Upload 80.60%
22:19:58:WU01:FS00:Upload 92.56%
22:20:01:WU01:FS00:Upload complete
22:20:01:WU01:FS00:Server responded PLEASE_WAIT (464)
22:20:01:WARNING:WU01:FS00:Failed to send results, will try again later
22:30:03:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:30:03:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:30:03:WU01:FS00:Connecting to 66.170.111.50:8080
22:30:09:WU01:FS00:Upload 4.41%
22:30:15:WU01:FS00:Upload 16.37%
22:30:21:WU01:FS00:Upload 23.30%
22:30:27:WU01:FS00:Upload 28.96%
22:30:33:WU01:FS00:Upload 37.15%
22:30:39:WU01:FS00:Upload 48.48%
22:30:45:WU01:FS00:Upload 57.93%
22:30:51:WU01:FS00:Upload 62.34%
22:30:57:WU01:FS00:Upload 68.00%
22:31:03:WU01:FS00:Upload 76.19%
22:31:09:WU01:FS00:Upload 84.38%
22:31:15:WU01:FS00:Upload 93.19%
22:31:21:WU01:FS00:Upload complete
22:31:21:WU01:FS00:Server responded PLEASE_WAIT (464)
22:31:21:WARNING:WU01:FS00:Failed to send results, will try again later
22:48:00:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
22:48:00:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
22:48:00:WU01:FS00:Connecting to 66.170.111.50:8080
22:48:06:WU01:FS00:Upload 13.85%
22:48:12:WU01:FS00:Upload 23.30%
22:48:18:WU01:FS00:Upload 34.00%
22:48:24:WU01:FS00:Upload 47.23%
22:48:30:WU01:FS00:Upload 60.45%
22:48:36:WU01:FS00:Upload 72.41%
22:48:42:WU01:FS00:Upload 82.49%
22:48:48:WU01:FS00:Upload 98.86%
22:48:49:WU01:FS00:Upload complete
22:48:49:WU01:FS00:Server responded PLEASE_WAIT (464)
22:48:49:WARNING:WU01:FS00:Failed to send results, will try again later
23:17:02:WU01:FS00:Sending unit results: id:01 state:SEND error:NO_ERROR project:17408 run:0 clone:124 gen:214 core:0xa7 unit:0x0000007c000000d60000440000000000
23:17:02:WU01:FS00:Uploading 9.93MiB to 66.170.111.50
23:17:02:WU01:FS00:Connecting to 66.170.111.50:8080
23:17:08:WU01:FS00:Upload 14.48%
23:17:14:WU01:FS00:Upload 26.45%
23:17:20:WU01:FS00:Upload 35.26%
23:17:26:WU01:FS00:Upload 50.37%
23:17:32:WU01:FS00:Upload 62.97%
23:17:38:WU01:FS00:Upload 77.45%
23:17:44:WU01:FS00:Upload 95.08%
23:17:46:WU01:FS00:Upload complete
23:17:46:WU01:FS00:Server responded PLEASE_WAIT (464)
23:17:46:WARNING:WU01:FS00:Failed to send results, will try again later

[Joe_H]

They are aware of the disk space having run out, and are working on it.

[Foxbat]

That's good to hear, my stuck WU {17415 (0, 14, 558)} for this server is on attempt 17 so far. The expiration is tomorrow at 13:24 UTC.

I'm curious, the Work Server is 66.170.111.50, but my FAH Client reports the Collection Server is 0.0.0.0, not an actual IPv4 address like my other WUs currently being Folded. Is this a "use any server" shortcut?

[Callaghan]

As above i too cannot upload to server 66.170.111.50 and have an expiration is tomorrow at 14:11:11Z. 17414 (0, 434, 480)

[Joe_H]

I'm curious, the Work Server is 66.170.111.50, but my FAH Client reports the Collection Server is 0.0.0.0, not an actual IPv4 address like my other WUs currently being Folded. Is this a "use any server" shortcut?

A CS reported as 0.0.0.0 means none was assigned at the time the WU was downloaded to your system. All WUs need to go back to their WS, a CS is not always set for use as a backup when the WS is unavailable.

[new08]

This is still sending out project work -as I wait for my last 17411 CPU unit to expire in two hours.
It's not the small points -it's the wasted effort and power usage.
I don't usually defer to F@H CPU work, but understood it was valued..
I never see this on BOINC, tbh.

[bruce]

When a WU passes the timeout, a duplicate is issued. I don't think there's a limit to the number of duplicates that might be reissued. Eventually one of them will be completed and the project will move on. (Only the first one is needed for science.)

Most or all of the other copies that are in-flight will eventually be completed and should receive credit. The duplicated logs and results will eventually be discarded. When that cleanup processing happens probably matters.

The server doesn't have a message saying "Somebody else completed that WU before you did so we're discarding your result" That message SHOULD NEVER be issued, because you did nothing wrong but the server should still silently discard the duplicate results after awarding you credit.

FAH was designed to avoid issuing duplicate WUs except when they're really needed. Shortening the timeout interval does increase the frequency of WU duplication. In any case, no more duplicates should be sent after the first result is accepted.

This sounds like an unforeseen result of shortened timeouts.

[susanreads]

I had this a few days ago, with the unit eventually being dumped. I've finished two different units since without a problem, then this morning I was assigned another unit for the same project, from the same work server. Can I expect the same thing to happen? I have to leave it running overnight in order to finish before the deadline, and I resent doing that if it's just going to be thrown away.