Issue with 129.74.85.15/16

[Agrovale]

I have 13 work units that are trying to upload to this collection server, they all seem to be getting partially completed then failing. Here's a log for one of the work units, they are all similar.

Code: Select all

*********************** Log Started 2012-11-14T13:34:34Z ***********************
13:56:50:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
13:56:50:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
13:56:50:WU01:FS01:Connecting to 129.74.85.15:8080
14:00:35:WU01:FS01:Upload 0.37%
14:00:35:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
14:00:35:WU01:FS01:Trying to send results to collection server
14:00:35:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
14:00:35:WU01:FS01:Connecting to 129.74.85.16:8080
14:04:20:WU01:FS01:Upload 0.37%
14:04:20:ERROR:WU01:FS01:Exception: Transfer failed
14:06:34:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
14:06:34:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
14:06:34:WU01:FS01:Connecting to 129.74.85.15:8080
14:10:19:WU01:FS01:Upload 0.37%
14:10:19:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
14:10:19:WU01:FS01:Trying to send results to collection server
14:10:19:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
14:10:19:WU01:FS01:Connecting to 129.74.85.16:8080
14:18:51:WU01:FS01:Upload 0.37%
14:19:51:WU01:FS01:Upload 0.75%
14:19:51:ERROR:WU01:FS01:Exception: Transfer failed
14:25:46:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
14:25:46:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
14:25:46:WU01:FS01:Connecting to 129.74.85.15:8080
14:33:31:WU01:FS01:Upload 0.37%
14:33:31:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
14:33:31:WU01:FS01:Trying to send results to collection server
14:33:31:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
14:33:31:WU01:FS01:Connecting to 129.74.85.16:8080
14:38:16:WU01:FS01:Upload 0.37%
14:38:16:ERROR:WU01:FS01:Exception: Transfer failed
15:25:46:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
15:25:46:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
15:25:46:WU01:FS01:Connecting to 129.74.85.15:8080
15:29:31:WU01:FS01:Upload 0.37%
15:29:31:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
15:29:31:WU01:FS01:Trying to send results to collection server
15:29:31:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
15:29:31:WU01:FS01:Connecting to 129.74.85.16:8080
15:33:16:WU01:FS01:Upload 0.37%
15:33:16:ERROR:WU01:FS01:Exception: Transfer failed
15:58:38:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
15:58:38:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
15:58:38:WU01:FS01:Connecting to 129.74.85.15:8080
16:04:28:WU01:FS01:Upload 0.37%
16:06:28:WU01:FS01:Upload 0.75%
16:06:28:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
16:06:28:WU01:FS01:Trying to send results to collection server
16:06:28:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
16:06:28:WU01:FS01:Connecting to 129.74.85.16:8080
16:14:14:WU01:FS01:Upload 0.37%
16:14:14:ERROR:WU01:FS01:Exception: Transfer failed
16:25:12:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
16:25:12:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
16:25:12:WU01:FS01:Connecting to 129.74.85.15:8080
16:32:57:WU01:FS01:Upload 0.37%
16:32:57:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
16:32:57:WU01:FS01:Trying to send results to collection server
16:32:57:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
16:32:57:WU01:FS01:Connecting to 129.74.85.16:8080
16:34:19:WU01:FS01:Upload 0.37%
16:34:19:ERROR:WU01:FS01:Exception: Transfer failed
17:17:19:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
17:17:19:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
17:17:19:WU01:FS01:Connecting to 129.74.85.15:8080
17:26:04:WU01:FS01:Upload 0.37%
17:26:04:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
17:26:04:WU01:FS01:Trying to send results to collection server
17:26:04:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
17:26:04:WU01:FS01:Connecting to 129.74.85.16:8080
17:43:51:WU01:FS01:Upload 0.37%
17:43:51:ERROR:WU01:FS01:Exception: Transfer failed
18:09:08:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
18:09:08:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
18:09:08:WU01:FS01:Connecting to 129.74.85.15:8080
18:16:53:WU01:FS01:Upload 0.37%
18:16:53:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
18:16:53:WU01:FS01:Trying to send results to collection server
18:16:53:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
18:16:53:WU01:FS01:Connecting to 129.74.85.16:8080
18:18:15:WU01:FS01:Upload 0.37%
18:18:15:ERROR:WU01:FS01:Exception: Transfer failed
19:03:15:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
19:03:15:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
19:03:15:WU01:FS01:Connecting to 129.74.85.15:8080
19:15:01:WU01:FS01:Upload 0.37%
19:15:01:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
19:15:01:WU01:FS01:Trying to send results to collection server
19:15:01:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
19:15:01:WU01:FS01:Connecting to 129.74.85.16:8080
19:19:46:WU01:FS01:Upload 0.37%
19:19:46:ERROR:WU01:FS01:Exception: Transfer failed
******************************** Date: 14/11/12 ********************************
19:52:53:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
19:52:53:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
19:52:53:WU01:FS01:Connecting to 129.74.85.15:8080
19:56:38:WU01:FS01:Upload 0.37%
19:56:38:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
19:56:38:WU01:FS01:Trying to send results to collection server
19:56:38:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
19:56:38:WU01:FS01:Connecting to 129.74.85.16:8080
20:13:27:WU01:FS01:Upload 0.37%
20:27:30:WU01:FS01:Upload 0.75%
20:27:30:ERROR:WU01:FS01:Exception: Transfer failed
20:47:52:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
20:47:52:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
20:47:52:WU01:FS01:Connecting to 129.74.85.15:8080
20:51:37:WU01:FS01:Upload 0.37%
20:51:37:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
20:51:37:WU01:FS01:Trying to send results to collection server
20:51:37:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
20:51:37:WU01:FS01:Connecting to 129.74.85.16:8080
20:55:22:WU01:FS01:Upload 0.37%
20:55:22:ERROR:WU01:FS01:Exception: Transfer failed
22:14:48:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
22:14:48:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
22:14:48:WU01:FS01:Connecting to 129.74.85.15:8080
22:20:34:WU01:FS01:Upload 0.37%
22:20:34:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
22:20:34:WU01:FS01:Trying to send results to collection server
22:20:34:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
22:20:34:WU01:FS01:Connecting to 129.74.85.16:8080
22:28:20:WU01:FS01:Upload 0.37%
22:28:20:ERROR:WU01:FS01:Exception: Transfer failed
00:17:48:WU01:FS01:Sending unit results: id:01 state:SEND error:OK project:7029 run:0 clone:286 gen:1 core:0xa4 unit:0x000000030001329c50294c62406156a9
00:17:48:WU01:FS01:Uploading 16.69MiB to 129.74.85.15
00:17:48:WU01:FS01:Connecting to 129.74.85.15:8080
00:25:33:WU01:FS01:Upload 0.37%
00:25:33:WARNING:WU01:FS01:Exception: Failed to send results to work server: Transfer failed
00:25:33:WU01:FS01:Trying to send results to collection server
00:25:33:WU01:FS01:Uploading 16.69MiB to 129.74.85.16
00:25:33:WU01:FS01:Connecting to 129.74.85.16:8080
00:42:21:WU01:FS01:Upload 0.37%
00:56:25:WU01:FS01:Upload 0.75%
00:56:25:ERROR:WU01:FS01:Exception: Transfer failed

[katakaio]

I'm running into similar issues with my SMP work units.

Code: Select all

03:04:52:WU00:FS01:Connecting to assign3.stanford.edu:8080
03:04:52:WU00:FS01:News: Welcome to Folding@Home
03:04:52:WU00:FS01:Assigned to work server 129.74.85.15
03:04:52:WU00:FS01:Requesting new work unit for slot 01: READY smp:4 from 129.74.85.15
03:04:52:WU00:FS01:Connecting to 129.74.85.15:8080
03:05:13:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
03:05:13:WU00:FS01:Connecting to 129.74.85.15:80
03:05:34:ERROR:WU00:FS01:Exception: Failed to connect to 129.74.85.15:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

This has been going on for the last few days. Any thoughts?

[abdulwahidc]

Hi

Thanks for posting. I'll look into this issue tomorrow.

[bruce]

Welcome to foldingforum.org, Agrovale

The log that you posted does not show 7 different WUs trying to upload so I can only search for the one mentioned:
project:7029 run:0 clone:286 gen:1

It has been returned and successfully credited:
Hi Agrovale (team 14033),
Your WU (P7029 R0 C286 G1) was added to the stats database on 2012-11-19 17:08:12 PST for 3543.44 points of credit.

[funkydude]

2 projects not uploading for me:

Code: Select all

23:46:20:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:7037 run:0 clone:912 gen:4 core:0xa4 unit:0x000000070001329c502950b9bfd702a6
23:46:20:WU00:FS01:Uploading 16.80MiB to 129.74.85.15
23:46:20:WU00:FS01:Connecting to 129.74.85.15:8080
23:46:20:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:7011 run:1 clone:3 gen:27 core:0xa4 unit:0x0000005a0001329c4dfb829b52aff54a
23:46:20:WU02:FS01:Uploading 1.67MiB to 129.74.85.15
23:46:20:WU02:FS01:Connecting to 129.74.85.15:8080
23:46:27:WU03:FS01:0xa4:Completed 22500 out of 250000 steps  (9%)
23:46:48:WU00:FS01:Upload 0.74%
23:46:48:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
23:46:48:WU00:FS01:Trying to send results to collection server
23:46:48:WU00:FS01:Uploading 16.80MiB to 129.74.85.16
23:46:48:WU00:FS01:Connecting to 129.74.85.16:8080
23:46:51:WU02:FS01:Upload 7.51%
23:46:51:WARNING:WU02:FS01:Exception: Failed to send results to work server: Transfer failed
23:46:51:WU02:FS01:Trying to send results to collection server
23:46:51:WU02:FS01:Uploading 1.67MiB to 129.74.85.16
23:46:51:WU02:FS01:Connecting to 129.74.85.16:8080
23:47:16:WU00:FS01:Upload 0.74%
23:47:16:ERROR:WU00:FS01:Exception: Transfer failed
23:47:18:WU02:FS01:Upload 7.51%
23:47:18:ERROR:WU02:FS01:Exception: Transfer failed

[bruce]

katakaio:
Have you tried the debugging techniques listed in the "do this first" post in this forum? Other than that, what I'm saying to funkydude probably applies to you, too.

--------------------------------------------------------------
Welcome to foldingforum.org, funkydude.

Neither of those WUs have been successfully uploaded by you (assuming you're folding as "funkydude"). The logs do show a poor connection, but it's difficult to sort out what might be on the server end or if part of the problem might be near your local site.

I understand that the servers had some networking problems last evening (EST), but they had apparently been resolved this morning (EST). I can't tell if they are having more problems this evening. From what I can see, the servers are responding as expected but I can't really test everything since I don't really have any data to upload.

[funkydude]

Any sign of a "Poor connection" is nothing other than my upload bandwidth being maxed during results transfer, thus slowing the transfer. I have a high bandwidth expensive connection so it's nothing to do with it.

Just to test, I shut down all connections, restarted the router, and left F@H as the only program using my upload stream, still the same issue.

This is definitely a server problem, especially considering other work has (and is) uploading fine.

[bruce]

Checking your end of the connection was a reasonable thing to do since others seem to be able to use those server and the server managers have not found anything wrong on their end either. At this point it's a unsolved mystery.

[P5-133XL]

If your BW is maxed out from something other than folding (like torrenting) then that will produce poor connection symptoms regardless of how expensive or how high the BW is. Saturated BW causes delays and delays cause timeouts which cause packets to be lost and that causes applications to sit there waiting for packets that are never going to come just like a bad connection.

Your test of folding with a free upload stream is invalid because the DL stream is just as important for sending status messages back to the client. You can not just isolate one direction because data communication is two-way.

When dealing with a saturated connection you need to setup and use a QOS (quality of service) router that can prioritize streams (ports, IP addresses, applications) and/or leave some reserved BW (in both directions) for other non-specified traffic.

[funkydude]

If your BW is maxed out from something other than folding (like torrenting) then that will produce poor connection symptoms regardless of how expensive or how high the BW is. Saturated BW causes delays and delays cause timeouts which cause packets to be lost and that causes applications to sit there waiting for packets that are never going to come just like a bad connection.

Your test of folding with a free upload stream is invalid because the DL stream is just as important for sending status messages back to the client. You can not just isolate one direction because data communication is two-way.

When dealing with a saturated connection you need to setup and use a QOS (quality of service) router that can prioritize streams (ports, IP addresses, applications) and/or leave some reserved BW (in both directions) for other non-specified traffic.

Did you even bother reading my post or just skip to Mr. know-it-all mode? Please read it and realize that your entire post is pointless.

Checking your end of the connection was a reasonable thing to do since others seem to be able to use those server and the server managers have not found anything wrong on their end either. At this point it's a unsolved mystery.

In that case, how does one ditch these work units?

[bollix47]

Your log shows two WUs trying to upload at the same time. This may be part of the problem.

Rather than deleting them try the following:

Pause all the slots (right click on each slot and select Pause ... you need to be in the Advanced or Expert view for this ... i.e. not Novice). You could use the Pause button to do this but you're going to be selecting individual slots later so best to start the routine now.

Click on Start>All Programs>FAHClient>Data Directory>work

Copy the two client files to a backup location.

Pick one of the folders 00(preferred since that WU has been returned successfully by another folder) or 02, right click on it and select cut ... then go to the same backup location, right click there and select paste which should leave only one ready for upload in your work folder.

Restart the slot (right click on slot 01 and select Fold).

If it uploads then Pause the slot again and put the moved folder and the two client files back where they came from and restart the slot again(right click on the slot and select Fold).

If one or both upload great and if they don't you at least now know what needs to be deleted after pausing the slot ... just the folders (00 & 02) ... do not delete the two client files.

Restart any other slots that you may have paused ... right click on the slot and select Fold.

[brityank]

I noticed that I uploaded Project: 7010 (Run 2, Clone 117, Gen 44) at around 16:30GMT, but have no points update to show. I looked at the server and saw it is in Reject mode - so where did my points go?

Here's the pertinent info of the log:

Code: Select all

# Windows CPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.23

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\F@H2
Executable: C:\F@H2\F@H2.exe
Arguments: -local -verbosity 9 

[18:03:03] - Ask before connecting: No
[18:03:03] - User name: brityank (Team 36120)
[18:03:03] - User ID: 3B*******************
[18:03:03] - Machine ID: 2
[18:03:03] 
[18:03:03] Loaded queue successfully.
[18:03:03] 

 + + + + + + + + + + + +

[12:21:21] - Connecting to assignment server
[12:21:21] Connecting to http://assign.stanford.edu:8080/
[12:21:23] Posted data.
[12:21:23] Initial: 4A81; - Successful: assigned to (129.74.85.15).
[12:21:23] + News From Folding@Home: Welcome to Folding@Home
[12:21:23] Loaded queue successfully.
[12:21:23] Connecting to http://129.74.85.15:8080/
[12:21:24] Posted data.
[12:21:24] Initial: 0000; - Receiving payload (expected size: 55838)
[12:21:25] - Downloaded at ~54 kB/s
[12:21:25] - Averaged speed for that direction ~154 kB/s
[12:21:25] + Received work.
[12:21:25] Trying to send all finished work units
[12:21:25] + No unsent completed units remaining.
[12:21:25] + Closed connections
[12:21:25] 
[12:21:25] + Processing work unit
[12:21:25] Core required: FahCore_a4.exe
[12:21:25] Core found.
[12:21:25] Working on queue slot 03 [November 27 12:21:25 UTC]
[12:21:25] + Working ...
[12:21:25] - Calling '.\FahCore_a4.exe -dir work/ -suffix 03 -checkpoint 15 -verbose -lifeline 1112 -version 623'

[12:21:25] 
[12:21:25] *------------------------------*
[12:21:25] Folding@Home Gromacs GB Core
[12:21:25] Version 2.27 (Dec. 15, 2010)
[12:21:25] 
[12:21:25] Preparing to commence simulation
[12:21:25] - Looking at optimizations...
[12:21:25] - Created dyn
[12:21:25] - Files status OK
[12:21:25] - Expanded 55326 -> 208564 (decompressed 376.9 percent)
[12:21:25] Called DecompressByteArray: compressed_data_size=55326 data_size=208564, decompressed_data_size=208564 diff=0
[12:21:25] - Digital signature verified
[12:21:25] 
[12:21:25] Project: 7010 (Run 2, Clone 117, Gen 44)
[12:21:25] 
[12:21:25] Assembly optimizations on if available.
[12:21:25] Entering M.D.
[12:21:31] Mapping NT from 1 to 1 
[12:21:31] Completed 0 out of 10000000 steps  (0%)
[12:38:17] Completed 100000 out of 10000000 steps  (1%)

 + + + + + + + + + + + +

[16:30:12] Completed 10000000 out of 10000000 steps  (100%)
[16:30:12] DynamicWrapper: Finished Work Unit: sleep=10000
[16:30:22] 
[16:30:22] Finished Work Unit:
[16:30:22] - Reading up to 2153724 from "work/wudata_03.trr": Read 2153724
[16:30:22] trr file hash check passed.
[16:30:22] - Reading up to 223712 from "work/wudata_03.xtc": Read 223712
[16:30:22] xtc file hash check passed.
[16:30:22] edr file hash check passed.
[16:30:22] logfile size: 80711
[16:30:22] Leaving Run
[16:30:22] - Writing 2482619 bytes of core data to disk...
[16:30:23] Done: 2482107 -> 1843180 (compressed to 74.2 percent)
[16:30:23]   ... Done.
[16:30:23] - Shutting down core
[16:30:23] 
[16:30:23] Folding@home Core Shutdown: FINISHED_UNIT
[16:30:25] CoreStatus = 64 (100)
[16:30:25] Unit 3 finished with 89 percent of time to deadline remaining.
[16:30:25] Updated performance fraction: 0.879525
[16:30:25] Sending work to server
[16:30:25] Project: 7010 (Run 2, Clone 117, Gen 44)


[16:30:25] + Attempting to send results [November 28 16:30:25 UTC]
[16:30:25] - Reading file work/wuresults_03.dat from core
[16:30:25]   (Read 1843692 bytes from disk)
[16:30:25] Connecting to http://129.74.85.15:8080/
[16:30:58] Posted data.
[16:30:58] Initial: 0000; - Uploaded at ~54 kB/s
[16:30:58] - Averaged speed for that direction ~56 kB/s
[16:30:58] + Results successfully sent
[16:30:58] Thank you for your contribution to Folding@Home.
[16:30:58] + Number of Units Completed: 1045

Or do you do this to make sure I'm still around??

[bruce]

....Or do you do this to make sure I'm still around??

Oh, are you still around?

I do know that you're not the only one who is missing credits for WUs which were uploaded to those servers during that time period. As you can see, there were known server problems then.

Since there's no record that either you or the Mods can see, it will (probably) require someone manually searching specific server logs to figure out what happens. In the past, I've seen situations that appeared to be similar. The credits do seem to appear EVENTUALLY but that can take from a few days to several weeks. The credit will probably appear in rather large batches.

[brityank]

Oh, are you still around?

Thanks Bruce. Seems the older I get, the rounder I become


Appreciate the support and follow-up.