.WU not uploading

Moderators: Site Moderators, FAHC Science Team

Post Reply
silverpulser
Posts: 107
Joined: Sat Nov 10, 2012 9:06 am

.WU not uploading

Post by silverpulser »

Is this a temporary issue?

16:17:32:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18117 run:0 clone:66 gen:62 core:0x22 unit:0x000000420000003e000046c500000000
16:17:32:WU00:FS01:Uploading 11.01MiB to 34.72.228.44
16:17:32:WU00:FS01:Connecting to 34.72.228.44:8080
16:17:38:WU00:FS01:Upload 27.24%
16:17:44:WU00:FS01:Upload 56.75%
16:17:50:WU00:FS01:Upload 86.26%
16:17:52:WU00:FS01:Upload complete
16:17:52:WU00:FS01:Server responded PLEASE_WAIT (464)
16:17:52:WARNING:WU00:FS01:Failed to send results, will try again later
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: .WU not uploading

Post by Joe_H »

The WS is out of space, not an issue with the WU. See this topic - viewtopic.php?f=18&t=37549
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
silverpulser
Posts: 107
Joined: Sat Nov 10, 2012 9:06 am

Re: .WU not uploading

Post by silverpulser »

So we just sit and wait for a resolution. OK but will the proper credit be given?
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: .WU not uploading

Post by Neil-B »

Completed wus are credited based on the time it gets successfully uploaded to the work server ... if for some reason like now it is delayed then the credit will reduce to reflect this delay ... if it times out before the server receives it then no credit is given
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: .WU not uploading

Post by Neil-B »

The WS has been taken down to be sorted out ... however the CS issue was resolved and anyone with uploads that were not uploading to 34.72.228.44 should find the uploads have now succeeded to the CS.
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Oleg F
Posts: 7
Joined: Tue Nov 16, 2021 5:29 pm

Re: .WU not uploading

Post by Oleg F »

Persisting problem

03:45:53:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18201 run:27331 clone:0 gen:17 core:0x22 unit:0x00000000000000110000471900006ac3
03:45:53:WU00:FS01:Uploading 25.42MiB to 128.252.203.11
03:45:53:WU00:FS01:Connecting to 128.252.203.11:8080
03:45:59:WU00:FS01:Upload 1.72%
03:46:05:WU00:FS01:Upload 3.44%
03:46:11:WU00:FS01:Upload 5.16%
03:46:17:WU00:FS01:Upload 6.88%
03:46:38:WU00:FS01:Upload 7.62%
03:46:38:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
03:46:38:WU00:FS01:Trying to send results to collection server
03:46:38:WU00:FS01:Uploading 25.42MiB to 128.252.203.10
03:46:38:WU00:FS01:Connecting to 128.252.203.10:8080
03:46:44:WU00:FS01:Upload 1.48%
03:46:50:WU00:FS01:Upload 3.20%
03:46:56:WU00:FS01:Upload 4.92%
03:47:02:WU00:FS01:Upload 6.64%
03:47:24:WU00:FS01:Upload 7.62%
03:47:24:ERROR:WU00:FS01:Exception: Transfer failed
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: .WU not uploading

Post by Neil-B »

Different server ... Different Issue as this one is interrupting early in transfer ... Checked server and has ample space ... Looks potentially like some form of comms interruption ... I am getting uploads to that server going through regularly so might not be at server end ... If you have any A/V or DPI running it might be worth checking if that is causing the issue as it appears to keep failing at same point - or some form of traffic shaping ... stats don't show it as having been received already so doesn't look like a "short response but really uploaded then keeps retrying scenario"... When you say persistent problem do you mean this is still happening on this wu/rig - has it cleared yet?
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
jjmiller
Scientist
Posts: 137
Joined: Fri Apr 09, 2021 4:43 pm

Re: .WU not uploading

Post by jjmiller »

Oleg- thanks for posting! Are you still having this issue? We have had some rare issues with WUs not being returned to the server for reasons we haven't been able to track down yet. If you're still having this issue, we'd like to try a TCP capture to see if we can get an idea of what's going on.

Thanks,
Oleg F
Posts: 7
Joined: Tue Nov 16, 2021 5:29 pm

Re: .WU not uploading

Post by Oleg F »

Good Day!

Turned AV(Comodo) off, problem persists.

11:06:00:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18201 run:27331 clone:0 gen:17 core:0x22 unit:0x00000000000000110000471900006ac3
11:06:00:WU00:FS01:Uploading 25.42MiB to 128.252.203.11
11:06:00:WU00:FS01:Connecting to 128.252.203.11:8080
11:06:06:WU00:FS01:Upload 1.48%
11:06:12:WU00:FS01:Upload 3.20%
11:06:18:WU00:FS01:Upload 4.92%
11:06:24:WU00:FS01:Upload 6.64%
11:06:46:WU00:FS01:Upload 7.62%
11:06:46:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
11:06:46:WU00:FS01:Trying to send results to collection server
11:06:46:WU00:FS01:Uploading 25.42MiB to 128.252.203.10
11:06:46:WU00:FS01:Connecting to 128.252.203.10:8080
11:06:52:WU00:FS01:Upload 1.48%
11:06:58:WU00:FS01:Upload 3.20%
11:07:04:WU00:FS01:Upload 4.92%
11:07:10:WU00:FS01:Upload 6.39%
11:07:32:WU00:FS01:Upload 7.62%
11:07:33:ERROR:WU00:FS01:Exception: Transfer failed
Oleg F
Posts: 7
Joined: Tue Nov 16, 2021 5:29 pm

Re: .WU not uploading

Post by Oleg F »

Don't understand clearly, what is DPI
gunnarre
Posts: 559
Joined: Sun May 24, 2020 7:23 pm
Location: Norway

Re: .WU not uploading

Post by gunnarre »

Deep Packet Inspection - in other words a firewall in a router somewhere along the way (at your home network or ISP) which allows regular HTTP access to http://128.252.203.10 on port 80 and 8080, but finds the WU data itself to be suspicious. To check if that is the case, you might try a VPN, tethering your computer to your cell phone or using some other way to test with an alternative network connection.
Image
Online: GTX 1660 Super + occasional CPU folding in the cold.
Offline: Radeon HD 7770, GTX 1050 Ti 4G OC, RX580
jjmiller
Scientist
Posts: 137
Joined: Fri Apr 09, 2021 4:43 pm

Re: .WU not uploading

Post by jjmiller »

Ok- one additional suggestion while I hunt down whether we can do a TCP.

If you are on a Desktop computer, could you try navigating to "advanced settings" on FAHcontrol (configure-> Advanced) and see if the "Power, pause work while on battery power." box is checked? The one case we have documented of a "stuck" WU (if this is the same problem) being fixed was when a user unchecked this check box: viewtopic.php?f=18&t=37508&start=15#p353744.

In the meantime, I'll continue to try setting up a TCP on our end to see if we can get a better idea why the transfer is failing.
jjmiller
Scientist
Posts: 137
Joined: Fri Apr 09, 2021 4:43 pm

Re: .WU not uploading

Post by jjmiller »

Oleg,

I did manage to get a TCP capture setup. Would you be willing to post one of the following to help in our TCP analysis?

1) Windows for attempted returns from UTC: 2021-11-19 15:40 to 18:40. (e.g. in your most recent post 11:06:00-11:07:33)
2) Log file from the stuck WU trimmed to the time UTC: 2021-11-19 15:40 to 18:40

Thanks!
Oleg F
Posts: 7
Joined: Tue Nov 16, 2021 5:29 pm

Re: .WU not uploading

Post by Oleg F »

Good Day!
"Pause work while on battery power" checked, didn't touch it.
Yes, using desktop PC, Win7, Ryzen 3400G, WU done on AMD RX 570.
Deep Packet Inspection - well, doubtful, router or ISP aren't expected to prevent anything, it's Ukraine.
Cell phone internet - sorry, don't have one available now

17:06:01:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18201 run:27331 clone:0 gen:17 core:0x22 unit:0x00000000000000110000471900006ac3
17:06:01:WU00:FS01:Uploading 25.42MiB to 128.252.203.11
17:06:01:WU00:FS01:Connecting to 128.252.203.11:8080
17:06:13:WU00:FS01:Upload 0.49%
17:06:19:WU00:FS01:Upload 2.21%
17:06:25:WU00:FS01:Upload 3.93%
17:06:31:WU00:FS01:Upload 5.65%
17:06:37:WU00:FS01:Upload 7.38%
17:06:56:WU00:FS01:Upload 7.62%
17:06:56:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
17:06:56:WU00:FS01:Trying to send results to collection server
17:06:56:WU00:FS01:Uploading 25.42MiB to 128.252.203.10
17:06:56:WU00:FS01:Connecting to 128.252.203.10:8080
17:07:02:WU00:FS01:Upload 1.48%
17:07:08:WU00:FS01:Upload 2.95%
17:07:14:WU00:FS01:Upload 4.67%
17:07:20:WU00:FS01:Upload 6.39%
17:07:43:WU00:FS01:Upload 7.62%
17:07:43:ERROR:WU00:FS01:Exception: Transfer failed
18:06:01:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18201 run:27331 clone:0 gen:17 core:0x22 unit:0x00000000000000110000471900006ac3
18:06:01:WU00:FS01:Uploading 25.42MiB to 128.252.203.11
18:06:01:WU00:FS01:Connecting to 128.252.203.11:8080
18:06:07:WU00:FS01:Upload 1.48%
18:06:13:WU00:FS01:Upload 3.20%
18:06:19:WU00:FS01:Upload 4.92%
18:06:25:WU00:FS01:Upload 6.39%
18:06:48:WU00:FS01:Upload 7.62%
18:06:48:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
18:06:48:WU00:FS01:Trying to send results to collection server
18:06:48:WU00:FS01:Uploading 25.42MiB to 128.252.203.10
18:06:48:WU00:FS01:Connecting to 128.252.203.10:8080
18:06:54:WU00:FS01:Upload 1.72%
18:07:00:WU00:FS01:Upload 3.44%
18:07:06:WU00:FS01:Upload 4.92%
18:07:12:WU00:FS01:Upload 6.64%
18:07:34:WU00:FS01:Upload 7.62%
18:07:34:ERROR:WU00:FS01:Exception: Transfer failed
19:06:02:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18201 run:27331 clone:0 gen:17 core:0x22 unit:0x00000000000000110000471900006ac3
19:06:02:WU00:FS01:Uploading 25.42MiB to 128.252.203.11
19:06:02:WU00:FS01:Connecting to 128.252.203.11:8080
19:06:08:WU00:FS01:Upload 1.72%
19:06:14:WU00:FS01:Upload 3.44%
19:06:20:WU00:FS01:Upload 5.16%
19:06:26:WU00:FS01:Upload 6.64%
19:06:48:WU00:FS01:Upload 7.62%
19:06:48:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
19:06:48:WU00:FS01:Trying to send results to collection server
19:06:48:WU00:FS01:Uploading 25.42MiB to 128.252.203.10
19:06:48:WU00:FS01:Connecting to 128.252.203.10:8080
19:06:54:WU00:FS01:Upload 1.48%
19:07:00:WU00:FS01:Upload 3.20%
19:07:06:WU00:FS01:Upload 4.92%
19:07:12:WU00:FS01:Upload 6.15%
19:07:42:WU00:FS01:Upload 7.62%
19:07:43:ERROR:WU00:FS01:Exception: Transfer failed
20:06:02:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18201 run:27331 clone:0 gen:17 core:0x22 unit:0x00000000000000110000471900006ac3
20:06:02:WU00:FS01:Uploading 25.42MiB to 128.252.203.11
20:06:02:WU00:FS01:Connecting to 128.252.203.11:8080
20:06:08:WU00:FS01:Upload 1.72%
20:06:14:WU00:FS01:Upload 3.44%
20:06:20:WU00:FS01:Upload 5.16%
20:06:26:WU00:FS01:Upload 6.64%
20:06:48:WU00:FS01:Upload 7.62%
20:06:48:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
20:06:48:WU00:FS01:Trying to send results to collection server
20:06:48:WU00:FS01:Uploading 25.42MiB to 128.252.203.10
20:06:48:WU00:FS01:Connecting to 128.252.203.10:8080
20:06:54:WU00:FS01:Upload 1.72%
20:07:00:WU00:FS01:Upload 3.44%
20:07:06:WU00:FS01:Upload 4.92%
20:07:12:WU00:FS01:Upload 6.64%
20:07:34:WU00:FS01:Upload 7.62%
20:07:34:ERROR:WU00:FS01:Exception: Transfer failed

Don't see "file attach" form, uploaded WU here, for just-in-case case.

Code: Select all

https://fex.net/s/x6feolt
file:00.7z
pass:123FAH
Oleg F
Posts: 7
Joined: Tue Nov 16, 2021 5:29 pm

Re: .WU not uploading

Post by Oleg F »

Finally:

08:06:14:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:18201 run:27331 clone:0 gen:17 core:0x22 unit:0x00000000000000110000471900006ac3
08:06:14:WU00:FS01:Uploading 25.42MiB to 128.252.203.11
08:06:14:WU00:FS01:Connecting to 128.252.203.11:8080
08:06:20:WU00:FS01:Upload 1.72%
08:06:26:WU00:FS01:Upload 3.44%
08:06:32:WU00:FS01:Upload 5.16%
08:06:38:WU00:FS01:Upload 6.88%
08:07:00:WU00:FS01:Upload 7.62%
08:07:00:WARNING:WU00:FS01:Exception: Failed to send results to work server: Transfer failed
08:07:00:WU00:FS01:Trying to send results to collection server
08:07:00:WU00:FS01:Uploading 25.42MiB to 128.252.203.10
08:07:00:WU00:FS01:Connecting to 128.252.203.10:8080
08:07:07:WU00:FS01:Upload 1.72%
08:07:13:WU00:FS01:Upload 3.44%
08:07:19:WU00:FS01:Upload 5.16%
08:07:25:WU00:FS01:Upload 7.13%
08:07:46:WU00:FS01:Upload 7.62%
08:07:46:ERROR:WU00:FS01:Exception: Transfer failed
08:19:48:WARNING:WU00:FS01:Past final deadline 2021-11-22T08:19:47Z, dumping
08:19:48:WU00:FS01:Cleaning up
Post Reply