Upload to 66.170.111.50:8080 failing repeatedly

[jrsynow]

... The first major question is, does the problem lie in a) the WS software, b) the Linux kernel or c) the VMWare network.
...

I noticed the same issues with the Temple University servers back in November of last year(viewtopic.php?t=38869, Discord support channel 11/10/2022), so that could rule out point C. Up until ~Nov. 2022 uploads were always rock solid on my end.

[Lazvon]

I am happy to take a look. But sounds like server side capture is same as client side. Drop a PM on where to grab it.

Things I would look at:

• VMWare: are there 2 (or more) virtual network cards listed in the virtual machine itself? If so, can it plumbed for a single NIC as far as Linux is concerned?
• VMWare: what packet stats does vCenter or ESXi directly show for the virtual and physical NICs?
• Linux: I assume you have already looked at the Linux interface stats (netstat -i (and -s, and -s -t), ethtool -S, ip -s link show (never used this myself)), to validate you aren’t dropping in Linux itself?
• Linux: does sar or iostat show any IOPS to the disk problems? VMWare view of IO? (And CPU and Mem too?)
• Switch: any drops/overruns/discard counters increasing for the VMWare host?
• Switch/Router/Firewall/ISP: assume none of those components are set for per packet load balancing, and they are set for per flow, right?

[BobWilliams757]

I know nothing at all about the proper ways to troubleshoot such things, but would gladly help comb through data if anyone can break it down to what I need to look for within that data.

In my case, it has been inconsistent and occasional slowdowns, and they often have "bursts" of decent file transfer followed by periods of very slow transfer.

I'll refrain from posting any logs unless it shows a longer slowdown that is having a negative impact on the returns. I haven't had delays any longer than a few minutes total lately, and no more issues of the one hour until it throws an error and switches to a new server.

[Lazvon]

Is this on VMWare Cloud? Is it a donated VM?

I spend about $60m a year with VMWare. Couple of the folks who work me are on their CAB. Generally a good partner for me and they like to keep me happy so I don’t go somewhere else. I can probably get one of their better engineers responsible for their cloud instances to help me troubleshoot from ESXi to the physical host to the network to the Internet of so.

Their ISP’s (NTT) routers are certainly busy (slow ICMP TTL Expired messages while tracerouting) assuming this is in Seattle and doesn’t go somewhere else. So could totally just be an ISP issue.

Again, happy to take a look at the PCAP. It is something I have been doing since the mid-90s (well, not PCAP, was Network General Sniffers back then). Not sure it will show anything different than I have seen on client side.

Essentially, either there are 2+ links and per packet load balancing is turned on so a longer path and shorter path get used, and thus Out of Order packets. Or, there is severe caching along the path with congestion and buffers aren’t being served in order for the OOO issues, and/or some kind of random early discard dropping packets on the floor to try and sliver congestion, but if congestion is too bad… this is what we get.

[Lazvon]

Was able to do 50M PPD yesterday. Past week has been 40-45M, from what I can tell just because of the constant transfer rate problems with this server. Hopefully things are improving and I get back to 53-55M again.

[Lazvon]

So, slow transfers still very much going on. This capture was started on June 2nd at 7:23pm EDT. I mention the time, in case you want to figure out the "Relative Start Time" which is seconds for any of these connections over the last 6-days. I only show about 70 of the connections out of the "134" a fair number of the 134 were 0-bytes though since I had the FW blocking for a while. I'm gonna leave it open, but certainly slower than all the other servers which are instant (or nearly instant) download/uploads versus 10-30 minutes.

There are a few fast downloads in there, but don't think there are ever any fast uploads.

[Lazvon]

3 out of 7 folders all downloading during this particular 10-minute poll...

[toTOW]

Is this on VMWare Cloud? Is it a donated VM?

Yes, it's a donated server.

[FrancoV78]

I also opened a topic, I generally have problems with uploading these days.