Trouble with vav2x.fah.temple.edu

[Wedge009]

This may be a regional issue, I'm located in Sydney, Australia. Internet connections seem to be working fine for everything else, but starting at least at ~05:00 UTC, 2025-05-23, my clients could not download tasks from vav21.fah.temple.edu or vav23.fah.temple.edu (there may be other temple.edu servers I'd have trouble with but these were the servers my clients were assigned to.

I tried restarting F@h and rebooting my router. When I run ping against these servers I note responses in excess of 300 ms (I think this is quite slow) and a lot of dropped packets (over 75%). So to me this suggests an issue somewhere between Sydney and wherever temple.edu is located, given that the status page lists these servers as on-line.

In the end I had to keep dumping tasks from these servers until I get assigned to tasks from the highland servers. Strangely, there's another thread about those servers being a problem for others, but for me, right now, these servers deliver tasks as normal.

[Wedge009]

Looks like it was a transient issue. I was able to download again from vav21.fah.temple.edu, but ping times are still very high, nearly 250 ms. But no dropped packets, so presumably that's why the downloads resumed.

[muziqaz]

Temple is university in Philly, Pensilvania. So you are looking at transatlantic (Pacific) fibre, plus exchanges within USA, between you and the uni

[Joe_H]

I am less than 300 miles from Temple University, and my ping still is about 30 ms. I would need an IP in Sydney that would respond to a ping to get an idea of a "normal" ping for that distance.

[Wedge009]

I was concerned mainly because I'd never seen this kind of issue so consistently for so long before (about 2 hours). Testing again just now, vav21 and vav23 are around 225 ms for me.

I tried upenn.edu as a point of comparison, but it looks like that server is based in San Francisco, as I'm getting ping times of around 35 ms. It looks like drexel.edu is actually based in Philadelphia, and ping times are also around 225 ms, so maybe that's 'normal' for my connections to eastern US.

As I said before, no more dropped packets, so I'm not seeing any issues with downloading F@h tasks for the time being.

[appepi]

Also in Sydney, also 225ms to Temple Vs Telstra Speedtest Ping from TPG (124.168.94.74) to Telstra Chatswood 11ms, and Ping to TPG 2ms.

Pinging vav21.fah.temple.edu [129.32.209.204] with 32 bytes of data:
Ping statistics for 129.32.209.204:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 223ms, Maximum = 225ms, Average = 224ms

PS C:\WINDOWS\system32> ping 124.168.94.74
Pinging 124.168.94.74 with 32 bytes of data:
Ping statistics for 124.168.94.74:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 4ms, Average = 2ms

[appepi]

by Joe_H » Sat May 24, 2025 1:27 am
I am less than 300 miles from Temple University, and my ping still is about 30 ms. I would need an IP in Sydney that would respond to a ping to get an idea of a "normal" ping for that distance.

I am within 3 Km of Sydney University:

Pinging sydney.edu.au [20.248.131.216] with 32 bytes of data:
Ping statistics for 20.248.131.216:
Approximate round trip times in milli-seconds:
Minimum = 12ms, Maximum = 16ms, Average = 14ms

I am not quite 10,000 miles from :

City of Philadelphia (.gov)
https://www.phila.gov
Official website of the City of Philadelphia, includes information on municipal services, permits, licenses, and records for citizens and businesses.

Pinging phila.gov [108.158.32.38] with 32 bytes of data:
Approximate round trip times in milli-seconds:
Minimum = 9ms, Maximum = 11ms, Average = 10ms

But ...

Pinging temple.edu [2607:4a80::f5:60] with 32 bytes of data:
Ping statistics for 2607:4a80::f5:60:
Approximate round trip times in milli-seconds:
Minimum = 223ms, Maximum = 266ms, Average = 235ms

And ...

Pinging harvard.edu [192.0.66.20] with 32 bytes of data:
Approximate round trip times in milli-seconds:
Minimum = 9ms, Maximum = 11ms, Average = 10ms

Which is much the same as ...

Pinging ucsd.edu [75.2.44.127] with 32 bytes of data:
Approximate round trip times in milli-seconds:
Minimum = 10ms, Maximum = 15ms, Average = 11ms

or ...

Pinging washu.edu [2620:12a:8000::3] with 32 bytes of data:
Approximate round trip times in milli-seconds:
Minimum = 10ms, Maximum = 13ms, Average = 11ms

[arisu]

A traceroute would be more useful than a ping.

[Wedge009]

What I found in my limited research is that the server's location may not be the in the same location as the institution represented. For example, I get 35 ms for washu.edu but its IP address is apparently in San Francisco. Western US is obviously closer to eastern Australia than eastern US.

Anyhow, I think we established that ~230 ms between Sydney and the F@h servers at temple.edu is reasonable. There must have been something going on yesterday somewhere along the lines of communications that dropped so many packets, causing the (from my perspective) outage.

[Wedge009]

A traceroute would be more useful than a ping.

I did that earlier today as well:

Code: Select all

traceroute to vav21.fah.temple.edu (129.32.209.204), 30 hops max, 60 byte packets
 1  _gateway (192.168.1.1)  2.683 ms  2.711 ms  2.698 ms
 2  [ISP]  13.294 ms  13.233 ms  13.318 ms
 3  [ISP]  13.002 ms  13.037 ms  13.030 ms
 4  [ISP]  13.040 ms  13.076 ms  12.950 ms
 5  202.167.230.25 (202.167.230.25)  13.225 ms  13.257 ms  13.202 ms
 6  * * *
 7  51.162.232.153 (51.162.232.153)  13.329 ms  13.197 ms 51.162.232.159 (51.162.232.159)  13.138 ms
 8  * * *
 9  Bundle-Ether42.222.bdr02-ipt-639garde-syd.au.as38195.net (103.200.13.70)  161.129 ms Bundle-Ether25.130.bdr01-ipt-55smarke-sjc.us.as38195.net (103.200.13.118)  161.037 ms  161.057 ms
10  hu0-4-0-2.rcr21.b001848-1.sjc01.atlas.cogentco.com (38.142.245.25)  167.369 ms Bundle-Ether25.130.bdr01-ipt-55smarke-sjc.us.as38195.net (103.200.13.118)  161.128 ms hu0-4-0-2.rcr21.b001848-1.sjc01.atlas.cogentco.com (38.142.245.25)  167.436 ms
11  hu0-4-0-2.rcr21.b001848-1.sjc01.atlas.cogentco.com (38.142.245.25)  167.365 ms be3528.ccr21.sjc01.atlas.cogentco.com (154.54.94.49)  165.711 ms  165.638 ms
12  be3144.ccr41.sjc03.atlas.cogentco.com (154.54.5.102)  163.836 ms  161.076 ms be3528.ccr21.sjc01.atlas.cogentco.com (154.54.94.49)  163.630 ms
13  be3670.ccr22.sfo01.atlas.cogentco.com (154.54.43.13)  167.363 ms be3142.ccr41.sjc03.atlas.cogentco.com (154.54.1.194)  167.303 ms  167.423 ms
14  be3669.ccr21.sfo01.atlas.cogentco.com (154.54.43.9)  168.623 ms  168.654 ms be3670.ccr22.sfo01.atlas.cogentco.com (154.54.43.13)  167.296 ms
15  be5822.ccr31.slc01.atlas.cogentco.com (154.54.167.141)  194.728 ms  194.671 ms be3110.ccr32.slc01.atlas.cogentco.com (154.54.44.142)  193.626 ms
16  be3802.ccr31.oma02.atlas.cogentco.com (154.54.165.78)  202.016 ms be5456.ccr22.den01.atlas.cogentco.com (154.54.45.165)  194.742 ms be5821.ccr21.den01.atlas.cogentco.com (154.54.167.138)  194.681 ms
17  be4995.ccr32.oma02.atlas.cogentco.com (154.54.165.214)  197.375 ms  197.010 ms  202.524 ms
18  port-channel2717.ccr91.cle04.atlas.cogentco.com (154.54.6.222)  222.643 ms port-channel2718.ccr92.cle04.atlas.cogentco.com (154.54.7.130)  222.281 ms be5068.ccr42.ord01.atlas.cogentco.com (154.54.166.74)  209.218 ms
19  be4985.ccr41.jfk02.atlas.cogentco.com (154.54.162.166)  220.757 ms  249.371 ms port-channel2717.ccr91.cle04.atlas.cogentco.com (154.54.6.222)  221.960 ms
20  be2333.rcr22.phl01.atlas.cogentco.com (154.54.5.2)  223.172 ms be2364.rcr21.phl01.atlas.cogentco.com (154.54.3.142)  223.084 ms be2333.rcr22.phl01.atlas.cogentco.com (154.54.5.2)  223.067 ms
21  be3163.rcr21.phl03.atlas.cogentco.com (154.54.7.2)  220.392 ms be3164.rcr21.phl03.atlas.cogentco.com (154.54.25.142)  219.723 ms be2364.rcr21.phl01.atlas.cogentco.com (154.54.3.142)  218.929 ms
22  be3163.rcr21.phl03.atlas.cogentco.com (154.54.7.2)  220.474 ms  219.465 ms be2326.nr51.b003003-4.phl03.atlas.cogentco.com (154.24.67.233)  224.826 ms
23  38.190.146.67 (38.190.146.67)  224.539 ms be2326.nr51.b003003-4.phl03.atlas.cogentco.com (154.24.67.233)  224.282 ms  222.914 ms
24  38.190.146.67 (38.190.146.67)  221.508 ms  221.435 ms 38.190.146.66 (38.190.146.66)  222.347 ms
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

[appepi]

You mean like ?

Code: Select all

PS C:\WINDOWS\system32> tracert 129.32.209.204
Tracing route to vav21.fah.temple.edu [129.32.209.204] over a maximum of 30 hops:
  1     2 ms     7 ms     2 ms  [my router]
  2    13 ms    19 ms    14 ms  10.20.26.131
  3    12 ms    11 ms    12 ms  syd-apt-ros-int3-be-200.tpg.com.au [203.29.134.122]
  4   165 ms     *      165 ms  sjo-b23-link.ip.twelve99.net [62.115.188.4]
  5   171 ms   169 ms     *     palo-bb2-link.ip.twelve99.net [62.115.139.116]
  6   165 ms   163 ms   161 ms  palo-b24-link.ip.twelve99.net [62.115.139.111]
  7     *        *        *     Request timed out.
  8   165 ms   164 ms   163 ms  be2379.ccr21.sfo01.atlas.cogentco.com [154.54.42.157]
  9   180 ms   178 ms   178 ms  be5822.ccr31.slc01.atlas.cogentco.com [154.54.167.141]
 10   189 ms   187 ms   187 ms  be5821.ccr21.den01.atlas.cogentco.com [154.54.167.138]
 11   198 ms   202 ms   200 ms  be3802.ccr31.oma02.atlas.cogentco.com [154.54.165.78]
 12   207 ms   208 ms   206 ms  be5214.ccr41.ord01.atlas.cogentco.com [154.54.165.134]
 13   212 ms   219 ms   228 ms  port-channel2717.ccr91.cle04.atlas.cogentco.com [154.54.6.222]
 14   223 ms   235 ms   223 ms  be4985.ccr41.jfk02.atlas.cogentco.com [154.54.162.166]
 15   223 ms   230 ms   233 ms  be2364.rcr21.phl01.atlas.cogentco.com [154.54.3.142]
 16   224 ms   225 ms   222 ms  be3163.rcr21.phl03.atlas.cogentco.com [154.54.7.2]
 17   229 ms   223 ms   225 ms  be2326.nr51.b003003-4.phl03.atlas.cogentco.com [154.24.67.233]
 18   233 ms   222 ms   222 ms  38.190.146.66
 19     *        *        *     Request timed out.
 20   224 ms   223 ms   224 ms  vav21.fah.temple.edu [129.32.209.204]
Trace complete.

Versus

Code: Select all

PS C:\WINDOWS\system32> tracert phila.gov
Tracing route to phila.gov [108.158.32.2]
over a maximum of 30 hops:
  1     3 ms     3 ms     3 ms  [my router]
  2    12 ms    11 ms    10 ms  10.20.26.131
  3    11 ms    11 ms    10 ms  syd-apt-ros-cdn4-be-200.tpgi.com.au [203.29.134.66]
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10    11 ms    11 ms    11 ms  server-108-158-32-2.syd3.r.cloudfront.net [108.158.32.2]
Trace complete.

Or this (for a recently completed job)

Code: Select all

 Tracing route to highland1.seas.upenn.edu [158.130.118.23]
over a maximum of 30 hops:
  1     3 ms     1 ms     1 ms  [my router]
  2    11 ms    11 ms    10 ms  10.20.26.131
  3    11 ms    11 ms    11 ms  syd-apt-ros-int3-be-200.tpg.com.au [203.29.134.122]
  4   203 ms   189 ms   187 ms  ix-hge-0-0-0-15.ecore1.lvw-losangeles.as6453.net [64.86.197.96]
  5   188 ms   188 ms   188 ms  ae15.edge7.lax1.sp.lumen.tech [4.68.111.53]
  6     *        *        *     Request timed out.
  7   238 ms   237 ms   238 ms  UNIVERSITY.edge5.Newark1.Level3.net [4.16.91.150]
  8   238 ms   240 ms   238 ms  hnt-brdr-dmz.hnt-fw.router.upenn.edu [128.91.48.6]
  9   238 ms   243 ms   238 ms  hnt-fw.hnt-brdr.router.upenn.edu [128.91.48.14]
 10   240 ms   239 ms   238 ms  hnt-brdr.vagelos.router.upenn.edu [128.91.244.34]
 11   246 ms   254 ms   264 ms  core-router.seas.upenn.edu [158.130.0.249]
 12   240 ms   244 ms   238 ms  tpn-dark-core.seas.upenn.edu [158.130.0.41]
 13   239 ms   239 ms   239 ms  highland1.seas.upenn.edu [158.130.118.23]
Trace complete.