Page 2 of 2
Re: Bziped summary files corrupted
Posted: Wed Apr 14, 2010 12:42 pm
by Kakao
HaloJones wrote:Can you detail the specific problem you have with the downloaded files and maybe someone here can host them for you as a relay to avoid the issue?
I can download the files from my web server in San Diego, California, where I keep those files backups. That is a ssh download. Although very slow, they arrive without errors. I think I have a problem with everything from California as I can watch youtube videos and download other content at the connection full speed.
The speed would be just an inconvenience if it weren't by the corrupted files. It looks like something changed about the Stanford's http server configuration.
I will have to change the KakaoStats code to download from the web server but that will obviously require a coding session. That is because the files are not http exposed as the current download code requires. I will need to either expose then or change the current code to make a ssh download. I can only start it on the weekend.
Re: Bziped summary files corrupted
Posted: Wed Apr 14, 2010 1:41 pm
by HaloJones
OK. Happy to hear you may have a solution.
Re: Bziped summary files corrupted
Posted: Wed Apr 14, 2010 6:13 pm
by noorman
.
Kakao,
I downloaded the file from your link (in my WinXP) and unpacked it with Winrar; to me the file seems uncorrupted ...
my traceroute gave me this:
Code: Select all
Tracing route to vspm27.stanford.edu [171.65.103.94]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms [router]
2 <1 ms <1 ms <1 ms [router]
3 7 ms 8 ms 7 ms d54C3F001.access.telenet.be [84.195.240.1]
4 9 ms 9 ms 9 ms dD5E0C521.access.telenet.be [213.224.197.33]
5 8 ms 10 ms 8 ms dD5E0FD59.access.telenet.be [213.224.253.89]
6 8 ms 7 ms 8 ms dD5E0FDB9.access.telenet.be [213.224.253.185]
7 10 ms 10 ms 9 ms ae0.anr11.ip4.tinet.net [77.67.65.177]
8 22 ms 42 ms 19 ms xe-1-0-0.lon11.ip4.tinet.net [89.149.185.170]
9 21 ms 25 ms 35 ms te7-6.mpd02.lon01.atlas.cogentco.com [130.117.15.49]
10 98 ms 96 ms 96 ms te0-0-0-1.mpd21.jfk02.atlas.cogentco.com [154.54.30.129]
11 141 ms 218 ms 211 ms te1-8.mpd01.ord01.atlas.cogentco.com [154.54.29.157]
12 168 ms 168 ms 163 ms te0-1-0-2.mpd21.mci01.atlas.cogentco.com [66.28.4.185]
13 292 ms 215 ms 206 ms te2-2.mpd01.sfo01.atlas.cogentco.com [154.54.6.38]
14 185 ms 215 ms 203 ms te7-4.mpd01.sjc04.atlas.cogentco.com [154.54.28.82]
15 170 ms 172 ms 170 ms Stanford_University2.demarc.cogentco.com [66.250.7.138]
16 172 ms 175 ms 176 ms boundarya-rtr.Stanford.EDU [68.65.168.33]
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 171 ms 171 ms 180 ms vspm27.Stanford.EDU [171.65.103.94]
Trace complete.
.
part of the unpacked file looks like this:
Code: Select all
Wed Apr 14 10:50:00 PDT 2010
name newcredit sum(total) team
PS3 1908894914 8037921 0
anonymous 1326616315 6495933 0
PDC 527070688 9644 1
mutsu 210771427 130924 11108
AtlasFolder 172276138 315802 36167
ChasR 153427598 231306 32
Leganfuh 143451489 140142 75255
eastms.edu 134667803 1111084 1714
_ 132702717 730830 0
anonymous 121951048 43475 1
Scott_H 118546443 353954 2654
wayne 116650005 133649 46429
[Zebulon.fr]_Cobra 103389154 111407 51
Christian_Bargmann 99501616 172194 1971
van_arnam 97621713 187973 50625
Michael_McCord,_M.D. 95168086 124027 11108
HayesK 87130337 102801 32
Tigerbiten 85996460 136098 33
kennish 77972716 41581 39340
wingrider 77145738 40012 53562
OverClocking-Masters.com 76076097 98262 51
Computekinc.us 74443526 102033 32
clamatowas 74410014 43090 111065
Spider_Monkey 71640893 119493 11108
.
seems normal enough to me
NOTE: must add that I downloaded that via my high-speed cable network
.
Re: Bziped summary files corrupted
Posted: Wed Apr 14, 2010 6:49 pm
by noorman
.
a while ago, someone thought that the extra delays after "boundarya-rtr.Stanford.EDU [68.65.168.33]" are from the Stanford firewall ...
my delay seems to be equal to that of toTOW (3 lines of time outs)
.
Re: Bziped summary files corrupted
Posted: Wed Apr 14, 2010 8:21 pm
by artoar_11
This is my traceroute, if it will help. PPPOE - 16Mbit/s.
Code: Select all
Tracing route to vspm27.stanford.edu [171.65.103.94]
over a maximum of 30 hops:
1 1 ms 1 ms 1 ms [router]
2 2 ms 1 ms 1 ms in-int.PPPoE2.escom.bg [195.24.89.9]
3 2 ms 1 ms 1 ms 195.24.88.2
4 5 ms 5 ms 5 ms 94.156.248.121
5 9 ms 5 ms 5 ms ae0-431.sof10.ip4.tinet.net [77.67.67.113]
6 42 ms 41 ms 41 ms xe-10-0-0.fra23.ip4.tinet.net [89.149.186.250]
7 56 ms 59 ms 41 ms te1-7.mpd02.fra03.atlas.cogentco.com [130.117.14.85]
8 137 ms 138 ms 137 ms te2-4.mpd01.ymq02.atlas.cogentco.com [154.54.28.93]
9 152 ms 152 ms 152 ms te8-7.mpd02.ord01.atlas.cogentco.com [66.28.4.57]
10 173 ms 174 ms 174 ms te0-4-0-0.mpd22.mci01.atlas.cogentco.com [154.54.30.178]
11 204 ms 204 ms 204 ms te4-7.mpd01.sfo01.atlas.cogentco.com [154.54.24.105]
12 216 ms 216 ms 216 ms te4-2.mpd01.sjc04.atlas.cogentco.com [154.54.2.166]
13 216 ms 232 ms 216 ms Stanford_University2.demarc.cogentco.com [66.250.7.138]
14 216 ms 214 ms 211 ms bnda-rtr-1.Stanford.EDU [68.65.168.33]
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 221 ms 210 ms 211 ms vspm27.Stanford.EDU [171.65.103.94]
Trace complete.
Re: Bziped summary files corrupted
Posted: Wed Apr 14, 2010 9:48 pm
by Kakao
noorman wrote:.
a while ago, someone thought that the extra delays after "boundarya-rtr.Stanford.EDU [68.65.168.33]" are from the Stanford firewall ...
my delay seems to be equal to that of toTOW (3 lines of time outs)
.
So everybody gets 3 timeouts before reaching the target and I never get there even with max hops set to 60
Code: Select all
$ traceroute -m 60 vspm27.stanford.edu
traceroute to vspm27.stanford.edu (171.65.103.94), 60 hops max, 60 byte packets
1 10.1.1.1 (10.1.1.1) 1.403 ms 2.042 ms 2.713 ms
2 BrT-L10-bsace704-vrdef.dsl.brasiltelecom.net.br (201.10.168.254) 25.361 ms 34.849 ms 41.281 ms
3 BrT-G7-1-2-740-bsace-core02.g.brasiltelecom.net.br (201.10.248.93) 33.660 ms 39.602 ms 41.954 ms
4 po2-0.core02.mia03.atlas.cogentco.com (154.54.10.13) 180.361 ms 180.655 ms 181.211 ms
5 te4-4.ccr01.mia03.atlas.cogentco.com (154.54.2.42) 185.834 ms 187.429 ms 191.833 ms
6 te3-3.ccr01.mia01.atlas.cogentco.com (154.54.2.153) 198.812 ms te2-3.ccr01.mia01.atlas.cogentco.com (154.54.24.233) 155.750 ms te9-3.ccr01.mia01.atlas.cogentco.com (154.54.28.241) 180.727 ms
7 te8-2.ccr01.iah01.atlas.cogentco.com (154.54.24.193) 216.769 ms 217.069 ms te7-3.mpd01.iah01.atlas.cogentco.com (154.54.28.77) 195.035 ms
8 te4-2.mpd01.lax01.atlas.cogentco.com (154.54.0.141) 234.378 ms te3-2.mpd02.lax01.atlas.cogentco.com (154.54.0.245) 236.895 ms te9-2.mpd02.lax01.atlas.cogentco.com (154.54.0.253) 235.875 ms
9 te7-4.mpd01.sfo01.atlas.cogentco.com (154.54.3.50) 278.402 ms te4-5.mpd01.sfo01.atlas.cogentco.com (154.54.3.94) 270.813 ms 271.234 ms
10 te7-4.mpd01.sjc04.atlas.cogentco.com (154.54.28.82) 367.097 ms * *
11 Stanford_University2.demarc.cogentco.com (66.250.7.138) 267.943 ms 272.032 ms 340.796 ms
12 bnda-rtr-1.Stanford.EDU (68.65.168.33) 323.310 ms 324.436 ms 333.791 ms
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
31 * * *
32 * * *
33 * * *
34 * * *
35 * * *
36 * * *
37 * * *
38 * * *
39 * * *
40 * * *
41 * * *
42 * * *
43 * * *
44 * * *
45 * * *
46 * * *
47 * * *
48 * * *
49 * * *
50 * * *
51 * * *
52 * * *
53 * * *
54 * * *
55 * * *
56 * * *
57 * * *
58 * * *
59 * * *
60 * * *
Re: Bziped summary files corrupted
Posted: Wed Apr 14, 2010 9:52 pm
by HaloJones
I get the three timeouts after boundarya-rtr.stanford.edu but then I get in.
Kakaostats #1
Posted: Wed Apr 14, 2010 11:10 pm
by JPinTO
Kakao: I just wanted to express my gratitude for your stats page. It's a very simple but effective design for conveying a lot of information. Thanks in advance for your work in making it operational again!!!
- JP
Re: Bziped summary files corrupted
Posted: Thu Apr 15, 2010 12:25 pm
by Kakao
I managed to implement a half solution. First I exposed the backup files at the web server through http. Then I pointed the download script to them. But the problems persisted showing that the problem is not with the Stanford's http server. Then I tried the hardened download script rewritten for the development version of KakaoStats and it proved to be more resilient to http connection problems successfully downloading the last few updates.
So the problem is not solved. I just have a more resilient script working and the download is very slow. My issue is very likely a routing problem which I will investigate during the next days.
There are many updates still missing in the database but I will manually feed then until the weekend.
Re: Bziped summary files corrupted
Posted: Thu Apr 15, 2010 3:26 pm
by Kijad
Glad to see it's at least partially up-and-running though, and isn't something like "oh the server HDDs all failed."
Keep on keepin' on, let us know if we can be of help in any way.
Re: Bziped summary files corrupted
Posted: Thu Apr 15, 2010 5:12 pm
by noorman
.
So, you are having trouble getting from the San Diego area to the San Jose area
Should be a small-hop trip though ...
( I had to get that file to the western seaboard of Europe )
.
Re: Bziped summary files corrupted
Posted: Thu Apr 15, 2010 5:18 pm
by Kakao
No. There is no problem from San Diego to San Jose. I can get the files extremely fast when downloading from San Jose to San Diego. The problem is downloading to Brasília, Brazil from both San Diego and San Jose. That is where the routing problem is. There is no problem getting content from anywhere else to Brasília.
Re: Bziped summary files corrupted
Posted: Thu Apr 15, 2010 5:42 pm
by noorman
Kakao wrote:No. There is no problem from San Diego to San Jose. I can get the files extremely fast when downloading from San Jose to San Diego. The problem is downloading to Brasília, Brazil from both San Diego and San Jose. That is where the routing problem is. There is no problem getting content from anywhere else to Brasília.
.
Why, then, does a traceroute to
kakaostats.com guide me to a server in the San Diego area ?
.
Re: Bziped summary files corrupted
Posted: Thu Apr 15, 2010 5:53 pm
by Kakao
The web side of KakaoStats lives on the web server in San Diego. That is where the pages are built. The data to build the pages live in the database server in Brasília. So whenever a page is built by the web server it sends a sql query to the database. Actually there is lots of caching so the queries are much less frequent.
A database server is a much more demanding software than a web server. It would be costly to rent a machine in a data center to run the database. So it sits right in my house.
The machine in my house has 8GB memory, two HDs in Raid 1. Nothing fancy. But in the data center they charge every bit you add to the base machine.
Re: Bziped summary files corrupted
Posted: Thu Apr 15, 2010 6:03 pm
by noorman
.
OK, I get it.
Now I see why you need this data packet to be sent from California to Brasil
That 's also the reason why my traceroute didn't detect any obvious problems
Hope you can trace the culprit (soon) / good luck
.