Page 1 of 3

Stats server is down, [and fixed] again

Posted: Wed Jun 03, 2015 9:27 pm
by Kent Irwin
Thread title says it all.

Re: Stats server is down, again

Posted: Wed Jun 03, 2015 10:56 pm
by sortofageek
What are you seeing that tells you stats are down? I just looked around at my account, a few team members and the team itself. It looked ok to me.

Re: Stats server is down, again

Posted: Wed Jun 03, 2015 11:11 pm
by Joe_H
I have seen some slow connections to the fah-web.stanford.edu sites today, and looking at my team stats on EOC it appears an update may not have been done in time for the 3:00 load. But the 6:00 one is there on EOC. SO if there was a problem, it may have been just a temporary one.

Re: Stats server is down, again [not]

Posted: Sun Jun 07, 2015 5:19 am
by sco01
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.

Apache/2.0.52 (CentOS) Server at fah-web.stanford.edu Port 80

Re: Stats server is down, again [not]

Posted: Sun Jun 07, 2015 6:20 am
by billford
Same here.

As far as I can see it was OK until the update run at 01:00 (UTC), then something went pear-shaped.

Re: Stats server is down, again [not]

Posted: Sun Jun 07, 2015 9:36 am
by Kent Irwin
Kent Irwin wrote:Thread title says it all.
Stats server is down, again [yes]
It's down again. It was down for a short period when I originally posted. It is a chronic problem.

Re: Stats server is down, again [not]

Posted: Sun Jun 07, 2015 10:46 am
by Skufer
I can confirm that I am also seeing an error 500 Internal Server Error on the stats page

Re: Stats server is down, again

Posted: Sun Jun 07, 2015 11:14 am
by toTOW
Yes it's down again, no data on 3 tast EOC updates :(

Re: Stats server is down, again

Posted: Sun Jun 07, 2015 11:40 am
by drougnor
My stats collector hasn't seen an update since the last one at 9pm EDT.

Re: Stats server is down, again

Posted: Sun Jun 07, 2015 2:19 pm
by Joe_H
I have sent emails to the appropriate PG members. The problems appear to be only with fah-web.stanford.edu which accesses and reports the stats, as best as I can tell the database server itself is not affected.

Re: Stats server is down, again

Posted: Sun Jun 07, 2015 2:46 pm
by VijayPande
Yes, the db is on a separate server and is isolated from these issues. This is related to the NaCl issue I posted about yesterday.

Short term, this should be fixed on Monday. Longer term, as mentioned in that post, we need to migrate out some services onto a new server to take care of this. We have those servers ready so it’s about migrating now.

Re: Stats server is down, again

Posted: Sun Jun 07, 2015 2:56 pm
by drougnor
*lame joke warning* That's what you get for putting table salt in your servers!

Lame joking aside, thanks for the info, sir. I know there's nothing we can do but wait, but we at EVGA had our overtake of the Default team estimated to be in the next 12-18 hours or so and are a little disappointed not to be able to track the 'official' time of overtake. However, it is what it is and we gladly continue to Fold for the cause!

d

Re: Stats server is down, again

Posted: Sun Jun 07, 2015 7:42 pm
by beylie
So this is why I've been seeing no stats for the last few days! Please tell me the work done isn't lost, but will show up when the stats servers are up again?

Re: Stats server is down, again

Posted: Sun Jun 07, 2015 8:15 pm
by cxh
@beylie: No need to worry! Work is being returned and points are still being updated in our internal database. It's just the front-end that's not working at the moment.

Re: Stats server is down, again

Posted: Sun Jun 07, 2015 8:30 pm
by beylie
Excellent! Thank you!