Folding Forum

It appears that at 10pm Eastern time last night, the stats flat files stopped updating. The dates are updating (Or else my system wouldn't be pulling and processing the new files) but the values within those files are stuck at the 10pm EDT update.

A quick look at EOC shows they're not processing new values either.

Thanks!

Yep.

Reported

Fixed

Ok, I know no one can fix it just now, but I want to put some stank on this!

We at the EVGA folding team have been folding under the name of one of our folder's who is currently battling with cancer - wrinvert. Our goal was to push him over 1 billion points and we are within 5-6 hours of doing so, if not less, and I just noticed that my stats software picked up a perfect duplicate flat file, which is usually the indication that the stats server has burped and needs to be restarted.

SO, if there is ANY way anyone can get ahold of the appropriate people and rattle some cages to get that server in line so we can actually track the 1 billion point crossover, Folding @ EVGA would be in your debt, hard core. Please, and thank you!

d

We've restarted the stats server. New reports should be generated within the hour. Thanks for bringing this to our attention!

Now it works again. Tnx

seems like the stats server is stuck again ...

Stats database was updated for a WU I turned in less than an hour ago, what are you basing your statement on that the server is stuck? The official stats reflect the turn in of that WU as well.

If you are referring to EOC stats, they were down for most of the day for a database upgrade and just came back up about 2 hours ago.

Could someone please check the stats server? Date and time of last work unit submitted was recorded on 2016-07-11 20:05:39.

Is the stats server status listed on this page - and if not, should it? http://fah-web.stanford.edu/pybeta/serverstat.html

I will notify the stats team about the outage. It appears the scripts have stopped, but the the server is up.

As for being on the Server Status page, how would you add it to a report on WS stats? In addition, the scripts that report on the stats run on fah-web, so being able to see that page shows the server itself is up. Also, at the current time the Server Status page does not include the AS's.

Can't the Server Status web page pull JSON or XML status updates from any IP address it's configured to communicate with?

Maybe I'm oversimplifying, but just add the Stats Server as a record (row) in the database that generates the web page.

For example, the field headings would be:
Server IP: "IP v4 Address"
Name: "Statistics Server"
Who: "Name of Admin responsible for maintaining S.S."
Client: Add new category: "Stats"
Status: "Accept" means the S.S. is configured to collect statistics data, "Standby" means not accepting statistic collection data due to scheduled maintenance or other planned downtime.
Connect: "Accepting" means the S.S. is currently accepting statistics data from reporting servers, "Reject" means the server scripts crashed or some other non-normal status and statistics collection has stopped

Other stats can be collected such as the CPU load and Network load to help troubleshoot a Reject connect status. Every other field can be null. I may be assuming too much, but if the Stats server was in the same reporting and system management database as the WS servers, the admin responsible would be immediately alerted if any abnormal event occurs and could fix it before it has to be reported on the forum. Again, assuming the same would be true if AS's were added to the database. If no such automated alerting and monitoring system exists, a very good solution is Solarwinds. I'm sure you could get a big non-profit academic discount.

From a PG member. No additional details available.

"IT was [...] to have migrated that server to a cloud service."

And stats are back up again.

Last updated: Wed Jul 13 10:00:13 PDT 2016

7im wrote:And stats are back up again.

Last updated: Wed Jul 13 10:00:13 PDT 2016

Actually they were back up by about noon on Tuesday PDT.