Stats not updating yet again

Moderators: Site Moderators, FAHC Science Team

Post Reply
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Stats not updating yet again

Post by drougnor »

It appears that at 10pm Eastern time last night, the stats flat files stopped updating. The dates are updating (Or else my system wouldn't be pulling and processing the new files) but the values within those files are stuck at the 10pm EDT update.

A quick look at EOC shows they're not processing new values either.

Thanks!
JonasTheMovie
Posts: 86
Joined: Wed Jan 06, 2016 4:16 am
Location: Northern Sweden

Re: Stats not updating yet again

Post by JonasTheMovie »

Yep.
Image
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Stats not updating yet again

Post by bruce »

Reported
ChristianVirtual
Posts: 1576
Joined: Tue May 28, 2013 12:14 pm
Location: Tokyo

Re: Stats not updating yet again

Post by ChristianVirtual »

Fixed
ImageImage
Please contribute your logs to http://ppd.fahmm.net
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating yet again

Post by drougnor »

Ok, I know no one can fix it just now, but I want to put some stank on this!

We at the EVGA folding team have been folding under the name of one of our folder's who is currently battling with cancer - wrinvert. Our goal was to push him over 1 billion points and we are within 5-6 hours of doing so, if not less, and I just noticed that my stats software picked up a perfect duplicate flat file, which is usually the indication that the stats server has burped and needs to be restarted.

SO, if there is ANY way anyone can get ahold of the appropriate people and rattle some cages to get that server in line so we can actually track the 1 billion point crossover, Folding @ EVGA would be in your debt, hard core. Please, and thank you!

d
cxh
Pande Group Member
Posts: 126
Joined: Thu Jan 23, 2014 1:16 am
Location: Stanford, CA
Contact:

Re: Stats not updating yet again

Post by cxh »

We've restarted the stats server. New reports should be generated within the hour. Thanks for bringing this to our attention!
Sn1ken
Posts: 53
Joined: Tue Feb 11, 2014 12:58 pm

Re: Stats not updating yet again

Post by Sn1ken »

Now it works again. Tnx
Image
schmiddi
Posts: 15
Joined: Tue Feb 10, 2015 6:43 am

Re: Stats not updating yet again

Post by schmiddi »

seems like the stats server is stuck again ...
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Stats not updating yet again

Post by Joe_H »

Stats database was updated for a WU I turned in less than an hour ago, what are you basing your statement on that the server is stuck? The official stats reflect the turn in of that WU as well.

If you are referring to EOC stats, they were down for most of the day for a database upgrade and just came back up about 2 hours ago.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
PS3EdOlkkola
Posts: 177
Joined: Tue Aug 26, 2014 9:48 pm
Hardware configuration: 10 SMP folding slots on Intel Phi "Knights Landing" system, configured as 24 CPUs/slot
9 AMD GPU folding slots
31 Nvidia GPU folding slots
50 total folding slots
Average PPD/slot = 459,500
Location: Dallas, TX

Re: Stats not updating yet again

Post by PS3EdOlkkola »

Could someone please check the stats server? Date and time of last work unit submitted was recorded on 2016-07-11 20:05:39.

Is the stats server status listed on this page - and if not, should it? http://fah-web.stanford.edu/pybeta/serverstat.html
Image
Hardware config viewtopic.php?f=66&t=17997&p=277235#p277235
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Stats not updating yet again

Post by Joe_H »

I will notify the stats team about the outage. It appears the scripts have stopped, but the the server is up.

As for being on the Server Status page, how would you add it to a report on WS stats? In addition, the scripts that report on the stats run on fah-web, so being able to see that page shows the server itself is up. Also, at the current time the Server Status page does not include the AS's.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
PS3EdOlkkola
Posts: 177
Joined: Tue Aug 26, 2014 9:48 pm
Hardware configuration: 10 SMP folding slots on Intel Phi "Knights Landing" system, configured as 24 CPUs/slot
9 AMD GPU folding slots
31 Nvidia GPU folding slots
50 total folding slots
Average PPD/slot = 459,500
Location: Dallas, TX

Re: Stats not updating yet again

Post by PS3EdOlkkola »

Can't the Server Status web page pull JSON or XML status updates from any IP address it's configured to communicate with?

Maybe I'm oversimplifying, but just add the Stats Server as a record (row) in the database that generates the web page.

For example, the field headings would be:
Server IP: "IP v4 Address"
Name: "Statistics Server"
Who: "Name of Admin responsible for maintaining S.S."
Client: Add new category: "Stats"
Status: "Accept" means the S.S. is configured to collect statistics data, "Standby" means not accepting statistic collection data due to scheduled maintenance or other planned downtime.
Connect: "Accepting" means the S.S. is currently accepting statistics data from reporting servers, "Reject" means the server scripts crashed or some other non-normal status and statistics collection has stopped

Other stats can be collected such as the CPU load and Network load to help troubleshoot a Reject connect status. Every other field can be null. I may be assuming too much, but if the Stats server was in the same reporting and system management database as the WS servers, the admin responsible would be immediately alerted if any abnormal event occurs and could fix it before it has to be reported on the forum. Again, assuming the same would be true if AS's were added to the database. If no such automated alerting and monitoring system exists, a very good solution is Solarwinds. I'm sure you could get a big non-profit academic discount.
Image
Hardware config viewtopic.php?f=66&t=17997&p=277235#p277235
7im
Posts: 10179
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:

Re: Stats not updating yet again

Post by 7im »

From a PG member. No additional details available.

"IT was [...] to have migrated that server to a cloud service."
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
7im
Posts: 10179
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:

Re: Stats not updating yet again

Post by 7im »

And stats are back up again.

Last updated: Wed Jul 13 10:00:13 PDT 2016
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: Stats not updating yet again

Post by Joe_H »

7im wrote:And stats are back up again.

Last updated: Wed Jul 13 10:00:13 PDT 2016
Actually they were back up by about noon on Tuesday PDT.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Post Reply