Not getting credit

Moderators: Site Moderators, FAHC Science Team

drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

We got two hourly updates this time, the 2pm Eastern and 3pm Eastern, before it stopped again.

What I've been able to piece together of how this system works is that the flat file gets generated with the current timestamp in PST and then gets populated with the current hour's worth of data (Name, Points total, WUs Total, Team number for the Users and Team Number, Team Name, Points total and WU total for the Teams). It appears that the crash in this particular case is happening after the blank files get generated with the timestamp, but before the data gets inserted. The system sees a file that it then zips up and publishes the empty file and zipped archive.

However, as the process manages to complete at least a few times before the next error, this is telling me that the database itself IS indeed whole and hale, esp since we can view the data through the interactive web portal on the official stats page.

Not that any of this is really relevant to most of you, but it was fun for me to piece together over a bit of time of examining error messages and figuring out what I was doing wrong and what that meant for the backend system and how it was apparently constructed.

d
ChristianVirtual
Posts: 1576
Joined: Tue May 28, 2013 12:14 pm
Location: Tokyo

Re: Stats not updating?

Post by ChristianVirtual »

It is also relevant to me. How else I can send push notification on stats changes when the stats file is empty ... (maybe I should at least push a note that stats is down and hope one in PG has an Apple Watch ;-)
ImageImage
Please contribute your logs to http://ppd.fahmm.net
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: Stats not updating?

Post by bruce »

It's strange, but we're not seeing a single type of failure over the last few weeks.

In one case, http://fah-web.stanford.edu which contains /cgi-bin and other sub-files was off-line. Last time, the message was that the server was too busy. Now I can get to certain scripts that are inside of fah-web.stanford.edu/cgi-bin. I wonder how many DIFFERENT problems we're actually looking at.
SombraGuerrero
Posts: 117
Joined: Mon Mar 16, 2009 3:06 am

Re: Stats not updating?

Post by SombraGuerrero »

That's the problem with HTTP500s. Even when you can drill down to some level of error detail, they're still so generic, they can mean anything. It seems to me that there's almost a cascading failure in the file system. Some sort of dirty process or cluster of files that's corrupting permissions, or page faulting or locking down shared resources in an access violation type of way. If it were a hardware failure, it wouldn't so consistently right itself with a simple reboot. That is, unless the storage devices are failing and there are literal read/write or file I/O problems happening. Do we know if the server has a something like a RAID array set up? Something to at least provide file system redundancy if no other type of redundancy?
QuintLeo
Posts: 51
Joined: Sat Jun 17, 2017 5:43 am

Re: Stats not updating?

Post by QuintLeo »

Hopefully they'll get the move to new hardware done SOON, since the old hardware has been showing WAY too much downtime the last couple weeks to be trusted.
rwh202
Posts: 410
Joined: Mon Nov 15, 2010 8:51 pm
Hardware configuration: 8x GTX 1080
3x GTX 1080 Ti
3x GTX 1060
Various other bits and pieces
Location: South Coast, UK

Re: Stats not updating?

Post by rwh202 »

And 'Trust' is what a lot of this is about. If donors can't trust that the project is being run and managed in a competent way, then what's the incentive of putting in their own effort?

Yes, I realise that the stats server isn't the same as the 'science', but it provides the most visible and understandable evidence of output to the majority of donors. Also, it it considered trivial to get right, so if something 'simple' can't be done right, what faith is there in the rest of the system?

To suggest that money or resource is the problem is not believable - A few days effort and a couple $100 a month on Azure / AWS and I'm sure they could be up and running (plus, if they spoke nicely, I bet it wouldn't cost a penny).
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

A possible potential as well may be a result of the open ended nature of letting people pick names as long and mostly complicated as they want. I'm not sure what kind of error trapping is being done on name creation, but with the influx of names that have a long hash in them that are being used for the Folding-Coin mining operations, there is a slim chance that someone accidentally created a name that is crashing one of the processes.

I'm not saying it's absolutely LIKELY, but it is something I hope the Folding programmers have looked at and eliminated as a potential cause.

d
SombraGuerrero
Posts: 117
Joined: Mon Mar 16, 2009 3:06 am

Re: Stats not updating?

Post by SombraGuerrero »

That is feasible. I recently had one of our vendors at work have to change the data type of a field from like varchar to blob2 because users were simply trying to stuff too much in there.
Mike.Barr
Posts: 6
Joined: Thu Aug 18, 2016 1:41 pm

Re: Stats not updating?

Post by Mike.Barr »

There was an error accessing/using the database.
The Folding@home team is working to fix this issue.

At least 24 hours since this message posted...Team updates not being posted.

Mike
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

Official page is back up. Waiting to see if my system picks up the flat file now.

**edit** So, as I said earlier, the official stats page is presenting, but now the .txt file is no longer being published. The .bz2, while still empty, is there, however.
EmperiuM
Posts: 4
Joined: Tue Jan 06, 2015 2:22 pm

Re: Stats not updating?

Post by EmperiuM »

There is data on the .bz file, but mine, for instance, is the same that I had on the Date of last work unit 2017-09-08 09:06:40.
For me, it didn't count my points since that day :(
nsummy
Posts: 9
Joined: Fri Aug 11, 2017 1:46 pm

Re: Stats not updating?

Post by nsummy »

On my stats page it says: Date of last work unit 2017-09-11 11:00:16

:(
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

I can verify with A-B comparisons on the last 4 data sets/flat files the system produced have been the identical data.

d
msultan
Pande Group Member
Posts: 134
Joined: Mon Jun 24, 2013 10:27 pm

Re: Stats not updating?

Post by msultan »

We are looking into the stats issue. Please give me a few hours to see what might be up. Our initial attempt to debug this didnt work.
drougnor
Posts: 147
Joined: Tue Dec 29, 2009 2:21 am

Re: Stats not updating?

Post by drougnor »

By all means take all the time you need. I'm much happier with a complete and correct solution that takes some time than a 'we think it's working, oooops, guess not' rushed patch job.
Post Reply