Flat file stats vs 'live' stats out of sync?

[drougnor]

Folks, I've been running a small scale statistics collection program for Team EVGA for close to two weeks now in prep for an intrateam competition that's going on now. Previously, the flat file statistics were in line with the live statistics you can pull from each individual donors' scripted pages, but participants in the contest informed me that my stats fell behind at around 9pm EST(6pm PST) on the 17th.

Is this a planned thing, or is there an issue with the stats servers?

Thanks for any info!

[drougnor]

Just did some follow up on this - Previously, the flat file that would go live for 11:50 am would show the points that would appear on the users 'live' pages at noon (well, after processing time.

I have just confirmed that what went live at 5:00am PST this morning just now showed up in the 5:50am PST flat file, now making the flat stats an hour behind.

[drougnor]

Ok, no responses here . . . Is there a more appropriate area I should post this to? This is actually a pretty big deal as all of the stats websites use that flat file system, and being an hour behind on the stats may end up driving more to using the scripted pages, putting undue and unneeded strain on the Pande Group servers.

[VijayPande]

On a quick look, I don't see any issue with our servers. Could you be more specific with links for the files you're talking about?

[drougnor]

Sure thing- http://fah-web.stanford.edu/daily_user_summary.txt and http://fah-web.stanford.edu/daily_user_summary.txt.bz2

When I do a spot check of several folks scores, they are no longer synced up to the 'live' points from the scripted pages.

[bruce]

The flat files have never been synced with the scripted pages.

Recently, the scripted pages have been updated hourly. The flat-files are updated less frequently and the script to generate them must be run AFTER all the pending updates are incorporated into the database. It's unrealistic to expect that the two would be instantaneously in sync.

I don't keep track of the details, so I don't know if there has been a change or not.

[drougnor]

Bruce, in this case, when I say 'in sync' what I mean is that the currently posted Flat File is providing the same numbers as what's displayed on the scripted pages. This is the way it's been for at least the last two months that I've been tracking the stats on my own pc by pulling the flat file, and for the last several years, EOC's updates, which also use the same flat files, have produced current numbers that matched up with the numbers displayed on the scripted stats page (With the exception of the hours between updates, obviously).

An example of how it was until last night, if I checked the flat file that was posted at 9:50am, and then checked my points on the scripted page after the 10am update, those numbers would be the same. NOW, the flat file that would have updated and been posted at 9:50 am will show the same points that the scripted pages displayed at 9am, instead of the points that would have displayed at 10 am.

Now, not knowing the specifics of what drives what on the back end, I can only report what was observed and let Dr. Pande and the rest determine if there is indeed an issue that needs fixing, or if those of us who use the flat files need to change how we've been doing things.

[bruce]

So you're saying that the data that is incorporated into the database between hh:00 and about hh:10 which is available to the web query at 10 past the hour was being published in flat files 20 minute EARLIER? I don't know how that was happening unless there was a bug that was somehow delaying the web data for another hour -- and you liked it better with the bug.

[drougnor]

I can't say one way or another without knowing the process that actually generates the flat file and stats database on the back end.

All I can say for certain is that the way things were has changed and I wanted to know if it was an intended change, or if this is an error that needed to be corrected. To this point, I've delivered what I know from experience and will wait for Dr. Pande to let us know the final determination of the situation.

[drougnor]

Ok, update on the Flat Files - They appear to have stalled out. The timestamp is changing, but I haven't gotten any new numbers from the flat file since 4 am EST (1 am PST).

Trying to check the 'live' pages and there is a message stating the update began at 1 am PST.

Just thought I'd update.

This is also being reported here: viewtopic.php?f=18&t=16807

[Ravage7779]

The eoc stat site is showing 0 for the last two updates. Something is amiss.

[Mstenholm]

eoc and all other stats draws from folding@home and that has been down for hours so no surprise there plus it's Sunday...

[bruce]

Surprise .... your sarcastic comment is unwarranted. It's 8am on a Sunday morning and the guy in charge is working.
viewtopic.php?f=18&t=16807

[drougnor]

Given the amount of work the Pande Group are dealing with, I consider myself lucky that he's taking time out of a Sunday morning to take a look at something like the Stats server, let alone anything else. This is a sign of dedication deeper than most, that's for sure.

Thanks to everyone, Pande Group and Mods for any updates on the situation when they come in. Your hard work is greatly appreciated.

[VijayPande]

The flat files have never been synced with the scripted pages.

Recently, the scripted pages have been updated hourly. The flat-files are updated less frequently and the script to generate them must be run AFTER all the pending updates are incorporated into the database. It's unrealistic to expect that the two would be instantaneously in sync.

I don't keep track of the details, so I don't know if there has been a change or not.

Yes, these will not be in sync intentionally (we update the flat files less frequently).