Folding Forum

drougnor wrote:As I said above, the Team flat file was publishing user data, but with different cases in some names, for several hours. My thought is that Jason's system processed those entries as if they were standard user entries. Either way, I sent him a PM on the EOC forum and a thread was started over there as well. The only thing we can do is wait to see if he cleans up his database. Otherwise, we deal with seeing extra names.

d

I was referring to the daily_user_summary flat file, not the team flat file. In all of the user flat files that have been published since the outage starting last Friday, some user's names have had a random letter change it's case.

In addition, 3 team members on team 439 have been dropped: Lawless, momofunk, and _varikin. All 3 can still be found on team 439 when checking the official Folding@Home donor page.

Interesting... I'm trying to understand what's happened here. Just my hypothesis based on the observations:

It would appear that the Stanford stats are case insensitive for username, yet preserve it.

In the case where a username has appeared with different case, these get aggregated in the stats file, with one of the capitalisations returned as the key. However, it would appear that something has changed with regard to which capitalisation has been chosen.

This will affect anything downstream (3rd party stat sites) that are case sensitive.

Ahhhh, ok. I had only seen the change when comparing between the two files, I hadn't realized it was carrying over into the current User file. My apologies.

Yeah, it looks more and more like whatever was done on the DB, there's now a lot of issues with the flat files, and this is going to cause issues for those of us paying attention to the stats flat files fora while until it gets properly cleaned up.

Crossing my fingers that the right information makes it to the right people.

d

goodyca wrote:In addition, 3 team members on team 439 have been dropped: Lawless, momofunk, and _varikin. All 3 can still be found on team 439 when checking the official Folding@Home donor page.

I think they all have 0 points, in which case a recent update at EOC would have seen them purged, so this is expected and probably not related to the current stat behaviour.

drougnor wrote:Ahhhh, ok. I had only seen the change when comparing between the two files, I hadn't realized it was carrying over into the current User file. My apologies.

Yeah, it looks more and more like whatever was done on the DB, there's now a lot of issues with the flat files, and this is going to cause issues for those of us paying attention to the stats flat files fora while until it gets properly cleaned up.

Crossing my fingers that the right information makes it to the right people.

d

OK. Now that you (sort-of) understand it, what happens if there are 3 accounts in the case-sensitive Stanford data: say Bruce and bruce and BRUCE. EOC used to give you the option of seeing them as combined or not. What changed from a couple weeks ago. (Use your own example, please.)

All I can do is show what my collection of stats shows using a specific example from Team EVGA - The user '1pcracer' has, up until the recent change in data, was exactly as it is showing - lowercase p. Now it is showing as '1Pcracer'. I don't know what caused the change, but the change has happened and it is effecting one of the stats sites we all use.

As for the inner workings of EOC? I'm not the dev, so I don't know. My best guess is that he's using a Case Sensitive approach that defines '1pcracer' differently than '1Pcracer'.

You'll notice that I'm not claiming duplication of data IN the flat files, but that there's a change in the data presented in the flat files. (meaning that some names are capped differently)

I also see that my verbiage earlier isn't specific enough - The primary issue is that there was a change made that no down stream (AKA, us end users) were made aware of. EOC now is showing duplicated information for reasons out of mine and your control. I'm simply looking to put the reason together in a place that we can then point to later for Jason at EOC to use as a means to fix his presentation of the data without it being unusable for the end user.

d

bruce wrote:

drougnor wrote:Ahhhh, ok. I had only seen the change when comparing between the two files, I hadn't realized it was carrying over into the current User file. My apologies.

Yeah, it looks more and more like whatever was done on the DB, there's now a lot of issues with the flat files, and this is going to cause issues for those of us paying attention to the stats flat files fora while until it gets properly cleaned up.

Crossing my fingers that the right information makes it to the right people.

d

OK. Now that you (sort-of) understand it, what happens if there are 3 accounts in the case-sensitive Stanford data: say Bruce and bruce and BRUCE. EOC used to give you the option of seeing them as combined or not. What changed from a couple weeks ago. (Use your own example, please.)

I think what would have happened is that the stats file would have previously contained the combined results under (for example) 'bruce'. However, they are now all under 'Bruce'. This appears to be the case for a real user who has used both those names for team 3213 and is now duplicated on EOC.
However, I'd need an older flat file to comprehensively prove that.

I don't keep the original flat files, but the way I pull data for my own use, I can go back every hour there was a flat file update (Where my internet connection was solid and allowed the download to happen) and I can tell you that a random selection of Midnight (eastern) on October 4th, the user 'bruce' on team 3213 was all lowercase, and the most recent pull which reflects the current user flat file (Oct. 24th at 11:25am, Eastern) the user 'Bruce' for team 3213 is now capitalized.

Again, I don't keep the original flat files, but my data pulls EXACTLY what is published from those files, as far as how the name is presented and all the appropriate Points, WUs and Team numbers information goes.

(I trim the data set down to only the top 50 teams, not counting the aggregate teams, for my own purposes).

Interesting. However if EOC picks them up as different folders then the team would have all 3 listed with individual point totals as team members (which isn't happening in my case at least as I'm the only rickoic listed in my team.). So the individual stats appear where the problem lies and I'm not sure how PG compiles them. Could the have been once none case sensitive and all versions placed under the correct name, and something made them case sensitive but failed to place them in the correct team number. With that thought in mind I think I'll check the default team number of 0 to see if I'm listed that under different versions of rickoic.

Well tried to check but at this time all I get back is 'gateway timeout'. So will try later. Ok tried later and searched for folders 'like' rickoic and only came up with two teams which is what I expected.

We're back to hourly updates, it appears - the internal timestamp is now in GMT, so I had to adjust my own code to account for that, and the time of publication appears to have moved from 25 minutes past the hour to about quarter of the hour. This is mostly an FYI of what I've seen in the last few hours.

d

Sorry guys for not dealing with this sooner... I had exactly one (1) person email me about an issue with their username... I'm sorry but I don't check forums regularly (nor do I have any sort of email notifications enabled), but I DO check my email daily. I've made numerous comments on the front page of the stats to always use the 'contact us' page on the site so that it gets emailed to me directly... any other means is no guarantee I will see it in a timely fashion, if ever...

So yeah, as everyone has noticed, a lot of things got borked recently... The flat files have always been pretty consistent for the past... 10+ years? This is probably one of the largest 'oops' someone (not me) did in a long time.

First, there were 3 updates on October 22 where the contents of the files for the 'team' data got swapped with the 'user' data instead. I've been glancing through the DB and I don't think anything got corrupted from that mishap. I still need to look it over more before being satisfied nothing happened.

Second, there's another noticeable change back on October 21 in the file size where it actually shrunk a sizeable amount from what it normally is. I have no idea what's going on there I just noticed it while typing this post.

Third, it seems the case of some users might have randomly changed... Yes, I do match CaSe-SeNsItIvE... It used to matter as they could be separate user names. Now, I have no idea what is going on... I don't know if they are still case sensitive but the new name is what it's going to be going forward, or if they might randomly change one day in the future, or if matching case insensitive will be okay, or what... I'm going to have to take a long look at the data before jumping to any conclusion and making changes to the code and screwing things up even worse...

As long as the data is being recorded right now, even under the 'wrong' or 'new' username, I can always move the historical data.

Guess it's time to put on another pot of coffee... This is going to be a long day...

Unless there was something going on that I'm not aware of (quite possible), there's something very odd in the stats. Without trying very hard I've so far found six teams which have had huge spikes in output between 21st and 25th October then lapsed back to normal production levels. For example choderlab (229192) went from zero to 134 million points on the 21st then gradually slipped back to nothing again and Refearn User Experience (159301) put on 159 million over three days then went back towards zero. I am not accusing these teams of doing anything wrong, just seeking an explanation for what I see.

When there's an outage in the stats, like there was at that point, the first update that comes out after the stats straighten out ALWAYS look like massive spikes in production. In reality, it's simply all the accumulated production that wasn't being reported at that time showing up in one update.

No, that's not even anything like the case here as the spike is several times the team's previous total score i.e choderlab had a previous total of around 100 million and a spike of 134 million.

Please see Jason's news/update at EOC:
https://folding.extremeoverclocking.com/

That should fix some of the anomalies we've seen ... after he's finished please check again to see if the problem you've reported has been cleared up ... if not please report again.