Ever shrinking window of availabiliy of stats.

dschief · Post by **dschief** » Fri Mar 20, 2009 4:50 pm

Just tried to view stats at Fah home site. time was 08:48. and the data base is still locked. This update started at 06:00, and another update is scheduled to start at 09:00 !. This scenario is constantly repeating, and it's only going to get worse, as the data base continues to grow. time needed for updates is going to increase. You guys with math degrees probably remember the exact formula, as the number of records { N } goes up , the time to sort { T } goes up and the rate of change for T is not
Linear

Throwing this out for public discussion ; Brain storming kinda, any ideas to help with the process.

My 2 cents to get the ball rolling.

1. bring a 2nd. stat server on line, so we have server A & server B. Both have mirror images of each other .
start off with "A" available to the public. while "B" does all the updates coming in.. Then in 3 hrs updates are locked ,
"B" ports over the "New" Image to "A". B is now the Public server and "A" assumes the role of updater.
1a. another 2 server option , keep the 2 server configuration, "A" the only one viewable with "B" handling the updates
then write the updated file to "A" rather than take "A" down . the time to write the file is way quicker than the time to update the whole database.

2. Reduce the size of the data base, { I mentioned this last year, when the problem was'nt as bad}

Archive all records with no activity for 90 days. { my guess is that you'll reduce the data base by @ 70% }
possibly add an active/inactive flag to the records. Then put a "stickie" in the forum letting people know they need to request re-activation

Post by **VijayPande** » Fri Mar 20, 2009 8:39 pm

We have a plan to address this issue. Right now, other issues (mainly rolling out the new server code and some other backend consolidation) has taken precedence, especially since some of the backend changes will be needed to roll out our stats code update. But, I wanted to stress that this something I am very concerned about.

Post by **VijayPande** » Fri Mar 20, 2009 9:59 pm

PS We have had one plan waiting to be released. I've been dragging my feet on this plan, since it means giving up some data, but I think it's time to roll it out. Please see

http://folding.typepad.com/news/2009/03 ... ended.html

for details.

MtM · Post by **MtM** » Fri Mar 20, 2009 10:52 pm

VijayPande wrote:PS We have had one plan waiting to be released. I've been dragging my feet on this plan, since it means giving up some data, but I think it's time to roll it out. Please see

http://folding.typepad.com/news/2009/03 ... ended.html

for details.

Could you give us some more details then those details? I'm sorry for sounding confused, but after reading that I'm still wondering?

kelliegang · Post by **kelliegang** » Fri Mar 20, 2009 11:08 pm

Looks like they're just turning off the tracking of work units in each project range MtM...

Detailed listing for projects 0-999
Detailed listing for projects 1000-1999
Detailed listing for projects 2000-2999
Detailed listing for projects 3000-3999
Detailed listing for projects 4000-4999
Detailed listing for projects 5000-5999

^ That stuff right?

What do people use that information for anyhow?

Post by **VijayPande** » Sat Mar 21, 2009 12:33 am

kelliegang wrote:Looks like they're just turning off the tracking of work units in each project range MtM...
Detailed listing for projects 0-999
Detailed listing for projects 1000-1999
Detailed listing for projects 2000-2999
Detailed listing for projects 3000-3999
Detailed listing for projects 4000-4999
Detailed listing for projects 5000-5999
^ That stuff right?

What do people use that information for anyhow?

That's right. We had a poll some time ago and some people asked to keep it, but there was overwhelming desire to speed up stats over keeping this info.

MtM · Post by **MtM** » Sat Mar 21, 2009 9:11 am

Remembering vaguely, sorry bad memory. If it would help speed things up ok, I think I voted against since I felt it a way of seeing what kind of hw a certain user has ( you can base that on their projects which run either on smp, uniprocessor, nvidia or ati ). This is going to be impossible now, but not important enough to outweigh stats processing

Amaruk · Post by **Amaruk** » Sun Mar 22, 2009 7:11 am

I find it somewhat ironic that I will have greater access to stats that no longer have the information I access them for. Oh well, if it makes everyone else happy...*shrug*

kelliegang wrote:What do people use that information for anyhow?

How does one verify that their PS3s have successfully returned their WUs? Since the PS3 has no FAHlog, without that detailed listing you can't.

Those stats can be useful for other clients too, as MtM alluded to.

Trotador · Post by **Trotador** » Sun Mar 22, 2009 10:03 am

Amaruk wrote:I find it somewhat ironic that I will have greater access to stats that no longer have the information I access them for. Oh well, if it makes everyone else happy...*shrug*

kelliegang wrote:What do people use that information for anyhow?
How does one verify that their PS3s have successfully returned their WUs? Since the PS3 has no FAHlog, without that detailed listing you can't.

Those stats can be useful for other clients too, as MtM alluded to.

I do agree with Amaruk here, actually I've have noted that individual WU counts are not being longer updated without reading Prof. Pande blog because I do like checking my daily figures at that level and not just the big one. That's the reason why I was searching through the forum before posting a question.

My view is that this way of "solving" the stats problem is not a good solution, losing the track of the folded WUs will be (is?)a big disappointment for me.

Please look for alternatives.

Thanks

MtM · Post by **MtM** » Sun Mar 22, 2009 10:50 am

Well offcourse I agree, I did side against it. But, the stats processing serve more people then the group which uses the detailed listings so I think we just don't carry enough weight to sway this our way. And afaik it's a temporary solution untill the whole new server and stats code is rolled out and we will be able to see it again ( provided we use a passkey ).

It's ashame it's still not possible to monitor ps3's with a 3rd party program, I think the best way to keep track of it would be putting some more pressure on the sony development team to allow the clients folder to a> have a shared network path and b> I'm not even sure the client creates logfiles so that would be needed as well

kelliegang · Post by **kelliegang** » Sun Mar 22, 2009 11:33 am

1: Is there some reason the stats need to update more than once per day?
Surely the costs of doing so are a bit large [or will end up getting very large].

2: Is this disappointment in losing the extended work unit details directed at the right people?
I understand people's ... "curiosity" but there are addons you can get to monitor your own computer's contributions from YOUR side rather than using scarse resources server side... as for the Ps3... you can see how many you've sent back to the server, no you cant tell which individual projects were returned but Sony Online Entertainment has a lot more resources to blow than does the Pande group, if this is your reason for "needing" these stats surely petitioning SOE to give some form of log for their "life with playstation" client would be more appropriate.

Competition for points is great when it inspires people to increase their output of VALID work units but 3 hourly updates just seems like a bandwith sucker and power draw [the calculating of stats and updating the servers + the hundred thousand users continually refreshing a points page]

Trotador · Post by **Trotador** » Sun Mar 22, 2009 12:07 pm

Glad to read that it is not a definitive position. Are then WU counts being still stored?

Kelliegang, I use that information for tracking my clients and for much fun sure, and believe me that it also helps to keep me contributing and thinking in upgrading my resources. Last but not least for a long time folder like me it is a little part of my memories

Sure that detailed stats could be updated just once a day or even less often if that becomes a bottleneck for the system. I would not complain for that.

kelliegang · Post by **kelliegang** » Sun Mar 22, 2009 12:12 pm

I was referring to the stats as a whole but I'm just going to have to chalk it up as something I dont understand.

The points, I get.
The work units contributed [not specific projects], I get.
Needing more detail displayed SOMEWHERE for PS3... I get [but think that is an issue to take up with SOE not the Pande Group]

Why you need a record of the detail of every project you've ever worked on to be displayed on a website, and you cant just keep record of it on your own system..... I dont get

Ah well hehe.

Trotador · Post by **Trotador** » Sun Mar 22, 2009 1:53 pm

Even if I could track all clients folding under my name (actually I can't) I would still ask F@H to do it, because:

1) It has got the actual data and can do it,
2) I'd want to see my detailed stats on a reputed website, the originator of all this effort
3) I love to see how others get their points (haven´t you ever wonder which WUs fold Wayne, or eastms.edu or Hispalense)
4) Apart from the science that is assumed, it is the only reward back to the donors
5) I spend, pleasantly, not little time keeping the clients working and really can not afford much more

Hope it helps you to get it

,

Please do not reply point to point to this post, thanks,

Ok I've said enough and will not post again, i realise it is my personal view and do not want to convert anybody

Keep folding folks!

Post by **VijayPande** » Mon Mar 23, 2009 8:12 pm

thanks for the feedback. We are still considering how to reinstate the project WU stats, so this isn't necessarily gone forever. However, this will likely take us some time, given our other pressing priorities at the moment.

Folding Forum

Ever shrinking window of availabiliy of stats.

Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.

Re: Ever shrinking window of availabiliy of stats.