ideas on speeding up stats

[hrsetrdr]

I almost never use the Contributions by team and project stats, and don't envision ever having a compelling need for them to be updated, on an up-to-the-minute basis. I'd still like for the db information to be available.

[v00d00]

First, I question why this pole exists. If anything the stats site should include more information and it should update more often.

As for speed I find it hard to believe that a program such as folding supported by Stanford University can’t afford a proper server. I also find it even harder to believe that with all the computer courses Stanford has a proper programmer can’t be found to maintain not only the servers but help with the code problems as well.

Just today I read this:

We try to be as transparent as possible with released projects, code, etc. We don't like publicizing things that are still under development precisely because of the fluid nature of that development--plans change, projects get delayed, we find bugs, etc. But requests for more communication are understandable and appreciated.

One note about the quad-core issue: the performance of A1 work units on quad-core machines is something that took us by surprise. We expected much more efficient utilization. We've been working very hard to improve this, and we anticipate releasing an update to the A2 core in the near future that has very close to full utilization of all four (or more) cores. [One of our rare pre-release announcements.]

One other response: we understand that many folders use points yield as a way of assessing the scientific impact of their contributions. We try to keep things as consistent as we can, but there are challenges both of inter-machine variation and of balancing points/effort and points/science.

How on earth can a staff member be surprised by something that was supposed to be tested?

If anything, we the folders need more accountability, not less.

Stop trolling across multiple threads. If you want to discuss your view, do it on the original thread.

Also, if the project doesnt live up to your expectations. You can always leave and find another one.

[Leoslocks]

Can I change my vote. The stats are not that important. Downgrade a 5 to a 3 1/2.

On Friday, Stanford launched the Pervasive Parallelism Lab (PPL). There's been lots of press describing it. The general plan for the lab is to develop a common paradigm for programming new architectures like GPU's, the Cell processor, Intel's Larabee, as well as multi-core CPUs. This is something we at FAH are very interested in, as we have had to have a unique code path for each of these (i.e. a separate code for the high performance part of the ATI GPU, NVIDIA GPU, PS3, and SMP). Having a single code path for all would be very, very exciting to keep FAH code development onto new hardware going smoothly.

Now PPL is important.

[MoneyGuyBK]

V.P.
I do look at these pages too, but sporadically.
So, I don;t need them updated all the time.... maybe weekly updating would be beneficial and sufficient for all.


Peace

[Mitsimonsta]

Also happy for a lower frequency on this part of the stats, be it daily or weekly.

As long as each of the 'Contributions by Team & Project' pages shows our points and WU total count as at the time of that the Project counts were updated, then we can work out "Percentage of total WU's by this project" and "Percentage of total points by this project". Which is info I want to know. Or better still, add that info to the pages. Could be done via Javascript/CSS or from the DB direct. Not a big overhead at all.

Right now I see I have a total 2636417 points, and completed 2641 WU's.
I've done 928 WU's of P2653, which equates to 35.14% of my total WU's, and 61.95% of my total points.

It would be nice if we could get the contributions by project summary text file when it updates also. This could go into my own record, or even into the 3rd-part stats like EOC and do all sorts of nice stuff.

[Foxery]

I thought of a different suggestion that might be an inbetween solution.

What about only showing me the last 10 WUs I have turned in, and when, rather than my entire history? Or alternatively, WUs I have turned in during the past 48 hours? This could potentially be more helpful than a master list. If you include the Core name, and I could see what kind of WUs I've been working on recently, and perhaps notice when a remote box is behaving in an unexpected manner, or crashed. Whereas, who cares what I submitted a year ago?

I'm not sure if this would actually save you any time, since similar data would still be parsed, but thought I'd throw it out there.

[WangFeiHong]

Hi here to dig up an old thread, just want to say, main its hardcore FAH users who check this, so i suggest, in addition to the previous, to have it in a txt form, which would make it less resource-consuming.

[VijayPande]

Hi here to dig up an old thread, just want to say, main its hardcore FAH users who check this, so i suggest, in addition to the previous, to have it in a txt form, which would make it less resource-consuming.

it's not the lack of a txt form which makes it slow (BTW, we have txt pages), it's the entering of lots of different info into a db.

We've been doing a lot (by some people's measures, too many different things) and so we've decided to concentrate on a few areas (right now, it's GPU core and next gen SMP -- getting stability and ease of use to where the classic client is), thus putting issues like this on hold for a bit. It does come up in our meetings from time to time, but I've intentionally been fighting the temptation to work on it in order to keep us focused and get the other issues done first before taking on more.

[spazzychalk]

these stats really are important. i use the team and personal and project by project breakdown. i find it to be the biggest inspiration/motivator its just extremely more fulfilling to be able to go into your history and read by project # exactly what youve helped with oh hey i did something with cancer hey look i did something about viruses. we all believe in what were doing here, but we need motivation. positive reinforcement. for some its the points. a reward system. competition. for others, they need to know what theyre doing is making a difference since they cant see it themselves. its crutial to be able to see those projects to KNOW youve done something, knowing what, not just that you are. i check several times a day. when i see someone on the team hasnt changed in a while i use it as tool to know theres a problem. because of the stats ive corrected 2 firewall problems and a core error i was able to catch because i saw their stats werent updating. i dont know where you guys are seeing the % stats Mitsimonsta is talking ive never seen it so i dont use that. but i use every bit available from right clicking the client / team stats

i still maintain that the best way to improve the stats is to get rid of the stats updating page. leave the previous stats page up with just a simple timestamp (or not) on the exisitng page showing last time stats were updated. and it stays there until the new page comes up. if the time stamps will take too much power than leave em off, but just leave the stats page up. id bet that would satisfy most people since they could still see there they're at, and for them it might even still be an update depending on when they last checked. but there is NOTHING more irritating about the stats page than not being able to see anything because its updating, and especially if it does it again later when you come back

[bruce]

i still maintain that the best way to improve the stats is to get rid of the stats updating page. leave the previous stats page up with just a simple timestamp (or not) on the exisitng page showing last time stats were updated. and it stays there until the new page comes up. if the time stamps will take too much power than leave em off, but just leave the stats page up. id bet that would satisfy most people since they could still see there they're at, and for them it might even still be an update depending on when they last checked. but there is NOTHING more irritating about the stats page than not being able to see anything because its updating, and especially if it does it again later when you come back

Either I don't understand you or you don't understand how the stats work. When you enter your donor-name or your Team-Number into the stats page you're submitting a query to a database. That entire database must be on-line for the query to work. It takes 45 minutes to add the previous block of updates to the database when nobody is allowed to run queries and if queries were allowed, it would not finish before the next batch of updates arrived.

Perhaps you're saying that after the database is updated, that a backup copy be created with a new time-stamp and that queries only be allowed on the backup copy. Making duplicate copies of a database that large is a VERY challenging undertaking and managing those duplications without error adds to the challenge. Since introducing any process that can potentially create errors in the stats database is unacceptable, changes of this magnitude cannot happen while most of the Pande Group are busy working on both scientific research and on the critical bugs in the SMP and GPU beta versions. When those versions reach a stable (released) status, changes to the stats can be considered. In the meantime, we're all thankful for the 3rd party stats sites that provide the sort of access that you're seeking while the official stats are being updated.

[spazzychalk]

actually ive never queried stats that way. i just right click on the client in the systray, status / team stats. but no not a backup copy. more of a .... freeze on the current while its being updated? im sure this isnt being explained as well as i see it in my head. just leaving the most recent stats viewable during the update. then there is something to look at, even if a little old, instead a screen saying try back later

[bruce]

Aha. I see why you see it that way. That would require the stats update not only to update the database, but to do a query for every client and to freeze that page. In fact, the present system only generates the pages when someone request it, so you'd be adding a lot of processing for people who don't look at their stats between those two updates plus you'd have to find a place to store all those static pages.

[spazzychalk]

these stats really are important. i use the team and personal and project by project breakdown. i find it to be the biggest inspiration/motivator its just extremely more fulfilling to be able to go into your history and read by project # exactly what youve helped with oh hey i did something with cancer hey look i did something about viruses. we all believe in what were doing here, but we need motivation. positive reinforcement. for some its the points. a reward system. competition. for others, they need to know what theyre doing is making a difference since they cant see it themselves. its crutial to be able to see those projects to KNOW youve done something, knowing what, not just that you are. i check several times a day. when i see someone on the team hasnt changed in a while i use it as tool to know theres a problem. because of the stats ive corrected 2 firewall problems and a core error i was able to catch because i saw their stats werent updating. i dont know where you guys are seeing the % stats Mitsimonsta is talking ive never seen it so i dont use that. but i use every bit available from right clicking the client / team stats

ok then. well then just the first half of my statement