Too Many Connections

[deesy58]

I wasn't sure where to post this, so please move it if that would be appropriate.

For several weeks, I have been seeing the error message "The database server is currently serving too many connections. Please try again in a few minutes" whenever I try to check my team's stats. I now see this result at least 90% of the time (not exaggerating) that I attempt to see how my team and I are doing. I also noted that one of my work units was apparently lost by the database, even after it was properly posted and visible for a short time.

My question is: Is the FAH system now becoming overloaded and unable to handle the number of folders who are trying to contribute to the project? Many potential folders participate in order to contribute to the search for knowledge that might assist in the search for treatments and cures for deadly diseases. Has the project now become overextended? Should we pause our folding activities until such time as Pande Group can resolve these technical problems?

Just wondering.

Deesy

[bruce]

The stats server has been overloaded for quite a while now and it has been taken off-line regularly while the updates are being processed. There's a piece in the News Blog about new servers, and one of them is slated as a replacement for the stats server. I suspect that this upgrade will solve the problem rather quickly when it comes on-line.

To add to that, they have been processing a rather large manual update to correct for some missing credits, so you've probably seen it off-line more that usual. (The "couple of days" was WAY too optimistic.)

By the way, you should be able to see a cached version of your team stats even during an update.

[Nathan_P]

I wasn't sure where to post this, so please move it if that would be appropriate.

For several weeks, I have been seeing the error message "The database server is currently serving too many connections. Please try again in a few minutes" whenever I try to check my team's stats. I now see this result at least 90% of the time (not exaggerating) that I attempt to see how my team and I are doing. I also noted that one of my work units was apparently lost by the database, even after it was properly posted and visible for a short time.

My question is: Is the FAH system now becoming overloaded and unable to handle the number of folders who are trying to contribute to the project? Many potential folders participate in order to contribute to the search for knowledge that might assist in the search for treatments and cures for deadly diseases. Has the project now become overextended? Should we pause our folding activities until such time as Pande Group can resolve these technical problems?

Just wondering.

Deesy

As bruce says its only the stats servers that are affected - i would suggest using on of the 3rd party stats sites instead - they do run a few hours behind stanford though. Rest assured that the work servers run fine 99% of the time so fold on

[Grandpa_01]

As bruce says its only the stats servers that are affected - i would suggest using on of the 3rd party stats sites instead - they do run a few hours behind stanford though. Rest assured that the work servers run fine 99% of the time so fold on

99% that highley depends upon the server

[bruce]

99% that highley depends upon the server

Bruce wouldn't have said 99% and he only mentioned that ONE of the new servers was slated to run the stats database. There are other new servers that will be allocated toward fixing that other ~1% (plus ) congestion problem.

Yes, FAH grows rapidly, but I've heard Vijay say more than once, that's the kind of problem we WANT to have, and we do know how to fix it.

[deesy58]

Thanks for the info. I just got the same message twice more in the past ten minutes, so I guess the servers are not, yet, up and running.

Happy Thanksgiving to all who celebrate this holiday!

Deesy

[codysluder]

Thanks for the info. I just got the same message twice more in the past ten minutes, so I guess the servers are not, yet, up and running.

Perhaps you didn't understand what Bruce was saying. The stats will still be disabled during updates, and checking again 10 minutes later proves nothing. The real question is whether the stats are brought on-line between updates. Right now, the message says "update started at Thu Nov 26 18:00:00 UTC 2009 (Thu Nov 26 10:00:00 PST 2009)" and it's 10:40 PST. Being off-line for 40+ minutes is not a change, but I don't know if the stats can be queried again soon. I don't keep track, but let's say that the update takes 60 minutes. To do an on-line query, I will need to check again after 11:00 PST.

If on-line queries were enabled during the update, it would probably take several hours, rather than maybe one hour. Then the next update would possibly need to start before the current one finishes. That's the problem that forced them to disable on-line queries during the update.

The new server should reduce this hour to a much smaller number, but there will still be periods when they're off-line for updates.

[deesy58]

Hmm. I believe that I did understand what Bruce was saying. My observations have indicated to me that stats updates typically take place during the even numbered hours (PST), and that my stats are most likely to be available during odd numbered hours. I usually see the message regarding stats updates taking place if I happen to check during an even numbered hour, so I seldom check during an even numbered hour.

The message I am referring to says nothing at all about stats updates. It says simply: "The database server is currently serving too many connections. Please try again in a few minutes."

This would imply that the server is overloaded - not offline for update entries. Is this not correct?

If, on the other hand, the error message is incorrect and misleading, perhaps is should be changed and made more meaningful ...

Deesy

[bruce]

If we assume they were off-line between 10:00 and 11:00 there would be some pent-up demand so you may have gotten the "too many connections" right after 11:00 because we WERE putting too much load on the server. Unfortunately I can't tell you how long that lasted. The fact that they're currently available (11:51 PST) isn't really saying much except that they're working as intended within the limitations of the current hardware.

Did anybody happen to check at 11:10 or 11:20 or 11:30 or 11:40?

We'll all be glad when the new hardware is ready, and there's not much point in figuring out the extent of the current limitations.

[Nathan_P]

As bruce says its only the stats servers that are affected - i would suggest using on of the 3rd party stats sites instead - they do run a few hours behind stanford though. Rest assured that the work servers run fine 99% of the time so fold on

99% that highley depends upon the server

Hey - I was trying to be an optomist

[deesy58]

I think you might be correct, Bruce, about the "pent up demand." I was able to successfully access my stats at 11:45 AM PST today. However, it is not always the case that the "pent up demand" is satisfied within the first 45 minutes or so after the server is made available. I have received this message as much as 55 minutes past an odd hour.

Just out of curiosity, is this particular error message the result of hardware overload, or is it the database broker that can't handle the connections? Not that it's especially important, but if it is the latter, will upgraded hardware solve the problem?

Deesy

[Pette Broad]

I wonder if there's any chance of reinstating the detailed stats once the new server comes on line. It's something I really miss

Pete

[bruce]

Just out of curiosity, is this particular error message the result of hardware overload, or is it the database broker that can't handle the connections? Not that it's especially important, but if it is the latter, will upgraded hardware solve the problem?

I believe it's a question of how much RAM the server has. Each connection allocates a certain amount of RAM and so there's a limit on the number of simultaneous connections that can be supported. Each active query allocates more RAM, and the database, itself, needs a lot of RAM, mostly for cache, and performance depends on how well that disk cache works. (I'm not sure of the actual numbers, but the new db server will be bigger/faster/etc. and will also have more RAM.)

[deesy58]

Sounds reasonable. Progress is seldom easy!

Deesy