Stanford Network Issue {Resolved}

JimboPalmer · Post by **JimboPalmer** » Tue Dec 13, 2011 12:24 am

*hondo* wrote:With no announcement from officials, I have shut down late today, being retired & on a tight budget I'm afraid I cannot justify continuation especially with no indication that WUs are actually getting through, so from me after, well, around 9 years of F@H its over & out for a while

I wonder what was wrong with the announcement in the first post of this thread? It has been updated as PG knew something.

derrickmcc · Post by **derrickmcc** » Tue Dec 13, 2011 12:27 am

Running the beta v7 client, the WU's show as waiting to be returned if the relevant collection server is down.
I had a p7622 worth 5187 points completed on 10th December which could not be collected. This has now been collected and I have received and completed furthe large WU's since.
So the assignment servers and collection servers are working (well, most of them) but the stats server is not updating.
I am happy to keep folding because the completed WUs are collected and that is progressing the science, hopefully the points will get allocated later but if not then we will all have lost points for the same length of time, they are only points and it's the science that is important.

orphic1 · Post by **orphic1** » Tue Dec 13, 2011 12:40 am

DR650SE wrote:
orphic1 wrote:
DR650SE wrote:@orphic1 I'm using the Beta 7 Client, but don't really know what the Core 16 is. I'll have to research that tonight.

Glad to know I can keep folding and not worry about losing work
You need to add the following extra slot option in expert view in the folding slot options screen:

client-type | advanced

For each ATI GPU slot and then it should start folding using the new 16 Core which is more efficient on ATI GPU's.

Im trying to add this, do I need to add, and then the name is GPU and value is advanced? I think i'm confused between the Slots tab and the Expert tab. Do I need one for each GPU?

Did you get it figured out?

Better yet if you have any questions or need help PM me or email me, I don't want to take over this thread to much.

texxasmike · Post by **texxasmike** » Tue Dec 13, 2011 2:48 am

If I am reading the Server Stat page correctly then there are over 8000 WU that have not posted to the Stats Servers. That means keep on folding.

Access the Server Status page from the Folding Forum header and look at the Totals row and the column: WUS RCVD.

According to the legend (below the chart) WUS RCVD shows how many WUs have been received since the last time the servers WUs were updated into the stats. This shows the relative number of WUs being received on the different servers (if all is fine) or which servers are not being inputted into the stats db if there is some problem.

Hikyuuri · Post by **Hikyuuri** » Tue Dec 13, 2011 5:04 am

I have also been watching the Server Stat page. If I am reading the numbers the same as texxasmike, there are now over 16000 WUs waiting to be processed. This number has greatly jumped in the past couple hours.

This is probably a sign that the F@H team are getting their servers back online, and that all the data is being processed. It will probably take a while though, I know their hourly updates can take some time, and now they are trying to process a couple days worth of data. Hopefully, everything gets updated shortly.

Alperian · Post by **Alperian** » Tue Dec 13, 2011 12:04 pm

Yes. In the meantime keep folding, keep folding, keep folding. Another quite nice idea (unless you have a treadmill generator rig and can't leave the building), would be to go out and spend an hour with a lonely old person who might not have anyone for company this Christmas.

Can you imagine 1,500,000 user hours helping the old? Great for Pande PR, still great for work units, great for your secular sainthood.

When you return to your rig the points might not mean quite as much.

caintry_boy · Post by **caintry_boy** » Tue Dec 13, 2011 12:50 pm

bruce wrote:See http://folding.typepad.com/ for more information.

That tells us nothing. When is the Stats server gonna be back up and running?
BTW, some of us still cannot upload finished WU's so when are the other servers gonna be up again??

aliasvee · Post by **aliasvee** » Tue Dec 13, 2011 2:48 pm

It would be nice if we could get some information about when the servers are back up again. There are many out here who is getting very frustrated because lack of information?

Mactin · Post by **Mactin** » Tue Dec 13, 2011 2:58 pm

I agree,
The outage has being going on for almost 60 hours and weve have had no feedback from PG for nearly 48.
When I hear Dr. Pande talking about expanding the donor base, he should talk more abour donor relations, a donor not lost, is exactly the same as a new donor.
Maybe, he should take on a business school MBA or DBA student to help.

new08 · Post by **new08** » Tue Dec 13, 2011 3:25 pm

Looking under Data sources from the header, I see that the servers allocated to my work for days have received no units.

Over 4000 received units on one server 171.65.103.160 classic VSPMF93 -accept [3000 on ONE other, that's nearly the 8,000 quoted]
shows an instance of what TexasDave stated- but, IF not getting to be collected, like my work CS's
171.67.108.25 [&26] CS 5vsp19v pande standby Not accept [shows Zero collected- but I have NO unsent finished units

]
then it is possible that work could be wasted for these servers- If so, it would be fairer if work was not sent out till they are back-
I have to rely on PG not to waste my fuel costs.I AM old

[PS:I agree with the previous posters drift- but 2 wrongs don't make a right!

KroontjesPen · Post by **KroontjesPen** » Tue Dec 13, 2011 4:20 pm

Is are really no more news than this?

December 10, 2011
Network issue at Stanford

The whole Stanford campus is having a major issue due to the lack of chilled water on campus. This is causing issues for many of our servers in different server rooms. As of this moment, this is still not resolved. We cannot bring servers back on line until this is resolved. It's getting late (~10:30pm PST), so we'll check back tomorrow morning to see hopefully that this is resolved. Until then, several of FAH's servers are down unfortunately.

UPDATE 4:30am Pacific Time: Chilled water came back on line at 11pm, but several of our servers are still down. Our sysadmins will work to get them back up, but it may not be until Monday, depending on their availability on Sunday.

UPDATE 11:30am Pacific time: Our sysadmins have been in the office getting machines back on line. We're almost there, although it looks like there are a few machines which have issues resulting from the outage.

k1net1cs · Post by **k1net1cs** » Tue Dec 13, 2011 4:25 pm

In any case, it'd be great if there's anyone from PG gives a rough update regarding the issue.
The silence is a bit deafening.

I mean, what's the point of having a sticky 'Stanford Network Issue' thread if it's mostly users' speculation of what's going on instead of situation updates?

orphic1 · Post by **orphic1** » Tue Dec 13, 2011 4:26 pm

k1net1cs wrote:In any case, it'd be great if there's anyone from PG gives a rough update regarding the issue.
The silence is a bit deafening.

I mean, what's the point of having a sticky 'Stanford Network Issue' thread if it's mostly users' speculation of what's going on instead of situation updates?

Agreed, an official update would be nice, at the very least a rough ETA for stats to be updated so I can quit refreshing the page every hour.

Post by **Joe_H** » Tue Dec 13, 2011 4:27 pm

caintry_boy wrote:
BTW, some of us still cannot upload finished WU's so when are the other servers gonna be up again??

Almost all of the work servers are back up and accepting according to the serverstat pages. So if you have specific work servers that are not accepting and should be, then I would suggest posting that information in the forum for issues with specific servers. All of my pending uploads from the servers going down have since uploaded, others have reported the same.

GreyWhiskers · Post by **GreyWhiskers** » Tue Dec 13, 2011 4:34 pm

Code: Select all

                                              Server IP           Name        Client     WU Avail      WUs Rcv
      Tue Dec 13 04:50:01 PST 2011          171.64.65.102        vspg2v2       GPU         16012         9292
      Tue Dec 13 05:00:00 PST 2011          171.64.65.102        vspg2v2       GPU         16040         220
      Tue Dec 13 05:10:00 PST 2011          171.64.65.102        vspg2v2       GPU         16860         2205
      Tue Dec 13 05:20:00 PST 2011          171.64.65.102        vspg2v2       GPU         16035         4215
      Tue Dec 13 05:30:00 PST 2011          171.64.65.102        vspg2v2       GPU         16865         6166
      Tue Dec 13 05:40:00 PST 2011          171.64.65.102        vspg2v2       GPU         15983         8160
      Tue Dec 13 05:50:00 PST 2011          171.64.65.102        vspg2v2       GPU         15970        10139
      Tue Dec 13 06:00:00 PST 2011          171.64.65.102        vspg2v2       GPU         16790         197
      Tue Dec 13 06:10:00 PST 2011          171.64.65.102        vspg2v2       GPU         15935         2132
      Tue Dec 13 06:20:00 PST 2011          171.64.65.102        vspg2v2       GPU         15949         4052
      Tue Dec 13 06:30:00 PST 2011          171.64.65.102        vspg2v2       GPU         15944         5975
      Tue Dec 13 06:40:00 PST 2011          171.64.65.102        vspg2v2       GPU         15901         7863
      Tue Dec 13 06:50:00 PST 2011          171.64.65.102        vspg2v2       GPU         16659         9706
      Tue Dec 13 07:00:00 PST 2011          171.64.65.102        vspg2v2       GPU         15877         169
      Tue Dec 13 07:10:01 PST 2011          171.64.65.102        vspg2v2       GPU         15943         2084
      Tue Dec 13 07:20:00 PST 2011          171.64.65.102        vspg2v2       GPU         16812         4047
      Tue Dec 13 07:30:00 PST 2011          171.64.65.102        vspg2v2       GPU         16891         6076
      Tue Dec 13 07:40:00 PST 2011          171.64.65.102        vspg2v2       GPU         16934         8145
      Tue Dec 13 07:50:00 PST 2011          171.64.65.102        vspg2v2       GPU         16895        10157

I grabbed a few hours of the details for one of the GPU servers that seemed in the stats to have a lot (10,157) of WUs received.

The WU Available count is going up and down a bit - what one would expect from the replenishment process.

But, the WU Received is cycling. The top row shows 9292, then ten minutes later, its down to 220, then monotonically goes up till it reaches 10139, then down to 197. For this sample, the cycle is hourly.

Per the definition of WU Received, that would tell me that a batch of WUs are being sent to the Stats server at the top of the hour. I presume the stats server is receiving these, but Stanford has chosen not to make the stats available quite yet.

NOTE: this is pure supposition on my part from the data.

Folding Forum

Stanford Network Issue {Resolved}

Re: Stanford Network Issue

Re: Stanford Network Issue

Re: Stanford Network Issue

Re: Stanford Network Issue

Re: Stanford Network Issue

Re: Stanford Network Issue

Re: Are you missing points??? - please read this!!!

Re: Stanford Network Issue

Re: Are you missing points??? - please read this!!!

Re: Stanford Network Issue

Re: Are you missing points??? - please read this!!!

Re: Stanford Network Issue

Re: Stanford Network Issue

Re: Are you missing points??? - please read this!!!

Re: Stanford Network Issue