GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Moderators: Site Moderators, FAHC Science Team

Tobit
Posts: 342
Joined: Thu Apr 17, 2008 2:35 pm
Location: Manchester, NH USA

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by Tobit »

Project: 5781 (Run 10, Clone 80, Gen 4) - Both the WS and the CS have no record of the unit.

Code: Select all

Slot 01  Done
Project: 5781 (Run 10, Clone 80, Gen 4), Core: 11
Work server: 171.67.108.21:8080
Collection server: 171.67.108.26
Download date: February 19 08:07:36
Finished date: February 19 12:28:11
Failed uploads: 11

Code: Select all

Launch directory: C:\fah\gpu1
Executable: Folding@home-Win32-GPU.exe
Arguments: -send all -verbosity 9 

[20:13:56] - Ask before connecting: No
[20:13:56] - User name: Tobit (Team 33)
[20:13:56] - User ID: 1FA10FEE5F260BA4
[20:13:56] - Machine ID: 3
[20:13:56] 
[20:13:56] Loaded queue successfully.
[20:13:56] Attempting to return result(s) to server...
[20:13:56] Trying to send all finished work units
[20:13:56] Project: 5781 (Run 10, Clone 80, Gen 4)
[20:13:56] - Read packet limit of 540015616... Set to 524286976.

[20:13:56] + Attempting to send results [February 19 20:13:56 UTC]
[20:13:56] - Reading file work/wuresults_01.dat from core
[20:13:56]   (Read 168458 bytes from disk)
[20:13:56] Connecting to http://171.67.108.21:8080/
[20:13:58] Posted data.
[20:13:58] Initial: 0000; - Uploaded at ~82 kB/s
[20:13:58] - Averaged speed for that direction ~90 kB/s
[20:13:58] - Server does not have record of this unit. Will try again later.
[20:13:58] - Error: Could not transmit unit 01 (completed February 19) to work server.
[20:13:58] - 11 failed uploads of this unit.
[20:13:58] - Read packet limit of 540015616... Set to 524286976.

[20:13:58] + Attempting to send results [February 19 20:13:58 UTC]
[20:13:58] - Reading file work/wuresults_01.dat from core
[20:13:58]   (Read 168458 bytes from disk)
[20:13:58] Connecting to http://171.67.108.26:8080/
[20:13:59] Posted data.
[20:13:59] Initial: 0000; - Uploaded at ~165 kB/s
[20:13:59] - Averaged speed for that direction ~105 kB/s
[20:13:59] - Server does not have record of this unit. Will try again later.
[20:13:59]   Could not transmit unit 01 to Collection server; keeping in queue.
[20:13:59] + Sent 0 of 1 completed units to the server
[20:13:59] - Failed to send all units to server
[20:13:59] ***** Got a SIGTERM signal (2)
[20:13:59] Killing all core threads

Folding@Home Client Shutdown.
MichaelO
Posts: 50
Joined: Tue Jan 01, 2008 8:59 pm

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by MichaelO »

noorman wrote:.
- Server does not have record of this unit. Will try again later.
.

This has been reported before; I 've passed this on to the Pande Group because it sometimes used to happen in the past too, but I had not seen it in years.

It 's a problem with the list of outgoing WU's that is not fully known (or incorrect) at the Collecting server, so it has no reference to it and doesn't accept it.


.
Noorman,

I understand that. My issue is that this has been happening for a full week now. I am not the first to report this in this thread, which I think was only 4 or 5 pages long when I made my first post.
Image
noorman
Posts: 270
Joined: Sun Dec 02, 2007 2:26 pm
Hardware configuration: Folders: Intel C2D E6550 @ 3.150 GHz + GPU XFX 9800GTX+ @ 765 MHZ w. WinXP-GPU
AMD A2X64 3800+ @ stock + GPU XFX 9800GTX+ @ 775 MHZ w. WinXP-GPU
Main rig: an old Athlon Barton 2500+ @2.25 GHz & 2* 512 MB RAM Apacer, Radeon 9800Pro, WinXP SP3+
Location: Belgium, near the International Sea-Port of Antwerp

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by noorman »

MichaelO wrote:
noorman wrote:.

MORE Official news: http://foldingforum.org/viewtopic.php?f=24&t=13474


.
I am not seeing any progress on this front. If they have rolled out the fix, it is not working yet. I just had two more clients hang, with the now infamous:

"Server has no record of this WU" message.

And I have also witnessed that these clients will subsequently hang when trying to resend the WU on subsequent retries. My only success in then restarting these clients has been to delete the queue and lose the work. This is becoming incredibly and increasingly frustrating. I have only 6 GPU clients but I am considering quiting GPU folding altogether if this situation does not improve shortly. Its a waste of my time and the electricity to keep the cards running and to have to constantly babysit them.

If a situation like this happened in a corporate environment someone would have already lost their job. The network instability and what appears to be a lack of any quality assurance on software changes is appalling and the current situation is the worst I have seen it in the 3 years that I have been folding.

Apologies for the rant but its just been sitting here simmering and it finally boiled over.
.


It 's the first major problem in years alright (except for some Power issues from time to time).

It 's related to a bug in the server software; like I said before, those happen in all software programs that are longer than a page (or two).
The programmer in charge seems to have found the cause of it, now they need a solution, preferably a final fix in stead of a plaster ...

In the past this would not have the same repercussions because the volume of work was much much smaller and Results came in less frequent.
So, this bug creates a major problem because of that difference.
I too hope for a quick fix now and for some reparations for those who lost work or still have Results in queue.


.
- stopped Linux SMP w. HT on i7-860@3.5 GHz
....................................
Folded since 10-06-04 till 09-2010
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by PantherX »

Hope they find and fix that bug soon. I am not having any client hang ups or unusual problems so it continues to fold and only had problems when uploading. I think that MichaelO might be having some other issues as his client hangs up.

PS-
is there a thread where i can post info about a similar WU uploading problem of CPU uni Client ( Project: 6318 (Run 2665, Clone 0, Gen 1))

Thanks for your help
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
MichaelO
Posts: 50
Joined: Tue Jan 01, 2008 8:59 pm

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by MichaelO »

Panther-X wrote:Hope they find and fix that bug soon. I am not having any client hang ups or unusual problems so it continues to fold and only had problems when uploading. I think that MichaelO might be having some other issues as his client hangs up.
I am not certain what my issues would be given that these same 6 clients were working fine prior to last weekend. And the 7 SMP clients are all working as well. My network is fine and is monitored all day long since I work at home. There are also the prior 17 pages of issues with people having similar problems as well as other members on our team. This NOT an isolated problem.
Image
noorman
Posts: 270
Joined: Sun Dec 02, 2007 2:26 pm
Hardware configuration: Folders: Intel C2D E6550 @ 3.150 GHz + GPU XFX 9800GTX+ @ 765 MHZ w. WinXP-GPU
AMD A2X64 3800+ @ stock + GPU XFX 9800GTX+ @ 775 MHZ w. WinXP-GPU
Main rig: an old Athlon Barton 2500+ @2.25 GHz & 2* 512 MB RAM Apacer, Radeon 9800Pro, WinXP SP3+
Location: Belgium, near the International Sea-Port of Antwerp

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by noorman »

Panther-X wrote:Hope they find and fix that bug soon. I am not having any client hang ups or unusual problems so it continues to fold and only had problems when uploading. I think that MichaelO might be having some other issues as his client hangs up.

PS-
is there a thread where i can post info about a similar WU uploading problem of CPU uni Client ( Project: 6318 (Run 2665, Clone 0, Gen 1))

Thanks for your help
.

You should look up to which server it tried to upload to (in the Client log) and look for or start a thread with that issue on that specific server in "Issues with a specific server" (in fact this sub-forum)

.
- stopped Linux SMP w. HT on i7-860@3.5 GHz
....................................
Folded since 10-06-04 till 09-2010
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by PantherX »

well, i guess that i am the lucky for now but really hope that this gets sorted out soon.
I was thinking that Dr. Vajay Pande should post a small update on his blog (for easier RSS Feed ) linking it to the thread that he has started as it clearly isn't a small problem.
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
cdb
Posts: 26
Joined: Sat Feb 02, 2008 8:56 pm

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by cdb »

I'm starting to get abit fed up with this. It's gone from a total meltdown last weekend to getting better and now slowly going down hill again.
I'm getting WUs completing , but not sending (not all, just a few) and then these constantly try to send and stop me downloading new ones until I quit and restart the client. So if I leave my pc alone, it gets to the point where it's sat there doing nothing but using electric trying to upload.
The fix to send WUs back if installed this morning isn't working at my end.

It might be the first major problem in years, but everytime I folded on my ps3 for a couple of weeks or more at a time there seemed to be problems with the servers and now I'm trying to do it 24/7 the system goes into meltdown.

An update on the actual news page might be nice rather than a tread that you have to track down in the forum.
noorman
Posts: 270
Joined: Sun Dec 02, 2007 2:26 pm
Hardware configuration: Folders: Intel C2D E6550 @ 3.150 GHz + GPU XFX 9800GTX+ @ 765 MHZ w. WinXP-GPU
AMD A2X64 3800+ @ stock + GPU XFX 9800GTX+ @ 775 MHZ w. WinXP-GPU
Main rig: an old Athlon Barton 2500+ @2.25 GHz & 2* 512 MB RAM Apacer, Radeon 9800Pro, WinXP SP3+
Location: Belgium, near the International Sea-Port of Antwerp

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by noorman »

Panther-X wrote:well, i guess that i am the lucky for now but really hope that this gets sorted out soon.
I was thinking that Dr. Vajay Pande should post a small update on his blog (for easier RSS Feed ) linking it to the thread that he has started as it clearly isn't a small problem.
.


He recently posted again here: http://foldingforum.org/viewtopic.php?f=24&t=13474.

I guess he 's very busy right now trying to fix the problems before writing reports on that work.
It clearly isn't a simple bug (to find) and if found, a 'good' solution is needed too, then needs testing before a roll-out on all affected servers (in this case WS)

He hasn't even been on Twitter (today) and he posts there more often than in his Blog.


.
- stopped Linux SMP w. HT on i7-860@3.5 GHz
....................................
Folded since 10-06-04 till 09-2010
PantherX
Site Moderator
Posts: 6986
Joined: Wed Dec 23, 2009 9:33 am
Hardware configuration: V7.6.21 -> Multi-purpose 24/7
Windows 10 64-bit
CPU:2/3/4/6 -> Intel i7-6700K
GPU:1 -> Nvidia GTX 1080 Ti
§
Retired:
2x Nvidia GTX 1070
Nvidia GTX 675M
Nvidia GTX 660 Ti
Nvidia GTX 650 SC
Nvidia GTX 260 896 MB SOC
Nvidia 9600GT 1 GB OC
Nvidia 9500M GS
Nvidia 8800GTS 320 MB

Intel Core i7-860
Intel Core i7-3840QM
Intel i3-3240
Intel Core 2 Duo E8200
Intel Core 2 Duo E6550
Intel Core 2 Duo T8300
Intel Pentium E5500
Intel Pentium E5400
Location: Land Of The Long White Cloud
Contact:

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by PantherX »

noorman is it possible that you give me his twitter URL so that i can follow F@H more closely?
Thanks for your help
ETA:
Now ↞ Very Soon ↔ Soon ↔ Soon-ish ↔ Not Soon ↠ End Of Time

Welcome To The F@H Support Forum Ӂ Troubleshooting Bad WUs Ӂ Troubleshooting Server Connectivity Issues
*hondo*
Posts: 102
Joined: Sat Mar 08, 2008 9:50 am
Hardware configuration: Geoforce 250 GTS Active for F@H

4 Core Intel not used at all for F@H
Location: England UK

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by *hondo* »

Hello there folks :) Is there anyway someone within this forum can either tell / let me know if this WU has hit the collection server please? 18:28:01] Project: 10105 (Run 26, Clone 9, Gen 19)[

If it has, I may have stumbled on a rough neck solution


[18:28:01] + Attempting to send results [February 19 18:28:01 UTC]
[18:28:05] - Server does not have record of this unit. Will try again later.
[18:28:05] - Error: Could not transmit unit 06 (completed February 19) to work server.
[18:28:05] Keeping unit 06 in queue.
[18:28:05] Project: 10105 (Run 26, Clone 9, Gen 19)


[18:28:05] + Attempting to send results [February 19 18:28:05 UTC]
[18:28:08] - Server does not have record of this unit. Will try again later.
[18:28:08] - Error: Could not transmit unit 06 (completed February 19) to work server.



[18:28:08] + Attempting to send results [February 19 18:28:08 UTC]
[18:30:40] - Server does not have record of this unit. Will try again later.
[18:30:40] Could not transmit unit 06 to Collection server; keeping in queue.
[18:30:40] - Preparing to get new work unit...
Last edited by *hondo* on Fri Feb 19, 2010 10:36 pm, edited 1 time in total.
noorman
Posts: 270
Joined: Sun Dec 02, 2007 2:26 pm
Hardware configuration: Folders: Intel C2D E6550 @ 3.150 GHz + GPU XFX 9800GTX+ @ 765 MHZ w. WinXP-GPU
AMD A2X64 3800+ @ stock + GPU XFX 9800GTX+ @ 775 MHZ w. WinXP-GPU
Main rig: an old Athlon Barton 2500+ @2.25 GHz & 2* 512 MB RAM Apacer, Radeon 9800Pro, WinXP SP3+
Location: Belgium, near the International Sea-Port of Antwerp

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by noorman »

Panther-X wrote:noorman is it possible that you give me his twitter URL so that i can follow F@H more closely?
Thanks for your help
.

This is the one I found: http://twitter.com/vijaypande.

.
- stopped Linux SMP w. HT on i7-860@3.5 GHz
....................................
Folded since 10-06-04 till 09-2010
derrickmcc
Posts: 221
Joined: Fri Jul 24, 2009 12:30 am
Hardware configuration: 2 x GTX 460 (825/1600/1650)
AMD Athlon II X2 250 3.0Ghz
Kingston 2Gb DDR2 1066 Mhz
MSI K9A2 Platinum
Western Digital 500Gb Sata II
LiteOn DVD
Coolermaster 900W UCP
Antec 902
Windows XP SP3
Location: Malvern, UK

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by derrickmcc »

OK, I PM-ed Vijay and he says:
There was no code change but rather a settings change that makes WU accepts more flexible.
And I know one swallow doesn't make a summer, but on completion of my last WU on GPU3, it loaded the results not just for that WU but also the previous one in the queue:

Code: Select all

[21:12:09] Folding@home Core Shutdown: FINISHED_UNIT
[21:12:13] CoreStatus = 64 (100)
[21:12:13] Sending work to server
[21:12:13] Project: 5771 (Run 0, Clone 266, Gen 1032)
[21:12:13] - Read packet limit of 540015616... Set to 524286976.

[21:12:13] + Attempting to send results [February 19 21:12:13 UTC]
[21:12:16] + Results successfully sent
[21:12:16] Thank you for your contribution to Folding@Home.
[21:12:16] + Number of Units Completed: 1015

[21:12:20] Project: 10104 (Run 88, Clone 5, Gen 31)
[21:12:20] - Read packet limit of 540015616... Set to 524286976.

[21:12:20] + Attempting to send results [February 19 21:12:20 UTC]
[21:12:24] + Results successfully sent
[21:12:24] Thank you for your contribution to Folding@Home.
[21:12:24] + Number of Units Completed: 1016
But now 2 swallows (GPU 4) ...

Code: Select all

[21:18:04] Folding@home Core Shutdown: FINISHED_UNIT
[21:18:08] CoreStatus = 64 (100)
[21:18:08] Sending work to server
[21:18:08] Project: 5767 (Run 3, Clone 96, Gen 1001)
[21:18:08] - Read packet limit of 540015616... Set to 524286976.

[21:18:08] + Attempting to send results [February 19 21:18:08 UTC]
[21:18:12] + Results successfully sent
[21:18:12] Thank you for your contribution to Folding@Home.
[21:18:12] + Number of Units Completed: 991

[21:18:16] Project: 10104 (Run 48, Clone 6, Gen 34)
[21:18:16] - Read packet limit of 540015616... Set to 524286976.

[21:18:16] + Attempting to send results [February 19 21:18:16 UTC]
[21:18:23] + Results successfully sent
[21:18:23] Thank you for your contribution to Folding@Home.
[21:18:23] + Number of Units Completed: 992

[21:18:23] Project: 5783 (Run 6, Clone 69, Gen 49)
[21:18:23] - Read packet limit of 540015616... Set to 524286976.

[21:18:23] + Attempting to send results [February 19 21:18:23 UTC]
[21:18:32] + Results successfully sent
[21:18:32] Thank you for your contribution to Folding@Home.
[21:18:32] + Number of Units Completed: 993
Things might be looking up. :D
Image
Image
VijayPande
Pande Group Member
Posts: 2058
Joined: Fri Nov 30, 2007 6:25 am
Location: Stanford

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by VijayPande »

We're constantly trying new fixes for this. My hope is that we'll get this fixed this weekend. All the donors are understandably upset by this and Joe and I have been working long hours to fix it, so all of us would like to get this behind us!
Prof. Vijay Pande, PhD
Departments of Chemistry, Structural Biology, and Computer Science
Chair, Biophysics
Director, Folding@home Distributed Computing Project
Stanford University
Teddy
Posts: 134
Joined: Tue Feb 12, 2008 3:05 am
Location: Canberra, Australia
Contact:

Re: GPU server status 171.67.108.21, 171.64.65.71,171.67.108.26

Post by Teddy »

Well when it is fixed you can have my time & resources again, until then you can only use my resources in a limited fashion.

Edit Oh & if it is any consolation, the GPU machines that I have left running seem to be behaving themselves OK.

Regards Teddy
Post Reply