New Assignment Server feedback/problem

Moderators: Site Moderators, FAHC Science Team

Post Reply
DutchForce
Posts: 60
Joined: Sun Sep 08, 2013 12:43 pm
Location: Netherlands

New Assignment Server feedback/problem

Post by DutchForce »

I noticed that the new Assignment Server is back in action and wanted to give some feedback:

I've just got a Core15 (P7627) again on my GTX780 Ti, just like the previous time when the new AS code was running, instead of a Core17 WU (Project 13000/13001, which I normally get with the "advanced" flag). I'm using FAHClient v7.3.6 with the "advanced" flag on all my GPUs (2x GTX780 Ti's and 3x GTX660 Ti's).

Edit: I've just got another Core15 WU (P9621) on my other GTX780 Ti.
Last edited by DutchForce on Mon Sep 29, 2014 9:17 pm, edited 1 time in total.
Image
Flaschie
Posts: 69
Joined: Sun Mar 11, 2012 5:52 pm

Re: New Assignment Server feedback/problem

Post by Flaschie »

I suddenly got a core 18 (P10473), which should not be possible for an AMD/ATi-card. Is this related to the new AS? Using beta-flag...
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: New Assignment Server feedback/problem

Post by Joe_H »

DutchForce wrote:I've just got a Core15 (P7627) again on my GTX780 Ti, just like the previous time when the new AS code was running, instead of a Core17 WU (Project 13000/13001, which I normally get with the "advanced" flag). I'm using FAHClient v7.3.6 with the "advanced" flag on all my GPUs (2x GTX780 Ti's and 3x GTX660 Ti's).
The server with Project 7627 has settings for Full, Advanced and Beta, so you getting a WU from that project assigned is entirely possible. A setting of advanced is not a guaranty of getting particular projects.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
DutchForce
Posts: 60
Joined: Sun Sep 08, 2013 12:43 pm
Location: Netherlands

Re: New Assignment Server feedback/problem

Post by DutchForce »

Joe_H wrote:
DutchForce wrote:I've just got a Core15 (P7627) again on my GTX780 Ti, just like the previous time when the new AS code was running, instead of a Core17 WU (Project 13000/13001, which I normally get with the "advanced" flag). I'm using FAHClient v7.3.6 with the "advanced" flag on all my GPUs (2x GTX780 Ti's and 3x GTX660 Ti's).
The server with Project 7627 has settings for Full, Advanced and Beta, so you getting a WU from that project assigned is entirely possible. A setting of advanced is not a guaranty of getting particular projects.
Project 13000/13001 server has a much higher "advanced" weighting setting (4000) than the Project 762x server (100).
For the past 11 weeks I did get ~650 Core17 (P13000/13001) WUs with the "advanced" flag and only got ~15 Core15 WUs (when the new AS code was running the previous time).

BTW: I did edit my first post, because I got another Core15 WU on my other GTX780 Ti.

Edit: And I've just got Core15 WUs on my all my (3x) GTX660 Ti's (P7624, P7621 and P8018).
Image
PS3EdOlkkola
Posts: 177
Joined: Tue Aug 26, 2014 9:48 pm
Hardware configuration: 10 SMP folding slots on Intel Phi "Knights Landing" system, configured as 24 CPUs/slot
9 AMD GPU folding slots
31 Nvidia GPU folding slots
50 total folding slots
Average PPD/slot = 459,500
Location: Dallas, TX

Re: New Assignment Server feedback/problem

Post by PS3EdOlkkola »

If the weightings on the AS are 40:1 in favor of Project 13000/13001, it appears the algorithm being used may have an issue with assigning work units. I'm also getting many more Core 15 work units on 780ti, 780's. After installing a 980 over the weekend, it has only been getting Core 15 work units, not one Core 17 (has "advanced" flag set). Joe_H, I think you may want to look again at the code to see if the weighting factor for the AS is operating as designed.
Image
Hardware config viewtopic.php?f=66&t=17997&p=277235#p277235
Joe_H
Site Admin
Posts: 7927
Joined: Tue Apr 21, 2009 4:41 pm
Hardware configuration: Mac Pro 2.8 quad 12 GB smp4
MacBook Pro 2.9 i7 8 GB smp2
Location: W. MA

Re: New Assignment Server feedback/problem

Post by Joe_H »

A 40:1 ratio means nothing if there are a limited number of WU's available for a particular system configuration. And at times in the past few months people have been getting Core_15 work instead, before any of the recent AS changes. As a forum moderator I have no additional access to the code or the servers, so I can't examine it any more than regular folders.

The current test of the updated AS code could be connected to these assignments, or not related at all. Joe Coffland is responsible for the coding and testing of the AS code changes and has posted elsewhere that he did fix where persons with ATI cards were getting Core_18 assignments when they shouldn't. If he identifies a problem related to this type of assignment, then he may post about it when fixed.

P.S. The only guaranteed way of getting Core_17 WU's when they are available is to run GPU folding on a Linux system. Of course when they are unavailable the GPU will not get any assignment at all
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Calcii
Posts: 57
Joined: Fri Dec 16, 2011 12:47 pm

Re: New Assignment Server feedback/problem

Post by Calcii »

Only 15 cores on 780 ti with flag advanced. Plz anyone tell doctors or v.j. pande about low quantity of 17 core units. I hate 15 cores, believe in remove that jobs at all
EXT64
Posts: 323
Joined: Mon Apr 09, 2012 11:54 pm

Re: New Assignment Server feedback/problem

Post by EXT64 »

It sounds like there is still a configuration problem with the Core 17 server (Joe Coffland did say researchers were still getting use to the new system, so this is not a surprise). When running the old AS I only get 1300x, when the new AS turns on I get only Core 15. It sounds like the new AS is a really great upgrade (better visibility of the entire fah network) but as with any major upgrade there will be some teething to get through. We just need to be patient for a week and report what we see.

Edit: Also there is nothing "Wrong" with Core 15, it is doing useful science. It is unfortunate though that PG has decided to not re-benchmark it with QRB as I imagine that causes a lot of donor resentment and WU dumping.

Edit2: My 780ti in windows has been happily chugging through Core15 WUs for about a day now.
PS3EdOlkkola
Posts: 177
Joined: Tue Aug 26, 2014 9:48 pm
Hardware configuration: 10 SMP folding slots on Intel Phi "Knights Landing" system, configured as 24 CPUs/slot
9 AMD GPU folding slots
31 Nvidia GPU folding slots
50 total folding slots
Average PPD/slot = 459,500
Location: Dallas, TX

Re: New Assignment Server feedback/problem

Post by PS3EdOlkkola »

@Joe_H, my apologies, I confused you with Joe Coffland.

Over the last 8 hours, Core 17 units are being replaced by both Core 15 (on Nvidia) and Core 16 (on AMD). I clearly understand that all work units have to get completed, but it seems terribly coincidental that all these older work units suddenly have superior priority over Core 17 with a simultaneous change of AS code. It may be a lack of Core 17 work units, but unless notified differently, the only announced and visible change to donors is the AS code change.

I'm simply suggesting -- to Joe Coffland now -- to look at the AS code one more time.
Image
Hardware config viewtopic.php?f=66&t=17997&p=277235#p277235
billford
Posts: 1003
Joined: Thu May 02, 2013 8:46 pm
Hardware configuration: Full Time:

2x NVidia GTX 980
1x NVidia GTX 780 Ti
2x 3GHz Core i5 PC (Linux)

Retired:

3.2GHz Core i5 PC (Linux)
3.2GHz Core i5 iMac
2.8GHz Core i5 iMac
2.16GHz Core 2 Duo iMac
2GHz Core 2 Duo MacBook
1.6GHz Core 2 Duo Acer laptop
Location: Near Oxford, United Kingdom
Contact:

Re: New Assignment Server feedback/problem

Post by billford »

PS3EdOlkkola wrote: I'm simply suggesting -- to Joe Coffland now -- to look at the AS code one more time.
Preferably in the comfort of his office whilst the old code runs on the server.
Image
PS3EdOlkkola
Posts: 177
Joined: Tue Aug 26, 2014 9:48 pm
Hardware configuration: 10 SMP folding slots on Intel Phi "Knights Landing" system, configured as 24 CPUs/slot
9 AMD GPU folding slots
31 Nvidia GPU folding slots
50 total folding slots
Average PPD/slot = 459,500
Location: Dallas, TX

Re: New Assignment Server feedback/problem

Post by PS3EdOlkkola »

Across all my systems using three different internet providers in two physically different locations, they are all unable to connect to the assignment server. Half my GPUs are idle at the moment, and I suspect they all will be in a couple of hours. All other tools I use (both automated and manual) show the problem is not with either the rigs or the internet connections. The message all systems are receiving is this:

"19:31:33:WARNING:WU02:FS01:Failed to get assignment from 'assign-GPU.stanford.edu:8080': Failed to connect to assign-GPU.stanford.edu:8080: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."

Post edited by Mod.
Forum rules expressly prohibit any kind of recruiting or the advertisement of services or products.
Image
Hardware config viewtopic.php?f=66&t=17997&p=277235#p277235
7im
Posts: 10179
Joined: Thu Nov 29, 2007 4:30 pm
Hardware configuration: Intel i7-4770K @ 4.5 GHz, 16 GB DDR3-2133 Corsair Vengence (black/red), EVGA GTX 760 @ 1200 MHz, on an Asus Maximus VI Hero MB (black/red), in a blacked out Antec P280 Tower, with a Xigmatek Night Hawk (black) HSF, Seasonic 760w Platinum (black case, sleeves, wires), 4 SilenX 120mm Case fans with silicon fan gaskets and silicon mounts (all black), a 512GB Samsung SSD (black), and a 2TB Black Western Digital HD (silver/black).
Location: Arizona
Contact:

Re: New Assignment Server feedback/problem

Post by 7im »

@ PS3EdOlkkola, please provide client version, slot types, hardware config, etc.

They probably won't roll back any more, but will fix going forward, and they need that info to fix it. Even an outsourced tester would tell you that. ;)
How to provide enough information to get helpful support
Tell me and I forget. Teach me and I remember. Involve me and I learn.
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: New Assignment Server feedback/problem

Post by bruce »

When something in a server is down, rolling back the code is an inappropriate action until somebody has had a chance to observe which component(s) were associated with the crash and hopefully gather some clues about WHY it crashed. After the capture of that information is complete, rolling back may or may not be necessary.
DutchForce
Posts: 60
Joined: Sun Sep 08, 2013 12:43 pm
Location: Netherlands

Re: New Assignment Server feedback/problem

Post by DutchForce »

@ PS3EdOlkkola, I think you are still using FAHClient v7.3.6. I was still using this older version and I had the same problem and message as you and decided to upgrade to v7.4.4, which has the capability to access the second Assignment server when you can not connect to the first AS. After the upgrade it failed to connect to the first AS, but could get an assignment from the second AS. So I think the first AS was (temporarily) offline to do some work.

BTW, I still get only Core15 WUs (P8018 and P762x) on all my GPUs.
Image
billford
Posts: 1003
Joined: Thu May 02, 2013 8:46 pm
Hardware configuration: Full Time:

2x NVidia GTX 980
1x NVidia GTX 780 Ti
2x 3GHz Core i5 PC (Linux)

Retired:

3.2GHz Core i5 PC (Linux)
3.2GHz Core i5 iMac
2.8GHz Core i5 iMac
2.16GHz Core 2 Duo iMac
2GHz Core 2 Duo MacBook
1.6GHz Core 2 Duo Acer laptop
Location: Near Oxford, United Kingdom
Contact:

Re: New Assignment Server feedback/problem

Post by billford »

7im wrote: They probably won't roll back any more, but will fix going forward
So we're stuck with high-end GPUs running low-value Core15's until Joe finds the bug(s)… :(

Ah well, such is life. Please ask those concerned to ensure he has a plentiful supply of coffee :D
Image
Post Reply