It's getting better now, isn't it?

Moderators: Site Moderators, FAHC Science Team

ajm
Posts: 750
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

It's getting better now, isn't it?

Post by ajm »

Is it just me or are the servers becoming smoother, at last? I'm reaching some 50% of my Total Estimated Points Per Day! And most of the time all my ressources are now used.
The servers stats are now constantly over 100,000 jobs per hour: https://apps.foldingathome.org/serverstats - kudos to LTT (I suppose) for their linus1 server!
Feels great! :D
JimboPalmer
Posts: 2522
Joined: Mon Feb 16, 2009 4:12 am
Location: Greenwood MS USA

Re: It's getting better now, isn't it?

Post by JimboPalmer »

I was up to 1/2 expected PPD, late last week, back to 1/4 now. When researchers get back Monday I am optimistic more resources will get online.

As to how well they are doing, look at this graph, click on Weekly and Users Active
https://folding.extremeoverclocking.com ... ary.php?s=
That is impressive!
Tsar of all the Rushers
I tried to remain childlike, all I achieved was childish.
A friend to those who want no friends
ajm
Posts: 750
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: It's getting better now, isn't it?

Post by ajm »

Yay!

Image
WTS
Posts: 97
Joined: Sun May 19, 2019 5:49 pm
Location: Arkansas

Re: It's getting better now, isn't it?

Post by WTS »

It seems to be better here, although I do have to reboot now and then to get a work unit or two.
But, may I ask, are the scientists learning anything from the huge increase in computing power, and how long
will it take to get the information into the hands of the researchers that need the information? It'd be nice to
know how forward we're moving in solving this. Is F@H regular work also progressing faster? Thanks!
Reality is what you stumble over when you walk around with your eyes closed.
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: It's getting better now, isn't it?

Post by Neil-B »

I expect we will get some form of News announcement as/when the team have a chance to do one - but I expect resource is focused at resolving the server throughput challenges ... I would expect the knowledge is already filtering out to the researchers as this is I believe shaping the directions and scale of the new projects - which are spawning fast and furious at the moment (keeping the volunteer beta testers well busy).

Given the current pandemic work is being given priority in the queues and as I understanding the issues are not lack of WUs but more lack of ability to farm out quick enough to keep all the expanded community busy I believe that for the most part it is Covid related WUs being released ... Some regular work may well be getting out at the moment and once expansion of
throughput on the servers is resolved then I expect that as long as regular science WUs are being generated they will get folded.

(Brain wanders off at a tangent)

I wish it were possible to "solve" Covid ... I am simply hoping that all the myriad efforts inc FAH, and lifestyle changes, and shifts in societal behaviours can get us to the point where a new normal can be established that doesn't require vast groups of people to be isolated and where enough has been learned that ways of living develop and sustain that addresses some of the deep seated causes of the pandemic (and I don't mean the virus itself).
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
Vester
Posts: 71
Joined: Sun Dec 02, 2007 4:29 pm
Hardware configuration: Gigabyte Aorus Z590 Pro AX, Intel i9-10850K, 32GB Crucial Ballistix DDR4-2600, Samsung NVMe EVO 980 Pro 256GB, CoolerMaster liquid cooler ML360, Nvidia Titan X (Pascal), Dell Nvidia RTX 3080 10GB 4Y12V, Pop!_OS.
Location: Fair Play, SC

Re: It's getting better now, isn't it?

Post by Vester »

I am getting plenty of work, but my backlog of uploading jobs is twenty at this time. FAH has used more than 20GB of bandwidth in the past few days trying to upload work units. Most fail repeatedly at 40% to 99% uploaded. If the servers would limit their number of simultaneous connections, the throughput would improve.
Frogging101
Posts: 78
Joined: Wed Mar 25, 2020 2:39 am
Location: Canada

Re: It's getting better now, isn't it?

Post by Frogging101 »

Vester wrote:I am getting plenty of work, but my backlog of uploading jobs is twenty at this time. FAH has used more than 20GB of bandwidth in the past few days trying to upload work units. Most fail repeatedly at 40% to 99% uploaded. If the servers would limit their number of simultaneous connections, the throughput would improve.
Possibly. But I imagine that to some extent, it's just the sheer volume of incoming requests. It's like a friendly DDoS attack. Even if the server were configured to ignore half of the requests directed at it, all of those incoming packets are still filling up the "pipe".

I have never been involved with operating a server that deals with any significant amount of traffic, but I would be surprised if mitigating something like this were as simple as limiting the number of connections.
Neil-B
Posts: 1996
Joined: Sun Mar 22, 2020 5:52 pm
Hardware configuration: 1: 2x Xeon E5-2697v3@2.60GHz, 512GB DDR4 LRDIMM, SSD Raid, Win10 Ent 20H2, Quadro K420 1GB, FAH 7.6.21
2: Xeon E3-1505Mv5@2.80GHz, 32GB DDR4, NVME, Win10 Pro 20H2, Quadro M1000M 2GB, FAH 7.6.21 (actually have two of these)
3: i7-960@3.20GHz, 12GB DDR3, SSD, Win10 Pro 20H2, GTX 750Ti 2GB, GTX 1080Ti 11GB, FAH 7.6.21
Location: UK

Re: It's getting better now, isn't it?

Post by Neil-B »

It iisn't !!
2x Xeon E5-2697v3, 512GB DDR4 LRDIMM, SSD Raid, W10-Ent, Quadro K420
Xeon E3-1505Mv5, 32GB DDR4, NVME, W10-Pro, Quadro M1000M
i7-960, 12GB DDR3, SSD, W10-Pro, GTX1080Ti
i9-10850K, 64GB DDR4, NVME, W11-Pro, RTX3070

(Green/Bold = Active)
bruce
Posts: 20824
Joined: Thu Nov 29, 2007 10:13 pm
Location: So. Cal.

Re: It's getting better now, isn't it?

Post by bruce »

There are periods of server-overload and gaps where work can be downloaded or uploaded and I think the gaps are becoming more plentiful.

It's still a supply_and_demand issue. FAH has dramatically increased supply and concurrently I suppose demand has backed off. It's hard to measure the relative significance of either one based on looking at the status of my clients a few times a day and I don't closely monitor my points like ajm did in his second post.
I would be surprised if mitigating something like this were as simple as limiting the number of connections.
When the bandwidth limiters kick in, you get the message "No WUs available for this configuration" but you get exactly the same message when there's something wrong with your configuration or the servers really have run out of a class of WUs. :(

My impression is that downloading has improved somewhat but reduced congestion there (if that's what happened) has allowed us to see congestion on back-end systems ... like the points DB.
ajm
Posts: 750
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: It's getting better now, isn't it?

Post by ajm »

Hum, my second post doesn't show my points. It's the Aggregate Teams Summary, with a view of FAH's total daily points in the course of last month.

EDIT: And it shows - that was the point - that the upwards progression has been accelerating these last days.
fernandg
Posts: 1
Joined: Mon Mar 30, 2020 8:35 pm
Location: California

Re: It's getting better now, isn't it?

Post by fernandg »

It's been a lot better within the past two to three days compared to last week.
IkkeDus
Posts: 14
Joined: Wed Jun 18, 2008 10:42 am
Hardware configuration: Q9550 @ 2.8 GHz
WIN10 x64
2x Radeon R9 280X-3GB
1x Radeon R9 7950-3GB
Location: Amsterdam, The Netherlands

Re: It's getting better now, isn't it?

Post by IkkeDus »

It's getting better in the sense of getting work, less idle time, BUT I do see a problem with uploading the results ("Server responded PLEASE_WAIT") which is not good for the PPD as well.

Although it is likely worse if the results are not getting uploaded at all (in time)...
Q9550 @ 2.8 GHz | 2x R9 280X-3GB | HD 7950-3GB | Win10 x64

Image
ajm
Posts: 750
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: It's getting better now, isn't it?

Post by ajm »

Browsing my logs, I see that the upload to 13.90.152.57 was often slow to very slow lately.
37.187.12.48 was fast.
13.82.98.119 was fast.
140.163.4.231 was ok.
Is that kind of feedback useful?
WTS
Posts: 97
Joined: Sun May 19, 2019 5:49 pm
Location: Arkansas

Re: It's getting better now, isn't it?

Post by WTS »

Help!!! F@H is getting spoofed badly. Here are the details: <https://www.proofpoint.com/us/threat-in ... l-campaign>
Reality is what you stumble over when you walk around with your eyes closed.
ajm
Posts: 750
Joined: Sat Mar 21, 2020 5:22 am
Location: Lucerne, Switzerland

Re: It's getting better now, isn't it?

Post by ajm »

Well, there's not much FAH can do. And, don't worry, if you're here, you have chosen the real one.
Post Reply