New Servers Idle?

[ajm]

There has been a new spike these last hours: https://apps.foldingathome.org/credit-log

[UofM.MartinK]

Yes, noticed that - and a very prominent dip exactly before that. Another good indication that this is when the the stats system finally processes the data. And where no abrupt spikes happen, the data are a pretty good representation of what's going on, I believe. the daily repeating "waveform" is also pretty interesting, a big group of clients seems to have a strong dependence on the time of the day

[bruce]

Not everybody is a FAH "gearhead" Remember FAH runs on home computers and it's moderately common for home computers to be turned off when people are sleeping or at work or ...

Yes, there are a lot of FAH Clients that run 24x7, but not 100%. That also means that there are certain times during a 24hr cycle when you can expect to get a new assignment.

In the early days of FAH, the daily cycles were probably more pronounced with a steeper rise and a more gradual descent (which can be seen today) because folks had to connect their modem to upload whatever was completed during the night.

[Neil-B]

Given that FAH is a "global phenomenon" I am slightly surprised at how pronounced the cycle is - but I guess is shows where the most folders are concentrated .. and the dips may well not be as low as they could be due to not just 24/7 folders but the effect of the global folding community?

[Endgame124]

Could it be based on the researchers? Every morning, they come in, do something to upload more work, which populates a bunch of clients. The fast clients run out of work quickly, and then the totals decrease over time until the next batch upload?

[UofM.MartinK]

I wasn't surprised by the 24 hour cycles, I just find them interesting... either mostly client/user-dependent (which region(s) dominate, who of the folders shuts down their computer at night, or only lets it fold at night because it's doing something else during the day), but perhaps there is a WU availability dependece as well, as suggested by Endgame124 - will try to correlate this later today

[PantherX]

Could it be based on the researchers? Every morning, they come in, do something to upload more work, which populates a bunch of clients. The fast clients run out of work quickly, and then the totals decrease over time until the next batch upload?

Generally speaking, researchers are involved when they are generating new Projects or additional simulations of an existing project if they need more data. Once the initial "batch" of WUs are done, the researchers are out of the loop, i.e. the Servers hand the WUs out and once the completed ones arrive, automatically generate the next sequence and will assign it.

Once there's enough statistical data, researchers can further analyses it and may tweak the settings (more sample collection or spin up a new series of Projects) depending on the data.

[UofM.MartinK]

This is a first ugly plot of my "experimental composite stats", combining key aspects of the server stats and credit logs:

It visualizes some interesting trends, including a "sneak peak" into my suspicion that the global F@H "supercomputer" is "only" about 60-80% as efficient as it (maybe) could be given the current number of clients, and interestingly, this does no longer seem to be due to the WU availability or WS speed.



Plot Walk-through:

The 3 solid lines show available WUs as reported by server-stats, split into GPU, CPU and total.
For the last 10+ days, there were always 300-500k WU available, cool!
And always over 200k CPU WUs, which is important because most of Covid-19 is CPU as of now.

The dashed purple line with the daily fluctuations is the hours/hours as reported by credit-log,
with a little forward-looking smoothing to deal with the short-term spikes due to delayed processing.
The cool thing here:
Trendline is going up, currently on average there are 500.000 hrs/hrs logged -
just slightly above what the WS report is available. In other words:
F@H always has WUs ready for the next 30-60 minutes.

The dotted lines towards the bottom are the assign rates (actual reported ones, not what the servers are configured for),
and in blue the Credits logged/hr, which is kinda an assign rate, just at the other end of the "pileline" ("collect rate"?)


WARNING! The rest of the post goes into nitty-gritty details!

And this is where things become interesting:
The green dotted line is the assign rates as reported by AS1 and AS2, I assume that is the number of assignments they hand out.
The purple line is all the individual assign rates reported by the WS's added up. Far less than what the two AS hand out!

The blue line ("collect rate") tracks the purple line ("WS assign rate") very well, a strong indication that almost no WUs are lost by the clients.
Big Kudos to all folders for being so reliable!!

(The JSON objects, which server-stats builds its numbers from, include more detail than visualized in the end. That's how I was able to derive some of the above stats)


Now my quest continues....
Hunting down why the AS assign rate and the WS cumulative assign rates are so different.
BTW, I think you can see that difference in real-time on the server-stats page: In the first table,
the cumulative assign rate for CPU & GPU are listed, but they are always a lot smaller than the TOTALS right beneath it.

Different Hypothesis/possible reasons so far:

#1)
There could be other things the AS count as assignment, for example, the JSON file includes things like:
"id_rate":3.156667,
"assign_rate":14.06,
"no_assign_rate":132.873333,
"blacklist_rate":2.49,
And "assign_rate" is what is reported - is it the total, and one would subtract "id_rate" and "blacklist_rate" to derive the net assign rate actually handed off to the WS?

#2) After clients receive an assignment from the AS, they fail to ask the WS. Unlikely that this happens on such a large scale.
UNLESS some "rogue folders" use such techniques to "cherry pick" WUs (or WSs) they like best.

#3) After clients receive an assignment from the AS, the WS says "no WUs available"
I definitely have evidence in my log files that this happens, but I need more statistics to see if that happens often enough to explain the difference.
If that would be the case, F@H should be able to tweak the "indirect interaction" between AS and WS relatively easily.

In a next step, I will analyze my own client logs to see how their data correlates with above plot, especially building statistics for the reported reasons of not getting WUs (and how much of the times my clients are idle due to that.)

Hypothesis #4):
Perhaps the 24-hours "oscillating" behavior is not only a function of when the folding computers are online or available for folding, but also of "WU request storms" during certain times of the day which in turn kick a significant amount of clients into a waiting-for-WUs state for more hours than necessary.

I am looking for any insight or feedback!

[Tohya]

Try asking on discord

https://discord.gg/97Z2wBF