Problems downloading statistics files?

[agrikk]

In the last few days I noticed that my regular download of files from apps.foldingathome.org have been failing or taking an incredibly long time to complete.

https://apps.foldingathome.org/daily_us ... ry.txt.bz2
https://apps.foldingathome.org/daily_te ... ry.txt.bz2

normally take a few seconds to download, but now it's taking upwards of twenty minutes with multiple retries and connection closures. I'm on the end of a ten gigabyte pipe and I'm getting 44kB/s download speeds.

Is there something going on that I should know about? Is this a temporary phenomenon?

[PantherX]

Welcome to the F@H Forum agrikk,

I have asked the people responsible about this so let's see what happens. I am aware of another report about the Individual Donor Stats not loading (viewtopic.php?f=16&t=35987).

[UofM.MartinK]

If many people need these files regularly, and the front-end is not easy to scale up for whatever reason, would be publishing the stats as torrent a potential solution?

This could even be community-driven, if there are no legal implications against it...

[PantherX]

The update is that everything seems fine from the back-end. If it is a routing or a network issue, there's almost nothing that F@H can do about it.

[agrikk]

Welcome to the F@H Forum agrikk,

I have asked the people responsible about this so let's see what happens. I am aware of another report about the Individual Donor Stats not loading (viewtopic.php?f=16&t=35987).

Thank you!

I've been pulling these stats files since 2013 and since covid, I understand that availability (both in data files and actual work units) has been taxed.

[UofM.MartinK]

The update is that everything seems fine from the back-end. If it is a routing or a network issue, there's almost nothing that F@H can do about it.

Other than (also) distributing it as torrent?

Each file with date tag - why not? Is it worth an experiment?

Update: add a fancy way of signing to guarantee authenticity... perhaps even as part of the filename?

[PantherX]

...Other than (also) distributing it as torrent?

Each file with date tag - why not? Is it worth an experiment?

Update: add a fancy way of signing to guarantee authenticity... perhaps even as part of the filename?

I didn't mention the feature during my conversation. However, you're more than welcomed to submit your ideas here (assuming that it hasn't already been suggested before): https://github.com/FoldingAtHome/fah-issues/issues

[UofM.MartinK]

Doesn't seem to be in high demand, based on the reaction here, and the files are smaller than I thought

If bandwidth or availability at various locations would be a problem regularly, once could also distribute them with a CDN like cloudflare.

But that doesn't seem to be a case, do no need to fix something that ain't broken

[Neil-B]

phew ... the torrent elephant in the room seems to have wandered off of its own accord ... relaxes, smiles, heart rate lowers ...

[cine.chris]

I'm trying to obtain a daily archive of FAH user/team bzipped files for 2020 Jan-Nov.
Hoping that someone on this thread might be able to help me locate a source.

[bruce]

You may be out-of-luck.

As far as FAH is concerned, the database is constantly changing and always up-to-date. There's no need for a history. (Well, there are backups, of course, but they're not distributed because they're obsolete.) The data are updated minute-to-minute as new reports of completed WUs are come in. No part of the database is considered as static. (Well, except for the current statistics for Donors who have died or left FAH, but there's no attempt to identify them.)

The third party stats sites do retain one (or more) previous flat-file and calculate the differences between the latest and that previous version. From that, they are able to calculate trends and generate the kind of history you're looking for. They may be able to provide the historic information you're looking for.

[cine.chris]

I connected with agrikk.
He has data back to 2014, which I think will be more than adequate for the patterns that I want to look at.
@Bruce
What part of So Cal?
I'm a So Cal native, born in Burbank, raised in the 'valley', North Hollywood.
Lockheed moved me to Georgia in '93 with the F22 program.