A plea to open the queue.dat

[smoking2000]

Folding@Home is a great project with one of the most worthwhile causes, by using Open Source tools groundbreaking research is done and released into the open for all to benefit. Unfortunately the Folding@Home client and its data files as Closed Source as Microsoft junk. Some of the arguments for this are good, like the ability to guarantee the integrity of the research results. Others are not, like the Security Through Obscurity arguments.

I hereby make a plea to the Folding@Home staff, specifically Vijay Pande and Adam Beberg, to open the format of the queue.dat so that the enthusiastic 3rd party developers can build tools to monitor the FAH client which are as robust as the FAH client itself.

As is understood by all developers with much experience in the Folding@Home realm, the current data files which are human readable are too unreliable to build software around. The unitinfo.txt is the prime example in this case. Too often than we'd like it contains invalid or incorrect data, whereas the FAH client does have the correct data in its queue.dat. Another exaple is the FAHlog.txt, it is hard to parse and misses many of the useful bits of information contained in the queue.dat.

Because the queue.dat is such a useful and reliable source of data on a running FAH client it's used by all major Folding@Home client monitors. InCrease, the leading FAH client monitor on Mac OS X, has been using qd to parse the queue.dat for a long time, qd was ported to Mac OS X for that purpose. Recently the most popular FAH client on Windows & Linux, FahMon, incorporated code from qd to parse the queue.dat to finally have a reliable source of information. And I myself rewrote my platform independent and webbased client monitor, FCI, to use qd to parse the queue.dat and build upon that information. qd, in being the only tool to be able to read the queue.dat
has proven so useful that the code lives on even after its author has passed away (you're welcome).

I hope this illustrates the need of the community to have the ability to reliably monitor their FAH clients, and the limited resources we have to accomplish this. The FAH client itself does not provide in this need unfortunately.

This plea was triggered by the changes to the queue.dat since the release of the 5.91 & 6.00 FAH clients which write new data to the
queue.dat but which I haven't figured out yet, bytes 612-615 of each queue index being the last unknown bit (pun intended), all other new pieces of data I had the luck to being able to figure out. But this luck has run out.

All I ask is that you release the documentation of the queue.dat, the C struct you use to define it and the meaning of its members. Are they Big Endian, or Little Endian, or can it be both. How should the binary value be manipulated to make it into a useful form (e.g. for humans to read). I do not expect the full source of the FAH client to be released, as this is a futile endeavor (with good reasons, even though most Open Source authors don't agree), I only ask to open the parts we in the community need to work together with the client. Like how Microsoft is required by the EU now to document its protocols so 3rd parties can build software that seamlessly works together with it.

To put more weight behind this plea I also announce a challenge. If the Folding@home staff discards this plea like all others before it have, I will award a brand new 60GB Playstation 3 to the person who can tell me how to read the remaining field in the queue.dat. So that qd remains as useful to the community as it has always been. This is also in the best interest of the FAH staff looking at the amount of usage qfix receives, which is based on the knowledge that was gained in the development of qd. I cannot guarantee that qfix will not break the FAH client by writing invalid data to the queue.dat without proper understanding of the queue.dat (as I do now, and being the only person to take care of this software it's up to me to keep it working properly). I would hate for software which is distributed from my personal website, and thereby has my name and reputation attached to it, to harm the scientific research done by FAH, but I also refuse to let these immensely useful pieces of software wither and die, and leave the FAH community (the most valuable resource the project has IMHO) without these useful tools and have their abilities in this project set back in time to the point where we didn't have ways to work with the FAH client in these ways.

Vijay, please consider this request seriously. But be warned, I am fed up by the empty promises made by the FAH staff. The old FCF content, another invaluable resource to the community, is still unavailable to this same community, and this loss is not made up by the existence of foldingforum.org as much as I would like it to be. We also live without complete project lists, the other psummary pages are still not merged. The silliness continues, which I regard as a crying same and major loss to the community. The Folding@Home ecosystem can be so many times better, if we were able to work together in more ways than just running the FAH client. The community wants this badly, please help us help the project better.

[John Naylor]

I agree entirely.

[7im]

I disagree.

He has some valid questions, but also makes invalid statements. However, one has to be able to accept that the answer to some questions will always be NO.

Integrity of results is paramount. The argument smoking2000 makes against Security through Obscurity is crap. Any measure taken to help assure data integrity will always be considered reasonable. Obscurity is not the only Security the project uses, so claiming the use of obscurity makes the project vulnerable is ****. The project does not have to prove a negative to show that it is not vulnerable. Can you prove that God does not exist? Can you prove that Martians do not exist? Can you prove that obscurity does not help keep data secure?

Yes, an open queue.dat may be helpful to 3rd party applications, but could also be hurtful. How do you balance that? And the data in the queue.dat is rarely more helpful that what is already in the unitinfo.txt file and the fahlog.txt file. When the unitinfo.txt file contains incomplete info, so does the queue.dat file. And the 3rd party tools seem to be doing just fine without an open format, so that's not a convincing argument either. Yes, their work hasn't been easy lately, but what value is not worth the work?

Which maybe leads us to your true motivation... you finally failed to reverse engineer some part of the queue.dat file, so you feel slighted, and your frustration leads you to this challenge. You lost the battle, so you are advertising for a hacker mercenary to do your dirty work for you.

(you're welcome)? That's rather conceited, don't you think? Several others offered to keep the qd application up to date as well as you. You took up the task, so the others let you continue. The programmer side of you does an excellent job. The public relations side of you could use some improvement. If you can't continue updating qd for the benefit of the project users without getting more than self-gratification, then maybe you aren't the right person to continue doing it.

And while we do not advertise the fact that the old forum is still up and available for anyone to read, it is not sturdy enough to use on a regular basis by a large number of people. FOR THE LAST TIME, THE DATA IN THE OLD FORUM IS NOT LOST!!! If you can't figure out how to find the site (which hasn't moved) then maybe you're not as accomplished a programmer as you might think. So I'll repeat my usual challenge to anyone else that says the old data is lost. Ask me anything about the old forum. Your last post, your post count, thread name, etc.

Yes, I agree that more standardized data formats between projects, clients, etc. would be very beneficial. I would love to see a better unitinfo.txt file to make 3rd party apps more useful. But to assume that is always possible across platforms like CPUs, GPUs, and PS3s is ridiculous.

If Pande Group wants to comment, then they will or they won't. Pleading won't change policy. Positive tangible proof that change is beneficial to project would be your best argument, and so far, that has been lacking, IMO.

Edit: Remember this is a family forum -UF

[Ivoshiee]

I will welcome the opening up of the queue.dat format.
As long as the other WU-client tracking abilities are somewhat unreliable (unitinfo.txt and FAHlog.txt) the 3rd party tools need to have better way to do it - parsing the queue.dat.
I would like to see more information if the future queue.dat as well. For starters there should be WU points listed.

One thing I do not understand is the presence and large usage of the qfix. Why it is being promoted? Someone may argue that it is for restoring a finished WU, but is this restoring needed? If the FAH client is broken then the Pande Group are perfectly able to fix the FAH client itself and will need no hack from the community.

Also, http://folding.stanford.edu/English/License:

You may not alter the software or associated data files.

[smoking2000]

I disagree.

That is your right. Just like it is mine to think otherwise.

Yes, an open queue.dat may be helpful to 3rd party applications, but could also be hurtful. How do you balance that?

I call **** on the 'hurtful' part. There is no data in the queue.dat that has to be kept secret, there is none. Even the bit of data that I have figured out, released information about & code for, even got kicked out of the beta team for, and kept in a private code branch on request of Vijay, the arguments to not release this information are crap. They show that the security model of FAH is broken by design if the release of this information is considered a vulnerability. All I did was connect the dots of information which is public. That's how I figured this out, and if I can do it, anyone who is willing can. You don't need any special reverse engineering skills for that. The fact that the client.cfg is human readable & editable breaks the security of this bit of information.

The main point I'm trying to make in the plea is that we as a community shouldn't have to reverse engineer a data file get decent interoperability out of the FAH client. The means to do this should be provided by upstream. Robust client monitors are in their best interest too. They allow for serious folders and beta testers to easily spot problems, and to some extent even debug these problems (I have code for that too). Unreliable client monitors reduce the effectiveness contributions by serious folders.

And the data in the queue.dat is rarely more helpful that what is already in the unitinfo.txt file and the fahlog.txt file. When the unitinfo.txt file contains incomplete info, so does the queue.dat file. And the 3rd party tools seem to be doing just fine without an open format, so that's not a convincing argument either. Yes, their work hasn't been easy lately, but what value is not worth the work?

What you say here is just wrong, the queue.dat is generally more helpful, the unitinfo.txt is unreliable and the FAHlog.txt is errorprone to parse. When the unitinfo.txt contains incomplete information this is because the work/wuinfo_??.dat contains bad data by the FahCore, the queue.dat at that time has correct information by the FAH client.

The queue.dat also provides a wealth of information that is neither in the FAHlog.txt or in the unitinfo.txt, and if the information is in the FAHlog.txt it's hard to reliably extract it. The WU timestamps for instance are a good example, all 5 timestamps can be easily read from the queue.dat but not from the FAHlog.txt.

The 'issue' time is not recorded in the FAHlog.txt or not easily found in it (I believe it's only printed to the FAHlog when WU uploads have failed and are retried).
The 'begin' time can be found in the FAHlog.txt in the line:

Code: Select all

[05:53:25] Working on Unit 06 [January 18 05:53:25]

The 'end' time is not easily found but is in this section of logmessages in the FAHlog:

Code: Select all

[22:28:02] CoreStatus = 64 (100)
[22:28:02] Unit 6 finished with 83 percent of time to deadline remaining.
[22:28:02] Updated performance fraction: 0.826848

Although this doesn't list the date explicitly, only the time.

All these values are easily read from the queue.dat but not from the FAHlog.txt and it's impossible to get all these values from the unitinfo.txt.

For me this information helped me to save a few WUs when I noticed an extremely long time between the 'issue' timestamp and 'download' timestamp, turned out there was a collision storm on the network where those clients were, which I could then quickly get fixed. It would have taken hours or more to upload those 50+ MBs of wuresults from the FAH clients on that network.

Another piece of information easily read from the queue.dat but which is not stored elsewhere is the Packet Size Limit, which used to cause problems with bigger than expected wuresults for some advanced bigWU project, before the upload restrictions were loosened in the fah6 client. This is one of the problems qfix could fix for you before the release of fah6.

Fact is that we 3rd party devs are not doing just fine, we have to resort to reverse engineering to come to a level of interoperability with the FAH client. To me this is fundamentally wrong. The only data source easily read by a machine and not humans is the queue.dat, and the only way we can do that is through the results of reverse engineering which is wrong IMO.

Which maybe leads us to your true motivation... you finally failed to reverse engineer some part of the queue.dat file, so you feel slighted, and your frustration leads you to this challenge. You lost the battle, so you are advertising for a hacker mercenary to do your dirty work for you.

My motivations behind my plea to open the queue.dat are manifold. The fact that I ran into a wall figuring out the last bit is one of them indeed. I believe that this work benefits the community greatly, and that this work needs to be done. Because I think that way, I did it myself initially (as in the saying: If you want something done (right/at all), you gotta do it yourself), but I've run into my own limitations so called for the help of others. I couldn't think of a better way to motivate others to help solve my problem than by offering a shiny PS3 as a reward for their efforts on my and the communities behalf. As I said, I'm putting my weight behind this issue. I can spare the money to donate a brand new PS3 to a fellow folder to add to the horde of undead working for the cause, I see it as a challenge in which everybody wins, me, the community, and the project.

FWIW, my main trigger to actually post this plea was the edit by Ivo to the Unitinfo.txt Wiki page, in which I wrote about the unreliability of this file. This combined with the recent changelog is saw of FahMon, were my primary motivations. In my opinion the community deserves a reliable way of monitoring their FAH clients, as they're allowing FAH to run on their systems after all. I think it would benefit the FAH project in whole if the FAH staff and 3rd party devs could work together better, by helping us monitor their software, to better the community now and for the future.

(you're welcome)? That's rather conceited, don't you think? Several others offered to keep the qd application up to date as well as you. You took up the task, so the others let you continue. The programmer side of you does an excellent job. The public relations side of you could use some improvement. If you can't continue updating qd for the benefit of the project users without getting more than self-gratification, then maybe you aren't the right person to continue doing it.

Being a software developer is a very ungrateful profession, while I know that there are whole lot of people appreciating my work and efforts, by reading the webforums that link to my website, by watching the downloads of qd & friends by happy finstall users everyday, etc., there are very few users that actually tell you that they do. I took the liberty to say to all the users that they're welcome, and that I am glad to have been of service in providing these useful tools and maintaining them both for my own well being and that of the whole community.

I take great pride in my work, and work on it with great passion as can be seen in my posts. Where more than I'd like my limitations in social interaction show as well. I have accepted this limitation as I have been a typical geek for over 24 years, it's in my blood and genes. The rest of the world will also just have to live with that, just like I have to live with their own respective limitations.

Because I feel that the role qd fulfills in the FAH community is of such importance, I released qd-tools as Open Source, with full documentation and source code. It provides the tools one needs to keep qd and qdinfo.dat up to date with the changes to the Folding@Home project summary, but not with changes in the queue.dat itself unfortunately. Where Dick Howell had a tool to update the project data too but didn't publish it for unknown reason, I have published this tool, so that if I get hit by a truck or suffer an unfortunate heart attack while posting a letter, that the community can continue on without my help. And that the community does not have to figure out from scratch how those tools work and keep them updated. I released this because I didn't want to be the maintainer of qd, I initially got involved with the maintaining qd effort after Plutronics of Team Mac OS X send me regularly updated qdinfo.dat files which he updated by hand. I got involved more deeply after I learned the amount of manual labor that had to be done by Plutronics to provide these updates. When he sent me the script Jackrabbit wrote to show the difference between two project summary files, I immediately saw an opportunity for myself to return my gratitude for the service provided to me in improving this script to ease the labor involved in updating the qdinfo.dat. From one thing came another the more I tried to make sense of the code and information we had access to, and I turned out to be the new maintainer and developer of qd. A role I still think I'm far from the most qualified person to fulfill, my mastery of C was not much more than needed to do the things I did in improving and maintaining qd. Although I wasn't the right man for the job, I did it anyway because I felt that someone had to do it, and by thinking that way I needed to take that responsibility even though I'd rather not, but my principals overrule my desires. I did get a wonderful challenge with the responsibility of maintaining qd, it helped me improve my C and other skills immensely, something I got great satisfaction from.

And while there were indeed others that offered to take care of qd, but none actually did, even before I got involved. So if there are others willing to maintain and develop qd, by all means feel free to do so! All I have learned of qd & friends I've shared in the qd-tools tarball, available for all to download. I've thought about throwing in the towel many times, especially when frustrating encounters with you or Bruce were involved. But I haven't done it yet, because I don't believe there is anyone who will pickup the towel if I throw it in. If I am wrong in this belief please correct me! I took the responsibility for the well being qd & friends the moment I released my code and associated my name and reputation to it, and I'll keep doing this until there is someone else willing to take care of this widely used and relied on tool. I cannot abandon its users just like that, my principals don't allow for that.

And while we do not advertise the fact that the old forum is still up and available for anyone to read, it is not sturdy enough to use on a regular basis by a large number of people. FOR THE LAST TIME, THE DATA IN THE OLD FORUM IS NOT LOST!!! If you can't figure out how to find the site (which hasn't moved) then maybe you're not as accomplished a programmer as you might think. So I'll repeat my usual challenge to anyone else that says the old data is lost. Ask me anything about the old forum. Your last post, your post count, thread name, etc.

Yes, I know that the old forum is accessible, but it's still broken and cannot be used really. Just to prove I do know, I fixed all the broken links to FCF in the FAH Wiki last night.

I had already figured out that it still existed some time ago, noticing that the A record on the domain wasn't redirected didn't require any programming skills though. Just competence with Internet and networking technology, and as I work in the UNIX department of an ISP, I happen to have those skills at a pretty high level but even normal users can figure this out, intimate UNIX and networking skills are not a requirement.

Yes, I agree that more standardized data formats between projects, clients, etc. would be very beneficial. I would love to see a better unitinfo.txt file to make 3rd party apps more useful. But to assume that is always possible across platforms like CPUs, GPUs, and PS3s is ridiculous.

The queue.dat provides just this, it works for the CPU client (old & new), the GPU clients, v4 to v6 clients the whole pile of clients.

The only thing hard to monitor is the PS3, but I'm working on an OCR/screenscraping idea to monitor the PS3 client with a webcam and feed the queue.dat info which is on the screen to a client monitor. And I also suspect that you can monitor the FAH client from an alternative OS installed on the PS3, but I have to verify that (I don't have a TV to hook a PS3 to, so I don't own one yet, I donated one to a teammate who does own a TV though ). Being able to monitor the PS3 remotely is only a matter of time IMO, but here openness would greatly help too.

If Pande Group wants to comment, then they will or they won't. Pleading won't change policy. Positive tangible proof that change is beneficial to project would be your best argument, and so far, that has been lacking, IMO.

Edit: Remember this is a family forum -UF

I hope they do comment, as I've put much time and effort into this issue (I've been writing this reply since 12:00 CET for instance, ~4,5 hours ago). I'm not looking for yet another flamewar with you or Bruce. I'm looking for a long time solution where the FAH staff helps the community to monitor their FAH clients, in such a way that it doesn't have to rely on unreliable data files or the work of reverse engineering. And from which we will have a better community as a result. Which has been my primary motivation for contributing to this community in the first place, the desire to help build a better world for all to enjoy, something which also drives my other Open Source related activities.

One thing I do not understand is the presence and large usage of the qfix. Why it is being promoted? Someone may argue that it is for restoring a finished WU, but is this restoring needed? If the FAH client is broken then the Pande Group are perfectly able to fix the FAH client itself and will need no hack from the community.

I think I can explain the popularity of qfix a bit. One part is that it is perceived a "magic" fix for some WU upload problems, which are partially founded, e.g. when the Packet Size Limit bug is the case in the older v5 clients. As qfix was designed carefully not to change the queue.dat (unless explicitly being required after fixing a know bug), it doesn't harm to run it if you're having problems uploading a WU. qfix has also proven to be able to recover wuresults in some cases, less cases than perceived by qfix users IMO though.

I would like for qfix to remain this useful and reliable, but I fear that I am unable to do so without proper understanding of the queue.dat. This is another one of the reasons for this plea. 3rd party devs are here to provide means where Pande Group cannot put in the development effort, client monitors are a perfect example in my opinion. Pande Group only has to provide the monitor authors the means to monitor the FAH client and they will do the rest. No need to build, host and maintain another bunch of webservers and service like BOINC for example, we can host our own FCI servers. Pande Group also won't have to invest the man hours for developing monitoring functionality into the FAH client, as we have FahMon, InCrease, finstall & co, the time saved can be better spent on developing improved MD software or more efficient project assignments.

Also, http://folding.stanford.edu/English/License:

You may not alter the software or associated data files.

FWIW, I think the Pande Group implicitly approves of the license violation of qfix by the lack of their public disapproval of its usage. If they do publicly disapprove of its usage or existence maybe even, I will stop distributing it immediately. I do care about software licensing, or my use of the GPL would be a facade, but I also want to do my piece and my very best in helping building a better world for myself, the world and all the future generations that will come after us. This grand purpose in life I put in practice by participating in the Folding@Home project, a cause I greatly support for ways of enabling this greater future I strife for, but with which I also have some moral dilemmas like those regarding secrecy, or the (perceived) lacking support for the community, which is IMO among the most important building blocks of our common cause to better us all. Alone we achieve very little, but with combined effort we can help each other achieve great things. Which to me is the essence of Distribution Computing.

Edit: Again, this is a family forum. Watch your language. -UF

[smoking2000]

A small update.

Vijay wrote in his email of 2008-02-27:

While I think the idea of opening up the queue.dat could be useful, I've decided to postpone any further progress here until we get v6 released and final. Once the genie is out of the bottle, we can't put it back in, and it makes sense to get to v6 final before going further.

In regard of the challenge I put out, there is no need anymore, I've figured out all the pieces of data in the queue.dat, including the bytes 612-615

This does not mean, however, that the plea can be disregarded, the need for a documented queue.dat still exists!