Folding Forum

New release of FCI: v1.4.1

Minor bugfix release, I missing this one:
The lastest FCI release (and previous releases) can be downloaded from the project website:
http://fci.bile.nl/?target=downloads.plc

Or you can use the direct link:
http://fci.bile.nl/downloads/fci-1.4.1.tar.gz

I have updated to the new 1.4.1 and again I have to say the --upgrade feature is really really nice Thank You again!! Also its nice to see the GPU2 projects saved into the projects page I have not had time today to go though and setup the client.conf and modify the crontab yet I will try to get that done either later tonight or early tomorrow to ensure that also works without any errors. I will let you know as soon I have done that but I am sure it will not be a problem one thing can you configure the client.conf on one computer to handle two or more clients running on that machine in multiple directories I have a smp and gpu2 client both running on one computer and I wanted to know if I needed to setup multiple client.conf and crontab entries one for each client or can one client handle multiple fah clients running on the same machine. Currently I have separate crontab entries for each fah client with specific command line interface options for each.

Shadowtester wrote:I have updated to the new 1.4.1 and again I have to say the --upgrade feature is really really nice Thank You again!! Also its nice to see the GPU2 projects saved into the projects page I have not had time today to go though and setup the client.conf and modify the crontab yet I will try to get that done either later tonight or early tomorrow to ensure that also works without any errors. I will let you know as soon I have done that but I am sure it will not be a problem one thing can you configure the client.conf on one computer to handle two or more clients running on that machine in multiple directories I have a smp and gpu2 client both running on one computer and I wanted to know if I needed to setup multiple client.conf and crontab entries one for each client or can one client handle multiple fah clients running on the same machine. Currently I have separate crontab entries for each fah client with specific command line interface options for each.

You can have a fci-client.conf for each client, or you can have a single fci-client.conf with the shared settings and use the commandline arguments for the client specific settings.

I personally would use the latter method, and have the shared arguments in the fci-client.conf (--url, --username, --password, & --skip-unitinfo) and use the commandline arguments for specific clients (e.g. --verbose, --dir, --name, etc).

But you can also choose to create an fci-client.conf with all the settings for one client (e.g. your SMP client) and one with all the settings for the other client. E.g. create the fci-client.conf for your SMP client as ~/.fci/fci-client.conf as if it would have been the only client you'd run, then copy the file to ~/.fci/fci-client.conf.GPU and modify it with the settings specific to your GPU client. You can then schedule the two jobs in cron as: Personally I think this is ugly, and prefer: But in the spirit of Perl, There Is More Than One Way To Do It, and you can choose which way you like best.


PS, the FCI client always had the ability to upload to a non-standard port. I tested a setup similar to your yesterday and it worked fine without having to modify the code at all.

The config file setup worked great I ended up using the second method I think I prefer that as well so I am using a client.conf and then specific command line switches when the client is run from the crontab. Thank you again for all the help and a great application for monitoring my folding at home clients both locally and remotely over the internet.

I thought of another possible future feature which would be helpful if possible. It would be nice if you could have the server check the client logs and maybe compare the time stamps intervals and report if the time stamp interval exceeds maybe twice the time per frame to help alert the person using fci to monitor the clients that that client has a problem and may be hung. I know fahmon does something like that and will report the client is hung it would be great if fci could do the same so remotely monitored client that became hung could be attended too.

Another possible future feature which would be nice if possible would be a benchmark section to be saved with the project data which would list the average time per frame for each client which has worked on that project in the past.

Thanks again for all your help and for a great piece of software which has become my primary method of monitoring my fah clients even when I am at home on my lan.

What you mention regarding using the TPF to detect hung clients and/or other notable events, is one of the reasons I'm working on integrating RRDTool in FCI. Currently only for the EOC data, but I also intent to use it for TPF, PPD and other useful data. But as I mentioned before, this all depends on an improved FAHlog parsers which takes the date into account too (because RRDTool uses epoch timestamps, so only the time reported in FAHlog is not sufficient). I'll keep these feature request in mind for when I've improved the FAHlog parser.

As I said be for just thinking of possible improvement I think would be helpful for things I run across while using the software on a daily basis. The hung client detection is one of the only reasons I use fahmon at all anymore. I had a gpu client hang on me yesterday for about 6 hours I had checked it a couple of times with fci but did not notice that it was not making any progress only after the I got home and the client should have finished I checked it again and seen that it was still not finished. I then checked in fahmon and it showed the client as hung so I had to stop the client and then restart so it would continue to fold. I realize this is just a hobby for you so I am not expecting any immediate action or updates. All that could be asked or expected from you is for you to acknowledge the request and keep it in mind for possible future development as your time allows. Your development and support of this software has been exceptional no one could expect or ask for any more!!!

Thank you again.

New release of FCI: v1.4.2

Minor bugfix release, touching both the FCI client and server: The lastest FCI release (and previous releases) can be downloaded from the project website:
http://fci.bile.nl/?target=downloads.plc

Or you can use the direct link:
http://fci.bile.nl/downloads/fci-1.4.2.tar.gz

The upgrade went fine again no problems Thank You again for the update.

When trying to update the xml files I get this:
Any ideas? And if that's not what's stopping it from updating, I'm not sure what is.

The error is generated when fci-update-xml-files.pl parses the known-projects.xml.

The known-projects.xml has likely become corrupted for some reason, which will need manual fixing.

Open the known-projects.xml file (/var/www/fci/xml-data/known-projects.xml) with your favorite text editor and remove XML for the project in question.

It may also be that there is no data in the known-projects.xml yet, because no FCI clients have done an upload yet.

Are you a bluewin.ch or comcast.net user? If so I can have a look at your FCI server if it's avaible to the Internet. There are two FCI servers I know about (because they download the qd files from my server) that are not publicly accessible on port 80. Making the server available (by configuring NAT and/or firewalls correctly) greatly enhances my troubleshooting abilities.

You can also post the content of your known-projects.xml if you are not able to provide remote access.

It appears that known-projects.xml is empty. Clients have made uploads, though. It shows them as unprocessed in settings -> clients. Now I noticed that running the update also causes errors.
Edit: I went through and checked for errors with all the things in crontab and this is what I found:

PeddlerOfFlesh wrote:It appears that known-projects.xml is empty. Clients have made uploads, though. It shows them as unprocessed in settings -> clients.

Try running one or more of your FCI clients with --verbose and check if in the Composing list of files to upload: section what the status for the work/current.xyz is. This file is used by the FCI server in the final stage of the upload procedure to generate an image using rasmol for the project in question. This project is then added to the known-projects.xml file.

Also check the #####[ Server Response ]##### section, which normally includes something like this near the end: Or in the case of a project not yet listed in the known-projects.xml: This output will be missing if you're uploading data of a GPU client.

Recently (since FCI v1.4, 2009-03-21) projects being worked on by clients that are missing a current.xyz in the upload are also added to the known-projects.xml, to have these projects also show up in the project list under the Projects menu option in the main web interface. This change was made because the GPU clients don't produce the current.xyz anymore, and whose projects would therefor never be preserved in the known-projects list.

I tried to reproduce your situation, but I cannot trigger the warnings you reported using the following procedure:

* Clean install of the latest FCI client & server (v1.4.2)
* Upload Linux SMP client data to FCI server by FCI client
* Executing the FCI server cronjobs listed in documentation/INSTALL.Debian in that order, but twice:
** fci-update-stanford-files.pl
** fci-update-qd-files.pl
** fci-update-fahstats.pl
** fci-jmol-missing-projects.pl
** fci-update-xml-files.pl
** fci-generate-queue-graphs.pl
** fci-update-jmol-projects.pl
** fci-update-eoc-stats.pl
** fci-update-project-images.pl

When you do a clean install of an FCI server, most of the data still needs to be collected.

fci-update-stanford-files.pl needs to be run first to download the data files from Stanford so the FCI server knows about the various projects, and the user & team stats. The FCI server only has empty files serving as placeholders for the actual data from Stanford. The first time this script is download it will not also download the teamstats files because fci-update-xml-files.pl has not run yet, which updates the team-list.xml file listing the teams known on the FCI server. This also means that the first time fci-update-xml-files.pl is run, it will not be able to fetch the stats for a username from the much smaller teamstats text file, and will have to get it by parsing the much bigger daily_user_summary.txt. Because the performance overhead of parsing the daily_user_summary.txt is quite big, this is only done if there are no teamstats files for the usernames known on the FCI server.

fci-update-qd-files.pl is not as immediately required, but I don't always include the latest qdinfo.dat and qd binaries with FCI, so you'll want to download the latest and greatest, so your FCI clients will download the most recent qdinfo.dat containing the most recent projects. You want this to avoid missing PPD info for your clients which is provided by qd.

fci-update-fahstats.pl is not useful at this point yet, but running it anyway is harmless. It will download the UserID used in the Kakaostats (formerly FAHstats) for the usernames listed in the username-list.xml (this XML file is updated by fci-update-xml-files.pl (the FCI server hearth) with the usernames listed in the FAH client configuration uploaded by the FCI clients). But because the client data has not been processed by fci-update-xml-files.pl yet, the script won't have any usernames to download the UserID for. It will in the next run.

fci-jmol-missing-projects.pl needs to be run too, to download the latest copy of the Jmol missingProjects.txt file and to check the FCI known-projects.xml file for projects listed in the Jmol missingProjects.txt. If the script finds Jmol missing projects in the known-projects.xml file, and these projects have and xyz file. The xyz file will be send via email to NicoV so he can include it in http://jmol.sourceforge.net/fah/. This script uses the Email Settings you can configured via the Jmol Missing Projects in the FCI Settings web interface. You'll want to change the From address to your email address so NicoV can credit you properly, and to contact you in case of problems.

fci-update-xml-files.pl is the hearth of the FCI server which combines the various data sources and updates the xml-data files used by the FCI Main web interface. It processes the uploaded client-data, the stanford data, and more (see the INSTALL file). As mentioned before, some scripts can only do something useful after fci-update-xml-files.pl has processed all the data and updated the XML files whose content they require.

fci-generate-queue-graphs.pl is also only useful after fci-update-xml-files.pl has run at least once, because it uses the client-list.xml composed by fci-update-xml-files.pl listing the clients who data has been processed.

fci-update-jmol-projects.pl is needed to download the Jmol fah-projects.xml which is used to update the FCI project-summary.xml with projects not listed on the Stanford psummary.html. This keeps the project data available in FCI useful for approved betateam members too, or those that run with -advmethods whose projects tend to be listed only on the alternate psummary pages every now an then.

fci-update-eoc-stats.pl is something you really want to run, especially in the next release with the inclusion of RRD graphs using the EOC 24hr Average. This script also depends on fci-update-xml-files.pl to have done its job, because like fci-update-fahstats.pl it downloads the EOC stats for all the users listed in the usernames.xml updated by fci-update-xml-files.pl.

fci-update-project-images.pl does not require fci-update-xml-files.pl to have done its job, it only generates the missing image types for all the projects listed in the known-projects.xml that have an xyz file. During the FCI client upload only the still image type is generated because it's not very demanding to generate a single image using rasmol. Generating the animated GIFs for the rotate & xyz180 image types is bit more demanding because all the frames need to be generated, and compiled into the actual animation. This is best done at a quiet time, not when all the FCI clients are uploading (most probably all at the same time or nearly).


Because of the dependencies between the various FCI server scripts, I highly recommend that you do not enable the FCI client nor the FCI server cronjobs before you've run all the commands in question (both client and server-side) at least once manually. When you've verified that everything works correctly, you can enable the cronjobs. This way you are sure that scripts do not run before their dependencies, because you hardly ever install an FCI server at xx:59:59 so that all the scripts run in the correct sequence automatically. You'll need to run the sequence manually at least once, but preferably twice, to get all the data in their most complete form. When this is done, you can let the schedulers on the FCI clients and server handle all the subsequent runs of the scripts.

PeddlerOfFlesh wrote:Now I noticed that running the update also causes errors.

Code: Select all

Constant subroutine main::NULL redefined at D:/Programs/Misc/PERL/lib/Exporter.pm line 66.
 at C:\Program Files\FCI\fci-client.pl line 98

This is a harmless warning in a module used by FCI client on Windows only, caused by the diverted use of the Win32 Perl module (to avoid requiring this module be installed on non-Windows OSes):

PeddlerOfFlesh wrote:Edit: I went through and checked for errors with all the things in crontab and this is what I found:

Code: Select all

user@computer:/var/www/fci$ /usr/local/bin/fci-update-stanford-files.pl --dir /var/www/fci/
Warning: Cannot parse project line (252):
<snipped a bunch of HTML>

These warnings are shown because the psummary.html downloaded from the Stanford servers contains invalid project lines. Projects listed on the psummary are considered invalid if they don't have data in all its colums.

This has been the case for p5732-5747 for quite some time now, the script generating the psummary pages at stanford contains a bug whereby its includes both the valid and invalid project lines: This warning is unrelated to the one reported by fci-update-xml-files due to an empty known-projects.xml file.

PeddlerOfFlesh wrote:

Code: Select all

user@computer:/var/www/fci$ /usr/local/bin/fci-jmol-missing-projects.pl --dir /var/www/fci/
Warning: <project> element has no 'number' key attribute at /usr/local/bin/fci-jmol-missing-projects.pl line 488
Not a HASH reference at /usr/local/bin/fci-jmol-missing-projects.pl line 258.

user@computer:/var/www/fci$ /usr/local/bin/fci-update-project-images.pl --dir /var/www/fci/
Warning: <project> element has no 'number' key attribute at /usr/local/bin/fci-update-project-images.pl line 548
Not a HASH reference at /usr/local/bin/fci-update-project-images.pl line 174.

These warnings are also due to the empty known-projects.xml file, like you encounted before while running fci-update-xml-files.pl.

Wow, thanks for the huge response. Looking at the output it says current .xyz can't be read. Not sure what that's about.

Edit: I guess not existing makes it hard to read. (for GPU)

with SMP I get this:

New release of FCI: v1.4.3

PeddlerOfFlesh wrote:with SMP I get this:

Code: Select all

Checking if project of xyz file is seen before.
Extracted from xyz file:
 Work Unit : p2653_Protein in POPC
   Project : 2653
<div style='color: #FF0000'>Error running upload.plc: <pre>Not a HASH reference
at (eval 209) line 430.
</pre></div><br><hr>

Ah, that error is caused by the empty known-projects.xml too.

It's triggered during the process_xyz_file() stage of the upload handling, which does what its name implies, processes the uploaded current.xyz. It checks if the project listed in the xyz file is listed in the known-projects file, and if not imports it into the project-data directory and generates a still image for it as you could see in my previous example. The error occurs because the data structure build from the known-projects XML content is not in it expected form.

I've just changed the way this check for a project listed in known-projects.xml is done, to be a bit more fault tolerant of unexpected XML content in this new release of FCI. It also contains a few other changes from the active development branch that I made separately from this issue.

The lastest FCI release (and previous releases) can be downloaded from the project website:
http://fci.bile.nl/?target=downloads.plc

Or you can use the direct link:
http://fci.bile.nl/downloads/fci-1.4.3.tar.gz