FCI: Folding@Home Client Info (Web based client monitor)

[Shadowtester]

Just did the upgrade for both my server and clients everything went great so far no problems that I can see. Thank you again for your hard work and generosity sharing this with the folding community.

[PeddlerOfFlesh]

I tried upgrading to 1.4.3. I'm still not sure how I'm supposed to get data into the known-projects.xml when everything is complaining about known-projects.xml being empty.

I've re-uploaded my clients multiple times and ran everything on the server a bunch of times. I tried wiping it out and starting over again, but that didn't help either. As far as I can tell, everything is going fine until it gets to something that wants the known-projects.xml

[smoking2000]

Can you describe what you did exactly to reinstall FCI? The full commands used to install, etc.

It would also be helpful if you could post the FCI client output (using --verbose) for your clients after a clean install of the FCI server.

[rhavern]

I'm seeing similar issues with a fresh install of FCI 1.4.3. As advised, I went through and manually ran each of the server scripts after a client had uploaded successfully. I don't get any username or team info available and the data seems to be unprocessed.

Code: Select all

Client	Directory	Size	Status	Last Update
Server1-GPU	client-data/Server1-GPU/	118.71	KB	* New/Unprocessed Client	n/a
Server2-GPU1	client-data/Server2-GPU1/	59.26	KB	* New/Unprocessed Client	n/a
Server2-GPU2	client-data/Server2-GPU2/	138.24	KB	* New/Unprocessed Client	n/a
Total :	316.21	KB	 

Stanford Files
Name	File	Size	Modified	Last Update
Project Summary	psummary.html	92.39	KB	Thu Apr 9 15:03:21 2009	2009-04-09 15:03:20
Team Summary	daily_team_summary.txt	1.72	MB	Thu Apr 9 15:01:14 2009	2009-04-09 15:01:10
User Summary	daily_user_summary.txt	25.92	MB	Thu Apr 9 15:03:20 2009	2009-04-09 15:01:14
Total :	27.73	MB	 

Team Stats Files
No team-stats found!

I ran all the server scripts again manually:

Code: Select all

rick@linux-1:~$ /usr/local/bin/fci-update-stanford-files.pl --dir /var/www/fci/ --verbose
Changing working directory to: /var/www/fci/

Warning: <team> element has no 'number' key attribute at /usr/local/bin/fci-update-stanford-files.pl line 184
Updating Team Stats ...
Not a HASH reference at /usr/local/bin/fci-update-stanford-files.pl line 277.
rick@linux-1:~$ /usr/local/bin/fci-update-qd-files.pl --dir /var/www/fci/ --verbose
Updating Data File (data_file) ...
Checking URL for modification: http://linuxminded.xs4all.nl/software/qd-tools/source/qdinfo/qdinfo.dat ... Not Modified.
Data File is up-to-date, skipping download.

Updating Windows Binary (bin_win32_i386) ...
Checking URL for modification: http://linuxminded.xs4all.nl/software/qd-tools/binaries/win32/x86/qd.exe ... Not Modified.
Windows Binary is up-to-date, skipping download.

Updating OpenBSD Binary (bin_openbsd_i386) ...
Checking URL for modification: http://linuxminded.xs4all.nl/software/qd-tools/binaries/openbsd/x86/qd ... Not Modified.
OpenBSD Binary is up-to-date, skipping download.

Updating Linux Binary (bin_linux_i386) ...
Checking URL for modification: http://linuxminded.xs4all.nl/software/qd-tools/binaries/linux/x86/qd ... Not Modified.
Linux Binary is up-to-date, skipping download.

Updating FreeBSD Binary (bin_freebsd_i386) ...
Checking URL for modification: http://linuxminded.xs4all.nl/software/qd-tools/binaries/freebsd/x86/qd ... Not Modified.
FreeBSD Binary is up-to-date, skipping download.

Updating MacOS X Binary (bin_darwin_universal) ...
Checking URL for modification: http://linuxminded.xs4all.nl/software/qd-tools/binaries/darwin/universal/qd ... Not Modified.
MacOS X Binary is up-to-date, skipping download.

rick@linux-1:~$ /usr/local/bin/fci-update-fahstats.pl --dir /var/www/fci/ --verbose
Changing working directory to: /var/www/fci/

Warning: <username> element has no 'userid' key attribute at /usr/local/bin/fci-update-fahstats.pl line 509
Not a HASH reference at /usr/local/bin/fci-update-fahstats.pl line 117.
rick@linux-1:~$ /usr/local/bin/fci-jmol-missing-projects.pl --dir /var/www/fci/ --verbose
Changing working directory to: /var/www/fci/

Warning: <project> element has no 'number' key attribute at /usr/local/bin/fci-jmol-missing-projects.pl line 488
Looking up Last-Modified HTTP header ... Found.
Last-Modified: Tue, 07 Apr 2009 19:21:39 GMT

Checking URL for modification: http://jmol.sourceforge.net/fah/missingProjects.txt ... Not Modified.
Local copy is up-to-date, skipping download.

Compiling list of sent projects ... Done.

Checking for missing projects in known projects list.
Not a HASH reference at /usr/local/bin/fci-jmol-missing-projects.pl line 258.
rick@linux-1:~$ /usr/local/bin/fci-update-xml-files.pl --dir /var/www/fci/ --verbose
Changing working directory to: /var/www/fci/

Reading XML files ...Warning: <project> element has no 'number' key attribute at /usr/local/bin/fci-update-xml-files.pl line 130
 Done.

Processing client: Server1-GPU

Checking last update ...
 Current: 2009-04-09 18:30:05
     Now: 2009-04-09 19:03:25
The last upload is less than 1 hour old.
This client has not been processed before.
Client is considered: new (First upload: 33 minutes 20 seconds ago)
Client was first seen on: 2009-04-09 18:30:05

Files for this client:
 client-data/Server1-GPU/client.xml (client_xml)
 client-data/Server1-GPU/qd-data.xml.new (qd_data_new)
 client-data/Server1-GPU/FAHlog.txt (fahlog)
 client-data/Server1-GPU/unitinfo.txt (unitinfo)
 client-data/Server1-GPU/FAHlog-Complete.txt (fahlog_complete)
 client-data/Server1-GPU/client.xml.new (client_xml_new)
 client-data/Server1-GPU/client.cfg (client_cfg)
 client-data/Server1-GPU/MyFolding.html (myfolding)

File to parse: client-data/Server1-GPU/qd-data.xml.new
Parsing qd-data.xml file: client-data/Server1-GPU/qd-data.xml.new

Not a HASH reference at /usr/local/bin/fci-update-xml-files.pl line 629.
rick@linux-1:~$ /usr/local/bin/fci-generate-queue-graphs.pl --dir /var/www/fci/ --verbose
Changing working directory to: /var/www/fci/

All clients mode.

Saving updated queue-graphs XML file ...
rick@linux-1:~$ /usr/local/bin/fci-update-jmol-projects.pl --dir /var/www/fci/ --verbose
Changing working directory to: /var/www/fci/

Looking up Last-Modified HTTP header ... Found.
Last-Modified: Tue, 07 Apr 2009 19:21:35 GMT

Checking URL for modification: http://jmol.sourceforge.net/fah/fah-projects.xml ... Not Modified.
Local copy is up-to-date, skipping download.

rick@linux-1:~$ /usr/local/bin/fci-update-eoc-stats.pl --dir /var/www/fci/ --verbose
Changing working directory to: /var/www/fci/

Warning: <username> element has no 'userid' key attribute at /usr/local/bin/fci-update-eoc-stats.pl line 535
Not a HASH reference at /usr/local/bin/fci-update-eoc-stats.pl line 129.
rick@linux-1:~$ /usr/local/bin/fci-update-project-images.pl --dir /var/www/fci/ --verbose
Changing working directory to: /var/www/fci/

Warning: <project> element has no 'number' key attribute at /usr/local/bin/fci-update-project-images.pl line 548
Not a HASH reference at /usr/local/bin/fci-update-project-images.pl line 174.

I had a load of help installing this by a SysAdmin friend at work, so I'm a bit stuck until next week.

Many thanks,
Rick

[smoking2000]

The verbose output of any FCI script is essential to debugging this issue. So please post the output of your FCI client and FCI server commands you ran before, but also append the --verbose argument to each command.

[rhavern]

I've updated the previous posting with --verbose outputs.

Thanks!

[smoking2000]

Your xml-data files have been updated incorrectly, and now contain invalid content.

To alleviate the current messed up state, first disable all FCI cronjobs, because you don't cron to run the scripts before you have made sure that they run successfully. This means first running the scripts manually, using --verbose to see what they're doing. Only after you've successfully run all the scripts should you enable the cronjobs.

After a fresh install of FCI, the scripts expect the XML files to have meaningful content (which they don't now) after having been updated by fci-update-xml-files.pl, or be in their initial state as shipped in the FCI tarball. You can recreate this state by copying the files in the www/xml-data/ directory in the FCI tarball over those in /var/www/fci/xml-data/. Make sure to set the permissions correctly after you've copied the files using: chown -R <owner>:www-data /var/www/fci/xml-data/ && chmod -R ug+rw /var/www/fci/xml-data/ (where <owner> is the username you used for install.pl too, your username that is a member of the www-data group)

[smoking2000]

I've just found out how to reproduce this problem:
* Clean install of FCI server
* Do not upload client data using fci-client.pl
* Run the FCI server scripts as per documentation

Because there is no client-data yet, fci-update-xml-files.pl cannot write useful data to the XML files, but does write more than it reads from the XML files in question. This introduces the corruption of the XML files which causes the scripts in the subsequent runs to complain about unexpected content.

You can avoid this problem by deploying FCI client & server as follows:
* Complete clean install (no left over cronjobs, scripts in /usr/local/bin/, data in /var/www/fci/, etc)
* Install FCI client & server, but do not schedule the cronjobs yet!
* Run FCI client using --verbose to verify that uploads work correctly. This will also give the FCI server meaningful data to process later. It's also recommended to tail the apache error_log to while the FCI client does its upload to check for server side errors not reported in the server response.
* Run the FCI server scripts in sequence twice, using --verbose to verify they work correctly.
* Schedule the cronjobs, if the manual runs were successful.

[rhavern]

Okay, I'll give that a try tomorrow, thanks for the info. Is there somewhere that all this forum chat is being captured in your docs? It may save someone else some pain in the future and reduce the number of questions you get asked

[smoking2000]

I will improve the installation instructions in the INSTALL.<OS> files shipped in the documentation directory of the FCI tarball. This already describes the full installation instructions, but not these special precautions yet.

[rhavern]

I've followed your directions (completely and correctly this time and everything is now working as expected. Your software rocks and you are exceptionally helpful, many thanks!

Rick

[smoking2000]

Good to hear it's working properly now.

I have a couple of questions for you now that you're using FCI and are only monitoring GPU clients currently:

* Do the Expected dates match the date and time on which the FAH clients actually completed the WUs in question? The GPU client behaves differently from the other FAH clients, which causes qd to calculate an incorrect Expected date and PPD. I've only seen this with the GPU client, but not all. I need to know if I need to fix this in qd or not.

* I see that you're clients are missing the CPU & RAM information in the Hardware section, this is retrieved from using the systeminfo command on Windows. The output of this command is language specific, are you running an English or other language version of Windows? And can you post the output of the systeminfo command? (Open a cmd.exe promt via Start -> Run, execute systeminfo and copy the output (see the qd howto for how to copy & paste output from a command prompt).

Also, I see that you have not enabled the settings, client-data & client-uploads ACLs yet. This means that anyone can upload data to your FCI server, and they can also download your uploaded client files from your FCI server. This includes your client.cfg with the passkey, which you don't want to share with the world, but only with authorized users (you). And because the settings interface is not protected yet, anyone can lock you out of you server by configuring the ACLs to deny anyone access except the account they created.

Not having the ACLs enabled allowed me to troubleshoot your issue much better yesterday, because I could check your client files for the expected data, but you don't want to allow just anyone to do this!

[rhavern]

I've compared the dates of fahspy and fci for the GPU clients and the expected dates are within half an hour of each other. Of course fahspy updates far more frequently at the moment, so that more than likely explains the difference.

Here's the output of one of the devices, all of them are running English:

Code: Select all

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Rick>systeminfo

Host Name:                 ASUS
OS Name:                   Microsoft Windows XP Professional
OS Version:                5.1.2600 Service Pack 3 Build 2600
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Workstation
OS Build Type:             Multiprocessor Free
Registered Owner:          Rick
Registered Organization:
Product ID:                55274-640-5418136-23729
Original Install Date:     05/01/2006, 14:38:25
System Up Time:            0 Days, 5 Hours, 49 Minutes, 44 Seconds
System Manufacturer:       System manufacturer
System Model:              System Product Name
System type:               X86-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: x86 Family 15 Model 35 Stepping 2 AuthenticAMD
~2412 Mhz
BIOS Version:              Nvidia - 42302e31
Windows Directory:         C:\WINDOWS
System Directory:          C:\WINDOWS\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-us;English (United States)
Input Locale:              en-gb;English (United Kingdom)
Time Zone:                 (GMT) Greenwich Mean Time : Dublin, Edinburgh, Lisbon
, London
Total Physical Memory:     2,047 MB
Available Physical Memory: 1,027 MB
Virtual Memory: Max Size:  2,048 MB
Virtual Memory: Available: 2,008 MB
Virtual Memory: In Use:    40 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    HELL
Logon Server:              \\ASUS
Hotfix(s):                 117 Hotfix(s) Installed.
                           [01]: File 1
                           [02]: File 1
                           [03]: File 1
                           [04]: File 1
                           [05]: File 1
                           [06]: File 1
                           [07]: File 1
                           [08]: File 1
                           [09]: File 1
                           [10]: File 1
                           [11]: File 1
                           [12]: File 1
                           [13]: File 1
                           [14]: File 1
                           [15]: File 1
                           [16]: File 1
                           [17]: File 1
                           [18]: File 1
                           [19]: File 1
                           [20]: File 1
                           [21]: File 1
                           [22]: File 1
                           [23]: File 1
                           [24]: File 1
                           [25]: File 1
                           [26]: File 1
                           [27]: File 1
                           [28]: File 1
                           [29]: File 1
                           [30]: File 1
                           [31]: File 1
                           [32]: File 1
                           [33]: File 1
                           [34]: File 1
                           [35]: File 1
                           [36]: File 1
                           [37]: File 1
                           [38]: File 1
                           [39]: File 1
                           [40]: File 1
                           [41]: File 1
                           [42]: File 1
                           [43]: File 1
                           [44]: File 1
                           [45]: File 1
                           [46]: File 1
                           [47]: File 1
                           [48]: File 1
                           [49]: Q147222
                           [50]: M928366 - Update
                           [51]: S867460 - Update
                           [52]: Q827429
                           [53]: Q927978
                           [54]: Q936181
                           [55]: Q954430
                           [56]: IDNMitigationAPIs - Update
                           [57]: NLSDownlevelMapping - Update
                           [58]: KB929399
                           [59]: KB952069_WM9
                           [60]: KB911565
                           [61]: KB917734_WMP10
                           [62]: KB936782_WMP11
                           [63]: KB939683
                           [64]: KB954154_WM11
                           [65]: KB959772_WM11
                           [66]: KB925398_WMP64
                           [67]: KB941569
                           [68]: KB928090-IE7 - Update
                           [69]: KB929969 - Update
                           [70]: KB931768-IE7 - Update
                           [71]: KB933566-IE7 - Update
                           [72]: KB937143-IE7 - Update
                           [73]: KB938127-IE7 - Update
                           [74]: KB939653-IE7 - Update
                           [75]: KB942615-IE7 - Update
                           [76]: KB944533-IE7 - Update
                           [77]: KB947864-IE7 - Update
                           [78]: KB950759-IE7 - Update
                           [79]: KB953838-IE7 - Update
                           [80]: KB956390-IE7 - Update
                           [81]: KB958215-IE7 - Update
                           [82]: KB960714-IE7 - Update
                           [83]: KB961260-IE7 - Update
                           [84]: MSCompPackV1 - Update
                           [85]: KB936929 - Service Pack
                           [86]: KB938464 - Update
                           [87]: KB946648 - Update
                           [88]: KB950760 - Update
                           [89]: KB950762 - Update
                           [90]: KB950974 - Update
                           [91]: KB951066 - Update
                           [92]: KB951072-v2 - Update
                           [93]: KB951376 - Update
                           [94]: KB951376-v2 - Update
                           [95]: KB951698 - Update
                           [96]: KB951748 - Update
                           [97]: KB951978 - Update
                           [98]: KB952287 - Update
                           [99]: KB952954 - Update
                           [100]: KB953839 - Update
                           [101]: KB954211 - Update
                           [102]: KB954459 - Update
                           [103]: KB954600 - Update
                           [104]: KB955069 - Update
                           [105]: KB955839 - Update
                           [106]: KB956391 - Update
                           [107]: KB956802 - Update
                           [108]: KB956803 - Update
                           [109]: KB956841 - Update
                           [110]: KB957095 - Update
                           [111]: KB957097 - Update
                           [112]: KB958644 - Update
                           [113]: KB958687 - Update
                           [114]: KB958690 - Update
                           [115]: KB960225 - Update
                           [116]: KB960715 - Update
                           [117]: KB967715 - Update
NetWork Card(s):           2 NIC(s) Installed.
                           [01]: NVIDIA nForce Networking Controller
                                 Connection Name: Local Area Connection 2
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: 192.168.1.36
                           [02]: Cisco Systems VPN Adapter
                                 Connection Name: Local Area Connection 12

C:\Documents and Settings\Rick>

One thing I noticed in the docs at http://fci.bile.nl/?target=install.plc&os=Windows was in the perl module section,

Code: Select all

Client:                 |                     
- LWP::UserAgent        | libwww-perl     
- HTTP::Request::Common | libwww-perl     
- Date::Calc            | Date-Calc         (ActivePerl)    
- Getopt::Mixed         | Getopt-Mixed    
- POSIX                 | ActivePerl
- Win32                 | Win32             (ActivePerl)
- Win32::TieRegistry    | Win32-TieRegistry (ActivePerl)

I didn't see the POSIX one listed in the ActivePerl Perl Package Manager, view all packages.

Since things appear to be more or less working, I've secured the items you mentioned. I didn't want to secure it until it was working, in case you did need to access it. Thanks again, this software rocks.

[smoking2000]

First, thanks for the systeminfo output.

Second, the reason the POSIX module is not listed in the INSTALL.Windows file is because this module is installed by default. This is not only the case for ActivePerl, but also for perl 5.10 on UNIX which includes several modules by default that used to be separate CPAN modules in perl 5.8 (Data::Dumper among others). If the package/module listed in the INSTALL file is perl or in case of Windows ActivePerl this means that this module is included by default in the perl version I tested and is included in that release of the OS in question. But because you may have an older release of the OS and therefor an older perl, you may have to install this module manually, so it is included in the INSTALL file.

Regarding the systeminfo output and the missing CPU and RAM info for your clients, this is likely caused by the line wrap in the systeminfo output which cause the regular expression not to recognise the CPU line:

Code: Select all

                           [01]: x86 Family 15 Model 35 Stepping 2 AuthenticAMD
~2412 Mhz

The amount of RAM installed is not detected because that regex doesn't match either, because your locale uses , instead of . as kilo separator:

Code: Select all

Total Physical Memory:     2,047 MB

I have WMI based code in development to discover the hardware and software info FCI client needs, but this is not finished yet. Using WMI avoids the need for systeminfo which is not installed on XP Home. On XP Home, or if systeminfo does not return any ouput for some other reason, the FCI client falls back to the Windows registry for this information, but this case is not triggered in your case since you do have systeminfo available, only is FCI client not able to parse its output correctly. This WMI code will be included in the next release, which will also allow FCI client to report which VGA cards are installed.

[rhavern]

Those improvements all sound excellent. Is there a best practice guide on configuration of the web portion of FCI, particularly the ACL piece? I activated the ACL, but it must have been too secure, as the clients were no longer updating the server. What is the function of registering the clients? I'm contemplating offering this server to fellow team members, but am unsure how to best proceed and keep access and client info secure.

Thanks for helping me get this far!