How to set up Folding@home on FreeNAS/TrueNAS - a guide

[Jesse_V]

This is a guide on how to configure a PC to manage a large storage pool using FreeNAS and running Folding@home at the same time. Up until now, to the extent of my research on this forum and across the Internet, they seemed fairly mutually exclusive. There are alternatives, such as Ansible, but FreeNAS is the most popular and widely-recommended OS for a NAS. However, it is based on FreeBSD, which is not supported by Folding@home, and it is not possible to run F@h inside a FreeBSD jail in FreeNAS. Therefore another solution is needed.

First, I tried setting up a lightweight headless Debian VM inside FreeNAS. I installed the Nvidia drivers and set it up for F@h as normal. The BSD hypervisor, bhyve, does support GPU passthrough. However, Phoronix reported that performance is around 66% compared to bare metal. This is not good for F@h and not really acceptable for the high-priority COVID Moonshot sprint projects. I therefore gave up on FreeNAS for F@h at first.

However, over the past few months, iXsystems started unifying the FreeNAS and TrueNAS platforms and announced TrueNAS SCALE as a new project. As TrueNAS SCALE is based on Debian, F@h is supported and I took a closer look at it. Here is how you can get Folding@home up and running on TrueNAS SCALE:

1. Follow the guides to install TrueNAS onto a USB flashdrive.
2. Use the web interface to import or create a storage pool and get TrueNAS set up.
3. Use the terminal interface on the PC to drop into a shell.
4. Install the Nvidia drivers with:

   Code: Select all

   apt-get --no-install-recommends install nvidia-driver nvidia-opencl-icd nvidia-cuda-mps nvidia-smi nvidia-tesla-440-opencl-icd

   You can also use nvidia-detect to list detected hardware and suggest the correct driver. Its possible that ocl-icd-opencl-dev or nvidia-opencl-dev could be the packages strictly necessary for a minimal system, but the above sets up everything as needed.
5. Use the web interface to restart the system.
6. Double-check that the drivers are correctly installed. You can run nvidia-smi to see the status of connected graphics cards. I also installed "clinfo" and ran

   Code: Select all

   clinfo | less

   to run a test of OpenCL and CUDA.
7. Install FAHClient on the system. As of writing, the FAHClient package for Debian can be found from the download page at specifically with this link. This URL can be fetched with wget and installed with

   Code: Select all

   dpkg -i fahclient*.deb

8. Follow the initial configuration guide to set up the F@h client for the first time. I suggest selecting "Full" to ensure that the client is set up for all hardware.
9. Modify the config.xml file at /etc/fahclient/config.xml. Add the following lines:

   Code: Select all

   <allow v='127.0.0.1 192.168.0.0/16'/>
   <password v='myfahpassword/>
   

   The network subnet can be customized for your LAN to allow a connection from FAHControl on another PC. The password should also be changed.
10. Start F@h with either

    Code: Select all

    systemctl start FAHClient

    or

    Code: Select all

    /etc/init.d/FAHClient start

    I prefer the latter.
11. Examine the logs with

    Code: Select all

    tail -30 /var/lib/fahclient/log.txt

    and ensure that it is running correctly. The beginning of the log should list detected GPUs and find the OpenCL and CUDA platforms. If workunits are running, especially GPU workunits, you've got it set up correctly!
12. Connect FAHControl. On another PC, open FAHControl, click "Add" in the bottom-left corner, add the IP address of the NAS and the above password, and click Save. Assuming no firewalls are in the way for TCP/36330, it should connect. This will make it easy to control this instance on the NAS.
13. Back on the TrueNAS terminal, navigate to the root directory, which should be the default home location for the root user.
14. Back up the fahclient user with

    Code: Select all

    tail -1 /etc/passwd > fah_user.txt

    This is critical as TrueNAS SCALE seems to regenerate /etc/passwd at boot. Without this entry, FAHClient will not be able to start.
15. Use the web interface to set up a startup script. As of writing, this can be found at Data Protection > Init/Shutdown Scripts. Create a POSTINIT script that runs the following command:

    Code: Select all

    cat /root/fah_user.txt >> /etc/passwd; /etc/init.d/FAHClient start

16. Create a SHUTDOWN script that runs the following command to shut down F@h:

    Code: Select all

    /etc/init.d/FAHClient stop

17. Exit the shell in the TrueNAS terminal to return to the main menu.
18. Use the web interface to restart the system. From the terminal, see that F@h is shutting down. At startup, you should hear the fans kick up, or check the logs to double-check that F@h is running again. If not, review /etc/passwd to verify that the entry for fahclient is present at the bottom of the file.
19. Write a F@h monitoring script to run on the TrueNAS terminal screen. I created monitor.sh:

    Code: Select all

    #!/usr/bin/env bash
    
    watch -t -c -p -n 3 " \
        nvidia-smi | grep -v -e '^\s*$' \
        && echo \
        && top -b -n 1 | head -13 \
        && echo \
        && tail -5 /var/lib/fahclient/log.txt"
    

    I also added some lines that used netstat, vnstat, and df to display network activity and disk consumption, but the above works for F@h.
20. It should be set now!

I am seeing about a 20 to 25% gain in improvement over Windows, which is fairly common for Linux, so I'm happy. Another option is to use a Docker instance. I tried both the official Docker container and a third-party container, and while both recognized the GPUs, they failed to accept the OpenCL drivers or open a port for remote control. I suspect (but did not prove) that this is due to the early development of TrueNAS SCALE, so its something that I might circle back to next year when TrueNAS SCALE is fully released. In any case, installing F@h natively on the system works too. Please post below if this also works for you or if you have any suggestions for this guide.

[PantherX]

Fantastic write-up, Jesse_V! Thanks for sharing the detailed steps and rational as to why it's needed

[Jesse_V]

Caution: the F@h installation, configuration of Docker images, and other changes are specific to the running version of TrueNAS SCALE and do not seem to follow from one version to the next. After an update, the default home directory (/root) seems normal, but the system is using a fresh image, so F@h does not seem to be installed, /etc/fahclient is not found, and Docker reports that no images are available. I need to further test this to see if this is only happening to the version update I just did, or if this is a general issue. I believe it's possible to get these services to stay within /root/ or a ZFS volume, which should survive between updates.

[Trevor68]

Hi,
The Truecharts guys have added Folding to their catalog for Truenas Scale as an app. It runs very well, but is running my CPU and GPU by default.
I was hoping someone cleverer than I might be able to help me figure out how to remove the CPU slot?

[Dr_Rob]

As someone with little Linux knowledge this guide by Jesse_V is welcome. However I was lost on step 9 not knowing how to navigate to the config file, or saving the modification. I also could not find FAH within the TrueNAS Scale app catalog as Trevor68 posted. Trevor68 is also correct in wanting cpu/gpu control. Should anyone be able to clarify information or steps within the guide, it is appreciated.