First, I tried setting up a lightweight headless Debian VM inside FreeNAS. I installed the Nvidia drivers and set it up for F@h as normal. The BSD hypervisor, bhyve, does support GPU passthrough. However, Phoronix reported that performance is around 66% compared to bare metal. This is not good for F@h and not really acceptable for the high-priority COVID Moonshot sprint projects. I therefore gave up on FreeNAS for F@h at first.
However, over the past few months, iXsystems started unifying the FreeNAS and TrueNAS platforms and announced TrueNAS SCALE as a new project. As TrueNAS SCALE is based on Debian, F@h is supported and I took a closer look at it. Here is how you can get Folding@home up and running on TrueNAS SCALE:
- Follow the guides to install TrueNAS onto a USB flashdrive.
- Use the web interface to import or create a storage pool and get TrueNAS set up.
- Use the terminal interface on the PC to drop into a shell.
- Install the Nvidia drivers with:
You can also use nvidia-detect to list detected hardware and suggest the correct driver. Its possible that ocl-icd-opencl-dev or nvidia-opencl-dev could be the packages strictly necessary for a minimal system, but the above sets up everything as needed.
Code: Select all
apt-get --no-install-recommends install nvidia-driver nvidia-opencl-icd nvidia-cuda-mps nvidia-smi nvidia-tesla-440-opencl-icd
- Use the web interface to restart the system.
- Double-check that the drivers are correctly installed. You can run nvidia-smi to see the status of connected graphics cards. I also installed "clinfo" and ran to run a test of OpenCL and CUDA.
Code: Select all
clinfo | less
- Install FAHClient on the system. As of writing, the FAHClient package for Debian can be found from the download page at specifically with this link. This URL can be fetched with wget and installed with
Code: Select all
dpkg -i fahclient*.deb
- Follow the initial configuration guide to set up the F@h client for the first time. I suggest selecting "Full" to ensure that the client is set up for all hardware.
- Modify the config.xml file at /etc/fahclient/config.xml. Add the following lines:
The network subnet can be customized for your LAN to allow a connection from FAHControl on another PC. The password should also be changed.
Code: Select all
<allow v='127.0.0.1 192.168.0.0/16'/> <password v='myfahpassword/>
- Start F@h with either or
Code: Select all
systemctl start FAHClient
I prefer the latter.Code: Select all
/etc/init.d/FAHClient start
- Examine the logs with and ensure that it is running correctly. The beginning of the log should list detected GPUs and find the OpenCL and CUDA platforms. If workunits are running, especially GPU workunits, you've got it set up correctly!
Code: Select all
tail -30 /var/lib/fahclient/log.txt
- Connect FAHControl. On another PC, open FAHControl, click "Add" in the bottom-left corner, add the IP address of the NAS and the above password, and click Save. Assuming no firewalls are in the way for TCP/36330, it should connect. This will make it easy to control this instance on the NAS.
- Back on the TrueNAS terminal, navigate to the root directory, which should be the default home location for the root user.
- Back up the fahclient user with This is critical as TrueNAS SCALE seems to regenerate /etc/passwd at boot. Without this entry, FAHClient will not be able to start.
Code: Select all
tail -1 /etc/passwd > fah_user.txt
- Use the web interface to set up a startup script. As of writing, this can be found at Data Protection > Init/Shutdown Scripts. Create a POSTINIT script that runs the following command:
Code: Select all
cat /root/fah_user.txt >> /etc/passwd; /etc/init.d/FAHClient start
- Create a SHUTDOWN script that runs the following command to shut down F@h:
Code: Select all
/etc/init.d/FAHClient stop
- Exit the shell in the TrueNAS terminal to return to the main menu.
- Use the web interface to restart the system. From the terminal, see that F@h is shutting down. At startup, you should hear the fans kick up, or check the logs to double-check that F@h is running again. If not, review /etc/passwd to verify that the entry for fahclient is present at the bottom of the file.
- Write a F@h monitoring script to run on the TrueNAS terminal screen. I created monitor.sh:
I also added some lines that used netstat, vnstat, and df to display network activity and disk consumption, but the above works for F@h.
Code: Select all
#!/usr/bin/env bash watch -t -c -p -n 3 " \ nvidia-smi | grep -v -e '^\s*$' \ && echo \ && top -b -n 1 | head -13 \ && echo \ && tail -5 /var/lib/fahclient/log.txt"
- It should be set now!