HOWTO: How I got my AMD card folding on Linux

[Frogging101]

My setup is as follows:

Code: Select all

Radeon R9 290 (Hawaii) with free software drivers (amdgpu/mesa)
Intel Core i7-4790
Linux Mint 18.1 64-bit
Linux kernel v5.5.1

I created this guide to share my findings and experience in setting up Folding@Home on this computer. I hope that it can provide some insight and help to others.

Installing the AMDGPU-PRO OpenCL implementation

I use the free software GPU driver stack. Unfortunately, I could not get Folding@Home to work with the free software OpenCL implementations from ROCm or Mesa. So I will not bother explaining how to set those up. If you have them, you may want to remove them in case they interfere.

What did work was something suggested by dschuermann on IRC (#radeon on freenode; great people there). This allows you to use the OpenCL implementation from the proprietary AMDGPU-PRO driver, without installing the whole driver.

1. Get the AMDGPU-PRO driver package:
   1. Go to https://www.amd.com/en/support and select your graphics card.
   2. Scroll down to Ubuntu x86 64-bit and download the file. (currently: amdgpu-pro-19.50-967956-ubuntu-18.04.tar.xz)
   3. Extract it to an empty directory somewhere. Do not run the install script or anything else.
2. Extract the OpenCL driver from the appropriate .deb file.
   For Vega 10 and later products, this is the .deb file matching the pattern opencl-amdgpu-pro-icd_*_amd64.deb.
   For products older than Vega 10, this is the .deb file matching the pattern opencl-orca-amdgpu-pro-icd_*_amd64.deb.
   1. In a terminal in the directory with the .deb files, run the following:

      Code: Select all

      # Extract the contents of the .deb package to the current directory
      dpkg -x DEB_FILE . # replace DEB_FILE with the appropriate pattern from above
      
      # Replace the path in the .icd file with the full path to where we're going to put the library
      sed -i 's|^|/opt/amdgpu-pro/lib/x86_64-linux-gnu/|' etc/OpenCL/vendors/amdocl*.icd
      
      # Copy the etc and opt directories extracted from the package to the system root.
      sudo cp -r etc opt /
      

Fixing the permissions issue (clGetDeviceIDs() returned -1)

Add the fahclient user to the "video" and "render" (if you have it) groups:

Code: Select all

sudo usermod -aG video fahclient
sudo usermod -aG render fahclient # don't worry if you don't have a "render" group

This is necessary because FAHClient must be able to access devices in /dev/dri, such as /dev/dri/card0 and /dev/dri/renderD128. Some distributions place the latter under the "render" group instead of the "video" group, so you may need both if your system is like that.

Some donors have been getting "OpenCL: Not detected: clGetDeviceIDs() returned -1" unless they run FAHClient as root, and adding the fahclient user to the video group doesn't fix it.

I observed this myself, and upon investigating, I discovered the problem. The official Folding@Home packages (at least the Debian-based ones) set up FAHClient to be launched by an init.d script, which launches FAHClient with "--run-as fahclient", so that it runs as the "fahclient" user.

The problem is that when FAHClient is launched with the --run-as option, it does not give itself the supplementary groups of the user passed into the --run-as option. Thus the FAHClient process will not have the permissions of the groups that the "fahclient" user is a member of. You can see this for yourself:

Code: Select all

$ pidof FAHClient
26801 26796
$ sudo cat /proc/26796/status | grep 'Groups:'
Groups:

The lack of any group numbers after "Groups:" shows that the FAHClient process does not have any supplementary groups, even though it is running as the "fahclient" user. This is due to a bug in FAHClient.

For the developers: If I had to guess (I have not verified this), this is probably happening because FAHClient uses setuid() to drop privileges with the "--run-as" argument. This is not sufficient. You must call initgroups() before setuid() to give the process the supplementary groups of the target user.

TL;DR For the users: To work around this, I created a systemd unit to start FAHClient instead of the init.d script. This works by using the systemd unit's "User" option to change users instead of having FAHClient do it.

First, stop FAHClient ("sudo systemctl stop FAHClient").

Create /etc/systemd/system/FAHClient.service (The name matters; calling it this will automatically supersede the init.d script of the same name), with the following contents:

Code: Select all

[Unit]
Description=Folding@Home Client
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/FAHClient /etc/fahclient/config.xml --pid-file=/var/run/fahclient/fahclient.pid --daemon
PIDFile=/var/run/fahclient/fahclient.pid
User=fahclient
RuntimeDirectory=fahclient
WorkingDirectory=~

[Install]
WantedBy=multi-user.target

Then do the following:

Code: Select all

sudo systemctl daemon-reload
sudo systemctl start FAHClient.service
# And, if you want it to start automatically on boot...
sudo systemctl enable FAHClient.service

[bruce]

Thank you for the detailed HOWTO.

I suggest you make the same report at https://github.com/FoldingAtHome/fah-issues/issues. Someday the Developers will have time to examine those issues and fix most of them ... including things like your "This is a bug in FAH" mentioned above.

In the meantime, others will find it here.

[Paragon]

Nice fix, I'll try it out next time I'm tinkering with my Ubuntu box (currently folding under root). I'll try to keep an eye out for people asking about this and point them here

[Revenant]

Very useful. Thank you for the post. Trying this out now.

[dmk1000]

Thanks. This also worked for the Ryzen Vega 8 Mobile in my Ryzen 5 2400G.
I grabbed the AMD drives for Vega 56 an they've worked fine.

[Rainmaker]

I signed up just to say thank you for this! I primarily run Linux on my desktop/workstation (ThreadRipper 3960X, Vega 56) and ended up running FaH under Windows 10 for Workstations because it just kept failing to find OpenCL on Linux. I pulled out my hair for days, trying the FOSS opencl-icd as well as the (finally releaed) amdgpu-pro for 18.04.4 - headless *and* full - to no avail.

With your systemd service file I have now got everything running properly. Again, thanks!

[Juanro49]

My setup is as follows:

Code: Select all

Radeon R9 290 (Hawaii) with free software drivers (amdgpu/mesa)
Intel Core i7-4790
Linux Mint 18.1 64-bit
Linux kernel v5.5.1

I created this guide to share my findings and experience in setting up Folding@Home on this computer. I hope that it can provide some insight and help to others.

Installing the AMDGPU-PRO OpenCL implementation

I use the free software GPU driver stack. Unfortunately, I could not get Folding@Home to work with the free software OpenCL implementations from ROCm or Mesa. So I will not bother explaining how to set those up. If you have them, you may want to remove them in case they interfere.

What did work was something suggested by dschuermann on IRC (#radeon on freenode; great people there). This allows you to use the OpenCL implementation from the proprietary AMDGPU-PRO driver, without installing the whole driver.

1. Get the AMDGPU-PRO driver package:
   1. Go to https://www.amd.com/en/support and select your graphics card.
   2. Scroll down to Ubuntu x86 64-bit and download the file. (currently: amdgpu-pro-19.50-967956-ubuntu-18.04.tar.xz)
   3. Extract it to an empty directory somewhere. Do not run the install script or anything else.
2. Extract the OpenCL driver from the appropriate .deb file.
   For Vega 10 and later products, this is the .deb file matching the pattern opencl-amdgpu-pro-icd_*_amd64.deb.
   For products older than Vega 10, this is the .deb file matching the pattern opencl-orca-amdgpu-pro-icd_*_amd64.deb.
   1. In a terminal in the directory with the .deb files, run the following:

      Code: Select all

      # Extract the contents of the .deb package to the current directory
      dpkg -x DEB_FILE . # replace DEB_FILE with the appropriate pattern from above
      
      # Replace the path in the .icd file with the full path to where we're going to put the library
      sed -i 's|^|/opt/amdgpu-pro/lib/x86_64-linux-gnu/|' etc/OpenCL/vendors/amdocl*.icd
      
      # Copy the etc and opt directories extracted from the package to the system root.
      sudo cp -r etc opt /
      

Fixing the permissions issue (clGetDeviceIDs() returned -1)

Some donors have been getting "OpenCL: Not detected: clGetDeviceIDs() returned -1" unless they run FAHClient as root, and adding the fahclient user to the video group doesn't fix it.

I observed this myself, and upon investigating, I discovered the problem. The official Folding@Home packages (at least the Debian-based ones) set up FAHClient to be launched by an init.d script, which launches FAHClient with "--run-as fahclient", so that it runs as the "fahclient" user.

The problem is that when FAHClient is launched with the --run-as option, it does not give itself the supplementary groups of the user passed into the --run-as option. Thus the FAHClient process will not have the permissions of the groups that the "fahclient" user is a member of. You can see this for yourself:

Code: Select all

$ pidof FAHClient
26801 26796
$ sudo cat /proc/26796/status | grep 'Groups:'
Groups:

The lack of any group numbers after "Groups:" shows that the FAHClient process does not have any supplementary groups, even though it is running as the "fahclient" user. This is due to a bug in FAHClient.

For the developers: If I had to guess (I have not verified this), this is probably happening because FAHClient uses setuid() to drop privileges with the "--run-as" argument. This is not sufficient. You must call initgroups() before setuid() to give the process the supplementary groups of the target user.

TL;DR For the users: To work around this, I created a systemd unit to start FAHClient instead of the init.d script. This works by using the systemd unit's "User" option to change users instead of having FAHClient do it.

Create /etc/systemd/system/FAHClient.service (The name matters; calling it this will automatically supersede the init.d script of the same name), with the following contents:

Code: Select all

[Unit]
Description=Folding@Home Client
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/FAHClient /etc/fahclient/config.xml --pid-file=/var/run/fahclient/fahclient.pid --daemon
PIDFile=/var/run/fahclient/fahclient.pid
User=fahclient
RuntimeDirectory=fahclient
WorkingDirectory=~

[Install]
WantedBy=multi-user.target

Thanks, it work for me, but I also need to extract opencl-amdgpu-pro-comgr to detect GPU Navi

[Rainmaker]

Removed as the advice causes issues for some configurations.

[Juanro49]

Thanks, it work for me, but I also need to extract opencl-amdgpu-pro-comgr to detect GPU Navi

That reminds me...

OP, you don't need to dig out random .deb files to install the AMD OpenCL stuff. In your OP you say that you need/want the free AMD display driver. So do I, for vaapi to work in patched Chromium. If you install the full proprietary AMDGPU/AMDGPU-PRO driver stack Chromium can't use the driver with VAAPI and reverts to software decoding for video.

There is, however, an easier way. You can invoke the AMD installer script but tell it to only install OpenCL.

For Vega and later cards:

Code: Select all

amdgpu-pro-install --headless --opencl=pal

For cards before Vega:

Code: Select all

amd-pro-install --headless --opencl=legacy

The --headless flag tells it to not install the main graphics components. You can also combine opencl options if you needed to (opencl=legacy,pal). This leaves the open graphics stack working OK and installs the opencl stuff. Worked fine for me on Ubuntu 18.04.4. Just offering this in case it helps out you, or someone else.

This option installs some additional software instead of just OpenCL support, I tried this option weeks ago and I had problems viewtopic.php?f=81&t=32831

[Rainmaker]

This option installs some additional software instead of just OpenCL support, I tried this option weeks ago and I had problems viewtopic.php?f=81&t=32831

Ah, thanks for the heads up. It worked OK for me but in that case the manual way is better to be safe.

[Frogging101]

Yeah, I didn't use the install script because I don't expect it to be designed for this very specific use case, and I don't trust it to do it properly.

[warrelium]

Hey Frogging101, I installed the proprietary OpenCl. Thank you for the insight. I'm on Ubuntu 18.04.4

However, even after adding fahclient to video group and with your service unit I am unable to get the FAH Client to launch with the permissions needed to get past the (clGetDeviceIDs() returned -1) error.

After creating /etc/systemd/system/FAHClient.service the FAHClient process does not start at boot

If I run FAHClient with -u root then it sees the openCl device and I am able to properly fold.

Is there a different approach I need to take with the FAHClient.service to get it to work on Ubuntu 18.04.4?


EDIT:
Tonight I learned that you have to enable the service to get it to run. The service now launches the FAHClient on boot after I did the following:

Code: Select all

sudo chmod 644 /etc/systemd/system/FAHClient.service

Code: Select all

sudo systemctl enable FAHClient

[Frogging101]

Thanks warrelium. I forgot about that. I updated the OP.

[toTOW]

Stickyfied topic so that we can refer people to it.

[Ksenio]

I tried following this, but I have still not made it work. I am running RX550 and Mint 19.3 . I have installed orca, but I am still getting this error
16:05:49:ERROR:WU00:FS01:Failed to start core: OpenCL device matching slot 1 not found, try setting 'opencl-index' manually

Any ideas what I could explore more? I am really determined to make it work