Odd GPU behavior - terminal window + FAHClient

[SJC_Steve]

After getting the Client and Viewer loaded on Ubuntu 20.04, (Control is still broken), I noticed that CPU tasks were running but my GPU, (GTX 1060, 3GB) wasn't running tasks.

However, when I opened a Terminal Window and sent the command "FAHClient" I got what appears to be a log file with continually updating messages. I noticed that the GPU was now at full utilization. When I closed the terminal window, the GPU went back to 0% utilization. Upon opening the terminal window and typing the same command, the GPU went to 100% utilization. I tried this several times with the same results. I appears that the terminal window must be active and viewing the FAHClient log messages for the Client to activate GPU tasks.

This seems rather odd, am I doing something wrong in the setup for the this to be happening?

Thanks,
Steve

[bruce]

Two problems:

* Starting FAHClient manually omits the "hideconsole" step so you get the continually updating output and you have to leave the window open.

Check the shortcut that starts FAHClient.

* The "cd" command is required before starting FAHClient so that FAH knows where to put it's work files. In a terminal window that should be the first step.

Presumably when you manually started it, you created a new set of working files somewhere else and it ran successfully.

STRICTLY A GUESS, but I suspect that you manually edited the client.cfg file and created a syntax error. FAHClient will absolutely not run if that kind of error is present and the hideconsole functionality won't let you see that error message. All of this is a workaround for WIndows not having the concept of a GPU running in a DAEMON.

[Gnomuz]

Difficult to help you on why GPU tasks don't start when FAHClient runs as daemon, you should post your logs.

The command to run FAHClient as a daemon is '/etc/init.d/FAHClient start' and '/etc/init.d/FAHClient stop' to stop it.
The current log is '/var/lib/fahclient/log.txt', the previous ones are located in '/var/lib/fahclient/logs/' and named 'log-YYYYMMDD-HHMM.txt'

That should help you provide the required information to sort it out

[SJC_Steve]

Here is the current log after I rebooted the system. I haven't manually modified any FaH files and didn't issue any commands to the the FAHClient, therefore the GPU is not currently operating, 0% utilization, CPU is at 100%.

Also here's the my Donor stats which shows no completed tasks since 2/3/2021 even though it's been running 24/7 utilizing both the CPU and GPU.
Donor: SJC_Steve
Date of last Work Unit 2021-02-03 17:15:59
Total score 15,384
Total WUs 7
Overall rank (if points are combined) 2,791,739 of 2,791,739
Active clients (within 50 days) 1
Active clients (within 7 days) 1

Code: Select all

steve@r7-3700x:/var/lib/fahclient$ cat log.txt
*********************** Log Started 2021-02-08T18:54:48Z ***********************
18:54:48:******************************* libFAH ********************************
18:54:48:         Date: Oct 20 2020
18:54:48:         Time: 20:36:39
18:54:48:     Revision: 5ca109d295a6245e2a2f590b3d0085ad5e567aeb
18:54:48:       Branch: master
18:54:48:     Compiler: GNU 8.3.0
18:54:48:      Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
18:54:48:               -fdata-sections -O3 -funroll-loops -fno-pie
18:54:48:     Platform: linux2 5.8.0-1-amd64
18:54:48:         Bits: 64
18:54:48:         Mode: Release
18:54:48:****************************** FAHClient ******************************
18:54:48:      Version: 7.6.21
18:54:48:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:54:48:    Copyright: 2020 foldingathome.org
18:54:48:     Homepage: https://foldingathome.org/
18:54:48:         Date: Oct 20 2020
18:54:48:         Time: 20:39:00
18:54:48:     Revision: 6efbf0e138e22d3963e6a291f78dcb9c6422a278
18:54:48:       Branch: master
18:54:48:     Compiler: GNU 8.3.0
18:54:48:      Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
18:54:48:               -fdata-sections -O3 -funroll-loops -fno-pie
18:54:48:     Platform: linux2 5.8.0-1-amd64
18:54:48:         Bits: 64
18:54:48:         Mode: Release
18:54:48:         Args: --child /etc/fahclient/config.xml --run-as fahclient
18:54:48:               --pid-file=/var/run/fahclient.pid --daemon
18:54:48:       Config: /etc/fahclient/config.xml
18:54:48:******************************** CBang ********************************
18:54:48:         Date: Oct 20 2020
18:54:48:         Time: 18:37:59
18:54:48:     Revision: 7e4ce85225d7eaeb775e87c31740181ca603de60
18:54:48:       Branch: master
18:54:48:     Compiler: GNU 8.3.0
18:54:48:      Options: -faligned-new -std=c++11 -fsigned-char -ffunction-sections
18:54:48:               -fdata-sections -O3 -funroll-loops -fno-pie -fPIC
18:54:48:     Platform: linux2 5.8.0-1-amd64
18:54:48:         Bits: 64
18:54:48:         Mode: Release
18:54:48:******************************* System ********************************
18:54:48:          CPU: AMD Ryzen 7 3700X 8-Core Processor
18:54:48:       CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
18:54:48:         CPUs: 8
18:54:48:       Memory: 15.61GiB
18:54:48:  Free Memory: 14.63GiB
18:54:48:      Threads: POSIX_THREADS
18:54:48:   OS Version: 5.4
18:54:48:  Has Battery: false
18:54:48:   On Battery: false
18:54:48:   UTC Offset: -7
18:54:48:          PID: 1113
18:54:48:          CWD: /var/lib/fahclient
18:54:48:           OS: Linux 5.4.0-65-generic x86_64
18:54:48:      OS Arch: AMD64
18:54:48:         GPUs: 1
18:54:48:        GPU 0: Bus:8 Slot:0 Func:0 NVIDIA:7 GP106 [GeForce GTX 1060 3GB] 3935
18:54:48:CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:11.2
18:54:48:       OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
18:54:48:               libOpenCL.so: cannot open shared object file: No such file or
18:54:48:               directory
18:54:48:***********************************************************************
18:54:48:<config>
18:54:48:  <!-- Client Control -->
18:54:48:  <fold-anon v='true'/>
18:54:48:
18:54:48:  <!-- Folding Slot Configuration -->
18:54:48:  <gpu v='false'/>
18:54:48:
18:54:48:  <!-- Slot Control -->
18:54:48:  <power v='full'/>
18:54:48:
18:54:48:  <!-- User Information -->
18:54:48:  <user v='SJC_Steve'/>
18:54:48:
18:54:48:  <!-- Folding Slots -->
18:54:48:  <slot id='0' type='CPU'/>
18:54:48:</config>
18:54:48:Trying to access database...
18:54:48:Successfully acquired database lock
18:54:48:FS00:Initialized folding slot 00: cpu:8
18:54:48:WU00:FS00:Starting
18:54:48:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx-256/a7-0.0.19/Core_a7.fah/FahCore_a7 -dir 00 -suffix 01 -version 706 -lifeline 1113 -checkpoint 15 -np 8
18:54:48:WU00:FS00:Started FahCore on PID 1124
18:54:48:WU00:FS00:Core PID:1128
18:54:48:WU00:FS00:FahCore 0xa7 started
18:54:48:WU00:FS00:0xa7:*********************** Log Started 2021-02-08T18:54:48Z ***********************
18:54:48:WU00:FS00:0xa7:************************** Gromacs Folding@home Core ***************************
18:54:48:WU00:FS00:0xa7:       Type: 0xa7
18:54:48:WU00:FS00:0xa7:       Core: Gromacs
18:54:48:WU00:FS00:0xa7:       Args: -dir 00 -suffix 01 -version 706 -lifeline 1124 -checkpoint 15 -np 8
18:54:48:WU00:FS00:0xa7:************************************ CBang *************************************
18:54:48:WU00:FS00:0xa7:       Date: Nov 27 2019
18:54:48:WU00:FS00:0xa7:       Time: 11:26:54
18:54:48:WU00:FS00:0xa7:   Revision: d25803215b59272441049dfa05a0a9bf7a6e3c48
18:54:48:WU00:FS00:0xa7:     Branch: master
18:54:48:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
18:54:48:WU00:FS00:0xa7:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
18:54:48:WU00:FS00:0xa7:             -fno-pie -fPIC
18:54:48:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
18:54:48:WU00:FS00:0xa7:       Bits: 64
18:54:48:WU00:FS00:0xa7:       Mode: Release
18:54:48:WU00:FS00:0xa7:************************************ System ************************************
18:54:48:WU00:FS00:0xa7:        CPU: AMD Ryzen 7 3700X 8-Core Processor
18:54:48:WU00:FS00:0xa7:     CPU ID: AuthenticAMD Family 23 Model 113 Stepping 0
18:54:48:WU00:FS00:0xa7:       CPUs: 8
18:54:48:WU00:FS00:0xa7:     Memory: 15.61GiB
18:54:48:WU00:FS00:0xa7:Free Memory: 14.54GiB
18:54:48:WU00:FS00:0xa7:    Threads: POSIX_THREADS
18:54:48:WU00:FS00:0xa7: OS Version: 5.4
18:54:48:WU00:FS00:0xa7:Has Battery: false
18:54:48:WU00:FS00:0xa7: On Battery: false
18:54:48:WU00:FS00:0xa7: UTC Offset: -7
18:54:48:WU00:FS00:0xa7:        PID: 1128
18:54:48:WU00:FS00:0xa7:        CWD: /var/lib/fahclient/work
18:54:48:WU00:FS00:0xa7:******************************** Build - libFAH ********************************
18:54:48:WU00:FS00:0xa7:    Version: 0.0.19
18:54:48:WU00:FS00:0xa7:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
18:54:48:WU00:FS00:0xa7:  Copyright: 2019 foldingathome.org
18:54:48:WU00:FS00:0xa7:   Homepage: https://foldingathome.org/
18:54:48:WU00:FS00:0xa7:       Date: Nov 26 2019
18:54:48:WU00:FS00:0xa7:       Time: 00:41:42
18:54:48:WU00:FS00:0xa7:   Revision: d5b5c747532224f986b7cd02c968ed9a20c16d6e
18:54:48:WU00:FS00:0xa7:     Branch: master
18:54:48:WU00:FS00:0xa7:   Compiler: GNU 8.3.0
18:54:48:WU00:FS00:0xa7:    Options: -std=c++11 -ffunction-sections -fdata-sections -O3 -funroll-loops
18:54:48:WU00:FS00:0xa7:             -fno-pie
18:54:48:WU00:FS00:0xa7:   Platform: linux2 4.19.0-5-amd64
18:54:48:WU00:FS00:0xa7:       Bits: 64
18:54:48:WU00:FS00:0xa7:       Mode: Release
18:54:48:WU00:FS00:0xa7:************************************ Build *************************************
18:54:48:WU00:FS00:0xa7:       SIMD: avx_256
18:54:48:WU00:FS00:0xa7:********************************************************************************
18:54:48:WU00:FS00:0xa7:Project: 17422 (Run 0, Clone 424, Gen 346)
18:54:48:WU00:FS00:0xa7:Unit: 0x00000000000000000000000000000000
18:54:48:WU00:FS00:0xa7:Digital signatures verified
18:54:48:WU00:FS00:0xa7:Calling: mdrun -s frame346.tpr -o frame346.trr -x frame346.xtc -cpi state.cpt -cpt 15 -nt 8
18:54:48:WU00:FS00:0xa7:Steps: first=43250000 total=125000
18:54:49:WU00:FS00:0xa7:Completed 39712 out of 125000 steps (31%)
18:54:59:WU00:FS00:0xa7:Completed 40000 out of 125000 steps (32%)
18:55:39:WU00:FS00:0xa7:Completed 41250 out of 125000 steps (33%)
18:56:17:WU00:FS00:0xa7:Completed 42500 out of 125000 steps (34%)
18:56:55:WU00:FS00:0xa7:Completed 43750 out of 125000 steps (35%)
18:57:33:WU00:FS00:0xa7:Completed 45000 out of 125000 steps (36%)
18:58:12:WU00:FS00:0xa7:Completed 46250 out of 125000 steps (37%)
18:58:50:WU00:FS00:0xa7:Completed 47500 out of 125000 steps (38%)
18:59:28:WU00:FS00:0xa7:Completed 48750 out of 125000 steps (39%)
19:00:06:WU00:FS00:0xa7:Completed 50000 out of 125000 steps (40%)
19:00:44:WU00:FS00:0xa7:Completed 51250 out of 125000 steps (41%)
19:01:23:WU00:FS00:0xa7:Completed 52500 out of 125000 steps (42%)
19:02:01:WU00:FS00:0xa7:Completed 53750 out of 125000 steps (43%)
19:02:39:WU00:FS00:0xa7:Completed 55000 out of 125000 steps (44%)
19:03:17:WU00:FS00:0xa7:Completed 56250 out of 125000 steps (45%)
19:03:56:WU00:FS00:0xa7:Completed 57500 out of 125000 steps (46%)
19:04:34:WU00:FS00:0xa7:Completed 58750 out of 125000 steps (47%)
19:05:13:WU00:FS00:0xa7:Completed 60000 out of 125000 steps (48%)
19:05:51:WU00:FS00:0xa7:Completed 61250 out of 125000 steps (49%)
19:06:29:WU00:FS00:0xa7:Completed 62500 out of 125000 steps (50%)
19:07:07:WU00:FS00:0xa7:Completed 63750 out of 125000 steps (51%)
19:07:46:WU00:FS00:0xa7:Completed 65000 out of 125000 steps (52%)
19:08:24:WU00:FS00:0xa7:Completed 66250 out of 125000 steps (53%)
19:09:02:WU00:FS00:0xa7:Completed 67500 out of 125000 steps (54%)
19:09:40:WU00:FS00:0xa7:Completed 68750 out of 125000 steps (55%)
19:10:19:WU00:FS00:0xa7:Completed 70000 out of 125000 steps (56%)
steve@r7-3700x:/var/lib/fahclient$

Mod Edit: Added Code Tags - PantherX

[Neil-B]

That log appears to be saying that there is no gpu slot? .. Which would match with the single active user .. but not with the gpu working 100% .. do you have more than one client running - would running fahclient from window be firing up a second client somehow that is running off a default config that fires up a gpu slot

[Neil-B]

In Windows i would check the running processes in task manager .. if my guess is right I would see two fahclients and probably one fahcoe_22 and maybe two fahcore_a7 or a8 when that terminal window is open and reporting cpu and gpu usage .. but I have no clue how to do it in linux

[SJC_Steve]

The GPU is no longer working, it's currently at 0%. It only operates when I issue the command "FAHClient" which I didn't do after I rebooted. The log is from operation with no intervention from me.

However, here's a portion of the log above, which show the Client sees the GPU, it's just not using it.

18:54:48: GPUs: 1
18:54:48: GPU 0: Bus:8 Slot:0 Func:0 NVIDIA:7 GP106 [GeForce GTX 1060 3GB] 3935
18:54:48:CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:11.2
18:54:48: OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
18:54:48: libOpenCL.so: cannot open shared object file: No such file or
18:54:48: directory

[bruce]

18:54:48: GPU 0: Bus:8 Slot:0 Func:0 NVIDIA:7 GP106 [GeForce GTX 1060 3GB] 3935
18:54:48:CUDA Device 0: Platform:0 Device:0 Bus:8 Slot:0 Compute:6.1 Driver:11.2
18:54:48: OpenCL: Not detected: Failed to open dynamic library 'libOpenCL.so':
18:54:48: libOpenCL.so: cannot open shared object file: No such file or...

Installing the drivers from NVidia.com would fix this problem on Windows. In Linux, you can probably install OpenCL directly (See other Linux discussions).

Running two copies of FAHClient is not supported and if you do manage to get it working, you're likely to cause other problems.

[SJC_Steve]

Installing the drivers from NVidia.com would fix this problem on Windows. In Linux, you can probably install OpenCL directly (See other Linux discussions).

Running two copies of FAHClient is not supported and if you do manage to get it working, you're likely to cause other problems.

Are you saying that simply issuing the command "FAHClient" starts an additional client with different attributes from the already operating Client?

Thanks,
Steve

[SJC_Steve]

I looked at the tasks running with FAH in the name;

Before issuing the FAHClient command, 4 processes are running with 0% GPU utilization
After issuing the FAHClient command, 10 processes are running with 100% GPU untilization as long as the FAHClient window is open. It goes back to 4 when I close the window.

So it appears that issuing that command opens another Client with other attributes?

[Neil-B]

That is what I believe is happening .. the process names may confirm this .. I know in Windows how you start the fahclient is important or it can cause issues

[SJC_Steve]

So, I've spent a week trying to get the latest release of FaH running my GPU on Ubuntu OS with no luck. I appreciate all the help everyone has offered but at this point, it appears that FaH is not a project for average donors running on Ubuntu. I'll go back to Boinc and try again in the future.
Thanks and good crunching.
Steve

[bruce]

FAHControl or WebControl are designed for the average user to run in the foreground.

FAHClient is not designed to run in a terminal window (full stop). It is designed to run as a background daemon. The installation script creates that daemon and you are choosing to override that process. You do so at your own risk. Having done that, you also suggest that you are neither an average Linux donor nor a Linux guru.

[SJC_Steve]

FAHControl or WebControl are designed for the average user to run in the foreground.

FAHClient is not designed to run in a terminal window (full stop). It is designed to run as a background daemon. The installation script creates that daemon and you are choosing to override that process. You do so at your own risk. Having done that, you also suggest that you are neither an average Linux donor nor a Linux guru.

I only ran FAHClient in the terminal window in ignorance by fumbling around as it was the only way I ever get the GPU running. When I ran the install as directed in the Linux install pages, all three of the install scripts failed with various dependency problems. I am after all, just a lowly average Linux donor fumbling around contrary to your suggestion.

If you don't believe it, install a fresh Ubuntu 20.04 OS and then run those three deb packages and see how many steps it takes you to get the GPU running. Then do a write up to let us newbies in on how to get this operating. Speculating on my abilities is a waste of time.

Thanks,
Steve

[bruce]

Sorry. I didn't mean that in a critical way even though it sounded that way. We do (blindly) have to guess the best way to provide support to a wide variety of users.

I'd like to see a family of installation scripts named for the version of Ubuntu that it was tested on with the necessary changes so they all succeed on that version.

I suspect that FAHClient runs properly in any version of Ubuntu once it is installed in an optimum environment and that FAHControl works if Python's version is properly handled.