Using MPS to dramatically increase PPD on big GPUs (Linux guide)

[foldinghomealone]

I'm completely lost now and I need to take a break, lol.

So, it somehow worked to fold one WU:
I used the following commands, from your "3rd time is the charm" post, which yesterday didn't work:

Code: Select all

export LD_LIBRARY_PATH="$PWD"
./FahCore_22 -dir 00 -suffix 01 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0 --log-time=true

However, it's the first time and only difference to yesterday, that I didn't use those commands before:

Code: Select all

sudo rm /home/myxubuntu/Desktop/mps_test1/libstdc++.so.6
sudo ln -s /lib/x86_64-linux-gnu/libstdc++.so.6 /home/myxubuntu/Desktop/mps_test1/libstdc++.so.6

I stopped after a while, cleared the logs, copied to "mps_test1" and wanted to test two WUs using MPS.
MPS was already activated before.

So, for the WU of mps_test1, I started the command and it started to process the WU.

However, when I started the mps_test WU then it stopped again like all the times before.

[muziqaz]

Just to confirm, fahcore22 commands for mps_test and mps_test1 have to be identical, you have to have 2 linux terminals open for both of them, and in both terminals you need to run LD_LIBRARY_PATH command before running fahcore22 commands.
LD_LIBRARY_PATH command is only applied to the terminal window it is run in, and once you close that terminal, things reset again.
So basically, after you create symlink to libstdc++ on both mps_test(1) folders, run LD_LIBRARY_PATH on both terminal windows, make sure each window is in its own mps_test(1) folder before doing that. Then start fahcore22 command in one terminal window, let it run for 4-5 frames, record average frame times, then stop that run, clear the log of that run, then start fahcore22 commands on both terminal windows, and wait till they finish several frames each. Records times form each run. then compare

[muziqaz]

ok, stand down for a second. I'm running in to the same openmm issue myself all of a sudden. Let me ask around and figure it out

[foldinghomealone]

just to answer your question: Yes, I ran the ...Library... command on each terminal

[muziqaz]

just to answer your question: Yes, I ran the ...Library... command on each terminal

ok, this is not a fix, but a workaround, which worked on my system.
So after creating both folders, and copying WU and fahcore to each of them, and then symlinking libstdc++ library (in both fahcore folders), you have to run

sudo su

command, enter root password, then run

export LD_LIBRARY_PATH="$PWD"

command following with

./FahCore_22 -dir 00 -suffix 01 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0 --log-time=true

Same needs to be done with second terminal window.
Once you have enough frames generated for your comparison, CTRL+C to stop folding and then type exit on each of the terminal windows to exit root mode.

I remember having similar issue long ago, but cannot remember how I solved it since. But anyways. Try the above, see if it works

[foldinghomealone]

It doesn't work. Same issue.
I get one WU folding. When I start the second one, the second one aborts.

--> let me test a few things. I found out what might be the problem

[muziqaz]

It doesn't work. Same issue.
I get one WU folding. When I start the second one, the second one aborts.

so maybe MPS enablement is borked for standalone fahcore runs :/?

On my side both can be run on the same GPU at the same time (though mine is AMD, which supports this kind of thing out of the box):

[foldinghomealone]

--> let me test a few things. I found out what might be the problem

I found out the following:
Folding manually a WU works NEVER, when fah-client.service is running / started
Folding manually a WU works ALWAYS, when fah-client.service is stopped.

I was able to fold both WUs (mps_test and mps_test1) at the same time, however not with MPS mode.
(nvidia-smi shows for both fahcore_22 only "C" instead of "M+C")

Arisu wrote for enabling MPS:
"Restart FAH for the changes to take effect:
sudo systemctl restart fah-client.service"

But whenever I do that, the folding will be cancelled with the errors shown the last day(s)

--> So, I don't know how to run MPS and a disabled fah-client.service at the same time

[muziqaz]

Remember, my tests are unrelated to fah-client service. My tests are being run with fahcore as a standalone service.
Hopefully arisu can chime in, once they are back from GPU heist

[foldinghomealone]

Yes, unrelated, but I didn't know that the fah-client.service hinders manual folding.
That caused a lot of confusion on my side, because I never understood, why it didn't fold most of the time and sometimes it did fold.

I tried to start following commands from override.conf before manual folding:

Code: Select all

/usr/bin/nvidia-smi -c EXCLUSIVE_PROCESS
/usr/bin/nvidia-cuda-mps-control -d

But then, the folding was stopped again.

[muziqaz]

I cannot comment on that, since on my system fah-client.service is running, and when I took the screenshot I posted here, I was folding on a CPU through the client

[DarkFoss]

Wow Thank you everyone. I didn't even think this was possible. Working well over the last 24 hours with my 7900xtx. Went from 20 wu's processed per day to 28 cpu+gpu added around 2 million ppd. The extra 8 gpu wu's more than compensate for the individual reduction of qrb.

I had enabled the gpu in the default group for the last 24, switched at midnight to having to separate gpu clients instead. I noticed that everytime the gpu in the default group uploaded it would sigterm interrupt the cpu folding and stay paused until the gpu downloaded and reached checkpoint zero before resuming folding loss of 2min per gpu wu.

No such issue when having 2 separate gpu clients.

[DarkFoss]

Currently tuning around these settings using lact (sponsored by AMD)with the compute profile active

Code: Select all

~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
amd-pstate-epp

rocm-smi
WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status
============================================ ROCm System Management Interface ============================================
====================================================== Concise Info ======================================================
Device  Node  IDs              Temp    Power   Partitions          SCLK     MCLK     Fan    Perf    PwrCap  VRAM%  GPU%  
              (DID,     GUID)  (Edge)  (Avg)   (Mem, Compute, ID)                                                        
==========================================================================================================================
0       1     0x744c,   39379  59.0°C  285.0W  N/A, N/A, 0         2340Mhz  1249Mhz  40.0%  manual  339.0W  18%    100%  
==========================================================================================================================
================================================== End of ROCm SMI Log ===================================================

[DarkFoss]

Too bad it does not show the CU%

Code: Select all

sudo amd-smi
[sudo] password for darkfoss: 
+------------------------------------------------------------------------------+
| AMD-SMI 26.1.0+5df6c765      amdgpu version: 6.14.0-35 ROCm version: 7.1.0    |
| VBIOS version: 022.001.002.010.000001                                        |
| Platform: Linux Baremetal                                                    |
|-------------------------------------+----------------------------------------|
| BDF                        GPU-Name | Mem-Uti   Temp   UEC       Power-Usage |
| GPU  HIP-ID  OAM-ID  Partition-Mode | GFX-Uti    Fan               Mem-Usage |
|=====================================+========================================|
| 0000:0b:00.0     Radeon RX 7900 XTX | 10 %     77 °C   0           278/339 W |
|   0       0     N/A             N/A | 99 %    40.0 %           4439/24560 MB |
+-------------------------------------+----------------------------------------+
+------------------------------------------------------------------------------+
| Processes:                                                                   |
|  GPU        PID  Process Name          GTT_MEM  VRAM_MEM  MEM_USAGE     CU % |
|==============================================================================|
|    0       1860  fah-client             2.0 MB   50.8 KB      0.0 B  N/A     |
|    0       2624  lact                   4.0 MB   85.9 KB      0.0 B  N/A     |
|    0       4316  FahCore_27             2.0 MB  335.6 MB   430.7 MB  N/A     |
|    0       4365  FahCore_27             2.0 MB  214.6 MB   306.7 MB  N/A     |
+------------------------------------------------------------------------------+

[foldinghomealone]

I tried using MPS on a 5090 I've rented on vast.ai.
Unfortunately, I can't get Compute Mode running...

Code: Select all

nvidia-smi -c EXCLUSIVE_PROCESS
Unable to set the compute mode for GPU 00000000:03:00.0: Insufficient Permissions
Terminating early due to previous errors.

I assume that the "Linux Desktop Container" doesn't allow to change the operation mode of the GPU.

Anyone any idea?