Page 4 of 4

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sat Oct 18, 2025 9:41 pm
by foldinghomealone
I'm completely lost now and I need to take a break, lol.

So, it somehow worked to fold one WU:
I used the following commands, from your "3rd time is the charm" post, which yesterday didn't work:

Code: Select all

export LD_LIBRARY_PATH="$PWD"
./FahCore_22 -dir 00 -suffix 01 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0 --log-time=true
However, it's the first time and only difference to yesterday, that I didn't use those commands before:

Code: Select all

sudo rm /home/myxubuntu/Desktop/mps_test1/libstdc++.so.6
sudo ln -s /lib/x86_64-linux-gnu/libstdc++.so.6 /home/myxubuntu/Desktop/mps_test1/libstdc++.so.6
I stopped after a while, cleared the logs, copied to "mps_test1" and wanted to test two WUs using MPS.
MPS was already activated before.

So, for the WU of mps_test1, I started the command and it started to process the WU.

However, when I started the mps_test WU then it stopped again like all the times before.

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sat Oct 18, 2025 9:53 pm
by muziqaz
Just to confirm, fahcore22 commands for mps_test and mps_test1 have to be identical, you have to have 2 linux terminals open for both of them, and in both terminals you need to run LD_LIBRARY_PATH command before running fahcore22 commands.
LD_LIBRARY_PATH command is only applied to the terminal window it is run in, and once you close that terminal, things reset again.
So basically, after you create symlink to libstdc++ on both mps_test(1) folders, run LD_LIBRARY_PATH on both terminal windows, make sure each window is in its own mps_test(1) folder before doing that. Then start fahcore22 command in one terminal window, let it run for 4-5 frames, record average frame times, then stop that run, clear the log of that run, then start fahcore22 commands on both terminal windows, and wait till they finish several frames each. Records times form each run. then compare

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sat Oct 18, 2025 10:54 pm
by muziqaz
ok, stand down for a second. I'm running in to the same openmm issue myself all of a sudden. Let me ask around and figure it out

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sun Oct 19, 2025 4:08 am
by foldinghomealone
just to answer your question: Yes, I ran the ...Library... command on each terminal

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sun Oct 19, 2025 9:16 am
by muziqaz
foldinghomealone wrote: Sun Oct 19, 2025 4:08 am just to answer your question: Yes, I ran the ...Library... command on each terminal
ok, this is not a fix, but a workaround, which worked on my system.
So after creating both folders, and copying WU and fahcore to each of them, and then symlinking libstdc++ library (in both fahcore folders), you have to run
sudo su
command, enter root password, then run
export LD_LIBRARY_PATH="$PWD"
command following with
./FahCore_22 -dir 00 -suffix 01 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0 --log-time=true
Same needs to be done with second terminal window.
Once you have enough frames generated for your comparison, CTRL+C to stop folding and then type exit on each of the terminal windows to exit root mode.

I remember having similar issue long ago, but cannot remember how I solved it since. But anyways. Try the above, see if it works

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sun Oct 19, 2025 10:11 am
by foldinghomealone
It doesn't work. Same issue.
I get one WU folding. When I start the second one, the second one aborts.

--> let me test a few things. I found out what might be the problem

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sun Oct 19, 2025 10:42 am
by muziqaz
foldinghomealone wrote: Sun Oct 19, 2025 10:11 am It doesn't work. Same issue.
I get one WU folding. When I start the second one, the second one aborts.
so maybe MPS enablement is borked for standalone fahcore runs :/?

On my side both can be run on the same GPU at the same time (though mine is AMD, which supports this kind of thing out of the box):
Image

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sun Oct 19, 2025 10:57 am
by foldinghomealone
foldinghomealone wrote: Sun Oct 19, 2025 10:11 am --> let me test a few things. I found out what might be the problem
I found out the following:
Folding manually a WU works NEVER, when fah-client.service is running / started
Folding manually a WU works ALWAYS, when fah-client.service is stopped.

I was able to fold both WUs (mps_test and mps_test1) at the same time, however not with MPS mode.
(nvidia-smi shows for both fahcore_22 only "C" instead of "M+C")

Arisu wrote for enabling MPS:
"Restart FAH for the changes to take effect:
sudo systemctl restart fah-client.service"

But whenever I do that, the folding will be cancelled with the errors shown the last day(s)

--> So, I don't know how to run MPS and a disabled fah-client.service at the same time

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sun Oct 19, 2025 11:10 am
by muziqaz
Remember, my tests are unrelated to fah-client service. My tests are being run with fahcore as a standalone service.
Hopefully arisu can chime in, once they are back from GPU heist :D

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sun Oct 19, 2025 11:26 am
by foldinghomealone
Yes, unrelated, but I didn't know that the fah-client.service hinders manual folding.
That caused a lot of confusion on my side, because I never understood, why it didn't fold most of the time and sometimes it did fold.

I tried to start following commands from override.conf before manual folding:

Code: Select all

/usr/bin/nvidia-smi -c EXCLUSIVE_PROCESS
/usr/bin/nvidia-cuda-mps-control -d
But then, the folding was stopped again.

Re: Using MPS to dramatically increase PPD on big GPUs (Linux guide)

Posted: Sun Oct 19, 2025 11:30 am
by muziqaz
I cannot comment on that, since on my system fah-client.service is running, and when I took the screenshot I posted here, I was folding on a CPU through the client :D