Using MPS to dramatically increase PPD on big GPUs (Linux guide)

[foldinghomealone]

I'm completely lost now and I need to take a break, lol.

So, it somehow worked to fold one WU:
I used the following commands, from your "3rd time is the charm" post, which yesterday didn't work:

Code: Select all

export LD_LIBRARY_PATH="$PWD"
./FahCore_22 -dir 00 -suffix 01 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0 --log-time=true

However, it's the first time and only difference to yesterday, that I didn't use those commands before:

Code: Select all

sudo rm /home/myxubuntu/Desktop/mps_test1/libstdc++.so.6
sudo ln -s /lib/x86_64-linux-gnu/libstdc++.so.6 /home/myxubuntu/Desktop/mps_test1/libstdc++.so.6

I stopped after a while, cleared the logs, copied to "mps_test1" and wanted to test two WUs using MPS.
MPS was already activated before.

So, for the WU of mps_test1, I started the command and it started to process the WU.

However, when I started the mps_test WU then it stopped again like all the times before.

[muziqaz]

Just to confirm, fahcore22 commands for mps_test and mps_test1 have to be identical, you have to have 2 linux terminals open for both of them, and in both terminals you need to run LD_LIBRARY_PATH command before running fahcore22 commands.
LD_LIBRARY_PATH command is only applied to the terminal window it is run in, and once you close that terminal, things reset again.
So basically, after you create symlink to libstdc++ on both mps_test(1) folders, run LD_LIBRARY_PATH on both terminal windows, make sure each window is in its own mps_test(1) folder before doing that. Then start fahcore22 command in one terminal window, let it run for 4-5 frames, record average frame times, then stop that run, clear the log of that run, then start fahcore22 commands on both terminal windows, and wait till they finish several frames each. Records times form each run. then compare

[muziqaz]

ok, stand down for a second. I'm running in to the same openmm issue myself all of a sudden. Let me ask around and figure it out

[foldinghomealone]

just to answer your question: Yes, I ran the ...Library... command on each terminal

[muziqaz]

just to answer your question: Yes, I ran the ...Library... command on each terminal

ok, this is not a fix, but a workaround, which worked on my system.
So after creating both folders, and copying WU and fahcore to each of them, and then symlinking libstdc++ library (in both fahcore folders), you have to run

sudo su

command, enter root password, then run

export LD_LIBRARY_PATH="$PWD"

command following with

./FahCore_22 -dir 00 -suffix 01 -gpu-vendor nvidia -opencl-platform 0 -opencl-device 0 -cuda-device 0 -gpu 0 --log-time=true

Same needs to be done with second terminal window.
Once you have enough frames generated for your comparison, CTRL+C to stop folding and then type exit on each of the terminal windows to exit root mode.

I remember having similar issue long ago, but cannot remember how I solved it since. But anyways. Try the above, see if it works

[foldinghomealone]

It doesn't work. Same issue.
I get one WU folding. When I start the second one, the second one aborts.

--> let me test a few things. I found out what might be the problem

[muziqaz]

It doesn't work. Same issue.
I get one WU folding. When I start the second one, the second one aborts.

so maybe MPS enablement is borked for standalone fahcore runs :/?

On my side both can be run on the same GPU at the same time (though mine is AMD, which supports this kind of thing out of the box):

[foldinghomealone]

--> let me test a few things. I found out what might be the problem

I found out the following:
Folding manually a WU works NEVER, when fah-client.service is running / started
Folding manually a WU works ALWAYS, when fah-client.service is stopped.

I was able to fold both WUs (mps_test and mps_test1) at the same time, however not with MPS mode.
(nvidia-smi shows for both fahcore_22 only "C" instead of "M+C")

Arisu wrote for enabling MPS:
"Restart FAH for the changes to take effect:
sudo systemctl restart fah-client.service"

But whenever I do that, the folding will be cancelled with the errors shown the last day(s)

--> So, I don't know how to run MPS and a disabled fah-client.service at the same time

[muziqaz]

Remember, my tests are unrelated to fah-client service. My tests are being run with fahcore as a standalone service.
Hopefully arisu can chime in, once they are back from GPU heist

[foldinghomealone]

Yes, unrelated, but I didn't know that the fah-client.service hinders manual folding.
That caused a lot of confusion on my side, because I never understood, why it didn't fold most of the time and sometimes it did fold.

I tried to start following commands from override.conf before manual folding:

Code: Select all

/usr/bin/nvidia-smi -c EXCLUSIVE_PROCESS
/usr/bin/nvidia-cuda-mps-control -d

But then, the folding was stopped again.

[muziqaz]

I cannot comment on that, since on my system fah-client.service is running, and when I took the screenshot I posted here, I was folding on a CPU through the client