Using MPS to dramatically increase PPD on big GPUs (Linux guide)

[toTOW]

Use your own hardware if you want to try exotic things.

[foldinghomealone]

Parallel computing shouldn't be considered "exotic" on a cloud computing marketplace that rents out GPU and compute resources for AI workloads, machine learning training, rendering, and other high-performance computing tasks

[toTOW]

Docker instances will always have limitations to protect the host and other users on the same host. If you need total control of the hardware, you'll need dedicated hardware.

[muziqaz]

Try one layer of exotic only.
MPS is one layer
Docker is second layer
Vast.ai instance is 3rd layer

[foldinghomealone]

Sorry guys, even considered that you're most helpful here on the board, you're answers are not really useful in this case because that they don't give any actionable information.

Useful answeres would be:
- no reply at all, if there is no meaningful information to add.
- "In my experience, docker template XYZ allowes EXCLUSIVE_PROCESS."
- "I haven't found any docker template allowing EXCLUSIVE_PROCESS."
- "Pls contact our team member USER - he's a vast.ai pro."

[muziqaz]

Sorry guys, even considered that you're most helpful here on the board, you're answers are not really useful in this case because that they don't give any actionable information.

Useful answeres would be:
- no reply at all, if there is no meaningful information to add.
- "In my experience, docker template XYZ allowes EXCLUSIVE_PROCESS."
- "I haven't found any docker template allowing EXCLUSIVE_PROCESS."
- "Pls contact our team member USER - he's a vast.ai pro."

No,
You are asking us to debug (for you, remotely) two items which are not included in the OP's post. This thread assumes you own the hardware, and it is at your house, you are not using docker or some other form of environment.

[foldinghomealone]

No,
You are asking us to debug (for you, remotely) two items which are not included in the OP's post. This thread assumes you own the hardware, and it is at your house, you are not using docker or some other form of environment.

So you're suggesting that if I open a new thread asking the same question I will get a more helpful answer from you?

[muziqaz]

No,
You are asking us to debug (for you, remotely) two items which are not included in the OP's post. This thread assumes you own the hardware, and it is at your house, you are not using docker or some other form of environment.

So you're suggesting that if I open a new thread asking the same question I will get a more helpful answer from you?

No, you are free to use this one, but don't dismiss our suggestions, as they are valid suggestions. Whenever you do exotic stuff try to limit the variables. This is the same with troubleshooting of hardware

[toTOW]

- "Pls contact our team member USER - he's a vast.ai pro."

Officiel Vast.ai Discord : https://discord.gg/vast

You're welcome.

[foldinghomealone]

Regarding vast.ai and using MPS.

According to official support:
Vast docker are non privileged containers, meaning some permissions/access are not available to root users.
VMs should be tried instead for such tasks.
It is recommended to use following template (Ubuntu VM):
https://cloud.vast.ai?ref_id=108031&tem ... 830d32ab5b

However, it seems that not many GPUs are available for use with VM and they are more expensive than the one using the normal CUDA template.

[Albuquerquefx]

There's no way a docker host would instantiate Exclusive Mode for the hardware GPU; that completely breaks the concept (and base reasoning) for using a container ecosystem to begin with. Containerized workloads were designed and built as a very lightweight means to run multiple workloads on the same hardware, without the heavier Type 1 Hypervisor performance penalties (and also missing a lot of the featureset, too.)

Exclusive mode means only one CUDA context is permitted on the GPU. Thus, if a Docker host is providing GPU resources, why would you ever limit the card to a single docker instance? What are all the other docker containers gonna do? The short answer is: nothing with the GPU.

MPS combines all child contexts into a single parent context, which is how "Exclusive Mode" works with MPS. This decision to blend all contexts together makes precarious security assumptions... If it's your own Docker host at home, it's no problem at all because assumedly these are all your own CUDA contexts and you're OK with them hanging out together. In a production cloud (read: multitenant, or potentially so) environment, you should never amalgamate all these contexts together. MPS basically eliminates any data and process isolation mechanisms between the different CUDA contexts, meaning a bad actor can very easily reach across into processes and datasets which aren't theirs. It's great for eliminating terribly expensive context switches on the GPU, but it's terrible for process security.

CN: You'll need to rent your own piece of hardware, and then stack up Docker, if you want to play this game. It's not worth it.