Docker with access to your GPU

Every model server in this series runs in a container: it installs cleanly, it uninstalls cleanly, and you can run two versions side by side. Two pieces, neither of them ours: Docker, and NVIDIA's Container Toolkit.

1. Docker

Docker's own convenience script installs the latest version. Docker documents it for test and development machines; for a production server, use the repository method of their documentation.

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

2. NVIDIA Container Toolkit (Ubuntu, Debian)

From NVIDIA's installation guide:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Other distributions: follow the same guide on NVIDIA's site.

3. Test

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

The same table as on the host, printed from inside a container: your containers can see the card.

The trap after a driver update

If you use CDI (the /etc/cdi/nvidia.yaml file), it describes the driver libraries as they were when it was generated. After a driver update it can point to files that no longer exist, and no GPU container starts. Regenerate it: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml. We now regenerate it at every boot.