Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

docker run --gpus all fails with 'could not select device driver'

Errordocker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

Short answer

Docker only hands a GPU to a container when the host has an NVIDIA driver and the NVIDIA Container Toolkit is installed and registered as a Docker runtime. Check nvidia-smi on the host first, then the toolkit, then the runtime entry in daemon.json, and restart Docker after changing it.

  • Applies to NVIDIA Container Toolkit 1.x, Docker Engine 28, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Check the GPU on the host before blaming Docker. If nvidia-smi fails here, no container can see the GPU.

    nvidia-smi
    command -v nvidia-ctk
  2. 2Match what you see, then jump to its fix

    1. 1nvidia-smi on the host fails or is not foundNo working NVIDIA driver on the hostFix
    2. 2command -v nvidia-ctk prints nothingThe NVIDIA Container Toolkit is not installedFix
    3. 3The toolkit is installed and nvidia-smi works on the host, but docker run --gpus all still failsDocker was never told to use the NVIDIA runtime, or was not restartedFix

What does the error look like?

  • docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

Cause 1 of 3

No working NVIDIA driver on the host

1Check

The toolkit install guide lists a supported NVIDIA GPU driver as a prerequisite, installed on the host.

nvidia-smi

2Fix

Install the driver for your card from your distribution or from NVIDIA, reboot, and run nvidia-smi again until it prints the GPU table.

# install the driver for your card, then check it on the host
nvidia-smi

3Verify

nvidia-smi lists your GPU on the host.

nvidia-smi

Confirmed by docs.nvidia.com

Did this fix it?

Cause 2 of 3

The NVIDIA Container Toolkit is not installed

1Check

nvidia-ctk comes with the toolkit packages. If it is missing, Docker has nothing to attach the GPU with.

command -v nvidia-ctk

2Fix

Add NVIDIA's package repository and install nvidia-container-toolkit by following the install guide for your distribution. The guide's Ubuntu steps start with the repository key and list file.

# after adding NVIDIA's repository as the install guide describes:
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

3Verify

nvidia-ctk is on the PATH.

command -v nvidia-ctk

Confirmed by docs.nvidia.com

Did this fix it?

Cause 3 of 3

Docker was never told to use the NVIDIA runtime, or was not restarted

1Check

Look for an nvidia runtime in Docker's daemon configuration.

cat /etc/docker/daemon.json

2Fix

Let the toolkit write the runtime into daemon.json, then restart Docker. The restart is required.

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

3Verify

The sample workload from NVIDIA prints the same table as the host.

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Confirmed by docs.nvidia.com

Did this fix it?

How do I stop it coming back?

  • Reinstall or update the NVIDIA driver and the toolkit together, then restart Docker.
  • Keep nvidia-smi in a post-install checklist for every new GPU server.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.