docker run --gpus all fails with 'could not select device driver'
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].Short answer
Docker only hands a GPU to a container when the host has an NVIDIA driver and the NVIDIA Container Toolkit is installed and registered as a Docker runtime. Check nvidia-smi on the host first, then the toolkit, then the runtime entry in daemon.json, and restart Docker after changing it.
Fix it fast
1Find out which cause you have
Check the GPU on the host before blaming Docker. If nvidia-smi fails here, no container can see the GPU.
nvidia-smi command -v nvidia-ctk2Match what you see, then jump to its fix
- 1
nvidia-smi on the host fails or is not foundNo working NVIDIA driver on the hostFix - 2
command -v nvidia-ctk prints nothingThe NVIDIA Container Toolkit is not installedFix - 3
The toolkit is installed and nvidia-smi works on the host, but docker run --gpus all still failsDocker was never told to use the NVIDIA runtime, or was not restartedFix
- 1
What does the error look like?
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
Cause 1 of 3
No working NVIDIA driver on the host
1Check
The toolkit install guide lists a supported NVIDIA GPU driver as a prerequisite, installed on the host.
nvidia-smi2Fix
Install the driver for your card from your distribution or from NVIDIA, reboot, and run nvidia-smi again until it prints the GPU table.
# install the driver for your card, then check it on the host
nvidia-smi3Verify
nvidia-smi lists your GPU on the host.
nvidia-smiConfirmed by docs.nvidia.com
Cause 2 of 3
The NVIDIA Container Toolkit is not installed
1Check
nvidia-ctk comes with the toolkit packages. If it is missing, Docker has nothing to attach the GPU with.
command -v nvidia-ctk2Fix
Add NVIDIA's package repository and install nvidia-container-toolkit by following the install guide for your distribution. The guide's Ubuntu steps start with the repository key and list file.
# after adding NVIDIA's repository as the install guide describes:
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit3Verify
nvidia-ctk is on the PATH.
command -v nvidia-ctkConfirmed by docs.nvidia.com
Cause 3 of 3
Docker was never told to use the NVIDIA runtime, or was not restarted
1Check
Look for an nvidia runtime in Docker's daemon configuration.
cat /etc/docker/daemon.json2Fix
Let the toolkit write the runtime into daemon.json, then restart Docker. The restart is required.
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker3Verify
The sample workload from NVIDIA prints the same table as the host.
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smiConfirmed by docs.nvidia.com
How do I stop it coming back?
- Reinstall or update the NVIDIA driver and the toolkit together, then restart Docker.
- Keep nvidia-smi in a post-install checklist for every new GPU server.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
Related fixes
Servers
nvidia-smi works in a container at first, then fails with NVML Unknown Error
Failed to initialize NVML: Unknown Error
The container started with GPU access, then lost it.
3 causes, each with a fixServers
Apache logs SIGWINCH and exits on its own
the httpd container stops although nothing crashed
SIGWINCH is Apache's graceful-stop signal.
2 causes, each with a fixServers
sam local fails: Running AWS SAM projects locally requires Docker
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
sam local start-api and sam local invoke run your function inside a Docker container, so SAM stops if it cannot talk to Docker.
3 causes, each with a fixServers
dnf fails to download repo metadata in a CentOS 8 Docker build
The same Dockerfile built fine before 2022
CentOS Linux 8 reached end of life on 31 December 2021 and its content left the mirrors.
2 causes, each with a fixServers
pip install confluent-kafka fails because librdkafka/rdkafka.h is missing
pip stops while building confluent-kafka from source
The error means pip is compiling confluent-kafka from source, and the compiler cannot find the librdkafka C headers.
3 causes, each with a fixServers
Cypress says the npm package is installed but the binary is missing
Cypress binary is missing after npm install in CI or in a Docker build
The npm package is only a launcher.
3 causes, each with a fix