nvidia-smi works in a container at first, then fails with NVML Unknown Error
Failed to initialize NVML: Unknown ErrorShort answer
The container started with GPU access, then lost it. NVIDIA documents two triggers: a container update such as changing CPU or memory limits, and systemctl daemon-reload on hosts where systemd manages container cgroups. Recreating the container restores access. Switching Docker to the cgroupfs driver stops the daemon-reload trigger.
Fix it fast
1Find out which cause you have
Run nvidia-smi on the host and in the broken container. If the host works and the container fails, the container lost its GPU access. Also see which cgroup driver Docker uses.
nvidia-smi docker exec my-gpu-container nvidia-smi docker info --format '{{.CgroupDriver}}'2Match what you see, then jump to its fix
- 1
The failure starts some time after the container was healthy; the host ran a package update or daemon-reload around thensystemctl daemon-reload ran while the container was running (systemd cgroup driver)Fix - 2
The failure appears right after docker update, or after a tool changed CPU or memory limits on the running containerThe container was updated after it startedFix - 3
The failure repeats with the systemd cgroup driver, and the host has no nvidia entries in /dev/charrunc with the systemd driver cannot find /dev/char links for the GPUFix
- 1
What does the error look like?
Failed to initialize NVML: Unknown Error
Cause 1 of 3
systemctl daemon-reload ran while the container was running (systemd cgroup driver)
1Check
With the systemd cgroup driver, a daemon-reload can remove the GPU device access that the NVIDIA hook injected. The docker info command above prints systemd in that case.
docker info --format '{{.CgroupDriver}}'2Fix
Tell Docker to use the cgroupfs driver, restart Docker, then recreate the GPU containers.
# /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=cgroupfs"]
}
sudo systemctl restart docker3Verify
Start a fresh GPU container, run a daemon-reload on the host, and check the container again.
docker run -d --name gpu-test --gpus all ubuntu sleep infinity
sudo systemctl daemon-reload
docker exec gpu-test nvidia-smiConfirmed by docs.nvidia.com
Cause 2 of 3
The container was updated after it started
1Check
The runtime hook adds GPU access without the low-level runtime (runc) knowing about it. An explicit container update, such as new CPU or memory limits, can drop that access. The cgroupfs driver does not prevent this one.
docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}' my-gpu-container2Fix
Delete the affected container and start it again. A restarted container gets its GPU access back. Set limits in the run command or Compose file, not with an update afterwards.
docker rm -f my-gpu-container
docker run -d --name my-gpu-container --gpus all --memory 8g my-image3Verify
nvidia-smi lists the GPU inside the new container.
docker exec my-gpu-container nvidia-smiConfirmed by docs.nvidia.com
Cause 3 of 3
runc with the systemd driver cannot find /dev/char links for the GPU
1Check
NVIDIA notes that some runc versions with the systemd cgroup driver show this when the /dev/char symlinks for the devices are missing. Newer runc versions and newer NVIDIA drivers, which create those symlinks, make it less likely.
ls -l /dev/char | grep -i nvidia
runc --version2Fix
Update runc and the NVIDIA driver to current releases from your distribution or NVIDIA, then recreate the GPU containers.
sudo apt-get update
sudo apt-get upgrade3Verify
After a reboot and a fresh container, the error does not come back after a daemon-reload.
docker exec gpu-test nvidia-smiConfirmed by docs.nvidia.com
How do I stop it coming back?
- Set CPU and memory limits when the container is created, and do not call docker update on GPU containers.
- Use the cgroupfs driver on single-host Docker setups where daemon-reload runs from package updates.
- Let the platform recreate the container when it fails (restart policy or orchestrator) instead of fixing it by hand.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
Related fixes
Servers
Vulkan finds no NVIDIA GPU inside a Docker container
nvidia-smi lists the GPU in the container, but a Vulkan program finds no NVIDIA device
By default the NVIDIA Container Toolkit gives a container only the utility and compute capabilities.
4 causes, each with a fixServers
Apache logs SIGWINCH and exits on its own
the httpd container stops although nothing crashed
SIGWINCH is Apache's graceful-stop signal.
2 causes, each with a fixServers
sam local fails: Running AWS SAM projects locally requires Docker
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
sam local start-api and sam local invoke run your function inside a Docker container, so SAM stops if it cannot talk to Docker.
3 causes, each with a fixServers
dnf fails to download repo metadata in a CentOS 8 Docker build
The same Dockerfile built fine before 2022
CentOS Linux 8 reached end of life on 31 December 2021 and its content left the mirrors.
2 causes, each with a fixServers
pip install confluent-kafka fails because librdkafka/rdkafka.h is missing
pip stops while building confluent-kafka from source
The error means pip is compiling confluent-kafka from source, and the compiler cannot find the librdkafka C headers.
3 causes, each with a fixServers
Cypress says the npm package is installed but the binary is missing
Cypress binary is missing after npm install in CI or in a Docker build
The npm package is only a launcher.
3 causes, each with a fix