Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

nvidia-smi works in a container at first, then fails with NVML Unknown Error

ErrorFailed to initialize NVML: Unknown Error

Short answer

The container started with GPU access, then lost it. NVIDIA documents two triggers: a container update such as changing CPU or memory limits, and systemctl daemon-reload on hosts where systemd manages container cgroups. Recreating the container restores access. Switching Docker to the cgroupfs driver stops the daemon-reload trigger.

  • Applies to NVIDIA Container Toolkit, Docker Engine 28, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Run nvidia-smi on the host and in the broken container. If the host works and the container fails, the container lost its GPU access. Also see which cgroup driver Docker uses.

    nvidia-smi
    docker exec my-gpu-container nvidia-smi
    docker info --format '{{.CgroupDriver}}'
  2. 2Match what you see, then jump to its fix

    1. 1The failure starts some time after the container was healthy; the host ran a package update or daemon-reload around thensystemctl daemon-reload ran while the container was running (systemd cgroup driver)Fix
    2. 2The failure appears right after docker update, or after a tool changed CPU or memory limits on the running containerThe container was updated after it startedFix
    3. 3The failure repeats with the systemd cgroup driver, and the host has no nvidia entries in /dev/charrunc with the systemd driver cannot find /dev/char links for the GPUFix

What does the error look like?

  • Failed to initialize NVML: Unknown Error

Cause 1 of 3

systemctl daemon-reload ran while the container was running (systemd cgroup driver)

1Check

With the systemd cgroup driver, a daemon-reload can remove the GPU device access that the NVIDIA hook injected. The docker info command above prints systemd in that case.

docker info --format '{{.CgroupDriver}}'

2Fix

Tell Docker to use the cgroupfs driver, restart Docker, then recreate the GPU containers.

# /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=cgroupfs"]
}

sudo systemctl restart docker

3Verify

Start a fresh GPU container, run a daemon-reload on the host, and check the container again.

docker run -d --name gpu-test --gpus all ubuntu sleep infinity
sudo systemctl daemon-reload
docker exec gpu-test nvidia-smi

Confirmed by docs.nvidia.com

Did this fix it?

Cause 2 of 3

The container was updated after it started

1Check

The runtime hook adds GPU access without the low-level runtime (runc) knowing about it. An explicit container update, such as new CPU or memory limits, can drop that access. The cgroupfs driver does not prevent this one.

docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}' my-gpu-container

2Fix

Delete the affected container and start it again. A restarted container gets its GPU access back. Set limits in the run command or Compose file, not with an update afterwards.

docker rm -f my-gpu-container
docker run -d --name my-gpu-container --gpus all --memory 8g my-image

3Verify

nvidia-smi lists the GPU inside the new container.

docker exec my-gpu-container nvidia-smi

Confirmed by docs.nvidia.com

Did this fix it?

Cause 3 of 3

runc with the systemd driver cannot find /dev/char links for the GPU

1Check

NVIDIA notes that some runc versions with the systemd cgroup driver show this when the /dev/char symlinks for the devices are missing. Newer runc versions and newer NVIDIA drivers, which create those symlinks, make it less likely.

ls -l /dev/char | grep -i nvidia
runc --version

2Fix

Update runc and the NVIDIA driver to current releases from your distribution or NVIDIA, then recreate the GPU containers.

sudo apt-get update
sudo apt-get upgrade

3Verify

After a reboot and a fresh container, the error does not come back after a daemon-reload.

docker exec gpu-test nvidia-smi

Confirmed by docs.nvidia.com

Did this fix it?

How do I stop it coming back?

  • Set CPU and memory limits when the container is created, and do not call docker update on GPU containers.
  • Use the cgroupfs driver on single-host Docker setups where daemon-reload runs from package updates.
  • Let the platform recreate the container when it fails (restart policy or orchestrator) instead of fixing it by hand.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Vulkan finds no NVIDIA GPU inside a Docker container

nvidia-smi lists the GPU in the container, but a Vulkan program finds no NVIDIA device

By default the NVIDIA Container Toolkit gives a container only the utility and compute capabilities.

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.