Kubernetes cannot pull an image from a plain HTTP or self-signed registry
Pod stays in the Waiting state because the image cannot be pulledShort answer
Kubernetes nodes pull images through their container runtime, usually containerd, not through your Docker daemon. An insecure-registries entry on your PC changes nothing there. Configure each node's containerd with a hosts.toml file for the registry, naming either an http server or skip_verify for a self-signed certificate.
Fix it fast
1Find out which cause you have
Read the pull error in the pod events, and see which runtime the nodes use.
kubectl describe pod my-app-7d9c6b5f4-abcde kubectl get nodes -o wide2Match what you see, then jump to its fix
- 1
docker pull works on your machine after adding the registry to insecure-registries; pods still failRegistry settings were changed on Docker, not on the node runtimeFix - 2
Files exist in /etc/containerd/certs.d, but pulls behave as beforecontainerd is not told to read registry host filesFix - 3
Pulls from that registry fail while other registries workThe registry speaks plain HTTP and has no hosts.toml entryFix - 4
The registry is on HTTPS, with a self-signed certificate, and the node rejects itThe registry uses HTTPS with a certificate the node does not trustFix
- 1
What does the error look like?
Pod stays in the Waiting state because the image cannot be pulleddocker pull works on a laptop, the same image fails in the cluster
Cause 1 of 4
Registry settings were changed on Docker, not on the node runtime
1Check
The kubelet hands every pull to the container runtime on that node, so settings must live on each node that can run the pod. The CONTAINER-RUNTIME column shows which runtime that is.
kubectl get nodes -o wide2Fix
Configure the runtime on every node (the next causes show containerd). A change on your own computer helps only your own docker pull.
# repeat the containerd steps below on every node3Verify
Pull the image on the node, then recreate the pod.
kubectl delete pod my-app-7d9c6b5f4-abcdeConfirmed by kubernetes.io
Cause 2 of 4
containerd is not told to read registry host files
1Check
containerd only reads the certs.d directory when config_path is set in its config file.
grep -n config_path /etc/containerd/config.toml2Fix
Set config_path to the directory, then restart containerd. For containerd 2.x (version = 3) the section is io.containerd.cri.v1.images; for 1.x (version = 2) it is io.containerd.grpc.v1.cri.
# containerd 2.x, version = 3
[plugins."io.containerd.cri.v1.images".registry]
config_path = "/etc/containerd/certs.d"
# containerd 1.x, version = 2
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"3Verify
Restart containerd and make sure it is running.
sudo systemctl restart containerd
systemctl is-active containerdConfirmed by github.com
Cause 3 of 4
The registry speaks plain HTTP and has no hosts.toml entry
1Check
The directory name under certs.d must be the registry host and port exactly as in the image name.
ls /etc/containerd/certs.d/2Fix
Create a hosts.toml for the registry with an http:// server. containerd does not need a restart for changes in this directory.
sudo mkdir -p /etc/containerd/certs.d/192.168.31.250:5000
sudo tee /etc/containerd/certs.d/192.168.31.250:5000/hosts.toml <<'CONF'
server = "http://192.168.31.250:5000"
[host."http://192.168.31.250:5000"]
capabilities = ["pull", "resolve", "push"]
CONF3Verify
Recreate the pod and watch it start.
kubectl get pod -wConfirmed by github.com
Cause 4 of 4
The registry uses HTTPS with a certificate the node does not trust
1Check
A self-signed certificate fails verification on any node that does not have its CA.
curl -vI https://myregistry.io:5000/v2/2Fix
Best: install the CA on the node. For testing only, skip verification in hosts.toml. This skips the certificate chain and host name checks.
server = "https://myregistry.io:5000"
[host."https://myregistry.io:5000"]
capabilities = ["pull", "resolve", "push"]
skip_verify = true3Verify
Recreate the pod and check that the image pulls.
kubectl describe pod my-app-7d9c6b5f4-abcdeConfirmed by github.com
How do I stop it coming back?
- Give the registry a real certificate so that nodes need no exceptions.
- Put the hosts.toml in your node image or provisioning scripts, so new nodes get it.
- Use skip_verify only on test clusters.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
Related fixes
Servers
docker pull or push fails against a private registry without valid HTTPS
Error response from daemon: v1 ping attempt failed with error
Docker treats every registry as secure except local ones.
3 causes, each with a fixServers
Apache logs SIGWINCH and exits on its own
the httpd container stops although nothing crashed
SIGWINCH is Apache's graceful-stop signal.
2 causes, each with a fixServers
sam local fails: Running AWS SAM projects locally requires Docker
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
sam local start-api and sam local invoke run your function inside a Docker container, so SAM stops if it cannot talk to Docker.
3 causes, each with a fixServers
dnf fails to download repo metadata in a CentOS 8 Docker build
The same Dockerfile built fine before 2022
CentOS Linux 8 reached end of life on 31 December 2021 and its content left the mirrors.
2 causes, each with a fixServers
pip install confluent-kafka fails because librdkafka/rdkafka.h is missing
pip stops while building confluent-kafka from source
The error means pip is compiling confluent-kafka from source, and the compiler cannot find the librdkafka C headers.
3 causes, each with a fixServers
Cypress says the npm package is installed but the binary is missing
Cypress binary is missing after npm install in CI or in a Docker build
The npm package is only a launcher.
3 causes, each with a fix