Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

Kubernetes cannot pull an image from a plain HTTP or self-signed registry

ErrorPod stays in the Waiting state because the image cannot be pulled

Short answer

Kubernetes nodes pull images through their container runtime, usually containerd, not through your Docker daemon. An insecure-registries entry on your PC changes nothing there. Configure each node's containerd with a hosts.toml file for the registry, naming either an http server or skip_verify for a self-signed certificate.

  • Applies to Kubernetes 1.33, containerd 2.x, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the pull error in the pod events, and see which runtime the nodes use.

    kubectl describe pod my-app-7d9c6b5f4-abcde
    kubectl get nodes -o wide
  2. 2Match what you see, then jump to its fix

    1. 1docker pull works on your machine after adding the registry to insecure-registries; pods still failRegistry settings were changed on Docker, not on the node runtimeFix
    2. 2Files exist in /etc/containerd/certs.d, but pulls behave as beforecontainerd is not told to read registry host filesFix
    3. 3Pulls from that registry fail while other registries workThe registry speaks plain HTTP and has no hosts.toml entryFix
    4. 4The registry is on HTTPS, with a self-signed certificate, and the node rejects itThe registry uses HTTPS with a certificate the node does not trustFix

What does the error look like?

  • Pod stays in the Waiting state because the image cannot be pulled
  • docker pull works on a laptop, the same image fails in the cluster

Cause 1 of 4

Registry settings were changed on Docker, not on the node runtime

1Check

The kubelet hands every pull to the container runtime on that node, so settings must live on each node that can run the pod. The CONTAINER-RUNTIME column shows which runtime that is.

kubectl get nodes -o wide

2Fix

Configure the runtime on every node (the next causes show containerd). A change on your own computer helps only your own docker pull.

# repeat the containerd steps below on every node

3Verify

Pull the image on the node, then recreate the pod.

kubectl delete pod my-app-7d9c6b5f4-abcde

Confirmed by kubernetes.io

Did this fix it?

Cause 2 of 4

containerd is not told to read registry host files

1Check

containerd only reads the certs.d directory when config_path is set in its config file.

grep -n config_path /etc/containerd/config.toml

2Fix

Set config_path to the directory, then restart containerd. For containerd 2.x (version = 3) the section is io.containerd.cri.v1.images; for 1.x (version = 2) it is io.containerd.grpc.v1.cri.

# containerd 2.x, version = 3
[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"

# containerd 1.x, version = 2
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

3Verify

Restart containerd and make sure it is running.

sudo systemctl restart containerd
systemctl is-active containerd

Confirmed by github.com

Did this fix it?

Cause 3 of 4

The registry speaks plain HTTP and has no hosts.toml entry

1Check

The directory name under certs.d must be the registry host and port exactly as in the image name.

ls /etc/containerd/certs.d/

2Fix

Create a hosts.toml for the registry with an http:// server. containerd does not need a restart for changes in this directory.

sudo mkdir -p /etc/containerd/certs.d/192.168.31.250:5000
sudo tee /etc/containerd/certs.d/192.168.31.250:5000/hosts.toml <<'CONF'
server = "http://192.168.31.250:5000"

[host."http://192.168.31.250:5000"]
  capabilities = ["pull", "resolve", "push"]
CONF

3Verify

Recreate the pod and watch it start.

kubectl get pod -w

Confirmed by github.com

Did this fix it?

Cause 4 of 4

The registry uses HTTPS with a certificate the node does not trust

1Check

A self-signed certificate fails verification on any node that does not have its CA.

curl -vI https://myregistry.io:5000/v2/

2Fix

Best: install the CA on the node. For testing only, skip verification in hosts.toml. This skips the certificate chain and host name checks.

server = "https://myregistry.io:5000"

[host."https://myregistry.io:5000"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true

3Verify

Recreate the pod and check that the image pulls.

kubectl describe pod my-app-7d9c6b5f4-abcde

Confirmed by github.com

Did this fix it?

How do I stop it coming back?

  • Give the registry a real certificate so that nodes need no exceptions.
  • Put the hosts.toml in your node image or provisioning scripts, so new nodes get it.
  • Use skip_verify only on test clusters.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.