Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

Kubernetes pod shows Readiness probe failed and never becomes Ready

ErrorWarning Unhealthy kubelet Readiness probe failed:

Short answer

A failed readiness probe does not restart the pod. The kubelet marks it not ready and removes it from the Service. Read the probe event in kubectl describe pod, then check three things: the app needs longer to start, the probe path or port is wrong, or the endpoint returns a status outside 200 to 399.

  • Applies to Kubernetes 1.33, kubectl 1.33
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the pod events at the bottom of the describe output. The text after Readiness probe failed names what the kubelet saw.

    kubectl describe pod my-app-7d9c6b5f4-abcde
  2. 2Match what you see, then jump to its fix

    1. 1The failures appear only in the first minute, then stop or never stop on a slow appThe app needs longer to start than the probe allowsFix
    2. 2The failure line shows a status code or connection error immediately, on every attemptThe probe path, port or status code does not match what the app returnsFix
    3. 3The Pod is Running, the container is not restarting, and requests to the Service failThe Service has no endpoints because every pod is not readyFix

What does the error look like?

  • Warning Unhealthy kubelet Readiness probe failed:
  • READY 0/1 with STATUS Running

Cause 1 of 3

The app needs longer to start than the probe allows

1Check

Compare how long the app takes to be ready with initialDelaySeconds plus failureThreshold times periodSeconds. If the app needs more, the probe gives up too soon.

kubectl logs my-app-7d9c6b5f4-abcde
kubectl get pod my-app-7d9c6b5f4-abcde -o jsonpath='{.spec.containers[0].readinessProbe}'

2Fix

Add a startup probe on the same endpoint with a high failureThreshold. Kubernetes does not run the liveness or readiness probes until it succeeds.

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 30   # up to about 300 seconds to start
readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5

3Verify

Apply the change and watch the pod reach 1/1.

kubectl get pod -w

Confirmed by kubernetes.io

Did this fix it?

Cause 2 of 3

The probe path, port or status code does not match what the app returns

1Check

An HTTP probe succeeds for any code from 200 up to 399 and fails on all others, including a redirect to a login page that ends in 401 or a 404 for a path that does not exist. Call the endpoint from your machine through a port-forward.

kubectl port-forward pod/my-app-7d9c6b5f4-abcde 8080:8080 &
curl -i http://127.0.0.1:8080/healthz

2Fix

Point the probe at a path and port the app really serves, and make that route answer with a 2xx without needing a login.

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5

3Verify

The pod shows READY 1/1 after the next probe period.

kubectl get pod my-app-7d9c6b5f4-abcde

Confirmed by kubernetes.io

Did this fix it?

Cause 3 of 3

The Service has no endpoints because every pod is not ready

1Check

A pod failing its readiness probe is taken out of the Service's EndpointSlices. List them to see which pods are in.

kubectl get endpointslices -l kubernetes.io/service-name=my-service

2Fix

Fix the probe using the two causes above. Do not remove the readiness probe to get traffic flowing: pods that are not ready would then receive requests.

kubectl rollout restart deployment/my-app

3Verify

The pod IP appears in the endpoints again.

kubectl get endpointslices -l kubernetes.io/service-name=my-service

Confirmed by kubernetes.io

Did this fix it?

How do I stop it coming back?

  • Give every slow-starting app a startup probe instead of a long initial delay.
  • Use one cheap, login-free health route for probes.
  • Test the probe path with a port-forward before you commit the manifest.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.