Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

kubectl top node shows no metrics because metrics-server is missing or unhealthy

Errorkubectl top node or kubectl top pod prints an error instead of CPU and memory figures

Short answer

kubectl top reads the Metrics API, and only metrics-server (or a similar adapter) serves it. No metrics means the server is not installed, not registered, cannot trust the kubelet certificate, or cannot reach the kubelets. Check the APIService and the metrics-server log first.

  • Applies to Kubernetes 1.33, metrics-server 0.8
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Ask the cluster whether a Metrics API is registered and whether it reports as available. Then read the metrics-server log.

    kubectl get apiservices | grep metrics
    kubectl -n kube-system logs deploy/metrics-server --tail=30
  2. 2Match what you see, then jump to its fix

    1. 1The first command prints nothing at allmetrics-server is not installedFix
    2. 2The log shows a TLS or x509 error while scraping nodes; common on kind, minikube and kubeadm clustersmetrics-server does not trust the kubelet certificateFix
    3. 3kubectl get apiservices shows v1beta1.metrics.k8s.io as not available, or the log shows timeouts reaching a node on port 10250The kube-apiserver and metrics-server cannot reach each other, or the kubeletsFix

What does the error look like?

  • kubectl top node or kubectl top pod prints an error instead of CPU and memory figures
  • no metrics.k8s.io APIService in kubectl get apiservices, or it is not Available

Cause 1 of 3

metrics-server is not installed

1Check

Kubernetes does not ship the Metrics API. You must deploy metrics-server or another adapter that serves it, and the API aggregation layer must be on with an APIService for metrics.k8s.io registered.

kubectl -n kube-system get deploy metrics-server

2Fix

Install the latest release manifest from the project.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

3Verify

Wait for the deployment to become ready, then ask for node metrics.

kubectl -n kube-system rollout status deploy/metrics-server
kubectl top nodes

Confirmed by kubernetes-sigs.github.io

Did this fix it?

Cause 2 of 3

metrics-server does not trust the kubelet certificate

1Check

metrics-server requires the kubelet serving certificate to be signed by the cluster CA. Where it is not, scraping fails and the Metrics API has no data.

kubectl -n kube-system logs deploy/metrics-server --tail=50 | grep -i -E 'x509|tls|certificate'

2Fix

The proper fix is kubelet serving certificates signed by the cluster CA, or pointing metrics-server at the CA with --kubelet-certificate-authority. On a throw-away or test cluster you can skip verification with --kubelet-insecure-tls, which the project marks as testing only.

kubectl -n kube-system edit deploy metrics-server
# under the container args, add:
#   - --kubelet-insecure-tls

3Verify

Let the new pod start, then check node metrics.

kubectl -n kube-system rollout status deploy/metrics-server
kubectl top nodes

Confirmed by github.com

Did this fix it?

Cause 3 of 3

The kube-apiserver and metrics-server cannot reach each other, or the kubelets

1Check

The control plane must reach metrics-server by its container IP, and metrics-server must reach every kubelet on port 10250. Firewalls, security groups and network policies break one of the two.

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl -n kube-system get pods -o wide

2Fix

Open port 10250 between the nodes and the pod network. If the control plane cannot reach pod IPs (some managed or self-managed setups), run metrics-server with hostNetwork so it is reached by node IP on a custom port.

# in the metrics-server pod spec, only if the control plane cannot reach pod IPs
hostNetwork: true
# and set the container's --secure-port to a free node port

3Verify

The APIService reports Available True.

kubectl get apiservice v1beta1.metrics.k8s.io

Confirmed by github.com

Did this fix it?

How do I stop it coming back?

  • Install metrics-server as part of cluster bootstrap and check kubectl top nodes in the smoke test.
  • Use kubelet certificates signed by the cluster CA instead of --kubelet-insecure-tls outside test clusters.
  • Keep port 10250 open between nodes and the pod network in any firewall rules you write.
  • Check the compatibility matrix before upgrading Kubernetes or metrics-server.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.