An Elasticsearch pod keeps restarting with Back-off restarting failed container
Back-off restarting failed containerShort answer
Back-off restarting failed container only says the container keeps dying. The cause is in the previous run's log and in the pod's last state. For Elasticsearch the most common reason is a node setting vm.max_map_count that is too low; the next is a memory limit that kills the JVM.
Fix it fast
1Find out which cause you have
Read the log of the instance that crashed, not the new one that is starting. Then look at Last State and Events in describe.
kubectl logs elasticsearch-0 --previous kubectl describe pod elasticsearch-02Match what you see, then jump to its fix
- 1
The previous log mentions vm.max_map_count being too lowvm.max_map_count on the node is too lowFix - 2
It worked after you set it, then the pod crashed again after a node reboot or a new node joinedThe setting is not permanent on the nodeFix - 3
Managed nodes refuse privileged containers and sysctl changesYou cannot change the node, and mmap is the only blockerFix
- 1
What does the error look like?
Back-off restarting failed containerCrashLoopBackOffThe log of the crashed container mentions vm.max_map_count
Cause 1 of 3
vm.max_map_count on the node is too low
1Check
This is a kernel setting of the node, not of the pod. Elastic's own guidance is to raise it to 1048576 (262144 for Elasticsearch 8.15 and earlier). Check the value on the node that runs the pod.
kubectl get pod elasticsearch-0 -o wide
sysctl vm.max_map_count # run on that node2Fix
Set it on the node, or let a privileged init container set it before Elasticsearch starts. The init container needs a cluster that allows privileged containers.
# on the node, until reboot
sudo sysctl -w vm.max_map_count=1048576
# pod spec: init container
initContainers:
- name: sysctl
image: busybox:1.36
command: ["sysctl", "-w", "vm.max_map_count=1048576"]
securityContext:
privileged: true3Verify
The pod stays Running and the node reports the new value.
kubectl get pod elasticsearch-0 -w
sysctl vm.max_map_countConfirmed by elastic.co
Cause 2 of 3
The setting is not permanent on the node
1Check
sysctl -w lasts until reboot. Elastic documents the permanent route as an entry in /etc/sysctl.conf, applied with sysctl --system.
grep -R max_map_count /etc/sysctl.conf /etc/sysctl.d/2Fix
Write the value into /etc/sysctl.conf on every node that can run Elasticsearch, or manage it with a DaemonSet or a node pool setting.
echo "vm.max_map_count=1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl --system3Verify
The value survives a reboot.
sysctl vm.max_map_countConfirmed by elastic.co
Cause 3 of 3
You cannot change the node, and mmap is the only blocker
1Check
ECK's quickstart avoids the kernel change by turning mmap off in the node set config. The same documentation recommends raising vm.max_map_count for production and leaving this setting unset.
kubectl get elasticsearch -o yaml | grep -n allow_mmap2Fix
For a test cluster only, disable mmap in the Elasticsearch config. Do not combine it with the init container check.
spec:
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false3Verify
The pod leaves CrashLoopBackOff.
kubectl get pods -wConfirmed by elastic.co
How do I stop it coming back?
- Bake vm.max_map_count into the node image or node pool, not into a one-off command.
- Run kubectl logs --previous first on any CrashLoopBackOff.
- Use the same Elastic documentation version as your Elasticsearch version when you copy a value.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
Related fixes
Servers
Elasticsearch and Kibana will not start with Docker Compose
Elasticsearch 8 and Kibana do not start with Docker Compose
Elasticsearch in Docker usually fails for three documented reasons: the host kernel's vm.max_map_count is too low, Docker has less than the 4GB of memory the Compose guide asks for, or the passwords in .env break the rules (6+ alphanumeric characters).
4 causes, each with a fixServers
A Docker container exits with code 137
exited with code 137
Exit code 137 is 128 plus signal 9, so the process was killed with SIGKILL.
3 causes, each with a fixServers
Argo CD behind an nginx Ingress loops with too many redirects
307 Temporary Redirect on every request to the Argo CD host
The browser and Argo CD disagree about who handles TLS.
3 causes, each with a fixServers
A cert-manager Certificate stays at READY False with Let's Encrypt
Issuing certificate as Secret does not exist
A Certificate that stays not ready is waiting on a chain: Issuer, CertificateRequest, Order, Challenge.
3 causes, each with a fixServers
Kubernetes in Docker Desktop will not start or stays on starting
kubectl shows no contexts, or errors against the docker-desktop context
When the Docker Desktop cluster hangs on starting, the usual causes are too few resources, a broken cluster state, or the kubeadm provisioner failing.
4 causes, each with a fixServers
EKS managed node group ends in Create failed because nodes never joined
NodeCreationFailure
Nodes that do not register within 15 minutes cause NodeCreationFailure and a Create failed status.
4 causes, each with a fix