Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

An Elasticsearch pod keeps restarting with Back-off restarting failed container

ErrorBack-off restarting failed container

Short answer

Back-off restarting failed container only says the container keeps dying. The cause is in the previous run's log and in the pod's last state. For Elasticsearch the most common reason is a node setting vm.max_map_count that is too low; the next is a memory limit that kills the JVM.

  • Applies to Kubernetes, Elasticsearch 8.16 and later, ECK
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the log of the instance that crashed, not the new one that is starting. Then look at Last State and Events in describe.

    kubectl logs elasticsearch-0 --previous
    kubectl describe pod elasticsearch-0
  2. 2Match what you see, then jump to its fix

    1. 1The previous log mentions vm.max_map_count being too lowvm.max_map_count on the node is too lowFix
    2. 2It worked after you set it, then the pod crashed again after a node reboot or a new node joinedThe setting is not permanent on the nodeFix
    3. 3Managed nodes refuse privileged containers and sysctl changesYou cannot change the node, and mmap is the only blockerFix

What does the error look like?

  • Back-off restarting failed container
  • CrashLoopBackOff
  • The log of the crashed container mentions vm.max_map_count

Cause 1 of 3

vm.max_map_count on the node is too low

1Check

This is a kernel setting of the node, not of the pod. Elastic's own guidance is to raise it to 1048576 (262144 for Elasticsearch 8.15 and earlier). Check the value on the node that runs the pod.

kubectl get pod elasticsearch-0 -o wide
sysctl vm.max_map_count   # run on that node

2Fix

Set it on the node, or let a privileged init container set it before Elasticsearch starts. The init container needs a cluster that allows privileged containers.

# on the node, until reboot
sudo sysctl -w vm.max_map_count=1048576

# pod spec: init container
initContainers:
  - name: sysctl
    image: busybox:1.36
    command: ["sysctl", "-w", "vm.max_map_count=1048576"]
    securityContext:
      privileged: true

3Verify

The pod stays Running and the node reports the new value.

kubectl get pod elasticsearch-0 -w
sysctl vm.max_map_count

Confirmed by elastic.co

Did this fix it?

Cause 2 of 3

The setting is not permanent on the node

1Check

sysctl -w lasts until reboot. Elastic documents the permanent route as an entry in /etc/sysctl.conf, applied with sysctl --system.

grep -R max_map_count /etc/sysctl.conf /etc/sysctl.d/

2Fix

Write the value into /etc/sysctl.conf on every node that can run Elasticsearch, or manage it with a DaemonSet or a node pool setting.

echo "vm.max_map_count=1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl --system

3Verify

The value survives a reboot.

sysctl vm.max_map_count

Confirmed by elastic.co

Did this fix it?

Cause 3 of 3

You cannot change the node, and mmap is the only blocker

1Check

ECK's quickstart avoids the kernel change by turning mmap off in the node set config. The same documentation recommends raising vm.max_map_count for production and leaving this setting unset.

kubectl get elasticsearch -o yaml | grep -n allow_mmap

2Fix

For a test cluster only, disable mmap in the Elasticsearch config. Do not combine it with the init container check.

spec:
  nodeSets:
    - name: default
      count: 1
      config:
        node.store.allow_mmap: false

3Verify

The pod leaves CrashLoopBackOff.

kubectl get pods -w

Confirmed by elastic.co

Did this fix it?

How do I stop it coming back?

  • Bake vm.max_map_count into the node image or node pool, not into a one-off command.
  • Run kubectl logs --previous first on any CrashLoopBackOff.
  • Use the same Elastic documentation version as your Elasticsearch version when you copy a value.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Elasticsearch and Kibana will not start with Docker Compose

Elasticsearch 8 and Kibana do not start with Docker Compose

Elasticsearch in Docker usually fails for three documented reasons: the host kernel's vm.max_map_count is too low, Docker has less than the 4GB of memory the Compose guide asks for, or the passwords in .env break the rules (6+ alphanumeric characters).

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.