Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

MongoDB pod will not run on a Kubernetes node whose CPU lacks AVX

ErrorMongoDB deployment not working in Kubernetes because the processor does not have AVX support

Short answer

MongoDB 5.0 and later only runs on x86_64 CPUs that support the AVX instruction set. If your Kubernetes node has an older or virtualised CPU without AVX, the pod cannot start. Either run a MongoDB version below 5.0, or schedule the pod onto nodes whose CPU does have AVX.

  • Applies to MongoDB 5.0 and later, Kubernetes 1.30+, Docker mongo image
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Find out which node the pod landed on and which image tag it runs. The tag tells you whether the AVX rule applies at all (5.0 or later).

    kubectl get pod -l app=mongo -o wide
    kubectl get pod -l app=mongo -o jsonpath='{.items[*].spec.containers[*].image}'
  2. 2Match what you see, then jump to its fix

    1. 1The image tag is 5.0 or higher, and the same Deployment worked with an older tagThe image is MongoDB 5.0 or later and the node CPU has no AVXFix
    2. 2Some nodes in the cluster run MongoDB 5.0 fine, others do notThe cluster has AVX nodes, but the pod is scheduled onto one withoutFix

What does the error look like?

  • MongoDB deployment not working in Kubernetes because the processor does not have AVX support

Cause 1 of 2

The image is MongoDB 5.0 or later and the node CPU has no AVX

1Check

The MongoDB production notes say MongoDB 5.0 and later requires the AVX instruction set, and that mongod no longer supports x86_64 platforms below the minimum microarchitecture. Intel needs Haswell or later; AMD needs Bulldozer or later.

kubectl get pod -l app=mongo -o jsonpath='{.items[*].spec.containers[*].image}'

2Fix

If you cannot change the hardware, pin an image tag below 5.0, for example 4.4. Plan an upgrade, because the old release no longer gets new features.

# in the Deployment or StatefulSet
containers:
  - name: mongo
    image: mongo:4.4

3Verify

Roll the workload and watch the new pod reach Running.

kubectl rollout restart deployment/mongo
kubectl get pods -l app=mongo -w

Confirmed by mongodb.com

Did this fix it?

Cause 2 of 2

The cluster has AVX nodes, but the pod is scheduled onto one without

1Check

With Node Feature Discovery installed, nodes get a label per CPU capability. The AVX label is feature.node.kubernetes.io/cpu-cpuid.AVX with the value true. Nodes without it are the ones to avoid.

kubectl get nodes -l feature.node.kubernetes.io/cpu-cpuid.AVX=true

2Fix

Add a nodeSelector to the pod template so Kubernetes only places the pod on nodes carrying that label. Without Node Feature Discovery, label the good nodes yourself and select on your own label.

spec:
  template:
    spec:
      nodeSelector:
        feature.node.kubernetes.io/cpu-cpuid.AVX: "true"

3Verify

Check that the new pod runs on one of the labelled nodes.

kubectl get pod -l app=mongo -o wide

Confirmed by kubernetes.io

Did this fix it?

How do I stop it coming back?

  • Read the CPU requirement in the MongoDB production notes before moving to a new major version.
  • Pin the image tag instead of using latest, so a new major release never arrives unannounced.
  • Label nodes by CPU capability and select on the label for workloads that need it.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.