Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

A pod stays Pending with volume node affinity conflict

Error1 node(s) had volume node affinity conflict, 3 node(s) didn't match Pod's node affinity/selector

Short answer

The scheduler found that the pod's volume can only be reached from some nodes, and none of those nodes are allowed for the pod. The usual cause is a StorageClass that binds volumes immediately, in a zone the pod cannot use. Use WaitForFirstConsumer for new volumes, and align node selectors with the volume's node affinity.

  • Applies to Kubernetes 1.3x, kubectl 1.3x
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the pod's events to see which nodes the scheduler rejected, then read the volume's node affinity to see where it can be used.

    kubectl describe pod redis-0 | tail -n 15
    kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,AFFINITY:.spec.nodeAffinity.required
  2. 2Match what you see, then jump to its fix

    1. 1The volume's affinity names one zone or node, and the pod's other constraints exclude itThe StorageClass binds volumes before the pod is scheduledFix
    2. 2The event says the nodes did not match the pod's node affinity or selector, and the volume's affinity lists other nodesThe pod's selector or affinity excludes every node the volume can useFix
    3. 3A local volume's affinity names one node, and that node is unschedulableThe node that holds the volume is cordoned or goneFix

What does the error look like?

  • 1 node(s) had volume node affinity conflict, 3 node(s) didn't match Pod's node affinity/selector
  • STATUS Pending

Cause 1 of 3

The StorageClass binds volumes before the pod is scheduled

1Check

With the default Immediate mode, the volume is provisioned when the claim is created, without knowing where the pod will run, which can leave the pod unschedulable.

kubectl get storageclass -o custom-columns=NAME:.metadata.name,MODE:.volumeBindingMode

2Fix

Use a StorageClass with volumeBindingMode WaitForFirstConsumer for new volumes, so provisioning follows the pod's scheduling constraints. Existing bound volumes stay where they are, so delete the pending claim and its volume if the data is disposable.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-wait
provisioner: example.com/example   # use your cluster's provisioner
volumeBindingMode: WaitForFirstConsumer

3Verify

A new claim stays Pending until a pod uses it, then the pod schedules.

kubectl get pvc,pod -w

Confirmed by kubernetes.io

Did this fix it?

Cause 2 of 3

The pod's selector or affinity excludes every node the volume can use

1Check

Pods that use a volume are only scheduled to nodes selected by its node affinity, and the pod's own constraints must overlap with it.

kubectl get statefulset redis -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}{.spec.template.spec.affinity}{"\n"}'
kubectl get nodes --show-labels

2Fix

Change the selector or node affinity in the StatefulSet so it includes nodes in the volume's zone. When provisioning is meant to stay in certain zones, set allowedTopologies in the StorageClass.

allowedTopologies:
- matchLabelExpressions:
  - key: topology.kubernetes.io/zone
    values:
    - us-central-1a
    - us-central-1b

3Verify

The pod leaves Pending.

kubectl get pod redis-0 -o wide

Confirmed by kubernetes.io

Did this fix it?

Cause 3 of 3

The node that holds the volume is cordoned or gone

1Check

For local volumes, node affinity must be set explicitly and ties the pod to that node. If it was drained, the pod waits.

kubectl get nodes

2Fix

Make the node schedulable again, or recreate the volume elsewhere if the node is lost and the data can be restored.

kubectl uncordon <node name>

3Verify

The node shows Ready without SchedulingDisabled, and the pod is scheduled.

kubectl get nodes && kubectl get pod redis-0

Confirmed by kubernetes.io

Did this fix it?

How do I stop it coming back?

  • Set WaitForFirstConsumer on every StorageClass backed by zonal disks.
  • Spread StatefulSet replicas with the same zones your volumes use.
  • After node maintenance, run uncordon, and do not leave drained nodes behind.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.