Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

Pod stuck in ContainerCreating: unable to attach or mount EBS volumes

ErrorUnable to attach or mount volumes : timed out waiting for the condition

Short answer

The kubelet waited for the volume and gave up. With EBS on EKS the documented causes are: the CSI driver has no IAM permissions, a ReadWriteOnce volume is wanted by a second node, or the volume sits in a different Availability Zone from the node because binding was not delayed to the pod.

  • Applies to Amazon EKS, Kubernetes 1.3x, aws-ebs-csi-driver
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Describe the pod and the claim. The events say whether provisioning failed, whether the attach was refused, or whether the volume is in use elsewhere.

    kubectl describe pod <pod-name>
    kubectl describe pvc <pvc-name>
  2. 2Match what you see, then jump to its fix

    1. 1PVC events: failed to provision volume with StorageClass, with could not create volume in EC2: UnauthorizedOperationThe EBS CSI driver has no IAM permissionsFix
    2. 2Two replicas or a rolling update, and the new pod lands on a different node than the one using the volumeA ReadWriteOnce volume is requested from a second nodeFix
    3. 3The pod is scheduled, but the volume never attaches; the claim was bound before the pod was placedThe volume and the node are in different Availability ZonesFix

What does the error look like?

  • Unable to attach or mount volumes : timed out waiting for the condition
  • failed to provision volume with StorageClass

Cause 1 of 3

The EBS CSI driver has no IAM permissions

1Check

The driver calls AWS APIs for you. Without its IAM role, installing the add-on and creating a claim ends in exactly this event.

kubectl describe pvc <pvc-name> | grep -i -A3 "provision"

2Fix

Create the driver role with AmazonEBSCSIDriverPolicyV2 and attach it to the aws-ebs-csi-driver add-on. The same page walks through the role and trust policy.

aws eks update-addon --cluster-name <cluster> --addon-name aws-ebs-csi-driver \
  --service-account-role-arn arn:aws:iam::<account-id>:role/AmazonEKS_EBS_CSI_DriverRole

3Verify

Delete the stuck pod so it is recreated, then watch it start.

kubectl get pod <pod-name> -w

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 3

A ReadWriteOnce volume is requested from a second node

1Check

ReadWriteOnce means the volume can be mounted read-write by a single node. Pods on the same node can share it; pods on other nodes cannot. Compare the node of each pod.

kubectl get pods -o wide
kubectl get pvc <pvc-name> -o jsonpath='{.spec.accessModes}'

2Fix

Let only pods on one node use the claim: run a single replica, or give each replica its own claim. ReadWriteOncePod limits access to a single pod.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes:
    - ReadWriteOncePod
  resources:
    requests:
      storage: 10Gi

3Verify

Start the pod again and confirm it reaches Running.

kubectl get pod <pod-name> -o wide

Confirmed by kubernetes.io

Did this fix it?

Cause 3 of 3

The volume and the node are in different Availability Zones

1Check

EBS volumes attach only to instances in the same Availability Zone. With the default Immediate binding, the volume is created before the pod is scheduled, so the pod can land in another zone. Check the storage class.

kubectl get storageclass -o custom-columns=NAME:.metadata.name,MODE:.volumeBindingMode

2Fix

Use WaitForFirstConsumer, which delays binding and provisioning until a pod uses the claim, so the volume is created in the pod's zone. The volume of an already bound claim keeps its zone: recreate the claim, or schedule the pod into that zone.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-delayed
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

3Verify

Create a new claim with this class and start the pod.

kubectl get pvc,pod

Confirmed by kubernetes.io

Did this fix it?

How do I stop it coming back?

  • Create the IAM role and the EBS CSI add-on together before any workload uses a claim.
  • Use WaitForFirstConsumer in every storage class that points at zonal storage.
  • Plan for one pod per ReadWriteOnce volume, and give each replica its own claim.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Apache has a certificate, but https:// does not open

No line with :443 in the output of ss -ltnp

HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.