Pod stuck in ContainerCreating: unable to attach or mount EBS volumes
Unable to attach or mount volumes : timed out waiting for the conditionShort answer
The kubelet waited for the volume and gave up. With EBS on EKS the documented causes are: the CSI driver has no IAM permissions, a ReadWriteOnce volume is wanted by a second node, or the volume sits in a different Availability Zone from the node because binding was not delayed to the pod.
Fix it fast
1Find out which cause you have
Describe the pod and the claim. The events say whether provisioning failed, whether the attach was refused, or whether the volume is in use elsewhere.
kubectl describe pod <pod-name> kubectl describe pvc <pvc-name>2Match what you see, then jump to its fix
- 1
PVC events: failed to provision volume with StorageClass, with could not create volume in EC2: UnauthorizedOperationThe EBS CSI driver has no IAM permissionsFix - 2
Two replicas or a rolling update, and the new pod lands on a different node than the one using the volumeA ReadWriteOnce volume is requested from a second nodeFix - 3
The pod is scheduled, but the volume never attaches; the claim was bound before the pod was placedThe volume and the node are in different Availability ZonesFix
- 1
What does the error look like?
Unable to attach or mount volumes : timed out waiting for the conditionfailed to provision volume with StorageClass
Cause 1 of 3
The EBS CSI driver has no IAM permissions
1Check
The driver calls AWS APIs for you. Without its IAM role, installing the add-on and creating a claim ends in exactly this event.
kubectl describe pvc <pvc-name> | grep -i -A3 "provision"2Fix
Create the driver role with AmazonEBSCSIDriverPolicyV2 and attach it to the aws-ebs-csi-driver add-on. The same page walks through the role and trust policy.
aws eks update-addon --cluster-name <cluster> --addon-name aws-ebs-csi-driver \
--service-account-role-arn arn:aws:iam::<account-id>:role/AmazonEKS_EBS_CSI_DriverRole3Verify
Delete the stuck pod so it is recreated, then watch it start.
kubectl get pod <pod-name> -wConfirmed by docs.aws.amazon.com
Cause 2 of 3
A ReadWriteOnce volume is requested from a second node
1Check
ReadWriteOnce means the volume can be mounted read-write by a single node. Pods on the same node can share it; pods on other nodes cannot. Compare the node of each pod.
kubectl get pods -o wide
kubectl get pvc <pvc-name> -o jsonpath='{.spec.accessModes}'2Fix
Let only pods on one node use the claim: run a single replica, or give each replica its own claim. ReadWriteOncePod limits access to a single pod.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
spec:
accessModes:
- ReadWriteOncePod
resources:
requests:
storage: 10Gi3Verify
Start the pod again and confirm it reaches Running.
kubectl get pod <pod-name> -o wideConfirmed by kubernetes.io
Cause 3 of 3
The volume and the node are in different Availability Zones
1Check
EBS volumes attach only to instances in the same Availability Zone. With the default Immediate binding, the volume is created before the pod is scheduled, so the pod can land in another zone. Check the storage class.
kubectl get storageclass -o custom-columns=NAME:.metadata.name,MODE:.volumeBindingMode2Fix
Use WaitForFirstConsumer, which delays binding and provisioning until a pod uses the claim, so the volume is created in the pod's zone. The volume of an already bound claim keeps its zone: recreate the claim, or schedule the pod into that zone.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3-delayed
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer3Verify
Create a new claim with this class and start the pod.
kubectl get pvc,podConfirmed by kubernetes.io
How do I stop it coming back?
- Create the IAM role and the EBS CSI add-on together before any workload uses a claim.
- Use WaitForFirstConsumer in every storage class that points at zonal storage.
- Plan for one pod per ReadWriteOnce volume, and give each replica its own claim.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
- Dispatch ProA real-time logistics platform: Flutter apps, Spring Boot services and a Kafka event pipeline on AWS.
- RaheebA consumer app shipped to both stores from one Flutter codebase on a .NET and Azure back end.
- Loade DriverThe driver side of a delivery platform: live jobs, routing and earnings, past 5K Google Play downloads.
Related fixes
Servers
A pod stays Pending with volume node affinity conflict
STATUS Pending
The scheduler found that the pod's volume can only be reached from some nodes, and none of those nodes are allowed for the pod.
3 causes, each with a fixServers
EKS managed node group ends in Create failed because nodes never joined
NodeCreationFailure
Nodes that do not register within 15 minutes cause NodeCreationFailure and a Create failed status.
4 causes, each with a fixServers
Status code 460 on an AWS Application Load Balancer
elb_status_code 460 in ALB access logs
A 460 is generated by the load balancer, not by your app.
3 causes, each with a fixServers
Application Load Balancer returns 504 for requests to EC2 instances
HTTPCode_ELB_504_Count
A 504 from an Application Load Balancer means it could not get an answer from a target in time.
4 causes, each with a fixServers
Apache has a certificate, but https:// does not open
No line with :443 in the output of ss -ltnp
HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.
4 causes, each with a fixServers
Argo CD behind an nginx Ingress loops with too many redirects
307 Temporary Redirect on every request to the Argo CD host
The browser and Argo CD disagree about who handles TLS.
3 causes, each with a fix