Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

EKS EBS CSI driver cannot create volumes: NoCredentialProviders

Errorcould not create volume in EC2: NoCredentialProviders: no valid providers in chain

Short answer

The EBS CSI controller pod has no AWS credentials. With IAM roles for service accounts that means one of three gaps: no IAM OIDC provider for the cluster, no role attached to the aws-ebs-csi-driver add-on, or a trust policy whose audience or subject does not match the ebs-csi-controller-sa service account.

  • Applies to Amazon EKS, aws-ebs-csi-driver add-on, AWS CLI v2, eksctl
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the PVC events to confirm it is a credentials error, then check whether the service account carries a role.

    kubectl describe pvc <pvc-name>
    kubectl -n kube-system get sa ebs-csi-controller-sa -o yaml
  2. 2Match what you see, then jump to its fix

    1. 1The service account has a role annotation, but the role can never be assumedThe cluster has no IAM OIDC providerFix
    2. 2The ebs-csi-controller-sa service account has no role ARN, or the role was never createdNo IAM role is attached to the EBS CSI add-onFix
    3. 3The role exists and is attached, but the controller still has no credentialsThe role's trust policy does not match the service accountFix

What does the error look like?

  • could not create volume in EC2: NoCredentialProviders: no valid providers in chain
  • failed to provision volume with StorageClass

Cause 1 of 3

The cluster has no IAM OIDC provider

1Check

IAM roles for service accounts need an IAM OIDC provider for the cluster's issuer URL. Compare the issuer ID with the providers in your account.

oidc_id=$(aws eks describe-cluster --name <cluster> --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4

2Fix

If the check prints nothing, create the provider for the cluster.

eksctl utils associate-iam-oidc-provider --cluster <cluster> --approve

3Verify

Run the check again and see the provider ID printed. Then retry the claim and read its events.

kubectl describe pvc <pvc-name>

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 3

No IAM role is attached to the EBS CSI add-on

1Check

Without the IAM permissions the plugin cannot call AWS APIs. AWS notes that skipping the role step makes provisioning fail on the PVC. Look for the role annotation on the service account.

kubectl -n kube-system get sa ebs-csi-controller-sa -o jsonpath='{.metadata.annotations}'

2Fix

Create a role for the driver with the AWS managed policy AmazonEBSCSIDriverPolicyV2, then give the add-on that role.

aws iam attach-role-policy \
  --policy-arn arn:aws:iam::aws:policy/AmazonEBSCSIDriverPolicyV2 \
  --role-name AmazonEKS_EBS_CSI_DriverRole
aws eks update-addon --cluster-name <cluster> --addon-name aws-ebs-csi-driver \
  --service-account-role-arn arn:aws:iam::<account-id>:role/AmazonEKS_EBS_CSI_DriverRole

3Verify

Wait for the add-on update, then retry the PVC.

kubectl get pvc <pvc-name>

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 3 of 3

The role's trust policy does not match the service account

1Check

The trust policy must allow sts:AssumeRoleWithWebIdentity from the cluster's OIDC provider, with the audience sts.amazonaws.com and the subject system:serviceaccount:kube-system:ebs-csi-controller-sa.

aws iam get-role --role-name AmazonEKS_EBS_CSI_DriverRole --query Role.AssumeRolePolicyDocument

2Fix

Correct the Federated principal, the :aud and the :sub lines so they use your cluster's OIDC provider ID.

"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
  "StringEquals": {
    "oidc.eks.<region>.amazonaws.com/id/<OIDC-ID>:aud": "sts.amazonaws.com",
    "oidc.eks.<region>.amazonaws.com/id/<OIDC-ID>:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
  }
}

3Verify

Retry the claim and read its events.

kubectl describe pvc <pvc-name>

Confirmed by docs.aws.amazon.com

Did this fix it?

How do I stop it coming back?

  • Create the IAM OIDC provider as part of cluster creation, in the same infrastructure code.
  • Create the driver role and the add-on together, so the add-on never runs without a role.
  • Use a custom KMS key policy on the role when volumes are encrypted with your own key.
  • AWS suggests EKS Pod Identity for new setups; it avoids the OIDC trust policy entirely.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Apache has a certificate, but https:// does not open

No line with :443 in the output of ss -ltnp

HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.

4 causes, each with a fix

Servers

AWS CLI says you must specify a region

You must specify a region. You can also configure your region by running "aws configure".

The AWS CLI sends every request to one region and has none to use.

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.