Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

The Java SDK finds no AWS credentials on an EC2 instance or in a container on it

Errorsoftware.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain

Short answer

The SDK walks a fixed list of credential sources and the EC2 instance role is the last one. The error means every earlier source was empty and the metadata service gave nothing. Check that an instance profile is attached, then check the metadata request can reach the instance, which fails inside containers when the hop limit is 1.

  • Applies to AWS SDK for Java 2.x, Amazon EC2, AWS CLI v2, Docker 27
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Run these on the instance itself (inside the container if the app runs in one). They ask the metadata service the same question the SDK asks: first for a token, then for the name of the attached role.

    TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
    curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/
  2. 2Match what you see, then jump to its fix

    1. 1The second curl returns an empty body or 404, and the exception lists InstanceProfileCredentialsProvider lastNo IAM role is attached to the instanceFix
    2. 2The curl works on the host but hangs or returns nothing inside the containerThe app runs in a container and the metadata reply cannot reach itFix
    3. 3The app picks up the wrong account, or logs a timeout while reading from the metadata serviceAnother source in the chain answers first, or the metadata call times outFix

What does the error look like?

  • software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain
  • InstanceProfileCredentialsProvider()

Cause 1 of 3

No IAM role is attached to the instance

1Check

Ask EC2 which instance profile the instance has. An empty result means nothing is attached, so there is nothing for the SDK to read.

aws ec2 describe-iam-instance-profile-associations --filters Name=instance-id,Values=i-1234567890abcdef0

2Fix

Attach an instance profile (the container for the IAM role) to the running instance. An instance holds one role only.

aws ec2 associate-iam-instance-profile \
    --instance-id i-1234567890abcdef0 \
    --iam-instance-profile Name=my-app-role

3Verify

Repeat the second curl from the first step. It should now print the role name.

curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 3

The app runs in a container and the metadata reply cannot reach it

1Check

The SDK asks IMDSv2 for a token first. The reply to that PUT request has a hop limit, and going into a container counts as one more hop, so a limit of 1 drops it. Read the current value.

aws ec2 describe-instances --instance-id i-1234567890abcdef0 --query 'Reservations[].Instances[].MetadataOptions'

2Fix

Raise the hop limit to 2 for the instance (the endpoint must stay enabled).

aws ec2 modify-instance-metadata-options \
    --instance-id i-1234567890abcdef0 \
    --http-put-response-hop-limit 2 \
    --http-endpoint enabled

3Verify

Run the two curl commands from the first step again inside the container.

docker exec my-container curl -s -X PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 60"

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 3 of 3

Another source in the chain answers first, or the metadata call times out

1Check

The chain checks Java system properties, then environment variables, then web identity, then the shared credentials file, then ECS container credentials, and only then the EC2 role. A stale key in any earlier place wins over the role. List what is set.

env | grep -E '^AWS_'
ls -l ~/.aws/credentials ~/.aws/config

2Fix

Remove the stale variables or profile entries so the instance role is reached. If the metadata call is only slow, raise the timeout (seconds) with the environment variable the SDK for Java 2.x supports.

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
export AWS_METADATA_SERVICE_TIMEOUT=5

3Verify

Restart the app and make one call that needs credentials.

aws sts get-caller-identity

Confirmed by docs.aws.amazon.com

Did this fix it?

How do I stop it coming back?

  • Attach the role at launch through the launch template, so no instance starts without credentials.
  • Run containers with a hop limit of 2 set in the launch template from the start.
  • Keep long-lived access keys off instances; the role removes the need for them.
  • Build clients without an explicit credentials provider, so the default chain does the lookup.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.