Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

aws s3 cp fails with 'An error occurred (400) when calling the HeadObject operation'

ErrorA client error (400) occurred when calling the HeadObject operation: Bad Request

Short answer

A failed HEAD request returns only a generic status, so the 400 hides the reason. Repeat the call as a GET with s3api get-object to see the real error. The documented 400 causes are encryption headers that do not fit the object: SSE headers sent for an SSE-KMS or SSE-S3 object, or SSE-C objects read without the customer key.

  • Applies to AWS CLI v2, Amazon S3 general purpose buckets
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    HeadObject has no response body, so S3 can only give a generic code. A GET on the same object is identical except that it has a body, so the error code and message become visible.

    aws s3api get-object --bucket <bucket> --key <key> /tmp/out.bin
    aws s3 cp s3://<bucket>/<key> /tmp/out.bin --debug 2>&1 | grep -i "HTTP/"
  2. 2Match what you see, then jump to its fix

    1. 1The command includes --sse or --sse aws:kms, and the object is encrypted with SSE-S3 or SSE-KMSServer-side encryption headers are sent for an object that cannot take themFix
    2. 2Objects uploaded with --sse-c fail on download or copy, while ordinary objects workThe object uses SSE-C and the request has no customer keyFix

What does the error look like?

  • A client error (400) occurred when calling the HeadObject operation: Bad Request
  • Completed 1 part(s) with ... file(s) remaining

Cause 1 of 2

Server-side encryption headers are sent for an object that cannot take them

1Check

The x-amz-server-side-encryption header is meant for uploads. If it goes into a HEAD request for an object that uses SSE-S3, SSE-KMS or DSSE-KMS, S3 answers 400 Bad Request. Look for encryption options in the command, a profile or a wrapper script.

grep -rn -- "--sse" ./scripts 2>/dev/null
aws s3api head-object --bucket <bucket> --key <key> --query ServerSideEncryption

2Fix

Leave the --sse options off downloads and metadata calls. Use them only on the upload that writes the object.

# download: no encryption flags
aws s3 cp s3://<bucket>/<key> ./file
# upload: encryption flags belong here
aws s3 cp ./file s3://<bucket>/<key> --sse aws:kms

3Verify

Run the download again.

aws s3 cp s3://<bucket>/<key> ./file

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 2

The object uses SSE-C and the request has no customer key

1Check

For an object stored with a customer-provided key, the metadata request must send the algorithm, key and key MD5 headers. In the CLI those come from --sse-c and --sse-c-key.

grep -rn -- "--sse-c" ./scripts 2>/dev/null

2Fix

Pass the same customer key on the read. The key is the raw key, not base64 encoded, and --sse-c and --sse-c-key must be given together.

aws s3 cp s3://<bucket>/<key> ./file --sse-c AES256 --sse-c-key <raw-32-byte-key>

3Verify

The download completes without the 400.

ls -l ./file

Confirmed by docs.aws.amazon.com

Did this fix it?

How do I stop it coming back?

  • Wrap uploads and downloads in separate commands, so write-only flags never reach a read.
  • When a HEAD call fails, retry it as a GET before you guess; the GET shows the real S3 error code.
  • Keep customer-provided keys in a secrets store and pass them from a file at run time.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Apache has a certificate, but https:// does not open

No line with :443 in the output of ss -ltnp

HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.

4 causes, each with a fix

Servers

AWS CLI says you must specify a region

You must specify a region. You can also configure your region by running "aws configure".

The AWS CLI sends every request to one region and has none to use.

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.