Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

Application Load Balancer returns 504 for requests to EC2 instances

ErrorHTTP 504: Gateway timeout

Short answer

A 504 from an Application Load Balancer means it could not get an answer from a target in time. The documented causes are: no connection to the target within 10 seconds, a target that did not reply before the idle timeout, a network ACL blocking ephemeral ports, or a target sending a Content-Length larger than the body.

  • Applies to Application Load Balancer, AWS CLI v2, EC2 targets
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Open the target group and look at the health status of each target. A target that is unhealthy or never reached the load balancer points to the network causes first. If targets are healthy and only slow requests fail, start with the idle timeout.

    aws elbv2 describe-target-groups --load-balancer-arn <load-balancer-arn>
    aws elbv2 describe-target-health --target-group-arn <target-group-arn>
  2. 2Match what you see, then jump to its fix

    1. 1Targets show as unhealthy or time out on the health check, and every request to them gets a 504The load balancer cannot open a connection to the target in 10 secondsFix
    2. 2Security groups look right, but traffic still times out between the subnetsA network ACL blocks the ephemeral portsFix
    3. 3Only slow routes fail, and they fail after about the same number of seconds each timeThe target connects but does not answer before the idle timeoutFix
    4. 4Responses from one endpoint hang and then end in a 504, while other endpoints are fineThe target sends a Content-Length larger than the bodyFix

What does the error look like?

  • HTTP 504: Gateway timeout
  • HTTPCode_ELB_504_Count

Cause 1 of 4

The load balancer cannot open a connection to the target in 10 seconds

1Check

The security group on the instances must allow traffic from the load balancer on the health check port and protocol, and the load balancer's own security group must allow traffic out to the instances. First connect to the target directly from inside the network using its private IP.

curl -sv --max-time 5 http://<target-private-ip>:<port>/health
aws ec2 describe-security-groups --group-ids <instance-sg-id>

2Fix

Add a rule to the instance security group that allows the load balancer's security group on the target port. If the connection times out because the instance is overloaded, add targets to the target group.

aws ec2 authorize-security-group-ingress \
  --group-id <instance-sg-id> \
  --protocol tcp --port 8080 \
  --source-group <load-balancer-sg-id>

3Verify

Wait for the next health checks and confirm the targets turn healthy.

aws elbv2 describe-target-health --target-group-arn <target-group-arn>

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 4

A network ACL blocks the ephemeral ports

1Check

The network ACL on the target subnet must allow the targets to answer on the ephemeral ports 1024-65535, and the ACL on the load balancer subnets must allow those ports inbound. ACLs are stateless, so both directions need a rule.

aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<subnet-id>

2Fix

Add allow rules for the health check port and the ephemeral port range in both directions on both sets of subnets.

# In the VPC console, open Network ACLs, select the ACL of each subnet and add
# allow rules for TCP 1024-65535 (and the health check port) in the direction listed above.

3Verify

Send a request through the load balancer.

curl -i https://example.com/

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 3 of 4

The target connects but does not answer before the idle timeout

1Check

The load balancer closes a connection that has had no data for the idle timeout period. Long reports, uploads and slow queries run into this. Read the current value.

aws elbv2 describe-load-balancer-attributes --load-balancer-arn <load-balancer-arn>

2Fix

Raise idle_timeout.timeout_seconds (valid range 1 to 4000 seconds) and make sure your application's own idle timeout is longer than the load balancer's. For uploads, send at least one byte before each idle period ends.

aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn <load-balancer-arn> \
  --attributes "Key=idle_timeout.timeout_seconds,Value=120"

3Verify

Repeat the slow request.

curl -i -m 130 https://example.com/slow-report

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 4 of 4

The target sends a Content-Length larger than the body

1Check

If the target announces more bytes than it sends, the load balancer waits for the missing bytes until it times out. Compare the header with the bytes actually received.

curl -s -D - -o /tmp/body http://<target-private-ip>:<port>/path | grep -i content-length
wc -c /tmp/body

2Fix

Fix the application or middleware so the Content-Length matches the body, or use chunked transfer encoding when the length is not known up front.

# Example in Node.js: let the server set the header
res.setHeader('Content-Type', 'application/json');
res.end(JSON.stringify(data));

3Verify

Call the endpoint through the load balancer.

curl -i https://example.com/path

Confirmed by docs.aws.amazon.com

Did this fix it?

How do I stop it coming back?

  • Give every target group a cheap health check path that does not touch the database.
  • Set your application's keep-alive and idle timeouts longer than the load balancer's idle timeout.
  • Allow traffic by security group reference, not by IP address, so scaling does not break the rule.
  • Alarm on HTTPCode_ELB_504_Count so a timeout is seen before users report it.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Apache has a certificate, but https:// does not open

No line with :443 in the output of ss -ltnp

HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.

4 causes, each with a fix

Servers

AWS CLI says you must specify a region

You must specify a region. You can also configure your region by running "aws configure".

The AWS CLI sends every request to one region and has none to use.

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.