Application Load Balancer returns 504 for requests to EC2 instances
HTTP 504: Gateway timeoutShort answer
A 504 from an Application Load Balancer means it could not get an answer from a target in time. The documented causes are: no connection to the target within 10 seconds, a target that did not reply before the idle timeout, a network ACL blocking ephemeral ports, or a target sending a Content-Length larger than the body.
Fix it fast
1Find out which cause you have
Open the target group and look at the health status of each target. A target that is unhealthy or never reached the load balancer points to the network causes first. If targets are healthy and only slow requests fail, start with the idle timeout.
aws elbv2 describe-target-groups --load-balancer-arn <load-balancer-arn> aws elbv2 describe-target-health --target-group-arn <target-group-arn>2Match what you see, then jump to its fix
- 1
Targets show as unhealthy or time out on the health check, and every request to them gets a 504The load balancer cannot open a connection to the target in 10 secondsFix - 2
Security groups look right, but traffic still times out between the subnetsA network ACL blocks the ephemeral portsFix - 3
Only slow routes fail, and they fail after about the same number of seconds each timeThe target connects but does not answer before the idle timeoutFix - 4
Responses from one endpoint hang and then end in a 504, while other endpoints are fineThe target sends a Content-Length larger than the bodyFix
- 1
What does the error look like?
HTTP 504: Gateway timeoutHTTPCode_ELB_504_Count
Cause 1 of 4
The load balancer cannot open a connection to the target in 10 seconds
1Check
The security group on the instances must allow traffic from the load balancer on the health check port and protocol, and the load balancer's own security group must allow traffic out to the instances. First connect to the target directly from inside the network using its private IP.
curl -sv --max-time 5 http://<target-private-ip>:<port>/health
aws ec2 describe-security-groups --group-ids <instance-sg-id>2Fix
Add a rule to the instance security group that allows the load balancer's security group on the target port. If the connection times out because the instance is overloaded, add targets to the target group.
aws ec2 authorize-security-group-ingress \
--group-id <instance-sg-id> \
--protocol tcp --port 8080 \
--source-group <load-balancer-sg-id>3Verify
Wait for the next health checks and confirm the targets turn healthy.
aws elbv2 describe-target-health --target-group-arn <target-group-arn>Confirmed by docs.aws.amazon.com
Cause 2 of 4
A network ACL blocks the ephemeral ports
1Check
The network ACL on the target subnet must allow the targets to answer on the ephemeral ports 1024-65535, and the ACL on the load balancer subnets must allow those ports inbound. ACLs are stateless, so both directions need a rule.
aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<subnet-id>2Fix
Add allow rules for the health check port and the ephemeral port range in both directions on both sets of subnets.
# In the VPC console, open Network ACLs, select the ACL of each subnet and add
# allow rules for TCP 1024-65535 (and the health check port) in the direction listed above.3Verify
Send a request through the load balancer.
curl -i https://example.com/Confirmed by docs.aws.amazon.com
Cause 3 of 4
The target connects but does not answer before the idle timeout
1Check
The load balancer closes a connection that has had no data for the idle timeout period. Long reports, uploads and slow queries run into this. Read the current value.
aws elbv2 describe-load-balancer-attributes --load-balancer-arn <load-balancer-arn>2Fix
Raise idle_timeout.timeout_seconds (valid range 1 to 4000 seconds) and make sure your application's own idle timeout is longer than the load balancer's. For uploads, send at least one byte before each idle period ends.
aws elbv2 modify-load-balancer-attributes \
--load-balancer-arn <load-balancer-arn> \
--attributes "Key=idle_timeout.timeout_seconds,Value=120"3Verify
Repeat the slow request.
curl -i -m 130 https://example.com/slow-reportConfirmed by docs.aws.amazon.com
Cause 4 of 4
The target sends a Content-Length larger than the body
1Check
If the target announces more bytes than it sends, the load balancer waits for the missing bytes until it times out. Compare the header with the bytes actually received.
curl -s -D - -o /tmp/body http://<target-private-ip>:<port>/path | grep -i content-length
wc -c /tmp/body2Fix
Fix the application or middleware so the Content-Length matches the body, or use chunked transfer encoding when the length is not known up front.
# Example in Node.js: let the server set the header
res.setHeader('Content-Type', 'application/json');
res.end(JSON.stringify(data));3Verify
Call the endpoint through the load balancer.
curl -i https://example.com/pathConfirmed by docs.aws.amazon.com
How do I stop it coming back?
- Give every target group a cheap health check path that does not touch the database.
- Set your application's keep-alive and idle timeouts longer than the load balancer's idle timeout.
- Allow traffic by security group reference, not by IP address, so scaling does not break the rule.
- Alarm on HTTPCode_ELB_504_Count so a timeout is seen before users report it.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
- Dispatch ProA real-time logistics platform: Flutter apps, Spring Boot services and a Kafka event pipeline on AWS.
- RaheebA consumer app shipped to both stores from one Flutter codebase on a .NET and Azure back end.
- Loade DriverThe driver side of a delivery platform: live jobs, routing and earnings, past 5K Google Play downloads.
Related fixes
Servers
Status code 460 on an AWS Application Load Balancer
elb_status_code 460 in ALB access logs
A 460 is generated by the load balancer, not by your app.
3 causes, each with a fixServers
Apache has a certificate, but https:// does not open
No line with :443 in the output of ss -ltnp
HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.
4 causes, each with a fixServers
aws emr --steps fails: Error parsing parameter, Expected '=', received 'EOF'
Error parsing parameter '--steps': Expected: '=', received: 'EOF' for input:
The AWS CLI tried to read your option as shorthand key=value pairs and ran out of text where it expected an equals sign.
3 causes, each with a fixServers
AWS CLI says you must specify a region
You must specify a region. You can also configure your region by running "aws configure".
The AWS CLI sends every request to one region and has none to use.
4 causes, each with a fixServers
docker login to Amazon ECR fails with: Cannot perform an interactive login from a non TTY device
Cannot perform an interactive login from a non TTY device
Docker asked for a password because none reached it, and a pipe or script has no keyboard to answer.
3 causes, each with a fixServers
aws s3 cp fails with 'An error occurred (400) when calling the HeadObject operation'
Completed 1 part(s) with ... file(s) remaining
A failed HEAD request returns only a generic status, so the 400 hides the reason.
2 causes, each with a fix