nginx gives up waiting for the app and returns 504
504 Gateway Time-outShort answer
nginx returns 504 when the app behind it takes the request but sends nothing back within proxy_read_timeout (60 seconds by default), or when the connection cannot be opened in time. A bigger timeout only hides it. Find which request is slow and why: slow code, work that belongs in a queue, a full worker pool, an old upstream address, or a load balancer in front.
Fix it fast
1Find out which cause you have
Pull the timeout lines out of the error log. "while reading response header" means the app accepted the request and went quiet; "while connecting" means nginx never got a connection at all. The request field on each line names the slow URL.
sudo grep "upstream timed out" /var/log/nginx/error.log | tail -n 202Match what you see, then jump to its fix
- 1
while reading response header from upstream, always on the same few URLsOne route does slow work before it answersFix - 2
Timeouts on exports, imports, report builds, bulk mail or media processing, getting worse as the data growsA long job runs inside the HTTP requestFix - 3
server reached pm.max_children setting in the PHP-FPM log, with 504s on pages that are normally fastEvery PHP-FPM worker is busy, so new requests wait in lineFix - 4
while connecting to upstream, after a backend server, container or managed service got a new IPnginx still connects to an old upstream addressFix - 5
The 504 page is not nginx's, the error log is silent at that moment, and the access log shows 499 for the requestA load balancer in front of nginx gives up firstFix
- 1
What does the error look like?
504 Gateway Time-outupstream timed out (110: Connection timed out) while reading response header from upstreamupstream timed out (110: Connection timed out) while connecting to upstreamWARNING: [pool www] server reached pm.max_children setting (5), consider raising it
Cause 1 of 5
One route does slow work before it answers
1Check
Time the route against the app itself, with nginx out of the way. Then add $upstream_response_time to the access log so every slow route shows its real duration from now on.
curl -o /dev/null -s -w '%{time_total}\n' http://127.0.0.1:3000/reports/monthly
# nginx, http block
log_format timed '$remote_addr "$request" $status $request_time $upstream_response_time';
access_log /var/log/nginx/access.log timed;2Fix
Fix what is slow: add the missing index, cache the expensive result, or give the outside API call the route waits on its own short timeout. If one route really needs longer, raise the limit for that location only, never for the whole server.
location /reports/ {
proxy_pass http://127.0.0.1:3000;
proxy_read_timeout 180s;
}3Verify
Reload nginx and watch the upstream time for that route in the access log.
sudo nginx -t && sudo systemctl reload nginx
sudo tail -f /var/log/nginx/access.log | grep /reports/Cause 2 of 5
A long job runs inside the HTTP request
1Check
If the run time depends on how much data a user has, no fixed timeout will hold. proxy_read_timeout counts the silence between two reads from the app, so a request that sends nothing for minutes is cut off whatever the total limit is. With the timed log format above, list requests that took over 30 seconds.
sudo awk '$NF > 30' /var/log/nginx/access.log | tail -n 202Fix
Move the work to a background queue. The request stores a job and answers 202 with a job id in milliseconds; a worker process does the work, and the page polls a status route or gets notified when the file is ready.
import { Queue } from 'bullmq';
const reports = new Queue('reports', { connection: { host: '127.0.0.1', port: 6379 } });
app.post('/reports', async (req, res) => {
const job = await reports.add('build', { userId: req.user.id });
res.status(202).json({ jobId: job.id });
});3Verify
The route answers at once with 202; the worker's log shows the job being processed.
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' -X POST http://127.0.0.1:3000/reportsCause 3 of 5
Every PHP-FPM worker is busy, so new requests wait in line
1Check
PHP-FPM serves one request per child process. When all children are tied up by slow requests, new ones queue until nginx's fastcgi_read_timeout (also 60 seconds by default) runs out. Count the warnings and measure how much memory one child uses.
sudo grep -c "max_children" /var/log/php8.3-fpm.log
ps -o rss= -C php-fpm8.3 | awk '{s+=$1} END {print s/NR/1024 " MB per child"}'2Fix
Raise pm.max_children only as far as memory allows (free RAM divided by the size of one child), and turn on the slow log so the scripts that hold workers show up with a backtrace.
; /etc/php/8.3/fpm/pool.d/www.conf
pm = dynamic
pm.max_children = 20
pm.start_servers = 5
pm.min_spare_servers = 3
pm.max_spare_servers = 8
request_slowlog_timeout = 10s
slowlog = /var/log/php8.3-fpm.slow.log3Verify
Restart PHP-FPM and check that the warning stays away under normal traffic.
sudo systemctl restart php8.3-fpm
sudo tail -f /var/log/php8.3-fpm.logCause 4 of 5
nginx still connects to an old upstream address
1Check
nginx resolves a host name in proxy_pass once, when it loads its config, and keeps that IP until the next reload. When the name moves, nginx keeps knocking on the old address until the connect timeout runs out. Compare what DNS says now with the IP in the log line.
dig +short backend.internal.example.com
sudo grep "while connecting to upstream" /var/log/nginx/error.log | tail -n 32Fix
Let nginx re-resolve the name. Since nginx 1.27.3 the open-source build supports the resolve parameter in an upstream block; it needs a shared memory zone and a resolver. On older versions, put the name in a variable in proxy_pass and set a resolver. Inside Docker the resolver is 127.0.0.11.
upstream backend {
zone backend 64k;
resolver 10.0.0.2 valid=30s;
server backend.internal.example.com:8080 resolve;
}
server {
location / {
proxy_pass http://backend;
}
}3Verify
Reload once, then confirm requests succeed and that later DNS changes are followed without another reload.
sudo nginx -t && sudo systemctl reload nginx
curl -sI https://example.com | head -n 1Cause 5 of 5
A load balancer in front of nginx gives up first
1Check
Each proxy in the chain has its own clock. An AWS Application Load Balancer, for example, returns 504 when the target stays silent past its idle timeout (60 seconds unless changed). It then drops its connection to nginx, which nginx logs as 499, client closed request.
curl -sI https://example.com/reports/monthly | grep -i '^server'
sudo grep '" 499 ' /var/log/nginx/access.log | tail -n 52Fix
Make each outer timeout a little longer than the one inside it, so the failure comes from the layer that can log a reason, and keep long work out of the request with the queue fix above.
aws elbv2 modify-load-balancer-attributes \
--load-balancer-arn arn:aws:elasticloadbalancing:REGION:ACCOUNT:loadbalancer/app/NAME/ID \
--attributes Key=idle_timeout.timeout_seconds,Value=1203Verify
Repeat the slow request; if it still fails, nginx's error log now holds the reason.
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://example.com/reports/monthlyHow do I stop it coming back?
- Give every outbound call in the app (database, HTTP APIs, SMTP) a timeout shorter than nginx's, so the app fails with a clear error instead of hanging.
- Log $upstream_response_time and alert when the slowest requests on a route pass half of proxy_read_timeout.
- Send anything whose run time grows with data size to a queue from the start.
- Size PHP-FPM or app workers from measured memory per worker, and measure again after big releases.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
- TaxiDexMy own multi-tenant taxi dispatch SaaS: API, dispatcher portal, driver and passenger apps, and the servers underneath.
- WorkforceA field app, an admin console and a self-hosted Node.js server for push and reconciliation.
- stegnor.comMy company site: 804 pages from a Python and Jinja generator, with 15 free browser tools.
Related fixes
Servers
nginx 502 Bad Gateway in front of a Node.js app
connect() failed (111: Connection refused) while connecting to upstream
A 502 means nginx reached for your Node.js app and got nothing usable back.
5 causes, each with a fixServers
nginx and PHP-FPM show File not found or Primary script unknown
File not found.
PHP-FPM received a script path it could not turn into a file.
4 causes, each with a fixServers
Pages load unstyled because /_next/static files return 404 behind nginx
GET https://example.com/_next/static/chunks/main.js 404 (Not Found)
The browser asks for /_next/static/..
3 causes, each with a fixServers
nginx still returns 504 although the PHP script has a long time limit
upstream timed out (110: Connection timed out) while reading response header from upstream
set_time_limit() only controls PHP.
4 causes, each with a fixServers
nginx cannot open the PHP-FPM socket: Permission denied
502 Bad Gateway
nginx runs as one user and PHP-FPM creates its socket for another.
3 causes, each with a fixServers
Cookies from a PHP backend vanish behind an nginx proxy
Login works when you call the backend directly but fails through nginx
nginx forwards Set-Cookie headers from the backend unchanged, so lost cookies usually come from the cookie itself: a domain or path that does not match the public URL, or Secure and SameSite flags the browser rejects.
4 causes, each with a fix