Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

nginx gives up waiting for the app and returns 504

Error504 Gateway Time-out

Short answer

nginx returns 504 when the app behind it takes the request but sends nothing back within proxy_read_timeout (60 seconds by default), or when the connection cannot be opened in time. A bigger timeout only hides it. Find which request is slow and why: slow code, work that belongs in a queue, a full worker pool, an old upstream address, or a load balancer in front.

  • Applies to nginx 1.28, PHP-FPM 8.3, Node.js 22, BullMQ 5, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Pull the timeout lines out of the error log. "while reading response header" means the app accepted the request and went quiet; "while connecting" means nginx never got a connection at all. The request field on each line names the slow URL.

    sudo grep "upstream timed out" /var/log/nginx/error.log | tail -n 20
  2. 2Match what you see, then jump to its fix

    1. 1while reading response header from upstream, always on the same few URLsOne route does slow work before it answersFix
    2. 2Timeouts on exports, imports, report builds, bulk mail or media processing, getting worse as the data growsA long job runs inside the HTTP requestFix
    3. 3server reached pm.max_children setting in the PHP-FPM log, with 504s on pages that are normally fastEvery PHP-FPM worker is busy, so new requests wait in lineFix
    4. 4while connecting to upstream, after a backend server, container or managed service got a new IPnginx still connects to an old upstream addressFix
    5. 5The 504 page is not nginx's, the error log is silent at that moment, and the access log shows 499 for the requestA load balancer in front of nginx gives up firstFix

What does the error look like?

  • 504 Gateway Time-out
  • upstream timed out (110: Connection timed out) while reading response header from upstream
  • upstream timed out (110: Connection timed out) while connecting to upstream
  • WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

Cause 1 of 5

One route does slow work before it answers

1Check

Time the route against the app itself, with nginx out of the way. Then add $upstream_response_time to the access log so every slow route shows its real duration from now on.

curl -o /dev/null -s -w '%{time_total}\n' http://127.0.0.1:3000/reports/monthly

# nginx, http block
log_format timed '$remote_addr "$request" $status $request_time $upstream_response_time';
access_log /var/log/nginx/access.log timed;

2Fix

Fix what is slow: add the missing index, cache the expensive result, or give the outside API call the route waits on its own short timeout. If one route really needs longer, raise the limit for that location only, never for the whole server.

location /reports/ {
    proxy_pass http://127.0.0.1:3000;
    proxy_read_timeout 180s;
}

3Verify

Reload nginx and watch the upstream time for that route in the access log.

sudo nginx -t && sudo systemctl reload nginx
sudo tail -f /var/log/nginx/access.log | grep /reports/
Did this fix it?

Cause 2 of 5

A long job runs inside the HTTP request

1Check

If the run time depends on how much data a user has, no fixed timeout will hold. proxy_read_timeout counts the silence between two reads from the app, so a request that sends nothing for minutes is cut off whatever the total limit is. With the timed log format above, list requests that took over 30 seconds.

sudo awk '$NF > 30' /var/log/nginx/access.log | tail -n 20

2Fix

Move the work to a background queue. The request stores a job and answers 202 with a job id in milliseconds; a worker process does the work, and the page polls a status route or gets notified when the file is ready.

import { Queue } from 'bullmq';

const reports = new Queue('reports', { connection: { host: '127.0.0.1', port: 6379 } });

app.post('/reports', async (req, res) => {
  const job = await reports.add('build', { userId: req.user.id });
  res.status(202).json({ jobId: job.id });
});

3Verify

The route answers at once with 202; the worker's log shows the job being processed.

curl -s -o /dev/null -w '%{http_code} %{time_total}\n' -X POST http://127.0.0.1:3000/reports
Did this fix it?

Cause 3 of 5

Every PHP-FPM worker is busy, so new requests wait in line

1Check

PHP-FPM serves one request per child process. When all children are tied up by slow requests, new ones queue until nginx's fastcgi_read_timeout (also 60 seconds by default) runs out. Count the warnings and measure how much memory one child uses.

sudo grep -c "max_children" /var/log/php8.3-fpm.log
ps -o rss= -C php-fpm8.3 | awk '{s+=$1} END {print s/NR/1024 " MB per child"}'

2Fix

Raise pm.max_children only as far as memory allows (free RAM divided by the size of one child), and turn on the slow log so the scripts that hold workers show up with a backtrace.

; /etc/php/8.3/fpm/pool.d/www.conf
pm = dynamic
pm.max_children = 20
pm.start_servers = 5
pm.min_spare_servers = 3
pm.max_spare_servers = 8
request_slowlog_timeout = 10s
slowlog = /var/log/php8.3-fpm.slow.log

3Verify

Restart PHP-FPM and check that the warning stays away under normal traffic.

sudo systemctl restart php8.3-fpm
sudo tail -f /var/log/php8.3-fpm.log
Did this fix it?

Cause 4 of 5

nginx still connects to an old upstream address

1Check

nginx resolves a host name in proxy_pass once, when it loads its config, and keeps that IP until the next reload. When the name moves, nginx keeps knocking on the old address until the connect timeout runs out. Compare what DNS says now with the IP in the log line.

dig +short backend.internal.example.com
sudo grep "while connecting to upstream" /var/log/nginx/error.log | tail -n 3

2Fix

Let nginx re-resolve the name. Since nginx 1.27.3 the open-source build supports the resolve parameter in an upstream block; it needs a shared memory zone and a resolver. On older versions, put the name in a variable in proxy_pass and set a resolver. Inside Docker the resolver is 127.0.0.11.

upstream backend {
    zone backend 64k;
    resolver 10.0.0.2 valid=30s;
    server backend.internal.example.com:8080 resolve;
}

server {
    location / {
        proxy_pass http://backend;
    }
}

3Verify

Reload once, then confirm requests succeed and that later DNS changes are followed without another reload.

sudo nginx -t && sudo systemctl reload nginx
curl -sI https://example.com | head -n 1
Did this fix it?

Cause 5 of 5

A load balancer in front of nginx gives up first

1Check

Each proxy in the chain has its own clock. An AWS Application Load Balancer, for example, returns 504 when the target stays silent past its idle timeout (60 seconds unless changed). It then drops its connection to nginx, which nginx logs as 499, client closed request.

curl -sI https://example.com/reports/monthly | grep -i '^server'
sudo grep '" 499 ' /var/log/nginx/access.log | tail -n 5

2Fix

Make each outer timeout a little longer than the one inside it, so the failure comes from the layer that can log a reason, and keep long work out of the request with the queue fix above.

aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn arn:aws:elasticloadbalancing:REGION:ACCOUNT:loadbalancer/app/NAME/ID \
  --attributes Key=idle_timeout.timeout_seconds,Value=120

3Verify

Repeat the slow request; if it still fails, nginx's error log now holds the reason.

curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://example.com/reports/monthly
Did this fix it?

How do I stop it coming back?

  • Give every outbound call in the app (database, HTTP APIs, SMTP) a timeout shorter than nginx's, so the app fails with a clear error instead of hanging.
  • Log $upstream_response_time and alert when the slowest requests on a route pass half of proxy_read_timeout.
  • Send anything whose run time grows with data size to a queue from the start.
  • Size PHP-FPM or app workers from measured memory per worker, and measure again after big releases.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Cookies from a PHP backend vanish behind an nginx proxy

Login works when you call the backend directly but fails through nginx

nginx forwards Set-Cookie headers from the backend unchanged, so lost cookies usually come from the cookie itself: a domain or path that does not match the public URL, or Secure and SameSite flags the browser rejects.

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.