Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

HAProxy answers 503 "No server is available to handle this request"

Error503 Service Unavailable

Short answer

HAProxy sends 503 when it has no server to give a request to. Ask the stats socket for the server status first. Servers marked DOWN usually fail their health check; a frontend with no matching backend, a server put into maintenance, and a full queue that times out are the other documented causes.

  • Applies to HAProxy 3.0, nginx 1.26, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Ask the running HAProxy for the state of every server. The socket path comes from the stats socket line in the global section.

    echo "show servers state" | sudo socat /var/run/haproxy.sock stdio
  2. 2Match what you see, then jump to its fix

    1. 1status DOWN for each server in the stats outputEvery server in the backend is DOWN because its health check failsFix
    2. 2the frontend has use_backend rules and no default_backend; servers in the stats are healthyNo backend was selected for the requestFix
    3. 3status MAINT in the stats output while the application itself is healthyThe server was put into maintenance by hand or by a scriptFix
    4. 4503s during bursts only, with servers UPRequests waited in the queue longer than timeout queueFix

What does the error look like?

  • 503 Service Unavailable
  • No server is available to handle this request.

Cause 1 of 4

Every server in the backend is DOWN because its health check fails

1Check

With check alone HAProxy only opens a TCP connection. With option httpchk it sends a real request, by default OPTIONS / over HTTP/1.0, and only 2xx or 3xx answers count as healthy. A backend that rejects OPTIONS, wants a Host header, or listens on another port fails the check. Test the same request by hand.

curl -i -X OPTIONS http://10.0.0.11:8080/
grep -n -E "httpchk|server " /etc/haproxy/haproxy.cfg

2Fix

Point the check at a URL that returns 200 and send the Host header the backend expects. Use the server's real port in the server line.

backend app
    option httpchk GET /health HTTP/1.1
    http-check send hdr Host app.example.com
    server web1 10.0.0.11:8080 check
    server web2 10.0.0.12:8080 check

3Verify

Validate the file, reload, and read the server status again.

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
echo "show servers state" | sudo socat /var/run/haproxy.sock stdio

Confirmed by docs.haproxy.org

Did this fix it?

Cause 2 of 4

No backend was selected for the request

1Check

When no use_backend condition matches and the frontend has no default_backend, HAProxy uses no server and returns 503.

grep -n -E "^(frontend|backend|listen)|use_backend|default_backend" /etc/haproxy/haproxy.cfg

2Fix

Add a default_backend that catches everything the rules do not, or correct the ACL so the request matches a rule.

frontend www
    bind :80
    use_backend api if { path_beg /api }
    default_backend app

3Verify

Validate, reload and request a path that no rule matches.

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
curl -i http://127.0.0.1/

Confirmed by docs.haproxy.org

Did this fix it?

Cause 3 of 4

The server was put into maintenance by hand or by a script

1Check

disable server on the runtime socket marks a server DOWN for maintenance and stops its checks until it leaves that state.

echo "show stat" | sudo socat /var/run/haproxy.sock stdio | cut -d, -f1,2,18 | column -s, -t

2Fix

Re-enable the server on an admin-level socket; this also turns its checks back on.

echo "enable server app/web1" | sudo socat /var/run/haproxy.sock stdio

3Verify

The status column changes from MAINT to UP once a check passes.

echo "show servers state app" | sudo socat /var/run/haproxy.sock stdio

Confirmed by docs.haproxy.org

Did this fix it?

Cause 4 of 4

Requests waited in the queue longer than timeout queue

1Check

When a server's maxconn is reached, extra requests wait in a queue. Past timeout queue the request is dropped and a 503 is returned. If timeout queue is unset, the connect timeout is used.

grep -n -E "maxconn|timeout" /etc/haproxy/haproxy.cfg

2Fix

Raise the server's maxconn if the backend can take more, or lengthen timeout queue to what clients can accept. Adding servers is the real fix for sustained load.

backend app
    timeout queue 30s
    server web1 10.0.0.11:8080 check maxconn 200

3Verify

Validate and reload, then repeat the burst that failed.

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

Confirmed by docs.haproxy.org

Did this fix it?

How do I stop it coming back?

  • Give each backend a health-check URL that checks the application, and send the Host header it needs.
  • Always set default_backend on a frontend that uses use_backend rules.
  • Alert on servers leaving UP, so an empty backend is found before users see 503.
  • Set maxconn per server to what it can really carry, and size timeout queue on purpose.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.