Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages

Help for owners

My load balancer says my servers are unhealthy and the site gives errors

Short answer

A load balancer sends each server a small test request on a schedule and only sends visitors to servers that pass. Servers fail it because the test address does not exist, the answer is the wrong code, a firewall blocks the test, or the server is overloaded. Read the failure reason code before changing anything.

  • Written for business owners
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Does this sound like your situation?

  • The cloud console shows my servers as unhealthy, but the application seems to run.
  • Visitors see 502 or 503 errors, and it comes and goes.
  • A new server was added, but it never starts receiving traffic.
  • The site broke right after a deployment or a security change.
  • Servers keep being replaced automatically and nobody knows why.

Why does this happen?

  1. The health check asks for something the application does not answer

    AWS describes the check as an HTTP request to a path and port that you set. By default the path is / and only the response code 200 counts as success. If the application redirects, asks for a login, or has no such page, every check fails even though the site works for people.

  2. A firewall rule or network rule blocks the check

    AWS lists security groups and network access control lists that do not allow traffic from the load balancer on the health check port as a cause of a target not entering service.

  3. The server answers too slowly

    AWS says a target that does not respond within the health check timeout fails the check, and that an over-utilised instance can be too busy to respond. A busy server then loses its place, which pushes more load onto the others.

  4. The application needs a host name the check does not send

    AWS notes that some applications need a virtual host configuration to answer the health check, because the check sends the server's private IP address as the host header rather than your domain name.

  5. Errors in the web server in front of the application

    When the 502 comes from the web server rather than the load balancer, the application behind it is down or unreachable. The technical detail is on the fix page.

What can I do about it today?

  1. Do not delete or restart serversRemoving servers or the load balancer can turn a partial problem into a full outage. Leave the setup as it is until the reason is known.
  2. Write down what the console saysOpen the target group and note, for each server, the status and the reason shown next to it, for example a response code mismatch or a timeout. Send a screenshot to whoever manages the servers.
  3. Note what changed lastA deployment, a security rule, a new certificate, a server update. The failure usually starts right after one of them.
  4. Check the site as a visitorOpen your site on a phone using mobile data and tell your developer whether it fails always or only sometimes. Intermittent failure points to one unhealthy server among several.
  5. Keep access with the right personMake sure someone who owns the cloud account can give access to the person fixing it, rather than sharing a password.

When should I bring someone in?

  • All servers show unhealthy at once and the site is down.
  • Nobody can say who changed the health check or the security rules.
  • The problem repeats after each deployment.

How do I handle it?

  1. Read the reason codes firstThe target group tells which check failed: wrong code, timeout, or no answer. That decides where to look.
  2. Call the health check path myselfI request the same path from inside the network and compare the answer with what the load balancer expects.
  3. Fix the cause, not the symptomEither the application gets a simple health page that returns the expected code, or the check settings are adjusted, with the network rules corrected where they block it.
  4. Prove it under a deploymentA fix is finished when a deployment with a rollback leaves the servers healthy throughout.

The price depends on what I find, so I quote in writing after a short discovery call. Tell me what is happening and I reply within 24 hours.

What do owners ask about this?

If all servers are unhealthy, does the load balancer stop sending traffic?

AWS states that if a target group contains only unhealthy targets, the load balancer sends requests to all of them regardless. So the site may still partly answer, which makes the problem harder to notice.

How quickly is a server marked unhealthy?

It depends on the settings. AWS gives defaults of a check every 30 seconds and two consecutive failures to mark a server unhealthy, but these can be changed on your target group.

Is a health check failing the same as the site being down?

No. A site can work for visitors while the check fails, for example when the check asks for a page that does not exist.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Found a mistake? Tell me.

For your developer: the technical fixes

Next step

Want this off your desk?

Send me a few lines on where things stand. I reply within 24 hours.