Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Node.js, PHP and APIs

Kong Gateway returns 502 'An invalid response was received from the upstream server'

ErrorAn invalid response was received from the upstream server

Short answer

Kong returns 502 when it cannot get a valid HTTP response from the upstream. The Kong error log names the reason. The usual four are: the Service host, port or protocol is wrong, upstream TLS verification fails, the response headers exceed the proxy buffer, or the upstream repeats Content-Length or Transfer-Encoding.

  • Applies to Kong Gateway 3.x, Docker, Node.js 22 with Express 5
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the Kong error log at the moment of the failing request. The line after the 502 tells which cause applies. For a Docker install, read the container log.

    docker logs --tail 50 kong 2>&1 | grep -i upstream
  2. 2Match what you see, then jump to its fix

    1. 1502 comes with a connection failure in the log, or only for this one route. The Express app runs in another container or on another portThe Service points at the wrong host, port or protocolFix
    2. 2upstream SSL certificate verify error, for example (22:certificate chain too long), in the Kong logUpstream TLS verification failsFix
    3. 3upstream sent too big header while reading response header from upstreamThe upstream response headers are bigger than the proxy bufferFix
    4. 4502 only after upgrading to Kong 3.6 or later; the same upstream worked beforeThe upstream sends duplicate Content-Length or Transfer-Encoding headersFix

What does the error look like?

  • An invalid response was received from the upstream server
  • upstream sent too big header while reading response header from upstream
  • upstream SSL certificate verify error: (22:certificate chain too long)

Cause 1 of 4

The Service points at the wrong host, port or protocol

1Check

The Kong Service page documents host, port (default 80) and protocol (default http). Check that these three values match where the Express app really listens. A TLS upstream needs the https protocol and the matching port. Kong returns 502 when it cannot open the connection.

curl -i http://127.0.0.1:3000/health
docker logs --tail 20 kong 2>&1 | grep -i "connect"

2Fix

Set the Service to the name and port the app really listens on, from inside Kong's network. If the app speaks TLS, use https and its TLS port.

services:
  - name: api
    protocol: http
    host: api        # the name Kong can resolve for your app
    port: 3000

3Verify

Send a request through the Kong proxy port.

curl -i http://localhost:8000/api/health

Confirmed by developer.konghq.com

Did this fix it?

Cause 2 of 4

Upstream TLS verification fails

1Check

Kong's upstream TLS guide shows the exact 'An invalid response was received from the upstream server' body when the certificate chain is longer than the allowed verify depth. A leaf plus an intermediate needs depth 1.

kubectl logs -n kong <kong-pod> | grep -i 'certificate verify'

2Fix

On Kong Ingress Controller, raise the verify depth on the Service (or the BackendTLSPolicy) and make sure the CA certificate is configured. Kong keeps upstream connections alive for 60 seconds, so restart the pod to apply it at once.

kubectl annotate -n kong --overwrite service echo konghq.com/tls-verify-depth=1

3Verify

Repeat the request after the restart.

curl -i http://localhost:8000/echo

Confirmed by developer.konghq.com

Did this fix it?

Cause 3 of 4

The upstream response headers are bigger than the proxy buffer

1Check

Kong's large-payload guide says that if the upstream response headers exceed nginx_http_proxy_buffer_size, Kong returns 502 and logs this line. Big JWT cookies or many Set-Cookie headers from Express cause it.

curl -s -D - -o /dev/null http://api:3000/login | wc -c

2Fix

Raise nginx_http_proxy_buffer_size. Kong injects any nginx_http_<directive> setting into its http block, and the KONG_ environment variable form works in Docker.

# kong.conf
nginx_http_proxy_buffer_size = 16k

# or in Docker
KONG_NGINX_HTTP_PROXY_BUFFER_SIZE=16k

3Verify

Restart Kong (docker restart kong) and request the route that failed.

docker restart kong

Confirmed by developer.konghq.com

Did this fix it?

Cause 4 of 4

The upstream sends duplicate Content-Length or Transfer-Encoding headers

1Check

Kong's breaking-changes page says that since the OpenResty update in 3.6.0.0, nginx treats upstream responses with duplicate Content-Length, duplicate Transfer-Encoding, or both, as invalid and returns 502.

curl -s -D - -o /dev/null http://api:3000/data | grep -i -E 'content-length|transfer-encoding'

2Fix

Fix the upstream so it sends each header once. In Express, do not set Content-Length by hand on a response that Express or a compression middleware also sizes.

// remove manual header lines like this one
// res.setHeader('Content-Length', body.length);
res.json(data);

3Verify

The header check above prints each header at most once.

curl -s -D - -o /dev/null http://api:3000/data | grep -ic 'content-length'

Confirmed by developer.konghq.com

Did this fix it?

How do I stop it coming back?

  • Check every Service host, port and protocol against where the app really listens before adding the route.
  • Add a /health route to the upstream and call it through Kong in your monitoring.
  • Keep response cookies small; a large token belongs in a header or a server-side session.
  • Read the breaking-changes page before every Kong upgrade.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.