Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Node.js, PHP and APIs

API Gateway gives up on a slow Lambda or Node.js backend

ErrorThe request fails after roughly 29 seconds on a REST API (30 seconds on an HTTP API)

Short answer

API Gateway stops waiting for the backend at its integration timeout: 29 seconds by default on REST APIs, a fixed 30 seconds on HTTP APIs. Find which limit you hit. Then raise the REST limit (Regional or private APIs only), lift the Lambda timeout if it is the shorter one, or answer fast and finish the work asynchronously.

  • Applies to Amazon API Gateway REST and HTTP APIs, AWS Lambda Node.js 22, AWS CLI v2
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the function's configured timeout, then compare it with the time the work really takes and with the API type (REST or HTTP). The three numbers tell the causes apart.

    aws lambda get-function-configuration --function-name my-function --query Timeout
  2. 2Match what you see, then jump to its fix

    1. 1Requests fail at about 29 s (REST) or 30 s (HTTP API) no matter how high the Lambda timeout isThe work really takes longer than the API type allowsFix
    2. 2Only a REST API is affected, and its type is Regional or privateA Regional or private REST API still has the default 29 second timeoutFix
    3. 3Requests fail after a few seconds, long before 29 sThe Lambda timeout is shorter than the workFix
    4. 4You raised the timeout in the console or CLI but the old behaviour continuesA change to the integration was never deployedFix

What does the error look like?

  • The request fails after roughly 29 seconds on a REST API (30 seconds on an HTTP API)
  • $context.integrationErrorMessage in the access log shows the integration error

Cause 1 of 4

The work really takes longer than the API type allows

1Check

Time the slow operation in the function (a report build, a big export, a call to a slow third party). If it needs more than the API limit, no setting on a HTTP API or an edge-optimized REST API can help, because those limits cannot be raised.

console.time('work');
await doSlowThing();
console.timeEnd('work');

2Fix

Stop making the caller wait. For a REST API with a Lambda non-proxy integration, add the header X-Amz-Invocation-Type with the static value 'Event' in the integration request so Lambda runs the function asynchronously. In any API type you can also answer quickly with a job id and let the client poll for the result.

# OpenAPI: x-amazon-apigateway-integration
"requestParameters" : {
  "integration.request.header.X-Amz-Invocation-Type" : "'Event'"
}

3Verify

Call the route again. It should answer in well under the limit while the function keeps running in the background.

curl -s -o /dev/null -w '%{time_total}\n' https://abc123.execute-api.us-east-1.amazonaws.com/prod/report

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 4

A Regional or private REST API still has the default 29 second timeout

1Check

The quota table lists an integration timeout of 50 ms to 29 s, with a raise possible for Regional and private APIs only; edge-optimized APIs and HTTP APIs are marked as not raisable. Read the current value on the integration.

aws apigateway get-integration \
  --rest-api-id abc123 --resource-id a1b2c3 --http-method POST \
  --query timeoutInMillis

2Fix

Request the integration timeout quota increase in Service Quotas (AWS notes it may mean a lower account throttle quota), then set a larger timeoutInMillis on the integration and deploy the API again.

aws apigateway update-integration \
  --rest-api-id abc123 --resource-id a1b2c3 --http-method POST \
  --patch-operations op=replace,path=/timeoutInMillis,value=60000
aws apigateway create-deployment --rest-api-id abc123 --stage-name prod

3Verify

Read the value back and repeat the slow request after the deployment.

aws apigateway get-integration \
  --rest-api-id abc123 --resource-id a1b2c3 --http-method POST \
  --query timeoutInMillis

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 3 of 4

The Lambda timeout is shorter than the work

1Check

A new Lambda function has a 3 second timeout unless you changed it. The first step prints the value.

aws lambda get-function-configuration --function-name my-function --query Timeout

2Fix

Raise the function timeout, but keep it at or below the API limit if a person is waiting for the answer. Lambda accepts 1 to 900 seconds.

aws lambda update-function-configuration --function-name my-function --timeout 25

3Verify

Read the value again and repeat the request.

aws lambda get-function-configuration --function-name my-function --query Timeout

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 4 of 4

A change to the integration was never deployed

1Check

On REST APIs, edits to integrations only reach callers after a deployment to a stage.

aws apigateway get-integration --rest-api-id abc123 --resource-id a1b2c3 --http-method POST --query timeoutInMillis

2Fix

Deploy the API again to the stage your clients call.

aws apigateway create-deployment --rest-api-id abc123 --stage-name prod

3Verify

Repeat the slow request against that stage URL.

curl -i https://abc123.execute-api.us-east-1.amazonaws.com/prod/report

Confirmed by docs.aws.amazon.com

Did this fix it?

How do I stop it coming back?

  • Treat 29 seconds as a hard design limit: anything that can take longer returns a job id and finishes in the background.
  • Set the Lambda timeout deliberately for each function instead of leaving the 3 second default.
  • Add $context.integrationErrorMessage to the API access log so integration failures name themselves.
  • Put a timeout on every call your handler makes to another service, shorter than the API limit.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Back end

cdk synth says --app is required

--app is required either in command-line, in cdk.json or in ~/.cdk.json

The CDK CLI needs a command that runs your app and finds it in three places: the --app option, cdk.json in the project folder, or cdk.json in your home folder.

3 causes, each with a fix

Back end

The AWS SDK in Node.js cannot find any credentials

Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1

The AWS SDK walks a fixed credential chain: code, environment variables, the shared files in ~/.aws, then role sources such as ECS or EC2.

3 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.