Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

Gunicorn workers are killed with SIGKILL inside a container

ErrorWorker was sent SIGKILL

Short answer

A SIGKILL cannot be caught, so the worker never logs why. The first suspect is the kernel out-of-memory killer, because a container has a memory limit. The second is the heartbeat temp file on a slow or full disk. Look at the kernel log first, then raise the limit, or move the heartbeat to RAM.

  • Applies to Gunicorn, Docker Engine 28, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Look for the kernel's out-of-memory message about gunicorn on the Docker host.

    dmesg | grep gunicorn
  2. 2Match what you see, then jump to its fix

    1. 1dmesg shows Memory cgroup out of memory and Killed process (gunicorn)The container hit its memory limit and the kernel killed the workerFix
    2. 2workers die with no out-of-memory line in dmesg, under disk stallsThe worker heartbeat file sits on a slow or full diskFix

What does the error look like?

  • Worker was sent SIGKILL
  • [INFO] Booting worker with pid

Cause 1 of 2

The container hit its memory limit and the kernel killed the worker

1Check

The container may have a --memory limit lower than the workers need together.

dmesg | grep -i "out of memory"
docker stats --no-stream

2Fix

Give the container more memory, or run fewer workers so the total fits. Do not remove the limit by disabling the OOM killer; Docker warns against that.

docker run -m 1g my-image gunicorn -w 2 app:app

3Verify

Run a load against the app and see that the master stops logging new workers.

docker logs --since 10m my-container | grep -c "Booting worker"

Confirmed by gunicorn.org

Did this fix it?

Cause 2 of 2

The worker heartbeat file sits on a slow or full disk

1Check

Gunicorn touches a temp file as a heartbeat. On a disk-backed temp directory the call can block when I/O stalls or the disk is full. Check whether the temp directory is RAM-backed.

docker exec my-container df /tmp

2Fix

Put the heartbeat in a RAM-backed directory. In Docker, a tmpfs mount gives you one. Mount it and point gunicorn at it.

docker run --tmpfs /mem:rw,size=64m my-image gunicorn --worker-tmp-dir /mem app:app

3Verify

The worker stays up through the load that used to kill it.

docker logs --since 10m my-container

Confirmed by gunicorn.org

Did this fix it?

How do I stop it coming back?

  • Alert on container restarts and on kernel out-of-memory lines.
  • Keep the worker heartbeat directory on tmpfs in every container.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.