Gunicorn workers are killed with SIGKILL inside a container
Worker was sent SIGKILLShort answer
A SIGKILL cannot be caught, so the worker never logs why. The first suspect is the kernel out-of-memory killer, because a container has a memory limit. The second is the heartbeat temp file on a slow or full disk. Look at the kernel log first, then raise the limit, or move the heartbeat to RAM.
Fix it fast
1Find out which cause you have
Look for the kernel's out-of-memory message about gunicorn on the Docker host.
dmesg | grep gunicorn2Match what you see, then jump to its fix
What does the error look like?
Worker was sent SIGKILL[INFO] Booting worker with pid
Cause 1 of 2
The container hit its memory limit and the kernel killed the worker
1Check
The container may have a --memory limit lower than the workers need together.
dmesg | grep -i "out of memory"
docker stats --no-stream2Fix
Give the container more memory, or run fewer workers so the total fits. Do not remove the limit by disabling the OOM killer; Docker warns against that.
docker run -m 1g my-image gunicorn -w 2 app:app3Verify
Run a load against the app and see that the master stops logging new workers.
docker logs --since 10m my-container | grep -c "Booting worker"Confirmed by gunicorn.org
Cause 2 of 2
The worker heartbeat file sits on a slow or full disk
1Check
Gunicorn touches a temp file as a heartbeat. On a disk-backed temp directory the call can block when I/O stalls or the disk is full. Check whether the temp directory is RAM-backed.
docker exec my-container df /tmp2Fix
Put the heartbeat in a RAM-backed directory. In Docker, a tmpfs mount gives you one. Mount it and point gunicorn at it.
docker run --tmpfs /mem:rw,size=64m my-image gunicorn --worker-tmp-dir /mem app:app3Verify
The worker stays up through the load that used to kill it.
docker logs --since 10m my-containerConfirmed by gunicorn.org
How do I stop it coming back?
- Alert on container restarts and on kernel out-of-memory lines.
- Keep the worker heartbeat directory on tmpfs in every container.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
Related fixes
Servers
nginx shows 502 Bad Gateway in front of a Django app on Gunicorn
(13: Permission denied) while connecting to upstream
A 502 means nginx could not get an answer from Gunicorn.
4 causes, each with a fixServers
Apache logs SIGWINCH and exits on its own
the httpd container stops although nothing crashed
SIGWINCH is Apache's graceful-stop signal.
2 causes, each with a fixServers
sam local fails: Running AWS SAM projects locally requires Docker
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
sam local start-api and sam local invoke run your function inside a Docker container, so SAM stops if it cannot talk to Docker.
3 causes, each with a fixServers
dnf fails to download repo metadata in a CentOS 8 Docker build
The same Dockerfile built fine before 2022
CentOS Linux 8 reached end of life on 31 December 2021 and its content left the mirrors.
2 causes, each with a fixServers
pip install confluent-kafka fails because librdkafka/rdkafka.h is missing
pip stops while building confluent-kafka from source
The error means pip is compiling confluent-kafka from source, and the compiler cannot find the librdkafka C headers.
3 causes, each with a fixServers
Cypress says the npm package is installed but the binary is missing
Cypress binary is missing after npm install in CI or in a Docker build
The npm package is only a launcher.
3 causes, each with a fix