A Gunicorn worker timeout means the master process stopped hearing from a worker and killed it. The request that worker held dies with it, and your proxy returns a 502. At peak it happens to several workers at once, and the ones left take the whole queue.
What breaks at peak
Gunicorn runs one master and a fixed number of worker processes. Each worker updates a heartbeat file. If a worker stays silent for longer than timeout, 30 seconds by default, the master logs this line and sends it SIGABRT:
[2026-11-20 14:02:11 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:42)
[2026-11-20 14:02:11 +0000] [1] [ERROR] Worker (pid:42) exited with code 1
A worker stuck inside C code, such as a driver call with no timeout, may ignore the abort. The master then sends SIGKILL on its next check. The kernel OOM killer also uses SIGKILL, and both cases log this line. Gunicorn ends it with a guess that the worker ran out of memory:
[ERROR] Worker (pid:42) was sent SIGKILL
Three things cause most of these at peak.
Slow dependencies. With the default sync worker, one worker serves one request at a time. A database query or an outbound API call that hangs for 30 seconds holds that worker for 30 seconds. Once every worker waits on the same slow dependency, new requests queue in the listen backlog until the proxy gives up.
Too few workers for the CPU you pay for. workers defaults to 1, or to WEB_CONCURRENCY if your platform sets it. A pod with 4 CPUs and one sync worker serves one request at a time.
Too many workers for the memory you have. Each worker is a full copy of your app. Workers grow as they cache data and leak objects. At peak, the container hits its memory limit and the kernel kills the largest worker, which gives you the SIGKILL line above.
Gunicorn workers and threads for peak traffic
Start from a config file instead of command-line flags, so the values live in version control. This gunicorn.conf.py serves a Django or Flask app with threaded workers:
# gunicorn.conf.py
import os
cpu_limit = int(os.environ.get("CPU_LIMIT", "2"))
bind = "0.0.0.0:8000"
worker_class = "gthread"
workers = int(os.environ.get("WEB_CONCURRENCY", cpu_limit * 2 + 1))
threads = int(os.environ.get("GUNICORN_THREADS", "4"))
timeout = 30
graceful_timeout = 25
keepalive = 75
max_requests = 2000
max_requests_jitter = 200
worker_tmp_dir = "/dev/shm"
backlog = 2048
statsd_host = "localhost:9125"
def worker_abort(worker):
# Runs in the worker on SIGABRT, before it exits. Logs where every thread was stuck.
import sys
import traceback
for thread_id, frame in sys._current_frames().items():
stack = "".join(traceback.format_stack(frame))
worker.log.warning("Thread %s stack at timeout:\n%s", thread_id, stack)
Start it with gunicorn -c gunicorn.conf.py myproject.wsgi:application for Django, or with your Flask app path as shown further down.
worker_class and threads. Setting threads above 1 switches the sync worker to gthread, and I name it in the file anyway. Threads share one process’s memory, so four threads cost far less than four workers. They help when requests wait on I/O. Pure Python CPU work holds the GIL, and threads won’t run it in parallel.
workers. Gunicorn’s docs suggest (2 × CPU cores) + 1 as a start. Count cores from the container’s CPU limit. os.cpu_count() returns the host’s cores, so a 2-CPU pod on a 64-core node would start 129 workers. Pass the limit in with the downward API:
env:
- name: CPU_LIMIT
valueFrom:
resourceFieldRef:
resource: limits.cpu
divisor: "1"
With divisor: "1" Kubernetes rounds a fractional limit up, so 1500m reads as 2. For CPU-heavy endpoints, such as report rendering, I drop to one worker per CPU and one or two threads. For I/O-heavy APIs, I keep the formula and raise threads to 4 or 8.
Memory sets the ceiling. Measure one worker’s resident memory after it has served traffic for a while:
ps -o pid,rss,args -C gunicorn
The rule is workers × peak RSS per worker + master + headroom below the container’s memory limit. Leave a quarter of the limit as headroom, because workers grow between restarts. If the CPU formula gives more workers than memory allows, use fewer workers with more threads, or bigger pods.
max_requests and max_requests_jitter. A worker restarts after serving max_requests requests, which caps slow leaks. The default of 0 never restarts. The jitter spreads restarts so all workers don’t recycle in the same second.
timeout. Treat this as the last line of defence. Set timeouts on everything a request waits for first: a database statement_timeout, and connect and read timeouts on every HTTP client. Those fail one request with a clear error. A worker kill drops the request with no traceback, unless the worker_abort hook above logs one. I set outbound timeouts before I touch --timeout, and I raise it only when a known endpoint needs longer.
Order the limits from the inside out. Outbound timeouts sit below timeout, and timeout sits a few seconds below the proxy’s read timeout. If the proxy gives up first, the worker keeps working on a response nobody will read. On gthread workers the main thread sends the heartbeat, so a slow request in a thread won’t trigger the timeout. There, the proxy timeout is the only cap on request time.
graceful_timeout. On SIGTERM, workers get this long to finish in-flight requests. Keep it below Kubernetes’ terminationGracePeriodSeconds, which defaults to 30.
keepalive. The sync worker ignores this setting. On gthread, if a load balancer connects straight to Gunicorn, set keepalive above the balancer’s idle timeout. Otherwise the balancer reuses a socket Gunicorn already closed and returns a 502. An AWS ALB idles out at 60 seconds by default, as the ALB 502 and 504 recipe explains. The config above assumes a balancer in front. Behind nginx on the same host, the default of 2 seconds works.
worker_tmp_dir. The heartbeat file lives here. On a disk-backed /tmp, a slow disk can stall the heartbeat and kill healthy workers. /dev/shm is RAM-backed in most containers.
backlog. The kernel’s queue of connections waiting for a worker to accept them. The kernel caps it at net.core.somaxconn. Linux network tuning for web servers covers raising that.
Database connections multiply. Every thread in every worker can hold its own database connection. Total connections are pods × workers × threads. Django keeps them per thread, and the Django database connections recipe sizes them.
Flask. Flask’s app.run() and flask run start Werkzeug’s development server, which prints WARNING: This is a development server. Do not use it in a production deployment. In production, run the same config with your app factory: gunicorn -c gunicorn.conf.py "myapp:create_app()". Gunicorn calls the factory once per worker. Flask-SQLAlchemy builds one SQLAlchemy engine per worker, and SQLAlchemy’s default pool holds 5 connections plus 10 overflow. Under gthread, set SQLALCHEMY_ENGINE_OPTIONS = {"pool_size": threads, "max_overflow": 0}, so each thread gets a connection without overflow you didn’t budget for.
preload_app. preload_app = True imports your app once in the master, before the fork. Workers then share its memory pages until they write to them, and a broken import fails at startup instead of in every worker. Don’t open database connections or start threads at import time when you preload, because the forked workers would share those sockets.
Managed platforms. Heroku, Render and similar platforms set WEB_CONCURRENCY from the dyno or instance size, and the config above picks that up. AWS App Runner, Azure App Service and Cloud Run run your own start command, so pass the config file there. Cloud Run caps concurrent requests per instance with its own concurrency setting. Set it near workers × threads, so it doesn’t queue more requests than Gunicorn can serve.
Test it on your system
Run the test in staging with the same CPU and memory limits as production. This k6 script sends a fixed arrival rate, so a slow server doesn’t slow the test down with it:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: Number(__ENV.RATE || 200), // your peak requests per second
timeUnit: '1s',
duration: '15m',
preAllocatedVUs: 200,
maxVUs: 2000,
},
},
thresholds: {
http_req_failed: ['rate<0.001'], // your error budget
http_req_duration: ['p(99)<800'], // your latency target in ms
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/api/orders?limit=20`);
check(res, { 'status is 200': (r) => r.status === 200 });
}
Run k6 run -e BASE_URL=https://staging.example.com -e RATE=300 gunicorn-peak.js. Point it at your heaviest real endpoints, through the same proxy as production.
While it runs, watch three places:
- Gunicorn’s log, for
WORKER TIMEOUT,was sent SIGKILLand the stacks fromworker_abort. - The listen queue. Run
ss -ltn 'sport = :8000'inside the pod.Recv-Qcounts connections waiting for a worker. It should drain to 0 between bursts. - Container CPU and memory, with
kubectl top pod. CPU pinned at the limit withRecv-Qgrowing means you need more pods. Memory close to the limit means fewer workers per pod.
Halfway through, slow one dependency on purpose. Add latency to a staging database with pg_sleep in a test endpoint, or point an outbound call at a slow mock. A good config fails those requests at your outbound timeout and keeps serving the rest.
Set the bar before you start. A pass means no WORKER TIMEOUT or SIGKILL lines at your peak rate. Recv-Q drains, memory stays under your chosen share of the limit, and the k6 thresholds hold.
Keep it healthy
Gunicorn sends StatsD metrics when you set statsd_host. Run statsd_exporter as a sidecar listening on port 9125, with this mapping file:
mappings:
- match: "gunicorn.log.critical"
name: "gunicorn_log_critical_total"
- match: "gunicorn.workers"
name: "gunicorn_workers"
- match: "gunicorn.request.status.*"
name: "gunicorn_responses_total"
labels:
status: "$1"
The master logs WORKER TIMEOUT at critical level, so gunicorn_log_critical_total counts timeouts. gunicorn_workers reports the live worker count when it changes. On Gunicorn 24.1 or later, --enable-backlog-metric also sends gunicorn.backlog, the listen queue depth, as a StatsD histogram.
Pair it with cAdvisor’s container_oom_events_total for the OOM kills:
groups:
- name: gunicorn-workers
rules:
- alert: GunicornWorkerTimeout
expr: increase(gunicorn_log_critical_total[5m]) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Gunicorn on {{ $labels.instance }} killed a worker for timing out"
description: "Read the worker_abort stack in the log to find the stuck call. Add a timeout to that call before you raise Gunicorn's timeout."
- alert: GunicornWorkerTimeoutsRepeating
expr: increase(gunicorn_log_critical_total[10m]) > 5
for: 5m
labels:
severity: critical
annotations:
summary: "Gunicorn on {{ $labels.instance }} keeps killing workers"
description: "Requests are failing with 502s. Check the slowest dependency first, then CPU throttling and Recv-Q on the pod. Add pods if CPU sits at the limit."
- alert: GunicornContainerOOM
expr: increase(container_oom_events_total{container="web"}[10m]) > 0
for: 0m
labels:
severity: critical
annotations:
summary: "A Gunicorn worker in {{ $labels.pod }} was OOM-killed"
description: "Check peak RSS per worker with ps. Lower workers or max_requests, or raise the memory limit so workers × RSS fits with headroom."
Replace web with your container name. The OOM metric needs cAdvisor’s oom_event metrics. If yours doesn’t export it, node_exporter’s node_vmstat_oom_kill counts kills per node. Set the repeat threshold from your own baseline, since a busy app with a known slow endpoint may tolerate one timeout an hour.
Roll back
Keep the old config in Git, and copy the running values before you deploy:
kubectl exec deploy/web -- cat /app/gunicorn.conf.py > gunicorn.conf.py.before
kubectl get deploy web -o yaml > web-deployment-before.yaml
On Kubernetes, kubectl rollout undo deployment/web returns to the previous ReplicaSet. On a VM, put the old file back and send SIGHUP to the master. Gunicorn reloads the config, starts new workers and stops the old ones once they finish.
If the new worker count causes OOM kills or database connection errors, roll back now. Those failures grow with traffic. If the change only failed to help, revert in your next quiet window.
More recipes
- Django CONN_MAX_AGE, Connection Pooling and PgBouncer
Match CONN_MAX_AGE or Django's pool to your worker count, and add Django PgBouncer settings when pods multiply, so peak traffic stops exhausting Postgres.
- Fix FastAPI Blocking Calls in Async Routes
Keep blocking calls off the event loop with async clients, def routes or run_in_threadpool, so one slow call stops stalling every request in the process.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.