Django CONN_MAX_AGE, Connection Pooling and PgBouncer

Match CONN_MAX_AGE or Django's pool to your worker count, and add Django PgBouncer settings when pods multiply, so peak traffic stops exhausting Postgres.

Published

Applies to Django 4.2 and later on PostgreSQL; the built-in pool needs Django 5.1 or later with psycopg 3; PgBouncer 1.21 and later

Django’s CONN_MAX_AGE decides whether each request opens a new database connection or reuses one. The default of 0 opens and closes one per request. A high value keeps one open per thread, forever. Both settings fail at peak in different ways, and the fix depends on how many Django threads you run in total.

What breaks at peak

With CONN_MAX_AGE = 0, every request pays for a new connection. Django closes the connection at the end of each request. The next request opens a TCP connection, often TLS, runs authentication and makes Postgres fork a new backend process. At peak, your connection rate equals your request rate. Postgres spends CPU on forks and logins, and your p99 carries the setup time on every request.

With a long CONN_MAX_AGE, idle connections pile up. Django keeps one connection per worker thread. Total connections are pods × workers × threads, plus Celery workers, cron jobs and migrations. The autoscaler adds pods at peak, and Postgres refuses the next login:

django.db.utils.OperationalError: connection to server at "db.internal" (10.0.4.12), port 5432 failed: FATAL:  sorry, too many clients already

The PostgreSQL PgBouncer recipe explains that error and its second form for reserved slots. On MySQL the same math ends in Too many connections, which the MySQL too many connections recipe covers.

Long-lived connections also go stale. A database restart, a failover or a firewall that drops idle TCP sessions closes connections Django still holds. The next request on that thread fails with an error such as server closed the connection unexpectedly.

ASGI changes the rules. Django’s docs say to turn persistent connections off under ASGI and use a pool instead.

Django CONN_MAX_AGE, pooling and PgBouncer settings

Pick one of three setups by your total thread count:

Your setupUse
WSGI, and pods × workers × threads fits well under max_connectionsPersistent connections with health checks
ASGI, or many threads that each spend little time in queriesDjango’s built-in pool, Django 5.1 or later
Pod count that grows at peak past what Postgres can holdPgBouncer in transaction mode

Persistent connections with health checks. This settings.py block reads its values from the environment, so you can change them without a rebuild:

# settings.py (database section)
import os

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "HOST": os.environ["DB_HOST"],
        "PORT": os.environ.get("DB_PORT", "5432"),
        "NAME": os.environ["DB_NAME"],
        "USER": os.environ["DB_USER"],
        "PASSWORD": os.environ["DB_PASSWORD"],
        "CONN_MAX_AGE": int(os.environ.get("DB_CONN_MAX_AGE", "60")),
        "CONN_HEALTH_CHECKS": True,
        "OPTIONS": {
            "application_name": os.environ.get("DB_APP_NAME", "django-web"),
            "connect_timeout": 5,
            "options": "-c statement_timeout=25000",
        },
    }
}
  • CONN_MAX_AGE is the connection’s lifetime in seconds. Django closes it at the start or end of a request once it’s older than this. I use 60 seconds rather than None, so connections recycle after a failover and spread across replicas behind a DNS name.
  • CONN_HEALTH_CHECKS, added in Django 4.1, checks a reused connection once per request before the first query. If the database closed it, Django opens a new one instead of failing the request.
  • application_name labels the connections in pg_stat_activity, so you can count Django’s share. Give Celery and cron a different name.
  • connect_timeout and statement_timeout bound how long a request waits on the database. Keep the statement timeout below your Gunicorn timeout, as the Gunicorn workers and timeouts recipe explains.

Size it with this rule: pods at peak × workers × threads, plus Celery concurrency, plus a few for migrations and monitoring, must stay under max_connections minus the reserved slots. If it doesn’t fit, move to one of the next two setups.

Django’s built-in pool. Django 5.1 added a connection pool for PostgreSQL through psycopg_pool. It needs psycopg 3, installed as psycopg[pool]. Each process gets one pool for each database alias, shared by all its threads. Add this below the block above (changed lines only):

DATABASES["default"]["CONN_MAX_AGE"] = 0
DATABASES["default"]["OPTIONS"]["pool"] = {
    "min_size": 2,
    "max_size": int(os.environ.get("DB_POOL_MAX", "4")),
    "timeout": 10,
    "max_lifetime": 1800,
}
  • CONN_MAX_AGE must be 0, or Django raises ImproperlyConfigured: Pooling doesn't support persistent connections.
  • max_size caps connections per process. A thread holds a connection only while it uses one. If your views spend little time in queries, it can sit below the thread count.
  • timeout is how long a thread waits for a free connection. Past it, the request fails with psycopg_pool.PoolTimeout: couldn't get a connection after 10.00 sec. Keep it well below your request timeout.
  • max_lifetime recycles each connection after that many seconds, which does the job CONN_MAX_AGE did before.
  • With CONN_HEALTH_CHECKS = True, Django checks a connection each time it leaves the pool.

The budget becomes pods × workers × max_size. That’s smaller than one connection per thread, but it still grows with every pod.

PgBouncer in front of Postgres. Once the autoscaler can add pods faster than Postgres can take connections, put a pooler between them. PgBouncer in transaction mode lets thousands of client connections share a fixed set of server connections. Point Django at it and turn off server-side cursors (changed lines only):

DATABASES["default"]["HOST"] = os.environ.get("PGBOUNCER_HOST", "pgbouncer.internal")
DATABASES["default"]["PORT"] = "6432"
DATABASES["default"]["DISABLE_SERVER_SIDE_CURSORS"] = True

QuerySet.iterator() opens a server-side cursor that lives on one server connection. In transaction mode the next fetch can land on a different one, and Postgres returns cursor "_django_curs_..." does not exist. DISABLE_SERVER_SIDE_CURSORS makes .iterator() fetch through a normal cursor instead. Django turns off psycopg 3’s automatic prepared statements by default, so those don’t break transaction mode.

You can keep CONN_MAX_AGE at 60 behind PgBouncer, since a connection to PgBouncer costs little. Run migrations straight against Postgres on port 5432. Your migrate Job sets DB_HOST and DB_PORT to the database, and the web pods point at PgBouncer.

Managed databases. Amazon RDS and Aurora take RDS Proxy in place of PgBouncer, and the RDS Proxy recipe covers its pinning rules. Azure Database for PostgreSQL flexible server has PgBouncer built in. Set the pgbouncer.enabled server parameter, then connect on port 6432 with the same host name. On these services you change max_connections through the parameter group or server parameters, and the provider sets its default from the instance size.

Test it on your system

Use a staging database with the same max_connections as production, and run as many app pods as your peak needs. Send a fixed arrival rate at endpoints that query the database:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.RATE || 150), // your peak requests per second
      timeUnit: '1s',
      duration: '20m',
      preAllocatedVUs: 200,
      maxVUs: 1500,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(99)<600'],
  },
};

export default function () {
  const id = Math.floor(Math.random() * 50000) + 1;
  const res = http.get(`${__ENV.BASE_URL}/api/customers/${id}/orders/`);
  check(res, { 'status 200': (r) => r.status === 200 });
}

Partway through, scale the app to your peak pod count with kubectl scale deploy/web --replicas=<peak>. The scale-out is where connection limits break.

On Postgres, watch who holds connections and how fast new ones open:

-- connections by app and state
SELECT application_name, state, count(*)
FROM pg_stat_activity
GROUP BY 1, 2
ORDER BY 3 DESC;

-- sessions opened since the stats reset (PostgreSQL 14 and later); run twice and compare
SELECT sessions FROM pg_stat_database WHERE datname = current_database();

With persistent connections or a pool, sessions grows by a handful during the run. With CONN_MAX_AGE = 0 it climbs with every request. Behind PgBouncer, run SHOW POOLS; on its admin console and watch cl_waiting.

Fix your targets before the run. It passes when Django’s connection count stays inside your budget at peak pod count, and no too many clients or PoolTimeout errors appear. The k6 thresholds should hold through the scale-out.

Keep it healthy

Scrape Postgres with postgres_exporter. In current releases its pg_stat_activity_count metric carries application_name and state labels, so the application_name you set above lets you alert on Django alone. Replace django-web and the budget of 300 with your own values:

groups:
  - name: django-db-connections
    rules:
      - alert: DjangoConnectionsOverBudget
        expr: sum by (instance) (pg_stat_activity_count{application_name="django-web"}) > 300
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Django holds more Postgres connections than its budget on {{ $labels.instance }}"
          description: "Count pods, workers and threads against the budget. Lower DB_POOL_MAX or DB_CONN_MAX_AGE, or move the app behind PgBouncer before the next scale-out."
      - alert: PostgresConnectionSlotsLow
        expr: |
          sum by (instance) (pg_stat_activity_count)
            > on (instance) (pg_settings_max_connections - pg_settings_superuser_reserved_connections) * 0.9
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} has under 10% of its connection slots left"
          description: "New logins will fail with too many clients. Scale down workers that hold idle connections, or point them at PgBouncer. Check pg_stat_activity by application_name."
      - alert: DjangoIdleInTransaction
        expr: max by (instance, datname) (pg_stat_activity_max_tx_duration{state="idle in transaction", application_name="django-web"}) > 60
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "A Django connection sat idle in a transaction for over a minute on {{ $labels.datname }}"
          description: "Code is holding a transaction open while it waits on something else, often an API call inside atomic(). Find the query in pg_stat_activity and move the slow call outside the transaction."

Also alert on the app’s own errors. A rising count of OperationalError or PoolTimeout in your error tracker shows connection trouble before the database metrics do.

Roll back

Save what’s running before you change anything:

kubectl get deploy web -o jsonpath='{.spec.template.spec.containers[0].env}' > web-env-before.json
psql -h db.internal -U app -c "SHOW max_connections;" > pg-before.txt

The connection settings come from environment variables, so a rollback is kubectl set env deployment/web DB_CONN_MAX_AGE=0 DB_HOST=db.internal DB_PORT=5432 with your old values. The pool and DISABLE_SERVER_SIDE_CURSORS changes live in code, so revert the commit.

Before you roll back to persistent connections, check that the old total still fits under max_connections at your current pod count. If the new setup causes PoolTimeout or cursor errors, roll back right away. A change that only failed to help can wait for a low-traffic window.

More recipes

Further reading