Could Not Obtain a Connection From the Pool in Rails

Give each Rails process a pool that covers every thread that can query, so ActiveRecord::ConnectionTimeoutError stops at peak and the database keeps headroom.

Published

Applies to Rails 7.0 and later; the database.yml key is max_connections from Rails 8.1, pool before that; Sidekiq 7 and later

could not obtain a connection from the pool within 5.000 seconds means a Rails thread waited for a database connection and found none free. Every connection in its process was busy. The fix starts with counting threads, and the count tends to run higher than the number in database.yml.

What breaks at peak

Active Record keeps one connection pool per database per process. A thread checks out a connection for its work, and other threads wait when none is free. After checkout_timeout, 5 seconds by default, the waiting thread gives up and raises:

ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use

The two numbers change with your timeout and with how long the thread waited, so search for the text between them. Rails 6.1 to 8.1 print the same wording.

The request that hit it fails with a 500. Every request behind it waits up to 5 seconds first, so p99 jumps to the timeout before the errors start. Three things run the pool dry:

  • More threads than connections. Puma threads, Sidekiq threads, Action Cable workers and async query threads all draw from the same pool in their process. The pool default is 5, and a busy process can run more threads than that.
  • Connections held too long. A slow query, a long transaction or an HTTP call made inside a transaction keeps the connection checked out. At peak, queries slow down, each thread holds its connection longer and the pool empties without any change in thread count.
  • Threads you started yourself. Thread.new in app code takes a connection the first time it queries. Wrap that work in ActiveRecord::Base.connection_pool.with_connection so the connection goes back when the block ends. Active Record’s reaper reclaims connections from dead threads every reaping_frequency, 60 seconds by default.

Raising the pool everywhere can push the problem onto the database. Postgres refuses logins past max_connections with FATAL: sorry, too many clients already, and MySQL returns ERROR 1040 (HY000): Too many connections.

The Active Record pool config

Set the pool in config/database.yml. Rails 8.1 renamed the key from pool to max_connections and still reads pool as a fallback:

# config/database.yml, the default and production sections
default: &default
  adapter: postgresql
  encoding: unicode
  max_connections: <%= ENV.fetch("DB_POOL") { ENV.fetch("RAILS_MAX_THREADS", 3) } %>
  checkout_timeout: 5
  idle_timeout: 300

production:
  primary:
    <<: *default
    database: app_production

On Rails 8.0 and older, write pool: in place of max_connections:. Each key does one job:

  • max_connections caps the connections one process may open for this database. The pool opens them as threads ask, so a large cap costs nothing until traffic arrives. Rails generates RAILS_MAX_THREADS with a fallback of 5 here.
  • checkout_timeout sets how long a thread waits for a free connection, in seconds. A shorter wait fails requests sooner. A longer one hides a starved pool behind slow responses.
  • idle_timeout closes connections left unused for that many seconds, 300 by default. It lets a quiet process hand connections back to the database.

DB_POOL lets you give web and worker processes different pool sizes from the same file. Set it per deployment.

Size the pool for each process type. Count every thread that can query in that process:

ProcessPool should cover
Puma workerRAILS_MAX_THREADS
Puma worker with async_query_executor = :global_thread_poolRAILS_MAX_THREADS + global_executor_concurrency (default 4) + 1
Puma worker running Action Cable in-processadd config.action_cable.worker_pool_size (default 4)
Sidekiq processSidekiq’s concurrency (default 5 from Sidekiq 7)
Your own threadsadd one per thread that queries

The async row comes from the Rails configuration guide, and load_async uses that pool. Sidekiq reads RAILS_MAX_THREADS for its concurrency when you don’t set it, so a shared variable keeps the two in step. A Sidekiq process with concurrency: 10 and a pool of 5 hits this error as soon as all ten threads run jobs that query.

Size the database side. The pool is per process, and you run many processes. Add up the worst case:

web:    pods × WEB_CONCURRENCY × web pool
jobs:   sidekiq processes × sidekiq pool
other:  migrations, consoles, cron, monitoring
total ≤ max_connections − superuser_reserved_connections

Each extra database in database.yml adds its own pool to the sum. If Solid Queue, Solid Cache or a replica live on the same Postgres server, every process can hold a pool for each of them. The Puma workers and threads recipe sets the WEB_CONCURRENCY side of the sum.

If the total doesn’t fit, raising max_connections on the database moves the problem. Put PgBouncer in transaction mode in front of Postgres, and set prepared_statements: false in database.yml if your PgBouncer is older than 1.21. For MySQL, ProxySQL plays the same role.

If this error shows up while the pool already equals the thread count, I don’t raise the pool first. I look for a connection held outside the request. The usual suspects are a Thread.new, a gem with its own background thread and a transaction wrapped around an API call. Raising the pool hides that until the next peak.

Managed databases. Amazon RDS, Aurora, Azure Database for PostgreSQL and MySQL, and Cloud SQL set max_connections from the instance size. Read the live value with SHOW max_connections;, and change it through the parameter group or the server parameters page, which may need a restart. For many pods or Lambda functions on AWS, the RDS Proxy recipe covers the managed pooler.

Test it on your system

Run a load test against staging with production’s pod count, WEB_CONCURRENCY, RAILS_MAX_THREADS and pool size. Use k6 at a fixed arrival rate and pick endpoints that query the way real traffic does:

// pool-peak.js
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.RATE || 50),
      timeUnit: '1s',
      duration: '10m',
      preAllocatedVUs: 100,
      maxVUs: 500,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<1000'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/orders?status=open`);
  check(res, { 'status 200': (r) => r.status === 200 });
}
k6 run -e BASE_URL=https://staging.example.com -e RATE=300 pool-peak.js

Set RATE to your peak requests per second and the p99 threshold to your own budget. Start a burst of Sidekiq jobs during the run too, because a production peak tends to fill the job queue as well.

Watch three places while it runs:

  • The app log. Count ActiveRecord::ConnectionTimeoutError lines with kubectl logs -l app=web --since=10m | grep -c ConnectionTimeoutError.
  • The pool. ActiveRecord::Base.connection_pool.stat returns size, connections, busy, dead, idle and waiting for the current process. The exporter below publishes the same numbers per process.
  • The database. On Postgres, count connections by state:
SELECT application_name, state, count(*)
FROM pg_stat_activity
WHERE datname = 'app_production'
GROUP BY application_name, state
ORDER BY count DESC;

Write down your pass criteria before the run. No ConnectionTimeoutError lines, waiting back at 0 between bursts and database connections under the budget you computed are a fair start. Connections stuck in idle in transaction point to code that holds a transaction open while it does other work.

To prove the alert works, lower the pool below RAILS_MAX_THREADS on one staging pod and run the test again. The error should appear in the log and the alert below should fire.

Keep it healthy

Publish pool stats with prometheus_exporter. In cluster mode, start the Active Record instrumentation in each worker:

# config/puma.rb, changed lines only (use on_worker_boot on Puma 6)
before_worker_boot do
  require "prometheus_exporter/instrumentation"
  PrometheusExporter::Instrumentation::ActiveRecord.start(config_labels: [:database, :host])
end

The collector adds a ruby_ prefix. Scrape Postgres with postgres_exporter for the server side:

groups:
  - name: activerecord-pool
    rules:
      - alert: ActiveRecordPoolThreadsWaiting
        expr: sum by (instance) (ruby_active_record_connection_pool_waiting) > 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Rails threads are waiting for a database connection on {{ $labels.instance }}"
          description: "The pool is empty for minutes at a time. Compare pool size with the threads that query in each process, then check for slow queries or transactions held open."
      - alert: ActiveRecordPoolExhausted
        expr: |
          sum by (instance) (ruby_active_record_connection_pool_busy)
            >= sum by (instance) (ruby_active_record_connection_pool_size)
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Every Active Record connection on {{ $labels.instance }} is busy"
          description: "Requests now wait up to checkout_timeout and then fail with ConnectionTimeoutError. Scale out if the database has headroom; otherwise find the long transactions in pg_stat_activity."
      - alert: ActiveRecordDeadConnections
        expr: sum by (instance) (ruby_active_record_connection_pool_dead) > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Connections owned by dead threads on {{ $labels.instance }}"
          description: "A thread died while it held a connection. Look for Thread.new without connection_pool.with_connection, and for gems that run their own threads."
      - alert: PostgresConnectionsNearMax
        expr: sum by (instance) (pg_stat_activity_count) / on (instance) pg_settings_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Postgres {{ $labels.instance }} is past 80% of max_connections"
          description: "Rails pools across all pods are close to the server limit. Recompute pods x workers x pool plus Sidekiq, then lower a pool or add PgBouncer before scaling out."

The first rule catches a pool that runs short for a few minutes. The second fires when the pool stays full, which is when users start getting errors. Set the 0.8 on the database rule from the headroom you want for migrations and consoles.

Roll back

The pool lives in database.yml and an environment variable. Save the running values first:

kubectl exec deploy/web -- printenv DB_POOL RAILS_MAX_THREADS > pool-before.txt

To revert, set the old value and roll the pods:

kubectl set env deploy/web DB_POOL=5   # your previous value

If the bigger pools pushed the database into too many clients errors, roll back now and scale the web deployment down until the total fits. Any other revert can wait for the quiet hours, since rolling the pods reconnects every process at once.

More recipes

Further reading