could not obtain a connection from the pool within 5.000 seconds means a Rails thread waited for a database connection and found none free. Every connection in its process was busy. The fix starts with counting threads, and the count tends to run higher than the number in database.yml.
What breaks at peak
Active Record keeps one connection pool per database per process. A thread checks out a connection for its work, and other threads wait when none is free. After checkout_timeout, 5 seconds by default, the waiting thread gives up and raises:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use
The two numbers change with your timeout and with how long the thread waited, so search for the text between them. Rails 6.1 to 8.1 print the same wording.
The request that hit it fails with a 500. Every request behind it waits up to 5 seconds first, so p99 jumps to the timeout before the errors start. Three things run the pool dry:
- More threads than connections. Puma threads, Sidekiq threads, Action Cable workers and async query threads all draw from the same pool in their process. The pool default is 5, and a busy process can run more threads than that.
- Connections held too long. A slow query, a long transaction or an HTTP call made inside a transaction keeps the connection checked out. At peak, queries slow down, each thread holds its connection longer and the pool empties without any change in thread count.
- Threads you started yourself.
Thread.newin app code takes a connection the first time it queries. Wrap that work inActiveRecord::Base.connection_pool.with_connectionso the connection goes back when the block ends. Active Record’s reaper reclaims connections from dead threads everyreaping_frequency, 60 seconds by default.
Raising the pool everywhere can push the problem onto the database. Postgres refuses logins past max_connections with FATAL: sorry, too many clients already, and MySQL returns ERROR 1040 (HY000): Too many connections.
The Active Record pool config
Set the pool in config/database.yml. Rails 8.1 renamed the key from pool to max_connections and still reads pool as a fallback:
# config/database.yml, the default and production sections
default: &default
adapter: postgresql
encoding: unicode
max_connections: <%= ENV.fetch("DB_POOL") { ENV.fetch("RAILS_MAX_THREADS", 3) } %>
checkout_timeout: 5
idle_timeout: 300
production:
primary:
<<: *default
database: app_production
On Rails 8.0 and older, write pool: in place of max_connections:. Each key does one job:
max_connectionscaps the connections one process may open for this database. The pool opens them as threads ask, so a large cap costs nothing until traffic arrives. Rails generatesRAILS_MAX_THREADSwith a fallback of 5 here.checkout_timeoutsets how long a thread waits for a free connection, in seconds. A shorter wait fails requests sooner. A longer one hides a starved pool behind slow responses.idle_timeoutcloses connections left unused for that many seconds, 300 by default. It lets a quiet process hand connections back to the database.
DB_POOL lets you give web and worker processes different pool sizes from the same file. Set it per deployment.
Size the pool for each process type. Count every thread that can query in that process:
| Process | Pool should cover |
|---|---|
| Puma worker | RAILS_MAX_THREADS |
Puma worker with async_query_executor = :global_thread_pool | RAILS_MAX_THREADS + global_executor_concurrency (default 4) + 1 |
| Puma worker running Action Cable in-process | add config.action_cable.worker_pool_size (default 4) |
| Sidekiq process | Sidekiq’s concurrency (default 5 from Sidekiq 7) |
| Your own threads | add one per thread that queries |
The async row comes from the Rails configuration guide, and load_async uses that pool. Sidekiq reads RAILS_MAX_THREADS for its concurrency when you don’t set it, so a shared variable keeps the two in step. A Sidekiq process with concurrency: 10 and a pool of 5 hits this error as soon as all ten threads run jobs that query.
Size the database side. The pool is per process, and you run many processes. Add up the worst case:
web: pods × WEB_CONCURRENCY × web pool
jobs: sidekiq processes × sidekiq pool
other: migrations, consoles, cron, monitoring
total ≤ max_connections − superuser_reserved_connections
Each extra database in database.yml adds its own pool to the sum. If Solid Queue, Solid Cache or a replica live on the same Postgres server, every process can hold a pool for each of them. The Puma workers and threads recipe sets the WEB_CONCURRENCY side of the sum.
If the total doesn’t fit, raising max_connections on the database moves the problem. Put PgBouncer in transaction mode in front of Postgres, and set prepared_statements: false in database.yml if your PgBouncer is older than 1.21. For MySQL, ProxySQL plays the same role.
If this error shows up while the pool already equals the thread count, I don’t raise the pool first. I look for a connection held outside the request. The usual suspects are a Thread.new, a gem with its own background thread and a transaction wrapped around an API call. Raising the pool hides that until the next peak.
Managed databases. Amazon RDS, Aurora, Azure Database for PostgreSQL and MySQL, and Cloud SQL set max_connections from the instance size. Read the live value with SHOW max_connections;, and change it through the parameter group or the server parameters page, which may need a restart. For many pods or Lambda functions on AWS, the RDS Proxy recipe covers the managed pooler.
Test it on your system
Run a load test against staging with production’s pod count, WEB_CONCURRENCY, RAILS_MAX_THREADS and pool size. Use k6 at a fixed arrival rate and pick endpoints that query the way real traffic does:
// pool-peak.js
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: Number(__ENV.RATE || 50),
timeUnit: '1s',
duration: '10m',
preAllocatedVUs: 100,
maxVUs: 500,
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(99)<1000'],
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/orders?status=open`);
check(res, { 'status 200': (r) => r.status === 200 });
}
k6 run -e BASE_URL=https://staging.example.com -e RATE=300 pool-peak.js
Set RATE to your peak requests per second and the p99 threshold to your own budget. Start a burst of Sidekiq jobs during the run too, because a production peak tends to fill the job queue as well.
Watch three places while it runs:
- The app log. Count
ActiveRecord::ConnectionTimeoutErrorlines withkubectl logs -l app=web --since=10m | grep -c ConnectionTimeoutError. - The pool.
ActiveRecord::Base.connection_pool.statreturnssize,connections,busy,dead,idleandwaitingfor the current process. The exporter below publishes the same numbers per process. - The database. On Postgres, count connections by state:
SELECT application_name, state, count(*)
FROM pg_stat_activity
WHERE datname = 'app_production'
GROUP BY application_name, state
ORDER BY count DESC;
Write down your pass criteria before the run. No ConnectionTimeoutError lines, waiting back at 0 between bursts and database connections under the budget you computed are a fair start. Connections stuck in idle in transaction point to code that holds a transaction open while it does other work.
To prove the alert works, lower the pool below RAILS_MAX_THREADS on one staging pod and run the test again. The error should appear in the log and the alert below should fire.
Keep it healthy
Publish pool stats with prometheus_exporter. In cluster mode, start the Active Record instrumentation in each worker:
# config/puma.rb, changed lines only (use on_worker_boot on Puma 6)
before_worker_boot do
require "prometheus_exporter/instrumentation"
PrometheusExporter::Instrumentation::ActiveRecord.start(config_labels: [:database, :host])
end
The collector adds a ruby_ prefix. Scrape Postgres with postgres_exporter for the server side:
groups:
- name: activerecord-pool
rules:
- alert: ActiveRecordPoolThreadsWaiting
expr: sum by (instance) (ruby_active_record_connection_pool_waiting) > 0
for: 2m
labels:
severity: warning
annotations:
summary: "Rails threads are waiting for a database connection on {{ $labels.instance }}"
description: "The pool is empty for minutes at a time. Compare pool size with the threads that query in each process, then check for slow queries or transactions held open."
- alert: ActiveRecordPoolExhausted
expr: |
sum by (instance) (ruby_active_record_connection_pool_busy)
>= sum by (instance) (ruby_active_record_connection_pool_size)
for: 5m
labels:
severity: critical
annotations:
summary: "Every Active Record connection on {{ $labels.instance }} is busy"
description: "Requests now wait up to checkout_timeout and then fail with ConnectionTimeoutError. Scale out if the database has headroom; otherwise find the long transactions in pg_stat_activity."
- alert: ActiveRecordDeadConnections
expr: sum by (instance) (ruby_active_record_connection_pool_dead) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Connections owned by dead threads on {{ $labels.instance }}"
description: "A thread died while it held a connection. Look for Thread.new without connection_pool.with_connection, and for gems that run their own threads."
- alert: PostgresConnectionsNearMax
expr: sum by (instance) (pg_stat_activity_count) / on (instance) pg_settings_max_connections > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Postgres {{ $labels.instance }} is past 80% of max_connections"
description: "Rails pools across all pods are close to the server limit. Recompute pods x workers x pool plus Sidekiq, then lower a pool or add PgBouncer before scaling out."
The first rule catches a pool that runs short for a few minutes. The second fires when the pool stays full, which is when users start getting errors. Set the 0.8 on the database rule from the headroom you want for migrations and consoles.
Roll back
The pool lives in database.yml and an environment variable. Save the running values first:
kubectl exec deploy/web -- printenv DB_POOL RAILS_MAX_THREADS > pool-before.txt
To revert, set the old value and roll the pods:
kubectl set env deploy/web DB_POOL=5 # your previous value
If the bigger pools pushed the database into too many clients errors, roll back now and scale the web deployment down until the total fits. Any other revert can wait for the quiet hours, since rolling the pods reconnects every process at once.
More recipes
- Puma Workers and Threads for Rails at Peak
Size Puma workers to the CPU your Rails pod gets and threads to its I/O wait, so peak traffic queues less and the pod stays inside its memory limit.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.