Connection is not available, request timed out after means a request waited the whole connection-timeout for a HikariCP connection and got none. Every connection in the pool was busy. The fix is either a bigger pool or shorter holds, and the database decides how big the pool may grow.
What breaks at peak
The exception comes from HikariCP and reaches your code wrapped by Spring. On HikariCP 5.1 and later, the line ends with the pool’s state at the moment it gave up:
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms (total=10, active=10, idle=0, waiting=50)
HikariCP 4 ends the message at 30000ms. with no counts. The counts above are an example, but the pattern is the one to look for. active equals total, idle is zero, and threads are waiting. Spring wraps the exception in CannotGetJdbcConnectionException: Failed to obtain JDBC Connection, or in CannotCreateTransactionException when JPA opens a transaction.
30000ms is the default connection-timeout. With it, a request stalls for 30 seconds before it fails, and your load balancer or client has often given up by then. Users get a timeout and your logs fill with stack traces half a minute later.
The pool runs dry for one of four reasons:
- The pool is too small for the concurrency. Spring Boot’s default pool holds 10 connections per pod. Tomcat runs up to 200 request threads. At peak, more than 10 of them want the database at once, and the rest queue.
- Connections are held too long. A slow query, an HTTP call inside a
@Transactionalmethod or a long batch job keeps a connection out of the pool.spring.jpa.open-in-viewdefaults totrue, so a request that touched JPA keeps its connection until the response is written. - A leak. Code that borrows a connection outside Spring’s templates and never closes it shrinks the pool for good. The pool gets smaller over hours, then fails at the next spike.
- The database refuses new connections. Pods times pool size passes the database’s
max_connections. Postgres repliesFATAL: sorry, too many clients already, and HikariCP can’t fill its pool.
One more warning shows up next to pool timeouts:
HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=...).
HikariCP logs it when its housekeeping thread ran late. A long GC pause or CPU throttling froze the whole JVM. Fix the freeze first. The GC pause recipe and the CPU throttling recipe cover both causes.
The HikariCP connection pool config for Spring Boot
Changed lines only, in application.yaml. Replace the URL and pool size with yours:
spring:
datasource:
url: jdbc:postgresql://db.internal:5432/app
username: app
password: ${DB_PASSWORD}
hikari:
pool-name: app-db
maximum-pool-size: 20
connection-timeout: 3000
max-lifetime: 1500000
leak-detection-threshold: 20000
jpa:
open-in-view: false
management:
endpoints:
web:
exposure:
include: health,prometheus
What each line does:
pool-namenames the pool in logs and in thepoollabel of every metric. With several datasources, unnamed pools show up asHikariPool-1andHikariPool-2.maximum-pool-sizecaps connections per pod. Too low and requests queue for a connection. Too high and the database runs out of slots or spends its CPU switching between backends. Leaveminimum-idleunset, so the pool stays at this size. HikariCP recommends a fixed-size pool for spiky traffic.connection-timeoutsets how long a request waits for a connection, in milliseconds. Set it below your request timeout, so a starved request fails fast with a clear error. Set it too low and short bursts turn into errors.max-lifetimeretires each connection after this many milliseconds. The default is 30 minutes. HikariCP asks for a value several seconds shorter than any timeout the database, a proxy or a firewall applies. Otherwise something in the path closes a connection the pool still thinks is good.leak-detection-thresholdlogsConnection leak detection triggered for ... on thread ..., stack trace followswhen a connection stays out longer than this. The stack trace shows the code that borrowed it. Set it above your slowest legitimate transaction. HikariCP accepts 2000 ms at the lowest.open-in-view: falsereleases the connection when the service method returns, not when the response finishes. Lazy loading in controllers or templates then fails fast, which tells you where to add a fetch join.- The
prometheusendpoint exposes thehikaricp_*metrics for the test and the alerts below. It needsmicrometer-registry-prometheuson the classpath.
Size the pool from both ends. Start from the database, since it sets the ceiling:
pool per pod <= (max_connections - reserved slots - other clients) / max pods
Use the HPA’s maxReplicas for max pods, not today’s count. Other clients are migrations, cron jobs, admin tools and monitoring. During a rolling update, old and new pods run side by side, so count the surge pods too.
Then check that the pool covers your traffic. A connection busy for the length of each transaction gives you:
connections needed per pod = peak transactions per second per pod * average seconds a connection is held
If the second number is bigger than the first, raising the pool won’t help. Shorten the holds, or put a pooler such as PgBouncer between the pods and the database. The PgBouncer recipe covers that setup.
HikariCP’s own pool sizing guide gives a target for the database side: connections = ((core_count * 2) + effective_spindle_count). That’s the total across all pods. A bigger total adds queueing inside the database instead of in the pool.
If one request can hold two connections at once, for example a REQUIRES_NEW transaction inside another, a small pool can deadlock. The same guide gives the floor: pool size = Tn x (Cm - 1) + 1. Tn is the number of threads, and Cm is the most connections one thread holds.
I shorten the time each request holds a connection before I touch the pool size. Moving an HTTP call out of a @Transactional method often frees more capacity than ten extra connections.
Managed databases. On Amazon RDS and Aurora, max_connections comes from the instance class through the parameter group. Change it there, or move up an instance class. With RDS Proxy in front, keep max-lifetime below the proxy’s idle client timeout. Cloud SQL sets max_connections as a database flag. Azure Database for PostgreSQL flexible server exposes it as a server parameter and offers a built-in PgBouncer.
Test it on your system
Run a staging copy of the service against a database with production’s max_connections. Pick an endpoint that does real database work. Hold a fixed arrival rate with k6, so a slow pool doesn’t slow the load generator down with it:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
pool: {
executor: 'ramping-arrival-rate',
startRate: 50,
timeUnit: '1s',
preAllocatedVUs: 200,
maxVUs: 2000,
stages: [
{ target: 200, duration: '5m' },
{ target: 200, duration: '10m' },
{ target: 400, duration: '5m' },
{ target: 400, duration: '10m' },
],
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(99)<500'],
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/api/orders/recent`);
check(res, { 'status is 200': (r) => r.status === 200 });
}
Set the first plateau to your expected peak and the second above it. Scale the Deployment to its maxReplicas for the run, so you also prove the database can take every pod’s pool at once.
While it runs, watch the pool on one pod:
kubectl port-forward deploy/api 8080:8080 &
watch -n 5 "curl -s localhost:8080/actuator/prometheus | grep -E '^hikaricp_connections_(active|idle|pending|max|timeout_total|acquire_seconds_max)'"
And the database side, here on Postgres:
SELECT state, count(*) FROM pg_stat_activity WHERE datname = 'app' GROUP BY state;
hikaricp_connections_pending counts threads waiting for a connection. hikaricp_connections_acquire_seconds_max shows the longest wait in the last window. hikaricp_connections_timeout_total counts requests that gave up.
Pick your targets first. I call a run a pass when:
hikaricp_connections_timeout_totalstays flat and noSQLTransientConnectionExceptionappears in the logs.pendingreturns to zero between bursts, and the longest acquire time stays well underconnection-timeout.- Database connections stay under
max_connectionswith every pod at its full pool. - p99 and error rate stay inside the k6 thresholds you set.
If active sits at the maximum while the database’s CPU is low, connections are waiting on something other than SQL. Take a thread dump with jcmd 1 Thread.print and look for threads that hold a connection inside an HTTP call.
Keep it healthy
Spring Boot binds HikariCP’s metrics through Micrometer, and Prometheus scrapes them from /actuator/prometheus. These rules catch timeouts, a pool at its limit and slow acquires:
groups:
- name: hikaricp
rules:
- alert: HikariConnectionTimeouts
expr: sum by (job, pool) (increase(hikaricp_connections_timeout_total[5m])) > 0
for: 1m
labels:
severity: critical
annotations:
summary: "Requests timed out waiting for a {{ $labels.pool }} connection"
description: "Callers got SQLTransientConnectionException. Check pg_stat_activity for long transactions, then the leak detection log, before you raise maximum-pool-size."
- alert: HikariPoolSaturated
expr: |
max by (job, instance, pool) (hikaricp_connections_active)
/ max by (job, instance, pool) (hikaricp_connections_max) > 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.pool }} on {{ $labels.instance }} is above 90% busy"
description: "The pool has little room for the next spike. Find the slowest holders in hikaricp_connections_usage_seconds and the database's slow query log."
- alert: HikariThreadsWaiting
expr: max by (job, instance, pool) (hikaricp_connections_pending) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Threads have queued for {{ $labels.pool }} connections for 5 minutes"
description: "Requests are waiting on the pool. Compare pool size times pods with max_connections before you grow it, or shorten transactions."
- alert: HikariSlowAcquire
expr: max by (job, instance, pool) (hikaricp_connections_acquire_seconds_max) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "Getting a {{ $labels.pool }} connection takes over 1 s"
description: "Waits are approaching connection-timeout. Check for a leak in the logs and for a database that's slow to accept new connections."
Also alert on these log lines:
Connection is not available, request timed out after
Connection leak detection triggered for
Thread starvation or clock leap detected
Roll back
Note the current values before you change them. kubectl get deployment api -o yaml > api-before.yaml covers settings passed as environment variables. Keep application.yaml in git so the old values are one revert away.
You can also override the pool size without a rebuild. Spring Boot maps SPRING_DATASOURCE_HIKARI_MAXIMUMPOOLSIZE to maximum-pool-size:
kubectl set env deployment/api SPRING_DATASOURCE_HIKARI_MAXIMUMPOOLSIZE=10
If the bigger pool brings too many clients errors from the database, roll back now. A pool that can’t connect fails every request, which is worse than a queue. If the change only moved latency the wrong way, revert it in a quiet hour.
More recipes
- G1 vs ZGC and How to Catch Java GC Pause Spikes
Alert on Java GC pause latency and full GCs before users notice p99 spikes, then use the same metrics to decide between G1 and ZGC.
- Fix Java Heap Space OOM in Kubernetes Containers
Size the JVM heap from the container limit with MaxRAMPercentage and leave room for native memory, so pods stop failing with heap errors or OOMKilled.
- Spring Boot Virtual Threads and server.tomcat.threads.max
Size server.tomcat.threads.max from your traffic, or turn on Spring Boot virtual threads, so requests stop queueing inside Tomcat at peak.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.