Fix HikariCP Connection Is Not Available, Request Timed Out

Size the HikariCP connection pool in Spring Boot from your database limit and pod count, so requests stop timing out while they wait for a connection.

Published

Applies to Spring Boot 3.x and later with HikariCP 5 and later; HikariCP 4 prints the same error without the pool counts

Connection is not available, request timed out after means a request waited the whole connection-timeout for a HikariCP connection and got none. Every connection in the pool was busy. The fix is either a bigger pool or shorter holds, and the database decides how big the pool may grow.

What breaks at peak

The exception comes from HikariCP and reaches your code wrapped by Spring. On HikariCP 5.1 and later, the line ends with the pool’s state at the moment it gave up:

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms (total=10, active=10, idle=0, waiting=50)

HikariCP 4 ends the message at 30000ms. with no counts. The counts above are an example, but the pattern is the one to look for. active equals total, idle is zero, and threads are waiting. Spring wraps the exception in CannotGetJdbcConnectionException: Failed to obtain JDBC Connection, or in CannotCreateTransactionException when JPA opens a transaction.

30000ms is the default connection-timeout. With it, a request stalls for 30 seconds before it fails, and your load balancer or client has often given up by then. Users get a timeout and your logs fill with stack traces half a minute later.

The pool runs dry for one of four reasons:

  • The pool is too small for the concurrency. Spring Boot’s default pool holds 10 connections per pod. Tomcat runs up to 200 request threads. At peak, more than 10 of them want the database at once, and the rest queue.
  • Connections are held too long. A slow query, an HTTP call inside a @Transactional method or a long batch job keeps a connection out of the pool. spring.jpa.open-in-view defaults to true, so a request that touched JPA keeps its connection until the response is written.
  • A leak. Code that borrows a connection outside Spring’s templates and never closes it shrinks the pool for good. The pool gets smaller over hours, then fails at the next spike.
  • The database refuses new connections. Pods times pool size passes the database’s max_connections. Postgres replies FATAL: sorry, too many clients already, and HikariCP can’t fill its pool.

One more warning shows up next to pool timeouts:

HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=...).

HikariCP logs it when its housekeeping thread ran late. A long GC pause or CPU throttling froze the whole JVM. Fix the freeze first. The GC pause recipe and the CPU throttling recipe cover both causes.

The HikariCP connection pool config for Spring Boot

Changed lines only, in application.yaml. Replace the URL and pool size with yours:

spring:
  datasource:
    url: jdbc:postgresql://db.internal:5432/app
    username: app
    password: ${DB_PASSWORD}
    hikari:
      pool-name: app-db
      maximum-pool-size: 20
      connection-timeout: 3000
      max-lifetime: 1500000
      leak-detection-threshold: 20000
  jpa:
    open-in-view: false
management:
  endpoints:
    web:
      exposure:
        include: health,prometheus

What each line does:

  • pool-name names the pool in logs and in the pool label of every metric. With several datasources, unnamed pools show up as HikariPool-1 and HikariPool-2.
  • maximum-pool-size caps connections per pod. Too low and requests queue for a connection. Too high and the database runs out of slots or spends its CPU switching between backends. Leave minimum-idle unset, so the pool stays at this size. HikariCP recommends a fixed-size pool for spiky traffic.
  • connection-timeout sets how long a request waits for a connection, in milliseconds. Set it below your request timeout, so a starved request fails fast with a clear error. Set it too low and short bursts turn into errors.
  • max-lifetime retires each connection after this many milliseconds. The default is 30 minutes. HikariCP asks for a value several seconds shorter than any timeout the database, a proxy or a firewall applies. Otherwise something in the path closes a connection the pool still thinks is good.
  • leak-detection-threshold logs Connection leak detection triggered for ... on thread ..., stack trace follows when a connection stays out longer than this. The stack trace shows the code that borrowed it. Set it above your slowest legitimate transaction. HikariCP accepts 2000 ms at the lowest.
  • open-in-view: false releases the connection when the service method returns, not when the response finishes. Lazy loading in controllers or templates then fails fast, which tells you where to add a fetch join.
  • The prometheus endpoint exposes the hikaricp_* metrics for the test and the alerts below. It needs micrometer-registry-prometheus on the classpath.

Size the pool from both ends. Start from the database, since it sets the ceiling:

pool per pod <= (max_connections - reserved slots - other clients) / max pods

Use the HPA’s maxReplicas for max pods, not today’s count. Other clients are migrations, cron jobs, admin tools and monitoring. During a rolling update, old and new pods run side by side, so count the surge pods too.

Then check that the pool covers your traffic. A connection busy for the length of each transaction gives you:

connections needed per pod = peak transactions per second per pod * average seconds a connection is held

If the second number is bigger than the first, raising the pool won’t help. Shorten the holds, or put a pooler such as PgBouncer between the pods and the database. The PgBouncer recipe covers that setup.

HikariCP’s own pool sizing guide gives a target for the database side: connections = ((core_count * 2) + effective_spindle_count). That’s the total across all pods. A bigger total adds queueing inside the database instead of in the pool.

If one request can hold two connections at once, for example a REQUIRES_NEW transaction inside another, a small pool can deadlock. The same guide gives the floor: pool size = Tn x (Cm - 1) + 1. Tn is the number of threads, and Cm is the most connections one thread holds.

I shorten the time each request holds a connection before I touch the pool size. Moving an HTTP call out of a @Transactional method often frees more capacity than ten extra connections.

Managed databases. On Amazon RDS and Aurora, max_connections comes from the instance class through the parameter group. Change it there, or move up an instance class. With RDS Proxy in front, keep max-lifetime below the proxy’s idle client timeout. Cloud SQL sets max_connections as a database flag. Azure Database for PostgreSQL flexible server exposes it as a server parameter and offers a built-in PgBouncer.

Test it on your system

Run a staging copy of the service against a database with production’s max_connections. Pick an endpoint that does real database work. Hold a fixed arrival rate with k6, so a slow pool doesn’t slow the load generator down with it:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    pool: {
      executor: 'ramping-arrival-rate',
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 2000,
      stages: [
        { target: 200, duration: '5m' },
        { target: 200, duration: '10m' },
        { target: 400, duration: '5m' },
        { target: 400, duration: '10m' },
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<500'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/orders/recent`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}

Set the first plateau to your expected peak and the second above it. Scale the Deployment to its maxReplicas for the run, so you also prove the database can take every pod’s pool at once.

While it runs, watch the pool on one pod:

kubectl port-forward deploy/api 8080:8080 &
watch -n 5 "curl -s localhost:8080/actuator/prometheus | grep -E '^hikaricp_connections_(active|idle|pending|max|timeout_total|acquire_seconds_max)'"

And the database side, here on Postgres:

SELECT state, count(*) FROM pg_stat_activity WHERE datname = 'app' GROUP BY state;

hikaricp_connections_pending counts threads waiting for a connection. hikaricp_connections_acquire_seconds_max shows the longest wait in the last window. hikaricp_connections_timeout_total counts requests that gave up.

Pick your targets first. I call a run a pass when:

  • hikaricp_connections_timeout_total stays flat and no SQLTransientConnectionException appears in the logs.
  • pending returns to zero between bursts, and the longest acquire time stays well under connection-timeout.
  • Database connections stay under max_connections with every pod at its full pool.
  • p99 and error rate stay inside the k6 thresholds you set.

If active sits at the maximum while the database’s CPU is low, connections are waiting on something other than SQL. Take a thread dump with jcmd 1 Thread.print and look for threads that hold a connection inside an HTTP call.

Keep it healthy

Spring Boot binds HikariCP’s metrics through Micrometer, and Prometheus scrapes them from /actuator/prometheus. These rules catch timeouts, a pool at its limit and slow acquires:

groups:
  - name: hikaricp
    rules:
      - alert: HikariConnectionTimeouts
        expr: sum by (job, pool) (increase(hikaricp_connections_timeout_total[5m])) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Requests timed out waiting for a {{ $labels.pool }} connection"
          description: "Callers got SQLTransientConnectionException. Check pg_stat_activity for long transactions, then the leak detection log, before you raise maximum-pool-size."
      - alert: HikariPoolSaturated
        expr: |
          max by (job, instance, pool) (hikaricp_connections_active)
            / max by (job, instance, pool) (hikaricp_connections_max) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.pool }} on {{ $labels.instance }} is above 90% busy"
          description: "The pool has little room for the next spike. Find the slowest holders in hikaricp_connections_usage_seconds and the database's slow query log."
      - alert: HikariThreadsWaiting
        expr: max by (job, instance, pool) (hikaricp_connections_pending) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Threads have queued for {{ $labels.pool }} connections for 5 minutes"
          description: "Requests are waiting on the pool. Compare pool size times pods with max_connections before you grow it, or shorten transactions."
      - alert: HikariSlowAcquire
        expr: max by (job, instance, pool) (hikaricp_connections_acquire_seconds_max) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Getting a {{ $labels.pool }} connection takes over 1 s"
          description: "Waits are approaching connection-timeout. Check for a leak in the logs and for a database that's slow to accept new connections."

Also alert on these log lines:

Connection is not available, request timed out after
Connection leak detection triggered for
Thread starvation or clock leap detected

Roll back

Note the current values before you change them. kubectl get deployment api -o yaml > api-before.yaml covers settings passed as environment variables. Keep application.yaml in git so the old values are one revert away.

You can also override the pool size without a rebuild. Spring Boot maps SPRING_DATASOURCE_HIKARI_MAXIMUMPOOLSIZE to maximum-pool-size:

kubectl set env deployment/api SPRING_DATASOURCE_HIKARI_MAXIMUMPOOLSIZE=10

If the bigger pool brings too many clients errors from the database, roll back now. A pool that can’t connect fails every request, which is worse than a queue. If the change only moved latency the wrong way, revert it in a quiet hour.

More recipes

Further reading