Spring Boot Virtual Threads and server.tomcat.threads.max

Size server.tomcat.threads.max from your traffic, or turn on Spring Boot virtual threads, so requests stop queueing inside Tomcat at peak.

Published

Applies to Spring Boot 3.2 and later with embedded Tomcat; virtual threads need Java 21, and Java 24 or later to stop synchronized pinning

server.tomcat.threads.max sets how many requests one Spring Boot pod works on at the same time, and it defaults to 200. On Java 21 and later you can skip the number and run each request on a virtual thread. Both choices move the limit somewhere else, so test where it lands.

What breaks at peak

Tomcat’s NIO connector splits the work in two. An acceptor takes connections, up to server.tomcat.max-connections, which defaults to 8192. A worker pool of up to server.tomcat.threads.max threads runs the requests.

When all 200 workers are busy, Tomcat parks new requests in the executor’s queue, whose capacity defaults to Integer.MAX_VALUE. Nothing logs an error. Latency climbs while the queue grows, and CPU can sit low if the workers wait on a database or an API.

A long queue hurts in three ways:

  • Health checks queue too. Liveness probes wait behind slow requests, miss their timeout, and the kubelet restarts a pod that was only busy. Its traffic moves to the remaining pods, which then queue harder.
  • Clients give up first. The load balancer times out and returns a 504, but Tomcat still runs the request later. Your pod spends threads on answers nobody reads.
  • The connection limit bites last. Past max-connections, new connections wait in the kernel’s backlog, sized by server.tomcat.accept-count (default 100). Past that, connection attempts time out or get refused.

Your clients get errors from the proxy in front of the pod. Behind nginx it’s upstream timed out (110: Connection timed out) while reading response header from upstream, which the nginx upstream timeouts recipe explains.

To confirm the pool is the limit, count busy worker threads in a thread dump. Tomcat names them after the connector:

kubectl exec deploy/api -- jcmd 1 Thread.print > threads.txt
grep -c '"http-nio-8080-exec-' threads.txt
grep -A 12 '"http-nio-8080-exec-' threads.txt | grep -E '^\s+at ' | sort | uniq -c | sort -rn | head -20

The second command groups the top frames. If most workers sit in HikariPool.getConnection, the database pool is the real limit, and the HikariCP recipe comes first. If they sit in socket reads to another service, that service sets your throughput.

The fix with Tomcat threads or Spring Boot virtual threads

You have two options. Platform threads with a sized pool work on any Java version and keep a hard cap per pod. Virtual threads remove the cap and push the limit to your downstream pools.

Option one, a sized platform thread pool. Changed lines only, in application.yaml:

server:
  shutdown: graceful
  tomcat:
    threads:
      max: 200
      min-spare: 20
    accept-count: 100
    max-connections: 8192
    mbeanregistry:
      enabled: true
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s
  • threads.max caps concurrent requests per pod. Too low and requests queue while CPU idles. Too high and each thread adds a stack, more context switches and more callers for a database pool that can’t serve them.
  • threads.min-spare keeps this many workers alive when idle. The default is 10. A higher floor saves thread creation during the first seconds of a spike.
  • accept-count is the kernel backlog once Tomcat holds max-connections. Raise it only if your load balancer retries refused connections on another pod.
  • max-connections caps open connections, idle keep-alive connections included. Keep it above your load balancer’s connection count to this pod.
  • mbeanregistry.enabled turns on the registry Micrometer reads Tomcat’s thread metrics from. It’s off by default, and without it the tomcat_threads_* metrics don’t exist.
  • shutdown: graceful, the default since Spring Boot 3.4, lets in-flight requests finish on a rolling deploy. timeout-per-shutdown-phase bounds that wait.

Size the pool from what one pod handles at peak. By Little’s law:

busy threads per pod = peak requests per second per pod * p95 response time in seconds

Set threads.max to that number plus headroom for bursts, about half again. Then check it against two limits. CPU-bound requests gain nothing from more threads than cores. Requests that need the database can’t run faster than the HikariCP pool, so threads beyond it wait there instead of in Tomcat.

I’d rather keep threads.max near what a pod can finish and add pods than raise it into the thousands. A queue you can see in metrics beats a thousand threads all waiting on ten connections.

Option two, Spring Boot virtual threads. One property, on Spring Boot 3.2 or later and Java 21 or later:

spring:
  threads:
    virtual:
      enabled: true

Tomcat then runs every request on a new virtual thread. A virtual thread that blocks on I/O releases its carrier thread, so thousands of waiting requests cost little memory. Spring Boot’s docs say threads.max and threads.min-spare have no effect once this is on. The same property moves @Async tasks and the task scheduler onto virtual threads too.

Three things change when you flip it:

  • max-connections becomes your concurrency cap. Every open connection can run a request at once, up to 8192 by default.
  • Downstream pools take the queue. A HikariCP pool of 20 now gets thousands of callers at peak instead of 200. They wait connection-timeout, then fail with Connection is not available. Put a Semaphore around calls to slow dependencies, or lower max-connections, so the queue stays bounded.
  • Pinning on Java 21 to 23. A virtual thread that blocks inside a synchronized block holds its carrier thread. There’s about one carrier per core, so a few pinned threads stall the whole pod. Java 24 removed this for synchronized (JEP 491), and Java 25 is the first LTS release with that change. Native calls still pin.

Find pinning with the JDK Flight Recorder. Record under load, then print the pinned events and their stack traces:

kubectl exec deploy/api -- jcmd 1 JFR.start name=pin duration=120s filename=/tmp/pin.jfr
# wait for the recording to finish, then
kubectl exec deploy/api -- jfr print --events jdk.VirtualThreadPinned /tmp/pin.jfr

On Java 21 to 23, I keep platform threads for any service whose JDBC driver or HTTP client blocks inside synchronized. On Java 25, I default to virtual threads for I/O-heavy services, with a semaphore in front of the database.

Test it on your system

Run a staging pod with production’s CPU and memory limits, its HikariCP pool and a real downstream. Ramp a fixed arrival rate in steps, so you see where queueing starts:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    steps: {
      executor: 'ramping-arrival-rate',
      startRate: 20,
      timeUnit: '1s',
      preAllocatedVUs: 300,
      maxVUs: 3000,
      stages: [
        { target: 100, duration: '3m' },
        { target: 100, duration: '7m' },
        { target: 200, duration: '3m' },
        { target: 200, duration: '7m' },
        { target: 300, duration: '3m' },
        { target: 300, duration: '7m' },
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<600'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/catalog/search?q=shoes`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}

Point it at one pod, through a port-forward or a test Service, and set the plateaus around that pod’s share of your peak. Run it once with platform threads and once with virtual threads if you’re comparing.

While each step holds, watch Tomcat and the database pool:

kubectl port-forward deploy/api 8080:8080 &
watch -n 5 "curl -s localhost:8080/actuator/prometheus | grep -E '^(tomcat_threads_(busy|config_max)|tomcat_connections_current|hikaricp_connections_(active|pending))'"

With platform threads, tomcat_threads_busy_threads reaching tomcat_threads_config_max_threads marks the start of the Tomcat queue. With virtual threads those two metrics mean nothing. Watch tomcat_connections_current_connections and hikaricp_connections_pending instead.

Set targets for each step before the run. A step passes when:

  • p99 and error rate stay inside the thresholds you chose.
  • Busy threads stay under the maximum with room for a burst, or pending connections return to zero on virtual threads.
  • No liveness probe failures or restarts show in kubectl get pod -w.
  • With virtual threads, no jdk.VirtualThreadPinned events point at a hot code path.

The step where p99 breaks tells you one pod’s real capacity. Use it to set your HPA target.

Keep it healthy

Micrometer exposes Tomcat’s thread and connection metrics through Spring Boot Actuator, once mbeanregistry.enabled is on. Turn on the request histogram with management.metrics.distribution.percentiles-histogram.http.server.requests=true for the latency rule.

Spring Boot 3.3.0 reported tomcat_threads_config_max_threads as -1. Upgrade to 3.3.1 or later, or the saturation rule never fires. The > 0 guard stops a -1 from turning into a negative ratio.

groups:
  - name: tomcat
    rules:
      - alert: TomcatThreadPoolSaturated
        expr: |
          (
            max by (job, instance) (tomcat_threads_busy_threads)
              / max by (job, instance) (tomcat_threads_config_max_threads) > 0.9
          )
          and on (job, instance) max by (job, instance) (tomcat_threads_config_max_threads) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tomcat on {{ $labels.instance }} is using over 90% of its worker threads"
          description: "New requests will queue inside Tomcat. Take a thread dump with jcmd 1 Thread.print to see where workers wait, then fix that dependency or add pods."
      - alert: TomcatConnectionsNearMax
        expr: |
          max by (job, instance) (tomcat_connections_current_connections)
            / max by (job, instance) (tomcat_connections_config_max_connections) > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tomcat on {{ $labels.instance }} holds over 80% of max-connections"
          description: "With virtual threads this is the concurrency cap. Check downstream pools for waiting callers before you raise server.tomcat.max-connections."
      - alert: HttpP99AboveBudget
        expr: |
          histogram_quantile(0.99,
            sum by (le, job) (rate(http_server_requests_seconds_bucket{uri!~"/actuator.*"}[5m]))
          ) > 1
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "p99 for {{ $labels.job }} is above 1 s"
          description: "Requests are slow or queued. Compare busy threads with the maximum, and HikariCP pending connections, to find which pool ran out."

Roll back

Both options are configuration, so keep application.yaml in git and note the running values first. If you pass them as environment variables, save the Deployment with kubectl get deployment api -o yaml > api-before.yaml.

Virtual threads switch off without a rebuild:

kubectl set env deployment/api SPRING_THREADS_VIRTUAL_ENABLED=false

If the change brings pool timeouts, probe failures or pinned carriers, roll back now. If it only failed to help, revert it during a low-traffic hour, since every pod restarts with cold caches.

More recipes

Further reading