Fix Java Heap Space OOM in Kubernetes Containers

Size the JVM heap from the container limit with MaxRAMPercentage and leave room for native memory, so pods stop failing with heap errors or OOMKilled.

Published

Applies to OpenJDK 17 and later; on cgroup v2 nodes, 11.0.16 and 8u372 or later also read the container limit

On Kubernetes, java.lang.OutOfMemoryError: Java heap space and an OOMKilled pod trace back to one decision: a heap sized without the container limit in mind. Set it too small and the heap fills. Set it too close to the limit and the kernel kills the pod.

What breaks at peak

A JVM in a container can run out of memory in two places, and each one fails in its own way.

The heap fills. Live objects outgrow the maximum heap, and the JVM throws this in whichever thread asked for memory:

Exception in thread "http-nio-8080-exec-7" java.lang.OutOfMemoryError: Java heap space

The JVM doesn’t exit by default. The thread that hit the error dies, the collector runs back to back, and the pod stays Ready. It passes health checks while it serves errors and slow responses. I find this worse than a crash, because nothing restarts it.

The container gets killed. Heap plus everything outside the heap passes the memory limit. The kernel’s OOM killer ends the process with exit code 137, and Kubernetes records OOMKilled as the last termination reason. The logs stop mid-line, with no Java stack trace.

A few other OutOfMemoryError messages point outside the heap. Raising the heap makes them worse, because it takes room from the memory they need:

java.lang.OutOfMemoryError: Metaspace
java.lang.OutOfMemoryError: Direct buffer memory
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

Java 8 words the last one unable to create new native thread.

Why peak traffic triggers it. More requests in flight means more live objects at once. More threads means more stack memory. Netty, gRPC and NIO clients allocate direct buffers per connection. A heap that held at normal traffic runs out when all three grow together.

Where the default goes wrong. Since JDK 10, and 8u191 on Java 8, the JVM reads the container’s memory limit. UseContainerSupport is on by default, and MaxRAMPercentage defaults to 25. A pod with a 2Gi limit gets a 512 MiB heap and leaves the other three quarters unused. Teams notice the heap errors and set -Xmx equal to the limit. That leaves nothing for native memory, and the pod gets OOMKilled instead.

Older JVMs make it worse on cgroup v2 nodes, which most current node images use, including Amazon Linux 2023 and Ubuntu 22.04. JDK 15 added cgroup v2 support, and it reached 11.0.16 and 8u372 as backports. An older build reads the host’s memory and sizes the heap from that.

The MaxRAMPercentage fix

This Deployment sizes the heap as a share of the container limit and makes a heap failure restart the pod. Replace the image and names with yours:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.4.2
          env:
            - name: JAVA_TOOL_OPTIONS
              value: >-
                -XX:MaxRAMPercentage=75.0
                -XX:InitialRAMPercentage=75.0
                -XX:+ExitOnOutOfMemoryError
                -XX:+HeapDumpOnOutOfMemoryError
                -XX:HeapDumpPath=/dumps
                -Xlog:gc:stdout:time,uptime,level,tags
          resources:
            requests:
              cpu: "2"
              memory: 2Gi
            limits:
              memory: 2Gi
          volumeMounts:
            - name: dumps
              mountPath: /dumps
      volumes:
        - name: dumps
          emptyDir:
            sizeLimit: 3Gi

Each setting does one job:

  • -XX:MaxRAMPercentage=75.0 caps the heap at that share of the container limit. Too low and the heap fills while memory sits idle. Too high and native memory pushes the container over its limit.
  • -XX:InitialRAMPercentage=75.0 starts the heap at full size. The collector then never grows the heap mid-spike. The pod’s memory use looks the same from the first minute.
  • -XX:+ExitOnOutOfMemoryError exits on the first OutOfMemoryError, so Kubernetes restarts the pod. The JVM prints Terminating due to java.lang.OutOfMemoryError: Java heap space and exits with code 3. Kubernetes reports that as Error with exit code 3. Exit code 3 points at the heap, and 137 points at the container limit.
  • -XX:+HeapDumpOnOutOfMemoryError writes a heap dump to /dumps before the exit. The dump is about as large as the heap. An emptyDir survives container restarts, so you can copy the file out with kubectl cp after the restart. Size sizeLimit above the heap, since a pod over its emptyDir limit gets evicted.
  • -Xlog:gc:stdout:time,uptime,level,tags logs one line per collection with its pause, so you can read GC pressure from kubectl logs.
  • Memory request equal to the limit. The scheduler then reserves on the node what the JVM will use. A pod running above its request is among the first evicted when a node runs short.

JAVA_TOOL_OPTIONS works with any launcher, and the JVM confirms it on startup with Picked up JAVA_TOOL_OPTIONS:. Check your entrypoint script and base image for an -Xmx too. -Xmx overrides MaxRAMPercentage, so a forgotten one undoes this change.

Size it from your own native memory. The container limit has to cover the heap plus metaspace, the code cache, thread stacks, direct buffers, GC data structures and the C heap. Thread stacks alone take up to -Xss each, 1 MiB by default on 64-bit Linux. A Tomcat pool of 200 platform threads can reserve 200 MiB.

Measure it in staging. Add -XX:NativeMemoryTracking=summary to the options, run your load test and read the totals at peak:

kubectl exec deploy/api -- jcmd 1 VM.native_memory summary

Add up the committed memory of every area except Java Heap, then add a margin of 10 to 20%. The heap gets the rest:

MaxRAMPercentage = (container limit - non-heap at peak - margin) / container limit * 100

As a starting point before you measure, use 75 for limits of 1 GiB and up with a normal thread count. Start at 50 to 60 for smaller containers, or for services with hundreds of threads or heavy direct buffer use. I size from the native memory numbers at peak, not from whatever percentage the last service used.

jcmd ships with the JDK, not with JRE-only or distroless images. If your image lacks it, run the measurement from a staging image built on the full JDK. Remove NativeMemoryTracking once you have the numbers, since tracking costs a little CPU and memory.

Two more settings help when native memory keeps growing. -XX:MaxDirectMemorySize caps direct buffers, which otherwise default to the heap’s maximum size. On glibc images with many cores, MALLOC_ARENA_MAX=2 as an environment variable stops the C heap from creating an arena per core.

Managed container platforms. The flags work the same on ECS and Fargate, Cloud Run, Azure Container Apps and App Service for containers. The limit the JVM reads is the task or container memory on ECS, and the instance memory on Cloud Run and Container Apps. On Cloud Run, files written to the container’s file system count against that memory. Send heap dumps to a mounted volume there, or leave the dump flag off.

Test it on your system

First check what the JVM sees inside the pod. The kubectl exec session inherits JAVA_TOOL_OPTIONS, so this prints the effective values:

kubectl exec deploy/api -- java -XX:+PrintFlagsFinal -version \
  | grep -E ' (MaxHeapSize|MaxRAMPercentage|UseContainerSupport) '
kubectl exec deploy/api -- java -XshowSettings:system -version

MaxHeapSize is in bytes and should equal the limit times your percentage. If it matches the host’s memory instead, the JVM can’t read the cgroup, and you need a newer build.

Then hold your peak arrival rate for long enough to fill the heap. Thirty minutes catches slow growth that a five-minute run misses. This k6 script holds a fixed rate, so a slow pod gets the same load as a fast one. Set rate to your peak requests per second for one pod times the replica count:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: 300,
      timeUnit: '1s',
      duration: '30m',
      preAllocatedVUs: 200,
      maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<800'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/orders?limit=50`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}

Run it with k6 run -e BASE_URL=https://api.staging.example.com heap-test.js. Pick an endpoint that builds large responses, since that’s where heap use peaks. While it runs, watch the pod and the JVM:

kubectl top pod -l app=api --containers
kubectl get pod -l app=api -w
kubectl logs -f deploy/api | grep -E 'Pause Full|OutOfMemoryError'
kubectl exec deploy/api -- jcmd 1 GC.heap_info

The GC log shows the heap before and after each collection. Watch the after-GC values. A healthy heap draws a sawtooth whose low points stay level. Low points that climb through the test mean live data still outgrows the heap.

Decide your pass marks before the run. Mine look like this:

  • No restarts, no OOMKilled and no OutOfMemoryError for the whole run.
  • No Pause Full lines in the GC log after warm-up.
  • Container working set stays under a share of the limit you pick, such as 90%.
  • Error rate and p99 stay inside the k6 thresholds you set for this endpoint.

Then test the failure path on purpose. Lower MaxRAMPercentage in staging until the heap runs out under load. The pod should restart with exit code 3, and a .hprof file should appear in /dumps.

If the heap holds but pauses grow with load, read the GC pause recipe. If the pod still gets OOMKilled with a sound heap, the Kubernetes OOMKilled recipe covers the container side.

Keep it healthy

Expose JVM metrics with Micrometer, which Spring Boot Actuator wires up when micrometer-registry-prometheus is on the classpath. Scrape pod metrics with cAdvisor and kube-state-metrics. Replace api with your container name:

groups:
  - name: java-memory
    rules:
      - alert: JvmLiveHeapHigh
        expr: |
          max by (job, instance) (jvm_gc_live_data_size_bytes)
            / max by (job, instance) (jvm_gc_max_data_size_bytes) > 0.85
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Live heap above 85% of the old generation on {{ $labels.instance }}"
          description: "Data that survives GC is close to the heap's maximum. Take a heap dump with jcmd 1 GC.heap_dump and look for growth, or raise MaxRAMPercentage if native memory has room."
      - alert: JavaContainerOOMKilled
        expr: |
          increase(kube_pod_container_status_restarts_total{container="api"}[10m]) > 0
            and on (namespace, pod, container)
          kube_pod_container_status_last_terminated_reason{container="api", reason="OOMKilled"} == 1
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.pod }} was OOMKilled"
          description: "Heap plus native memory passed the container limit. Lower MaxRAMPercentage or raise the limit, and check VM.native_memory summary for the area that grew."
      - alert: JavaContainerMemoryNearLimit
        expr: |
          max by (namespace, pod, container) (container_memory_working_set_bytes{container="api"})
            / on (namespace, pod, container)
          max by (namespace, pod, container) (kube_pod_container_resource_limits{container="api", resource="memory"}) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.pod }} is above 90% of its memory limit"
          description: "The next spike may OOMKill this pod. Compare the heap size with the limit and measure native memory before you change either."

Also alert on these lines in your logs. Each one means the JVM ran out somewhere:

java.lang.OutOfMemoryError: Java heap space
Terminating due to java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Metaspace
java.lang.OutOfMemoryError: Direct buffer memory

Roll back

Save the running spec before you apply the change:

kubectl get deployment api -o yaml > api-before.yaml
kubectl rollout history deployment/api

To undo it, run kubectl rollout undo deployment/api or apply the saved file. If pods start restarting with OOMKilled after the change, roll back now and lower the percentage before you try again. For any other reason, roll back during a quiet hour, since every pod restarts with an empty heap and cold caches.

Copy any heap dumps out first. The emptyDir goes away when its pod is deleted.

More recipes

Further reading