On Kubernetes, java.lang.OutOfMemoryError: Java heap space and an OOMKilled pod trace back to one decision: a heap sized without the container limit in mind. Set it too small and the heap fills. Set it too close to the limit and the kernel kills the pod.
What breaks at peak
A JVM in a container can run out of memory in two places, and each one fails in its own way.
The heap fills. Live objects outgrow the maximum heap, and the JVM throws this in whichever thread asked for memory:
Exception in thread "http-nio-8080-exec-7" java.lang.OutOfMemoryError: Java heap space
The JVM doesn’t exit by default. The thread that hit the error dies, the collector runs back to back, and the pod stays Ready. It passes health checks while it serves errors and slow responses. I find this worse than a crash, because nothing restarts it.
The container gets killed. Heap plus everything outside the heap passes the memory limit. The kernel’s OOM killer ends the process with exit code 137, and Kubernetes records OOMKilled as the last termination reason. The logs stop mid-line, with no Java stack trace.
A few other OutOfMemoryError messages point outside the heap. Raising the heap makes them worse, because it takes room from the memory they need:
java.lang.OutOfMemoryError: Metaspace
java.lang.OutOfMemoryError: Direct buffer memory
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Java 8 words the last one unable to create new native thread.
Why peak traffic triggers it. More requests in flight means more live objects at once. More threads means more stack memory. Netty, gRPC and NIO clients allocate direct buffers per connection. A heap that held at normal traffic runs out when all three grow together.
Where the default goes wrong. Since JDK 10, and 8u191 on Java 8, the JVM reads the container’s memory limit. UseContainerSupport is on by default, and MaxRAMPercentage defaults to 25. A pod with a 2Gi limit gets a 512 MiB heap and leaves the other three quarters unused. Teams notice the heap errors and set -Xmx equal to the limit. That leaves nothing for native memory, and the pod gets OOMKilled instead.
Older JVMs make it worse on cgroup v2 nodes, which most current node images use, including Amazon Linux 2023 and Ubuntu 22.04. JDK 15 added cgroup v2 support, and it reached 11.0.16 and 8u372 as backports. An older build reads the host’s memory and sizes the heap from that.
The MaxRAMPercentage fix
This Deployment sizes the heap as a share of the container limit and makes a heap failure restart the pod. Replace the image and names with yours:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: registry.example.com/api:1.4.2
env:
- name: JAVA_TOOL_OPTIONS
value: >-
-XX:MaxRAMPercentage=75.0
-XX:InitialRAMPercentage=75.0
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/dumps
-Xlog:gc:stdout:time,uptime,level,tags
resources:
requests:
cpu: "2"
memory: 2Gi
limits:
memory: 2Gi
volumeMounts:
- name: dumps
mountPath: /dumps
volumes:
- name: dumps
emptyDir:
sizeLimit: 3Gi
Each setting does one job:
-XX:MaxRAMPercentage=75.0caps the heap at that share of the container limit. Too low and the heap fills while memory sits idle. Too high and native memory pushes the container over its limit.-XX:InitialRAMPercentage=75.0starts the heap at full size. The collector then never grows the heap mid-spike. The pod’s memory use looks the same from the first minute.-XX:+ExitOnOutOfMemoryErrorexits on the firstOutOfMemoryError, so Kubernetes restarts the pod. The JVM printsTerminating due to java.lang.OutOfMemoryError: Java heap spaceand exits with code 3. Kubernetes reports that asErrorwith exit code 3. Exit code 3 points at the heap, and 137 points at the container limit.-XX:+HeapDumpOnOutOfMemoryErrorwrites a heap dump to/dumpsbefore the exit. The dump is about as large as the heap. AnemptyDirsurvives container restarts, so you can copy the file out withkubectl cpafter the restart. SizesizeLimitabove the heap, since a pod over itsemptyDirlimit gets evicted.-Xlog:gc:stdout:time,uptime,level,tagslogs one line per collection with its pause, so you can read GC pressure fromkubectl logs.- Memory request equal to the limit. The scheduler then reserves on the node what the JVM will use. A pod running above its request is among the first evicted when a node runs short.
JAVA_TOOL_OPTIONS works with any launcher, and the JVM confirms it on startup with Picked up JAVA_TOOL_OPTIONS:. Check your entrypoint script and base image for an -Xmx too. -Xmx overrides MaxRAMPercentage, so a forgotten one undoes this change.
Size it from your own native memory. The container limit has to cover the heap plus metaspace, the code cache, thread stacks, direct buffers, GC data structures and the C heap. Thread stacks alone take up to -Xss each, 1 MiB by default on 64-bit Linux. A Tomcat pool of 200 platform threads can reserve 200 MiB.
Measure it in staging. Add -XX:NativeMemoryTracking=summary to the options, run your load test and read the totals at peak:
kubectl exec deploy/api -- jcmd 1 VM.native_memory summary
Add up the committed memory of every area except Java Heap, then add a margin of 10 to 20%. The heap gets the rest:
MaxRAMPercentage = (container limit - non-heap at peak - margin) / container limit * 100
As a starting point before you measure, use 75 for limits of 1 GiB and up with a normal thread count. Start at 50 to 60 for smaller containers, or for services with hundreds of threads or heavy direct buffer use. I size from the native memory numbers at peak, not from whatever percentage the last service used.
jcmd ships with the JDK, not with JRE-only or distroless images. If your image lacks it, run the measurement from a staging image built on the full JDK. Remove NativeMemoryTracking once you have the numbers, since tracking costs a little CPU and memory.
Two more settings help when native memory keeps growing. -XX:MaxDirectMemorySize caps direct buffers, which otherwise default to the heap’s maximum size. On glibc images with many cores, MALLOC_ARENA_MAX=2 as an environment variable stops the C heap from creating an arena per core.
Managed container platforms. The flags work the same on ECS and Fargate, Cloud Run, Azure Container Apps and App Service for containers. The limit the JVM reads is the task or container memory on ECS, and the instance memory on Cloud Run and Container Apps. On Cloud Run, files written to the container’s file system count against that memory. Send heap dumps to a mounted volume there, or leave the dump flag off.
Test it on your system
First check what the JVM sees inside the pod. The kubectl exec session inherits JAVA_TOOL_OPTIONS, so this prints the effective values:
kubectl exec deploy/api -- java -XX:+PrintFlagsFinal -version \
| grep -E ' (MaxHeapSize|MaxRAMPercentage|UseContainerSupport) '
kubectl exec deploy/api -- java -XshowSettings:system -version
MaxHeapSize is in bytes and should equal the limit times your percentage. If it matches the host’s memory instead, the JVM can’t read the cgroup, and you need a newer build.
Then hold your peak arrival rate for long enough to fill the heap. Thirty minutes catches slow growth that a five-minute run misses. This k6 script holds a fixed rate, so a slow pod gets the same load as a fast one. Set rate to your peak requests per second for one pod times the replica count:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: 300,
timeUnit: '1s',
duration: '30m',
preAllocatedVUs: 200,
maxVUs: 1000,
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(99)<800'],
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/api/orders?limit=50`);
check(res, { 'status is 200': (r) => r.status === 200 });
}
Run it with k6 run -e BASE_URL=https://api.staging.example.com heap-test.js. Pick an endpoint that builds large responses, since that’s where heap use peaks. While it runs, watch the pod and the JVM:
kubectl top pod -l app=api --containers
kubectl get pod -l app=api -w
kubectl logs -f deploy/api | grep -E 'Pause Full|OutOfMemoryError'
kubectl exec deploy/api -- jcmd 1 GC.heap_info
The GC log shows the heap before and after each collection. Watch the after-GC values. A healthy heap draws a sawtooth whose low points stay level. Low points that climb through the test mean live data still outgrows the heap.
Decide your pass marks before the run. Mine look like this:
- No restarts, no
OOMKilledand noOutOfMemoryErrorfor the whole run. - No
Pause Fulllines in the GC log after warm-up. - Container working set stays under a share of the limit you pick, such as 90%.
- Error rate and p99 stay inside the k6 thresholds you set for this endpoint.
Then test the failure path on purpose. Lower MaxRAMPercentage in staging until the heap runs out under load. The pod should restart with exit code 3, and a .hprof file should appear in /dumps.
If the heap holds but pauses grow with load, read the GC pause recipe. If the pod still gets OOMKilled with a sound heap, the Kubernetes OOMKilled recipe covers the container side.
Keep it healthy
Expose JVM metrics with Micrometer, which Spring Boot Actuator wires up when micrometer-registry-prometheus is on the classpath. Scrape pod metrics with cAdvisor and kube-state-metrics. Replace api with your container name:
groups:
- name: java-memory
rules:
- alert: JvmLiveHeapHigh
expr: |
max by (job, instance) (jvm_gc_live_data_size_bytes)
/ max by (job, instance) (jvm_gc_max_data_size_bytes) > 0.85
for: 15m
labels:
severity: warning
annotations:
summary: "Live heap above 85% of the old generation on {{ $labels.instance }}"
description: "Data that survives GC is close to the heap's maximum. Take a heap dump with jcmd 1 GC.heap_dump and look for growth, or raise MaxRAMPercentage if native memory has room."
- alert: JavaContainerOOMKilled
expr: |
increase(kube_pod_container_status_restarts_total{container="api"}[10m]) > 0
and on (namespace, pod, container)
kube_pod_container_status_last_terminated_reason{container="api", reason="OOMKilled"} == 1
for: 0m
labels:
severity: critical
annotations:
summary: "{{ $labels.pod }} was OOMKilled"
description: "Heap plus native memory passed the container limit. Lower MaxRAMPercentage or raise the limit, and check VM.native_memory summary for the area that grew."
- alert: JavaContainerMemoryNearLimit
expr: |
max by (namespace, pod, container) (container_memory_working_set_bytes{container="api"})
/ on (namespace, pod, container)
max by (namespace, pod, container) (kube_pod_container_resource_limits{container="api", resource="memory"}) > 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.pod }} is above 90% of its memory limit"
description: "The next spike may OOMKill this pod. Compare the heap size with the limit and measure native memory before you change either."
Also alert on these lines in your logs. Each one means the JVM ran out somewhere:
java.lang.OutOfMemoryError: Java heap space
Terminating due to java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Metaspace
java.lang.OutOfMemoryError: Direct buffer memory
Roll back
Save the running spec before you apply the change:
kubectl get deployment api -o yaml > api-before.yaml
kubectl rollout history deployment/api
To undo it, run kubectl rollout undo deployment/api or apply the saved file. If pods start restarting with OOMKilled after the change, roll back now and lower the percentage before you try again. For any other reason, roll back during a quiet hour, since every pod restarts with an empty heap and cold caches.
Copy any heap dumps out first. The emptyDir goes away when its pod is deleted.
More recipes
- G1 vs ZGC and How to Catch Java GC Pause Spikes
Alert on Java GC pause latency and full GCs before users notice p99 spikes, then use the same metrics to decide between G1 and ZGC.
- Fix HikariCP Connection Is Not Available, Request Timed Out
Size the HikariCP connection pool in Spring Boot from your database limit and pod count, so requests stop timing out while they wait for a connection.
- Spring Boot Virtual Threads and server.tomcat.threads.max
Size server.tomcat.threads.max from your traffic, or turn on Spring Boot virtual threads, so requests stop queueing inside Tomcat at peak.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.