OOMKilled with exit code 137 means the kernel killed your container for using more memory than its limit. Exit code 137 is 128 plus signal 9, SIGKILL. Peak traffic brings more requests in flight, bigger buffers and fuller caches, so the kill lands at the worst time. The fix starts with finding which of three killers fired.
What breaks at peak
kubectl describe pod shows the last termination:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Three different mechanisms end a pod for memory, and each needs a different fix:
| What you see | What killed it | Fix |
|---|---|---|
Reason: OOMKilled, Exit Code: 137, restart count rising | The container’s memory cgroup hit its limit | Raise the limit or cut the app’s usage |
Pod Status: Failed, Reason: Evicted, message The node was low on resource: memory. | The kubelet evicted it under node memory pressure | Raise requests to match usage, or add node memory |
Exit Code: 137 with no OOMKilled reason | SIGKILL for another cause, such as a grace period that ran out | See the graceful shutdown recipe |
The node’s kernel log records each container kill. On the node, journalctl -k or dmesg shows a line that starts like this:
Memory cgroup out of memory: Killed process 48213 (java)
A system-wide OOM prints Out of memory: Killed process instead. That one means the whole node ran out. Pods that use far more than they request are the common cause. The Linux OOM killer recipe covers the host side.
The eviction message carries more detail. The kubelet appends the threshold and the container that used more than its request:
The node was low on resource: memory. Threshold quantity: 100Mi, available: 85432Ki. Container app was using 1843Mi, request is 512Mi, has larger consumption of memory.
The kubelet evicts pods whose usage exceeds their request first. A memory request far below real usage makes your pod the first to go.
The limit counts more than your heap. The kubelet and the kernel measure the container’s whole cgroup. That covers the runtime heap, native memory and thread stacks. Memory-backed emptyDir volumes and page cache count too. cAdvisor reports container_memory_working_set_bytes, which subtracts inactive page cache, and the eviction manager reads the same figure. A pod that writes large files can show a high working set while its process stays small, so compare with container_memory_rss.
One process or the whole container. On cgroup v2, Kubernetes 1.28 and later sets memory.oom.group, so an OOM kill takes down every process in the container together. Before that, or with the kubelet’s singleProcessOOMKill: true, the kernel kills one process. A Gunicorn or PHP-FPM worker can then die with no OOMKilled reason on the pod. The master logs the lost worker and starts another.
Restarts under load. Each OOM kill restarts the container, and repeated kills push it into CrashLoopBackOff with growing delays. The pod drops out of the endpoints each time. The remaining pods take its traffic, grow and hit their own limits.
Set Kubernetes memory limits from real usage
Measure before you size anything. Pull a week that includes your peak from Prometheus:
# peak working set per container over 7 days
max_over_time(container_memory_working_set_bytes{container!=""}[7d])
# working set as a share of the limit, right now
max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
/ on (namespace, pod, container)
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
Then set the resources. Changed lines only, in the Deployment’s container spec:
resources:
requests:
memory: 1280Mi # equal to the limit
cpu: 500m
limits:
memory: 1280Mi # peak working set × 1.25
Limit from the peak. Take the highest working set from a week that includes peak traffic, and add 20 to 30% for growth and spikes. A container that already hits the limit has a capped graph, so its true peak is unknown. Raise its limit for a while first, measure, then size it.
Request equal to the limit. I set the memory request equal to the limit on every request-serving pod. Memory can’t be throttled like CPU, so a pod that bursts above its request can only be evicted or killed. With the two equal, the scheduler reserves what the pod uses. The pod also drops down the eviction order. Set the CPU request equal to its limit as well, and the pod gets the Guaranteed QoS class and the lowest OOM score.
Cap the runtime below the limit. Most runtimes size their heap from the memory they see. The heap plus native memory must fit under the container limit, with room left over:
- Java. Set
-XX:MaxRAMPercentageso the heap leaves room for metaspace, threads and direct buffers. The JVM heap recipe has the sizing. - Node.js. Set
--max-old-space-sizeper process, as in the Node.js heap recipe. - Go. Set
GOMEMLIMITto 80 to 90% of the limit. The GOMAXPROCS and GOMEMLIMIT recipe explains the trade-off. - Python, Ruby and PHP. Memory grows with the worker count. Size workers from the limit, as in the Gunicorn workers recipe.
- Data stores. RabbitMQ and MongoDB can size their memory from the host’s RAM instead of the pod’s limit. The RabbitMQ memory alarm recipe and the MongoDB WiredTiger cache recipe set absolute limits.
A runtime that hits its own cap throws its own error, such as java.lang.OutOfMemoryError: Java heap space. That message points at the code. An OOM kill gives you nothing but exit code 137.
Memory-backed volumes. An emptyDir with medium: Memory is tmpfs, and its files count against the container’s memory. Set a sizeLimit on it, and count it in the limit.
Recommendations from VPA. The Vertical Pod Autoscaler watches usage and recommends requests. Run it in recommendation mode first, so it changes nothing:
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: api
namespace: shop
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: api
updatePolicy:
updateMode: "Off"
After a week with a peak in it, read the target and upperBound values with kubectl describe vpa api -n shop. Don’t let VPA and an HPA both act on memory for the same Deployment, because they fight. GKE has VPA built in, turned on with --enable-vertical-pod-autoscaling, and Autopilot runs it by default. AKS installs it with az aks update --enable-vpa. On EKS and OKE you install it from the Kubernetes autoscaler project.
Node memory. Every provider reserves memory for the kubelet and the OS, so Allocatable sits below the VM’s size. kubectl describe node shows both. The sum of memory requests on a node can’t exceed allocatable, but the sum of limits can. Requests equal to limits keep a node from overcommitting memory at all.
Resize without a restart. From Kubernetes 1.33, in-place pod resize lets you raise a running pod’s memory without a restart, through the pod’s resize subresource. It went stable in 1.35. It helps during an incident, but the Deployment’s template still sets the value for new pods, so fix it there too.
Test it on your system
Memory problems need two tests. A peak test shows the working set at your highest traffic. A soak test shows growth over hours, which a short run misses.
Run a fixed-rate k6 test against staging at your peak rate, with the same limits and runtime flags as production. Use the constant-arrival-rate script from the CPU throttling recipe, then run it again for several hours at your normal daily rate:
k6 run -e BASE_URL=https://staging.example.com -e RATE=300 peak-test.js
Use endpoints that allocate the most memory: large exports, image work, report generation and big JSON responses. Uploads and long-lived connections count too.
While it runs, watch the working set against the limit and the restart count:
watch -n 10 "kubectl top pods -n shop -l app=api --containers; \
kubectl get pods -n shop -l app=api \
-o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,LAST:.status.containerStatuses[0].lastState.terminated.reason"
Graph container_memory_working_set_bytes for the pods during the soak. A line that climbs and never levels off is a leak, and no limit fixes it.
Set the pass bar before the run. For example, no restarts, no OOMKilled and no evictions in either test. Peak working set stays below 80% of the limit, and the soak graph flattens within the first hour. If the runtime has its own cap, its heap graph should level off under that cap too.
Keep it healthy
Scrape cAdvisor through the kubelet and kube-state-metrics. These rules catch the kill itself, a container close to its limit, memory evictions and nodes under pressure:
groups:
- name: kubernetes-memory
rules:
- alert: ContainerOOMKilled
expr: |
increase(kube_pod_container_status_restarts_total[10m]) > 0
and on (namespace, pod, container)
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
for: 0m
labels:
severity: critical
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} {{ $labels.container }} was OOM-killed"
description: "The container outgrew its memory limit. Compare peak container_memory_working_set_bytes with the limit, check the runtime's heap cap, then raise the limit or fix the leak."
- alert: ContainerMemoryNearLimit
expr: |
max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
/ on (namespace, pod, container)
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
> 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} {{ $labels.container }} above 90% of its memory limit"
description: "An OOM kill is close. Compare with container_memory_rss to rule out page cache, then raise the limit or lower the runtime's heap cap."
- alert: PodEvictedForMemory
expr: |
sum by (namespace, pod) (kube_pod_status_reason{reason="Evicted"}) > 0
for: 0m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} was evicted"
description: "Run kubectl describe pod for the eviction message. If it says the node was low on memory, raise memory requests to match usage on the pods it names."
- alert: NodeMemoryPressure
expr: |
kube_node_status_condition{condition="MemoryPressure", status="true"} == 1
for: 5m
labels:
severity: warning
annotations:
summary: "Node {{ $labels.node }} reports MemoryPressure"
description: "The kubelet is evicting pods. Run kubectl top pods on the node to find pods above their requests, and raise their requests or add node memory."
ContainerOOMKilled uses for: 0m because a kill doesn’t recover on its own. PodEvictedForMemory matches any eviction reason, so read the message before acting. Set the 0.9 line from your own headroom target.
Roll back
Save the current resources first:
kubectl get deploy api -n shop \
-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}' \
> api-memory-before.txt
Revert through Git or Helm, or set the old values with kubectl:
kubectl set resources deploy/api -n shop -c app --requests=memory=512Mi --limits=memory=1Gi
Roll back now if higher requests leave new pods Pending because no node has room. Add nodes or shrink other requests, then try again. A lower limit brings the kills back, so revert one only during a quiet hour, after the soak test confirms the new figure.
More recipes
- Kubernetes CPU Throttling with Spare CPU on the Node
Size CPU requests from real usage and give limits room above them, or drop them, so CFS quota stops pausing pods on a node with CPU to spare.
- Kubernetes Graceful Shutdown Without 502s on Deploy
Hold each old pod in a preStop sleep until the load balancers drop it, then drain open requests on SIGTERM, so a rollout ends with no 502s.
- HPA Scaling Behavior That Keeps Up with Peak Traffic
Scale on a metric that leads the load, lower the target and keep spare node room, so the HPA adds pods before peak traffic starts to queue.
- Pod Disruption Budget That Survives Node Drains
Give each service a PodDisruptionBudget and spread its pods across nodes and zones, so drains and upgrades evict one pod at a time.