Fix OOMKilled and Exit Code 137 in Kubernetes

Set Kubernetes memory limits from measured peak usage plus headroom, and cap the runtime below the limit, so pods stop dying with exit code 137.

Published

Applies to Kubernetes 1.28 and later on cgroup v2 nodes; in-place resize needs 1.33 or later

OOMKilled with exit code 137 means the kernel killed your container for using more memory than its limit. Exit code 137 is 128 plus signal 9, SIGKILL. Peak traffic brings more requests in flight, bigger buffers and fuller caches, so the kill lands at the worst time. The fix starts with finding which of three killers fired.

What breaks at peak

kubectl describe pod shows the last termination:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

Three different mechanisms end a pod for memory, and each needs a different fix:

What you seeWhat killed itFix
Reason: OOMKilled, Exit Code: 137, restart count risingThe container’s memory cgroup hit its limitRaise the limit or cut the app’s usage
Pod Status: Failed, Reason: Evicted, message The node was low on resource: memory.The kubelet evicted it under node memory pressureRaise requests to match usage, or add node memory
Exit Code: 137 with no OOMKilled reasonSIGKILL for another cause, such as a grace period that ran outSee the graceful shutdown recipe

The node’s kernel log records each container kill. On the node, journalctl -k or dmesg shows a line that starts like this:

Memory cgroup out of memory: Killed process 48213 (java)

A system-wide OOM prints Out of memory: Killed process instead. That one means the whole node ran out. Pods that use far more than they request are the common cause. The Linux OOM killer recipe covers the host side.

The eviction message carries more detail. The kubelet appends the threshold and the container that used more than its request:

The node was low on resource: memory. Threshold quantity: 100Mi, available: 85432Ki. Container app was using 1843Mi, request is 512Mi, has larger consumption of memory.

The kubelet evicts pods whose usage exceeds their request first. A memory request far below real usage makes your pod the first to go.

The limit counts more than your heap. The kubelet and the kernel measure the container’s whole cgroup. That covers the runtime heap, native memory and thread stacks. Memory-backed emptyDir volumes and page cache count too. cAdvisor reports container_memory_working_set_bytes, which subtracts inactive page cache, and the eviction manager reads the same figure. A pod that writes large files can show a high working set while its process stays small, so compare with container_memory_rss.

One process or the whole container. On cgroup v2, Kubernetes 1.28 and later sets memory.oom.group, so an OOM kill takes down every process in the container together. Before that, or with the kubelet’s singleProcessOOMKill: true, the kernel kills one process. A Gunicorn or PHP-FPM worker can then die with no OOMKilled reason on the pod. The master logs the lost worker and starts another.

Restarts under load. Each OOM kill restarts the container, and repeated kills push it into CrashLoopBackOff with growing delays. The pod drops out of the endpoints each time. The remaining pods take its traffic, grow and hit their own limits.

Set Kubernetes memory limits from real usage

Measure before you size anything. Pull a week that includes your peak from Prometheus:

# peak working set per container over 7 days
max_over_time(container_memory_working_set_bytes{container!=""}[7d])

# working set as a share of the limit, right now
max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
  / on (namespace, pod, container)
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})

Then set the resources. Changed lines only, in the Deployment’s container spec:

resources:
  requests:
    memory: 1280Mi    # equal to the limit
    cpu: 500m
  limits:
    memory: 1280Mi    # peak working set × 1.25

Limit from the peak. Take the highest working set from a week that includes peak traffic, and add 20 to 30% for growth and spikes. A container that already hits the limit has a capped graph, so its true peak is unknown. Raise its limit for a while first, measure, then size it.

Request equal to the limit. I set the memory request equal to the limit on every request-serving pod. Memory can’t be throttled like CPU, so a pod that bursts above its request can only be evicted or killed. With the two equal, the scheduler reserves what the pod uses. The pod also drops down the eviction order. Set the CPU request equal to its limit as well, and the pod gets the Guaranteed QoS class and the lowest OOM score.

Cap the runtime below the limit. Most runtimes size their heap from the memory they see. The heap plus native memory must fit under the container limit, with room left over:

A runtime that hits its own cap throws its own error, such as java.lang.OutOfMemoryError: Java heap space. That message points at the code. An OOM kill gives you nothing but exit code 137.

Memory-backed volumes. An emptyDir with medium: Memory is tmpfs, and its files count against the container’s memory. Set a sizeLimit on it, and count it in the limit.

Recommendations from VPA. The Vertical Pod Autoscaler watches usage and recommends requests. Run it in recommendation mode first, so it changes nothing:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"

After a week with a peak in it, read the target and upperBound values with kubectl describe vpa api -n shop. Don’t let VPA and an HPA both act on memory for the same Deployment, because they fight. GKE has VPA built in, turned on with --enable-vertical-pod-autoscaling, and Autopilot runs it by default. AKS installs it with az aks update --enable-vpa. On EKS and OKE you install it from the Kubernetes autoscaler project.

Node memory. Every provider reserves memory for the kubelet and the OS, so Allocatable sits below the VM’s size. kubectl describe node shows both. The sum of memory requests on a node can’t exceed allocatable, but the sum of limits can. Requests equal to limits keep a node from overcommitting memory at all.

Resize without a restart. From Kubernetes 1.33, in-place pod resize lets you raise a running pod’s memory without a restart, through the pod’s resize subresource. It went stable in 1.35. It helps during an incident, but the Deployment’s template still sets the value for new pods, so fix it there too.

Test it on your system

Memory problems need two tests. A peak test shows the working set at your highest traffic. A soak test shows growth over hours, which a short run misses.

Run a fixed-rate k6 test against staging at your peak rate, with the same limits and runtime flags as production. Use the constant-arrival-rate script from the CPU throttling recipe, then run it again for several hours at your normal daily rate:

k6 run -e BASE_URL=https://staging.example.com -e RATE=300 peak-test.js

Use endpoints that allocate the most memory: large exports, image work, report generation and big JSON responses. Uploads and long-lived connections count too.

While it runs, watch the working set against the limit and the restart count:

watch -n 10 "kubectl top pods -n shop -l app=api --containers; \
  kubectl get pods -n shop -l app=api \
    -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,LAST:.status.containerStatuses[0].lastState.terminated.reason"

Graph container_memory_working_set_bytes for the pods during the soak. A line that climbs and never levels off is a leak, and no limit fixes it.

Set the pass bar before the run. For example, no restarts, no OOMKilled and no evictions in either test. Peak working set stays below 80% of the limit, and the soak graph flattens within the first hour. If the runtime has its own cap, its heap graph should level off under that cap too.

Keep it healthy

Scrape cAdvisor through the kubelet and kube-state-metrics. These rules catch the kill itself, a container close to its limit, memory evictions and nodes under pressure:

groups:
  - name: kubernetes-memory
    rules:
      - alert: ContainerOOMKilled
        expr: |
          increase(kube_pod_container_status_restarts_total[10m]) > 0
            and on (namespace, pod, container)
          kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} {{ $labels.container }} was OOM-killed"
          description: "The container outgrew its memory limit. Compare peak container_memory_working_set_bytes with the limit, check the runtime's heap cap, then raise the limit or fix the leak."
      - alert: ContainerMemoryNearLimit
        expr: |
          max by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
            / on (namespace, pod, container)
          max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
            > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} {{ $labels.container }} above 90% of its memory limit"
          description: "An OOM kill is close. Compare with container_memory_rss to rule out page cache, then raise the limit or lower the runtime's heap cap."
      - alert: PodEvictedForMemory
        expr: |
          sum by (namespace, pod) (kube_pod_status_reason{reason="Evicted"}) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} was evicted"
          description: "Run kubectl describe pod for the eviction message. If it says the node was low on memory, raise memory requests to match usage on the pods it names."
      - alert: NodeMemoryPressure
        expr: |
          kube_node_status_condition{condition="MemoryPressure", status="true"} == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.node }} reports MemoryPressure"
          description: "The kubelet is evicting pods. Run kubectl top pods on the node to find pods above their requests, and raise their requests or add node memory."

ContainerOOMKilled uses for: 0m because a kill doesn’t recover on its own. PodEvictedForMemory matches any eviction reason, so read the message before acting. Set the 0.9 line from your own headroom target.

Roll back

Save the current resources first:

kubectl get deploy api -n shop \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}' \
  > api-memory-before.txt

Revert through Git or Helm, or set the old values with kubectl:

kubectl set resources deploy/api -n shop -c app --requests=memory=512Mi --limits=memory=1Gi

Roll back now if higher requests leave new pods Pending because no node has room. Add nodes or shrink other requests, then try again. A lower limit brings the kills back, so revert one only during a quiet hour, after the soak test confirms the new figure.

More recipes

Further reading