Fix JavaScript Heap Out of Memory in Node.js

Size the V8 heap to the container with max-old-space-size, then find what grows at peak, so Node.js stops crashing with JavaScript heap out of memory.

Published

Applies to Node.js 20.3 and later; --max-old-space-size-percentage needs 22.21 or 24.6 and later

A Node.js process that runs out of JavaScript heap memory aborts with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory. Every request it held fails at once, and Kubernetes restarts the container. A bigger limit fixes some cases and only delays others. Find out which one you have before you change a flag.

What breaks at peak

V8 caps the heap at a size it picks at startup. As the heap nears that cap, the garbage collector runs more often and takes longer each time. Your event loop pauses for every collection, so latency climbs for a few minutes before the crash. Then V8 gives up and Node prints one of these lines to stderr:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

The first is the common form on Node 16 and later. All three still exist in the V8 that ships with Node 20 to 24. The part before “Allocation failed” names where V8 gave up, and the fix is the same for all three. Above them, Node prints a <--- Last few GCs ---> block with the last mark-compact runs.

The process aborts with SIGABRT, so it exits with code 134. That tells you which of two failures you have:

  • Heap limit reached. Exit code 134 and the FATAL ERROR line in the logs. V8’s own cap stopped the process. Kubernetes reports the last state as Error.
  • Container limit reached. Exit code 137 and OOMKilled, with no FATAL ERROR line. The heap cap sits above the container’s memory, or memory outside the heap grew. The Linux OOM killer recipe covers that side.

Peak traffic causes heap growth in three ways:

  • More work in flight. Each open request holds its body, its parsed JSON and its response. Twice the concurrency needs about twice the heap for that working set.
  • Caches and queues without a cap. An in-process Map used as a cache, or an array of pending jobs, grows with traffic and never shrinks.
  • A leak. Listeners, closures or timers that hold on to each request. Traffic makes it grow faster, so it crashes at peak first.

The max-old-space-size fix

Check the limit you run with now. Node reads the container’s cgroup memory limit, on cgroup v2 from Node 20.3, and V8 sizes its default heap from that. The default often leaves memory unused in a large container and can sit too close to the limit in a small one. Read it inside the running pod:

kubectl exec deploy/api -- node -e "const v8 = require('node:v8'); \
  const s = v8.getHeapStatistics(); \
  console.log('heap limit MiB', Math.round(s.heap_size_limit / 1048576)); \
  console.log('used MiB', Math.round(s.used_heap_size / 1048576))"

heap_size_limit comes out larger than the old-space flag you set, because it adds the young generation.

Set the old-space limit from the container. Old space holds long-lived objects and is where heap OOM happens. Set it through NODE_OPTIONS so every entry point picks it up. Changed lines only, in the Deployment’s container spec:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=1536"
resources:
  requests:
    memory: 2Gi
  limits:
    memory: 2Gi

--max-old-space-size takes MiB. As a rule of thumb for one Node process per container:

--max-old-space-size ≈ container memory limit × 0.75

The remaining quarter covers the young generation, Buffers, native addons and the Node binary itself. Raise that share if your app holds large Buffers or uses native modules such as sharp. Read process.memoryUsage() under load: rss minus heapTotal is what your process uses outside the heap. If you run several workers per pod, divide the result by the worker count, as the Node.js cluster mode recipe shows.

On Node 22.21 and 24.6 or later, --max-old-space-size-percentage=75 does the same without hard-coding MiB. It reads the memory Node sees, which is the container limit, and it wins over --max-old-space-size if you set both. I still prefer the fixed MiB value, because a reviewer can see the number in the manifest.

Set requests equal to limits for memory. With a lower request, the scheduler can pack the node so tight that the kernel kills your pod before V8 reaches its own cap.

Cap what grows with traffic. A bigger heap buys time. Bounding the working set stops the growth:

  • Request bodies. Express’s express.json() takes a limit, 100 KB by default. Fastify’s bodyLimit defaults to 1 MiB. In NestJS, call app.useBodyParser('json', { limit: '1mb' }) on the Express adapter, or pass bodyLimit to FastifyAdapter. Keep each limit at the largest payload your API accepts.
  • Large responses. Stream big exports and file downloads with stream.pipeline() in place of building the whole body as a string.
  • In-process caches. Give each cache a size cap, for example new LRUCache({ max: 10000 }) from lru-cache, or move it to Redis.
  • Concurrency. Some requests do memory-heavy work, such as PDF rendering. Put a semaphore around that path, so peak traffic queues there and leaves the heap alone.

Find the leak. If heap use after each full GC keeps rising at steady traffic, raising the limit only moves the crash. Take heap snapshots and compare them:

# NODE_OPTIONS for the pod you'll inspect
--heapsnapshot-signal=SIGUSR2 --diagnostic-dir=/tmp/diag --report-on-fatalerror

# node runs as PID 1 here; behind tini, send the signal to node's PID instead
# take one snapshot, wait ten minutes under load, take another
kubectl exec api-7d9f-abcde -- kill -USR2 1
kubectl cp api-7d9f-abcde:/tmp/diag ./diag

Open both .heapsnapshot files in Chrome DevTools, Memory tab, and pick the Comparison view. Sort by size delta. The constructors that grew between the two snapshots point at the leak, and the retainers panel shows what holds them.

Writing a snapshot pauses the process and needs memory close to twice the heap. Take the pod out of the Service first, by removing its label or failing its readiness check. Give it a higher memory limit while you inspect it. I take snapshots by signal before I trust --heapsnapshot-near-heap-limit. That flag writes a snapshot as the heap nears the cap. It stayed experimental until 22.22.1 and 24.13.1, and a fast leak can kill the process before the file lands.

--report-on-fatalerror writes a JSON report when the process aborts. It holds the heap space sizes, the JavaScript stack and the native stack at the crash, which tells you what the process was doing.

Test it on your system

Heap problems show up over time, so run a soak test at a fixed rate. Hold your peak rate for an hour or longer against staging, with the same memory limit and NODE_OPTIONS as production. Use k6 with the constant-arrival-rate executor, and point it at the endpoints with the largest payloads:

import http from 'k6/http';

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-arrival-rate',
      rate: 300,          // your peak requests per second
      timeUnit: '1s',
      duration: '90m',
      preAllocatedVUs: 200,
      maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<600'],
  },
};

export default function () {
  http.post(`${__ENV.TARGET}/api/reports`, JSON.stringify({ range: '30d' }), {
    headers: { 'Content-Type': 'application/json' },
  });
}

While it runs, watch these:

  • Heap used after GC. nodejs_heap_size_used_bytes from prom-client saw-tooths as GC runs. Look at the low points. A flat floor is healthy, and a floor that climbs every few minutes is a leak.
  • Time spent in GC. rate(nodejs_gc_duration_seconds_sum[1m]) gives seconds of GC per second. It climbs as the heap fills.
  • Container memory. container_memory_working_set_bytes against the pod’s limit, to catch growth outside the heap.
  • Restarts and exit codes from kubectl get pods -w.

Pass means heap after GC levels off below your target share of the limit, with no restart and your latency and error thresholds intact. Then run the same test with the concurrency your autoscaler allows per pod at its ceiling, because that’s where the working set peaks.

Keep it healthy

prom-client’s default metrics report heap use but not the limit. Export the limit yourself from prom-client, next to collectDefaultMetrics():

const v8 = require('node:v8');
const client = require('prom-client');

client.collectDefaultMetrics();
new client.Gauge({
  name: 'nodejs_heap_size_limit_bytes',
  help: 'V8 heap size limit from v8.getHeapStatistics()',
  collect() {
    this.set(v8.getHeapStatistics().heap_size_limit);
  },
});

These rules use that gauge, prom-client’s defaults and kube-state-metrics. They assume a scrape config that adds namespace and pod, and a container named app:

groups:
  - name: nodejs-heap
    rules:
      - alert: NodeHeapNearLimit
        expr: |
          max by (namespace, pod) (nodejs_heap_size_used_bytes)
            / max by (namespace, pod) (nodejs_heap_size_limit_bytes)
            > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Node.js heap on {{ $labels.pod }} above 85% of its limit"
          description: "The pod is heading for a heap out of memory crash. Take a heap snapshot with SIGUSR2 before it restarts, then compare it with a fresh pod."
      - alert: NodeGCTimeHigh
        expr: sum by (namespace, pod) (rate(nodejs_gc_duration_seconds_sum[5m])) > 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Node.js on {{ $labels.pod }} spends over 10% of its time in GC"
          description: "The heap is near full or allocation is heavy. Check heap use against the limit, then look for unbounded caches or large request bodies."
      - alert: NodeHeapOOMCrash
        expr: |
          increase(kube_pod_container_status_restarts_total{container="app"}[10m]) > 0
            and on (namespace, pod, container)
          kube_pod_container_status_last_terminated_exitcode{container="app"} == 134
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Node.js container on {{ $labels.pod }} aborted with exit code 134"
          description: "Check the previous logs with kubectl logs --previous for JavaScript heap out of memory. Raise --max-old-space-size if the container has room, or find the leak."

kube_pod_container_status_last_terminated_exitcode is marked experimental in kube-state-metrics, so check your version exports it. Exit code 134 can also come from another abort, and the logs tell you which. Set the 85% and 10% thresholds from your own baseline. A pod that runs at 80% heap all day for a reason, such as a large warm cache, needs its own threshold.

Roll back

Save the Deployment before the change with kubectl get deploy api -o yaml > api-before.yaml.

  • To undo the heap flag, remove --max-old-space-size from NODE_OPTIONS. V8 goes back to sizing its heap from the container limit.
  • To undo body or cache limits, revert the commit. Clients that sent larger payloads get errors until you do, with 413 Payload Too Large from Express and Fastify.
  • Remove the snapshot flags once you’ve found the leak. A signal or crash writes files that can fill the pod’s disk.

If pods start to get OOMKilled after you raise the heap limit, roll back now: the heap now outgrows the container. Lower the flag, or raise the memory limit by the same amount. Any other revert can wait until traffic drops.

More recipes