A Node.js process that runs out of JavaScript heap memory aborts with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory. Every request it held fails at once, and Kubernetes restarts the container. A bigger limit fixes some cases and only delays others. Find out which one you have before you change a flag.
What breaks at peak
V8 caps the heap at a size it picks at startup. As the heap nears that cap, the garbage collector runs more often and takes longer each time. Your event loop pauses for every collection, so latency climbs for a few minutes before the crash. Then V8 gives up and Node prints one of these lines to stderr:
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
The first is the common form on Node 16 and later. All three still exist in the V8 that ships with Node 20 to 24. The part before “Allocation failed” names where V8 gave up, and the fix is the same for all three. Above them, Node prints a <--- Last few GCs ---> block with the last mark-compact runs.
The process aborts with SIGABRT, so it exits with code 134. That tells you which of two failures you have:
- Heap limit reached. Exit code 134 and the
FATAL ERRORline in the logs. V8’s own cap stopped the process. Kubernetes reports the last state asError. - Container limit reached. Exit code 137 and
OOMKilled, with noFATAL ERRORline. The heap cap sits above the container’s memory, or memory outside the heap grew. The Linux OOM killer recipe covers that side.
Peak traffic causes heap growth in three ways:
- More work in flight. Each open request holds its body, its parsed JSON and its response. Twice the concurrency needs about twice the heap for that working set.
- Caches and queues without a cap. An in-process
Mapused as a cache, or an array of pending jobs, grows with traffic and never shrinks. - A leak. Listeners, closures or timers that hold on to each request. Traffic makes it grow faster, so it crashes at peak first.
The max-old-space-size fix
Check the limit you run with now. Node reads the container’s cgroup memory limit, on cgroup v2 from Node 20.3, and V8 sizes its default heap from that. The default often leaves memory unused in a large container and can sit too close to the limit in a small one. Read it inside the running pod:
kubectl exec deploy/api -- node -e "const v8 = require('node:v8'); \
const s = v8.getHeapStatistics(); \
console.log('heap limit MiB', Math.round(s.heap_size_limit / 1048576)); \
console.log('used MiB', Math.round(s.used_heap_size / 1048576))"
heap_size_limit comes out larger than the old-space flag you set, because it adds the young generation.
Set the old-space limit from the container. Old space holds long-lived objects and is where heap OOM happens. Set it through NODE_OPTIONS so every entry point picks it up. Changed lines only, in the Deployment’s container spec:
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=1536"
resources:
requests:
memory: 2Gi
limits:
memory: 2Gi
--max-old-space-size takes MiB. As a rule of thumb for one Node process per container:
--max-old-space-size ≈ container memory limit × 0.75
The remaining quarter covers the young generation, Buffers, native addons and the Node binary itself. Raise that share if your app holds large Buffers or uses native modules such as sharp. Read process.memoryUsage() under load: rss minus heapTotal is what your process uses outside the heap. If you run several workers per pod, divide the result by the worker count, as the Node.js cluster mode recipe shows.
On Node 22.21 and 24.6 or later, --max-old-space-size-percentage=75 does the same without hard-coding MiB. It reads the memory Node sees, which is the container limit, and it wins over --max-old-space-size if you set both. I still prefer the fixed MiB value, because a reviewer can see the number in the manifest.
Set requests equal to limits for memory. With a lower request, the scheduler can pack the node so tight that the kernel kills your pod before V8 reaches its own cap.
Cap what grows with traffic. A bigger heap buys time. Bounding the working set stops the growth:
- Request bodies. Express’s
express.json()takes alimit, 100 KB by default. Fastify’sbodyLimitdefaults to 1 MiB. In NestJS, callapp.useBodyParser('json', { limit: '1mb' })on the Express adapter, or passbodyLimittoFastifyAdapter. Keep each limit at the largest payload your API accepts. - Large responses. Stream big exports and file downloads with
stream.pipeline()in place of building the whole body as a string. - In-process caches. Give each cache a size cap, for example
new LRUCache({ max: 10000 })from lru-cache, or move it to Redis. - Concurrency. Some requests do memory-heavy work, such as PDF rendering. Put a semaphore around that path, so peak traffic queues there and leaves the heap alone.
Find the leak. If heap use after each full GC keeps rising at steady traffic, raising the limit only moves the crash. Take heap snapshots and compare them:
# NODE_OPTIONS for the pod you'll inspect
--heapsnapshot-signal=SIGUSR2 --diagnostic-dir=/tmp/diag --report-on-fatalerror
# node runs as PID 1 here; behind tini, send the signal to node's PID instead
# take one snapshot, wait ten minutes under load, take another
kubectl exec api-7d9f-abcde -- kill -USR2 1
kubectl cp api-7d9f-abcde:/tmp/diag ./diag
Open both .heapsnapshot files in Chrome DevTools, Memory tab, and pick the Comparison view. Sort by size delta. The constructors that grew between the two snapshots point at the leak, and the retainers panel shows what holds them.
Writing a snapshot pauses the process and needs memory close to twice the heap. Take the pod out of the Service first, by removing its label or failing its readiness check. Give it a higher memory limit while you inspect it. I take snapshots by signal before I trust --heapsnapshot-near-heap-limit. That flag writes a snapshot as the heap nears the cap. It stayed experimental until 22.22.1 and 24.13.1, and a fast leak can kill the process before the file lands.
--report-on-fatalerror writes a JSON report when the process aborts. It holds the heap space sizes, the JavaScript stack and the native stack at the crash, which tells you what the process was doing.
Test it on your system
Heap problems show up over time, so run a soak test at a fixed rate. Hold your peak rate for an hour or longer against staging, with the same memory limit and NODE_OPTIONS as production. Use k6 with the constant-arrival-rate executor, and point it at the endpoints with the largest payloads:
import http from 'k6/http';
export const options = {
scenarios: {
soak: {
executor: 'constant-arrival-rate',
rate: 300, // your peak requests per second
timeUnit: '1s',
duration: '90m',
preAllocatedVUs: 200,
maxVUs: 1000,
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(99)<600'],
},
};
export default function () {
http.post(`${__ENV.TARGET}/api/reports`, JSON.stringify({ range: '30d' }), {
headers: { 'Content-Type': 'application/json' },
});
}
While it runs, watch these:
- Heap used after GC.
nodejs_heap_size_used_bytesfrom prom-client saw-tooths as GC runs. Look at the low points. A flat floor is healthy, and a floor that climbs every few minutes is a leak. - Time spent in GC.
rate(nodejs_gc_duration_seconds_sum[1m])gives seconds of GC per second. It climbs as the heap fills. - Container memory.
container_memory_working_set_bytesagainst the pod’s limit, to catch growth outside the heap. - Restarts and exit codes from
kubectl get pods -w.
Pass means heap after GC levels off below your target share of the limit, with no restart and your latency and error thresholds intact. Then run the same test with the concurrency your autoscaler allows per pod at its ceiling, because that’s where the working set peaks.
Keep it healthy
prom-client’s default metrics report heap use but not the limit. Export the limit yourself from prom-client, next to collectDefaultMetrics():
const v8 = require('node:v8');
const client = require('prom-client');
client.collectDefaultMetrics();
new client.Gauge({
name: 'nodejs_heap_size_limit_bytes',
help: 'V8 heap size limit from v8.getHeapStatistics()',
collect() {
this.set(v8.getHeapStatistics().heap_size_limit);
},
});
These rules use that gauge, prom-client’s defaults and kube-state-metrics. They assume a scrape config that adds namespace and pod, and a container named app:
groups:
- name: nodejs-heap
rules:
- alert: NodeHeapNearLimit
expr: |
max by (namespace, pod) (nodejs_heap_size_used_bytes)
/ max by (namespace, pod) (nodejs_heap_size_limit_bytes)
> 0.85
for: 10m
labels:
severity: warning
annotations:
summary: "Node.js heap on {{ $labels.pod }} above 85% of its limit"
description: "The pod is heading for a heap out of memory crash. Take a heap snapshot with SIGUSR2 before it restarts, then compare it with a fresh pod."
- alert: NodeGCTimeHigh
expr: sum by (namespace, pod) (rate(nodejs_gc_duration_seconds_sum[5m])) > 0.1
for: 10m
labels:
severity: warning
annotations:
summary: "Node.js on {{ $labels.pod }} spends over 10% of its time in GC"
description: "The heap is near full or allocation is heavy. Check heap use against the limit, then look for unbounded caches or large request bodies."
- alert: NodeHeapOOMCrash
expr: |
increase(kube_pod_container_status_restarts_total{container="app"}[10m]) > 0
and on (namespace, pod, container)
kube_pod_container_status_last_terminated_exitcode{container="app"} == 134
for: 0m
labels:
severity: critical
annotations:
summary: "Node.js container on {{ $labels.pod }} aborted with exit code 134"
description: "Check the previous logs with kubectl logs --previous for JavaScript heap out of memory. Raise --max-old-space-size if the container has room, or find the leak."
kube_pod_container_status_last_terminated_exitcode is marked experimental in kube-state-metrics, so check your version exports it. Exit code 134 can also come from another abort, and the logs tell you which. Set the 85% and 10% thresholds from your own baseline. A pod that runs at 80% heap all day for a reason, such as a large warm cache, needs its own threshold.
Roll back
Save the Deployment before the change with kubectl get deploy api -o yaml > api-before.yaml.
- To undo the heap flag, remove
--max-old-space-sizefromNODE_OPTIONS. V8 goes back to sizing its heap from the container limit. - To undo body or cache limits, revert the commit. Clients that sent larger payloads get errors until you do, with
413 Payload Too Largefrom Express and Fastify. - Remove the snapshot flags once you’ve found the leak. A signal or crash writes files that can fill the pod’s disk.
If pods start to get OOMKilled after you raise the heap limit, roll back now: the heap now outgrows the container. Lower the flag, or raise the memory limit by the same amount. Any other revert can wait until traffic drops.
More recipes
- Node.js Event Loop Lag Monitoring with Prometheus
Alert on prom-client event loop lag, loop utilization and failed scrapes, so a blocked Node.js event loop pages you before your health checks start failing.
- Fix Socket Hang Up and ECONNRESET Behind a Load Balancer
Keep idle connections open longer on the side that should close them last, so Node.js and the load balancer stop racing to reuse a socket the other has closed.
- Next.js SSR Slow at Peak? Fix It with Caching and ISR
Turn per-request rendering into Next.js ISR caching with a cache that every replica shares, so peak traffic reads cached pages instead of rendering each one.
- Use Every CPU Core with Node.js Cluster Mode
Run one Node.js worker per CPU core, as pods or node:cluster workers, so peak traffic spreads across every core instead of queuing on one event loop.