Node.js event loop lag is the time a callback waits after it’s ready to run. When one handler runs synchronous work, every other request on that process waits behind it. CPU graphs show one busy core at most, so the dashboards look normal while p99 latency climbs. You find out from users, or from a pod that Kubernetes restarts for failing its probes.
What you miss today
Most Node.js dashboards show CPU, memory and request rate. None of them shows a blocked loop.
- CPU averages hide it. One process blocked for 400 ms every few seconds shows up as moderate CPU on a 5-minute graph.
- Lag percentiles can miss it too. Node samples the loop delay on a timer. One long block becomes a single sample, so p99 over a scrape window can stay low while the max shows the stall.
- The scrape fails with the loop.
/metricsruns on the same event loop. A process stuck for longer than the scrape timeout returns no metrics, and the graph shows a gap where the problem was.
The first symptom you hear about tends to be a timeout. Clients report slow responses, the load balancer logs 504s, or kubelet events show Liveness probe failed with context deadline exceeded. A restart clears the symptom and destroys the evidence.
The Node.js cluster mode recipe has one lag alert for a pod stuck on one core. The rules below cover the other stalls: one long block, a saturated loop, GC pauses and a loop too stuck to answer a scrape. Rust’s Tokio runtime has the same failure, which the Tokio spawn_blocking recipe covers.
The exporter and metrics
Use prom-client. Its default metrics include the event loop, the heap and GC. From version 0.16 the project publishes as @prometheus-io/client under the Prometheus organization, with the same metric names. Add a loop utilization gauge if you’re still on prom-client 15:
// metrics.js
const http = require('node:http');
const { performance } = require('node:perf_hooks');
const client = require('prom-client');
client.collectDefaultMetrics({ eventLoopMonitoringPrecision: 10 }); // ms between samples
let lastElu = performance.eventLoopUtilization();
new client.Gauge({
name: 'nodejs_eventloop_utilization_ratio',
help: 'Share of time the event loop was busy since the last scrape',
collect() {
const now = performance.eventLoopUtilization();
this.set(performance.eventLoopUtilization(now, lastElu).utilization);
lastElu = now;
},
});
http.createServer(async (_req, res) => {
res.writeHead(200, { 'Content-Type': client.register.contentType });
res.end(await client.register.metrics());
}).listen(Number(process.env.METRICS_PORT) || 9100);
Require it once at startup, before your app. A separate port keeps /metrics out of your public routes. It doesn’t help when the loop blocks, because the metrics server shares the loop.
These are the metrics that matter, and what each one tells you:
| Metric | What it tells you |
|---|---|
nodejs_eventloop_lag_max_seconds | The longest single delay since the last scrape. Your best signal for one long synchronous block. |
nodejs_eventloop_lag_p99_seconds | The delay most requests wait behind. It tracks steady pressure. |
nodejs_eventloop_lag_seconds | One setImmediate delay taken at scrape time. It’s noisy, so I chart it but don’t alert on it. |
nodejs_eventloop_utilization_ratio | The share of time the loop was busy. Near 1 means the process has no room left for more work. |
nodejs_gc_duration_seconds | A histogram of GC pauses by kind. Major collections block the loop. |
nodejs_heap_size_used_bytes | Heap in use. A full heap means more GC and longer pauses. |
up | Prometheus sets it to 0 when a scrape fails. A blocked loop fails the scrape. |
prom-client resets the lag histogram after each collection. The min, max, mean and percentile gauges each cover the time since the previous scrape. Two Prometheus servers scraping the same pod each see about half the window. Their numbers differ, and each one misses blocks the other caught.
@prometheus-io/client 0.16 and later adds nodejs_eventloop_utilization_summary and nodejs_eventloop_utilization_histogram by default. On that version you can drop the custom gauge and use one of those in the rules.
Express. Load metrics.js from your entry file before app.listen(). express-prom-bundle adds per-route request histograms on top, which tell you which route was running when the lag rose.
Fastify. @fastify/under-pressure checks the same signals inside the server. With maxEventLoopDelay or maxEventLoopUtilization set, it answers new requests with 503 while the loop is over the limit. Your load balancer then sends traffic elsewhere. Keep the Prometheus rules as well, so a 503 storm pages someone.
NestJS. Require metrics.js at the top of main.ts, or use @willsoto/nestjs-prometheus, which registers the same default metrics behind a /metrics controller.
The alert rules
These rules assume the scrape job is called api and that your scrape config adds namespace and pod labels. The startup filter skips the first five minutes of a process, when require() and JIT compilation cause lag you don’t need to hear about:
groups:
- name: nodejs-event-loop
rules:
- alert: NodeEventLoopBlocked
expr: |
max by (namespace, pod) (nodejs_eventloop_lag_max_seconds{job="api"}) > 0.5
and on (namespace, pod)
max by (namespace, pod) (time() - process_start_time_seconds{job="api"}) > 300
for: 5m
labels:
severity: warning
annotations:
summary: "Event loop on {{ $labels.pod }} blocked for over 500 ms at a time"
description: "A synchronous call holds the loop. Record a CPU profile with node --cpu-prof on one pod under load, and look for JSON.parse, sync crypto, sync fs or a large regex."
- alert: NodeEventLoopLagP99Critical
expr: max by (namespace, pod) (nodejs_eventloop_lag_p99_seconds{job="api"}) > 0.25
for: 5m
labels:
severity: critical
annotations:
summary: "Event loop p99 lag above 250 ms on {{ $labels.pod }}"
description: "Requests queue behind the loop and users wait. Check loop utilization and GC time on this pod, then add replicas or workers while you find the hot path."
- alert: NodeEventLoopSaturated
expr: max by (namespace, pod) (nodejs_eventloop_utilization_ratio{job="api"}) > 0.9
for: 10m
labels:
severity: warning
annotations:
summary: "Event loop on {{ $labels.pod }} busy over 90% of the time"
description: "The process has no spare loop time for a traffic spike. Add replicas, run one worker per core, or move CPU-heavy work to worker_threads."
- alert: NodeMajorGCPressure
expr: sum by (namespace, pod) (rate(nodejs_gc_duration_seconds_sum{job="api", kind="major"}[5m])) > 0.05
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.pod }} spends over 5% of its time in major GC"
description: "Major collections pause the loop. Compare nodejs_heap_size_used_bytes with the heap limit and follow the heap out of memory recipe."
- alert: NodeMetricsUnreachable
expr: up{job="api"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Prometheus can't scrape {{ $labels.pod }}"
description: "A blocked event loop can't answer /metrics. Check the pod's events for failed probes and its last logs, then capture a CPU profile before it restarts."
NodeMajorGCPressure points to the heap out of memory recipe, which covers sizing the heap and finding a leak.
Test the alerts
Fire each alert on purpose in staging, then confirm it clears. Add a route that blocks the loop, and only mount it when a staging flag is set:
if (process.env.ENABLE_BLOCK_ROUTE === 'true') {
app.get('/debug/block', (req, res) => {
const ms = Math.min(Number(req.query.ms) || 200, 5000);
const end = Date.now() + ms;
while (Date.now() < end) { /* hold the event loop */ }
res.end(`blocked ${ms} ms\n`);
});
}
Then drive it with k6 at a fixed rate. Each scenario targets one alert:
import http from 'k6/http';
export const options = {
scenarios: {
long_blocks: { // NodeEventLoopBlocked
executor: 'constant-arrival-rate',
exec: 'longBlock', rate: 1, timeUnit: '2s', duration: '10m',
preAllocatedVUs: 5,
},
saturate: { // NodeEventLoopSaturated and the p99 alert
executor: 'constant-arrival-rate',
exec: 'shortBlock', rate: 40, timeUnit: '1s', duration: '15m', startTime: '12m',
preAllocatedVUs: 50, maxVUs: 200,
},
},
};
export function longBlock() {
http.get(`${__ENV.TARGET}/debug/block?ms=800`);
}
export function shortBlock() {
http.get(`${__ENV.TARGET}/debug/block?ms=24`);
}
Run it with k6 run -e TARGET=http://api.staging.svc:3000 block.js, aimed at a single pod. Pick the shortBlock rate and duration so their product sits close to one second of work per second on that pod. Watch the Alerts page in Prometheus as each scenario runs. Each alert should go pending, fire after its for duration, then clear within a few scrapes of the scenario ending.
For NodeMetricsUnreachable, freeze the process. Raise the liveness probe’s failureThreshold on the test pod first, or kubelet restarts it before the alert fires:
kubectl exec api-7d9f-abcde -- kill -STOP 1 # node as PID 1
# wait three minutes, check the alert, then resume
kubectl exec api-7d9f-abcde -- kill -CONT 1
For NodeMajorGCPressure, run the soak test from the heap recipe with a lowered --max-old-space-size on one pod. Note any alert that doesn’t fire, and check its labels against the series in the Prometheus expression browser.
Tune the noise
The thresholds above are starting points. Set yours from a week of production data:
quantile_over_time(0.99, (max by (pod) (nodejs_eventloop_lag_max_seconds{job="api"}))[7d:5m])
Put the warning threshold a little above that line and the critical one at the lag your latency budget can’t absorb. A service with a 200 ms p99 target can’t tolerate a 250 ms loop stall, so its critical threshold sits lower.
fordurations. Single long blocks recur on a steady schedule, such as a cache rebuild on a timer. Aforof 5m skips one-offs and still catches a block that comes back on every scrape. Saturation needs a longerfor, because a short burst after a deploy or a cache flush is normal.- Scrape interval. The lag gauges cover the time between scrapes. A 60-second interval keeps the max but dilutes the percentiles. I keep 15 seconds on Node.js services for this reason.
- Known heavy routes. If a report endpoint blocks by design, move that work to
worker_threadsbefore you raise the threshold. A higher threshold hides the next block too. - Startup. Raise the 300-second filter if your app warms caches for longer after start.
Review the alert history after each launch. An alert that fired with no user impact needs a higher threshold or a longer for. An incident with no alert needs a lower one.
More recipes
- Fix JavaScript Heap Out of Memory in Node.js
Size the V8 heap to the container with max-old-space-size, then find what grows at peak, so Node.js stops crashing with JavaScript heap out of memory.
- Fix Socket Hang Up and ECONNRESET Behind a Load Balancer
Keep idle connections open longer on the side that should close them last, so Node.js and the load balancer stop racing to reuse a socket the other has closed.
- Next.js SSR Slow at Peak? Fix It with Caching and ISR
Turn per-request rendering into Next.js ISR caching with a cache that every replica shares, so peak traffic reads cached pages instead of rendering each one.
- Use Every CPU Core with Node.js Cluster Mode
Run one Node.js worker per CPU core, as pods or node:cluster workers, so peak traffic spreads across every core instead of queuing on one event loop.