The nginx Prometheus exporter, nginx-prometheus-exporter, reads the stub_status page of open source nginx. That page holds seven counters about connections and requests. It has no status codes and no latency, so the exporter can’t tell you when nginx returns 502s or when a backend slows down. Keep that exporter for connections, and add the VTS module for the two signals it lacks.
What you miss today
A backend that fails half its requests leaves stub_status looking normal. Active connections stay flat and the request counter keeps climbing, because a 502 is still a request. nginx_up stays at 1, since it only reports whether the exporter could read stub_status.
So the first alert you get comes from a person. A customer sends a screenshot of a 502 Bad Gateway page, or support reports a slow checkout. By then the error log has held upstream timed out or no live upstreams lines for some time. The upstream timeouts recipe explains those lines and the settings behind them.
Two signals close the gap:
- The 5xx rate per virtual host. It shows users getting errors now.
- Upstream response time per backend. It shows a backend slowing down before its requests start to time out.
The exporter and metrics
Run two sources side by side. Each covers what the other lacks.
| Source | What it gives you | Cost |
|---|---|---|
nginx-prometheus-exporter reading stub_status | Connection counts, accepted and handled totals, total requests | A small sidecar or service. stub_status ships in the official packages and images. |
nginx-module-vts | Requests per status class, per host and per upstream server, plus latency histograms | A third-party module you build against your exact nginx version |
Build the VTS module. The module comes from vozlt/nginx-module-vts. Its README documents a static build with ./configure --add-module=/path/to/nginx-module-vts. You can also build it as a dynamic module with --add-dynamic-module and --with-compat, matching the version from nginx -v. Some distributions and third-party repositories package it, so check yours before you compile. A dynamic module loads with one line at the top of nginx.conf:
# /etc/nginx/nginx.conf, changed lines only, outside the http block
load_module modules/ngx_http_vhost_traffic_status_module.so;
Turn it on. Add the shared zone and histogram buckets inside the http block:
# /etc/nginx/nginx.conf, changed lines only, inside http {}
vhost_traffic_status_zone shared:vhost_traffic_status:32m;
vhost_traffic_status_histogram_buckets 0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 2.5 5 10;
The zone holds the counters. The README suggests 32m or more once you add filters; the default is 1m. Histograms stay empty until you set buckets, and the module accepts up to 32 of them. Put a bucket near your latency target and one near proxy_read_timeout, so the alerts below have a boundary to read.
Expose both pages on a private port. Save this as /etc/nginx/conf.d/metrics.conf and change the addresses to your network:
server {
listen 10.0.1.5:8081;
allow 127.0.0.1;
allow 10.0.0.0/8; # your Prometheus subnet
deny all;
location = /stub_status {
stub_status;
}
location /vts {
vhost_traffic_status_display;
vhost_traffic_status_display_format prometheus;
}
}
Run the exporter. It listens on port 9113 by default and needs the stub_status URL:
docker run -d --name nginx-exporter --network host \
nginx/nginx-prometheus-exporter:1.5.1 \
--nginx.scrape-uri=http://10.0.1.5:8081/stub_status
Scrape both. The VTS page already speaks the Prometheus format, so Prometheus scrapes it as is:
scrape_configs:
- job_name: nginx
static_configs:
- targets: ['10.0.1.5:9113']
- job_name: nginx-vts
metrics_path: /vts
static_configs:
- targets: ['10.0.1.5:8081']
These are the metrics that matter, and what each one tells you:
| Metric | Source | What it tells you |
|---|---|---|
nginx_up | exporter | 0 means the exporter can’t read stub_status, often because nginx is down |
nginx_connections_active | exporter | Open client connections, to compare against worker_connections |
nginx_connections_accepted, nginx_connections_handled | exporter | A gap between them means workers ran out of connection slots |
nginx_http_requests_total | exporter | Total request rate, your traffic baseline |
nginx_vts_server_requests_total{host, code} | VTS | Requests per virtual host by status class, 1xx to 5xx |
nginx_vts_server_request_duration_seconds_bucket{host} | VTS | Full request time as nginx sees it, histogram |
nginx_vts_upstream_requests_total{upstream, backend, code} | VTS | Requests per backend server by status class |
nginx_vts_upstream_response_duration_seconds_bucket{upstream, backend} | VTS | Backend response time per server, histogram |
I alert on the server 5xx rate and page on it, and I keep the per-backend numbers for the dashboard and a lower-severity alert. The server counter covers 502s and 504s that nginx makes itself, which are the ones users see.
Other routes to the same signals. NGINX Plus exposes status codes through its API, and the same exporter reads them with --nginx.plus, as nginxplus_upstream_server_responses{code="5xx"}. Without a module, you can turn the access log into counters with Grafana Alloy or mtail. Log the status and $upstream_response_time, then count them. ingress-nginx exposes its own request metrics, but the project retired in March 2026.
Prometheus alert rules for the nginx 5xx error rate
The recording rules keep the alert expressions short and make the baseline queries cheap. Save the file and load it with rule_files in prometheus.yml:
groups:
- name: nginx-recording
rules:
- record: nginx:server_5xx_ratio:rate5m
expr: |
sum by (instance, host) (rate(nginx_vts_server_requests_total{code="5xx"}[5m]))
/ sum by (instance, host) (rate(nginx_vts_server_requests_total[5m]))
- record: nginx:server_requests:rate5m
expr: sum by (instance, host) (rate(nginx_vts_server_requests_total[5m]))
- record: nginx:upstream_backend_5xx_ratio:rate5m
expr: |
sum by (instance, upstream, backend) (rate(nginx_vts_upstream_requests_total{code="5xx"}[5m]))
/ sum by (instance, upstream, backend) (rate(nginx_vts_upstream_requests_total[5m]))
- record: nginx:upstream_response_p99:rate5m
expr: |
histogram_quantile(0.99,
sum by (instance, upstream, le) (rate(nginx_vts_upstream_response_duration_seconds_bucket[5m])))
- name: nginx-alerts
rules:
- alert: NginxDown
expr: nginx_up == 0
for: 2m
labels:
severity: critical
annotations:
summary: "nginx-prometheus-exporter can't read stub_status on {{ $labels.instance }}"
description: "Check that nginx runs with systemctl status nginx, then curl the stub_status URL from the exporter host. A firewall or allow rule change also causes this."
- alert: NginxHigh5xxRate
expr: nginx:server_5xx_ratio:rate5m > 0.02 and nginx:server_requests:rate5m > 1
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.host }} on {{ $labels.instance }} returns 5xx on over 2% of requests"
description: "Grep the nginx error log for 'upstream timed out', 'no live upstreams' and 'connect() failed', then check the backend with the highest 5xx ratio on the dashboard."
- alert: NginxBackendErrors
expr: nginx:upstream_backend_5xx_ratio:rate5m > 0.05
for: 10m
labels:
severity: warning
annotations:
summary: "Backend {{ $labels.backend }} in {{ $labels.upstream }} fails over 5% of requests"
description: "One server is failing while nginx retries on others. Check that server's app logs and health, and take it out of the pool if it doesn't recover."
- alert: NginxUpstreamSlow
expr: nginx:upstream_response_p99:rate5m > 1
for: 10m
labels:
severity: warning
annotations:
summary: "p99 backend response time for {{ $labels.upstream }} is over 1s"
description: "Backends are slowing down before they time out. Check database and dependency latency, then compare the p99 with proxy_read_timeout for that location."
- alert: NginxVtsScrapeFailing
expr: up{job="nginx-vts"} == 0
for: 10m
labels:
severity: warning
annotations:
summary: "Prometheus can't scrape the VTS page on {{ $labels.instance }}"
description: "The 5xx and latency alerts are blind until this clears. Check the allow rules in metrics.conf and that the module still loads after an nginx upgrade."
- alert: NginxTrafficDrop
expr: |
sum by (instance) (rate(nginx_http_requests_total[10m]))
< 0.5 * sum by (instance) (rate(nginx_http_requests_total[10m] offset 1w))
for: 15m
labels:
severity: warning
annotations:
summary: "Requests to {{ $labels.instance }} fell below half of the same time last week"
description: "Traffic isn't reaching nginx. Check DNS, the load balancer in front and its health checks before you look at nginx itself."
NginxHigh5xxRate skips hosts with less than one request a second, so one failed request on a quiet host doesn’t page anyone. NginxTrafficDrop catches the failure the others miss: users can’t reach nginx at all, so it records no errors. Connection-slot alerts live in the nginx high-traffic recipe.
Test the alerts
Cause each failure on purpose in staging, under a steady load. A fixed arrival rate from k6 gives you that, and the k6 breakpoint recipe shows how to build one. Run the load at a normal daytime rate, then break one thing at a time:
| Alert | How to cause it | How to clear it |
|---|---|---|
NginxHigh5xxRate | Stop every backend in one upstream group | Start them again |
NginxBackendErrors | Stop one backend of several | Start it again |
NginxUpstreamSlow | Add delay on one backend with tc qdisc add dev eth0 root netem delay 1500ms | tc qdisc del dev eth0 root netem |
NginxDown | systemctl stop nginx on a staging host with no load | systemctl start nginx |
NginxVtsScrapeFailing | Remove the Prometheus subnet from allow and reload | Restore the line and reload |
Watch each alert move through its states. Prometheus shows them on its /alerts page, or you can query the ALERTS series:
ALERTS{alertname=~"Nginx.*"}
An alert passes when it goes pending, turns firing once its for time is up, and reaches your pager or channel. Then clear the fault and confirm it resolves. Note how long each step took. The time from fault to page is your real time to detect, and it’s the number to improve.
NginxTrafficDrop needs a week of history, so test it with a shorter offset in a copy of the rule.
Tune the noise
The thresholds above are starting points. Read your own baseline before you trust them. With the recording rules in place for a few weeks, ask Prometheus what normal looks like:
# the worst 5xx ratio per host in a normal week
max_over_time(nginx:server_5xx_ratio:rate5m[7d])
# the p99 backend response time at your busiest hour
quantile_over_time(0.95, nginx:upstream_response_p99:rate5m[7d])
Set each threshold above what a normal week reaches and below what your users notice. For the error rate, the SLO behind the endpoint gives you the ceiling. A 99.9% target leaves 0.1% of requests for errors across the whole window, so a page at 2% for 5 minutes is generous. For latency, set the threshold under your target and well under proxy_read_timeout.
The for duration trades speed for noise. A deploy that restarts backends can cause a minute of 502s, so a for shorter than your rollout pages every deploy. If that happens, fix the deploy first with readiness checks and connection draining, then shorten for again. Keep critical for alerts where users get errors now, and send warning to a channel someone reads during work hours.
Low-traffic hosts need the minimum-rate guard more than a high threshold. Raise the > 1 request-per-second floor until a quiet host can’t trip the alert with one bad request.
Roll back
The VTS module adds a shared memory zone and a little work per request. To take it out, remove the vhost_traffic_status_* lines and the load_module line, then test and reload:
cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak # before any change
nginx -t && nginx -s reload
If nginx fails to start after an upgrade because the module was built for the old version, remove the load_module line now and rebuild the module later. Your connection metrics keep flowing from the exporter while you do.
More recipes
- Nginx Upstream Timed Out, 502 and 504 Errors
Set proxy timeouts from your slowest real endpoint, cap retries and stop one slow response from ejecting a server, so peak load gives fewer 502s and 504s.
- Nginx Tuning for High Traffic
Size worker_connections for two sockets per request and keep a right-sized upstream keepalive pool, so nginx stops dropping connections at peak.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.