Nginx Prometheus Exporter and 5xx Error Rate Alerts

Scrape stub_status for connections and the VTS module for status codes and upstream latency, so a rising nginx 5xx error rate pages you before users do.

Published

Applies to nginx 1.18 and later with nginx-module-vts; nginx-prometheus-exporter 1.x

The nginx Prometheus exporter, nginx-prometheus-exporter, reads the stub_status page of open source nginx. That page holds seven counters about connections and requests. It has no status codes and no latency, so the exporter can’t tell you when nginx returns 502s or when a backend slows down. Keep that exporter for connections, and add the VTS module for the two signals it lacks.

What you miss today

A backend that fails half its requests leaves stub_status looking normal. Active connections stay flat and the request counter keeps climbing, because a 502 is still a request. nginx_up stays at 1, since it only reports whether the exporter could read stub_status.

So the first alert you get comes from a person. A customer sends a screenshot of a 502 Bad Gateway page, or support reports a slow checkout. By then the error log has held upstream timed out or no live upstreams lines for some time. The upstream timeouts recipe explains those lines and the settings behind them.

Two signals close the gap:

  • The 5xx rate per virtual host. It shows users getting errors now.
  • Upstream response time per backend. It shows a backend slowing down before its requests start to time out.

The exporter and metrics

Run two sources side by side. Each covers what the other lacks.

SourceWhat it gives youCost
nginx-prometheus-exporter reading stub_statusConnection counts, accepted and handled totals, total requestsA small sidecar or service. stub_status ships in the official packages and images.
nginx-module-vtsRequests per status class, per host and per upstream server, plus latency histogramsA third-party module you build against your exact nginx version

Build the VTS module. The module comes from vozlt/nginx-module-vts. Its README documents a static build with ./configure --add-module=/path/to/nginx-module-vts. You can also build it as a dynamic module with --add-dynamic-module and --with-compat, matching the version from nginx -v. Some distributions and third-party repositories package it, so check yours before you compile. A dynamic module loads with one line at the top of nginx.conf:

# /etc/nginx/nginx.conf, changed lines only, outside the http block
load_module modules/ngx_http_vhost_traffic_status_module.so;

Turn it on. Add the shared zone and histogram buckets inside the http block:

# /etc/nginx/nginx.conf, changed lines only, inside http {}
vhost_traffic_status_zone shared:vhost_traffic_status:32m;
vhost_traffic_status_histogram_buckets 0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 2.5 5 10;

The zone holds the counters. The README suggests 32m or more once you add filters; the default is 1m. Histograms stay empty until you set buckets, and the module accepts up to 32 of them. Put a bucket near your latency target and one near proxy_read_timeout, so the alerts below have a boundary to read.

Expose both pages on a private port. Save this as /etc/nginx/conf.d/metrics.conf and change the addresses to your network:

server {
    listen 10.0.1.5:8081;
    allow 127.0.0.1;
    allow 10.0.0.0/8;     # your Prometheus subnet
    deny all;

    location = /stub_status {
        stub_status;
    }

    location /vts {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format prometheus;
    }
}

Run the exporter. It listens on port 9113 by default and needs the stub_status URL:

docker run -d --name nginx-exporter --network host \
  nginx/nginx-prometheus-exporter:1.5.1 \
  --nginx.scrape-uri=http://10.0.1.5:8081/stub_status

Scrape both. The VTS page already speaks the Prometheus format, so Prometheus scrapes it as is:

scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: ['10.0.1.5:9113']
  - job_name: nginx-vts
    metrics_path: /vts
    static_configs:
      - targets: ['10.0.1.5:8081']

These are the metrics that matter, and what each one tells you:

MetricSourceWhat it tells you
nginx_upexporter0 means the exporter can’t read stub_status, often because nginx is down
nginx_connections_activeexporterOpen client connections, to compare against worker_connections
nginx_connections_accepted, nginx_connections_handledexporterA gap between them means workers ran out of connection slots
nginx_http_requests_totalexporterTotal request rate, your traffic baseline
nginx_vts_server_requests_total{host, code}VTSRequests per virtual host by status class, 1xx to 5xx
nginx_vts_server_request_duration_seconds_bucket{host}VTSFull request time as nginx sees it, histogram
nginx_vts_upstream_requests_total{upstream, backend, code}VTSRequests per backend server by status class
nginx_vts_upstream_response_duration_seconds_bucket{upstream, backend}VTSBackend response time per server, histogram

I alert on the server 5xx rate and page on it, and I keep the per-backend numbers for the dashboard and a lower-severity alert. The server counter covers 502s and 504s that nginx makes itself, which are the ones users see.

Other routes to the same signals. NGINX Plus exposes status codes through its API, and the same exporter reads them with --nginx.plus, as nginxplus_upstream_server_responses{code="5xx"}. Without a module, you can turn the access log into counters with Grafana Alloy or mtail. Log the status and $upstream_response_time, then count them. ingress-nginx exposes its own request metrics, but the project retired in March 2026.

Prometheus alert rules for the nginx 5xx error rate

The recording rules keep the alert expressions short and make the baseline queries cheap. Save the file and load it with rule_files in prometheus.yml:

groups:
  - name: nginx-recording
    rules:
      - record: nginx:server_5xx_ratio:rate5m
        expr: |
          sum by (instance, host) (rate(nginx_vts_server_requests_total{code="5xx"}[5m]))
            / sum by (instance, host) (rate(nginx_vts_server_requests_total[5m]))
      - record: nginx:server_requests:rate5m
        expr: sum by (instance, host) (rate(nginx_vts_server_requests_total[5m]))
      - record: nginx:upstream_backend_5xx_ratio:rate5m
        expr: |
          sum by (instance, upstream, backend) (rate(nginx_vts_upstream_requests_total{code="5xx"}[5m]))
            / sum by (instance, upstream, backend) (rate(nginx_vts_upstream_requests_total[5m]))
      - record: nginx:upstream_response_p99:rate5m
        expr: |
          histogram_quantile(0.99,
            sum by (instance, upstream, le) (rate(nginx_vts_upstream_response_duration_seconds_bucket[5m])))

  - name: nginx-alerts
    rules:
      - alert: NginxDown
        expr: nginx_up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "nginx-prometheus-exporter can't read stub_status on {{ $labels.instance }}"
          description: "Check that nginx runs with systemctl status nginx, then curl the stub_status URL from the exporter host. A firewall or allow rule change also causes this."
      - alert: NginxHigh5xxRate
        expr: nginx:server_5xx_ratio:rate5m > 0.02 and nginx:server_requests:rate5m > 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.host }} on {{ $labels.instance }} returns 5xx on over 2% of requests"
          description: "Grep the nginx error log for 'upstream timed out', 'no live upstreams' and 'connect() failed', then check the backend with the highest 5xx ratio on the dashboard."
      - alert: NginxBackendErrors
        expr: nginx:upstream_backend_5xx_ratio:rate5m > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Backend {{ $labels.backend }} in {{ $labels.upstream }} fails over 5% of requests"
          description: "One server is failing while nginx retries on others. Check that server's app logs and health, and take it out of the pool if it doesn't recover."
      - alert: NginxUpstreamSlow
        expr: nginx:upstream_response_p99:rate5m > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p99 backend response time for {{ $labels.upstream }} is over 1s"
          description: "Backends are slowing down before they time out. Check database and dependency latency, then compare the p99 with proxy_read_timeout for that location."
      - alert: NginxVtsScrapeFailing
        expr: up{job="nginx-vts"} == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus can't scrape the VTS page on {{ $labels.instance }}"
          description: "The 5xx and latency alerts are blind until this clears. Check the allow rules in metrics.conf and that the module still loads after an nginx upgrade."
      - alert: NginxTrafficDrop
        expr: |
          sum by (instance) (rate(nginx_http_requests_total[10m]))
            < 0.5 * sum by (instance) (rate(nginx_http_requests_total[10m] offset 1w))
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Requests to {{ $labels.instance }} fell below half of the same time last week"
          description: "Traffic isn't reaching nginx. Check DNS, the load balancer in front and its health checks before you look at nginx itself."

NginxHigh5xxRate skips hosts with less than one request a second, so one failed request on a quiet host doesn’t page anyone. NginxTrafficDrop catches the failure the others miss: users can’t reach nginx at all, so it records no errors. Connection-slot alerts live in the nginx high-traffic recipe.

Test the alerts

Cause each failure on purpose in staging, under a steady load. A fixed arrival rate from k6 gives you that, and the k6 breakpoint recipe shows how to build one. Run the load at a normal daytime rate, then break one thing at a time:

AlertHow to cause itHow to clear it
NginxHigh5xxRateStop every backend in one upstream groupStart them again
NginxBackendErrorsStop one backend of severalStart it again
NginxUpstreamSlowAdd delay on one backend with tc qdisc add dev eth0 root netem delay 1500mstc qdisc del dev eth0 root netem
NginxDownsystemctl stop nginx on a staging host with no loadsystemctl start nginx
NginxVtsScrapeFailingRemove the Prometheus subnet from allow and reloadRestore the line and reload

Watch each alert move through its states. Prometheus shows them on its /alerts page, or you can query the ALERTS series:

ALERTS{alertname=~"Nginx.*"}

An alert passes when it goes pending, turns firing once its for time is up, and reaches your pager or channel. Then clear the fault and confirm it resolves. Note how long each step took. The time from fault to page is your real time to detect, and it’s the number to improve.

NginxTrafficDrop needs a week of history, so test it with a shorter offset in a copy of the rule.

Tune the noise

The thresholds above are starting points. Read your own baseline before you trust them. With the recording rules in place for a few weeks, ask Prometheus what normal looks like:

# the worst 5xx ratio per host in a normal week
max_over_time(nginx:server_5xx_ratio:rate5m[7d])

# the p99 backend response time at your busiest hour
quantile_over_time(0.95, nginx:upstream_response_p99:rate5m[7d])

Set each threshold above what a normal week reaches and below what your users notice. For the error rate, the SLO behind the endpoint gives you the ceiling. A 99.9% target leaves 0.1% of requests for errors across the whole window, so a page at 2% for 5 minutes is generous. For latency, set the threshold under your target and well under proxy_read_timeout.

The for duration trades speed for noise. A deploy that restarts backends can cause a minute of 502s, so a for shorter than your rollout pages every deploy. If that happens, fix the deploy first with readiness checks and connection draining, then shorten for again. Keep critical for alerts where users get errors now, and send warning to a channel someone reads during work hours.

Low-traffic hosts need the minimum-rate guard more than a high threshold. Raise the > 1 request-per-second floor until a quiet host can’t trip the alert with one bad request.

Roll back

The VTS module adds a shared memory zone and a little work per request. To take it out, remove the vhost_traffic_status_* lines and the load_module line, then test and reload:

cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak   # before any change
nginx -t && nginx -s reload

If nginx fails to start after an upgrade because the module was built for the old version, remove the load_module line now and rebuild the module later. Your connection metrics keep flowing from the exporter while you do.

More recipes

Further reading