Fix Application Gateway 502 Bad Gateway at Peak

Give Application Gateway a cheap health probe and a request timeout sized from your p99, so busy backends stay in the pool and peak traffic stops getting 502.

Published

Applies to Azure Application Gateway v2 (Standard_v2 and WAF_v2), v1 where noted; Azure CLI 2.x; azure-metrics-exporter 24.9 and later

An Application Gateway 502 Bad Gateway at peak often starts with the health probe. A busy backend answers the probe late, the gateway marks it unhealthy, and its traffic lands on the others. They slow down, fail their probes too, and the pool empties. The gateway then has nowhere to send requests.

What breaks at peak

Your users get this error page from the gateway:

502 - Web server received an invalid response while acting as a gateway or proxy server.

The Azure portal and docs show it with a Server Error: prefix. Microsoft lists the common causes:

  • An NSG, user-defined route or custom DNS blocks the gateway from the backend.
  • Backends don’t answer the default health probe.
  • A custom health probe is misconfigured.
  • The backend pool is empty, or every instance in it is unhealthy.
  • The backend took longer than the request timeout.
  • The backend certificate doesn’t match the host name the gateway expects.

The first and last causes break things all day, so you’d notice them in testing. The middle ones show up at peak.

The probe cascade. The default probe sends a request to http://127.0.0.1/ every 30 seconds, with a 30-second timeout and an unhealthy threshold of 3. It hits the app’s root path, which often renders a full page and queries the database. At peak, that page slows down like every other page. A backend that misses three probes in a row leaves the pool. The remaining backends take its share, slow down further and drop out next. Once the pool is empty, every request gets a 502.

The request timeout. Each backend setting has a request timeout, 20 seconds by default. On v1, a backend slower than that returns a 502 to the client. On v2, the gateway retries the request against a second backend member, and if that also times out, the client gets a 504.

Closed connections. A backend that crashes, restarts or closes a keep-alive connection while the gateway sends a request on it also produces a 502.

The access log tells you which case you have. Turn on the ApplicationGatewayAccessLog diagnostic setting and read the serverStatus and errorInfo fields. Microsoft documents these errorInfo values for backend failures:

  • ERRORINFO_UPSTREAM_NO_LIVE means the gateway found no healthy backend to send to. That’s the probe cascade.
  • ERRORINFO_UPSTREAM_TIMED_OUT means the backend took longer than the configured timeout.
  • ERRORINFO_UPSTREAM_CLOSED_CONNECTION means the backend closed the connection before the request finished, for example after a crash or at a connection limit.

The same failures on other platforms are in the AWS ALB 502 and 504 recipe and the nginx upstream timeouts recipe.

Application gateway backend health unhealthy fixes

Start by saving what’s there now:

RG=web-prod
AGW=agw-web
az network application-gateway show -g "$RG" -n "$AGW" \
  --query "{probes: probes, settings: backendHttpSettingsCollection, autoscale: autoscaleConfiguration}" \
  > agw-before.json
az network application-gateway show-backend-health -g "$RG" -n "$AGW" > agw-health-before.json

show-backend-health lists each backend server as Healthy, Unhealthy or Unknown, with the reason for each failure. Read the reasons before you change anything. A TCP connect error points at an NSG or a stopped service, and a probe timeout points at a slow health endpoint.

Give the probe a cheap endpoint. Add a /healthz path to your app that returns 200 without touching the database or any downstream service. It answers one question: can this process take requests? Then create a custom probe for it:

az network application-gateway probe create -g "$RG" --gateway-name "$AGW" \
  --name healthz \
  --protocol Http \
  --host-name-from-http-settings true \
  --path /healthz \
  --interval 10 \
  --timeout 5 \
  --threshold 3 \
  --match-status-codes 200-399

Each setting, and what goes wrong when it’s off:

  • --path points the probe at the cheap endpoint. Probe a path that does real work, and peak load fails it.
  • --interval is the gap between probes. Shorter catches a dead backend sooner. Each gateway instance probes on its own, so the backend sees more probe traffic than one per interval.
  • --timeout is how long the gateway waits for an answer. Keep it below the interval. A cheap endpoint answers in milliseconds, so a few seconds is generous.
  • --threshold is how many failures in a row mark a backend down. Too low, and one slow garbage collection pause pulls a healthy backend. Too high, and a dead backend keeps getting traffic for interval times threshold seconds.
  • --host-name-from-http-settings sends the host name from the backend setting. App Service and most multi-tenant backends need a real host header.

I keep a deep health check too, one that does touch the database, but I point monitoring at it, never the gateway. If the database goes down, every backend fails a deep probe at once and the gateway returns 502 for pages that didn’t need the database.

Attach the probe and size the timeout. Update the backend setting:

az network application-gateway http-settings update -g "$RG" --gateway-name "$AGW" \
  --name web-settings \
  --probe healthz \
  --enable-probe true \
  --timeout 30 \
  --connection-draining-timeout 60
  • --timeout is the request timeout, from 1 to 86400 seconds. Set it above the p99 of your slowest normal endpoint at peak, with room to spare. Too low, and slow but valid requests become 502 on v1 or 504 on v2. Too high, and a hung backend holds gateway connections for that long. A request that needs minutes belongs in a background job.
  • --connection-draining-timeout lets in-flight requests finish when a backend leaves the pool, for a deploy or a scale-in. Without it, a removed backend drops its open requests, and they come back as 502. Set it above your slowest normal request.

Keep-alive on the backend. The gateway reuses connections to your backends. Microsoft documents a 120-second keep-alive for client connections on both v1 and v2, but doesn’t publish the backend-side idle timeout. I set the backend’s keep-alive above 120 seconds anyway, for example keepalive_timeout 130s; in nginx or server.keepAliveTimeout = 130000 in Node.js. If ERRORINFO_UPSTREAM_CLOSED_CONNECTION appears in idle gaps between bursts, this is the first thing I check.

Autoscale floor. A v2 gateway scales its own instances, and new instances take a few minutes to come up. Set a minimum instance count that covers your normal peak, so the gateway doesn’t need to scale out during the spike itself. In Terraform:

resource "azurerm_application_gateway" "web" {
  # changed lines only
  autoscale_configuration {
    min_capacity = 3
    max_capacity = 20
  }
}

Backends that can’t keep up. If backends fail even the cheap probe at peak, the problem is capacity. Scale the backend pool out earlier. On AKS behind the Application Gateway Ingress Controller, the HPA recipe covers scaling out early enough. On App Service, scale out the plan on a schedule before known peaks.

Test it on your system

Run k6 against a staging gateway with the same SKU, probe and backend settings as production. Use a fixed arrival rate so a slow backend can’t throttle the test:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: 400,             // requests per second; set this to your peak
      timeUnit: '1s',
      duration: '20m',
      preAllocatedVUs: 300,
      maxVUs: 1500,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(99)<1000'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/products?page=${Math.ceil(Math.random() * 50)}`);
  check(res, { 'no 502 or 504': (r) => r.status !== 502 && r.status !== 504 });
}
k6 run -e BASE_URL=https://staging.example.com agw-peak.js

Halfway through, stop one backend on purpose, or drain it from the pool, and bring it back five minutes later. That shows you how long the probe takes to notice and whether draining holds. While it runs, watch:

  • az network application-gateway show-backend-health every minute, to see when each backend changes state.
  • The UnhealthyHostCount, ResponseStatus and BackendLastByteResponseTime metrics.
  • The access log, filtered to httpStatus 502 and 504, grouped by errorInfo.

Write your targets first. It passes when no backend leaves the pool at peak except the one you stopped. The error rate should stay inside your budget through the stop and restart, with p99 under your target. Run the same test against the old probe to see the cascade it fixes.

Keep it healthy

azure-metrics-exporter from webdevops reads Application Gateway metrics from Azure Monitor and serves them to Prometheus:

scrape_configs:
  - job_name: azure-appgw-status
    scrape_interval: 1m
    scrape_timeout: 55s
    metrics_path: /probe/metrics/list
    params:
      name: ["azure_appgw_status"]
      subscription: ["00000000-0000-0000-0000-000000000000"]
      resourceType: ["Microsoft.Network/applicationGateways"]
      metric: ["ResponseStatus", "BackendResponseStatus"]
      aggregation: ["total"]
      interval: ["PT5M"]
      timespan: ["PT5M"]
      metricFilter: ["HttpStatusGroup eq '*'"]
    static_configs:
      - targets: ["azure-metrics-exporter:8080"]
  - job_name: azure-appgw-hosts
    scrape_interval: 1m
    scrape_timeout: 55s
    metrics_path: /probe/metrics/list
    params:
      name: ["azure_appgw_hosts"]
      subscription: ["00000000-0000-0000-0000-000000000000"]
      resourceType: ["Microsoft.Network/applicationGateways"]
      metric: ["UnhealthyHostCount", "HealthyHostCount"]
      aggregation: ["average"]
      interval: ["PT5M"]
      timespan: ["PT5M"]
      metricFilter: ["BackendSettingsPool eq '*'"]
    static_configs:
      - targets: ["azure-metrics-exporter:8080"]

The exporter puts the Azure metric name in the metric label and the split dimension’s value in dimension, such as 5xx for a status group. ResponseStatus counts every code the gateway returned. BackendResponseStatus counts only codes the backends sent. The difference is what the gateway generated itself.

groups:
  - name: azure-application-gateway
    rules:
      - alert: AppGatewayGenerated5xx
        expr: |
          (
            sum by (resourceName) (azure_appgw_status{metric="ResponseStatus",dimension="5xx"})
            - sum by (resourceName) (azure_appgw_status{metric="BackendResponseStatus",dimension="5xx"})
          ) / sum by (resourceName) (azure_appgw_status{metric="ResponseStatus"}) > 0.001
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Application Gateway {{ $labels.resourceName }} is returning 5xx errors it generated itself"
          description: "Run az network application-gateway show-backend-health and group the access log by errorInfo. ERRORINFO_UPSTREAM_NO_LIVE means the probe emptied the pool, and ERRORINFO_UPSTREAM_TIMED_OUT means the request timeout is too low for a slow path."
      - alert: AppGatewayNoHealthyBackends
        expr: azure_appgw_hosts{metric="HealthyHostCount",aggregation="average"} < 1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Pool {{ $labels.dimension }} on {{ $labels.resourceName }} has no healthy backends"
          description: "Every request to this pool gets a 502. Check show-backend-health for the failure reason, then check the backends' NSGs and the probe path."
      - alert: AppGatewayUnhealthyBackends
        expr: azure_appgw_hosts{metric="UnhealthyHostCount",aggregation="average"} > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Pool {{ $labels.dimension }} on {{ $labels.resourceName }} has unhealthy backends"
          description: "The pool runs short of capacity. Check whether the probe path does real work, and compare backend CPU and latency with the healthy members."

Set the 0.1% threshold from your own error budget. Azure Monitor metrics arrive a few minutes late, so these alerts trail the first errors.

Roll back

agw-before.json holds the old probes and backend settings. To go back to the default probe, detach the custom one from the backend setting and restore the old timeout:

az network application-gateway http-settings update -g "$RG" --gateway-name "$AGW" \
  --name web-settings \
  --enable-probe false \
  --timeout 20
az network application-gateway probe delete -g "$RG" --gateway-name "$AGW" --name healthz

Use the timeout value from your saved file in place of 20 if it differed. A gateway update takes a few minutes to apply. If the new probe marks healthy backends as unhealthy and causes 502s, roll back now. For any other revert, wait for an off-peak window.

More recipes

Further reading