An Application Gateway 502 Bad Gateway at peak often starts with the health probe. A busy backend answers the probe late, the gateway marks it unhealthy, and its traffic lands on the others. They slow down, fail their probes too, and the pool empties. The gateway then has nowhere to send requests.
What breaks at peak
Your users get this error page from the gateway:
502 - Web server received an invalid response while acting as a gateway or proxy server.
The Azure portal and docs show it with a Server Error: prefix. Microsoft lists the common causes:
- An NSG, user-defined route or custom DNS blocks the gateway from the backend.
- Backends don’t answer the default health probe.
- A custom health probe is misconfigured.
- The backend pool is empty, or every instance in it is unhealthy.
- The backend took longer than the request timeout.
- The backend certificate doesn’t match the host name the gateway expects.
The first and last causes break things all day, so you’d notice them in testing. The middle ones show up at peak.
The probe cascade. The default probe sends a request to http://127.0.0.1/ every 30 seconds, with a 30-second timeout and an unhealthy threshold of 3. It hits the app’s root path, which often renders a full page and queries the database. At peak, that page slows down like every other page. A backend that misses three probes in a row leaves the pool. The remaining backends take its share, slow down further and drop out next. Once the pool is empty, every request gets a 502.
The request timeout. Each backend setting has a request timeout, 20 seconds by default. On v1, a backend slower than that returns a 502 to the client. On v2, the gateway retries the request against a second backend member, and if that also times out, the client gets a 504.
Closed connections. A backend that crashes, restarts or closes a keep-alive connection while the gateway sends a request on it also produces a 502.
The access log tells you which case you have. Turn on the ApplicationGatewayAccessLog diagnostic setting and read the serverStatus and errorInfo fields. Microsoft documents these errorInfo values for backend failures:
ERRORINFO_UPSTREAM_NO_LIVEmeans the gateway found no healthy backend to send to. That’s the probe cascade.ERRORINFO_UPSTREAM_TIMED_OUTmeans the backend took longer than the configured timeout.ERRORINFO_UPSTREAM_CLOSED_CONNECTIONmeans the backend closed the connection before the request finished, for example after a crash or at a connection limit.
The same failures on other platforms are in the AWS ALB 502 and 504 recipe and the nginx upstream timeouts recipe.
Application gateway backend health unhealthy fixes
Start by saving what’s there now:
RG=web-prod
AGW=agw-web
az network application-gateway show -g "$RG" -n "$AGW" \
--query "{probes: probes, settings: backendHttpSettingsCollection, autoscale: autoscaleConfiguration}" \
> agw-before.json
az network application-gateway show-backend-health -g "$RG" -n "$AGW" > agw-health-before.json
show-backend-health lists each backend server as Healthy, Unhealthy or Unknown, with the reason for each failure. Read the reasons before you change anything. A TCP connect error points at an NSG or a stopped service, and a probe timeout points at a slow health endpoint.
Give the probe a cheap endpoint. Add a /healthz path to your app that returns 200 without touching the database or any downstream service. It answers one question: can this process take requests? Then create a custom probe for it:
az network application-gateway probe create -g "$RG" --gateway-name "$AGW" \
--name healthz \
--protocol Http \
--host-name-from-http-settings true \
--path /healthz \
--interval 10 \
--timeout 5 \
--threshold 3 \
--match-status-codes 200-399
Each setting, and what goes wrong when it’s off:
--pathpoints the probe at the cheap endpoint. Probe a path that does real work, and peak load fails it.--intervalis the gap between probes. Shorter catches a dead backend sooner. Each gateway instance probes on its own, so the backend sees more probe traffic than one per interval.--timeoutis how long the gateway waits for an answer. Keep it below the interval. A cheap endpoint answers in milliseconds, so a few seconds is generous.--thresholdis how many failures in a row mark a backend down. Too low, and one slow garbage collection pause pulls a healthy backend. Too high, and a dead backend keeps getting traffic for interval times threshold seconds.--host-name-from-http-settingssends the host name from the backend setting. App Service and most multi-tenant backends need a real host header.
I keep a deep health check too, one that does touch the database, but I point monitoring at it, never the gateway. If the database goes down, every backend fails a deep probe at once and the gateway returns 502 for pages that didn’t need the database.
Attach the probe and size the timeout. Update the backend setting:
az network application-gateway http-settings update -g "$RG" --gateway-name "$AGW" \
--name web-settings \
--probe healthz \
--enable-probe true \
--timeout 30 \
--connection-draining-timeout 60
--timeoutis the request timeout, from 1 to 86400 seconds. Set it above the p99 of your slowest normal endpoint at peak, with room to spare. Too low, and slow but valid requests become 502 on v1 or 504 on v2. Too high, and a hung backend holds gateway connections for that long. A request that needs minutes belongs in a background job.--connection-draining-timeoutlets in-flight requests finish when a backend leaves the pool, for a deploy or a scale-in. Without it, a removed backend drops its open requests, and they come back as 502. Set it above your slowest normal request.
Keep-alive on the backend. The gateway reuses connections to your backends. Microsoft documents a 120-second keep-alive for client connections on both v1 and v2, but doesn’t publish the backend-side idle timeout. I set the backend’s keep-alive above 120 seconds anyway, for example keepalive_timeout 130s; in nginx or server.keepAliveTimeout = 130000 in Node.js. If ERRORINFO_UPSTREAM_CLOSED_CONNECTION appears in idle gaps between bursts, this is the first thing I check.
Autoscale floor. A v2 gateway scales its own instances, and new instances take a few minutes to come up. Set a minimum instance count that covers your normal peak, so the gateway doesn’t need to scale out during the spike itself. In Terraform:
resource "azurerm_application_gateway" "web" {
# changed lines only
autoscale_configuration {
min_capacity = 3
max_capacity = 20
}
}
Backends that can’t keep up. If backends fail even the cheap probe at peak, the problem is capacity. Scale the backend pool out earlier. On AKS behind the Application Gateway Ingress Controller, the HPA recipe covers scaling out early enough. On App Service, scale out the plan on a schedule before known peaks.
Test it on your system
Run k6 against a staging gateway with the same SKU, probe and backend settings as production. Use a fixed arrival rate so a slow backend can’t throttle the test:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: 400, // requests per second; set this to your peak
timeUnit: '1s',
duration: '20m',
preAllocatedVUs: 300,
maxVUs: 1500,
},
},
thresholds: {
http_req_failed: ['rate<0.001'],
http_req_duration: ['p(99)<1000'],
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/products?page=${Math.ceil(Math.random() * 50)}`);
check(res, { 'no 502 or 504': (r) => r.status !== 502 && r.status !== 504 });
}
k6 run -e BASE_URL=https://staging.example.com agw-peak.js
Halfway through, stop one backend on purpose, or drain it from the pool, and bring it back five minutes later. That shows you how long the probe takes to notice and whether draining holds. While it runs, watch:
az network application-gateway show-backend-healthevery minute, to see when each backend changes state.- The
UnhealthyHostCount,ResponseStatusandBackendLastByteResponseTimemetrics. - The access log, filtered to
httpStatus502 and 504, grouped byerrorInfo.
Write your targets first. It passes when no backend leaves the pool at peak except the one you stopped. The error rate should stay inside your budget through the stop and restart, with p99 under your target. Run the same test against the old probe to see the cascade it fixes.
Keep it healthy
azure-metrics-exporter from webdevops reads Application Gateway metrics from Azure Monitor and serves them to Prometheus:
scrape_configs:
- job_name: azure-appgw-status
scrape_interval: 1m
scrape_timeout: 55s
metrics_path: /probe/metrics/list
params:
name: ["azure_appgw_status"]
subscription: ["00000000-0000-0000-0000-000000000000"]
resourceType: ["Microsoft.Network/applicationGateways"]
metric: ["ResponseStatus", "BackendResponseStatus"]
aggregation: ["total"]
interval: ["PT5M"]
timespan: ["PT5M"]
metricFilter: ["HttpStatusGroup eq '*'"]
static_configs:
- targets: ["azure-metrics-exporter:8080"]
- job_name: azure-appgw-hosts
scrape_interval: 1m
scrape_timeout: 55s
metrics_path: /probe/metrics/list
params:
name: ["azure_appgw_hosts"]
subscription: ["00000000-0000-0000-0000-000000000000"]
resourceType: ["Microsoft.Network/applicationGateways"]
metric: ["UnhealthyHostCount", "HealthyHostCount"]
aggregation: ["average"]
interval: ["PT5M"]
timespan: ["PT5M"]
metricFilter: ["BackendSettingsPool eq '*'"]
static_configs:
- targets: ["azure-metrics-exporter:8080"]
The exporter puts the Azure metric name in the metric label and the split dimension’s value in dimension, such as 5xx for a status group. ResponseStatus counts every code the gateway returned. BackendResponseStatus counts only codes the backends sent. The difference is what the gateway generated itself.
groups:
- name: azure-application-gateway
rules:
- alert: AppGatewayGenerated5xx
expr: |
(
sum by (resourceName) (azure_appgw_status{metric="ResponseStatus",dimension="5xx"})
- sum by (resourceName) (azure_appgw_status{metric="BackendResponseStatus",dimension="5xx"})
) / sum by (resourceName) (azure_appgw_status{metric="ResponseStatus"}) > 0.001
for: 5m
labels:
severity: critical
annotations:
summary: "Application Gateway {{ $labels.resourceName }} is returning 5xx errors it generated itself"
description: "Run az network application-gateway show-backend-health and group the access log by errorInfo. ERRORINFO_UPSTREAM_NO_LIVE means the probe emptied the pool, and ERRORINFO_UPSTREAM_TIMED_OUT means the request timeout is too low for a slow path."
- alert: AppGatewayNoHealthyBackends
expr: azure_appgw_hosts{metric="HealthyHostCount",aggregation="average"} < 1
for: 2m
labels:
severity: critical
annotations:
summary: "Pool {{ $labels.dimension }} on {{ $labels.resourceName }} has no healthy backends"
description: "Every request to this pool gets a 502. Check show-backend-health for the failure reason, then check the backends' NSGs and the probe path."
- alert: AppGatewayUnhealthyBackends
expr: azure_appgw_hosts{metric="UnhealthyHostCount",aggregation="average"} > 0
for: 10m
labels:
severity: warning
annotations:
summary: "Pool {{ $labels.dimension }} on {{ $labels.resourceName }} has unhealthy backends"
description: "The pool runs short of capacity. Check whether the probe path does real work, and compare backend CPU and latency with the healthy members."
Set the 0.1% threshold from your own error budget. Azure Monitor metrics arrive a few minutes late, so these alerts trail the first errors.
Roll back
agw-before.json holds the old probes and backend settings. To go back to the default probe, detach the custom one from the backend setting and restore the old timeout:
az network application-gateway http-settings update -g "$RG" --gateway-name "$AGW" \
--name web-settings \
--enable-probe false \
--timeout 20
az network application-gateway probe delete -g "$RG" --gateway-name "$AGW" --name healthz
Use the timeout value from your saved file in place of 20 if it differed. A gateway update takes a few minutes to apply. If the new probe marks healthy backends as unhealthy and causes 502s, roll back now. For any other revert, wait for an off-peak window.
More recipes
- Fix StackExchange.Redis RedisTimeoutException on Azure
Read each Azure Cache for Redis timeout message, then fix the thread pool, the multiplexer or the server load it points to, so peak traffic stops timing out.
- Azure SNAT Port Exhaustion Alerts with Prometheus
Alert on failed SNAT connections and used ports per instance, so you catch App Service intermittent outbound connection errors before users report them.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.