Azure SNAT port exhaustion happens when your instances open outbound connections to the same destination faster than Azure can hand out source ports. New connections wait or fail until a port frees up. Azure doesn’t log an error with that name in your app, so you need the right metrics and alerts to see it.
What you miss today
Your instances look healthy. CPU and memory are normal, and the load balancer’s health probes pass. Microsoft lists what your app shows instead, for apps and functions on App Service:
- Slow response times on all or some of the instances in a service plan.
- Intermittent 5xx or Bad Gateway errors.
- Time-out error messages.
- Inability to connect to external endpoints, such as SQL databases or other app services.
Each one points somewhere else first, at the database, the partner API or the network. The real cause is the number of outbound flows to one destination.
The port math. Every outbound flow to a public address takes a source port from a fixed pool. On App Service, each instance starts with 128 preallocated SNAT ports. Ports only run out for repeated connections to the same destination address and port. A closed connection doesn’t return its port at once either. Azure Load Balancer reclaims SNAT ports from closed connections only after four minutes. An app that opens a new connection per request to one API can burn through 128 ports in seconds at peak. Then it waits minutes to get them back.
VMs and scale sets behind a Standard Load Balancer get a fixed number of ports per instance. The outbound rule or the default allocation table sets that number. Azure NAT Gateway gives each public IP about 64,000 ports, shared by every resource in its subnets. It supports up to 50,000 connections per public IP to one destination.
Why App Service is the hard case. Microsoft says SNAT port usage on App Service isn’t available as a metric. You can’t graph it, alert on it or autoscale on it. The SNAT Port Exhaustion detector under Diagnose and solve problems, in the Availability and Performance category, shows allocation after the fact. So I route a busy App Service’s outbound traffic through a NAT gateway with virtual network integration. Then the ports show up in metrics I can alert on.
The same failure exists on the other clouds as NAT gateway ErrorPortAllocation on AWS and Cloud NAT port exhaustion on Google Cloud.
The exporter and metrics
Azure Monitor holds the metrics. azure-metrics-exporter from webdevops reads them through the Azure Monitor API and serves them to Prometheus. Run it as a container with a managed identity or a service principal that has the Monitoring Reader role on the subscription. Prometheus scrapes its /probe/metrics/list endpoint, and the scrape parameters choose the resource type, metrics, aggregation and dimension split.
The metrics that matter:
| Resource | Metric (REST name) | What it tells you |
|---|---|---|
| Load Balancer | SnatConnectionCount, split by ConnectionState | New outbound flows. Microsoft’s guidance is that a failed count above zero indicates SNAT port exhaustion. |
| Load Balancer | UsedSnatPorts, split by BackendIPAddress | Ports each backend instance holds now. |
| Load Balancer | AllocatedSnatPorts, split by BackendIPAddress | Ports each backend instance may use. Divide used by allocated for the risk. |
| NAT Gateway | SNATConnectionCount, split by ConnectionState | New connections, with Failed as the exhaustion signal. |
| NAT Gateway | TotalConnectionCount | Active connections through the gateway, against its total connection limit. |
| NAT Gateway | PacketDropCount | Dropped packets. Check whether drops line up with failed connections. |
The scrape jobs, one per metric shape, since each needs its own dimension split:
scrape_configs:
- job_name: azure-lb-snat-ports
scrape_interval: 1m
scrape_timeout: 55s
metrics_path: /probe/metrics/list
params:
name: ["azure_lb_snat_ports"]
subscription: ["00000000-0000-0000-0000-000000000000"]
resourceType: ["Microsoft.Network/loadBalancers"]
metric: ["UsedSnatPorts", "AllocatedSnatPorts"]
aggregation: ["average"]
interval: ["PT5M"]
timespan: ["PT5M"]
metricFilter: ["BackendIPAddress eq '*'"]
metricTop: ["100"]
static_configs:
- targets: ["azure-metrics-exporter:8080"]
- job_name: azure-lb-snat-connections
scrape_interval: 1m
scrape_timeout: 55s
metrics_path: /probe/metrics/list
params:
name: ["azure_lb_snat_connections"]
subscription: ["00000000-0000-0000-0000-000000000000"]
resourceType: ["Microsoft.Network/loadBalancers"]
metric: ["SnatConnectionCount"]
aggregation: ["total"]
interval: ["PT5M"]
timespan: ["PT5M"]
metricFilter: ["ConnectionState eq '*'"]
static_configs:
- targets: ["azure-metrics-exporter:8080"]
- job_name: azure-natgw
scrape_interval: 1m
scrape_timeout: 55s
metrics_path: /probe/metrics/list
params:
name: ["azure_natgw"]
subscription: ["00000000-0000-0000-0000-000000000000"]
resourceType: ["Microsoft.Network/natGateways"]
metric: ["SNATConnectionCount"]
aggregation: ["total"]
interval: ["PT5M"]
timespan: ["PT5M"]
metricFilter: ["ConnectionState eq '*'"]
static_configs:
- targets: ["azure-metrics-exporter:8080"]
- job_name: azure-natgw-totals
scrape_interval: 1m
scrape_timeout: 55s
metrics_path: /probe/metrics/list
params:
name: ["azure_natgw_totals"]
subscription: ["00000000-0000-0000-0000-000000000000"]
resourceType: ["Microsoft.Network/natGateways"]
metric: ["TotalConnectionCount", "PacketDropCount"]
aggregation: ["total"]
interval: ["PT5M"]
timespan: ["PT5M"]
static_configs:
- targets: ["azure-metrics-exporter:8080"]
With its default name template, the exporter emits one Prometheus metric per job, named after the name parameter. The Azure metric goes in the metric label, the aggregation in aggregation, and the resource in resourceID and resourceName. A split on one dimension puts its value in a label called dimension. So failed NAT gateway connections arrive as azure_natgw{metric="SNATConnectionCount",aggregation="total",dimension="Failed"}. Run the exporter’s /query web UI once against your own resources and check the labels before you write rules on them.
The alert rules
groups:
- name: azure-snat
rules:
- alert: AzureLbSnatConnectionsFailing
expr: |
sum by (resourceName) (
azure_lb_snat_connections{metric="SnatConnectionCount",aggregation="total",dimension="Failed"}
) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Outbound SNAT connections are failing on load balancer {{ $labels.resourceName }}"
description: "Instances behind this load balancer can't open outbound connections. Check UsedSnatPorts per backend, find the destination the app calls most and pool connections to it, or move outbound traffic to a NAT gateway."
- alert: AzureLbSnatPortsHigh
expr: |
azure_lb_snat_ports{metric="UsedSnatPorts",aggregation="average"}
/ ignoring (metric)
azure_lb_snat_ports{metric="AllocatedSnatPorts",aggregation="average"} > 0.75
for: 10m
labels:
severity: warning
annotations:
summary: "Backend {{ $labels.dimension }} on {{ $labels.resourceName }} uses over 75% of its SNAT ports"
description: "This instance is near SNAT exhaustion. Raise allocated outbound ports on the outbound rule, add frontend IPs or scale out, then fix connection reuse in the app."
- alert: AzureNatGatewaySnatFailing
expr: |
sum by (resourceName) (
azure_natgw{metric="SNATConnectionCount",aggregation="total",dimension="Failed"}
) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "NAT gateway {{ $labels.resourceName }} is failing outbound connections"
description: "Check that the TCP idle timeout is still the 4-minute default, then add public IPs or a public IP prefix to the NAT gateway. Find the top destination and pool connections to it."
- alert: AzureNatGatewayConnectionsNearLimit
expr: |
azure_natgw_totals{metric="TotalConnectionCount",aggregation="total"} > 1600000
for: 10m
labels:
severity: warning
annotations:
summary: "NAT gateway {{ $labels.resourceName }} is above 80% of its 2 million connection limit"
description: "Find the subnets that open the most connections and split them onto a second NAT gateway, or cut idle connections with shorter pool idle timeouts."
The two Failed rules page, because users are already hitting errors. The ports rule follows Microsoft’s guidance for a low-severity alert at 75% of allocated ports. Add a second copy at 90% with severity: critical if your outbound traffic grows in fast bursts.
What fixes the alerts point to. Reuse connections first. One pooled HTTP client per process stops most SNAT exhaustion on its own. On .NET that’s IHttpClientFactory, covered in the HttpClient socket exhaustion recipe. Then take Azure traffic off SNAT. Private endpoints and service endpoints don’t use SNAT ports for the services they cover, such as Azure SQL and Storage. For the rest, route outbound traffic through a NAT gateway and add public IPs as you grow.
Test the alerts
Use a staging load balancer with an outbound rule that gives each backend a small, fixed port allocation, such as 1,024. Put one VM in its backend pool. Outside that VNet, run a target you own that accepts many TCP connections on one port. From the VM, open connections and hold them, so they pile up against one destination:
const net = require('node:net');
const [host, port, count] = [process.argv[2], Number(process.argv[3]), Number(process.argv[4])];
const sockets = [];
let failed = 0;
for (let i = 0; i < count; i++) {
const s = net.connect({ host, port });
s.setTimeout(10000); // connect timeout only
s.on('connect', () => s.setTimeout(0)); // then hold the socket open
s.on('timeout', () => s.destroy(new Error('connect timeout')));
s.on('error', () => { failed++; });
sockets.push(s);
}
setTimeout(() => {
console.log(`failed connects: ${failed} of ${count}`);
sockets.forEach((s) => s.destroy());
}, 600000);
node hold-connections.js 203.0.113.10 443 1500
Ask for more connections than the allocation. Within a few minutes, UsedSnatPorts for that VM should reach AllocatedSnatPorts and SnatConnectionCount should show Failed flows. AzureLbSnatPortsHigh fires first, then AzureLbSnatConnectionsFailing. Stop the script and confirm both clear. Ports come back about four minutes after the connections close, so give it time.
For a NAT gateway, run the same script from a VM in a subnet behind it, with one public IP on the gateway. A NAT gateway has far more ports, so spread the load across several VMs and one destination. For App Service, deploy a staging build that creates a new HttpClient per request against one endpoint. Drive it with k6 at your peak rate. The NAT gateway alerts should fire if the app routes outbound traffic through it. Compare the timing with the SNAT Port Exhaustion detector afterwards. Only run these against endpoints you own.
Check the rule logic offline as well, with promtool test rules and made-up series. Then a later edit can’t break a label match unnoticed.
Tune the noise
Keep the two Failed rules at > 0. A failed SNAT connection is a request your app couldn’t send. Tune the for instead. Azure Monitor publishes these metrics with a delay of a few minutes, and a 5-minute timespan smooths single blips. Shorter values add noise and don’t page you any sooner.
Set the port threshold from your own peak. Pull two weeks of history and find each backend’s busiest hour:
max_over_time(
(azure_lb_snat_ports{metric="UsedSnatPorts",aggregation="average"}
/ ignoring (metric)
azure_lb_snat_ports{metric="AllocatedSnatPorts",aggregation="average"})[14d:5m]
)
If normal peaks already reach 60%, a 75% alert fires on every busy day. Fix the allocation or the app first, then set the alert above the new peak. A scale set that adds instances changes the allocation per instance under the default table, so check the ratio again after each scale-out.
One backend IP may fire over and over while its peers stay low. The cause then sits in that instance’s workload, often a background job with its own client. Silence the alert for the duration of a known batch run rather than raising the threshold for everyone.
More recipes
- Fix Application Gateway 502 Bad Gateway at Peak
Give Application Gateway a cheap health probe and a request timeout sized from your p99, so busy backends stay in the pool and peak traffic stops getting 502.
- Fix StackExchange.Redis RedisTimeoutException on Azure
Read each Azure Cache for Redis timeout message, then fix the thread pool, the multiplexer or the server load it points to, so peak traffic stops timing out.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.