Azure SNAT Port Exhaustion Alerts with Prometheus

Alert on failed SNAT connections and used ports per instance, so you catch App Service intermittent outbound connection errors before users report them.

Published

Applies to Standard Load Balancer, NAT Gateway Standard and StandardV2, App Service with VNet integration; azure-metrics-exporter 24.9 and later

Azure SNAT port exhaustion happens when your instances open outbound connections to the same destination faster than Azure can hand out source ports. New connections wait or fail until a port frees up. Azure doesn’t log an error with that name in your app, so you need the right metrics and alerts to see it.

What you miss today

Your instances look healthy. CPU and memory are normal, and the load balancer’s health probes pass. Microsoft lists what your app shows instead, for apps and functions on App Service:

  • Slow response times on all or some of the instances in a service plan.
  • Intermittent 5xx or Bad Gateway errors.
  • Time-out error messages.
  • Inability to connect to external endpoints, such as SQL databases or other app services.

Each one points somewhere else first, at the database, the partner API or the network. The real cause is the number of outbound flows to one destination.

The port math. Every outbound flow to a public address takes a source port from a fixed pool. On App Service, each instance starts with 128 preallocated SNAT ports. Ports only run out for repeated connections to the same destination address and port. A closed connection doesn’t return its port at once either. Azure Load Balancer reclaims SNAT ports from closed connections only after four minutes. An app that opens a new connection per request to one API can burn through 128 ports in seconds at peak. Then it waits minutes to get them back.

VMs and scale sets behind a Standard Load Balancer get a fixed number of ports per instance. The outbound rule or the default allocation table sets that number. Azure NAT Gateway gives each public IP about 64,000 ports, shared by every resource in its subnets. It supports up to 50,000 connections per public IP to one destination.

Why App Service is the hard case. Microsoft says SNAT port usage on App Service isn’t available as a metric. You can’t graph it, alert on it or autoscale on it. The SNAT Port Exhaustion detector under Diagnose and solve problems, in the Availability and Performance category, shows allocation after the fact. So I route a busy App Service’s outbound traffic through a NAT gateway with virtual network integration. Then the ports show up in metrics I can alert on.

The same failure exists on the other clouds as NAT gateway ErrorPortAllocation on AWS and Cloud NAT port exhaustion on Google Cloud.

The exporter and metrics

Azure Monitor holds the metrics. azure-metrics-exporter from webdevops reads them through the Azure Monitor API and serves them to Prometheus. Run it as a container with a managed identity or a service principal that has the Monitoring Reader role on the subscription. Prometheus scrapes its /probe/metrics/list endpoint, and the scrape parameters choose the resource type, metrics, aggregation and dimension split.

The metrics that matter:

ResourceMetric (REST name)What it tells you
Load BalancerSnatConnectionCount, split by ConnectionStateNew outbound flows. Microsoft’s guidance is that a failed count above zero indicates SNAT port exhaustion.
Load BalancerUsedSnatPorts, split by BackendIPAddressPorts each backend instance holds now.
Load BalancerAllocatedSnatPorts, split by BackendIPAddressPorts each backend instance may use. Divide used by allocated for the risk.
NAT GatewaySNATConnectionCount, split by ConnectionStateNew connections, with Failed as the exhaustion signal.
NAT GatewayTotalConnectionCountActive connections through the gateway, against its total connection limit.
NAT GatewayPacketDropCountDropped packets. Check whether drops line up with failed connections.

The scrape jobs, one per metric shape, since each needs its own dimension split:

scrape_configs:
  - job_name: azure-lb-snat-ports
    scrape_interval: 1m
    scrape_timeout: 55s
    metrics_path: /probe/metrics/list
    params:
      name: ["azure_lb_snat_ports"]
      subscription: ["00000000-0000-0000-0000-000000000000"]
      resourceType: ["Microsoft.Network/loadBalancers"]
      metric: ["UsedSnatPorts", "AllocatedSnatPorts"]
      aggregation: ["average"]
      interval: ["PT5M"]
      timespan: ["PT5M"]
      metricFilter: ["BackendIPAddress eq '*'"]
      metricTop: ["100"]
    static_configs:
      - targets: ["azure-metrics-exporter:8080"]
  - job_name: azure-lb-snat-connections
    scrape_interval: 1m
    scrape_timeout: 55s
    metrics_path: /probe/metrics/list
    params:
      name: ["azure_lb_snat_connections"]
      subscription: ["00000000-0000-0000-0000-000000000000"]
      resourceType: ["Microsoft.Network/loadBalancers"]
      metric: ["SnatConnectionCount"]
      aggregation: ["total"]
      interval: ["PT5M"]
      timespan: ["PT5M"]
      metricFilter: ["ConnectionState eq '*'"]
    static_configs:
      - targets: ["azure-metrics-exporter:8080"]
  - job_name: azure-natgw
    scrape_interval: 1m
    scrape_timeout: 55s
    metrics_path: /probe/metrics/list
    params:
      name: ["azure_natgw"]
      subscription: ["00000000-0000-0000-0000-000000000000"]
      resourceType: ["Microsoft.Network/natGateways"]
      metric: ["SNATConnectionCount"]
      aggregation: ["total"]
      interval: ["PT5M"]
      timespan: ["PT5M"]
      metricFilter: ["ConnectionState eq '*'"]
    static_configs:
      - targets: ["azure-metrics-exporter:8080"]
  - job_name: azure-natgw-totals
    scrape_interval: 1m
    scrape_timeout: 55s
    metrics_path: /probe/metrics/list
    params:
      name: ["azure_natgw_totals"]
      subscription: ["00000000-0000-0000-0000-000000000000"]
      resourceType: ["Microsoft.Network/natGateways"]
      metric: ["TotalConnectionCount", "PacketDropCount"]
      aggregation: ["total"]
      interval: ["PT5M"]
      timespan: ["PT5M"]
    static_configs:
      - targets: ["azure-metrics-exporter:8080"]

With its default name template, the exporter emits one Prometheus metric per job, named after the name parameter. The Azure metric goes in the metric label, the aggregation in aggregation, and the resource in resourceID and resourceName. A split on one dimension puts its value in a label called dimension. So failed NAT gateway connections arrive as azure_natgw{metric="SNATConnectionCount",aggregation="total",dimension="Failed"}. Run the exporter’s /query web UI once against your own resources and check the labels before you write rules on them.

The alert rules

groups:
  - name: azure-snat
    rules:
      - alert: AzureLbSnatConnectionsFailing
        expr: |
          sum by (resourceName) (
            azure_lb_snat_connections{metric="SnatConnectionCount",aggregation="total",dimension="Failed"}
          ) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Outbound SNAT connections are failing on load balancer {{ $labels.resourceName }}"
          description: "Instances behind this load balancer can't open outbound connections. Check UsedSnatPorts per backend, find the destination the app calls most and pool connections to it, or move outbound traffic to a NAT gateway."
      - alert: AzureLbSnatPortsHigh
        expr: |
          azure_lb_snat_ports{metric="UsedSnatPorts",aggregation="average"}
            / ignoring (metric)
          azure_lb_snat_ports{metric="AllocatedSnatPorts",aggregation="average"} > 0.75
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Backend {{ $labels.dimension }} on {{ $labels.resourceName }} uses over 75% of its SNAT ports"
          description: "This instance is near SNAT exhaustion. Raise allocated outbound ports on the outbound rule, add frontend IPs or scale out, then fix connection reuse in the app."
      - alert: AzureNatGatewaySnatFailing
        expr: |
          sum by (resourceName) (
            azure_natgw{metric="SNATConnectionCount",aggregation="total",dimension="Failed"}
          ) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "NAT gateway {{ $labels.resourceName }} is failing outbound connections"
          description: "Check that the TCP idle timeout is still the 4-minute default, then add public IPs or a public IP prefix to the NAT gateway. Find the top destination and pool connections to it."
      - alert: AzureNatGatewayConnectionsNearLimit
        expr: |
          azure_natgw_totals{metric="TotalConnectionCount",aggregation="total"} > 1600000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "NAT gateway {{ $labels.resourceName }} is above 80% of its 2 million connection limit"
          description: "Find the subnets that open the most connections and split them onto a second NAT gateway, or cut idle connections with shorter pool idle timeouts."

The two Failed rules page, because users are already hitting errors. The ports rule follows Microsoft’s guidance for a low-severity alert at 75% of allocated ports. Add a second copy at 90% with severity: critical if your outbound traffic grows in fast bursts.

What fixes the alerts point to. Reuse connections first. One pooled HTTP client per process stops most SNAT exhaustion on its own. On .NET that’s IHttpClientFactory, covered in the HttpClient socket exhaustion recipe. Then take Azure traffic off SNAT. Private endpoints and service endpoints don’t use SNAT ports for the services they cover, such as Azure SQL and Storage. For the rest, route outbound traffic through a NAT gateway and add public IPs as you grow.

Test the alerts

Use a staging load balancer with an outbound rule that gives each backend a small, fixed port allocation, such as 1,024. Put one VM in its backend pool. Outside that VNet, run a target you own that accepts many TCP connections on one port. From the VM, open connections and hold them, so they pile up against one destination:

const net = require('node:net');

const [host, port, count] = [process.argv[2], Number(process.argv[3]), Number(process.argv[4])];
const sockets = [];
let failed = 0;

for (let i = 0; i < count; i++) {
  const s = net.connect({ host, port });
  s.setTimeout(10000); // connect timeout only
  s.on('connect', () => s.setTimeout(0)); // then hold the socket open
  s.on('timeout', () => s.destroy(new Error('connect timeout')));
  s.on('error', () => { failed++; });
  sockets.push(s);
}

setTimeout(() => {
  console.log(`failed connects: ${failed} of ${count}`);
  sockets.forEach((s) => s.destroy());
}, 600000);
node hold-connections.js 203.0.113.10 443 1500

Ask for more connections than the allocation. Within a few minutes, UsedSnatPorts for that VM should reach AllocatedSnatPorts and SnatConnectionCount should show Failed flows. AzureLbSnatPortsHigh fires first, then AzureLbSnatConnectionsFailing. Stop the script and confirm both clear. Ports come back about four minutes after the connections close, so give it time.

For a NAT gateway, run the same script from a VM in a subnet behind it, with one public IP on the gateway. A NAT gateway has far more ports, so spread the load across several VMs and one destination. For App Service, deploy a staging build that creates a new HttpClient per request against one endpoint. Drive it with k6 at your peak rate. The NAT gateway alerts should fire if the app routes outbound traffic through it. Compare the timing with the SNAT Port Exhaustion detector afterwards. Only run these against endpoints you own.

Check the rule logic offline as well, with promtool test rules and made-up series. Then a later edit can’t break a label match unnoticed.

Tune the noise

Keep the two Failed rules at > 0. A failed SNAT connection is a request your app couldn’t send. Tune the for instead. Azure Monitor publishes these metrics with a delay of a few minutes, and a 5-minute timespan smooths single blips. Shorter values add noise and don’t page you any sooner.

Set the port threshold from your own peak. Pull two weeks of history and find each backend’s busiest hour:

max_over_time(
  (azure_lb_snat_ports{metric="UsedSnatPorts",aggregation="average"}
    / ignoring (metric)
   azure_lb_snat_ports{metric="AllocatedSnatPorts",aggregation="average"})[14d:5m]
)

If normal peaks already reach 60%, a 75% alert fires on every busy day. Fix the allocation or the app first, then set the alert above the new peak. A scale set that adds instances changes the allocation per instance under the default table, so check the ratio again after each scale-out.

One backend IP may fire over and over while its peers stay low. The cause then sits in that instance’s workload, often a background job with its own client. Silence the alert for the duration of a known batch run rather than raising the threshold for everyone.

More recipes

Further reading