NAT Gateway ErrorPortAllocation and Port Exhaustion

Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.

Published

Applies to AWS public and private NAT gateways, zonal and regional; YACE (yet-another-cloudwatch-exporter) 0.62 and later

ErrorPortAllocation is the CloudWatch metric that counts how often a NAT gateway couldn’t find a free source port for a new connection. AWS describes a value above zero this way: “too many concurrent connections are open through the NAT gateway.” Each failure is an outbound call from your app that never got out.

What you miss today

NAT gateway port exhaustion hides well. The NAT gateway shows as Available in the console. Your instances and pods look healthy. Most dashboards graph bytes and packets, which stay flat or even drop while the problem runs.

Your app sees something else. Calls to a payment provider, an identity provider or a partner API start failing at peak. Depending on the client library, they show up as connect timeouts, ETIMEDOUT or connection resets. Retries make it worse, since each retry asks for another port.

The limit is per destination. Each IPv4 address on a NAT gateway supports up to 55,000 simultaneous connections to each unique destination. AWS defines a destination as the combination of destination IP, destination port and protocol. Spreading calls across the internet rarely runs out. Sending most of your traffic to one API endpoint behind a few IPs can run out fast. Every pod shares the same pool of source ports for that one destination.

Three habits fill that pool at peak:

  • A new connection per request. HTTP clients without connection reuse open and close a socket for each call. Each closed socket holds its port for a short while before the NAT gateway can hand it out again.
  • Idle connections that never close. A NAT gateway drops a connection that sits idle for 350 seconds. If your side tries to use it again, the gateway sends back an RST. Pools that keep connections longer than that hold ports they can’t use. They also produce errors when they reuse them.
  • Retry storms. A failing dependency plus aggressive retries multiplies new connections at the moment the pool is tight. The timeouts and retries recipe covers backoff and jitter.

Azure and Google Cloud hit the same wall with different names. See Azure SNAT port exhaustion and Cloud NAT port exhaustion.

The exporter and metrics

CloudWatch publishes NAT gateway metrics every minute in the AWS/NATGateway namespace. Get them into Prometheus with YACE, the yet-another-cloudwatch-exporter maintained by prometheus-community. Run it as a Deployment or a container with an IAM role that allows cloudwatch:GetMetricData, cloudwatch:ListMetrics and tag:GetResources.

This is the YACE config for NAT gateways:

apiVersion: v1alpha1
discovery:
  jobs:
    - type: AWS/NATGateway
      regions: [eu-west-1]
      period: 60
      length: 300
      nilToZero: true
      metrics:
        - name: ErrorPortAllocation
          statistics: [Sum]
        - name: ActiveConnectionCount
          statistics: [Maximum]
        - name: ConnectionAttemptCount
          statistics: [Sum]
        - name: ConnectionEstablishedCount
          statistics: [Sum]
        - name: IdleTimeoutCount
          statistics: [Sum]
        - name: PacketsDropCount
          statistics: [Sum]
        - name: PacketsInFromSource
          statistics: [Sum]
        - name: PacketsInFromDestination
          statistics: [Sum]

nilToZero reports a zero when CloudWatch returns no data point, so an idle NAT gateway doesn’t leave a gap that breaks a ratio. YACE names each metric aws_natgateway_<metric>_<statistic> in snake case and puts the gateway ID in dimension_NatGatewayId. A regional NAT gateway also reports per zone, in dimension_AvailabilityZone.

Each metric answers one question:

Metric in PrometheusWhat it tells you
aws_natgateway_error_port_allocation_sumConnections that failed because no source port was free. Any value above zero is a user-facing failure.
aws_natgateway_active_connection_count_maximumConcurrent TCP connections through the gateway. Your trend line for capacity planning.
aws_natgateway_connection_attempt_count_sumNew connection attempts, counting the first SYN only.
aws_natgateway_connection_established_count_sumConnections that got a reply. A gap below attempts means destinations didn’t answer.
aws_natgateway_idle_timeout_count_sumConnections that went idle for 350 seconds without a clean close. A rising count points at clients reusing stale connections.
aws_natgateway_packets_drop_count_sumPackets the gateway dropped. AWS treats a drop rate above 0.01% of traffic as worth investigating.

ActiveConnectionCount can’t tell you how close you are to the limit, because the limit applies per destination and the metric sums all of them. VPC Flow Logs can. Group flows from the NAT gateway’s network interface by destination address and port, and you’ll see which destination takes most of the connections.

The alert rules

groups:
  - name: aws-nat-gateway
    rules:
      - alert: NatGatewayPortAllocationErrors
        expr: max by (dimension_NatGatewayId) (aws_natgateway_error_port_allocation_sum) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "NAT gateway {{ $labels.dimension_NatGatewayId }} can't allocate source ports"
          description: "Outbound connections are failing now. Find the top destination in VPC Flow Logs, then add secondary IPs with aws ec2 associate-nat-gateway-address or turn on connection reuse in the client that calls it."
      - alert: NatGatewayConnectionsNotEstablished
        expr: |
          (
            sum by (dimension_NatGatewayId) (aws_natgateway_connection_attempt_count_sum)
            - sum by (dimension_NatGatewayId) (aws_natgateway_connection_established_count_sum)
          ) / sum by (dimension_NatGatewayId) (aws_natgateway_connection_attempt_count_sum) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Over 5% of new connections through {{ $labels.dimension_NatGatewayId }} get no reply"
          description: "A destination is refusing or ignoring connections. Check Flow Logs for the destination with REJECT or no return traffic, and check that client's timeouts and retries."
      - alert: NatGatewayIdleTimeoutsRising
        expr: |
          sum by (dimension_NatGatewayId) (aws_natgateway_idle_timeout_count_sum)
            > 2 * sum by (dimension_NatGatewayId) (aws_natgateway_idle_timeout_count_sum offset 1d)
          and sum by (dimension_NatGatewayId) (aws_natgateway_idle_timeout_count_sum) > 100
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Idle timeouts on {{ $labels.dimension_NatGatewayId }} doubled against the same time yesterday"
          description: "Clients hold idle connections past 350 seconds. Set connection pool idle timeouts below 350 seconds, or turn on TCP keepalive under that value on the client hosts."
      - alert: NatGatewayPacketDrops
        expr: |
          sum by (dimension_NatGatewayId) (aws_natgateway_packets_drop_count_sum)
            / (
              sum by (dimension_NatGatewayId) (aws_natgateway_packets_in_from_source_sum)
              + sum by (dimension_NatGatewayId) (aws_natgateway_packets_in_from_destination_sum)
            ) > 0.0001
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "NAT gateway {{ $labels.dimension_NatGatewayId }} is dropping over 0.01% of packets"
          description: "Check the AWS Health Dashboard for VPC issues in the region, then check PeakPacketsPerSecond against the gateway's packet limit. Split heavy subnets onto a second NAT gateway if it's near the limit."

The first rule is the one that pages. The other three give you time to act. A NAT gateway that runs near its limit for weeks often shows rising idle timeouts and a widening attempt gap first.

Fixes the alerts point to. Pick them in this order, cheapest first:

  • Reuse connections in the client that talks to the top destination. One pooled HTTP client per process does more than any infrastructure change. On .NET, that’s the HttpClient socket exhaustion recipe.
  • Send AWS traffic around the NAT gateway. A gateway endpoint for S3 and DynamoDB costs nothing. Interface endpoints for ECR, STS, SQS and other services take that traffic off the NAT gateway too.
  • Add IPs. A NAT gateway takes up to 8 IPv4 addresses, 1 primary and 7 secondary, and each one adds 55,000 connections per destination. The default Elastic IP quota allows 2 per public NAT gateway, so request more through Service Quotas first.
  • Split subnets across more NAT gateways, one per Availability Zone at least.
aws ec2 describe-nat-gateways --nat-gateway-ids nat-0abc123 \
  --query "NatGateways[0].NatGatewayAddresses" > nat-addresses-before.json
aws ec2 associate-nat-gateway-address --nat-gateway-id nat-0abc123 \
  --allocation-ids eipalloc-0def456

Save the current address list first, as above, so you can see what changed. To undo, aws ec2 disassociate-nat-gateway-address takes the association ID from the describe output. It drops connections that use that address, so do it off-peak unless the new address is causing errors.

Test the alerts

Start with the rule logic. promtool test rules feeds made-up series through each rule and checks what fires, with no AWS account involved:

rule_files:
  - nat-gateway-rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'aws_natgateway_error_port_allocation_sum{dimension_NatGatewayId="nat-0abc123"}'
        values: '0 0 12 40 40 40 0 0 0 0'
    alert_rule_test:
      - eval_time: 5m
        alertname: NatGatewayPortAllocationErrors
        exp_alerts:
          - exp_labels:
              severity: critical
              dimension_NatGatewayId: nat-0abc123
            exp_annotations:
              summary: "NAT gateway nat-0abc123 can't allocate source ports"
              description: "Outbound connections are failing now. Find the top destination in VPC Flow Logs, then add secondary IPs with aws ec2 associate-nat-gateway-address or turn on connection reuse in the client that calls it."
      - eval_time: 9m
        alertname: NatGatewayPortAllocationErrors
        exp_alerts: []

Save the rules as nat-gateway-rules.yml and the test as nat-gateway-test.yml, then run promtool test rules nat-gateway-test.yml. The second check proves the alert clears once errors stop.

Then prove the whole path in staging. Give a staging NAT gateway a single IP. Stand up a target you own outside the VPC, such as a small instance in another VPC with a public IP. Raise its open file limit so it can hold the connections. From two or three hosts behind the NAT gateway, open and hold connections to one port on that target:

import asyncio
import sys

HOST, PORT, COUNT = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])

async def hold(sem):
    async with sem:
        try:
            _, writer = await asyncio.wait_for(asyncio.open_connection(HOST, PORT), 10)
        except (OSError, asyncio.TimeoutError):
            return 1
    await asyncio.sleep(600)
    writer.close()
    return 0

async def main():
    sem = asyncio.Semaphore(500)
    failed = sum(await asyncio.gather(*(hold(sem) for _ in range(COUNT))))
    print(f"failed connects: {failed} of {COUNT}")

asyncio.run(main())
ulimit -n 65535
python3 hold_connections.py 203.0.113.10 443 25000

Each client host can only hold as many connections to one destination as its own local port range allows, so spread the total across hosts. Push the combined count past 55,000 per NAT gateway IP. Within a few minutes ErrorPortAllocation should go above zero in CloudWatch, then in Prometheus, and NatGatewayPortAllocationErrors should fire. Stop the scripts and confirm it clears after the for window. Run this only against a target you own, and only in staging.

Tune the noise

Keep NatGatewayPortAllocationErrors at > 0. Each count is a failed outbound connection, so there’s no safe baseline to tune it against. Tune its for instead. Two minutes skips a single stray data point. CloudWatch data reaches YACE a few minutes late, so a shorter for doesn’t page you any sooner.

For the warnings, pull two weeks of history and set thresholds from it:

quantile_over_time(0.99,
  (sum by (dimension_NatGatewayId) (aws_natgateway_idle_timeout_count_sum))[14d:5m])

Set the idle timeout floor, > 100 here, a little above that line, so a quiet gateway doubling from 3 to 6 doesn’t page anyone. The 5% attempt gap suits gateways that call APIs for most of their traffic. A gateway that runs crawlers or health checks against flaky hosts needs a higher value, or a filter on the subnets that matter.

My own default is one NAT gateway per zone with an alert on each, plus a Flow Logs query saved for the top destinations. The alert tells me where to look, and the query tells me which client to fix.

More recipes

Further reading