Cloud NAT port exhaustion fails without an error message. Cloud NAT drops the outbound packet, and your app waits for a connection that never opens. The alerts below read Cloud NAT’s own metrics. You hear about exhaustion before calls to a payment provider or partner API start timing out.
What you miss today
Each VM behind Cloud NAT gets a block of source ports on a NAT IP address. With static allocation the default block is 64 ports, and with dynamic allocation it starts at 32. A VM can reuse its ports across different destinations. To any single destination IP, port and protocol, though, it can hold one connection per port.
Peak traffic eats those ports in two ways:
- Concurrent connections to one destination. Every pod on a GKE node shares the node’s NAT ports. Twenty pods calling the same API endpoint draw from one block.
- Connection churn. A closed connection keeps its NAT port through the TCP
TIME_WAITtimeout. Google gives that default as 30 or 120 seconds, depending on when you created the gateway. A client that opens a new connection per request burns a port per request for that long.
Once the block runs out, Cloud NAT drops the SYN. The VM’s kernel retries it until the client library gives up, and your logs show a connect timeout with no other clue:
dial tcp 203.0.113.10:443: i/o timeout
connect ETIMEDOUT 203.0.113.10:443
Connection to api.example.com timed out. (connect timeout=5)
Those are Go, Node.js and Python (urllib3) wordings. CPU looks normal, your load balancer shows no 5xx from the edge, and the failing calls are outbound, so default dashboards miss them. The gateway’s metrics tell you what happened. Cloud NAT counts each dropped packet with a reason label, and OUT_OF_RESOURCES means it had no port to give.
A second failure hits the whole gateway. With NAT IPs you assign by hand, the gateway can run out of ports to hand to new VMs. Each NAT IP offers 64,512 ports, so one IP covers 1,008 VMs at 64 ports each. An autoscaler that adds nodes past that limit leaves the new ones with no NAT ports at all.
AWS and Azure have their own version of this problem, covered in the NAT gateway and SNAT recipes.
The exporter and metrics
Cloud NAT metrics live in Cloud Monitoring. To alert on them in Prometheus, run stackdriver_exporter with a service account that has roles/monitoring.viewer:
stackdriver_exporter \
--google.project-ids=my-project \
--monitoring.metrics-prefixes=compute.googleapis.com/nat/ \
--monitoring.metrics-prefixes=router.googleapis.com/nat/ \
--monitoring.aggregate-deltas
Google Cloud Managed Service for Prometheus is the other route. It queries Cloud Monitoring metrics with PromQL under its own naming scheme, so the rule expressions below would need translating.
The exporter names each metric stackdriver_<monitored resource>_<metric type>, with dots and slashes turned into underscores. It adds the resource labels and the metric labels to each series. --monitoring.aggregate-deltas turns Google’s DELTA metrics into counters, so rate() and increase() work on them. Scrape it every 60 seconds, since Google samples these metrics once a minute.
The per-VM metrics carry the gce_instance resource, with project_id, instance_id and zone:
| Metric | What it tells you |
|---|---|
stackdriver_gce_instance_compute_googleapis_com_nat_allocated_ports | Ports the gateway gave this VM, one series per nat_ip |
stackdriver_gce_instance_compute_googleapis_com_nat_port_usage | Most connections from this VM to a single destination IP and port. This is the number that runs into the allocation. |
stackdriver_gce_instance_compute_googleapis_com_nat_dropped_sent_packets_count | Outbound packets dropped, with reason set to OUT_OF_RESOURCES or ENDPOINT_INDEPENDENCE_CONFLICT |
The gateway metrics carry the nat_gateway resource, with project_id, region, router_id and gateway_name:
| Metric | What it tells you |
|---|---|
stackdriver_nat_gateway_router_googleapis_com_nat_dropped_sent_packets_count | The same drops, summed for the gateway |
stackdriver_nat_gateway_router_googleapis_com_nat_nat_allocation_failed | 1 when the gateway failed to give NAT IPs to some VM |
stackdriver_nat_gateway_router_googleapis_com_nat_allocated_ports | Ports handed out from each nat_ip |
stackdriver_nat_gateway_router_googleapis_com_nat_open_connections | Connections open through the gateway now |
ENDPOINT_INDEPENDENCE_CONFLICT appears only with endpoint-independent mapping on. Two internal endpoints wanted the same NAT IP and port at once for one destination.
The metrics name a VM by instance_id. Map it to a node name with gcloud compute instances list --filter="id=<instance_id>". Then turn on NAT logging for drops only, so you can see which destination ran the VM out of ports:
gcloud compute routers nats update nat-gw --router=nat-router --region=europe-west1 \
--enable-logging --log-filter=ERRORS_ONLY
gcloud logging read \
'resource.type="nat_gateway" AND jsonPayload.allocation_status="DROPPED"' \
--freshness=1h --limit=1000 \
--format='value(jsonPayload.connection.src_ip,jsonPayload.connection.dest_ip,jsonPayload.connection.dest_port)' \
| sort | uniq -c | sort -rn | head
Alert rules for OUT_OF_RESOURCES dropped packets
groups:
- name: gcp-cloud-nat
rules:
- alert: CloudNatDroppingOutOfResources
expr: |
sum by (project_id, region, gateway_name) (
increase(stackdriver_nat_gateway_router_googleapis_com_nat_dropped_sent_packets_count{reason="OUT_OF_RESOURCES"}[10m])
) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Cloud NAT {{ $labels.gateway_name }} is dropping packets for lack of ports"
description: "Outbound connections are failing now. Find the VM and destination in the NAT logs (allocation_status DROPPED), then raise min ports per VM or turn on dynamic port allocation."
- alert: CloudNatVmPortsNearLimit
expr: |
max by (project_id, zone, instance_id) (stackdriver_gce_instance_compute_googleapis_com_nat_port_usage)
/
sum by (project_id, zone, instance_id) (stackdriver_gce_instance_compute_googleapis_com_nat_allocated_ports)
> 0.8
for: 15m
labels:
severity: warning
annotations:
summary: "VM {{ $labels.instance_id }} uses over 80% of its NAT ports to one destination"
description: "Check the client for connection reuse and keepalive. If the connections are needed, raise min ports per VM or the dynamic maximum before the next peak."
- alert: CloudNatEndpointIndependenceConflict
expr: |
sum by (project_id, region, gateway_name) (
increase(stackdriver_nat_gateway_router_googleapis_com_nat_dropped_sent_packets_count{reason="ENDPOINT_INDEPENDENCE_CONFLICT"}[10m])
) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "Cloud NAT {{ $labels.gateway_name }} drops packets from endpoint-independent mapping"
description: "Turn off endpoint-independent mapping unless a workload needs it, or raise min ports per VM so endpoints stop competing for the same NAT port."
- alert: CloudNatAllocationFailed
expr: max by (project_id, region, gateway_name) (stackdriver_nat_gateway_router_googleapis_com_nat_nat_allocation_failed) == 1
for: 10m
labels:
severity: critical
annotations:
summary: "Cloud NAT {{ $labels.gateway_name }} could not give NAT IPs to every VM"
description: "New VMs have no outbound internet access. Add NAT IPs, switch to automatic IP allocation, or lower min ports per VM."
- alert: CloudNatIpPoolNearFull
expr: |
sum by (project_id, region, gateway_name) (stackdriver_nat_gateway_router_googleapis_com_nat_allocated_ports)
/
(count by (project_id, region, gateway_name) (stackdriver_nat_gateway_router_googleapis_com_nat_allocated_ports) * 64512)
> 0.8
for: 30m
labels:
severity: warning
annotations:
summary: "Cloud NAT {{ $labels.gateway_name }} has handed out over 80% of its NAT IP ports"
description: "The next node scale-up may get no ports. Add NAT IPs or move to automatic allocation before the next peak."
The pool rule counts one series per NAT IP that has ports in use, and each IP holds 64,512. An IP with nothing allocated yet doesn’t show up, so the rule overstates usage on a gateway with idle spare IPs. That’s the safe side.
When an alert fires. Start in the app. A client that opens a new connection per request burns ports through TIME_WAIT, and a keepalive pool ends that. The Go connection reuse and .NET HttpClient recipes cover the two clients I see do this most. I fix reuse before I buy more NAT IPs, because more ports only delay a churn problem.
Then size the gateway. A VM needs at least its peak concurrent connections to its busiest destination. Add that destination’s new connections per second times the TIME_WAIT timeout. Google APIs don’t need NAT ports at all once you turn on Private Google Access for the subnet.
# Save the current settings first
gcloud compute routers nats describe nat-gw --router=nat-router --region=europe-west1 \
--format=yaml > nat-gw-before.yaml
# Dynamic allocation, so each VM starts at the minimum and grows to the maximum
gcloud compute routers nats update nat-gw --router=nat-router --region=europe-west1 \
--enable-dynamic-port-allocation --min-ports-per-vm=64 --max-ports-per-vm=4096
# Free ports sooner after a connection closes
gcloud compute routers nats update nat-gw --router=nat-router --region=europe-west1 \
--tcp-time-wait-timeout=30s
With dynamic allocation, both port values must be powers of two, and the maximum can go up to 65,536. You can’t combine it with endpoint-independent mapping. Google warns that switching between static and dynamic allocation breaks every active NAT connection, so make that switch in a quiet window. On GKE I turn dynamic allocation on for every NAT gateway, because pods with unrelated call patterns share one node’s ports. Check the pool math afterwards: a VM that grows to its maximum takes ports that new nodes would need.
Test the alerts
Use a staging subnet with its own Cloud NAT gateway, so the test can’t starve production. Set static allocation with a small block, and stand up a test HTTP server on a public IP outside the VPC:
gcloud compute routers nats update nat-staging --router=nat-router-staging --region=europe-west1 \
--no-enable-dynamic-port-allocation --min-ports-per-vm=64
From one VM behind that gateway, run k6 with connection reuse off. Each request then opens a new connection to the same destination, and each closed one holds a port through TIME_WAIT:
import http from 'k6/http';
export const options = {
noConnectionReuse: true,
scenarios: {
churn: {
executor: 'constant-arrival-rate',
rate: 20, timeUnit: '1s', duration: '15m',
preAllocatedVUs: 50, maxVUs: 500,
},
},
};
export default function () {
http.get('http://203.0.113.10:8080/ping', { timeout: '5s' });
}
Pick a rate above the block size divided by the gateway’s TIME_WAIT timeout. k6 starts reporting request timeouts once the ports run out. Within a few minutes, CloudNatVmPortsNearLimit should go pending, then CloudNatDroppingOutOfResources. Stop the test and confirm both clear. NAT logs should show DROPPED entries for the test server’s IP.
To test CloudNatAllocationFailed, give the staging gateway one manual NAT IP. Then raise min ports per VM so far that the IP can’t cover every VM in the subnet. The gateway can’t place them all and sets the flag. Put the setting back afterwards.
Cloud Monitoring publishes these metrics up to a few minutes after each sample, and the exporter adds a scrape interval on top. Expect alerts to fire several minutes after the first drop. If that delay is too long for you, alert on the client side too, on connect timeouts per destination.
Tune the noise
Set thresholds from your own peak. Run this for two weeks, covering your busiest days, and look at the top VMs:
topk(10,
max_over_time(
(max by (project_id, instance_id) (stackdriver_gce_instance_compute_googleapis_com_nat_port_usage)
/ sum by (project_id, instance_id) (stackdriver_gce_instance_compute_googleapis_com_nat_allocated_ports))[14d:5m]
)
)
If your busiest VMs sit at half their block at peak, the 80% warning gives you room to act. If some already touch 80% on a normal day, fix them first. Raising the threshold to silence them means the first drop arrives with no warning.
With dynamic allocation the ratio behaves differently. The gateway grows a VM’s allocation as usage climbs, so the ratio can sit near 1 with no trouble. On those gateways, compare port_usage with your --max-ports-per-vm value as a fixed number.
Keep OUT_OF_RESOURCES at > 0 and critical. Each dropped packet is a failed or delayed call. The 5-minute for absorbs one late data point from Cloud Monitoring. Raise it if retries in your clients hide brief drops and you’ve decided those are acceptable.
Route the instance_id to a team. On GKE, add the node pool as a label in your scrape config, or join on a metric that carries it. A page that says “VM 4721… is out of ports” without a workload name costs you the first ten minutes of the incident.
Roll back
The alert rules change nothing in your project, so removing them is safe at any time. The gateway settings are different. You saved them to nat-gw-before.yaml before the change. Put back the values you changed:
gcloud compute routers nats update nat-gw --router=nat-router --region=europe-west1 \
--no-enable-dynamic-port-allocation --min-ports-per-vm=64 --clear-tcp-time-wait-timeout
Switching back to static allocation breaks every active NAT connection, the same as the switch forward. Your clients reconnect, and any call in flight fails. If the new settings cause drops or allocation failures, roll back now. Otherwise do it in a quiet hour, when fewer outbound calls are in flight.
More recipes
- Fix Cloud Run "No Available Instance" Errors
Set Cloud Run concurrency from one instance's real limit and size min and max instances for your peak, so cold starts and scaling stop dropping requests.
- Fix GCP Load Balancer 502s and backend_timeout
Size the backend service timeout to your slowest real request and hold backend keepalive at 620 seconds, so the load balancer stops answering 502.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.