Diagnosing and Fixing Outbound SNAT Port Exhaustion on Azure VMs, VMSS, AKS, and App Service

Recognizable Symptom: Intermittent Outbound Timeouts Under Load

When an application on an Azure VM, VM scale set, AKS node, or App Service suddenly drops outbound connections to an external endpoint, the first clue is that inbound traffic and CPU/memory look healthy. The failure pattern usually clusters during high request rates, especially when many short-lived TCP connections are opened per second to the same destination IP and port.

Root Cause: Limited SNAT Port Pool and TIME_WAIT Accumulation

Every outbound TCP connection passes through source network address translation (SNAT). The default outbound path preallocates a finite number of SNAT ports per instance, and each connection consumes one port per destination tuple. Closed sockets linger in the TIME_WAIT state before the port is reusable. When the application opens many short connections—such as per-request HTTP calls without keep-alive—the pool can be exhausted, and new connections fail with timeout errors. Exact per-instance port allocations depend on the load balancer rule and backend pool size, so check current documentation rather than relying on memorized figures.

Cause and Diagnostic Table

Finding	Likely cause	Next check
Failed SNAT connections spike under load	Port pool exhaustion from connection churn	Count TIME_WAIT sockets on the instance
High utilization with modest traffic	Too few instances sharing the frontend IP	Review backend pool size and outbound rules
Failures persist after NAT Gateway	Asymmetric routing or mixed outbound methods	Verify which outbound mechanism the subnet uses

Ordered Checks

1. Verify the failure is outbound-only. Confirm inbound requests succeed while outbound calls fail, and that failures correlate with load.
2. Inspect SNAT metrics. On the Azure Load Balancer resource, open Metrics and chart SNAT Connection Count split by connection state (including Failed) and SNAT port utilization. Spikes in failed SNAT connections during the incident window confirm exhaustion. For AKS, also check node-level conntrack usage.
3. Check connection patterns on the instance. SSH to a VM (or kubectl debug/kubectl exec on a node, with appropriate permissions) and run ss -tan state time-wait | wc -l. A rapidly growing count during load confirms churn-driven exhaustion.
4. Review the subnet's outbound mechanism. Determine whether the subnet uses default outbound access, a NAT Gateway, or load balancer outbound rules. Default outbound access for new subnets is being retired in favor of explicit outbound methods, so behavior differs by deployment date—verify what your resources actually use.
5. Assess frontend IP count. Ports are preallocated per instance per frontend IP; many instances sharing one frontend reduces the per-instance allocation.

Fixes Tied to Findings

1. Attach an Azure NAT Gateway

A NAT Gateway provides a much larger, on-demand SNAT port pool per subnet and handles idle and TIME_WAIT sockets more efficiently. Run these Azure CLI commands with a network contributor role on the resource group:

az network nat gateway create \
  --resource-group MyRG \
  --name MyNatGateway \
  --public-ip-addresses MyPublicIP1 \
  --idle-timeout 120

az network vnet subnet update \
  --resource-group MyRG \
  --vnet-name MyVNet \
  --name MySubnet \
  --nat-gateway MyNatGateway

Replace placeholders with your values. This changes the subnet's outbound path, so plan for the effective public IP to change—downstream firewalls that allowlist your old IP will need updates. Verify afterward that the subnet's effective outbound IP is the NAT Gateway's address and that failed SNAT connections drop under the same load.

2. Reduce Connection Churn

• Enable HTTP keep-alive and reuse clients instead of creating one per request (for example, a singleton HttpClient in .NET or an agent with keepAlive: true in Node.js).
• Configure database and service client connection pools with sensible maximum sizes and lifetimes.

3. Scale Out the Backend

Adding instances increases the total preallocated SNAT ports. Use az vmss scale or scale the AKS node pool when NAT Gateway is not an option.

4. Avoid Mixing Outbound Methods

NAT Gateway and load balancer outbound rules serving the same flows can produce confusing asymmetric behavior. Pick one explicit outbound method per subnet and verify the effective route before and after the change.

Escalation Criteria

1. SNAT metrics show failures, but the observed connection rate is well below what the pool should sustain.
2. Exhaustion persists after NAT Gateway adoption and connection-reuse fixes.
3. A firewall or third-party appliance in the path may be dropping translated flows.

Open a Microsoft support case with the SNAT metrics, the socket counts, and the remediation steps already taken.

Practical Verification

1. Reproduce under controlled load (for example, repeated short-lived requests to one destination) and watch failed SNAT connections before and after each fix.
2. Count TIME_WAIT sockets on an affected instance during the test to confirm churn is the driver.
3. After attaching a NAT Gateway, confirm the effective outbound public IP changed and failed SNAT connections drop to zero under the same load.

Key Takeaways

• SNAT exhaustion shows up as outbound-only timeouts under load with healthy CPU and memory.
• Load balancer SNAT metrics, split by connection state, are the primary evidence.
• NAT Gateway is the most robust fix; connection reuse and scale-out reduce pressure.
• Escalate only when metrics confirm exhaustion persists after mitigation.