Fix HttpClient Socket Exhaustion in .NET

Reuse connections through IHttpClientFactory or one shared SocketsHttpHandler, so outbound calls stop running out of local ports at peak traffic.

Published

Applies to .NET 8 and later as written; IHttpClientFactory and SocketsHttpHandler date back to .NET Core 2.1

HttpClient socket exhaustion starts with code that creates a new HttpClient for each request and disposes it. Each disposal closes a TCP connection, and each closed connection holds a local port for a while. At peak, the pod runs out of ports and outbound calls fail.

What breaks at peak

This is the pattern behind most cases. It looks tidy, which is why it survives code review:

using var client = new HttpClient();
var body = await client.GetStringAsync("https://payments.internal/status");

HttpClient owns a handler, and the handler owns a connection pool. Dispose the client and you dispose the pool, so the next request opens a new connection with a new TCP and TLS handshake. The side that closes a TCP connection keeps it in TIME_WAIT, on Linux for 60 seconds. During that time the local port stays taken for that destination.

Linux hands out local ports from net.ipv4.ip_local_port_range, 32768 to 60999 by default. Open new connections to one host faster than TIME_WAIT frees them and the range runs dry. Then every new connection fails.

On Linux images built on glibc, the exception reads:

System.Net.Http.HttpRequestException: Cannot assign requested address (payments.internal:443)
 ---> System.Net.Sockets.SocketException (99): Cannot assign requested address

On Alpine, musl words errno 99 as Address not available. On Windows, the same exhaustion shows up as one of these:

System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
System.Net.Sockets.SocketException (10055): An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

Before the errors start, latency climbs. Every request pays for a new handshake, and TLS handshakes cost CPU on both ends.

The opposite mistake. A single static HttpClient with default settings fixes the port problem. But it keeps its connections open forever, and HttpClient resolves DNS only when it opens a connection. After a blue-green switch or a database failover behind a DNS name, it keeps calling the old IPs.

NAT makes it worse. Pods that call out through a NAT gateway share its ports with every other pod. Connection churn there hits a limit long before the pod’s own range. The NAT Gateway recipe and the Azure SNAT recipe cover those limits.

The IHttpClientFactory and SocketsHttpHandler fix

Microsoft’s guidance gives two correct shapes. Use short-lived clients from IHttpClientFactory, or one long-lived client with PooledConnectionLifetime set. Either way, the connection pool lives across requests.

In ASP.NET Core, use a typed client. This Program.cs registers one with a configured SocketsHttpHandler:

using System.Net;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHttpClient<PaymentsClient>(client =>
    {
        client.BaseAddress = new Uri("https://payments.internal/");
        client.Timeout = TimeSpan.FromSeconds(10);
    })
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        PooledConnectionLifetime = TimeSpan.FromMinutes(2),
        PooledConnectionIdleTimeout = TimeSpan.FromSeconds(60),
        MaxConnectionsPerServer = 100,
        ConnectTimeout = TimeSpan.FromSeconds(3),
        AutomaticDecompression = DecompressionMethods.All,
    })
    .SetHandlerLifetime(Timeout.InfiniteTimeSpan);

var app = builder.Build();

app.MapGet("/checkout/{id}", async (string id, PaymentsClient payments, CancellationToken ct) =>
    Results.Ok(await payments.GetStatusAsync(id, ct)));

app.Run();

public sealed class PaymentsClient(HttpClient http)
{
    public Task<string> GetStatusAsync(string id, CancellationToken ct) =>
        http.GetStringAsync($"payments/{Uri.EscapeDataString(id)}/status", ct);
}

Each setting has one job:

  • AddHttpClient<PaymentsClient> makes the factory build PaymentsClient with an HttpClient that shares a pooled handler. Disposing a factory-made client doesn’t dispose that handler, so using is safe again.
  • PooledConnectionLifetime retires each connection after this long, once its current request ends. The next connection resolves DNS again, so traffic follows a DNS change. The default is infinite. Set it to how fast you need DNS changes picked up.
  • PooledConnectionIdleTimeout closes connections that sat unused this long. Keep it below the server’s or load balancer’s keep-alive timeout, or the server may close a connection the pool still lends out.
  • MaxConnectionsPerServer caps open connections to one origin. The default is int.MaxValue, so a burst can open thousands of sockets at once. Too low a cap makes requests queue for a connection, which the http.client.request.time_in_queue metric shows.
  • ConnectTimeout fails a TCP and TLS connect that hangs, instead of letting it use up the whole request timeout.
  • HttpClient.Timeout bounds the whole request. The default is 100 seconds, far beyond most latency budgets.
  • SetHandlerLifetime(Timeout.InfiniteTimeSpan) stops the factory from rotating handlers every two minutes, its default. The handler’s PooledConnectionLifetime does that job now, without dropping a whole pool at once.

Don’t inject a typed client into a singleton. The singleton keeps one HttpClient forever, and the factory can’t rotate it. With PooledConnectionLifetime set, as above, that’s safe. Without it, DNS changes never reach that client.

Outside dependency injection, share one client. A console app, a worker or an older code base can use a static instance:

using System.Net;

public static class Http
{
    public static readonly HttpClient Shared = new(new SocketsHttpHandler
    {
        PooledConnectionLifetime = TimeSpan.FromMinutes(2),
        MaxConnectionsPerServer = 100,
        AutomaticDecompression = DecompressionMethods.All,
    })
    {
        Timeout = TimeSpan.FromSeconds(10),
    };
}

Size the connection cap. For HTTP/1.1, each in-flight request needs its own connection. By Little’s law:

connections per origin per pod = peak outbound requests per second to that origin * p99 response time in seconds

Add headroom for bursts and set MaxConnectionsPerServer there. Check that pods times the cap stays under what the downstream service or your NAT gateway accepts. HTTP/2 multiplexes many requests on one connection, so the cap matters less there.

I search the code base for new HttpClient( before I touch any sysctl. Raising the port range or reusing TIME_WAIT ports only delays the failure:

grep -rn --include='*.cs' 'new HttpClient(' .

Any hit outside startup code or a static field is a candidate. Also check for using around clients that come from somewhere other than the factory.

Serverless and managed platforms. On AWS Lambda, create the client in a static field outside the handler. The execution environment reuses it across invocations. On Azure Functions, register it through IHttpClientFactory in Program.cs, the same way as above. App Service and Container Apps run your code unchanged, but their outbound ports are shared and limited.

Test it on your system

Run a staging pod that calls a real downstream at peak rates. Hold a fixed arrival rate, so slow outbound calls don’t slow the generator:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    outbound: {
      executor: 'constant-arrival-rate',
      rate: 400,
      timeUnit: '1s',
      duration: '15m',
      preAllocatedVUs: 300,
      maxVUs: 2000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.005'],
    http_req_duration: ['p(99)<700'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/checkout/${__ITER % 1000}`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}

Set rate to the peak your pod gets for an endpoint that makes outbound calls. Run it on the old build first to see the problem, then on the fix.

While it runs, count sockets inside the pod’s network namespace. /proc/net/sockstat shows TIME_WAIT as tw:

watch -n 5 'kubectl exec deploy/api -- cat /proc/net/sockstat'

On Windows, run netstat -ano | find /c "TIME_WAIT" on the host. On .NET 8 and later, watch the connection pool with dotnet-counters inside the container:

dotnet-counters monitor -n Api --counters System.Net.Http

http.client.open_connections should settle at a steady count. Let your own targets set the pass line. A good run shows:

  • No SocketException or HttpRequestException in the logs.
  • The tw count stays flat after the first minute, instead of climbing with load.
  • Open connections settle at a steady count instead of climbing with each request.
  • p99 and error rate stay inside the k6 thresholds you set.

Then test DNS. Point the downstream’s DNS name at a new set of IPs in staging during the run. Traffic should move within PooledConnectionLifetime, without errors.

Keep it healthy

Export .NET’s built-in System.Net.Http metrics with OpenTelemetry. Add the OpenTelemetry.Extensions.Hosting and OpenTelemetry.Exporter.Prometheus.AspNetCore packages, then register the meter:

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics => metrics
        .AddMeter("System.Net.Http", "System.Runtime")
        .AddPrometheusExporter());

// after builder.Build()
app.MapPrometheusScrapingEndpoint();

The exporter turns dots into underscores and adds unit suffixes, so http.client.connection.duration becomes http_client_connection_duration_seconds. Check your /metrics output for the exact names before you load the rules. Each closed connection records one sample in that histogram. Its count against the request count tells you how many requests opened a fresh connection.

groups:
  - name: dotnet-httpclient
    rules:
      - alert: HttpClientConnectionChurn
        expr: |
          sum by (job, server_address) (rate(http_client_connection_duration_seconds_count[5m]))
            / sum by (job, server_address) (rate(http_client_request_duration_seconds_count[5m])) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} opens a new connection for most calls to {{ $labels.server_address }}"
          description: "Connections aren't being reused, so local ports fill with TIME_WAIT. Search for new HttpClient( per request, and check PooledConnectionIdleTimeout against the server's keep-alive."
      - alert: HttpClientConnectErrors
        expr: |
          sum by (job, server_address) (rate(http_client_request_duration_seconds_count{error_type="connection_error"}[5m])) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} can't connect to {{ $labels.server_address }}"
          description: "Outbound connects are failing. Check /proc/net/sockstat in the pod for TIME_WAIT and the logs for SocketException 99, then check the downstream's health."
      - alert: HttpClientQueueing
        expr: |
          histogram_quantile(0.99,
            sum by (le, job, server_address) (rate(http_client_request_time_in_queue_seconds_bucket[5m]))
          ) > 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Calls to {{ $labels.server_address }} wait over 100 ms for a connection"
          description: "MaxConnectionsPerServer is too low for the load, or the downstream holds connections too long. Raise the cap within the downstream's limits, or fix its latency."

Also alert on these log lines:

Cannot assign requested address
Address not available
Only one usage of each socket address

Roll back

The fix is a code change, so roll back with your deploy tool or kubectl rollout undo deployment/api. Save the running spec first with kubectl get deployment api -o yaml > api-before.yaml.

If the new MaxConnectionsPerServer cap makes calls queue and p99 climb, roll back now or raise the cap. If you’re reverting for any other reason, wait for a quiet hour. Old pods with the per-request client will run out of ports again at the next peak.

For other runtimes with the same problem, see the Node.js keep-alive recipe and the Linux network tuning recipe.

More recipes

Further reading