Go Too Many Open Files and HTTP Connection Reuse

Share one Go HTTP client per service and set server timeouts, so peak traffic stops leaking file descriptors and your service keeps accepting connections.

Published

Applies to Go 1.21 and later; from Go 1.27, Close drains small unread bodies so the connection can be reused; Gin, Echo v4 and Echo v5

accept4: too many open files means your Go server can’t accept a new connection because the process has used every file descriptor it may open. I check outbound HTTP connections first, before the ulimit, because a client that never reuses a connection fills any limit you set. The fix sits in how you build http.Client and http.Server.

What breaks at peak

Every TCP connection holds a file descriptor, in both directions. The server’s own connections, the client connections to your dependencies and open files all count against one limit, RLIMIT_NOFILE. Since Go 1.19, a Go program on Linux and other Unix systems raises its soft limit to the hard limit at startup. Past that, net/http logs this line and backs off, from 5 ms doubling up to 1 second:

http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms

While it backs off, new connections wait in the kernel’s listen queue and then time out. Your load balancer marks the pod unhealthy or returns 502s. Outbound calls fail at the same time, with the error coming from socket():

dial tcp 10.0.3.7:443: socket: too many open files

On macOS and BSD the server line reads accept: too many open files, since those systems have no accept4.

Four habits in Go code use up descriptors at peak:

  • A new client per request. &http.Client{} with a new http.Transport each time opens a new connection each time. The old transport keeps its idle connections open until IdleConnTimeout, 90 seconds on http.DefaultTransport. Under load you hold one idle socket per request from the last minute and a half. Closed ones pile up in TIME_WAIT, and the host can run out of local ports with connect: cannot assign requested address.
  • A response body you never close. The connection stays checked out of the pool, so the transport dials a new one for the next request. Each open connection also keeps two goroutines alive, a read loop and a write loop.
  • The default idle limit per host. MaxIdleConnsPerHost defaults to 2. With 100 requests in flight to one service, the transport keeps 2 connections after the burst and closes the other 98. The next burst dials them all again.
  • A server with no timeouts. A zero-value http.Server waits forever on idle keep-alive clients and slow headers. Gin’s r.Run() and http.ListenAndServe both start one with no timeouts set.

The .NET HttpClient socket exhaustion recipe covers the same failure in another runtime.

Go HTTP client connection reuse

Build one client per downstream service at startup and share it across every goroutine. http.Client and http.Transport are safe for concurrent use:

package main

import (
	"context"
	"fmt"
	"io"
	"net"
	"net/http"
	"time"
)

// One client per downstream service, created once and shared by every handler.
var inventory = newClient(50, 100)

func newClient(idlePerHost, maxPerHost int) *http.Client {
	t := http.DefaultTransport.(*http.Transport).Clone()
	t.DialContext = (&net.Dialer{Timeout: 3 * time.Second, KeepAlive: 30 * time.Second}).DialContext
	t.MaxIdleConns = 200 // at least the sum of MaxIdleConnsPerHost across hosts
	t.MaxIdleConnsPerHost = idlePerHost
	t.MaxConnsPerHost = maxPerHost
	t.IdleConnTimeout = 60 * time.Second
	t.ResponseHeaderTimeout = 5 * time.Second
	return &http.Client{Transport: t, Timeout: 10 * time.Second}
}

func stock(ctx context.Context, sku string) ([]byte, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://inventory.internal/stock/"+sku, nil)
	if err != nil {
		return nil, err
	}
	resp, err := inventory.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		io.Copy(io.Discard, io.LimitReader(resp.Body, 64<<10))
		return nil, fmt.Errorf("inventory: %s", resp.Status)
	}
	return io.ReadAll(resp.Body)
}

Clone() keeps the defaults you want, such as proxy support from the environment and HTTP/2 over TLS. Each field you change does one job:

  • MaxIdleConnsPerHost sets how many idle connections the pool keeps for one host. Too low, and every burst dials again. Size it to the requests you have in flight to that host at peak, which is the request rate times the average latency. At 500 requests per second and 80 ms, that’s 40.
  • MaxIdleConns caps idle connections across all hosts. The default of 100 starts closing connections once several services each want 50.
  • MaxConnsPerHost caps all connections to one host, idle or busy. Requests over the cap wait for a free connection rather than dialing. That protects the downstream service and your descriptor budget. A wait longer than the request’s context returns an error.
  • IdleConnTimeout closes connections left idle that long. Keep it shorter than the downstream server’s keep-alive timeout, or the server closes first and your next request on that connection fails.
  • ResponseHeaderTimeout and Client.Timeout stop one slow dependency from holding connections and goroutines forever. Client.Timeout covers the whole exchange, body included.

Close every body, and read it. defer resp.Body.Close() returns the connection only once the body is read to the end. Before Go 1.27, closing an unread body closed the connection. Go 1.27 drains up to 256 KiB for at most 50 ms on Close and reuses the connection if that reaches the end. The io.Copy on the error path keeps reuse working on older versions too. On an error from Do, there’s no body to close.

Use the request’s context. Pass r.Context() from the incoming request. If the caller hangs up, Go cancels the outbound call and frees its connection.

With HTTPS, ForceAttemptHTTP2 stays on from the default transport. One HTTP/2 connection carries many requests at once, so idle limits matter less. Plain http:// calls between services inside a cluster use HTTP/1.1, where each in-flight request needs its own connection.

Server timeouts. Set them on every http.Server:

srv := &http.Server{
	Addr:              ":8080",
	Handler:           mux,
	ReadHeaderTimeout: 5 * time.Second,
	ReadTimeout:       30 * time.Second,
	WriteTimeout:      30 * time.Second,
	IdleTimeout:       75 * time.Second, // longer than the load balancer's idle timeout
}
log.Fatal(srv.ListenAndServe())

ReadHeaderTimeout drops clients that open a connection and take too long to send headers. IdleTimeout closes keep-alive connections nobody uses. Keep it longer than the idle timeout on the load balancer in front, so the balancer closes first. If Go closes first, the balancer can send a request into a closing socket. It then returns a 502, as the ALB 502 and 504 recipe explains. A zero IdleTimeout falls back to ReadTimeout. WriteTimeout cuts off streaming and server-sent events. For those handlers, extend the deadline per request with http.NewResponseController(w).SetWriteDeadline.

Gin. r.Run() builds an http.Server with only Addr and Handler. Build your own and pass the engine as the handler:

r := gin.New()
r.Use(gin.Recovery())
srv := &http.Server{
	Addr:              ":8080",
	Handler:           r,
	ReadHeaderTimeout: 5 * time.Second,
	IdleTimeout:       75 * time.Second,
}
log.Fatal(srv.ListenAndServe())

In handlers, pass c.Request.Context() to outbound calls.

Echo. Echo v5’s StartConfig sets a 30-second ReadTimeout, so idle keep-alive connections close after 30 seconds unless you set IdleTimeout. That’s shorter than many load balancer defaults. Set the rest in BeforeServeFunc:

sc := echo.StartConfig{
	Address: ":8080",
	BeforeServeFunc: func(s *http.Server) error {
		s.ReadHeaderTimeout = 5 * time.Second
		s.IdleTimeout = 75 * time.Second
		return nil
	},
}
if err := sc.Start(ctx, e); err != nil {
	log.Fatal(err)
}

In Echo v4, set the same fields on e.Server before e.Start. Pass c.Request().Context() to outbound calls.

Size the descriptor limit. Add peak inbound connections, the sum of MaxConnsPerHost across clients, open files and a margin. The total must fit under the hard limit. Read it inside the container with cat /proc/1/limits | grep 'open files'. On a VM, raise it with LimitNOFILE in the systemd unit, as the systemd LimitNOFILE recipe shows. In a container, the runtime sets it. Raising the limit buys room, but a leak still fills it. I fix reuse before I touch the limit.

Test it on your system

First prove reuse in a unit test. httptrace reports whether each request got a pooled connection:

package reusecheck

import (
	"io"
	"net/http"
	"net/http/httptest"
	"net/http/httptrace"
	"testing"
)

func TestConnectionReuse(t *testing.T) {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("ok"))
	}))
	defer srv.Close()

	client := srv.Client() // swap in your own client constructor
	reused := 0
	trace := &httptrace.ClientTrace{
		GotConn: func(info httptrace.GotConnInfo) {
			if info.Reused {
				reused++
			}
		},
	}
	for i := 0; i < 20; i++ {
		req, _ := http.NewRequest(http.MethodGet, srv.URL, nil)
		req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
		resp, err := client.Do(req)
		if err != nil {
			t.Fatal(err)
		}
		io.Copy(io.Discard, resp.Body)
		resp.Body.Close()
	}
	if reused < 19 {
		t.Fatalf("only %d of 20 requests reused a connection", reused)
	}
}

Replace the loop body with your own call path, such as stock(), to catch a missing Close in real code.

Then load test staging with k6 at a fixed arrival rate, on an endpoint that calls a dependency:

// fd-peak.js
import http from 'k6/http';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.RATE || 200),
      timeUnit: '1s',
      duration: '15m',
      preAllocatedVUs: 200,
      maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<400'],
  },
};

export default function () {
  http.get(`${__ENV.BASE_URL}/stock/sku-${Math.floor(Math.random() * 5000)}`);
}
k6 run -e BASE_URL=https://staging.example.com -e RATE=800 fd-peak.js

Set RATE to your peak and the p99 to your own budget. While it runs, watch the pod:

kubectl exec deploy/api -- sh -c 'ls /proc/1/fd | wc -l'
kubectl exec deploy/api -- sh -c "grep 'open files' /proc/1/limits"
# on the node or in a debug container with ss
ss -s
ss -tan state time-wait | wc -l

Write your targets down before the run. The descriptor count should level off well below the limit within the first minute. TIME_WAIT should stay flat rather than climb with each burst. No too many open files line should show up in the log. A count that keeps rising at a steady request rate points to a body left open or a client built per request. If p99 rises while connections sit at MaxConnsPerHost, the dependency is the bottleneck, and a higher cap only moves the queue there.

Keep it healthy

The Prometheus Go client’s process collector exports process_open_fds and process_max_fds on Linux. These rules assume Kubernetes service discovery adds namespace and pod labels:

groups:
  - name: go-file-descriptors
    rules:
      - alert: GoFileDescriptorsHigh
        expr: max by (namespace, pod) (process_open_fds / process_max_fds) > 0.8
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Go pod {{ $labels.pod }} is using over 80% of its file descriptors"
          description: "Run ls /proc/1/fd | wc -l in the pod and compare with ss -s. Look for an http.Client built per request or a response body that is never closed."
      - alert: GoFileDescriptorsExhausted
        expr: max by (namespace, pod) (process_open_fds / process_max_fds) > 0.95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Go pod {{ $labels.pod }} is about to refuse connections"
          description: "Accept errors with too many open files start at 100%. Restart the pod to buy time, then fix reuse before the next peak."
      - alert: GoFileDescriptorLeak
        expr: |
          max by (namespace, pod) (predict_linear(process_open_fds[1h], 4 * 3600))
            > on (namespace, pod)
          max by (namespace, pod) (process_max_fds)
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Go pod {{ $labels.pod }} will run out of file descriptors within 4 hours"
          description: "The count grows at a steady rate, which points to a leak. Take a goroutine profile and look for persistConn readLoop and writeLoop goroutines piling up."

The leak rule fires on steady growth long before the limit. A leaked connection leaves goroutines behind as well, and the pprof goroutine leak recipe shows how to find them.

Roll back

The client and server settings live in code, so rollback is a redeploy of the previous image. Keep its tag before you ship:

kubectl get deploy api -o jsonpath='{.spec.template.spec.containers[0].image}' > api-image-before.txt
kubectl set image deploy/api api="$(cat api-image-before.txt)"

If the change brought new errors, such as context deadline exceeded from a tight MaxConnsPerHost or timeouts cutting off a streaming endpoint, roll back now. Then raise the cap or exempt that handler. For any other revert, pick your quietest hour, because the old code brings the descriptor growth back with it.

More recipes

Further reading