Linux Network Tuning for Web Servers

Size the listen queue, file limits, port range and conntrack table to your peak, so a busy web server or proxy keeps accepting connections.

Published

Applies to Linux kernel 4.19 and later; pod sysctls need Kubernetes 1.27 or later

A busy web server or reverse proxy hits kernel limits before it runs out of CPU. The listen queue fills, the proxy runs out of local ports, or the conntrack table drops packets. Each limit has its own error string and its own fix. Pair this with nginx tuning for high traffic, which fixes the proxy side.

Read the caveats below before you paste it.

# /etc/sysctl.d/90-web-server.conf
net.core.somaxconn = 4096                  # default on kernel 5.4+; 128 before
net.ipv4.tcp_max_syn_backlog = 8192
# net.core.netdev_max_backlog = 16384  # only if softnet drops grow
net.ipv4.ip_local_port_range = 1024 65535
# your listening ports; add 30000-32767 on Kubernetes nodes
net.ipv4.ip_local_reserved_ports = 5432,6379,8080,9100
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_slow_start_after_idle = 0

The conntrack limit goes in its own pair of files, explained under Conntrack below.

What breaks at peak

The listen queue overflows. The kernel parks finished handshakes in an accept queue until your server calls accept(). Once the queue is full, the kernel drops the handshake and the client retries after a second or more. You see slow connects with nothing in the app logs. nstat shows TcpExtListenOverflows and TcpExtListenDrops climbing. A full SYN queue logs Possible SYN flooding on port ... Sending cookies. in the kernel log.

The NIC backlog drops packets. Packets wait in a per-CPU queue before the TCP stack processes them. A full queue drops them without a log line, and TCP retransmits.

File descriptors run out. Each socket is a file descriptor. At the limit, accept() fails with Too many open files, and nginx logs accept4() failed (24: Too many open files).

A proxy runs out of local ports. Each outbound connection takes a local port from net.ipv4.ip_local_port_range. The side that closes first holds the socket in TIME_WAIT for 60 seconds, a value the kernel hard-codes. If a proxy closes upstream connections faster than ports free up, connect() fails with EADDRNOTAVAIL and the proxy answers with 502s. On glibc systems such as Debian and Ubuntu, nginx logs this at [crit]:

connect() to 10.0.3.7:8080 failed (99: Cannot assign requested address) while connecting to upstream

Alpine and other musl builds print (99: Address not available) for the same errno. Grep for '\(99: ' to catch both.

The conntrack table fills. Hosts running Docker, Kubernetes or stateful iptables rules track every flow in a table. Once it’s full, the kernel logs nf_conntrack: table full, dropping packet. New connections time out at random while existing ones keep working.

The Linux sysctl config for web servers

Save your current values first (see Roll back), then apply the file with sysctl --system. I run nstat and ss -lnt before I touch any sysctl, so I know which limit the server hits today.

Listen backlog. net.core.somaxconn caps the accept queue of each listening socket, and net.ipv4.tcp_max_syn_backlog caps half-open connections still in the handshake. Your server asks for its own backlog, and the kernel uses the smaller of that and somaxconn. nginx asks for 511 unless you set listen ... backlog=, so raise both together.

Size the queue to the connections that can arrive while your workers are busy. Peak new connections per second × the longest accept stall gives you a floor. Check it with ss -lnt: for a listening socket, Send-Q shows the backlog in use and Recv-Q shows the queue right now. If overflows continue after the raise, the fix is more workers or more instances.

NIC backlog. net.core.netdev_max_backlog sets the per-CPU input queue, 1000 by default. Raise it only if the second column of /proc/net/softnet_stat grows; each row is one CPU, in hex. The setting is global, so you can’t set it from inside a container.

File descriptors. Three limits stack up: fs.file-max for the whole system, fs.nr_open as the ceiling for any one process, and the process’s own nofile limit. On recent kernels fs.file-max is huge, so the per-process limit bites first. systemd 240 and later raises fs.nr_open at boot, so run sysctl fs.nr_open before you decide it needs a change.

  • Under systemd, set LimitNOFILE=65535 in a drop-in with systemctl edit nginx, then restart the service.
  • In Docker, pass --ulimit nofile=65535:65535. Kubernetes has no pod field for it, so the container runtime’s default applies. Check with ulimit -n inside the container.
  • Check a running process with grep 'open files' /proc/<pid>/limits. For nginx, check a worker, since worker_rlimit_nofile sets the worker’s limit.

Ephemeral ports. Widen the range and reuse TIME_WAIT sockets for new outbound connections. Use 1024 65535. If both ends share parity, such as 1024 65000, the kernel logs ip_local_port_range: prefer different parity for start/end values.

The limit applies per destination address and port. With 64,512 ports and a 60-second TIME_WAIT, one proxy can close about 1,000 connections a second to one backend before it runs dry.

tcp_tw_reuse = 1 lets the kernel reuse a TIME_WAIT socket for a new outbound connection when TCP timestamps prove it safe. It needs tcp_timestamps = 1, the default, and it does nothing for inbound connections. Kernels since about 2018 default tcp_tw_reuse to 2, which reuses sockets on loopback only. Traffic to another host, pod or container gets no relief from that default.

Reserve your listening ports with ip_local_reserved_ports before you widen the range. Otherwise an outbound connection can grab a port that a service needs at startup. On Kubernetes nodes, reserve the NodePort range 30000-32767 as well.

Two old tips don’t help. tcp_fin_timeout doesn’t shorten TIME_WAIT; it controls orphaned sockets in FIN_WAIT_2. tcp_tw_recycle broke clients behind NAT, and Linux removed it in 4.12. Leave tcp_max_tw_buckets alone too. Past it, the kernel logs time wait bucket table overflow and skips TIME_WAIT, which risks stray packets landing on new connections.

A wider port range and tcp_tw_reuse treat the symptom. I’d rather fix the cause, too many closed connections. Use an upstream keepalive pool sized to your load or HTTP clients that reuse connections.

Conntrack. net.netfilter.nf_conntrack_max caps tracked flows. The key exists only after the nf_conntrack module loads. At boot, systemd-sysctl can run before anything loads the module, and sysctl --system then fails with cannot stat /proc/sys/net/netfilter/nf_conntrack_max. Load the module first with a modules-load file:

# /etc/modules-load.d/conntrack.conf
nf_conntrack
# /etc/sysctl.d/91-conntrack.conf
net.netfilter.nf_conntrack_max = 524288

Skip both files on a plain web server without firewall state; it has no table to fill.

  • Size it to peak concurrent connections plus recent closed ones, since conntrack keeps closed TCP flows for two minutes. Then add headroom. Each entry costs a few hundred bytes of kernel memory, so 524,288 entries fit in under 200 MB.
  • Keep the hash table at a quarter of the max or more. Set it with net.netfilter.nf_conntrack_buckets on recent kernels, or /sys/module/nf_conntrack/parameters/hashsize.
  • On Kubernetes, kube-proxy sets nf_conntrack_max at startup from conntrack.maxPerCore and overwrites your sysctl. Set it in the kube-proxy config, or set maxPerCore: 0 to keep your own value.
  • Check it with cat /proc/sys/net/netfilter/nf_conntrack_count against the max, and conntrack -S for drop counters.

TCP buffers. Leave tcp_rmem and tcp_wmem alone unless you move bulk data over long paths, such as cross-region replication. The kernel autotunes buffers up to the max value. A connection needs bandwidth × round-trip time of buffer, so 1 Gbit/s at 100 ms needs 12.5 MB. Raise only the max, because a bigger default costs memory on each of your thousands of short connections.

Slow start after idle. tcp_slow_start_after_idle = 0 keeps a connection’s congestion window after an idle gap. Long-lived keepalive and HTTP/2 connections then send a burst at full speed instead of starting slow. On a congested path that burst can cause loss.

BBR. BBR is a congestion control that paces sends from its own model of the path.

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

It needs kernel 4.9 or later and the tcp_bbr module, so check sysctl net.ipv4.tcp_available_congestion_control first. It changes how your server sends, which helps responses to distant or lossy clients most. Inside one data center you may see no change, and I leave it off on servers that only talk to internal clients.

BBR v1 can take bandwidth from CUBIC flows and retransmit more on shallow buffers, so measure retransmits and client latency before and after. Since kernel 4.13, BBR paces itself in TCP, which makes the fq qdisc optional. If you want fq anyway, default_qdisc applies only to new qdiscs. Find your interface with ip route get 1, then run tc qdisc del dev <iface> root. The kernel rebuilds the root qdisc, mq with fq children on a multi-queue NIC.

Kubernetes pods. Each pod gets its own network namespace. A new namespace starts somaxconn and ip_local_port_range at kernel defaults, whatever the node has. Set namespaced sysctls on the pod:

spec:
  securityContext:
    sysctls:
      - name: net.ipv4.ip_local_port_range
        value: "1024 65535"
      - name: net.ipv4.ip_local_reserved_ports
        value: "8080,9113"

Safe sysctls work on any node. The list includes ip_local_port_range, tcp_syncookies, and since Kubernetes 1.27 ip_local_reserved_ports. Version 1.29 added tcp_fin_timeout and the keepalive timers, 1.32 added tcp_rmem and tcp_wmem, and 1.37 added tcp_slow_start_after_idle.

net.core.somaxconn, tcp_tw_reuse and tcp_max_syn_backlog are namespaced but unsafe. The node’s kubelet needs allowedUnsafeSysctls, or the pod fails to start. Global settings such as netdev_max_backlog, nf_conntrack_max, fs.nr_open and default_qdisc belong on the node, so bake them into the node image or user data.

  • On EKS, set node sysctls and kubelet flags through the launch template’s user data.
  • GKE Autopilot rejects pods that use unsafe sysctls.
  • Pods with hostNetwork: true share the node’s values and can’t set net.* sysctls.

Out of scope here. The tcp_keepalive_* timers matter for idle connections through NAT and load balancers, not peak load. Keep tcp_syncookies at its default of 1, so a full SYN queue degrades instead of dropping. Leave tcp_abort_on_overflow at 0; at 1 the kernel resets clients during a short burst it could have absorbed. RSS and RPS spread packet processing across CPUs and belong in a NIC tuning guide.

Test it on your system

Load a staging server at your expected peak, then push past it. The nginx recipe has wrk and k6 commands at a fixed arrival rate. While the load runs, watch the kernel on the server and on any proxy in front of it:

nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtSyncookiesSent
ss -s
ss -Htan state time-wait | wc -l
cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max
sudo dmesg -T | grep -E 'SYN flooding|table full|time wait bucket|parity'

dmesg needs root on distributions that set kernel.dmesg_restrict = 1, which most do.

Run the same test on the old values first, so you know which limit you hit and at what load. Set your targets before either run. The new values hold when the listen counters stay flat and TIME_WAIT levels off well below your port range. Conntrack should stay under 80% of its max. The kernel log, the app log and the proxy log should show no errno 24 or 99 lines.

If you’re newer to these tools, my Linux for developers post covers the basics.

Keep it healthy

node_exporter’s default collectors export the kernel metrics below. Its netstat collector includes the TcpExt_Listen* counters, and its conntrack collector exports the table size and drop counter.

groups:
  - name: linux-network
    rules:
      - alert: ListenQueueOverflow
        expr: rate(node_netstat_TcpExt_ListenOverflows[5m]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} is dropping handshakes from a full accept queue"
          description: "Raise somaxconn and the server's listen backlog together. If overflows continue, add workers or instances."
      - alert: TimeWaitNearPortRange
        # ip_local_port_range 1024-65535 holds 64,512 ports; alert at 70%
        expr: node_sockstat_TCP_tw > 45000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} holds over 45,000 TIME_WAIT sockets"
          description: "Outbound connections may soon fail with errno 99. Check the proxy's upstream keepalive pool and widen ip_local_port_range."
      - alert: ConntrackTableNearFull
        expr: node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Conntrack table on {{ $labels.instance }} is over 80% full"
          description: "Raise nf_conntrack_max and the hash table size, or set it in the kube-proxy config on Kubernetes nodes."
      - alert: ConntrackDropping
        expr: rate(node_nf_conntrack_stat_drop[5m]) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is dropping packets because conntrack is full"
          description: "New connections are timing out. Raise nf_conntrack_max now, then find what filled the table."
      - alert: SoftnetDrops
        expr: rate(node_softnet_dropped_total[5m]) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU {{ $labels.cpu }} on {{ $labels.instance }} is dropping packets from a full backlog"
          description: "Raise net.core.netdev_max_backlog and check whether one CPU takes all the network interrupts."
      - alert: ProcessFileDescriptorsNearLimit
        expr: process_open_fds / process_max_fds > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} on {{ $labels.instance }} uses over 80% of its open-file limit"
          description: "Raise the service's LimitNOFILE or container ulimit before accept() starts failing with errno 24."

node_sockstat_TCP_tw counts TIME_WAIT across all destinations, but the port limit applies per destination. It also reads only the namespace node_exporter runs in, which is the node’s own network on Kubernetes. For a pod’s namespace, read /proc/net/sockstat inside the pod or run an exporter as a sidecar.

The file descriptor rule uses process_open_fds and process_max_fds, which most Prometheus client libraries export for their own process. nginx has no such metric, so log alerts on (24: Too many open files) and '\(99: ' cover it.

Check the values after every node image change. A new AMI or node pool brings back the kernel defaults without anyone noticing.

Roll back

Save the current values before you apply anything:

sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.core.netdev_max_backlog \
  net.ipv4.ip_local_port_range net.ipv4.ip_local_reserved_ports net.ipv4.tcp_tw_reuse \
  net.ipv4.tcp_slow_start_after_idle net.core.default_qdisc \
  net.ipv4.tcp_congestion_control > /root/sysctl-before.txt
sysctl net.netfilter.nf_conntrack_max >> /root/sysctl-before.txt 2>/dev/null

To roll back, delete your files and load the saved values:

rm -f /etc/sysctl.d/90-web-server.conf /etc/sysctl.d/91-conntrack.conf /etc/modules-load.d/conntrack.conf
sysctl -p /root/sysctl-before.txt

New connections pick up the old values at once, while existing connections keep their ports and buffers. A smaller nf_conntrack_max doesn’t evict entries, but new flows fail until the count drops below it. For a pod, remove the sysctls entries and redeploy.

Roll back straight away if a change is causing errors, such as a service that can’t bind its port or retransmits that jumped after BBR. Otherwise wait for an off-peak window, since the old limits break at the same load they broke at before.

More recipes

Further reading