Possible SYN flooding on port in the kernel log is seldom an attack. On a busy server it means the half-open queue for one listening socket filled up. The kernel then switched to SYN cookies to keep accepting. The fix is a bigger listen backlog. The alerts below warn you before the log line does, and tell a real flood apart from your own peak.
For the sysctls themselves, read Linux network tuning for web servers. This page is about seeing the problem.
What you miss today
Kernels from 6.2 print the listening address and port:
TCP: request_sock_TCP: Possible SYN flooding on port 0.0.0.0:443. Sending cookies.
TCP: request_sock_TCPv6: Possible SYN flooding on port [::]:443. Sending cookies.
Older kernels print the port alone and add a hint:
TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.
The TCPv6 form appears when your server listens on [::], which also takes IPv4 clients on a dual-stack socket. With net.ipv4.tcp_syncookies = 0 the line ends Dropping request instead. Grep for Possible SYN flooding on port to match every version.
The kernel prints that line once per listening socket. After the first one, it stays quiet until the socket closes, most often when the service restarts. A host that has sent cookies all week can show a single log line from Monday. Log-based alerting misses most of it, and the counters below don’t.
The cause at normal peak. I read the kernel’s TCP source to pin this down. A listening socket has a backlog, the smaller of the number the app passes to listen() and net.core.somaxconn. The same number caps two queues. One holds half-open connections that have sent a SYN and await the final ACK. The other holds finished connections waiting for accept(). Once the half-open queue holds more than the backlog, the kernel sends a SYN cookie in place of a queue entry and logs the warning.
The half-open queue holds about your new connections per second times the client round trip. 2,000 new connections a second from clients 100 ms away keeps about 200 in the queue. nginx asks for a backlog of 511 by default, and kernels before 5.4 default somaxconn to 128. A launch, a retry storm or a TLS reconnect wave after a deploy crosses that line without any attacker.
Raising tcp_max_syn_backlog doesn’t help. Most guides say to raise net.ipv4.tcp_max_syn_backlog. With SYN cookies on, the default, the kernel never reads it on this path. It only matters with cookies off, where it drops SYNs from unknown clients once the queue reaches three quarters of that limit. Raise the app’s backlog and somaxconn instead.
The cost to your users. The kernel docs say syncookies “do not allow to use TCP extensions”. They add that the warning without a real flood means your server “is seriously misconfigured”. A client that gets a cookie can lose window scaling or SACK when TCP timestamps are off, so large responses go slower. With cookies off, the kernel drops the SYN and the client resends it after a second or more. That shows up as connect times of one or three seconds in your p99, with idle CPU and nothing in the app log.
Default dashboards show CPU, memory and established connections. None of them show cookies.
The exporter and metrics
node_exporter’s netstat collector reads /proc/net/netstat. Its default field list already includes the counters that matter most:
| Metric | What it tells you |
|---|---|
node_netstat_TcpExt_SyncookiesSent | SYN-ACKs sent as cookies because a half-open queue was full. Any steady rate at peak means a backlog is too small. |
node_netstat_TcpExt_SyncookiesRecv | Valid cookies that came back in an ACK. Real clients complete the handshake, so this tracks SyncookiesSent at your own peak. |
node_netstat_TcpExt_SyncookiesFailed | ACKs carrying a cookie that didn’t validate |
node_netstat_TcpExt_ListenOverflows | Connections dropped because the accept queue was full. The app isn’t calling accept() fast enough. |
node_netstat_TcpExt_ListenDrops | All drops at a listening socket, overflows included |
Two more counters need a flag. TCPReqQFullDoCookies counts each time a full queue triggered a cookie. TCPReqQFullDrop counts SYNs dropped because the queue was full with cookies off. Add them by passing the default field regex with TCPReqQFull.* appended to the TcpExt_ group. On a current node_exporter that looks like this:
node_exporter --collector.netstat.fields='^(.*_(InErrors|InErrs)|Ip_Forwarding|Ip(6|Ext)_(InOctets|OutOctets)|Icmp6?_(InMsgs|OutMsgs)|TcpExt_(Listen.*|Syncookies.*|TCPSynRetrans|TCPTimeouts|TCPOFOQueue|TCPRcvQDrop|TCPReqQFull.*)|Tcp_(ActiveOpens|InSegs|OutSegs|OutRsts|PassiveOpens|RetransSegs|CurrEstab)|Udp6?_(InDatagrams|OutDatagrams|NoPorts|RcvbufErrors|SndbufErrors))$'
Copy the default from node_exporter --help on your version, since it changes between releases, and add only the new term.
Containers have their own counters. /proc/net/netstat belongs to a network namespace. node_exporter on a Kubernetes node runs with host networking and sees the node’s sockets only. An app pod listens in its own namespace, so its cookies never reach the node’s counters. The kernel log line still lands in the node’s dmesg. Run a node_exporter sidecar in the pod with --collector.disable-defaults --collector.netstat to read the pod’s counters. Pods with hostNetwork: true share the node’s.
On the host, ss shows the live queues per socket:
ss -ltn # Send-Q is the backlog, Recv-Q the accept queue now
ss -tn state syn-recv '( sport = :443 )' | wc -l # half-open connections on port 443
nstat -az TcpExtSyncookiesSent TcpExtSyncookiesRecv TcpExtListenOverflows
Distro differences. The counters match on every distro. The log format and the tooling don’t:
| Distro | Kernel and log line format | Read the kernel log | node_exporter package |
|---|---|---|---|
| Ubuntu and Debian | Ubuntu 24.04 (6.8) and Debian 13 (6.12) print the address. Ubuntu 22.04 (5.15) and Debian 12 (6.1) print the older form. | journalctl -k | prometheus-node-exporter, 1.7.0 on Ubuntu 24.04 and 1.9.0 on Debian 13 |
| RHEL, Rocky, AlmaLinux | RHEL 10 (6.12) prints the address. RHEL 9 runs 5.14 with backports, so match both forms. | journalctl -k | Upstream release binary |
| Amazon Linux 2023 | The default 6.1 kernel prints the older form. The 6.12 and 6.18 kernels print the address. | journalctl -k, since unprivileged dmesg is off | Upstream release binary; the AL2023 repository doesn’t carry it |
| Alpine | Current linux-lts kernels print the address | dmesg | prometheus-node-exporter from the community repository |
The alert rules
groups:
- name: linux-syn-backlog
rules:
- alert: SynCookiesAtPeak
expr: rate(node_netstat_TcpExt_SyncookiesSent[5m]) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} is sending SYN cookies"
description: "A listen backlog is too small for the connection rate. Run ss -ltn to find the socket whose Send-Q is lowest, then raise the app's listen backlog and net.core.somaxconn together."
- alert: SynFloodSuspected
expr: |
rate(node_netstat_TcpExt_SyncookiesSent[5m]) > 100
and
rate(node_netstat_TcpExt_SyncookiesRecv[5m]) < 0.1 * rate(node_netstat_TcpExt_SyncookiesSent[5m])
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} sends cookies that almost never come back"
description: "Clients aren't completing handshakes, which points to spoofed SYNs. Check ss -tn state syn-recv for source addresses, and turn on your provider's DDoS or rate-limit rules upstream."
- alert: SynRequestsDropped
expr: rate(node_netstat_TcpExt_TCPReqQFullDrop[5m]) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} is dropping SYNs from a full queue"
description: "SYN cookies are off, so clients wait a second or more and retry. Set net.ipv4.tcp_syncookies = 1 and raise the listen backlog."
- alert: AcceptQueueDrops
expr: rate(node_netstat_TcpExt_ListenDrops[5m]) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} drops connections at a listening socket"
description: "The app accepts connections slower than they arrive. Compare Recv-Q and Send-Q in ss -ltn, add workers or instances, then raise the backlog."
SynFloodSuspected separates an attack from your own peak. Real clients send back a valid cookie, so SyncookiesRecv follows SyncookiesSent. Spoofed SYNs never get an answer. SynRequestsDropped needs the extra netstat field from above.
Test the alerts
Run both tests on a staging host, never production. The first checks the cookie counters and alert wiring without a full queue. Mode 2 makes the kernel send a cookie for every SYN:
sysctl -w net.ipv4.tcp_syncookies=2
# drive normal traffic at the host for 15 minutes, then restore mode 1
sysctl -w net.ipv4.tcp_syncookies=1
SyncookiesSent and SyncookiesRecv should rise together, SynCookiesAtPeak should fire after its for window, and SynFloodSuspected should stay quiet. Mode 2 skips the log line, so expect none.
The second test fills a real queue. Start a listener with a small backlog on the staging host:
# slow_listener.py listens with a small backlog and answers HTTP slowly
import socket, sys, time
port, backlog = int(sys.argv[1]), int(sys.argv[2])
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("0.0.0.0", port))
srv.listen(backlog)
while True:
conn, _ = srv.accept()
conn.recv(4096)
conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok")
conn.close()
time.sleep(0.005)
python3 slow_listener.py 8080 16
On a separate client host, add round-trip delay with tc qdisc add dev eth0 root netem delay 100ms. Then open new connections at a fixed rate with k6, using noConnectionReuse: true and the constant-arrival-rate executor at a few hundred requests a second. The delay keeps each handshake in the half-open queue long enough to overflow 16.
Watch the server while it runs:
journalctl -k -f | grep 'Possible SYN flooding'
watch -n 2 nstat -az TcpExtSyncookiesSent TcpExtTCPReqQFullDoCookies TcpExtListenOverflows TcpExtListenDrops
The log line should appear once, the counters should climb, and SynCookiesAtPeak and AcceptQueueDrops should fire. Stop k6 and remove the delay with tc qdisc del dev eth0 root. The rates fall to zero, and the alerts should resolve within their for windows. Raise the listener’s backlog to 1024 and repeat. The cookies should stop.
Tune the noise
SynCookiesAtPeak starts at any nonzero rate, and I keep it there. Cookies at normal peak point at a backlog you can raise, and the fix is cheap. A deploy that reconnects all clients at once can trip it once a day. Raise for to cover that burst instead of setting a rate threshold.
SynFloodSuspected needs your own numbers. Look at a month of SyncookiesSent and pick a rate above your worst legitimate burst. The 10 percent return ratio works for most public services. Lower it if many of your clients sit on lossy mobile networks and drop their ACKs.
Route the two warnings to a ticket queue and the two critical rules to whoever is on call. If you fix the backlog and SynCookiesAtPeak keeps firing, the app can’t accept connections fast enough. That’s a capacity problem, and the k6 breakpoint recipe finds where it starts. For nginx, the listen backlog is the backlog= parameter on listen, covered in the nginx high-traffic recipe.
More recipes
- Alpine DNS Issues, musl Timeouts and Kubernetes ndots
Lower Kubernetes ndots, cache DNS on each node and use full names, so the musl resolver in Alpine images stops failing lookups at peak.
- Out of Memory Killed Process on Linux, Fix the OOM Killer
Cap each service's memory, add a little swap and set OOM scores, so the Linux OOM killer stops the process you can lose instead of your app.
- Tuned Profiles on RHEL, and When They Override Sysctls
Check which tuned profile runs, keep sysctls in /etc/sysctl.d and carry the rest in a custom profile, so a reboot or tuned restart can't undo your tuning.
- Systemd LimitNOFILE, Fix Open File Limits for Services
limits.conf isn't applied to a systemd service, so set LimitNOFILE in a drop-in, restart the unit and read the new limit back from /proc.