HAProxy maxconn exists at three levels, and each one fails in its own way at peak. The global limit caps the whole process. A frontend limit caps one listener. A server maxconn caps the requests HAProxy sends to one backend at once and queues the rest. Get the server limit and timeout queue wrong and users get 503s while your app servers sit half idle.
Peak traffic and the HAProxy 503 queue timeout
Server maxconn and the queue. In HTTP mode, a server’s maxconn limits concurrent requests to that server. Requests past the limit wait in a queue inside HAProxy. When a request waits longer than timeout queue, HAProxy drops it and returns a 503 with this body:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
The log line carries the termination state sQ: the request spent too long in the queue. If you never set timeout queue, HAProxy uses timeout connect instead. A short connect timeout, which is right for connects, then also becomes your queue limit. At peak that turns a short wait into a 503.
With no server maxconn at all, the default is unlimited. HAProxy then sends every request straight through, and the queue moves into your app. Gunicorn, Puma or PHP-FPM workers fill up, response times climb, and HAProxy logs sH when timeout server strikes first. The client gets a 504:
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
No servers left. When health checks mark every server down, HAProxy logs backend app has no server available! at emergency level. Every request then gets the same 503.
Frontend and global limits. A frontend’s maxconn defaults to 0, which means it takes the global value. Once a frontend or the process reaches its limit, HAProxy stops accepting. New connections wait in the kernel’s listen queue and then time out, with nothing in HAProxy’s log. If you leave the global maxconn unset, HAProxy computes it from the file descriptor limit and available memory. Check the number it picked before you assume it’s large enough:
echo "show info" | socat stdio /run/haproxy/admin.sock | grep -E '^(Maxconn|CurrConns|Nbthread):'
The HAProxy maxconn config for high traffic
This is a full haproxy.cfg for one HTTP service behind three app servers. Change the addresses, the health check path and the limits to fit yours.
global
log /dev/log local0
maxconn 60000
stats socket /run/haproxy/admin.sock mode 660 level admin
defaults
mode http
log global
option httplog
timeout connect 3s
timeout client 30s
timeout server 30s
timeout http-request 10s
timeout http-keep-alive 5s
timeout queue 5s
retries 3
option redispatch
frontend www
bind :80
maxconn 50000
default_backend app
backend app
balance leastconn
option httpchk
http-check send meth GET uri /healthz
default-server check inter 2s fall 3 rise 2 maxconn 64
server app1 10.0.3.7:8080
server app2 10.0.3.8:8080
server app3 10.0.3.9:8080
frontend metrics
bind 10.0.1.5:8405
http-request use-service prometheus-exporter if { path /metrics }
no log
Check it before you load it:
haproxy -c -f /etc/haproxy/haproxy.cfg
Global maxconn. It caps concurrent connections for the whole process, and HAProxy raises its own file descriptor limit to match. The configuration manual puts an established connection at about 33 kB of memory with default buffers. So size it from memory first: RAM you give HAProxy ÷ 33 kB is the ceiling. Then set it above your peak concurrent client connections, with room for a burst.
Frontend maxconn. Set it a little below the global value, so one frontend can’t take every slot from the others. With one frontend, the gap leaves room for the stats and metrics listeners.
Server maxconn. This is the setting that decides where requests wait. Set it to the number of requests the app server can work on at once. That’s workers × threads for Gunicorn or Puma, pm.max_children for PHP-FPM, or the connection pool size for an app bound by its database. I set it to the app’s real concurrency, so the queue builds in HAProxy where I can see it and measure it. A limit far above that figure lets requests queue inside the app, and you lose that view.
Timeout queue. It bounds how long a request waits for a free server slot. Size it from the latency budget: queue time plus a normal response must fit inside what the client will wait. If the budget for an endpoint is 6 seconds and responses take up to 1 second, a 5-second queue is the most you can spend. A longer queue doesn’t save requests at peak; it turns fast 503s into slow ones.
Balance leastconn. leastconn sends each request to the server with the fewest active connections. With uneven request times, it keeps one server from collecting the slow requests while others sit free. roundrobin suits short, even requests.
Health checks. check inter 2s fall 3 rise 2 marks a server down after three failed checks and back up after two good ones. Point /healthz at something cheap that proves the app can serve, not at a page that hits the database for every check.
Retries and redispatch. By default retries applies to failed connection attempts only, and 3 is the default. option redispatch lets the last retry go to a different server. Leave retry-on unset unless every request on the backend is safe to resend.
Timeouts. timeout http-request limits how long a client may take to send its headers, and it stops slow clients holding slots. timeout server should sit above your slowest legitimate response and below the timeout of whatever sits in front of HAProxy. A cloud load balancer in front needs an idle timeout above timeout client; the ALB 502 and 504 recipe covers that side.
Connection reuse. http-reuse safe is the default. HAProxy keeps idle server connections and reuses them for later requests, so you don’t need to set it.
Threads. HAProxy starts one thread per CPU it’s bound to when you leave nbthread unset. In a container with a CPU limit, compare Nbthread from show info with the CPUs the pod may use. Set nbthread to the limit if it reads higher.
Metrics. use-service prometheus-exporter serves Prometheus metrics from HAProxy itself. It needs a build with the Prometheus exporter. haproxy -vv prints “Built with the Prometheus exporter as a service” when it’s there, and the official Docker image includes it.
Test it on your system
Point a fixed arrival rate at a staging HAProxy with the same app servers and limits as production. Start at your expected peak, then raise the rate in steps until requests start to queue. The k6 breakpoint recipe covers the step plan and the script.
import http from 'k6/http';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: 1000, timeUnit: '1s', duration: '10m',
preAllocatedVUs: 1000, maxVUs: 8000,
},
},
thresholds: { http_req_failed: ['rate<0.01'], http_req_duration: ['p(99)<1500'] },
};
export default function () {
http.get('http://haproxy-staging.internal/your-busiest-endpoint');
}
Set rate to your own peak and the thresholds to your budget. While it runs, read the queue and session columns from the stats socket every two seconds:
watch -n2 "echo 'show stat' | socat stdio /run/haproxy/admin.sock \
| cut -d, -f1,2,3,4,5,6,7,18 | column -s, -t"
Those columns are proxy, server, current queue, max queue, current sessions, max sessions, session limit and status. Then count the termination states that matter in the log:
grep -oE ' (sQ|sC|SC|sH|SH)[-A-Za-z]{2} ' /var/log/haproxy.log | sort | uniq -c
sQ means the queue timed out, sC and SC mean the connect failed and sH and SH mean the server didn’t answer in time. Write your pass criteria down first. At your expected peak the queue should stay empty or drain within a second or two, with no sQ lines. Above peak, a short queue is fine, and 503s from sQ should start only past the rate you planned for. If sH shows up before the queue fills, the app is slower than maxconn assumes, so lower maxconn or fix the slow endpoint.
Keep it healthy
Scrape the metrics frontend with Prometheus. These rules use the metric names from HAProxy’s built-in exporter:
groups:
- name: haproxy-capacity
rules:
- alert: HAProxyBackendQueueing
expr: sum by (instance, proxy) (haproxy_backend_current_queue) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Requests have queued in backend {{ $labels.proxy }} for 5 minutes"
description: "Every server is at maxconn. Check app server latency first, then add servers or raise server maxconn if the app has spare capacity."
- alert: HAProxyBackend5xxRate
expr: |
sum by (instance, proxy) (rate(haproxy_backend_http_responses_total{code="5xx"}[5m]))
/ sum by (instance, proxy) (rate(haproxy_backend_http_responses_total[5m])) > 0.02
for: 5m
labels:
severity: critical
annotations:
summary: "Backend {{ $labels.proxy }} returns 5xx on over 2% of requests"
description: "Count sQ, sH and SC termination states in the HAProxy log to see if requests time out in the queue, at the server or on connect, then fix that layer."
- alert: HAProxyBackendNoServers
expr: haproxy_backend_active_servers == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Backend {{ $labels.proxy }} has no healthy servers"
description: "Every request gets a 503. Run 'show servers state' on the stats socket to see why the health checks fail, and check the app on each server."
- alert: HAProxyFrontendNearMaxconn
expr: haproxy_frontend_current_sessions / haproxy_frontend_limit_sessions > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Frontend {{ $labels.proxy }} is above 80% of its maxconn"
description: "New connections will wait in the kernel listen queue soon. Raise frontend and global maxconn if memory allows, or add HAProxy capacity."
- alert: HAProxyProcessNearMaxconn
expr: haproxy_process_current_connections / haproxy_process_max_connections > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "HAProxy on {{ $labels.instance }} is above 80% of global maxconn"
description: "Check show info for Maxconn and CurrConns, then raise global maxconn within the memory rule or add an instance."
The 2% and 80% figures are examples to replace with your own. A queue that fills for a few seconds at each spike is the design working. Five minutes of queueing means you’re short of app capacity.
Roll back
Save the config before the change:
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
To roll back, restore it, check it and reload:
cp /etc/haproxy/haproxy.cfg.bak /etc/haproxy/haproxy.cfg
haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy
A reload starts a new process on the restored config and lets the old one finish its connections. If the change is causing errors, such as sQ 503s after you lowered a server maxconn, roll back now. Otherwise wait for a quiet hour, since the old limits fail at the same load as before. When only one value is wrong, change that line alone and reload.
More recipes
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.
- NAT Gateway ErrorPortAllocation and Port Exhaustion
Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.