HAProxy maxconn Tuning for High Traffic

Size HAProxy maxconn at the global, frontend and server level and bound the queue time, so peak traffic waits briefly instead of failing with 503s.

Published

Applies to HAProxy 2.8 LTS and later; the Prometheus exporter needs a build with USE_PROMEX, as in the official image

HAProxy maxconn exists at three levels, and each one fails in its own way at peak. The global limit caps the whole process. A frontend limit caps one listener. A server maxconn caps the requests HAProxy sends to one backend at once and queues the rest. Get the server limit and timeout queue wrong and users get 503s while your app servers sit half idle.

Peak traffic and the HAProxy 503 queue timeout

Server maxconn and the queue. In HTTP mode, a server’s maxconn limits concurrent requests to that server. Requests past the limit wait in a queue inside HAProxy. When a request waits longer than timeout queue, HAProxy drops it and returns a 503 with this body:

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

The log line carries the termination state sQ: the request spent too long in the queue. If you never set timeout queue, HAProxy uses timeout connect instead. A short connect timeout, which is right for connects, then also becomes your queue limit. At peak that turns a short wait into a 503.

With no server maxconn at all, the default is unlimited. HAProxy then sends every request straight through, and the queue moves into your app. Gunicorn, Puma or PHP-FPM workers fill up, response times climb, and HAProxy logs sH when timeout server strikes first. The client gets a 504:

<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

No servers left. When health checks mark every server down, HAProxy logs backend app has no server available! at emergency level. Every request then gets the same 503.

Frontend and global limits. A frontend’s maxconn defaults to 0, which means it takes the global value. Once a frontend or the process reaches its limit, HAProxy stops accepting. New connections wait in the kernel’s listen queue and then time out, with nothing in HAProxy’s log. If you leave the global maxconn unset, HAProxy computes it from the file descriptor limit and available memory. Check the number it picked before you assume it’s large enough:

echo "show info" | socat stdio /run/haproxy/admin.sock | grep -E '^(Maxconn|CurrConns|Nbthread):'

The HAProxy maxconn config for high traffic

This is a full haproxy.cfg for one HTTP service behind three app servers. Change the addresses, the health check path and the limits to fit yours.

global
    log /dev/log local0
    maxconn 60000
    stats socket /run/haproxy/admin.sock mode 660 level admin

defaults
    mode http
    log global
    option httplog
    timeout connect 3s
    timeout client 30s
    timeout server 30s
    timeout http-request 10s
    timeout http-keep-alive 5s
    timeout queue 5s
    retries 3
    option redispatch

frontend www
    bind :80
    maxconn 50000
    default_backend app

backend app
    balance leastconn
    option httpchk
    http-check send meth GET uri /healthz
    default-server check inter 2s fall 3 rise 2 maxconn 64
    server app1 10.0.3.7:8080
    server app2 10.0.3.8:8080
    server app3 10.0.3.9:8080

frontend metrics
    bind 10.0.1.5:8405
    http-request use-service prometheus-exporter if { path /metrics }
    no log

Check it before you load it:

haproxy -c -f /etc/haproxy/haproxy.cfg

Global maxconn. It caps concurrent connections for the whole process, and HAProxy raises its own file descriptor limit to match. The configuration manual puts an established connection at about 33 kB of memory with default buffers. So size it from memory first: RAM you give HAProxy ÷ 33 kB is the ceiling. Then set it above your peak concurrent client connections, with room for a burst.

Frontend maxconn. Set it a little below the global value, so one frontend can’t take every slot from the others. With one frontend, the gap leaves room for the stats and metrics listeners.

Server maxconn. This is the setting that decides where requests wait. Set it to the number of requests the app server can work on at once. That’s workers × threads for Gunicorn or Puma, pm.max_children for PHP-FPM, or the connection pool size for an app bound by its database. I set it to the app’s real concurrency, so the queue builds in HAProxy where I can see it and measure it. A limit far above that figure lets requests queue inside the app, and you lose that view.

Timeout queue. It bounds how long a request waits for a free server slot. Size it from the latency budget: queue time plus a normal response must fit inside what the client will wait. If the budget for an endpoint is 6 seconds and responses take up to 1 second, a 5-second queue is the most you can spend. A longer queue doesn’t save requests at peak; it turns fast 503s into slow ones.

Balance leastconn. leastconn sends each request to the server with the fewest active connections. With uneven request times, it keeps one server from collecting the slow requests while others sit free. roundrobin suits short, even requests.

Health checks. check inter 2s fall 3 rise 2 marks a server down after three failed checks and back up after two good ones. Point /healthz at something cheap that proves the app can serve, not at a page that hits the database for every check.

Retries and redispatch. By default retries applies to failed connection attempts only, and 3 is the default. option redispatch lets the last retry go to a different server. Leave retry-on unset unless every request on the backend is safe to resend.

Timeouts. timeout http-request limits how long a client may take to send its headers, and it stops slow clients holding slots. timeout server should sit above your slowest legitimate response and below the timeout of whatever sits in front of HAProxy. A cloud load balancer in front needs an idle timeout above timeout client; the ALB 502 and 504 recipe covers that side.

Connection reuse. http-reuse safe is the default. HAProxy keeps idle server connections and reuses them for later requests, so you don’t need to set it.

Threads. HAProxy starts one thread per CPU it’s bound to when you leave nbthread unset. In a container with a CPU limit, compare Nbthread from show info with the CPUs the pod may use. Set nbthread to the limit if it reads higher.

Metrics. use-service prometheus-exporter serves Prometheus metrics from HAProxy itself. It needs a build with the Prometheus exporter. haproxy -vv prints “Built with the Prometheus exporter as a service” when it’s there, and the official Docker image includes it.

Test it on your system

Point a fixed arrival rate at a staging HAProxy with the same app servers and limits as production. Start at your expected peak, then raise the rate in steps until requests start to queue. The k6 breakpoint recipe covers the step plan and the script.

import http from 'k6/http';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: 1000, timeUnit: '1s', duration: '10m',
      preAllocatedVUs: 1000, maxVUs: 8000,
    },
  },
  thresholds: { http_req_failed: ['rate<0.01'], http_req_duration: ['p(99)<1500'] },
};

export default function () {
  http.get('http://haproxy-staging.internal/your-busiest-endpoint');
}

Set rate to your own peak and the thresholds to your budget. While it runs, read the queue and session columns from the stats socket every two seconds:

watch -n2 "echo 'show stat' | socat stdio /run/haproxy/admin.sock \
  | cut -d, -f1,2,3,4,5,6,7,18 | column -s, -t"

Those columns are proxy, server, current queue, max queue, current sessions, max sessions, session limit and status. Then count the termination states that matter in the log:

grep -oE ' (sQ|sC|SC|sH|SH)[-A-Za-z]{2} ' /var/log/haproxy.log | sort | uniq -c

sQ means the queue timed out, sC and SC mean the connect failed and sH and SH mean the server didn’t answer in time. Write your pass criteria down first. At your expected peak the queue should stay empty or drain within a second or two, with no sQ lines. Above peak, a short queue is fine, and 503s from sQ should start only past the rate you planned for. If sH shows up before the queue fills, the app is slower than maxconn assumes, so lower maxconn or fix the slow endpoint.

Keep it healthy

Scrape the metrics frontend with Prometheus. These rules use the metric names from HAProxy’s built-in exporter:

groups:
  - name: haproxy-capacity
    rules:
      - alert: HAProxyBackendQueueing
        expr: sum by (instance, proxy) (haproxy_backend_current_queue) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Requests have queued in backend {{ $labels.proxy }} for 5 minutes"
          description: "Every server is at maxconn. Check app server latency first, then add servers or raise server maxconn if the app has spare capacity."
      - alert: HAProxyBackend5xxRate
        expr: |
          sum by (instance, proxy) (rate(haproxy_backend_http_responses_total{code="5xx"}[5m]))
            / sum by (instance, proxy) (rate(haproxy_backend_http_responses_total[5m])) > 0.02
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Backend {{ $labels.proxy }} returns 5xx on over 2% of requests"
          description: "Count sQ, sH and SC termination states in the HAProxy log to see if requests time out in the queue, at the server or on connect, then fix that layer."
      - alert: HAProxyBackendNoServers
        expr: haproxy_backend_active_servers == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Backend {{ $labels.proxy }} has no healthy servers"
          description: "Every request gets a 503. Run 'show servers state' on the stats socket to see why the health checks fail, and check the app on each server."
      - alert: HAProxyFrontendNearMaxconn
        expr: haproxy_frontend_current_sessions / haproxy_frontend_limit_sessions > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Frontend {{ $labels.proxy }} is above 80% of its maxconn"
          description: "New connections will wait in the kernel listen queue soon. Raise frontend and global maxconn if memory allows, or add HAProxy capacity."
      - alert: HAProxyProcessNearMaxconn
        expr: haproxy_process_current_connections / haproxy_process_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "HAProxy on {{ $labels.instance }} is above 80% of global maxconn"
          description: "Check show info for Maxconn and CurrConns, then raise global maxconn within the memory rule or add an instance."

The 2% and 80% figures are examples to replace with your own. A queue that fills for a few seconds at each spike is the design working. Five minutes of queueing means you’re short of app capacity.

Roll back

Save the config before the change:

cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

To roll back, restore it, check it and reload:

cp /etc/haproxy/haproxy.cfg.bak /etc/haproxy/haproxy.cfg
haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy

A reload starts a new process on the restored config and lets the old one finish its connections. If the change is causing errors, such as sQ 503s after you lowered a server maxconn, roll back now. Otherwise wait for a quiet hour, since the old limits fail at the same load as before. When only one value is wrong, change that line alone and reload.

More recipes

Further reading