upstream timed out (110: Connection timed out) while reading response header from upstream means nginx sent a request and your app sent nothing back within proxy_read_timeout. nginx gives up and answers 504. At peak a slow backend sets off a chain: timeouts trigger retries, retries add load, and the failure counters pull servers out of the pool. Once every server is out, nginx logs no live upstreams and answers 502 without trying the backend at all.
The capacity side of the same proxy, worker connections and keepalive pools, lives in Nginx tuning for high traffic.
Upstream timed out and no live upstreams while connecting to upstream
Each line below comes from nginx’s error log. The client status is what nginx returns once it gives up on that request.
| Error log line | Client gets | What happened |
|---|---|---|
upstream timed out (110: Connection timed out) while connecting to upstream | 504 | The TCP connect didn’t finish within proxy_connect_timeout. The backend’s listen queue is full, or the host is gone. |
upstream timed out (110: Connection timed out) while reading response header from upstream | 504 | The backend accepted the request but sent no response header within proxy_read_timeout. |
connect() failed (111: Connection refused) while connecting to upstream | 502 | Nothing listens on that port. The app crashed, or a pod is restarting. |
no live upstreams while connecting to upstream | 502 | nginx has marked every server in the group as failed, or every server sits at max_conns. |
upstream prematurely closed connection while reading response header from upstream | 502 | The backend closed the connection mid-request, often a keepalive timeout mismatch. |
Alpine images use musl, which prints (110: Operation timed out) for the same errno. Grep for upstream timed out to catch both.
Under load the defaults turn one slow dependency into an outage. Your database slows down, so app responses slow down too. With proxy_read_timeout 60s, a user waits a full minute before the 504. proxy_next_upstream defaults to error timeout, so nginx resends each timed-out GET to the next server. proxy_next_upstream_tries defaults to 0, which means no limit, so one request can visit every server. Each retry lands on a backend that’s as busy as the first.
Every timeout also counts as a failed attempt. max_fails defaults to 1 and fail_timeout to 10 seconds. One slow response takes a server out of rotation for 10 seconds, and its traffic moves to the others. They slow down, fail and drop out in turn. With every server marked down, nginx logs no live upstreams and returns 502 at once.
A group with a single server behaves differently. nginx ignores max_fails and fail_timeout there, so the server is never marked unavailable. You get a wall of 504s instead of no live upstreams, unless you set max_conns.
The fix for nginx proxy timeouts and retries
This file replaces the upstream and proxy settings for one app. Change the addresses to your backends, and keep the keepalive lines from the high-traffic recipe if you already use them.
# /etc/nginx/conf.d/app.conf
log_format upstream_time '$remote_addr [$time_local] "$request" $status '
'rt=$request_time uct=$upstream_connect_time '
'uht=$upstream_header_time urt=$upstream_response_time '
'ua="$upstream_addr" us="$upstream_status"';
upstream app {
zone app 64k;
server 10.0.3.7:8080 max_fails=3 fail_timeout=10s max_conns=200;
server 10.0.3.8:8080 max_fails=3 fail_timeout=10s max_conns=200;
server 10.0.3.9:8080 max_fails=3 fail_timeout=10s max_conns=200;
keepalive 64;
}
server {
listen 80;
access_log /var/log/nginx/app.log upstream_time;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 2s;
proxy_send_timeout 10s;
proxy_read_timeout 15s;
proxy_next_upstream error timeout http_503;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 20s;
location / {
proxy_pass http://app;
}
# one slow endpoint gets its own budget and no retries
location /reports/ {
proxy_pass http://app;
proxy_read_timeout 120s;
proxy_next_upstream off;
}
}
The log format. $upstream_connect_time, $upstream_header_time and $upstream_response_time split each request into connect, wait and transfer. $upstream_status and $upstream_addr list every server nginx tried, comma separated, so a retried request shows up as us="504, 200". You’ll size every timeout below from this log.
Connect timeout. proxy_connect_timeout caps the TCP handshake with a backend. On the same network a healthy backend connects in milliseconds. The default of 60 seconds means a dead host holds each request for a minute before nginx moves on. A couple of seconds covers a busy listen queue. The nginx docs note the value can’t usually exceed 75 seconds.
Read timeout. proxy_read_timeout limits the gap between two reads from the backend, not the whole response. A response that streams a byte every 10 seconds never trips a 15-second timeout. Set it from your access log: take the slowest response time you accept for that location at peak, then add headroom. Too low, and nginx cuts off real requests with 504. Too high, and users wait out a stuck backend while it holds a connection slot.
Send timeout. proxy_send_timeout is the same gap for writing the request body to the backend. It matters for uploads, and the default of 60 seconds suits most of them.
Per-location budgets. One timeout for the whole site forces a bad trade. Give known slow paths, such as reports or exports, their own location with a longer read timeout. Keep the default tight for everything else.
Timeout order. The layer nearest the user should wait longest, and the app should give up first. The app’s own request timeout sits below proxy_read_timeout, so the app returns its own error and frees its worker. Gunicorn’s timeout defaults to 30 seconds, as the Gunicorn workers and timeouts recipe explains. In front of nginx, an AWS ALB idle timeout defaults to 60 seconds, and the ALB 502 and 504 recipe covers that layer. Write the chain down before you change any one value:
client or load balancer timeout > proxy_read_timeout > app request timeout > database statement timeout
Retries. proxy_next_upstream lists what counts as a failure worth retrying elsewhere. error and timeout always count toward max_fails, even when you leave them off the list. Adding http_503 lets a backend that’s draining for a deploy hand the request on. nginx only retries while nothing has reached the client. It won’t resend POST, PATCH or LOCK requests unless you add non_idempotent, and I leave that off for anything that writes.
Retry budget. proxy_next_upstream_tries 2 means the first attempt plus one retry. proxy_next_upstream_timeout 20s caps the time spent across all attempts. I cap both on any proxy that faces peak traffic. An unlimited retry turns one slow request into a visit to every server in the pool.
Failure counting. max_fails=3 fail_timeout=10s takes a server out only after three failures inside 10 seconds, and keeps it out for 10 seconds. With the default of 1, a single slow response ejects a healthy server. Set fail_timeout near the time your app needs to restart. Open source nginx checks health only through real requests. Active health checks with the health_check directive need NGINX Plus.
Connection caps. max_conns limits active connections to each server. Past the cap nginx tries the next server, and with every server full it logs no live upstreams and returns 502 at once. That is load shedding, and it protects a backend with fixed concurrency, such as a fixed number of Gunicorn workers or Puma threads. Size it to the backend’s real concurrency plus a little room. Without the zone line, each nginx worker counts its own connections, so the real cap is max_conns times your worker count. Open source nginx can’t queue requests at the cap; the queue directive is part of NGINX Plus. If you’d rather queue than shed, put HAProxy with server maxconn in that spot.
Kubernetes. On ingress-nginx, the same settings come from annotations such as nginx.ingress.kubernetes.io/proxy-read-timeout and nginx.ingress.kubernetes.io/proxy-next-upstream-tries. The project retired ingress-nginx in March 2026, so plan the move to a Gateway API controller and carry these values with you.
Test it on your system
Use a staging copy with at least two backends and the same nginx version and config as production. Run a fixed arrival rate at your peak against your busiest endpoints. The k6 breakpoint recipe explains how to build the request mix and pick the rate.
import http from 'k6/http';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: 500, timeUnit: '1s', duration: '15m',
preAllocatedVUs: 500, maxVUs: 5000,
},
},
thresholds: { http_req_failed: ['rate<0.01'], http_req_duration: ['p(99)<2000'] },
};
export default function () {
http.get('http://nginx-staging.internal/your-busiest-endpoint');
}
Set rate and the thresholds to your own peak and budget. Then break one backend on purpose while the test runs, one fault at a time:
# fault one, run on one backend, adds 3 seconds of delay
sudo tc qdisc add dev eth0 root netem delay 3000ms
sudo tc qdisc del dev eth0 root netem
# fault two stops one backend for two minutes, then starts it
sudo systemctl stop app && sleep 120 && sudo systemctl start app
Replace eth0 with the backend’s interface, and run tc only on a staging host you can reach another way. On Kubernetes, delete one backend pod instead of stopping a service.
Watch three things on the nginx host while each fault runs:
tail -f /var/log/nginx/error.log | grep -E 'upstream timed out|no live upstreams|connect\(\) failed|prematurely closed'
grep -c 'us="[^"]*,' /var/log/nginx/app.log
awk '{print $7}' /var/log/nginx/app.log | sort | uniq -c
The first shows which failure nginx hits. The second counts retried requests, whose us= field lists more than one status. The third counts status codes. Hold each fault for longer than fail_timeout, then clear it and watch the server return.
Agree the pass criteria before the run. With one backend slow or stopped, nginx should never log no live upstreams while another backend is healthy. 504s should come only from requests that were already in flight on the slow backend. Retried requests should stay a small share, and p99 should stay inside your budget once the bad server drops out. If no live upstreams shows up with healthy servers left, raise max_fails or max_conns.
Keep it healthy
Status codes and upstream latency need metrics that stock stub_status doesn’t have. The nginx Prometheus monitoring recipe adds the VTS module, which exports them per server and per upstream. These rules assume that setup, with vhost_traffic_status_histogram_buckets set:
groups:
- name: nginx-upstream-timeouts
rules:
- alert: NginxHigh5xxRate
expr: |
sum by (instance, host) (rate(nginx_vts_server_requests_total{code="5xx"}[5m]))
/ sum by (instance, host) (rate(nginx_vts_server_requests_total[5m])) > 0.02
for: 5m
labels:
severity: critical
annotations:
summary: "nginx {{ $labels.instance }} returns 5xx on over 2% of requests for {{ $labels.host }}"
description: "Grep the error log for 'upstream timed out' and 'no live upstreams' to see which failure it is, then check the backends named in the upstream_time log."
- alert: NginxUpstreamNearReadTimeout
# 12s is 80% of proxy_read_timeout 15s; change both together
expr: |
histogram_quantile(0.99,
sum by (instance, upstream, le) (rate(nginx_vts_upstream_response_duration_seconds_bucket[5m]))
) > 12
for: 10m
labels:
severity: warning
annotations:
summary: "p99 for upstream {{ $labels.upstream }} is close to proxy_read_timeout"
description: "Requests will start failing with 504 soon. Find the slow endpoint in the upstream_time log and fix the backend, or give that location its own timeout."
The 2% threshold is an example. Set yours from your error budget. Also alert on these lines in Loki or your log tool, since a short burst can hide inside a 5-minute rate:
no live upstreams while connecting to upstream
upstream timed out
connect() failed (111: Connection refused) while connecting to upstream
no live upstreams deserves its own page. When it fires and the backends look healthy, max_fails or max_conns is set too low for the traffic.
Roll back
Save the file before you edit it:
cp /etc/nginx/conf.d/app.conf /etc/nginx/conf.d/app.conf.bak
To roll back, restore it, test, then reload:
cp /etc/nginx/conf.d/app.conf.bak /etc/nginx/conf.d/app.conf
nginx -t && nginx -s reload
A reload lets old workers finish their open requests. The timeout change is the one most likely to hurt. If 504s jump on an endpoint that used to work, its responses take longer than the new proxy_read_timeout. Restore the old value now, or give that path its own location. Revert the retry and max_fails changes off-peak if they don’t cause errors, because the old values only show their cost at the next slowdown.
More recipes
- Nginx Prometheus Exporter and 5xx Error Rate Alerts
Scrape stub_status for connections and the VTS module for status codes and upstream latency, so a rising nginx 5xx error rate pages you before users do.
- Nginx Tuning for High Traffic
Size worker_connections for two sockets per request and keep a right-sized upstream keepalive pool, so nginx stops dropping connections at peak.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.