Exponential Backoff and Jitter for Timeouts and Retries

Give every call a deadline, a per-try timeout and jittered backoff under a retry budget, so a slow dependency can't set off a retry storm.

Published

Applies to Go 1.22 and later, Python 3.9 and later with requests, Node.js 20.3 and later; any HTTP or gRPC client

Exponential backoff and jitter spread retries out in time, so a struggling service gets room to recover. They need two other limits beside them, a timeout on every attempt and a cap on retry traffic. Without those, retries turn one slow dependency into an outage across every service that calls it.

What breaks at peak

A dependency slows down at peak. Maybe its database is busy, or a deploy left it short of pods. Its callers hold a worker, thread or socket for every request that waits. With no timeout, they wait as long as the dependency takes:

  • Go’s http.Client has no timeout unless you set Timeout. Its docs say “A Timeout of zero means no timeout.”
  • Python requests waits forever unless you pass timeout.
  • Node.js fetch has long default timeouts and no deadline from your code.

The caller’s pool fills with waiting requests. Its own latency climbs, and its callers start timing out too. Then the retries arrive, and the slowdown turns into a retry storm.

Retries multiply load. Three attempts per call means up to three times the traffic on a failing dependency. If three layers each retry three times, the service at the bottom can see 27 calls for one user request. The extra load lands on the one service that’s already struggling.

Retries without jitter arrive together. A fixed one-second backoff sends every failed client back at the same moment. The dependency takes the full retry wave each second. Exponential backoff spreads the waves out, but clients that failed together still retry together unless each waits a random time.

Retries outlive the user. A retry sent after the caller has given up does work nobody will read. A gateway with a 10-second timeout in front of a service that retries for 30 seconds wastes 20 seconds of capacity per request.

These are the timeout errors you’ll find in the logs, from Go, Python requests and Node.js:

context deadline exceeded (Client.Timeout exceeded while awaiting headers)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='pricing.internal', port=80): Read timed out. (read timeout=0.8)
TimeoutError: The operation was aborted due to timeout

The fix

Every outbound call gets four limits:

  1. A deadline for the whole call, retries included. It comes from the caller. Pass it down as a Go context, an AbortSignal or a time budget.
  2. A per-try timeout. Start near the dependency’s p99 at peak, plus some margin. Shorter fails good requests. Longer holds workers while the dependency hangs.
  3. Retries with full jitter. Wait a random time between zero and min(cap, base × 2^attempt). This is the “full jitter” scheme from the AWS Architecture Blog. It spreads retries better than adding a small random offset to a fixed backoff.
  4. A retry budget. Allow retries while they stay a small share of traffic, such as 10%. Past that, fail fast. A budget lets retries cover a brief blip and stops them in an outage.

Retry what’s safe and useful, and nothing else:

  • Retry idempotent requests: GET, PUT, DELETE, or a POST that carries an idempotency key the server checks.
  • Retry connection errors, timeouts, 429, 502, 503 and 504. A 400 or 404 won’t change on a second try.
  • Honor Retry-After when the server sends one, and stop if it asks for longer than your deadline allows.
  • Stop the moment the caller’s deadline passes.

Size the numbers as one budget. The deadline must cover attempts times the per-try timeout, plus the backoff waits. It must also sit below the caller’s own timeout. With a 2-second deadline, two retries and a per-try timeout of 600 ms, you leave about 200 ms for backoff.

Go. rand.N needs Go 1.22 or later. The budget works like gRPC’s retry throttling. A failure spends a token and a success earns back a tenth of one. Retries stop once the bucket drops below half.

package main

import (
	"context"
	"fmt"
	"math/rand/v2"
	"net/http"
	"sync"
	"time"
)

// Each attempt gets its own ceiling, which covers headers and body.
var client = &http.Client{Timeout: 800 * time.Millisecond}

type retryBudget struct {
	mu     sync.Mutex
	tokens float64
	max    float64
	ratio  float64
}

func newRetryBudget(max, ratio float64) *retryBudget {
	return &retryBudget{tokens: max, max: max, ratio: ratio}
}

func (b *retryBudget) record(ok bool) {
	b.mu.Lock()
	defer b.mu.Unlock()
	if ok {
		b.tokens = min(b.max, b.tokens+b.ratio)
	} else {
		b.tokens = max(0, b.tokens-1)
	}
}

func (b *retryBudget) allow() bool {
	b.mu.Lock()
	defer b.mu.Unlock()
	return b.tokens > b.max/2
}

// fullJitter waits a random time between zero and the capped exponential backoff.
func fullJitter(base, ceiling time.Duration, attempt int) time.Duration {
	return rand.N(min(ceiling, base<<attempt) + 1)
}

func retryableStatus(code int) bool {
	switch code {
	case http.StatusTooManyRequests, http.StatusBadGateway,
		http.StatusServiceUnavailable, http.StatusGatewayTimeout:
		return true
	}
	return false
}

// getWithRetry sends an idempotent GET. ctx carries the caller's whole deadline.
func getWithRetry(ctx context.Context, budget *retryBudget, url string) (*http.Response, error) {
	const maxAttempts = 3
	var lastErr error

	for attempt := 0; attempt < maxAttempts; attempt++ {
		if attempt > 0 {
			if !budget.allow() {
				return nil, fmt.Errorf("retry budget empty: %w", lastErr)
			}
			select {
			case <-time.After(fullJitter(100*time.Millisecond, 2*time.Second, attempt)):
			case <-ctx.Done():
				return nil, ctx.Err()
			}
		}

		req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
		if err != nil {
			return nil, err
		}
		resp, err := client.Do(req)
		switch {
		case ctx.Err() != nil:
			return nil, ctx.Err() // the caller gave up, so retrying is wasted work
		case err != nil:
			lastErr = err
		case retryableStatus(resp.StatusCode):
			resp.Body.Close()
			lastErr = fmt.Errorf("upstream returned %d", resp.StatusCode)
		default:
			budget.record(true)
			return resp, nil
		}
		budget.record(false)
	}
	return nil, lastErr
}

Create one budget per dependency, such as pricingBudget := newRetryBudget(10, 0.1), and share it across requests. Call it with the inbound request’s context, so the deadline flows through.

Python. The timeout tuple sets the connect and read timeouts separately. The function shrinks both to fit the time left in the deadline. It also counts attempts with prometheus_client, which the alerts below read.

import random
import time

import requests
from prometheus_client import Counter

ATTEMPTS = Counter("client_attempts_total", "Outbound attempts", ["target", "kind"])
RETRYABLE = {429, 502, 503, 504}

session = requests.Session()


def get_with_retry(url, target, deadline_s=2.0, per_try=(0.5, 0.8), max_attempts=3):
    """Idempotent GET with a total deadline, a per-try timeout and full jitter."""
    deadline = time.monotonic() + deadline_s
    last_error = None
    for attempt in range(max_attempts):
        if attempt:
            ceiling = min(2.0, 0.1 * 2**attempt)
            time.sleep(random.uniform(0, ceiling))
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            break
        ATTEMPTS.labels(target, "retry" if attempt else "first").inc()
        try:
            # (connect, read) timeouts, never longer than the time left
            resp = session.get(url, timeout=(min(per_try[0], remaining), min(per_try[1], remaining)))
        except (requests.ConnectionError, requests.Timeout) as exc:
            last_error = exc
            continue
        if resp.status_code not in RETRYABLE:
            return resp
        last_error = requests.HTTPError(f"{target} returned {resp.status_code}", response=resp)
    raise last_error or TimeoutError(f"{target}: deadline spent before the first try")

urllib3’s Retry class, which requests uses under HTTPAdapter, does the same job with less code. Two details matter. Its first retry goes out with no wait at all, because the backoff formula returns zero for it. And backoff_jitter adds a random amount on top of the exponential wait, which spreads retries less than full jitter.

Node.js. AbortSignal.any needs Node.js 20.3 or later. It combines the caller’s deadline with a per-try timeout, and the backoff sleep also stops when the caller gives up.

import { setTimeout as sleep } from 'node:timers/promises';

const RETRYABLE = new Set([429, 502, 503, 504]);

// signal is the caller's whole deadline, for example AbortSignal.timeout(2000)
export async function getWithRetry(url, { signal, perTryMs = 800, maxAttempts = 3 } = {}) {
  let lastError;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    if (attempt > 0) {
      const ceiling = Math.min(2000, 100 * 2 ** attempt);
      await sleep(Math.random() * ceiling, undefined, { signal });
    }
    try {
      const res = await fetch(url, {
        signal: AbortSignal.any([signal, AbortSignal.timeout(perTryMs)]),
      });
      if (!RETRYABLE.has(res.status)) return res;
      await res.body?.cancel();
      lastError = new Error(`upstream returned ${res.status}`);
    } catch (err) {
      if (signal.aborted) throw signal.reason; // the caller gave up
      lastError = err;
    }
  }
  throw lastError;
}

The Python and Node.js versions leave out the budget to stay short. Port the Go retryBudget before you ship either one.

Retry in one layer. I allow retries in the layer next to the dependency and set every other layer to zero retries. A service mesh counts as a layer. An Istio VirtualService with retries.attempts multiplies with the retries in your code. Pick one place and turn the other off. gRPC clients get the same budget from the service config, with retryThrottling set to maxTokens and tokenRatio.

A breaker stops calls to a dependency that keeps failing, which a retry budget alone doesn’t do. The circuit breaker recipe covers that next step. For a worked case of retrying overload errors per item, read the Elasticsearch rejections recipe, where bulk requests return 429s inside an HTTP 200.

Test it on your system

Test in staging, with the real call chain and a fault you control. Toxiproxy sits between your service and its dependency and adds latency or drops connections on command:

toxiproxy-cli create -l 0.0.0.0:18080 -u pricing.internal:8080 pricing
# point the service's PRICING_URL at the proxy on port 18080, then add latency
toxiproxy-cli toxic add -t latency -a latency=2000 -a jitter=500 pricing

Drive the front of the chain at a fixed arrival rate with k6, so a slower system doesn’t get less traffic:

import http from 'k6/http';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: 300, timeUnit: '1s', duration: '15m',
      preAllocatedVUs: 300, maxVUs: 3000,
    },
  },
};

export default function () {
  http.get('https://staging.example.com/checkout/quote', { timeout: '5s' });
}

Run it for five minutes clean, add the latency toxic for five minutes, then remove it with toxiproxy-cli toxic remove -n latency_downstream pricing. Repeat with -t timeout -a timeout=0, a toxic that holds connections open with no answer.

Watch the ratio of calls to the dependency over requests at the front. That’s your retry amplification. Watch the front’s p99 and error rate, and the worker or connection pool of the service in the middle.

Set the pass criteria before the run. With the fault on, amplification stays at or under 1 plus your budget, such as 1.1. The middle service’s pool doesn’t fill, and the front answers within its own timeout, with errors where the deadline ran out. After you remove the toxic, the dependency’s load returns to normal within a few seconds, with no burst of delayed retries.

Keep it healthy

The Python client above exports client_attempts_total with a kind label of first or retry. Add the same counter to the Go and Node.js clients with the Prometheus client library you already use. These rules watch the retry share per target:

groups:
  - name: client-retries
    rules:
      - alert: ClientRetryShareHigh
        expr: |
          sum by (job, target) (rate(client_attempts_total{kind="retry"}[5m]))
            /
          sum by (job, target) (rate(client_attempts_total{kind="first"}[5m]))
            > 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} retries over 10% of its calls to {{ $labels.target }}"
          description: "The dependency is failing or slow. Check its error rate and p99, and check that the client's per-try timeout still fits its peak latency."
      - alert: ClientRetryStorm
        expr: |
          sum by (target) (rate(client_attempts_total{kind="retry"}[5m]))
            /
          sum by (target) (rate(client_attempts_total{kind="first"}[5m]))
            > 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Retries are adding over 50% load on {{ $labels.target }}"
          description: "Retries outran the budget, or a layer retries without one. Find which callers retry most, turn their retries off, then fix the dependency."

Set the 10% to match your retry budget. Above it, the budget should already refuse retries, so a breach means a client without one. The storm rule sums every caller of a target, because the dependency feels all of them at once.

Roll back

Put the retry settings in config, such as an environment variable for attempts and per-try timeout, so you can change them without a code deploy. Note the current values before you change them.

Setting attempts to one turns retries off and keeps the timeouts. If the new settings raise the error rate, or the amplification grows past your budget, roll back now. A timeout tighter than the dependency’s real p99 fails good requests, so check that first. For any other reason, revert at a quiet hour, so you can watch the error rate settle before the next peak.

More recipes

Further reading