A cache stampede starts the moment a hot key expires. Every request that misses rebuilds the value itself, and they all hit the database at once. The fix is to let one caller rebuild while the rest wait or take the old copy, and to refresh hot keys before they expire.
What breaks at peak
Take a product page cached for 60 seconds. At peak it serves 500 requests a second from Redis. The key expires. In the time one rebuild takes, say 300 ms, about 150 requests miss. Each runs the same query.
You sized the database for the cache hit rate, and now it gets 150 copies of its heaviest query. That query slows down under the load, so the rebuild takes longer, and more requests miss while it runs. The herd grows until the first rebuild lands, or until the database runs out of connections. You’ll see it as one of these:
- Database CPU and active connections spike on a fixed rhythm that matches your TTLs.
- p99 latency jumps in a sawtooth while p50 stays flat.
FATAL: sorry, too many clients alreadyfrom Postgres, covered in the PgBouncer recipe.
Three events start a stampede besides normal expiry:
- Keys written together expire together. A deploy that warms the cache sets every key to expire in the same second. So does a batch job that writes thousands of keys.
- A cold cache. A Redis restart or failover, or a cache flush, turns every key into a miss at once.
- Eviction. A full Redis evicts keys under memory pressure, and a hot key can go with them. The Redis memory recipe covers that cap.
The same pattern hits a CDN when a popular object expires at the edge, which the CDN cache hit ratio recipe handles.
The fix
Four techniques, from cheapest to strongest:
- Request coalescing. Inside one process, the first miss rebuilds the key and the others wait for that result. Go’s
singleflightand a promise map in Node.js do this. With 20 pods, a stampede drops to at most 20 rebuilds. - TTL jitter. Add a random 10% to each TTL, so keys written together expire across a spread of seconds.
- A rebuild lock. Across processes, a Redis
SET key value NX PX ttllets one caller rebuild. The rest serve the stale copy or wait a short time. - Early refresh. Rebuild a hot key before it expires, so it never misses. XFetch decides by probability, and stale-while-revalidate decides by a stale window.
I start with coalescing and jitter, because they need no shared state and fix most of the damage. I add the lock and early refresh for keys whose rebuild is expensive enough that one per pod still hurts.
Go, with singleflight and jitter. DoChan lets each caller stop waiting when its own context ends, while the rebuild keeps going for the others. context.WithoutCancel needs Go 1.21.
package main
import (
"context"
"math/rand/v2"
"sync"
"time"
"golang.org/x/sync/singleflight"
)
type entry struct {
value []byte
expires time.Time
}
type Cache struct {
mu sync.RWMutex
items map[string]entry
group singleflight.Group
ttl time.Duration
}
func NewCache(ttl time.Duration) *Cache {
return &Cache{items: map[string]entry{}, ttl: ttl}
}
// Get returns the cached value or runs load once per key, however many callers miss together.
func (c *Cache) Get(ctx context.Context, key string, load func(context.Context) ([]byte, error)) ([]byte, error) {
c.mu.RLock()
e, ok := c.items[key]
c.mu.RUnlock()
if ok && time.Now().Before(e.expires) {
return e.value, nil
}
ch := c.group.DoChan(key, func() (any, error) {
// Detach from the first caller's cancel, so one impatient client
// doesn't fail every request waiting on this key.
v, err := load(context.WithoutCancel(ctx))
if err != nil {
return nil, err
}
// Spread expiry by up to 10% so keys written together don't expire together.
jitter := time.Duration(rand.Int64N(int64(c.ttl) / 10))
c.mu.Lock()
c.items[key] = entry{value: v, expires: time.Now().Add(c.ttl + jitter)}
c.mu.Unlock()
return v, nil
})
select {
case r := <-ch:
if r.Err != nil {
return nil, r.Err
}
return r.Val.([]byte), nil
case <-ctx.Done():
return nil, ctx.Err()
}
}
The same singleflight call works in front of Redis. Wrap the Redis read and the database rebuild in the DoChan function, and each pod sends one rebuild per key.
Python, with a Redis lock and XFetch. XFetch comes from the paper “Optimal Probabilistic Cache Stampede Prevention” by Vattani, Chierichetti and Lowenstein. Each read rolls a random number. The closer the key is to expiry, and the longer its last rebuild took, the more likely that read refreshes it early. The function stores the rebuild time next to the value for that reason.
import json
import math
import random
import time
import uuid
import redis
r = redis.Redis(host="redis.internal", port=6379, socket_timeout=0.2)
BETA = 1.0 # above 1 refreshes earlier, below 1 later
def cached(key, ttl_s, rebuild):
"""Serve from Redis; refresh early with XFetch and let one caller rebuild."""
raw, pttl_ms = r.get(key), r.pttl(key)
if raw is not None and pttl_ms > 0:
item = json.loads(raw)
# XFetch: the closer the expiry and the slower the rebuild, the likelier an early refresh.
early = -item["delta"] * BETA * math.log(1.0 - random.random())
if early < pttl_ms / 1000:
return item["value"]
if not _try_lock(key):
return item["value"] # someone else is refreshing; serve the current copy
return _rebuild(key, ttl_s, rebuild)
if _try_lock(key):
return _rebuild(key, ttl_s, rebuild)
# Cold miss and another worker holds the lock: wait briefly for its result.
for _ in range(20):
time.sleep(0.05)
raw = r.get(key)
if raw is not None:
return json.loads(raw)["value"]
return rebuild() # the lock holder is slow or gone; fall back to a direct read
def _try_lock(key):
return bool(r.set(f"lock:{key}", uuid.uuid4().hex, nx=True, px=10_000))
def _rebuild(key, ttl_s, rebuild):
start = time.monotonic()
try:
value = rebuild()
delta = time.monotonic() - start
ttl = int(ttl_s * random.uniform(0.9, 1.1)) # jitter the TTL by 10%
r.set(key, json.dumps({"value": value, "delta": delta}), ex=ttl)
return value
finally:
r.delete(f"lock:{key}")
The settings to size:
- Lock expiry (
px). Set it well above your slowest rebuild, such as twice the p99. Too short, and a second caller takes the lock while the first still runs. The release here deletes the lock without checking its owner, so keep rebuilds far under the expiry. - The wait loop. It waits up to one second, in 50 ms steps, for another worker’s result on a cold miss. Keep it under your request timeout. After it, the caller reads the database directly, so a crashed lock holder can’t hang every request. I cap every wait on a lock this way.
BETA. At 1.0, XFetch refreshes close to expiry for fast rebuilds and earlier for slow ones. Raise it for keys whose miss costs the most.
Node.js, with coalescing and stale-while-revalidate. Inside the stale window, callers get the old value at once while one refresh runs in the background. Past it, they share one rebuild:
const cache = new Map(); // key -> { value, expires, staleUntil }
const inFlight = new Map(); // key -> Promise
export async function getCached(key, ttlMs, staleMs, load) {
const hit = cache.get(key);
const now = Date.now();
if (hit && now < hit.expires) return hit.value;
if (hit && now < hit.staleUntil) {
refresh(key, ttlMs, staleMs, load).catch(() => {}); // stale-while-revalidate
return hit.value;
}
return refresh(key, ttlMs, staleMs, load);
}
function refresh(key, ttlMs, staleMs, load) {
let p = inFlight.get(key);
if (!p) {
p = load()
.then((value) => {
const ttl = ttlMs * (0.9 + Math.random() * 0.2); // jitter by 10%
cache.set(key, { value, expires: Date.now() + ttl, staleUntil: Date.now() + ttl + staleMs });
return value;
})
.finally(() => inFlight.delete(key));
inFlight.set(key, p);
}
return p;
}
Set staleMs to how old a value your product can show. A price may tolerate seconds; a stock count may tolerate none. A Node.js process in cluster mode holds one Map per worker, so coalescing works per worker, as the cluster mode recipe explains. Bound the Map with an LRU in production, or it grows with every key you’ve ever cached.
Test it on your system
Use staging with production-like data, the same cache TTLs and the same pod count. Drive the hottest endpoint at a fixed arrival rate, so a slow database doesn’t throttle the test:
import http from 'k6/http';
export const options = {
scenarios: {
hot: {
executor: 'constant-arrival-rate',
rate: 500, timeUnit: '1s', duration: '10m',
preAllocatedVUs: 300, maxVUs: 3000,
},
},
thresholds: { http_req_duration: ['p(99)<300'], http_req_failed: ['rate<0.001'] },
};
export default function () {
http.get('https://staging.example.com/products/123');
}
Set the rate and thresholds to your own peak and targets. Two minutes in, force the expiry by hand, then repeat for a whole key family to simulate a cold cache:
redis-cli UNLINK product:123
redis-cli --scan --pattern 'product:*' | xargs -n 500 redis-cli UNLINK
While it runs, count how often the database ran the rebuild query, from pg_stat_statements:
SELECT calls, mean_exec_time
FROM pg_stat_statements
WHERE query LIKE '%FROM products WHERE id = $1%';
Watch active connections in pg_stat_activity, database CPU and the k6 p99 next to it.
Run the test before and after the change, and decide the pass criteria first. After the change, one expiry adds at most one rebuild per pod with coalescing, or one in total with the lock. Active connections and database CPU stay flat through the forced expiry. The p99 stays under your target with no sawtooth on the TTL rhythm. On the cold-cache run, the database load rises and recovers without refused connections.
Keep it healthy
A stampede shows up as a jump in cache misses and database load at the same moment. Scrape Redis with redis_exporter and Postgres with postgres_exporter:
groups:
- name: cache-stampede
rules:
- alert: CacheMissRatioHigh
expr: |
rate(redis_keyspace_misses_total[5m])
/
(rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))
> 0.3
for: 10m
labels:
severity: warning
annotations:
summary: "Redis {{ $labels.instance }} is missing over 30% of reads"
description: "Check for a flush, a restart or heavy eviction with INFO stats. If hot keys expire together, add TTL jitter and coalescing per the cache stampede recipe."
- alert: DatabaseActiveConnectionsJump
expr: |
sum by (instance) (pg_stat_activity_count{state="active"})
>
2 * avg_over_time((sum by (instance) (pg_stat_activity_count{state="active"}))[1h:1m])
and
sum by (instance) (pg_stat_activity_count{state="active"}) > 20
for: 3m
labels:
severity: critical
annotations:
summary: "Active queries on {{ $labels.instance }} doubled against the last hour"
description: "Look for one query repeated many times in pg_stat_activity. If it rebuilds a cache key, add a rebuild lock or early refresh for that key."
Set the 30% from your own baseline hit ratio, since a session cache and a page cache miss at different rates. The > 20 floor keeps the second rule quiet at night, when a few queries double a small number. Add an app counter for rebuilds per key prefix if you can. It names the key that stampeded, which neither exporter can.
Roll back
Ship the change behind a flag, such as an environment variable that turns off coalescing and the lock and goes back to a plain read-through. Flip it and redeploy, or reload config.
If callers time out waiting on a lock whose holder died, roll back now. Then check the lock expiry against the rebuild time. If users see values older than they should, shorten the stale window before you turn the whole fix off. For any other reason, roll back in a quiet hour. A rollback brings the stampede risk back, so don’t do it before a known peak.
More recipes
- Circuit Breaker Pattern for Microservices with Resilience4j
Wrap each dependency in a circuit breaker with Resilience4j or its peers, so a failing service gets cut off fast and your product serves a fallback.
- Exponential Backoff and Jitter for Timeouts and Retries
Give every call a deadline, a per-try timeout and jittered backoff under a retry budget, so a slow dependency can't set off a retry storm.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.