Redis Sentinel failover promotes a replica when the primary dies. Sentinel handles the promotion. Your clients hold connections to the old primary, cache its address and keep writing there. You set up Sentinel, the primary and the client together below, then prove the failover on staging under load.
What happens when the Redis primary dies
Sentinels ping the primary every second. A Sentinel marks it subjectively down (+sdown) once it has gone down-after-milliseconds without a valid reply. The default in sentinel.conf is 30000, so nothing happens for 30 seconds. Once enough Sentinels agree to meet the quorum, the primary becomes objectively down (+odown). The Sentinels elect a leader, the leader promotes the best replica, and every Sentinel publishes +switch-master with the new address.
Your app sees three kinds of failure along the way:
- Connection errors and timeouts for the whole detection window. Writes fail, and reads fail too unless your client reads from replicas.
READONLY You can't write against a read only replica.This comes back when a client writes to a node that is now a replica. It happens when the old primary returns and Sentinel turns it into a replica, while your app still points at its address. A Kubernetes Service that selects every Redis pod causes the same thing on every write that lands on a replica.- Silent write loss. Redis replication is asynchronous. Writes the old primary acknowledged but hadn’t sent to a replica are gone after the promotion.
The worst case is a network partition. The old primary sits on one side with some clients, and it keeps accepting their writes. Sentinel promotes a replica on the other side. The partition heals, Sentinel reconfigures the old primary as a replica, and every write it took in the meantime disappears.
Sentinel does close client connections on an instance it reconfigures. It sends CLIENT KILL TYPE normal as part of the change. A client that reconnects to the same address still lands on a replica, though. The fix lives in three places: Sentinel, the primary and the client.
The Redis Sentinel failover setup
Run three Sentinels with a quorum of two. Put them on separate hosts or zones, away from a single failure domain. With two Sentinels, losing one host can leave you without a majority to authorize a failover. This is the full sentinel.conf for one Sentinel:
port 26379
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel monitor mymaster redis-0.redis.cache.svc.cluster.local 6379 2
sentinel auth-pass mymaster change-me
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
Each setting does one job:
sentinel monitor ... 2names the primary and sets the quorum. Two Sentinels must agree the primary is down before a failover starts. The leader also needs votes from a majority of all Sentinels.down-after-millisecondssets the detection window. A lower value fails over sooner. Set it too low and a GC pause on the host, a slowBGSAVEfork or a busy script triggers a failover nobody needed. I start at 5000 and raise it if Sentinel logs+sdownevents that clear without a failover.failover-timeoutlimits how long one failover attempt may take and how soon Sentinels retry the same primary. The default is 180000.parallel-syncs 1makes replicas resync with the new primary one at a time, so the rest keep serving reads.resolve-hostnamesandannounce-hostnameslet Sentinel track StatefulSet DNS names. Pod IPs change on every restart, and a Sentinel that tracks IPs ends up with stale entries.auth-passholds the Redis password. Sentinel rewrites this file at runtime, so mount it writable.
Cap writes on an isolated primary. Add two lines to redis.conf on every Redis node, changed lines only:
min-replicas-to-write 1
min-replicas-max-lag 10
The primary now refuses writes when fewer than one replica has acknowledged it within the last 10 seconds. Clients get NOREPLICAS Not enough good replicas to write. instead of a success that vanishes later. That bounds the partition window to about min-replicas-max-lag seconds. With a single replica, restarting it blocks writes, so run two replicas if you set this.
Make the client ask Sentinel. Your app must connect to the Sentinels, ask for the current primary and ask again after any error. A fixed hostname or a Service in front of all pods breaks that. In Node.js, ioredis does it with a few options:
import Redis from "ioredis";
const redis = new Redis({
sentinels: [
{ host: "sentinel-0.sentinel.cache.svc.cluster.local", port: 26379 },
{ host: "sentinel-1.sentinel.cache.svc.cluster.local", port: 26379 },
{ host: "sentinel-2.sentinel.cache.svc.cluster.local", port: 26379 },
],
name: "mymaster",
password: process.env.REDIS_PASSWORD,
connectTimeout: 2000,
maxRetriesPerRequest: 3,
failoverDetector: true,
reconnectOnError(err) {
// 2 = reconnect, ask Sentinel for the primary, resend the command
return err.message.startsWith("READONLY") ? 2 : false;
},
});
redis.on("error", (err) => console.error("redis error", err.message));
failoverDetector subscribes to +switch-master on the Sentinels and reconnects as soon as a failover finishes. reconnectOnError covers the case it misses: a write that reaches a replica. Resending is safe there, because Redis refused the write. maxRetriesPerRequest keeps commands from queuing forever while the primary is down. Add sentinelPassword if your Sentinels require a password.
In Python, redis-py’s Sentinel class does the same. Its connection turns READONLY into a ConnectionError and drops the connection, so a retry policy finds the new primary:
import os
from redis.backoff import ExponentialBackoff
from redis.exceptions import ConnectionError, TimeoutError
from redis.retry import Retry
from redis.sentinel import Sentinel
sentinel = Sentinel(
[
("sentinel-0.sentinel.cache.svc.cluster.local", 26379),
("sentinel-1.sentinel.cache.svc.cluster.local", 26379),
("sentinel-2.sentinel.cache.svc.cluster.local", 26379),
],
sentinel_kwargs={"socket_timeout": 0.5},
)
primary = sentinel.master_for(
"mymaster",
password=os.environ["REDIS_PASSWORD"],
socket_timeout=1.0,
socket_connect_timeout=1.0,
retry=Retry(ExponentialBackoff(cap=2, base=0.1), 6),
retry_on_error=[ConnectionError, TimeoutError],
)
Size the retry budget from your detection window. Six retries with a 2-second cap add up to 7 seconds of back-off, plus a connect timeout per attempt. That covers down-after-milliseconds 5000 plus the promotion. Keep it below your request timeout, so a caller gets an error instead of hanging. The timeouts and retries recipe covers how to budget both. In Go, go-redis has redis.NewFailoverClient, and in Java, Jedis has JedisSentinelPool and Lettuce takes a Sentinel RedisURI.
On Kubernetes, run Redis and Sentinel as StatefulSets, each with a PodDisruptionBudget so a node drain can’t take two Sentinels at once. The pod disruption budget recipe shows the manifest. Give Redis a headless Service for the pod DNS names and never a load-balanced Service across all pods for writes.
Managed Redis has no Sentinel. ElastiCache with Multi-AZ promotes a replica and repoints the primary endpoint’s DNS name, then closes connections to the old primary. Azure Cache for Redis on Standard and Premium, and its successor Azure Managed Redis, fail over to a replica behind the same hostname. Memorystore for Redis Standard tier does the same behind its primary endpoint. You don’t tune detection there. You do own the client: reconnect on connection errors, re-resolve DNS on each reconnect and treat READONLY as a reason to reconnect. A JVM that caches DNS forever keeps dialing the old node, so check networkaddress.cache.ttl.
Test the failover
Run this on staging first, with the same Redis version, replica count and Sentinel layout as production. Put your app under its normal peak load while you do it. A failover with no traffic proves only that Sentinel works.
Start a write probe that records every acknowledged value. It asks Sentinel for the primary on each loop, the way a correct client does:
while true; do
primary=$(redis-cli -h sentinel-0.sentinel.cache.svc.cluster.local -p 26379 \
SENTINEL GET-MASTER-ADDR-BY-NAME mymaster | head -1)
if v=$(redis-cli -h "$primary" INCR failover:probe 2>&1); then
echo "$(date +%T) $primary $v"
else
echo "$(date +%T) $primary ERROR $v"
fi
sleep 0.1
done | tee probe.log
Set REDISCLI_AUTH for the password. In a second terminal, watch the Sentinel events:
redis-cli -h sentinel-0.sentinel.cache.svc.cluster.local -p 26379 PSUBSCRIBE '*'
Then fail the primary three ways, one at a time, and let the cluster settle between them:
- A planned failover.
redis-cli -p 26379 SENTINEL FAILOVER mymasterpromotes a replica without waiting for agreement. It tests the client path with no detection delay. - A hard kill. Delete the primary pod with
kubectl delete pod redis-0 --grace-period=0 --force, or sendSIGKILLtoredis-serveron a VM. This tests detection. - A partition. On the primary’s host, drop traffic from the Sentinels and replicas with
iptables, but leave one app instance able to reach it. This testsmin-replicas-to-write.
Write your targets down before you start. A clean failover meets all of these:
- The Sentinel log shows
+sdown,+odown,+elected-leaderand+switch-master, with no-failover-abort-no-good-slave. - The error window in
probe.logstays under the budget you set, and your app’s error rate recovers without a restart. - After
+switch-master, no client logsREADONLY. - The last value the old primary acknowledged is at or below the first value the new primary returns. A gap means lost writes. In the partition test, the gap stays within
min-replicas-max-lagseconds of writes. - The old primary rejoins as a replica.
redis-cli -h redis-0... ROLEreportsslave.
If clients keep logging READONLY, they cache the address. If the error window runs far past down-after-milliseconds, check the retry budget in the client before you lower Sentinel’s timeouts.
Keep it healthy
Scrape every Redis node and every Sentinel with redis_exporter. Point one exporter target at each Sentinel’s port 26379. On a Sentinel, the exporter runs SENTINEL MASTERS and SENTINEL CKQUORUM and exports their results.
groups:
- name: redis-sentinel
rules:
- alert: RedisSentinelPrimaryNotOk
expr: redis_sentinel_master_status == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Sentinel {{ $labels.instance }} reports {{ $labels.master_name }} as {{ $labels.master_status }}"
description: "Sentinel sees the primary as down or failing over. Run SENTINEL MASTER on each Sentinel, check the Sentinel log for +odown and -failover-abort-no-good-slave, and confirm a replica is healthy."
- alert: RedisSentinelQuorumAtRisk
expr: redis_sentinel_master_ckquorum_status == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Sentinels cannot reach quorum for {{ $labels.master_name }}"
description: "A failover can't be authorized now. The message label holds the CKQUORUM error. Bring the missing Sentinels back before anything else fails."
- alert: RedisSentinelFewerThanThree
expr: redis_sentinel_master_ok_sentinels < 3
for: 5m
labels:
severity: warning
annotations:
summary: "Only {{ $value }} healthy Sentinels watch {{ $labels.master_name }}"
description: "One more lost Sentinel can block failover. Check the Sentinel pods and their PodDisruptionBudget, then run SENTINEL RESET mymaster if a Sentinel tracks stale replica addresses."
- alert: RedisSentinelFailoverHappened
expr: changes(redis_sentinel_master_config_epoch[15m]) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "A failover changed the primary of {{ $labels.master_name }}"
description: "Confirm clients recovered without READONLY errors, find the cause of the failover in the old primary's logs, and check that the old primary rejoined as a replica."
- alert: RedisPrimaryWithoutReplica
expr: redis_connected_slaves < 1 and on (instance) redis_instance_info{role="master"}
for: 5m
labels:
severity: critical
annotations:
summary: "Redis primary {{ $labels.instance }} has no connected replica"
description: "No replica can take over, and with min-replicas-to-write set the primary refuses writes. Check replica pods and the replica logs for sync errors."
- alert: RedisReplicaLinkDown
expr: redis_master_link_up == 0
for: 2m
labels:
severity: warning
annotations:
summary: "Replica {{ $labels.instance }} lost its link to the primary"
description: "This replica serves stale reads and can't be promoted safely. Check its log for sync errors and the network path to {{ $labels.master_host }}."
- alert: RedisReplicaAckLag
expr: redis_connected_slave_lag_seconds > 5
for: 5m
labels:
severity: warning
annotations:
summary: "Replica {{ $labels.slave_ip }} acknowledged the primary over 5 s ago"
description: "Writes start failing with NOREPLICAS when lag passes min-replicas-max-lag. Check the replica's CPU, network and output buffer limits on the primary."
redis_connected_slave_lag_seconds is the time since the replica’s last acknowledgement, the same clock min-replicas-max-lag reads. Set the 5-second threshold under your min-replicas-max-lag. Also alert on the client side. Count READONLY and NOREPLICAS errors in your app’s error logs. Neither should appear outside a failover test.
Roll back
Save the current settings before you change anything:
redis-cli -p 26379 SENTINEL MASTER mymaster > sentinel-before.txt
redis-cli -h redis-0.redis.cache.svc.cluster.local CONFIG GET 'min-replicas*' > redis-before.txt
Sentinel settings change at runtime on each Sentinel and persist to its config file:
redis-cli -p 26379 SENTINEL SET mymaster down-after-milliseconds 30000
redis-cli -h redis-0.redis.cache.svc.cluster.local CONFIG SET min-replicas-to-write 0
redis-cli -h redis-0.redis.cache.svc.cluster.local CONFIG REWRITE
Run the CONFIG SET on every Redis node. If min-replicas-to-write is refusing writes because a replica is down, roll it back now, then fix the replica. If Sentinel fails over on short pauses, raise down-after-milliseconds right away. Client changes roll back with a normal deploy. Ship those off-peak, since a bad Sentinel address list fails every Redis call from that release.
More recipes
- Find Redis High Latency, Slow Commands and Hot Keys
Alert on per-command p99, slow log growth and a busy main thread, then trace Redis hot keys and big keys, so p99 spikes get a cause before users notice.
- Fix Redis OOM with maxmemory and an Eviction Policy
Cap Redis with maxmemory below its container limit and match the eviction policy to its role, so peak traffic evicts keys or refuses writes without a crash.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.