Redis Sentinel Failover Without Lost or Refused Writes

Run three Sentinels and make clients ask Sentinel for the primary, so writes stop failing with READONLY You can't write against a read only replica.

Published

Applies to Redis 6.2 and later, Valkey 7.2 and later; Sentinel hostname support needs 6.2 or later

Redis Sentinel failover promotes a replica when the primary dies. Sentinel handles the promotion. Your clients hold connections to the old primary, cache its address and keep writing there. You set up Sentinel, the primary and the client together below, then prove the failover on staging under load.

What happens when the Redis primary dies

Sentinels ping the primary every second. A Sentinel marks it subjectively down (+sdown) once it has gone down-after-milliseconds without a valid reply. The default in sentinel.conf is 30000, so nothing happens for 30 seconds. Once enough Sentinels agree to meet the quorum, the primary becomes objectively down (+odown). The Sentinels elect a leader, the leader promotes the best replica, and every Sentinel publishes +switch-master with the new address.

Your app sees three kinds of failure along the way:

  • Connection errors and timeouts for the whole detection window. Writes fail, and reads fail too unless your client reads from replicas.
  • READONLY You can't write against a read only replica. This comes back when a client writes to a node that is now a replica. It happens when the old primary returns and Sentinel turns it into a replica, while your app still points at its address. A Kubernetes Service that selects every Redis pod causes the same thing on every write that lands on a replica.
  • Silent write loss. Redis replication is asynchronous. Writes the old primary acknowledged but hadn’t sent to a replica are gone after the promotion.

The worst case is a network partition. The old primary sits on one side with some clients, and it keeps accepting their writes. Sentinel promotes a replica on the other side. The partition heals, Sentinel reconfigures the old primary as a replica, and every write it took in the meantime disappears.

Sentinel does close client connections on an instance it reconfigures. It sends CLIENT KILL TYPE normal as part of the change. A client that reconnects to the same address still lands on a replica, though. The fix lives in three places: Sentinel, the primary and the client.

The Redis Sentinel failover setup

Run three Sentinels with a quorum of two. Put them on separate hosts or zones, away from a single failure domain. With two Sentinels, losing one host can leave you without a majority to authorize a failover. This is the full sentinel.conf for one Sentinel:

port 26379
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel monitor mymaster redis-0.redis.cache.svc.cluster.local 6379 2
sentinel auth-pass mymaster change-me
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

Each setting does one job:

  • sentinel monitor ... 2 names the primary and sets the quorum. Two Sentinels must agree the primary is down before a failover starts. The leader also needs votes from a majority of all Sentinels.
  • down-after-milliseconds sets the detection window. A lower value fails over sooner. Set it too low and a GC pause on the host, a slow BGSAVE fork or a busy script triggers a failover nobody needed. I start at 5000 and raise it if Sentinel logs +sdown events that clear without a failover.
  • failover-timeout limits how long one failover attempt may take and how soon Sentinels retry the same primary. The default is 180000.
  • parallel-syncs 1 makes replicas resync with the new primary one at a time, so the rest keep serving reads.
  • resolve-hostnames and announce-hostnames let Sentinel track StatefulSet DNS names. Pod IPs change on every restart, and a Sentinel that tracks IPs ends up with stale entries.
  • auth-pass holds the Redis password. Sentinel rewrites this file at runtime, so mount it writable.

Cap writes on an isolated primary. Add two lines to redis.conf on every Redis node, changed lines only:

min-replicas-to-write 1
min-replicas-max-lag 10

The primary now refuses writes when fewer than one replica has acknowledged it within the last 10 seconds. Clients get NOREPLICAS Not enough good replicas to write. instead of a success that vanishes later. That bounds the partition window to about min-replicas-max-lag seconds. With a single replica, restarting it blocks writes, so run two replicas if you set this.

Make the client ask Sentinel. Your app must connect to the Sentinels, ask for the current primary and ask again after any error. A fixed hostname or a Service in front of all pods breaks that. In Node.js, ioredis does it with a few options:

import Redis from "ioredis";

const redis = new Redis({
  sentinels: [
    { host: "sentinel-0.sentinel.cache.svc.cluster.local", port: 26379 },
    { host: "sentinel-1.sentinel.cache.svc.cluster.local", port: 26379 },
    { host: "sentinel-2.sentinel.cache.svc.cluster.local", port: 26379 },
  ],
  name: "mymaster",
  password: process.env.REDIS_PASSWORD,
  connectTimeout: 2000,
  maxRetriesPerRequest: 3,
  failoverDetector: true,
  reconnectOnError(err) {
    // 2 = reconnect, ask Sentinel for the primary, resend the command
    return err.message.startsWith("READONLY") ? 2 : false;
  },
});

redis.on("error", (err) => console.error("redis error", err.message));

failoverDetector subscribes to +switch-master on the Sentinels and reconnects as soon as a failover finishes. reconnectOnError covers the case it misses: a write that reaches a replica. Resending is safe there, because Redis refused the write. maxRetriesPerRequest keeps commands from queuing forever while the primary is down. Add sentinelPassword if your Sentinels require a password.

In Python, redis-py’s Sentinel class does the same. Its connection turns READONLY into a ConnectionError and drops the connection, so a retry policy finds the new primary:

import os

from redis.backoff import ExponentialBackoff
from redis.exceptions import ConnectionError, TimeoutError
from redis.retry import Retry
from redis.sentinel import Sentinel

sentinel = Sentinel(
    [
        ("sentinel-0.sentinel.cache.svc.cluster.local", 26379),
        ("sentinel-1.sentinel.cache.svc.cluster.local", 26379),
        ("sentinel-2.sentinel.cache.svc.cluster.local", 26379),
    ],
    sentinel_kwargs={"socket_timeout": 0.5},
)

primary = sentinel.master_for(
    "mymaster",
    password=os.environ["REDIS_PASSWORD"],
    socket_timeout=1.0,
    socket_connect_timeout=1.0,
    retry=Retry(ExponentialBackoff(cap=2, base=0.1), 6),
    retry_on_error=[ConnectionError, TimeoutError],
)

Size the retry budget from your detection window. Six retries with a 2-second cap add up to 7 seconds of back-off, plus a connect timeout per attempt. That covers down-after-milliseconds 5000 plus the promotion. Keep it below your request timeout, so a caller gets an error instead of hanging. The timeouts and retries recipe covers how to budget both. In Go, go-redis has redis.NewFailoverClient, and in Java, Jedis has JedisSentinelPool and Lettuce takes a Sentinel RedisURI.

On Kubernetes, run Redis and Sentinel as StatefulSets, each with a PodDisruptionBudget so a node drain can’t take two Sentinels at once. The pod disruption budget recipe shows the manifest. Give Redis a headless Service for the pod DNS names and never a load-balanced Service across all pods for writes.

Managed Redis has no Sentinel. ElastiCache with Multi-AZ promotes a replica and repoints the primary endpoint’s DNS name, then closes connections to the old primary. Azure Cache for Redis on Standard and Premium, and its successor Azure Managed Redis, fail over to a replica behind the same hostname. Memorystore for Redis Standard tier does the same behind its primary endpoint. You don’t tune detection there. You do own the client: reconnect on connection errors, re-resolve DNS on each reconnect and treat READONLY as a reason to reconnect. A JVM that caches DNS forever keeps dialing the old node, so check networkaddress.cache.ttl.

Test the failover

Run this on staging first, with the same Redis version, replica count and Sentinel layout as production. Put your app under its normal peak load while you do it. A failover with no traffic proves only that Sentinel works.

Start a write probe that records every acknowledged value. It asks Sentinel for the primary on each loop, the way a correct client does:

while true; do
  primary=$(redis-cli -h sentinel-0.sentinel.cache.svc.cluster.local -p 26379 \
    SENTINEL GET-MASTER-ADDR-BY-NAME mymaster | head -1)
  if v=$(redis-cli -h "$primary" INCR failover:probe 2>&1); then
    echo "$(date +%T) $primary $v"
  else
    echo "$(date +%T) $primary ERROR $v"
  fi
  sleep 0.1
done | tee probe.log

Set REDISCLI_AUTH for the password. In a second terminal, watch the Sentinel events:

redis-cli -h sentinel-0.sentinel.cache.svc.cluster.local -p 26379 PSUBSCRIBE '*'

Then fail the primary three ways, one at a time, and let the cluster settle between them:

  1. A planned failover. redis-cli -p 26379 SENTINEL FAILOVER mymaster promotes a replica without waiting for agreement. It tests the client path with no detection delay.
  2. A hard kill. Delete the primary pod with kubectl delete pod redis-0 --grace-period=0 --force, or send SIGKILL to redis-server on a VM. This tests detection.
  3. A partition. On the primary’s host, drop traffic from the Sentinels and replicas with iptables, but leave one app instance able to reach it. This tests min-replicas-to-write.

Write your targets down before you start. A clean failover meets all of these:

  • The Sentinel log shows +sdown, +odown, +elected-leader and +switch-master, with no -failover-abort-no-good-slave.
  • The error window in probe.log stays under the budget you set, and your app’s error rate recovers without a restart.
  • After +switch-master, no client logs READONLY.
  • The last value the old primary acknowledged is at or below the first value the new primary returns. A gap means lost writes. In the partition test, the gap stays within min-replicas-max-lag seconds of writes.
  • The old primary rejoins as a replica. redis-cli -h redis-0... ROLE reports slave.

If clients keep logging READONLY, they cache the address. If the error window runs far past down-after-milliseconds, check the retry budget in the client before you lower Sentinel’s timeouts.

Keep it healthy

Scrape every Redis node and every Sentinel with redis_exporter. Point one exporter target at each Sentinel’s port 26379. On a Sentinel, the exporter runs SENTINEL MASTERS and SENTINEL CKQUORUM and exports their results.

groups:
  - name: redis-sentinel
    rules:
      - alert: RedisSentinelPrimaryNotOk
        expr: redis_sentinel_master_status == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Sentinel {{ $labels.instance }} reports {{ $labels.master_name }} as {{ $labels.master_status }}"
          description: "Sentinel sees the primary as down or failing over. Run SENTINEL MASTER on each Sentinel, check the Sentinel log for +odown and -failover-abort-no-good-slave, and confirm a replica is healthy."
      - alert: RedisSentinelQuorumAtRisk
        expr: redis_sentinel_master_ckquorum_status == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Sentinels cannot reach quorum for {{ $labels.master_name }}"
          description: "A failover can't be authorized now. The message label holds the CKQUORUM error. Bring the missing Sentinels back before anything else fails."
      - alert: RedisSentinelFewerThanThree
        expr: redis_sentinel_master_ok_sentinels < 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Only {{ $value }} healthy Sentinels watch {{ $labels.master_name }}"
          description: "One more lost Sentinel can block failover. Check the Sentinel pods and their PodDisruptionBudget, then run SENTINEL RESET mymaster if a Sentinel tracks stale replica addresses."
      - alert: RedisSentinelFailoverHappened
        expr: changes(redis_sentinel_master_config_epoch[15m]) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "A failover changed the primary of {{ $labels.master_name }}"
          description: "Confirm clients recovered without READONLY errors, find the cause of the failover in the old primary's logs, and check that the old primary rejoined as a replica."
      - alert: RedisPrimaryWithoutReplica
        expr: redis_connected_slaves < 1 and on (instance) redis_instance_info{role="master"}
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Redis primary {{ $labels.instance }} has no connected replica"
          description: "No replica can take over, and with min-replicas-to-write set the primary refuses writes. Check replica pods and the replica logs for sync errors."
      - alert: RedisReplicaLinkDown
        expr: redis_master_link_up == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Replica {{ $labels.instance }} lost its link to the primary"
          description: "This replica serves stale reads and can't be promoted safely. Check its log for sync errors and the network path to {{ $labels.master_host }}."
      - alert: RedisReplicaAckLag
        expr: redis_connected_slave_lag_seconds > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replica {{ $labels.slave_ip }} acknowledged the primary over 5 s ago"
          description: "Writes start failing with NOREPLICAS when lag passes min-replicas-max-lag. Check the replica's CPU, network and output buffer limits on the primary."

redis_connected_slave_lag_seconds is the time since the replica’s last acknowledgement, the same clock min-replicas-max-lag reads. Set the 5-second threshold under your min-replicas-max-lag. Also alert on the client side. Count READONLY and NOREPLICAS errors in your app’s error logs. Neither should appear outside a failover test.

Roll back

Save the current settings before you change anything:

redis-cli -p 26379 SENTINEL MASTER mymaster > sentinel-before.txt
redis-cli -h redis-0.redis.cache.svc.cluster.local CONFIG GET 'min-replicas*' > redis-before.txt

Sentinel settings change at runtime on each Sentinel and persist to its config file:

redis-cli -p 26379 SENTINEL SET mymaster down-after-milliseconds 30000
redis-cli -h redis-0.redis.cache.svc.cluster.local CONFIG SET min-replicas-to-write 0
redis-cli -h redis-0.redis.cache.svc.cluster.local CONFIG REWRITE

Run the CONFIG SET on every Redis node. If min-replicas-to-write is refusing writes because a replica is down, roll it back now, then fix the replica. If Sentinel fails over on short pauses, raise down-after-milliseconds right away. Client changes roll back with a normal deploy. Ship those off-peak, since a bad Sentinel address list fails every Redis call from that release.

More recipes

Further reading