Find Redis High Latency, Slow Commands and Hot Keys

Alert on per-command p99, slow log growth and a busy main thread, then trace Redis hot keys and big keys, so p99 spikes get a cause before users notice.

Published

Applies to Redis 7.0 and later, Valkey 7.2 and later; redis-cli --keystats needs 7.4, the HOTKEYS command needs Redis 8.6

Redis high latency hides on the default dashboard. Ops per second look normal, average latency stays under a millisecond, and host CPU sits at 30%. Meanwhile one command or one key holds the main thread, and every other client waits behind it. You add the metrics that expose that below, alert on them and prove each alert fires.

What you miss today

Redis runs commands one at a time on its main thread. I/O threads, when you enable them, handle sockets and leave command execution to that one thread. A command that takes 50 ms holds up every client for those 50 ms, whatever it was asking for. Your app sees timeouts on cheap GET calls and blames the network.

Four things cause most of it:

  • Slow commands. KEYS *, HGETALL or SMEMBERS on a collection with millions of members, LRANGE key 0 -1, SORT, and DEL on a big key. A Lua script or function that loops runs to completion on the same thread. Past busy-reply-threshold, 5 seconds by default, other clients get BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
  • Big keys. A hash, set or list that grew without a cap turns every read of the whole thing into a slow command. It also stalls replication and cluster slot migration.
  • Hot keys. One key takes a large share of the traffic, for example a feature flag, a rate-limit counter or a cached homepage. In Redis Cluster that key lives on one shard. That shard’s main thread saturates while the others idle.
  • Pauses outside commands. The fork for BGSAVE or an AOF rewrite, swapping, transparent huge pages and heavy eviction all stall the thread. Disable transparent huge pages on Redis hosts, and see the Redis memory recipe for eviction.

Host CPU hides all four. A 4-core host with the Redis main thread at 100% reports 25% CPU. Your users report it first, as slow pages and client timeout errors.

The exporter and metrics

Run redis_exporter next to each Redis node, or as one deployment with a target per node. Give it a user that may run INFO, SLOWLOG, LATENCY and CONFIG GET. The exporter’s README lists a ready-made ACL.

Turn on the two Redis features the exporter reads. Changed lines only, in redis.conf:

slowlog-log-slower-than 10000
slowlog-max-len 1024
latency-monitor-threshold 50
latency-tracking yes
  • slowlog-log-slower-than is in microseconds. The default of 10000 logs commands slower than 10 ms. Set it to a fraction of your Redis latency budget.
  • slowlog-max-len keeps the last N entries. The default is 128, which a burst of slow commands can overwrite before you read it.
  • latency-monitor-threshold is in milliseconds and defaults to 0, which turns the latency monitor off. Set it near the pause your app notices.
  • latency-tracking defaults to yes from Redis 7. It feeds LATENCY HISTOGRAM and INFO latencystats.

Apply them at runtime with CONFIG SET and persist with CONFIG REWRITE. These metrics matter, each for one question:

MetricWhat it tells you
redis_commands_latencies_usec_bucket{cmd}A histogram of server-side time per command, from LATENCY HISTOGRAM. Use it for p99 per command.
redis_commands_duration_seconds_total{cmd}Time spent per command. Its rate shows which commands eat the main thread.
redis_cpu_user_main_thread_seconds_total and redis_cpu_sys_main_thread_seconds_totalCPU of the main thread alone. A rate near 1 means the thread is saturated.
redis_slowlog_last_idThe ID of the newest slow log entry. Its growth is the slow command rate.
redis_last_slow_execution_duration_secondsHow long the newest slow command took.
redis_latency_spike_duration_seconds{event_name} and redis_latency_spike_last{event_name}The latest spike per event from LATENCY LATEST, such as command, fork or expire-cycle.
redis_commands_processed_totalThroughput per node. Compare shards to spot a hot one.

The histogram measures time inside Redis. Queueing behind a slow command shows up in your client’s latency, not here. Track client-side p99 in your app too, and read the two together. Server p99 low with client p99 high points at the network, the client pool or a single blocked thread on the Redis host.

For big keys, the exporter can export redis_key_size for named keys or patterns with --check-keys. It finds them with SCAN on every scrape, so keep the pattern narrow. On Redis 8.2 and later in cluster mode, --include-cluster-slot-stats exports CPU and network per slot, which points at the hot slot.

For an ad hoc hunt, redis-cli scans the keyspace and samples keys:

redis-cli --bigkeys -i 0.01        # largest collection per type, by element count
redis-cli --memkeys -i 0.01        # largest keys by MEMORY USAGE
redis-cli --keystats -i 0.01       # both, plus size distribution (redis-cli 7.4+)
redis-cli --hotkeys -i 0.01        # needs maxmemory-policy allkeys-lfu or volatile-lfu

-i 0.01 sleeps between SCAN batches so the scan doesn’t add its own latency. On Redis 8.6, the HOTKEYS command tracks the top keys by CPU time or network bytes without changing the eviction policy:

redis-cli HOTKEYS START METRICS 2 CPU NET COUNT 10 DURATION 60
redis-cli HOTKEYS GET
redis-cli HOTKEYS RESET

On older versions without an LFU policy, a few seconds of redis-cli MONITOR piped through sort | uniq -c shows the hottest keys. MONITOR costs throughput while it runs, so keep it short.

Managed Redis. ElastiCache publishes EngineCPUUtilization, which is the main thread’s CPU, and can ship the slow log to CloudWatch Logs or Kinesis Data Firehose. Set slowlog-log-slower-than in a custom parameter group. Azure Cache for Redis reports a Server Load metric and accepts SLOWLOG GET from redis-cli. Memorystore for Redis lets you change only the configs on its supported list, so check it before you plan on the latency monitor. redis_exporter works against all three for the metrics their command set allows, and skips the rest.

The alert rules

groups:
  - name: redis-latency
    rules:
      - alert: RedisCommandP99High
        expr: |
          histogram_quantile(0.99,
            sum by (instance, le) (
              rate(redis_commands_latencies_usec_bucket{cmd!~"info|slowlog.*|latency.*|config.*|client.*"}[5m])
            )
          ) > 2000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Redis {{ $labels.instance }} p99 command time above 2 ms"
          description: "Find the command with the biggest rate of redis_commands_duration_seconds_total, then run SLOWLOG GET 20 on the node. Look for full reads of big keys and scripts that loop."
      - alert: RedisMainThreadSaturated
        expr: |
          rate(redis_cpu_user_main_thread_seconds_total[5m])
            + rate(redis_cpu_sys_main_thread_seconds_total[5m]) > 0.8
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Redis {{ $labels.instance }} main thread above 80% CPU"
          description: "Every client queues behind this thread. Run HOTKEYS or redis-cli --hotkeys to find a hot key, check SLOWLOG GET, and split the key or add read replicas for the reads."
      - alert: RedisSlowCommandsGrowing
        expr: increase(redis_slowlog_last_id[10m]) > 50
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Redis {{ $labels.instance }} logged over 50 slow commands in 10 minutes"
          description: "Run SLOWLOG GET 50 and group the entries by command and key prefix. Replace KEYS with SCAN and cap collections that grow without bound."
      - alert: RedisLatencySpike
        expr: |
          redis_latency_spike_duration_seconds > 0.1
            and (time() - redis_latency_spike_last) < 300
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Redis {{ $labels.instance }} paused over 100 ms on {{ $labels.event_name }}"
          description: "Run LATENCY DOCTOR on the node. A fork event points at BGSAVE or AOF rewrite and transparent huge pages; a command event points at the slow log."
      - alert: RedisHotShard
        expr: |
          (
            sum by (job, instance) (rate(redis_commands_processed_total[5m]))
              and on (instance) redis_instance_info{role="master"}
          )
          > on (job) group_left ()
          (
            2 * avg by (job) (
              sum by (job, instance) (rate(redis_commands_processed_total[5m]))
                and on (instance) redis_instance_info{role="master"}
            )
          )
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Redis primary {{ $labels.instance }} serves over twice the average shard load"
          description: "A hot key or hot slot sits on this shard. Run HOTKEYS on it, or redis-cli --hotkeys with an LFU policy, then cache the key in the app or split it across slots."

RedisHotShard only makes sense for Redis Cluster or client-side sharding, where every primary in one job should carry similar load. Drop it for a single primary.

Test the alerts

Use a staging node with the same version and config, and run the exporter and Prometheus against it as in production.

Slow commands and p99. Build one big hash, then read it whole in a loop:

redis-cli EVAL "for i=1,500000 do redis.call('HSET', KEYS[1], i, i) end return 1" 1 test:bighash
for i in $(seq 1 600); do redis-cli HGETALL test:bighash > /dev/null; done

RedisSlowCommandsGrowing and RedisCommandP99High should fire within their for windows. SLOWLOG GET 5 shows hgetall test:bighash. Delete the key with UNLINK test:bighash, which frees it in the background, and both alerts should clear.

A latency spike. A script that spins holds the thread:

redis-cli EVAL "local i = 0 while i < 300000000 do i = i + 1 end return i" 0

Send a PING from a second terminal while it runs and watch it wait. LATENCY LATEST records a command event, and RedisLatencySpike fires. Size the loop so it stays under busy-reply-threshold, or the test also produces BUSY errors.

A hot key at a fixed rate. memtier_benchmark can send most traffic to a few keys. --key-pattern=G:G draws keys from a Gaussian, and a small --key-stddev narrows it. --rate-limiting caps requests per second per connection, so the load stays at a fixed rate:

memtier_benchmark -s redis-staging -p 6379 --cluster-mode \
  --ratio=1:10 --key-pattern=G:G --key-maximum=1000000 \
  --key-median=500000 --key-stddev=50 \
  --clients=20 --threads=4 --rate-limiting=500 --test-time=1200

On a cluster, RedisHotShard fires for the shard that owns the hot keys. Raise --rate-limiting in steps until that shard’s main thread passes 80% and RedisMainThreadSaturated fires. HOTKEYS GET or redis-cli --hotkeys on that shard should name the keys around 500000.

Check the client side during each test with redis-cli --latency-history -i 5 from an app host. It shows the round trip your app sees. Run redis-cli --intrinsic-latency 30 on the Redis host itself once. If that shows pauses with Redis idle, the host or hypervisor causes them, and no Redis setting fixes it.

An alert passes when it fires within its for window, names the right node and clears once you stop the cause.

Tune the noise

Set thresholds from a week of your own traffic. This query gives the daily worst p99 per node:

max_over_time(
  histogram_quantile(0.99,
    sum by (instance, le) (rate(redis_commands_latencies_usec_bucket[5m]))
  )[1d:5m]
)

Put the RedisCommandP99High threshold above your normal peak and below the point your app’s timeouts start. The 2 ms in the rule is a placeholder. LATENCY HISTOGRAM buckets grow in powers of two, so p99 moves in steps. Don’t chase a threshold finer than a bucket.

Exclude commands you run on purpose. Admin jobs, the exporter itself and nightly scans show up in the histogram and the slow log. The cmd!~ filter drops the exporter’s own commands. Add your batch job’s commands there, or run batch work on a replica.

Match for to how long users tolerate it. Ten minutes suits p99 drift. The main-thread alert pages as critical, since a saturated thread slows every request. Keep its for short enough to page before your error budget burns.

On a cache with LFU eviction, redis-cli --hotkeys always returns some keys. A hot key becomes a problem only when its shard saturates. I alert on shard imbalance and main-thread CPU, and use the hot key tools to explain the alert, not to raise it.

Roll back

The only server change here is the four logging settings. Save them before you set them:

redis-cli CONFIG GET 'slowlog*' > redis-latency-before.txt
redis-cli CONFIG GET 'latency*' >> redis-latency-before.txt

To revert, CONFIG SET each value from the file and run CONFIG REWRITE. If the latency monitor or a long slow log adds load you can measure, revert now. A bigger slowlog-max-len costs a little memory per entry, so trim it if memory is tight.

More recipes

Further reading