Fix StackExchange.Redis RedisTimeoutException on Azure

Read each Azure Cache for Redis timeout message, then fix the thread pool, the multiplexer or the server load it points to, so peak traffic stops timing out.

Published

Applies to StackExchange.Redis 2.x and 3.x on .NET 6 and later; Azure Cache for Redis and Azure Managed Redis; azure-metrics-exporter 24.9 and later

StackExchange.Redis.RedisTimeoutException means a command didn’t get a reply within the client’s timeout, 5,000 ms by default. On Azure Cache for Redis at peak, the cause sits on the client more often than on the server. The exception message carries the numbers that tell you which side to fix.

What breaks at peak

A synchronous call fails with a message in this shape. This one is illustrative, and the fields after the command vary by library version and settings:

StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET session:8f2c, inst: 0, qu: 0, qs: 41, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 65536, serverEndpoint: app-cache.redis.cache.windows.net:6380, mc: 1/1/0, clientName: web-7d9f, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=58,Free=32709,Min=4,Max=32767)

An async call on recent versions can fail with a shorter message:

Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5984ms elapsed, timeout is 5000ms)

Your users get slow pages, then errors, on every path that reads the cache. A session store or a rate limiter on Redis can take login and checkout down with it.

Read the message before you change anything. The StackExchange.Redis docs explain each field. These decide the fix:

  • qs counts commands sent and still waiting for a reply. A high qs with a high in means replies have arrived at your host and your app isn’t reading them.
  • in is bytes sitting in the socket, unread. A large value points at the client, not the server.
  • qu counts commands queued to write. A high qu points at a write backlog, often bandwidth or a reconnect.
  • IOCP and WORKER show the .NET thread pool. Busy above Min means the pool had to grow, and .NET adds threads above the minimum at a slow, fixed pace. Your Redis replies wait in line for a thread.
  • rs is the reader state. If it often reports CompletePendingMessage, some code is running long work on the reader’s thread.

The message above shows the classic peak pattern. 58 busy worker threads against a minimum of 4, 41 commands waiting and 64 KB unread. The server answered on time. The app had no thread free to read the answer.

Causes on the client:

  • Thread pool starvation. Code that blocks on async calls with .Result or .Wait() holds threads while it waits. At peak, the pool runs dry and Redis replies queue up behind it. The .NET thread pool starvation recipe covers finding and fixing those calls.
  • A new ConnectionMultiplexer per request. Each one opens connections and runs a handshake. Under load you get connection storms, high connectedclients and timeouts during connect.
  • Large values. StackExchange.Redis sends commands down one connection and reads replies in order. A 2 MB value delays every small GET queued behind it.
  • Client CPU above 80%. Microsoft’s guidance is to keep client CPU below that line, measured for the whole host, not one process.

Causes on the server:

  • High server load. Redis runs commands on one thread. Long commands such as KEYS *, big MGET calls or heavy Lua scripts block everything behind them.
  • Bandwidth. Each cache size has a network limit. Large values at peak can saturate it.
  • Maintenance and failover. Azure patches caches and fails over between nodes. Commands in flight during a failover time out.

The fix

One multiplexer per process. Create the ConnectionMultiplexer once and share it. In ASP.NET Core, register it as a singleton:

using StackExchange.Redis;

var builder = WebApplication.CreateBuilder(args);

var options = ConfigurationOptions.Parse(builder.Configuration["Redis:ConnectionString"]!);
options.AbortOnConnectFail = false;
options.ConnectRetry = 3;
options.SyncTimeout = 5000;
options.AsyncTimeout = 5000;
options.ConnectTimeout = 10000;

builder.Services.AddSingleton<IConnectionMultiplexer>(
    _ => ConnectionMultiplexer.Connect(options));

var app = builder.Build();

app.MapGet("/session/{id}", async (string id, IConnectionMultiplexer redis) =>
{
    var value = await redis.GetDatabase().StringGetAsync($"session:{id}");
    return value.IsNullOrEmpty ? Results.NotFound() : Results.Text(value.ToString());
});

app.Run();

Each option, and what goes wrong when it’s off:

  • AbortOnConnectFail = false keeps the multiplexer retrying in the background if the cache can’t be reached for a moment at startup. The library already defaults to false for Azure endpoints. Set it in code so the behavior doesn’t depend on the host name.
  • ConnectRetry sets how many times the first connect retries. Too low, and one failed handshake during a failover breaks startup.
  • SyncTimeout and AsyncTimeout are the command timeouts. Both default to 5,000 ms, and AsyncTimeout follows SyncTimeout unless you set it. Raising them hides starvation instead of fixing it. Set them to what your request budget can afford, and fail fast below it.
  • ConnectTimeout limits each connect attempt. A TLS handshake to a busy cache can take longer than a command, so give it more room.

Call the async methods all the way through. StringGetAsync with await frees the thread while the reply is in flight. StringGet blocks it.

Raise the thread pool minimum, after you fix blocking code. .NET creates threads up to the minimum on demand, then adds them one at a time with a delay. A burst that needs more threads than the minimum waits on that ramp. Set the minimum in the project file:

<PropertyGroup>
  <ThreadPoolMinThreads>200</ThreadPoolMinThreads>
</PropertyGroup>

There’s no universal value. Start near the Busy count you see in timeout messages at peak, then let the load test move it. A higher minimum buys time during a burst. If your code blocks on async calls, it will use up any minimum you set.

On StackExchange.Redis 2.x the library ran its own thread pool for async completions, shown as mgr in the message. Version 3.0 removed it, so on 3.x the global thread pool numbers are the ones that count.

Keep values small and commands short. Find big keys and slow commands from a host in the same region:

redis-cli -h app-cache.redis.cache.windows.net -p 6380 --tls -a "$REDIS_KEY" --bigkeys
redis-cli -h app-cache.redis.cache.windows.net -p 6380 --tls -a "$REDIS_KEY" SLOWLOG GET 20

Split large values into smaller keys or compress them. Replace KEYS with SCAN. The Redis latency and hot keys recipe covers hot keys and slow commands in depth. If large values are part of the design, a second multiplexer for the bulk traffic stops one large reply from delaying session reads.

Size the cache from its metrics. If server load stays high after the client fixes, scale up to a size with more vCPUs. On a clustered tier you can add shards instead. Check Cache Read against your size’s bandwidth too. If connectedclients climbs at peak, look for a multiplexer created per request before you buy a bigger cache.

Managed service notes. On Azure Cache for Redis you change memory policy and maxmemory-reserved through the portal or ARM, and Microsoft locks most other Redis settings. You can’t change the server’s own timeout or thread settings, so the fixes above live in your client. Microsoft has announced the retirement of Azure Cache for Redis in favor of Azure Managed Redis. The client advice is the same on both, and on Amazon ElastiCache and Google Memorystore too. Set the cache’s Schedule updates window outside your peak hours. If your cache runs out of memory rather than time, the Redis maxmemory recipe covers eviction.

Test it on your system

First, check the cache itself from a VM in the same region. redis-benchmark waits for each reply before it sends the next, so treat it as a server capacity check, not a latency test:

redis-benchmark -h app-cache.redis.cache.windows.net -p 6380 --tls -a "$REDIS_KEY" \
  -c 50 -n 1000000 -d 1024 -t get,set -q

Then load test the app, since most of these timeouts start in the client. Drive your heaviest Redis-backed endpoint at a fixed arrival rate:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: 800,            // requests per second; set this to your peak
      timeUnit: '1s',
      duration: '15m',
      preAllocatedVUs: 400,
      maxVUs: 2000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(99)<300'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/session/${Math.floor(Math.random() * 50000)}`);
  check(res, { 'not a 5xx': (r) => r.status < 500 });
}
k6 run -e BASE_URL=https://staging.example.com redis-peak.js

While it runs, watch:

  • Your app logs for RedisTimeoutException and Timeout awaiting response. Save a few full messages and compare Busy with Min.
  • dotnet-counters monitor -n YourApp System.Runtime on one instance, for thread pool thread count and queue length.
  • Server Load, Connected Clients, Cache Read and Errors on the cache.
  • Host CPU on the app instances, not process CPU.

Write your targets down first. It passes when no timeouts appear and the thread pool queue drains between bursts. Server load should stay below the ceiling you set, and p99 should meet your budget. Run a failover in staging too, with Reboot on the primary node in the portal, and check that the app recovers without a restart.

Keep it healthy

azure-metrics-exporter from webdevops brings the cache metrics into Prometheus:

scrape_configs:
  - job_name: azure-redis
    scrape_interval: 1m
    scrape_timeout: 55s
    metrics_path: /probe/metrics/list
    params:
      name: ["azure_redis"]
      subscription: ["00000000-0000-0000-0000-000000000000"]
      resourceType: ["Microsoft.Cache/redis"]
      metric: ["serverLoad", "usedmemorypercentage", "connectedclients"]
      aggregation: ["maximum"]
      interval: ["PT5M"]
      timespan: ["PT5M"]
    static_configs:
      - targets: ["azure-metrics-exporter:8080"]
  - job_name: azure-redis-errors
    scrape_interval: 1m
    scrape_timeout: 55s
    metrics_path: /probe/metrics/list
    params:
      name: ["azure_redis_errors"]
      subscription: ["00000000-0000-0000-0000-000000000000"]
      resourceType: ["Microsoft.Cache/redis"]
      metric: ["errors"]
      aggregation: ["maximum"]
      interval: ["PT5M"]
      timespan: ["PT5M"]
      metricFilter: ["ErrorType eq '*'"]
    static_configs:
      - targets: ["azure-metrics-exporter:8080"]

Count timeouts in the app as well, since the cache can’t see them. With prometheus-net, wrap the call site:

using Prometheus;
using StackExchange.Redis;

public static class RedisMetrics
{
    public static readonly Counter Timeouts = Metrics.CreateCounter(
        "app_redis_timeouts_total", "RedisTimeoutException count from StackExchange.Redis");
}

// at the call site
try
{
    value = await db.StringGetAsync(key);
}
catch (RedisTimeoutException)
{
    RedisMetrics.Timeouts.Inc();
    throw;
}
groups:
  - name: azure-redis
    rules:
      - alert: RedisClientTimeouts
        expr: sum by (job) (rate(app_redis_timeouts_total[5m])) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} is getting RedisTimeoutException"
          description: "Pull a full timeout message from the app logs. If WORKER Busy is above Min, fix blocking calls and the thread pool minimum; if qs and in are low, check server load and failover events on the cache."
      - alert: AzureRedisServerLoadHigh
        expr: azure_redis{metric="serverLoad",aggregation="maximum"} > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Azure Redis {{ $labels.resourceName }} server load is above 80%"
          description: "Run SLOWLOG GET 20 and redis-cli --bigkeys to find expensive commands. If load stays high after that, scale up or add shards before the next peak."
      - alert: AzureRedisUnresponsiveClients
        expr: azure_redis_errors{metric="errors",dimension="UnresponsiveClients"} > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Azure Redis {{ $labels.resourceName }} reports clients that can't keep up"
          description: "Client hosts aren't reading replies fast enough. Check client CPU and thread pool counters on the app instances, then the timeout messages for high in values."
      - alert: AzureRedisFailover
        expr: azure_redis_errors{metric="errors",dimension="Failover"} > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Azure Redis {{ $labels.resourceName }} failed over"
          description: "Expect a short burst of timeouts. If they don't stop within a few minutes, restart the app instances so connections spread across both nodes again, and check the update schedule."

Set the server load line from your own baseline. 80 is a starting point, not a limit Azure enforces.

Roll back

Keep the client changes in one commit and save the cache’s settings first:

az redis show -g web-prod -n app-cache > redis-before.json

The multiplexer and async changes fix bugs, so you seldom need to undo them. The thread pool minimum is the setting to watch. A very high minimum can raise memory use and context switching on small instances. If timeouts or CPU get worse after a change, revert that commit and redeploy now. Scaling the cache down or changing its tier causes a brief disruption on Azure, so schedule that revert off-peak.

More recipes

Further reading