StackExchange.Redis.RedisTimeoutException means a command didn’t get a reply within the client’s timeout, 5,000 ms by default. On Azure Cache for Redis at peak, the cause sits on the client more often than on the server. The exception message carries the numbers that tell you which side to fix.
What breaks at peak
A synchronous call fails with a message in this shape. This one is illustrative, and the fields after the command vary by library version and settings:
StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET session:8f2c, inst: 0, qu: 0, qs: 41, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 65536, serverEndpoint: app-cache.redis.cache.windows.net:6380, mc: 1/1/0, clientName: web-7d9f, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=58,Free=32709,Min=4,Max=32767)
An async call on recent versions can fail with a shorter message:
Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5984ms elapsed, timeout is 5000ms)
Your users get slow pages, then errors, on every path that reads the cache. A session store or a rate limiter on Redis can take login and checkout down with it.
Read the message before you change anything. The StackExchange.Redis docs explain each field. These decide the fix:
qscounts commands sent and still waiting for a reply. A highqswith a highinmeans replies have arrived at your host and your app isn’t reading them.inis bytes sitting in the socket, unread. A large value points at the client, not the server.qucounts commands queued to write. A highqupoints at a write backlog, often bandwidth or a reconnect.IOCPandWORKERshow the .NET thread pool.BusyaboveMinmeans the pool had to grow, and .NET adds threads above the minimum at a slow, fixed pace. Your Redis replies wait in line for a thread.rsis the reader state. If it often reportsCompletePendingMessage, some code is running long work on the reader’s thread.
The message above shows the classic peak pattern. 58 busy worker threads against a minimum of 4, 41 commands waiting and 64 KB unread. The server answered on time. The app had no thread free to read the answer.
Causes on the client:
- Thread pool starvation. Code that blocks on async calls with
.Resultor.Wait()holds threads while it waits. At peak, the pool runs dry and Redis replies queue up behind it. The .NET thread pool starvation recipe covers finding and fixing those calls. - A new
ConnectionMultiplexerper request. Each one opens connections and runs a handshake. Under load you get connection storms, highconnectedclientsand timeouts during connect. - Large values. StackExchange.Redis sends commands down one connection and reads replies in order. A 2 MB value delays every small
GETqueued behind it. - Client CPU above 80%. Microsoft’s guidance is to keep client CPU below that line, measured for the whole host, not one process.
Causes on the server:
- High server load. Redis runs commands on one thread. Long commands such as
KEYS *, bigMGETcalls or heavy Lua scripts block everything behind them. - Bandwidth. Each cache size has a network limit. Large values at peak can saturate it.
- Maintenance and failover. Azure patches caches and fails over between nodes. Commands in flight during a failover time out.
The fix
One multiplexer per process. Create the ConnectionMultiplexer once and share it. In ASP.NET Core, register it as a singleton:
using StackExchange.Redis;
var builder = WebApplication.CreateBuilder(args);
var options = ConfigurationOptions.Parse(builder.Configuration["Redis:ConnectionString"]!);
options.AbortOnConnectFail = false;
options.ConnectRetry = 3;
options.SyncTimeout = 5000;
options.AsyncTimeout = 5000;
options.ConnectTimeout = 10000;
builder.Services.AddSingleton<IConnectionMultiplexer>(
_ => ConnectionMultiplexer.Connect(options));
var app = builder.Build();
app.MapGet("/session/{id}", async (string id, IConnectionMultiplexer redis) =>
{
var value = await redis.GetDatabase().StringGetAsync($"session:{id}");
return value.IsNullOrEmpty ? Results.NotFound() : Results.Text(value.ToString());
});
app.Run();
Each option, and what goes wrong when it’s off:
AbortOnConnectFail = falsekeeps the multiplexer retrying in the background if the cache can’t be reached for a moment at startup. The library already defaults tofalsefor Azure endpoints. Set it in code so the behavior doesn’t depend on the host name.ConnectRetrysets how many times the first connect retries. Too low, and one failed handshake during a failover breaks startup.SyncTimeoutandAsyncTimeoutare the command timeouts. Both default to 5,000 ms, andAsyncTimeoutfollowsSyncTimeoutunless you set it. Raising them hides starvation instead of fixing it. Set them to what your request budget can afford, and fail fast below it.ConnectTimeoutlimits each connect attempt. A TLS handshake to a busy cache can take longer than a command, so give it more room.
Call the async methods all the way through. StringGetAsync with await frees the thread while the reply is in flight. StringGet blocks it.
Raise the thread pool minimum, after you fix blocking code. .NET creates threads up to the minimum on demand, then adds them one at a time with a delay. A burst that needs more threads than the minimum waits on that ramp. Set the minimum in the project file:
<PropertyGroup>
<ThreadPoolMinThreads>200</ThreadPoolMinThreads>
</PropertyGroup>
There’s no universal value. Start near the Busy count you see in timeout messages at peak, then let the load test move it. A higher minimum buys time during a burst. If your code blocks on async calls, it will use up any minimum you set.
On StackExchange.Redis 2.x the library ran its own thread pool for async completions, shown as mgr in the message. Version 3.0 removed it, so on 3.x the global thread pool numbers are the ones that count.
Keep values small and commands short. Find big keys and slow commands from a host in the same region:
redis-cli -h app-cache.redis.cache.windows.net -p 6380 --tls -a "$REDIS_KEY" --bigkeys
redis-cli -h app-cache.redis.cache.windows.net -p 6380 --tls -a "$REDIS_KEY" SLOWLOG GET 20
Split large values into smaller keys or compress them. Replace KEYS with SCAN. The Redis latency and hot keys recipe covers hot keys and slow commands in depth. If large values are part of the design, a second multiplexer for the bulk traffic stops one large reply from delaying session reads.
Size the cache from its metrics. If server load stays high after the client fixes, scale up to a size with more vCPUs. On a clustered tier you can add shards instead. Check Cache Read against your size’s bandwidth too. If connectedclients climbs at peak, look for a multiplexer created per request before you buy a bigger cache.
Managed service notes. On Azure Cache for Redis you change memory policy and maxmemory-reserved through the portal or ARM, and Microsoft locks most other Redis settings. You can’t change the server’s own timeout or thread settings, so the fixes above live in your client. Microsoft has announced the retirement of Azure Cache for Redis in favor of Azure Managed Redis. The client advice is the same on both, and on Amazon ElastiCache and Google Memorystore too. Set the cache’s Schedule updates window outside your peak hours. If your cache runs out of memory rather than time, the Redis maxmemory recipe covers eviction.
Test it on your system
First, check the cache itself from a VM in the same region. redis-benchmark waits for each reply before it sends the next, so treat it as a server capacity check, not a latency test:
redis-benchmark -h app-cache.redis.cache.windows.net -p 6380 --tls -a "$REDIS_KEY" \
-c 50 -n 1000000 -d 1024 -t get,set -q
Then load test the app, since most of these timeouts start in the client. Drive your heaviest Redis-backed endpoint at a fixed arrival rate:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: 800, // requests per second; set this to your peak
timeUnit: '1s',
duration: '15m',
preAllocatedVUs: 400,
maxVUs: 2000,
},
},
thresholds: {
http_req_failed: ['rate<0.001'],
http_req_duration: ['p(99)<300'],
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/session/${Math.floor(Math.random() * 50000)}`);
check(res, { 'not a 5xx': (r) => r.status < 500 });
}
k6 run -e BASE_URL=https://staging.example.com redis-peak.js
While it runs, watch:
- Your app logs for
RedisTimeoutExceptionandTimeout awaiting response. Save a few full messages and compareBusywithMin. dotnet-counters monitor -n YourApp System.Runtimeon one instance, for thread pool thread count and queue length.Server Load,Connected Clients,Cache ReadandErrorson the cache.- Host CPU on the app instances, not process CPU.
Write your targets down first. It passes when no timeouts appear and the thread pool queue drains between bursts. Server load should stay below the ceiling you set, and p99 should meet your budget. Run a failover in staging too, with Reboot on the primary node in the portal, and check that the app recovers without a restart.
Keep it healthy
azure-metrics-exporter from webdevops brings the cache metrics into Prometheus:
scrape_configs:
- job_name: azure-redis
scrape_interval: 1m
scrape_timeout: 55s
metrics_path: /probe/metrics/list
params:
name: ["azure_redis"]
subscription: ["00000000-0000-0000-0000-000000000000"]
resourceType: ["Microsoft.Cache/redis"]
metric: ["serverLoad", "usedmemorypercentage", "connectedclients"]
aggregation: ["maximum"]
interval: ["PT5M"]
timespan: ["PT5M"]
static_configs:
- targets: ["azure-metrics-exporter:8080"]
- job_name: azure-redis-errors
scrape_interval: 1m
scrape_timeout: 55s
metrics_path: /probe/metrics/list
params:
name: ["azure_redis_errors"]
subscription: ["00000000-0000-0000-0000-000000000000"]
resourceType: ["Microsoft.Cache/redis"]
metric: ["errors"]
aggregation: ["maximum"]
interval: ["PT5M"]
timespan: ["PT5M"]
metricFilter: ["ErrorType eq '*'"]
static_configs:
- targets: ["azure-metrics-exporter:8080"]
Count timeouts in the app as well, since the cache can’t see them. With prometheus-net, wrap the call site:
using Prometheus;
using StackExchange.Redis;
public static class RedisMetrics
{
public static readonly Counter Timeouts = Metrics.CreateCounter(
"app_redis_timeouts_total", "RedisTimeoutException count from StackExchange.Redis");
}
// at the call site
try
{
value = await db.StringGetAsync(key);
}
catch (RedisTimeoutException)
{
RedisMetrics.Timeouts.Inc();
throw;
}
groups:
- name: azure-redis
rules:
- alert: RedisClientTimeouts
expr: sum by (job) (rate(app_redis_timeouts_total[5m])) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} is getting RedisTimeoutException"
description: "Pull a full timeout message from the app logs. If WORKER Busy is above Min, fix blocking calls and the thread pool minimum; if qs and in are low, check server load and failover events on the cache."
- alert: AzureRedisServerLoadHigh
expr: azure_redis{metric="serverLoad",aggregation="maximum"} > 80
for: 10m
labels:
severity: warning
annotations:
summary: "Azure Redis {{ $labels.resourceName }} server load is above 80%"
description: "Run SLOWLOG GET 20 and redis-cli --bigkeys to find expensive commands. If load stays high after that, scale up or add shards before the next peak."
- alert: AzureRedisUnresponsiveClients
expr: azure_redis_errors{metric="errors",dimension="UnresponsiveClients"} > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Azure Redis {{ $labels.resourceName }} reports clients that can't keep up"
description: "Client hosts aren't reading replies fast enough. Check client CPU and thread pool counters on the app instances, then the timeout messages for high in values."
- alert: AzureRedisFailover
expr: azure_redis_errors{metric="errors",dimension="Failover"} > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Azure Redis {{ $labels.resourceName }} failed over"
description: "Expect a short burst of timeouts. If they don't stop within a few minutes, restart the app instances so connections spread across both nodes again, and check the update schedule."
Set the server load line from your own baseline. 80 is a starting point, not a limit Azure enforces.
Roll back
Keep the client changes in one commit and save the cache’s settings first:
az redis show -g web-prod -n app-cache > redis-before.json
The multiplexer and async changes fix bugs, so you seldom need to undo them. The thread pool minimum is the setting to watch. A very high minimum can raise memory use and context switching on small instances. If timeouts or CPU get worse after a change, revert that commit and redeploy now. Scaling the cache down or changing its tier causes a brief disruption on Azure, so schedule that revert off-peak.
More recipes
- Fix Application Gateway 502 Bad Gateway at Peak
Give Application Gateway a cheap health probe and a request timeout sized from your p99, so busy backends stay in the pool and peak traffic stops getting 502.
- Azure SNAT Port Exhaustion Alerts with Prometheus
Alert on failed SNAT connections and used ports per instance, so you catch App Service intermittent outbound connection errors before users report them.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.