Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Databases and Redis

StackExchange.Redis RedisTimeoutException in a .NET app

ErrorStackExchange.Redis.RedisTimeoutException

Short answer

A RedisTimeoutException means a command did not finish in time. The message itself shows why: qs, in, and the IOCP and WORKER thread counts. Busy above Min points to thread pool starvation, a growing qs to a large value ahead of the queue, and slow commands or a failover point at the server.

  • Applies to StackExchange.Redis 2.x, .NET 8, Azure Cache for Redis
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Copy the full exception message and read three things: qs (sent, awaiting reply), in (bytes not yet read), and Busy against Min in the IOCP and WORKER pools.

    redis-cli --bigkeys
  2. 2Match what you see, then jump to its fix

    1. 1Busy is greater than Min in the IOCP or WORKER part of the messageThread pool starvation on the clientFix
    2. 2qs keeps growing, or SLOWLOG shows commands such as KEYS or large MGETA large value or a slow command blocks the queueFix
    3. 3Timeouts line up with CPU spikes, or a high in: value (bytes waiting to be read)The client host or the cache is out of CPU, memory or bandwidthFix
    4. 4A short burst of timeouts, then everything recovers; the Errors metric shows a Failover countA failover or patch on Azure Cache for RedisFix

What does the error look like?

  • StackExchange.Redis.RedisTimeoutException
  • Timeout performing GET MyKey (1000ms), inst: 2, qs: 6, in: 0
  • IOCP: (Busy=6,Free=994,Min=4,Max=1000), WORKER: (Busy=3,Free=997,Min=4,Max=1000)

Cause 1 of 4

Thread pool starvation on the client

1Check

When the pool has to grow during a burst, data the server already sent waits to be read. The documentation says the most common reason is blocking on async calls, and raising the minimum does not fix that.

grep -R -n -E "\.Result|\.Wait\(\)|GetAwaiter\(\)\.GetResult" --include=*.cs .

2Fix

First remove sync-over-async calls. If threads are still short during bursts, raise the minimum thread counts, starting around 200 to 300 and then tuning.

ThreadPool.SetMinThreads(200, 200);

3Verify

Re-run the load that timed out and read the Busy and Min values in any new message.

dotnet run -c Release

Confirmed by raw.githubusercontent.com

Did this fix it?

Cause 2 of 4

A large value or a slow command blocks the queue

1Check

One TCP connection reads one response at a time, so a big response holds up the small ones behind it. Redis runs commands on one thread, so a slow command blocks the rest.

redis-cli --bigkeys
redis-cli SLOWLOG GET 10

2Fix

Split large values into smaller ones, replace KEYS with SCAN, and use a pool of ConnectionMultiplexer objects if one connection is saturated.

redis-cli SLOWLOG GET 10

3Verify

SLOWLOG should no longer list the slow commands, and qs should stay low.

redis-cli SLOWLOG GET 10

Confirmed by learn.microsoft.com

Did this fix it?

Cause 3 of 4

The client host or the cache is out of CPU, memory or bandwidth

1Check

Keep client CPU under 80 percent. Check system-wide CPU and memory on the client and the server load metric of the cache.

redis-cli INFO cpu

2Fix

Move the client to a larger VM, reduce the data sent per request, or scale the cache up or out.

redis-cli INFO memory

3Verify

Watch server load and client CPU during the same traffic.

redis-cli INFO cpu

Confirmed by learn.microsoft.com

Did this fix it?

Cause 4 of 4

A failover or patch on Azure Cache for Redis

1Check

During a failover, an operation that was sent but not answered can time out. New requests on the closed connection fail until it reconnects.

echo Check Azure portal: cache > Metrics > Errors, split by ErrorType

2Fix

Restart the client applications so all connections are recreated and redistributed between the nodes.

# restart each client app after the failover, for example:
sudo systemctl restart my-app

3Verify

Compare timeout times with the Failover count.

echo Same time window in the Errors chart

Confirmed by learn.microsoft.com

Did this fix it?

How do I stop it coming back?

  • Never block on async Redis calls with .Result or .Wait().
  • Keep values small and avoid KEYS in production.
  • Alert on server load and client CPU.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.