Thread pool starvation in .NET means every thread pool thread is busy waiting, and new work has nowhere to run. ASP.NET Core requests queue, latency climbs into seconds and CPU stays low. The usual cause is sync over async code, such as .Result or .Wait() on a task.
What breaks at peak
ASP.NET Core runs request code on the shared thread pool. An await hands the thread back while I/O is in flight. A blocking call keeps it:
app.MapGet("/orders/{id}", (int id, OrdersRepository repo) =>
{
var order = repo.GetAsync(id).Result;
return Results.Ok(order);
});
Each request now holds a thread for the whole database call. The pool starts with a minimum equal to the processor count. Past that minimum, the runtime adds threads on a delay while work keeps arriving. Microsoft’s starvation guide shows the count jumping to two or three times the cores, then growing by one or two threads per second.
At peak, requests arrive faster than threads appear. Work items queue, every request waits behind them, and CPU sits idle because the threads aren’t working. .NET 6 made thread injection faster for blocking task waits, so modern versions recover sooner. They still starve during a sudden burst.
Kestrel often names the problem for you. Its heartbeat timer can’t get a thread either, and it logs this template:
warn: Microsoft.AspNetCore.Server.Kestrel[22]
As of "{now}", the heartbeat has been running for "{heartbeatDuration}" which is longer than "{interval}". This could be caused by thread pool starvation.
Older ASP.NET Core releases word it Heartbeat took longer than "00:00:01" at ..., with the same last sentence. Other signs show up next to it:
- p99 jumps from milliseconds to seconds while CPU stays well under the limit.
- StackExchange.Redis timeouts whose message shows
WORKER: (Busy=...,Free=...,Min=...,Max=...)with Busy above Min. The client couldn’t get a thread to read the reply. - Liveness probes time out, the kubelet restarts the pod, and the fresh pod starts at the minimum again. Each restart sends it back into starvation.
Small pods starve first. Environment.ProcessorCount follows the container’s CPU limit. A pod limited to one CPU starts with a minimum of one worker thread. A handful of blocked requests starves it. The CPU throttling recipe covers how CPU limits interact with the runtime.
Where the blocking hides. Search for these patterns:
.Result,.Wait()and.GetAwaiter().GetResult()on tasks, often inside constructors, DI factories andIConfigureOptionscode.Task.Run(...).Result, which uses a second thread to block the first.- Synchronous APIs that do I/O, such as
HttpClient.Send,File.ReadAllTexton a network share, or a sync database driver call. lockblocks around I/O, so one slow call holds a lock other threads wait on.- Synchronous request body reads. Kestrel rejects them by default, so look for
AllowSynchronousIO = truein your code.
The sync over async fix in ASP.NET Core
Make the call path async, end to end. This is the real fix. Change the endpoint and everything it calls to await:
app.MapGet("/orders/{id}", async (int id, OrdersRepository repo, CancellationToken ct) =>
{
var order = await repo.GetAsync(id, ct);
return order is null ? Results.NotFound() : Results.Ok(order);
});
The thread goes back to the pool while the query runs, and any free thread picks up the rest. Pass the CancellationToken through, so a request the client abandoned stops using database time.
For code that has to run at startup, move the async work into an IHostedService or do it before builder.Build() with await. A DI factory that blocks on a task runs on a request thread the first time someone resolves it.
Find the blocking calls with a search, then confirm them under load:
grep -rnE --include='*.cs' '\.Result\b|\.Wait\(\)|GetAwaiter\(\)\.GetResult\(\)' .
Not every hit is a bug, since .Result on a task that already finished doesn’t block. Under load, the stacks tell you which ones matter. dotnet-stack prints every managed thread’s stack from a running process:
dotnet-stack report -n Api > stacks.txt
grep -B 2 -A 8 'GetResultCore\|InternalWaitCore\|ManualResetEventSlim.Wait' stacks.txt | less
Threads whose bottom frames show ThreadPoolWorkQueue.Dispatch are pool threads. The frames above the wait name the method that blocked. For waits that come and go, record a trace on .NET 9 and later. It logs every blocking wait with its stack:
dotnet-trace collect -n Api --clrevents waithandle --clreventlevel verbose --duration 00:00:30
The diagnostic tools talk to the runtime over a socket in the container’s /tmp. Install them in a debug image, or add them to a staging image with dotnet tool install. Then run them in the same container as the app.
Raise the minimum as a stopgap. While the async rewrite is in progress, a higher minimum lets the pool create threads at once instead of on a delay. Set it in the project file:
<PropertyGroup>
<ThreadPoolMinThreads>64</ThreadPoolMinThreads>
</PropertyGroup>
Or in runtimeconfig.template.json next to the project file:
{
"configProperties": {
"System.Threading.ThreadPool.MinThreads": 64
}
}
On .NET 10 and later, DOTNET_ThreadPool_ForceMinWorkerThreads sets it without a rebuild. It takes a hexadecimal value, so 0x40 means 64. That’s handy on App Service or Container Apps, where you set it as an app setting.
Size the minimum from the blocking you know about:
min threads = peak requests per second per pod * seconds each request blocks + processor count
Every thread costs memory for its stack and adds context switches, so don’t jump to thousands. I treat a raised minimum as a tourniquet. It buys time while the team removes the blocking calls, and it comes back out once they’re gone.
ThreadPool.SetMinThreads(workerThreads, completionPortThreads) does the same from code at startup. Read the current values with ThreadPool.GetMinThreads first and keep the completion port value unless you have a reason to change it.
Test it on your system
Starvation shows up during bursts, so the test needs a sudden ramp as well as a steady load. Use a fixed arrival rate, so slow responses don’t slow the generator and hide the queue:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
burst: {
executor: 'ramping-arrival-rate',
startRate: 0,
timeUnit: '1s',
preAllocatedVUs: 300,
maxVUs: 3000,
stages: [
{ target: 300, duration: '10s' },
{ target: 300, duration: '10m' },
{ target: 0, duration: '1m' },
{ target: 300, duration: '10s' },
{ target: 300, duration: '5m' },
],
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(99)<500'],
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/orders/${(__ITER % 5000) + 1}`);
check(res, { 'status is 200': (r) => r.status === 200 });
}
Set the target to one pod’s share of your peak. The ten-second ramps copy what a deploy or an autoscaler event sends to a cold pod. Run it on a pod with production’s CPU limit.
While it runs, watch the pool inside the container:
dotnet-counters monitor -n Api --counters System.Runtime
On .NET 9 and later, watch dotnet.thread_pool.queue.length, dotnet.thread_pool.thread.count and dotnet.process.cpu.time. Older versions show the same data under different counter names. Also follow the logs for the heartbeat warning:
kubectl logs -f deploy/api | grep -E 'heartbeat|Heartbeat|thread pool starvation'
Starvation looks like a thread count that climbs step by step while the queue stays long and CPU stays low. Set your pass marks before the run:
- No Kestrel heartbeat warnings during either burst.
- Queue length returns to zero within seconds of each ramp.
- Thread count settles below about three times the processor count. Microsoft’s guide reads a steady count above that as threads blocked in your code.
- p99 and error rate stay inside the k6 thresholds you set.
Run it once before the fix and once after. The async build should hold a low thread count through both bursts.
Keep it healthy
Export the runtime’s built-in metrics with OpenTelemetry and the Prometheus exporter, as in the HttpClient socket exhaustion recipe. Register the System.Runtime meter, which exists on .NET 9 and later. On .NET 8, the OpenTelemetry.Instrumentation.Runtime package publishes similar metrics under other names.
The exporter names the series dotnet_thread_pool_queue_length, dotnet_thread_pool_thread_count, dotnet_process_cpu_time_seconds_total and dotnet_process_cpu_count. Check your /metrics output before you load these rules:
groups:
- name: dotnet-threadpool
rules:
- alert: DotnetThreadPoolStarvation
expr: |
avg_over_time(dotnet_thread_pool_queue_length[5m]) > 10
and on (job, instance)
(
sum by (job, instance) (rate(dotnet_process_cpu_time_seconds_total[5m]))
/ on (job, instance) max by (job, instance) (dotnet_process_cpu_count)
) < 0.5
for: 5m
labels:
severity: critical
annotations:
summary: "Work is queueing in the thread pool on {{ $labels.instance }} while CPU is low"
description: "Threads are blocked, not busy. Run dotnet-stack report on the pod and look for .Result, .Wait() or GetResult frames above ThreadPoolWorkQueue.Dispatch."
- alert: DotnetThreadPoolThreadsHigh
expr: |
max by (job, instance) (dotnet_thread_pool_thread_count)
> on (job, instance) 3 * max by (job, instance) (dotnet_process_cpu_count)
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} runs more than three thread pool threads per core"
description: "The pool is compensating for blocked threads. The next burst or restart will starve it. Find the blocking calls with dotnet-stack and make them async."
Also alert on the shared tail of Kestrel’s heartbeat warning, which matches every version:
This could be caused by thread pool starvation.
Roll back
Code changes roll back through your deploy tool or kubectl rollout undo deployment/api. Keep a copy of the current spec before you deploy: kubectl get deployment api -o yaml > api-before.yaml.
A raised minimum comes out by removing the property and redeploying. If it pushes memory toward the limit or brings OOMKilled restarts, roll it back now and lower the value. If you remove it because the async fix landed, do that in a quiet hour and watch the heartbeat warning through the next peak.
More recipes
- Fix HttpClient Socket Exhaustion in .NET
Reuse connections through IHttpClientFactory or one shared SocketsHttpHandler, so outbound calls stop running out of local ports at peak traffic.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.