Fix Thread Pool Starvation in .NET and ASP.NET Core

Remove sync over async calls in ASP.NET Core and set a thread pool minimum, so requests stop queueing behind blocked threads while CPU sits idle.

Published

Applies to .NET 6 and later; the built-in thread pool metrics need .NET 9, and the WaitHandleWait trace event needs .NET 9

Thread pool starvation in .NET means every thread pool thread is busy waiting, and new work has nowhere to run. ASP.NET Core requests queue, latency climbs into seconds and CPU stays low. The usual cause is sync over async code, such as .Result or .Wait() on a task.

What breaks at peak

ASP.NET Core runs request code on the shared thread pool. An await hands the thread back while I/O is in flight. A blocking call keeps it:

app.MapGet("/orders/{id}", (int id, OrdersRepository repo) =>
{
    var order = repo.GetAsync(id).Result;
    return Results.Ok(order);
});

Each request now holds a thread for the whole database call. The pool starts with a minimum equal to the processor count. Past that minimum, the runtime adds threads on a delay while work keeps arriving. Microsoft’s starvation guide shows the count jumping to two or three times the cores, then growing by one or two threads per second.

At peak, requests arrive faster than threads appear. Work items queue, every request waits behind them, and CPU sits idle because the threads aren’t working. .NET 6 made thread injection faster for blocking task waits, so modern versions recover sooner. They still starve during a sudden burst.

Kestrel often names the problem for you. Its heartbeat timer can’t get a thread either, and it logs this template:

warn: Microsoft.AspNetCore.Server.Kestrel[22]
      As of "{now}", the heartbeat has been running for "{heartbeatDuration}" which is longer than "{interval}". This could be caused by thread pool starvation.

Older ASP.NET Core releases word it Heartbeat took longer than "00:00:01" at ..., with the same last sentence. Other signs show up next to it:

  • p99 jumps from milliseconds to seconds while CPU stays well under the limit.
  • StackExchange.Redis timeouts whose message shows WORKER: (Busy=...,Free=...,Min=...,Max=...) with Busy above Min. The client couldn’t get a thread to read the reply.
  • Liveness probes time out, the kubelet restarts the pod, and the fresh pod starts at the minimum again. Each restart sends it back into starvation.

Small pods starve first. Environment.ProcessorCount follows the container’s CPU limit. A pod limited to one CPU starts with a minimum of one worker thread. A handful of blocked requests starves it. The CPU throttling recipe covers how CPU limits interact with the runtime.

Where the blocking hides. Search for these patterns:

  • .Result, .Wait() and .GetAwaiter().GetResult() on tasks, often inside constructors, DI factories and IConfigureOptions code.
  • Task.Run(...).Result, which uses a second thread to block the first.
  • Synchronous APIs that do I/O, such as HttpClient.Send, File.ReadAllText on a network share, or a sync database driver call.
  • lock blocks around I/O, so one slow call holds a lock other threads wait on.
  • Synchronous request body reads. Kestrel rejects them by default, so look for AllowSynchronousIO = true in your code.

The sync over async fix in ASP.NET Core

Make the call path async, end to end. This is the real fix. Change the endpoint and everything it calls to await:

app.MapGet("/orders/{id}", async (int id, OrdersRepository repo, CancellationToken ct) =>
{
    var order = await repo.GetAsync(id, ct);
    return order is null ? Results.NotFound() : Results.Ok(order);
});

The thread goes back to the pool while the query runs, and any free thread picks up the rest. Pass the CancellationToken through, so a request the client abandoned stops using database time.

For code that has to run at startup, move the async work into an IHostedService or do it before builder.Build() with await. A DI factory that blocks on a task runs on a request thread the first time someone resolves it.

Find the blocking calls with a search, then confirm them under load:

grep -rnE --include='*.cs' '\.Result\b|\.Wait\(\)|GetAwaiter\(\)\.GetResult\(\)' .

Not every hit is a bug, since .Result on a task that already finished doesn’t block. Under load, the stacks tell you which ones matter. dotnet-stack prints every managed thread’s stack from a running process:

dotnet-stack report -n Api > stacks.txt
grep -B 2 -A 8 'GetResultCore\|InternalWaitCore\|ManualResetEventSlim.Wait' stacks.txt | less

Threads whose bottom frames show ThreadPoolWorkQueue.Dispatch are pool threads. The frames above the wait name the method that blocked. For waits that come and go, record a trace on .NET 9 and later. It logs every blocking wait with its stack:

dotnet-trace collect -n Api --clrevents waithandle --clreventlevel verbose --duration 00:00:30

The diagnostic tools talk to the runtime over a socket in the container’s /tmp. Install them in a debug image, or add them to a staging image with dotnet tool install. Then run them in the same container as the app.

Raise the minimum as a stopgap. While the async rewrite is in progress, a higher minimum lets the pool create threads at once instead of on a delay. Set it in the project file:

<PropertyGroup>
  <ThreadPoolMinThreads>64</ThreadPoolMinThreads>
</PropertyGroup>

Or in runtimeconfig.template.json next to the project file:

{
  "configProperties": {
    "System.Threading.ThreadPool.MinThreads": 64
  }
}

On .NET 10 and later, DOTNET_ThreadPool_ForceMinWorkerThreads sets it without a rebuild. It takes a hexadecimal value, so 0x40 means 64. That’s handy on App Service or Container Apps, where you set it as an app setting.

Size the minimum from the blocking you know about:

min threads = peak requests per second per pod * seconds each request blocks + processor count

Every thread costs memory for its stack and adds context switches, so don’t jump to thousands. I treat a raised minimum as a tourniquet. It buys time while the team removes the blocking calls, and it comes back out once they’re gone.

ThreadPool.SetMinThreads(workerThreads, completionPortThreads) does the same from code at startup. Read the current values with ThreadPool.GetMinThreads first and keep the completion port value unless you have a reason to change it.

Test it on your system

Starvation shows up during bursts, so the test needs a sudden ramp as well as a steady load. Use a fixed arrival rate, so slow responses don’t slow the generator and hide the queue:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    burst: {
      executor: 'ramping-arrival-rate',
      startRate: 0,
      timeUnit: '1s',
      preAllocatedVUs: 300,
      maxVUs: 3000,
      stages: [
        { target: 300, duration: '10s' },
        { target: 300, duration: '10m' },
        { target: 0, duration: '1m' },
        { target: 300, duration: '10s' },
        { target: 300, duration: '5m' },
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<500'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/orders/${(__ITER % 5000) + 1}`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}

Set the target to one pod’s share of your peak. The ten-second ramps copy what a deploy or an autoscaler event sends to a cold pod. Run it on a pod with production’s CPU limit.

While it runs, watch the pool inside the container:

dotnet-counters monitor -n Api --counters System.Runtime

On .NET 9 and later, watch dotnet.thread_pool.queue.length, dotnet.thread_pool.thread.count and dotnet.process.cpu.time. Older versions show the same data under different counter names. Also follow the logs for the heartbeat warning:

kubectl logs -f deploy/api | grep -E 'heartbeat|Heartbeat|thread pool starvation'

Starvation looks like a thread count that climbs step by step while the queue stays long and CPU stays low. Set your pass marks before the run:

  • No Kestrel heartbeat warnings during either burst.
  • Queue length returns to zero within seconds of each ramp.
  • Thread count settles below about three times the processor count. Microsoft’s guide reads a steady count above that as threads blocked in your code.
  • p99 and error rate stay inside the k6 thresholds you set.

Run it once before the fix and once after. The async build should hold a low thread count through both bursts.

Keep it healthy

Export the runtime’s built-in metrics with OpenTelemetry and the Prometheus exporter, as in the HttpClient socket exhaustion recipe. Register the System.Runtime meter, which exists on .NET 9 and later. On .NET 8, the OpenTelemetry.Instrumentation.Runtime package publishes similar metrics under other names.

The exporter names the series dotnet_thread_pool_queue_length, dotnet_thread_pool_thread_count, dotnet_process_cpu_time_seconds_total and dotnet_process_cpu_count. Check your /metrics output before you load these rules:

groups:
  - name: dotnet-threadpool
    rules:
      - alert: DotnetThreadPoolStarvation
        expr: |
          avg_over_time(dotnet_thread_pool_queue_length[5m]) > 10
            and on (job, instance)
          (
            sum by (job, instance) (rate(dotnet_process_cpu_time_seconds_total[5m]))
              / on (job, instance) max by (job, instance) (dotnet_process_cpu_count)
          ) < 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Work is queueing in the thread pool on {{ $labels.instance }} while CPU is low"
          description: "Threads are blocked, not busy. Run dotnet-stack report on the pod and look for .Result, .Wait() or GetResult frames above ThreadPoolWorkQueue.Dispatch."
      - alert: DotnetThreadPoolThreadsHigh
        expr: |
          max by (job, instance) (dotnet_thread_pool_thread_count)
            > on (job, instance) 3 * max by (job, instance) (dotnet_process_cpu_count)
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} runs more than three thread pool threads per core"
          description: "The pool is compensating for blocked threads. The next burst or restart will starve it. Find the blocking calls with dotnet-stack and make them async."

Also alert on the shared tail of Kestrel’s heartbeat warning, which matches every version:

This could be caused by thread pool starvation.

Roll back

Code changes roll back through your deploy tool or kubectl rollout undo deployment/api. Keep a copy of the current spec before you deploy: kubectl get deployment api -o yaml > api-before.yaml.

A raised minimum comes out by removing the property and redeploying. If it pushes memory toward the limit or brings OOMKilled restarts, roll it back now and lower the value. If you remove it because the async fix landed, do that in a quiet hour and watch the heartbeat warning through the next peak.

More recipes

Further reading