Tokio spawn_blocking for Blocking Work in Axum and Actix

Move blocking calls to tokio spawn_blocking or a CPU pool, so one slow handler stops stalling every request that shares its Tokio worker thread.

Published

Applies to Tokio 1.45 and later (stable worker busy metrics), Rust 1.80 and later, Axum 0.8, Actix Web 4

Tokio spawn_blocking runs a blocking call on a separate thread pool, so it doesn’t hold one of the runtime’s few worker threads. Tokio starts one worker per CPU core by default. A handler that hashes a password, reads a file with std::fs or calls a sync database driver keeps its worker for the whole call. Every other task queued on that worker waits, including requests that would take a microsecond.

What breaks at peak

Tokio is cooperative. A task gives its thread back only at an .await that isn’t ready. Code that runs for a long time between two .await points owns the worker until it finishes. A useful rule of thumb from Tokio maintainer Alice Ryhl: stay under 10 to 100 microseconds between awaits.

At low traffic you won’t notice much. Tokio steals work across workers, so an idle worker picks up the queue. At peak, several requests hit the blocking handler at once and hold every worker. Then the whole service stalls:

  • Cheap endpoints, such as health checks, wait behind the slow ones, and probes start to fail.
  • Timers fire late, so tokio::time::timeout and tower’s timeout layer can’t cut a request short. They need a free worker to run.
  • Connection pools time out waiting for tasks that can’t get polled, which looks like a database problem.

Some blocking mistakes panic in place of stalling. These are the messages people search:

Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.

Recent Tokio releases append If you are in an async function, use `.await` on the future instead of blocking on it. You get it from calling Runtime::block_on or Handle::block_on inside async code, often through a sync wrapper around an async client.

Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.

The usual source is reqwest::blocking called from an async handler. In debug builds reqwest checks for this and panics. In release builds the check is compiled out, so the same code blocks the worker in production and never panics.

cannot call `block_in_place` from a `current_thread` runtime or from within a `LocalSet`

Tokio 1.47 and older word it can call blocking only when running on the multi-threaded runtime. Actix Web hits this one, because each Actix worker runs its own single-threaded runtime.

The spawn_blocking fix for an Axum blocking handler

Sort the blocking work into two kinds, because each goes to a different pool:

WorkExamplesWhere it goes
Blocking I/Ostd::fs, sync database drivers such as Diesel, reqwest::blocking, sync SDKstokio::task::spawn_blocking
CPU workpassword hashing, image resizing, compression, large JSON or CSV buildsspawn_blocking or rayon, behind a semaphore sized to your cores

Blocking I/O threads spend most of their time waiting, so a large pool of them is cheap. CPU work can’t go faster than your cores, so more threads only add context switches.

Axum. This handler verifies a password on the blocking pool and caps how many run at once:

use std::sync::Arc;

use axum::{extract::State, http::StatusCode, routing::post, Json, Router};
use tokio::sync::Semaphore;

#[derive(Clone)]
struct AppState {
    cpu_slots: Arc<Semaphore>,
}

#[derive(serde::Deserialize)]
struct Login {
    user: String,
    password: String,
}

async fn login(
    State(state): State<AppState>,
    Json(body): Json<Login>,
) -> Result<StatusCode, StatusCode> {
    // Wait here, on the async side, when every CPU slot is taken.
    let _permit = state
        .cpu_slots
        .clone()
        .acquire_owned()
        .await
        .map_err(|_| StatusCode::SERVICE_UNAVAILABLE)?;

    let ok = tokio::task::spawn_blocking(move || verify_password(&body.user, &body.password))
        .await
        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;

    if ok { Ok(StatusCode::NO_CONTENT) } else { Err(StatusCode::UNAUTHORIZED) }
}

fn verify_password(user: &str, password: &str) -> bool {
    // argon2 or bcrypt verification goes here
    !user.is_empty() && !password.is_empty()
}

fn main() {
    let cores = std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1);

    let runtime = tokio::runtime::Builder::new_multi_thread()
        .worker_threads(cores)
        .max_blocking_threads(64)
        .enable_all()
        .build()
        .expect("build runtime");

    runtime.block_on(async move {
        let state = AppState { cpu_slots: Arc::new(Semaphore::new(cores)) };
        let app = Router::new().route("/login", post(login)).with_state(state);
        let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.expect("bind");
        axum::serve(listener, app).await.expect("serve");
    });
}
  • spawn_blocking moves the closure to the blocking pool and returns a JoinHandle. Awaiting it frees the worker until the result is ready. The JoinError arm catches a panic inside the closure.
  • The semaphore caps concurrent hashes at the core count. Without it, a login spike queues hundreds of closures on the blocking pool. That queue applies no backpressure, as the Tokio docs warn, and every hash competes for the same cores.
  • worker_threads defaults to TOKIO_WORKER_THREADS if set, then to available_parallelism(). On Linux that reads the cgroup CPU quota, so a pod limited to 2 CPUs gets 2 workers. Set it yourself only if the count it reads is wrong.
  • max_blocking_threads caps the blocking pool, 512 by default. Each thread costs a stack and memory. I size it to the blocking I/O you allow at once: the database pool plus file and SDK calls, with headroom. Idle blocking threads exit after 10 seconds, set by thread_keep_alive.

A blocking task can’t be cancelled once it starts. If the client disconnects or a timeout fires, the closure still runs to the end and holds its thread. Put timeouts inside the blocking call too, for example on the sync driver’s query.

CPU work on rayon. For CPU-heavy work, hand it to rayon and await the answer over a oneshot channel, as the Tokio docs suggest:

async fn thumbnail(bytes: Vec<u8>) -> Result<Vec<u8>, tokio::sync::oneshot::error::RecvError> {
    let (tx, rx) = tokio::sync::oneshot::channel();
    rayon::spawn(move || {
        let _ = tx.send(make_thumbnail(&bytes));
    });
    rx.await
}

Rayon’s global pool runs one thread per core and queues the rest. Keep the semaphore in front of it, so a burst waits in your handler and doesn’t build a queue you can’t see.

block_in_place runs blocking code on the current worker and hands that worker’s other tasks to a new thread. It saves a move into a closure, but it pauses everything else in the same task, such as other branches of a join!. It also panics on a current-thread runtime. I use spawn_blocking unless a borrowed value makes the closure impossible.

Actix Web. Each Actix worker is an OS thread with its own single-threaded Tokio runtime, and HttpServer::workers defaults to available_parallelism(). A blocking handler stalls every connection on that worker, with no work stealing to help. Use web::block, which runs the closure on that worker’s blocking pool:

use actix_web::{post, web, App, HttpResponse, HttpServer};

#[derive(serde::Deserialize)]
struct Login {
    user: String,
    password: String,
}

#[post("/login")]
async fn login(body: web::Json<Login>) -> actix_web::Result<HttpResponse> {
    let Login { user, password } = body.into_inner();
    let ok = web::block(move || verify_password(&user, &password)).await?;
    Ok(if ok {
        HttpResponse::NoContent().finish()
    } else {
        HttpResponse::Unauthorized().finish()
    })
}

fn verify_password(user: &str, password: &str) -> bool {
    !user.is_empty() && !password.is_empty()
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| App::new().service(login))
        .workers(4)                     // match the pod's CPU limit
        .worker_max_blocking_threads(32) // per worker
        .bind(("0.0.0.0", 8080))?
        .run()
        .await
}

web::block returns BlockingError if the closure panics, and the ? turns that into a 500. worker_max_blocking_threads sets the pool per worker, and its default splits 512 threads across the workers. Each worker’s pool serves only that worker.

The Node.js event loop monitoring recipe covers the same failure on Node.js, where one thread serves everything.

Test it on your system

The blocked runtime shows up on the cheap endpoints, so test both at once. Hold the heavy endpoint at your peak rate and a cheap one at a steady low rate, both through the same load balancer as production:

import http from 'k6/http';

export const options = {
  scenarios: {
    heavy: {
      executor: 'constant-arrival-rate',
      exec: 'login', rate: 200, timeUnit: '1s', duration: '10m',
      preAllocatedVUs: 200, maxVUs: 800,
    },
    cheap: {
      executor: 'constant-arrival-rate',
      exec: 'health', rate: 20, timeUnit: '1s', duration: '10m',
      preAllocatedVUs: 20,
    },
  },
  thresholds: {
    'http_req_duration{scenario:cheap}': ['p(99)<50'],  // your budget for a trivial route
    'http_req_duration{scenario:heavy}': ['p(99)<800'],
    http_req_failed: ['rate<0.01'],
  },
};

export function login() {
  http.post(`${__ENV.TARGET}/login`, JSON.stringify({ user: 'load-test', password: 'x' }), {
    headers: { 'Content-Type': 'application/json' },
  });
}

export function health() {
  http.get(`${__ENV.TARGET}/healthz`);
}

Run it with k6 run -e TARGET=https://staging.example.com blocking.js against the old build, then the new one. On the old build, the cheap route’s p99 rises with the heavy load. On the fixed build it should stay flat while the heavy route slows down on its own.

While it runs, watch these:

  • The cheap route’s latency in the k6 output. That’s the clearest sign that workers are free.
  • Worker busy ratio and global queue depth from the metrics in the next section.
  • CPU per pod. With the semaphore, CPU should plateau near the core count while extra logins wait.
  • tokio-console in staging. Build with RUSTFLAGS="--cfg tokio_unstable" and the console-subscriber crate. Its never-yielded warning flags a task that has run for over a second without yielding, which points at the handler that blocks.

Pass means the cheap route holds its budget through the peak run, with no probe failures and your error threshold intact.

Keep it healthy

Tokio’s stable RuntimeMetrics give the worker busy time and the global queue depth. Sample them into gauges with the prometheus crate, and serve them on a /metrics route:

use std::sync::LazyLock;
use std::time::{Duration, Instant};

use prometheus::{register_gauge, register_int_gauge, Gauge, IntGauge};

static BUSY: LazyLock<Gauge> = LazyLock::new(|| {
    register_gauge!("tokio_runtime_busy_ratio", "Share of worker time spent polling tasks").unwrap()
});
static QUEUE: LazyLock<IntGauge> = LazyLock::new(|| {
    register_int_gauge!("tokio_runtime_global_queue_depth", "Tasks waiting in the global queue").unwrap()
});
static TASKS: LazyLock<IntGauge> = LazyLock::new(|| {
    register_int_gauge!("tokio_runtime_alive_tasks", "Tasks alive on the runtime").unwrap()
});

async fn sample_runtime(handle: tokio::runtime::Handle) {
    let m = handle.metrics();
    let workers = m.num_workers();
    let mut last: Vec<Duration> = (0..workers).map(|i| m.worker_total_busy_duration(i)).collect();
    let mut last_at = Instant::now();
    let mut tick = tokio::time::interval(Duration::from_secs(5));
    loop {
        tick.tick().await;
        let now = Instant::now();
        let elapsed = now.duration_since(last_at).as_secs_f64();
        last_at = now;
        let mut busy = Duration::ZERO;
        for (i, prev) in last.iter_mut().enumerate() {
            let total = m.worker_total_busy_duration(i);
            busy += total.saturating_sub(*prev);
            *prev = total;
        }
        BUSY.set(busy.as_secs_f64() / (elapsed * workers as f64));
        QUEUE.set(m.global_queue_depth() as i64);
        TASKS.set(m.num_alive_tasks() as i64);
    }
}

async fn metrics() -> String {
    prometheus::TextEncoder::new()
        .encode_to_string(&prometheus::gather())
        .unwrap_or_default()
}

Start the sampler with tokio::spawn(sample_runtime(tokio::runtime::Handle::current())) and add .route("/metrics", get(metrics)) to the router. worker_total_busy_duration and global_queue_depth are stable from Tokio 1.45. The blocking pool’s queue depth needs --cfg tokio_unstable. On Actix, each worker is its own runtime, so watch the cheap route’s latency there instead of runtime gauges.

These rules assume the scrape job is called api and your scrape config adds namespace and pod labels:

groups:
  - name: tokio-runtime
    rules:
      - alert: TokioWorkersBusy
        expr: max by (namespace, pod) (tokio_runtime_busy_ratio{job="api"}) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Tokio workers on {{ $labels.pod }} busy over 90% of the time"
          description: "Workers spend nearly all their time polling. Look for CPU work or blocking calls in handlers with tokio-console in staging, and move them to spawn_blocking or rayon."
      - alert: TokioRuntimeStalled
        expr: |
          max by (namespace, pod) (tokio_runtime_busy_ratio{job="api"}) > 0.95
            and on (namespace, pod)
          max by (namespace, pod) (tokio_runtime_global_queue_depth{job="api"}) > 50
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Tokio runtime on {{ $labels.pod }} is saturated with tasks queuing"
          description: "Requests wait for a free worker. Add replicas now, then find the handler that holds workers between awaits and move it off the runtime."
      - alert: TokioMetricsUnreachable
        expr: up{job="api"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus can't scrape {{ $labels.pod }}"
          description: "A runtime with every worker blocked can't answer /metrics. Check the pod's probe events and logs, and capture a thread dump with gdb or perf before it restarts."

Set the busy and queue thresholds from a week of your own baseline. A busy ratio near 1 on a CPU-bound service at peak can be fine, as long as the cheap route stays fast. The queue depth is what tells you requests wait.

Roll back

Tag the current release before you deploy, so you can redeploy it in one step.

  • To undo a handler change, revert the commit. The handler blocks its worker again, which you accepted before.
  • To undo runtime sizing, remove the worker_threads and max_blocking_threads calls, or set TOKIO_WORKER_THREADS back. Tokio returns to its defaults.
  • If requests wait too long on the semaphore at peak, raise the permit count before you remove it.

If memory climbs from blocking threads or the cheap route slows after the change, roll back now and check the pool sizes. Other reverts belong in an off-peak window.

More recipes

Further reading