A FastAPI blocking call inside an async def route stops the whole Uvicorn process, not one request. Every other request on that event loop waits until the call returns. At low traffic you never see it. At peak, p99 climbs while CPU stays low and nothing appears in the error log.
What breaks at peak
Uvicorn runs one event loop per process. The loop switches between requests only at an await. A call that blocks without awaiting holds the loop until it finishes:
import requests
from fastapi import FastAPI
app = FastAPI()
@app.get("/quote")
async def quote():
# Blocks the event loop for the whole HTTP call.
r = requests.get("https://pricing.internal/v1/quote", timeout=5)
return r.json()
If that call takes 300 ms, every request in the process waits up to 300 ms longer, including /healthz. Ten such calls in a burst stack into seconds of delay. The usual offenders are easy to miss in review:
requests,urllib3and sync SDKs such asboto3psycopg2, sync SQLAlchemy sessions and the Django ORMtime.sleep, file reads and writes, andsubprocess.run- CPU work: password hashing, PDF rendering, large JSON or pandas transforms
Three symptoms point to a blocked loop. Fast endpoints slow down whenever a slow one gets traffic. Kubernetes liveness probes time out, and the kubelet restarts pods that were healthy a minute earlier. Your CPU graphs stay low, because a process waiting on a socket uses no CPU.
Uvicorn won’t save you here. Its --workers supervisor pings each worker from a separate thread, so a blocked loop still answers. Under Gunicorn with a Uvicorn worker class, the heartbeat runs on the loop. A long block there ends in [CRITICAL] WORKER TIMEOUT, which the Gunicorn workers and timeouts recipe covers. Rust services stall the same way when a handler blocks a Tokio worker, and the Tokio spawn_blocking recipe covers that fix.
Find the blocking call. Python’s asyncio debug mode logs any step that holds the loop for longer than 100 ms. Turn it on in staging, with the standard asyncio loop instead of uvloop:
PYTHONASYNCIODEBUG=1 uvicorn app.main:app --loop asyncio --port 8000
Each slow step logs a warning from the asyncio logger:
Executing <Task pending name='Task-12' coro=<RequestResponseCycle.run_asgi() running at ...>> took 0.312 seconds
The task name tells you which request held the loop. Debug mode adds overhead, so keep it out of production.
Move blocking calls off the loop with run_in_threadpool
Pick the fix by what the call does. I go down this list in order.
Use an async client where one exists. It’s the only fix that adds no threads. Swap requests for httpx.AsyncClient, psycopg2 for psycopg 3 or asyncpg, and sync SQLAlchemy for AsyncSession. Create the client once in the lifespan and reuse it, so you keep connection pooling.
Declare the route with def when its body is sync. FastAPI runs def routes and def dependencies in a threadpool and awaits the result. The loop stays free:
@app.get("/quote")
def quote():
r = requests.get("https://pricing.internal/v1/quote", timeout=5)
return r.json()
Wrap one call with run_in_threadpool in an async route. Use this when the route awaits other things and has one sync call in the middle:
from fastapi.concurrency import run_in_threadpool
@app.post("/login")
async def login(form: LoginForm):
user = await users.get_by_email(form.email)
ok = await run_in_threadpool(password_hasher.verify, user.password_hash, form.password)
if not ok:
raise HTTPException(status_code=401)
return await sessions.create(user.id)
run_in_threadpool comes from Starlette and calls anyio.to_thread.run_sync. asyncio.to_thread also works, but it uses the loop’s default executor, a second pool capped at min(32, CPU count + 4) threads.
Send heavy CPU work to processes. Threads still share the GIL, so a CPU-bound function in the threadpool slows every other thread. Use a ProcessPoolExecutor with loop.run_in_executor, or move the work to a queue and a separate worker.
Size the threadpool. All of the above, except processes, share one AnyIO limiter per process. It defaults to 40 threads. The 41st sync call waits for a free thread, and your p99 grows with no error. Set it at startup:
import asyncio
import os
import time
from contextlib import asynccontextmanager
import anyio.to_thread
from fastapi import FastAPI
from prometheus_client import Gauge, Histogram, make_asgi_app
LOOP_LAG = Histogram(
"app_event_loop_lag_seconds",
"Delay between when a timer should fire and when the event loop ran it",
buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5),
)
THREADS_BUSY = Gauge("app_threadpool_busy_threads", "Threads in use in the AnyIO default limiter")
THREADS_MAX = Gauge("app_threadpool_max_threads", "Size of the AnyIO default limiter")
async def watch_loop(interval: float = 0.5) -> None:
limiter = anyio.to_thread.current_default_thread_limiter()
while True:
start = time.perf_counter()
await asyncio.sleep(interval)
LOOP_LAG.observe(max(0.0, time.perf_counter() - start - interval))
THREADS_BUSY.set(limiter.borrowed_tokens)
THREADS_MAX.set(limiter.total_tokens)
@asynccontextmanager
async def lifespan(app: FastAPI):
limiter = anyio.to_thread.current_default_thread_limiter()
limiter.total_tokens = int(os.environ.get("THREADPOOL_SIZE", "40"))
task = asyncio.create_task(watch_loop())
yield
task.cancel()
app = FastAPI(lifespan=lifespan)
app.mount("/metrics", make_asgi_app())
The size depends on what the threads wait for. If most sync calls hit the database, more threads than database connections only move the queue into the connection pool. I set the threadpool to the database pool size plus a few threads for other sync work. Raise it for sync calls to slow external APIs, where threads sit idle waiting.
Uvicorn settings. These matter once the loop is free:
--workersstarts several processes on a VM. On Kubernetes I run one process per pod and scale pods, so each pod’s CPU and the metrics above map to one loop. With several workers,prometheus_clientneeds its multiprocess mode.--limit-concurrencyreturns a 503 once a process holds that many connections or tasks. A fast 503 that a client retries beats a 30-second queue. Size it near what one process serves within your latency target.--timeout-keep-alivedefaults to 5 seconds. If a load balancer connects straight to Uvicorn, set it above the balancer’s idle timeout, or the balancer reuses closed sockets and returns 502s.--backlogdefaults to 2048, and the kernel caps it atnet.core.somaxconn.
On Cloud Run or App Runner, the platform’s per-instance concurrency setting caps requests per container. Keep it in line with --limit-concurrency, so the platform scales out before Uvicorn starts refusing.
Test it on your system
The test that exposes a blocked loop mixes a slow endpoint with a fast one. If the fast endpoint’s p99 rises when the slow one gets traffic, something still blocks. This k6 script runs both at fixed rates:
import http from 'k6/http';
import { check } from 'k6';
const BASE = __ENV.BASE_URL;
export const options = {
scenarios: {
heavy: {
executor: 'constant-arrival-rate',
exec: 'heavy',
rate: Number(__ENV.HEAVY_RATE || 50),
timeUnit: '1s',
duration: '10m',
preAllocatedVUs: 100,
maxVUs: 1000,
},
light: {
executor: 'constant-arrival-rate',
exec: 'light',
rate: Number(__ENV.LIGHT_RATE || 200),
timeUnit: '1s',
duration: '10m',
preAllocatedVUs: 100,
maxVUs: 1000,
},
},
thresholds: {
'http_req_duration{scenario:light}': ['p(99)<100'], // your target for the cheap endpoint
'http_req_duration{scenario:heavy}': ['p(99)<1500'],
http_req_failed: ['rate<0.001'],
},
};
export function heavy() {
const res = http.get(`${BASE}/quote`);
check(res, { 'heavy 200': (r) => r.status === 200 });
}
export function light() {
const res = http.get(`${BASE}/items/42`);
check(res, { 'light 200': (r) => r.status === 200 });
}
Run the light scenario alone first and note its p99. Then run both together at your peak rates. Pick the endpoints from your own traffic: one that calls a slow dependency, and one that only reads from memory or a cache.
While it runs, watch these:
app_event_loop_lag_secondson/metrics. Lag near zero means the loop stays free.app_threadpool_busy_threadsagainstapp_threadpool_max_threads. A pool pinned at its maximum means sync calls are queueing.- Liveness probe failures, with
kubectl get events --field-selector reason=Unhealthy. - Pod CPU. Low CPU with high latency points back to blocking or a full threadpool.
The run passes when the light endpoint’s p99 under the mixed load stays close to its solo run, within the margin you set. Loop lag stays under your threshold, and no pod restarts.
Keep it healthy
These rules use the metrics from the lifespan code above. If you also run prometheus-fastapi-instrumentator, its http_request_duration_seconds histogram, labelled by handler, shows which route slowed.
groups:
- name: fastapi-event-loop
rules:
- alert: FastAPIEventLoopLag
expr: |
histogram_quantile(0.99,
sum by (le, namespace, pod) (rate(app_event_loop_lag_seconds_bucket[5m]))
) > 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "Event loop p99 lag above 100 ms on {{ $labels.pod }}"
description: "A sync call is running on the loop. Run the pod's image in staging with PYTHONASYNCIODEBUG=1 and --loop asyncio to log the slow task, then move that call to a thread or an async client."
- alert: FastAPIEventLoopBlocked
expr: |
histogram_quantile(0.99,
sum by (le, namespace, pod) (rate(app_event_loop_lag_seconds_bucket[5m]))
) > 0.5
for: 2m
labels:
severity: critical
annotations:
summary: "Event loop on {{ $labels.pod }} is stalling every request"
description: "Every request in this process waits behind a blocking call. Check recent deploys for new sync calls in async routes, and roll back if one shipped."
- alert: FastAPIThreadpoolFull
expr: |
min_over_time(app_threadpool_busy_threads[5m])
>= on (namespace, pod) app_threadpool_max_threads
for: 5m
labels:
severity: warning
annotations:
summary: "The AnyIO threadpool on {{ $labels.pod }} has been full for 5 minutes"
description: "Sync routes and run_in_threadpool calls are queueing. Find the slow dependency first. Raise THREADPOOL_SIZE only if the database pool has room."
Set the lag thresholds from your own baseline. A loop that serves large JSON responses shows some lag from serialization alone.
Roll back
Each fix is a code change, so Git holds the old version. Record the running image and threadpool size before you deploy:
kubectl get deploy api -o jsonpath='{.spec.template.spec.containers[0].image}' > api-image-before.txt
kubectl get deploy api -o jsonpath='{.spec.template.spec.containers[0].env}' > api-env-before.json
kubectl rollout undo deployment/api returns to the previous ReplicaSet. If you only changed THREADPOOL_SIZE, set it back with kubectl set env deployment/api THREADPOOL_SIZE=40.
If a larger threadpool starts exhausting database connections, roll back at once, because that error spreads to every route. If a route moved to def runs slower than before, revert it in your next quiet window and look for an async client instead.
More recipes
- Django CONN_MAX_AGE, Connection Pooling and PgBouncer
Match CONN_MAX_AGE or Django's pool to your worker count, and add Django PgBouncer settings when pods multiply, so peak traffic stops exhausting Postgres.
- Fix Gunicorn WORKER TIMEOUT and Size Workers
Size Gunicorn workers and threads from the pod's CPU and memory, and order your timeouts, so peak traffic stops killing workers mid-request.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.