Kubernetes CPU throttling shows up as a slow p99 on a pod whose CPU graph looks calm. The node has idle cores, the pod sits under its limit on the dashboard, and requests still wait. The kernel enforces the CPU limit in 100 ms slices, and those slices cause the wait.
What breaks at peak
A CPU limit becomes a CFS bandwidth quota on the container’s cgroup. On cgroup v2 the kubelet writes it to cpu.max as quota and period. A limit of 1 gives the container 100 ms of CPU time in every 100 ms period. A limit of 500m gives it 50 ms.
The quota counts CPU time across every thread. Picture a service with 8 busy threads on an 8-core node and a limit of 1 CPU. The threads use up the 100 ms quota in about 12.5 ms of wall time. The kernel then pauses all of them for the remaining 87.5 ms of the period. Any request in flight waits out the pause.
Your monitoring averages that away. A one-minute rate of container_cpu_usage_seconds_total shows the pod at 40% of its limit, because most periods were quiet. The latency lives in the few periods that ran out of quota. Peak traffic makes those periods more common, and p99 climbs while the node still has free cores.
Kubernetes logs no error for throttling. You find it in the symptoms:
- p99 latency rises with traffic while node CPU stays low.
- Probes time out and the kubelet records an
Unhealthyevent such asLiveness probe failed: Get "http://10.0.3.17:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers). A throttled pod fails its liveness probe, restarts and makes the peak worse. nr_throttledin the container’scpu.statclimbs during traffic.
Two more causes make it worse:
- Runtimes that count host cores. A runtime that sizes its thread pools from the node’s core count starts more threads than the quota can feed. Go before 1.25 set
GOMAXPROCSto the host’s cores, and GC workers then burn the quota in bursts. The Go GOMAXPROCS and GOMEMLIMIT recipe covers that fix. - An old kernel. Kernels before Linux 5.4 had a CFS bug that throttled multi-threaded containers well under their quota. The fix landed in 5.4, with backports to older stable branches.
kubectl get nodes -o wideprints each node’s kernel in theKERNEL-VERSIONcolumn.
The fix for Kubernetes CPU limits
Requests and limits do different jobs. The CPU request reserves capacity for the scheduler and sets the container’s weight (cpu.weight on cgroup v2) when the node runs short. The limit caps the container even when the node is idle. Throttling comes from the limit alone.
Read the container’s own numbers before you change anything:
kubectl exec deploy/api -c app -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat
cpu.max prints the quota and period in microseconds, such as 100000 100000 for a limit of 1, or max 100000 for no limit. cpu.stat shows nr_periods, nr_throttled and throttled_usec. Run it twice a minute apart under load and compare. On a cgroup v1 node the files are /sys/fs/cgroup/cpu/cpu.cfs_quota_us and /sys/fs/cgroup/cpu/cpu.stat, where the time counter is throttled_time in nanoseconds.
Then pull a week of usage from Prometheus. cAdvisor, built into the kubelet, exports the counters:
# CPU used, 95th percentile over 7 days, per container
quantile_over_time(0.95,
sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))[7d:5m])
# share of CFS periods that hit the quota
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
/ sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))
Now set the resources. Changed lines only, in the Deployment’s container spec:
resources:
requests:
cpu: "1500m" # p95 usage at peak, plus 20 to 30% headroom
memory: 1Gi
limits:
memory: 1Gi # keep the memory limit
# no cpu limit: the request keeps this pod's share when the node is busy
Size the request from usage. Set the CPU request near the 95th percentile of usage at peak, plus 20 to 30% headroom. The scheduler packs nodes by requests, so a request below real usage overcommits the node. Under contention, the kernel shares CPU by weight, and a low request loses that contest.
Drop the limit, or give it room. For latency-sensitive services I remove the CPU limit and keep the memory limit. An honest request already protects the neighbors, because the kernel hands out contended CPU by weight. If your policy needs a limit, set it at 2 to 4 times the request and watch the throttled share. A ResourceQuota that sets limits.cpu forces every pod in its namespace to carry a CPU limit, so check for one first:
kubectl get resourcequota,limitrange -n shop
A LimitRange with a default CPU limit adds a limit to any container that omits one. Removing the limit from the Deployment does nothing while that default exists.
Match threads to the CPU you give. Set the runtime’s worker count from the request, not from the host. Go 1.25 and later reads the cgroup CPU limit for GOMAXPROCS. Without a limit, set GOMAXPROCS from the request through an environment variable. The JVM reads the cgroup quota for its processor count. Override it with -XX:ActiveProcessorCount when you run without a limit. For Node.js, one process uses about one core, as the Node.js cluster mode recipe explains.
Keep the HPA in mind. The HPA divides usage by the request. A higher request lowers the utilization it sees, so lower the HPA target with it. The HPA recipe covers how to pick that target.
Guaranteed pods with pinned cores. A few workloads need a whole core with no noisy neighbor. Set requests equal to limits with an integer CPU count, and run the node pool with the kubelet’s cpuManagerPolicy: static. Those containers get exclusive cores. Don’t use it for general web services, because each pinned pod holds its cores even when idle.
Managed node pools. The kubelet setting cpuCFSQuota turns quota enforcement on or off for the whole node, and cpuCFSQuotaPeriod changes the 100 ms period. I leave both at their defaults and fix the pod spec, because a node-wide change hits every tenant. On GKE Standard they sit in the node system config, applied with --system-config-from-file. On AKS they’re cpuCfsQuota and cpuCfsQuotaPeriod in a custom node configuration file passed with --kubelet-config. EKS managed node groups take kubelet settings through a custom launch template’s user data. OKE managed nodes take them as --kubelet-extra-args in a custom cloud-init script. GKE Autopilot doesn’t expose the kubelet, and it lets limits sit above requests only from version 1.30.2-gke.1394000.
Test it on your system
Run a fixed-rate load test against staging at your production peak. A closed-loop tool slows down with the server and hides the stalls you’re looking for, so use k6’s constant-arrival-rate executor:
import http from 'k6/http';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: Number(__ENV.RATE || 200),
timeUnit: '1s',
duration: '15m',
preAllocatedVUs: 200,
maxVUs: 1000,
},
},
thresholds: {
http_req_failed: ['rate<0.001'],
http_req_duration: ['p(99)<500'],
},
};
export default function () {
http.get(`${__ENV.BASE_URL}/api/products`);
}
k6 run -e BASE_URL=https://staging.example.com -e RATE=300 throttle-test.js
Replace the p(99) threshold and the rate with your own budget and peak. Run it twice: once with the old resources, once with the new ones. Keep the pod count fixed during both runs, so the HPA doesn’t hide the difference. Pause it with a fixed minReplicas equal to maxReplicas, or scale the Deployment by hand with the HPA removed.
While it runs, watch three things:
- The throttled share from the PromQL above, per container.
cpu.statinside one pod, read before and after the run, fornr_throttledandthrottled_usec.- Node CPU with
kubectl top nodes, to confirm the node has spare capacity and you’re testing the limit rather than a full node.
Decide the pass bar before the first run. For example, the throttled share stays under 5% of periods, p99 stays under the budget in the script, and no probe fails. With no limit, check the neighbors too. Other pods on the same nodes should keep their own latency while yours runs hot.
Keep it healthy
Scrape the kubelet’s cAdvisor endpoint and kube-state-metrics. These rules catch heavy throttling, a request that no longer matches usage and nodes that have run out of CPU to share:
groups:
- name: kubernetes-cpu
rules:
- alert: ContainerCPUThrottlingHigh
expr: |
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
/ sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))
> 0.25
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} {{ $labels.container }} throttled in over 25% of CFS periods"
description: "The CPU limit is pausing this container. Compare usage with the limit, then raise the limit, remove it or lower the runtime's thread count."
- alert: ContainerCPUAboveRequest
expr: |
sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
/ on (namespace, pod, container)
sum by (namespace, pod, container) (kube_pod_container_resource_requests{resource="cpu"})
> 1.5
for: 30m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} {{ $labels.container }} uses 1.5x its CPU request"
description: "The request no longer matches usage, so this pod loses CPU when the node gets busy. Raise the request to the 95th percentile of usage at peak, and lower the HPA target to match."
- alert: NodeCPUSaturated
expr: |
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) > 0.9
for: 15m
labels:
severity: warning
annotations:
summary: "Node {{ $labels.instance }} CPU above 90% for 15 minutes"
description: "Pods on this node now compete by CPU request. Check which pods run above their requests with kubectl top pods, and add nodes or raise those requests."
NodeCPUSaturated reads node_cpu_seconds_total from node_exporter. Set the 25% throttling line from your own baseline. A batch job can live with heavy throttling, so scope the rule to your request-serving namespaces with a label matcher.
Roll back
Save the current resources before you change them:
kubectl get deploy api -n shop \
-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}' \
> api-resources-before.txt
To put a CPU limit back, set it on the container. The Deployment rolls new pods:
kubectl set resources deploy/api -n shop -c app --limits=cpu=1,memory=1Gi --requests=cpu=500m
If you manage the Deployment with Helm or GitOps, revert the values change there instead, or the next sync undoes your fix.
Roll back now if a container without a limit starves its neighbors: node CPU pinned near 100% and other pods’ latency rising. That points to requests that are too low across the node, so raise them before you try again. For any other reason, revert during a quiet hour, because the change restarts every pod in the Deployment.
More recipes
- Kubernetes Graceful Shutdown Without 502s on Deploy
Hold each old pod in a preStop sleep until the load balancers drop it, then drain open requests on SIGTERM, so a rollout ends with no 502s.
- HPA Scaling Behavior That Keeps Up with Peak Traffic
Scale on a metric that leads the load, lower the target and keep spare node room, so the HPA adds pods before peak traffic starts to queue.
- Fix OOMKilled and Exit Code 137 in Kubernetes
Set Kubernetes memory limits from measured peak usage plus headroom, and cap the runtime below the limit, so pods stop dying with exit code 137.
- Pod Disruption Budget That Survives Node Drains
Give each service a PodDisruptionBudget and spread its pods across nodes and zones, so drains and upgrades evict one pod at a time.