Kubernetes CPU Throttling with Spare CPU on the Node

Size CPU requests from real usage and give limits room above them, or drop them, so CFS quota stops pausing pods on a node with CPU to spare.

Published

Applies to Kubernetes 1.27 and later on cgroup v2 nodes; Linux 5.4 and later for the CFS quota fix

Kubernetes CPU throttling shows up as a slow p99 on a pod whose CPU graph looks calm. The node has idle cores, the pod sits under its limit on the dashboard, and requests still wait. The kernel enforces the CPU limit in 100 ms slices, and those slices cause the wait.

What breaks at peak

A CPU limit becomes a CFS bandwidth quota on the container’s cgroup. On cgroup v2 the kubelet writes it to cpu.max as quota and period. A limit of 1 gives the container 100 ms of CPU time in every 100 ms period. A limit of 500m gives it 50 ms.

The quota counts CPU time across every thread. Picture a service with 8 busy threads on an 8-core node and a limit of 1 CPU. The threads use up the 100 ms quota in about 12.5 ms of wall time. The kernel then pauses all of them for the remaining 87.5 ms of the period. Any request in flight waits out the pause.

Your monitoring averages that away. A one-minute rate of container_cpu_usage_seconds_total shows the pod at 40% of its limit, because most periods were quiet. The latency lives in the few periods that ran out of quota. Peak traffic makes those periods more common, and p99 climbs while the node still has free cores.

Kubernetes logs no error for throttling. You find it in the symptoms:

  • p99 latency rises with traffic while node CPU stays low.
  • Probes time out and the kubelet records an Unhealthy event such as Liveness probe failed: Get "http://10.0.3.17:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers). A throttled pod fails its liveness probe, restarts and makes the peak worse.
  • nr_throttled in the container’s cpu.stat climbs during traffic.

Two more causes make it worse:

  • Runtimes that count host cores. A runtime that sizes its thread pools from the node’s core count starts more threads than the quota can feed. Go before 1.25 set GOMAXPROCS to the host’s cores, and GC workers then burn the quota in bursts. The Go GOMAXPROCS and GOMEMLIMIT recipe covers that fix.
  • An old kernel. Kernels before Linux 5.4 had a CFS bug that throttled multi-threaded containers well under their quota. The fix landed in 5.4, with backports to older stable branches. kubectl get nodes -o wide prints each node’s kernel in the KERNEL-VERSION column.

The fix for Kubernetes CPU limits

Requests and limits do different jobs. The CPU request reserves capacity for the scheduler and sets the container’s weight (cpu.weight on cgroup v2) when the node runs short. The limit caps the container even when the node is idle. Throttling comes from the limit alone.

Read the container’s own numbers before you change anything:

kubectl exec deploy/api -c app -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat

cpu.max prints the quota and period in microseconds, such as 100000 100000 for a limit of 1, or max 100000 for no limit. cpu.stat shows nr_periods, nr_throttled and throttled_usec. Run it twice a minute apart under load and compare. On a cgroup v1 node the files are /sys/fs/cgroup/cpu/cpu.cfs_quota_us and /sys/fs/cgroup/cpu/cpu.stat, where the time counter is throttled_time in nanoseconds.

Then pull a week of usage from Prometheus. cAdvisor, built into the kubelet, exports the counters:

# CPU used, 95th percentile over 7 days, per container
quantile_over_time(0.95,
  sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))[7d:5m])

# share of CFS periods that hit the quota
sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
  / sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))

Now set the resources. Changed lines only, in the Deployment’s container spec:

resources:
  requests:
    cpu: "1500m"     # p95 usage at peak, plus 20 to 30% headroom
    memory: 1Gi
  limits:
    memory: 1Gi      # keep the memory limit
    # no cpu limit: the request keeps this pod's share when the node is busy

Size the request from usage. Set the CPU request near the 95th percentile of usage at peak, plus 20 to 30% headroom. The scheduler packs nodes by requests, so a request below real usage overcommits the node. Under contention, the kernel shares CPU by weight, and a low request loses that contest.

Drop the limit, or give it room. For latency-sensitive services I remove the CPU limit and keep the memory limit. An honest request already protects the neighbors, because the kernel hands out contended CPU by weight. If your policy needs a limit, set it at 2 to 4 times the request and watch the throttled share. A ResourceQuota that sets limits.cpu forces every pod in its namespace to carry a CPU limit, so check for one first:

kubectl get resourcequota,limitrange -n shop

A LimitRange with a default CPU limit adds a limit to any container that omits one. Removing the limit from the Deployment does nothing while that default exists.

Match threads to the CPU you give. Set the runtime’s worker count from the request, not from the host. Go 1.25 and later reads the cgroup CPU limit for GOMAXPROCS. Without a limit, set GOMAXPROCS from the request through an environment variable. The JVM reads the cgroup quota for its processor count. Override it with -XX:ActiveProcessorCount when you run without a limit. For Node.js, one process uses about one core, as the Node.js cluster mode recipe explains.

Keep the HPA in mind. The HPA divides usage by the request. A higher request lowers the utilization it sees, so lower the HPA target with it. The HPA recipe covers how to pick that target.

Guaranteed pods with pinned cores. A few workloads need a whole core with no noisy neighbor. Set requests equal to limits with an integer CPU count, and run the node pool with the kubelet’s cpuManagerPolicy: static. Those containers get exclusive cores. Don’t use it for general web services, because each pinned pod holds its cores even when idle.

Managed node pools. The kubelet setting cpuCFSQuota turns quota enforcement on or off for the whole node, and cpuCFSQuotaPeriod changes the 100 ms period. I leave both at their defaults and fix the pod spec, because a node-wide change hits every tenant. On GKE Standard they sit in the node system config, applied with --system-config-from-file. On AKS they’re cpuCfsQuota and cpuCfsQuotaPeriod in a custom node configuration file passed with --kubelet-config. EKS managed node groups take kubelet settings through a custom launch template’s user data. OKE managed nodes take them as --kubelet-extra-args in a custom cloud-init script. GKE Autopilot doesn’t expose the kubelet, and it lets limits sit above requests only from version 1.30.2-gke.1394000.

Test it on your system

Run a fixed-rate load test against staging at your production peak. A closed-loop tool slows down with the server and hides the stalls you’re looking for, so use k6’s constant-arrival-rate executor:

import http from 'k6/http';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.RATE || 200),
      timeUnit: '1s',
      duration: '15m',
      preAllocatedVUs: 200,
      maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(99)<500'],
  },
};

export default function () {
  http.get(`${__ENV.BASE_URL}/api/products`);
}
k6 run -e BASE_URL=https://staging.example.com -e RATE=300 throttle-test.js

Replace the p(99) threshold and the rate with your own budget and peak. Run it twice: once with the old resources, once with the new ones. Keep the pod count fixed during both runs, so the HPA doesn’t hide the difference. Pause it with a fixed minReplicas equal to maxReplicas, or scale the Deployment by hand with the HPA removed.

While it runs, watch three things:

  • The throttled share from the PromQL above, per container.
  • cpu.stat inside one pod, read before and after the run, for nr_throttled and throttled_usec.
  • Node CPU with kubectl top nodes, to confirm the node has spare capacity and you’re testing the limit rather than a full node.

Decide the pass bar before the first run. For example, the throttled share stays under 5% of periods, p99 stays under the budget in the script, and no probe fails. With no limit, check the neighbors too. Other pods on the same nodes should keep their own latency while yours runs hot.

Keep it healthy

Scrape the kubelet’s cAdvisor endpoint and kube-state-metrics. These rules catch heavy throttling, a request that no longer matches usage and nodes that have run out of CPU to share:

groups:
  - name: kubernetes-cpu
    rules:
      - alert: ContainerCPUThrottlingHigh
        expr: |
          sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
            / sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))
            > 0.25
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} {{ $labels.container }} throttled in over 25% of CFS periods"
          description: "The CPU limit is pausing this container. Compare usage with the limit, then raise the limit, remove it or lower the runtime's thread count."
      - alert: ContainerCPUAboveRequest
        expr: |
          sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
            / on (namespace, pod, container)
          sum by (namespace, pod, container) (kube_pod_container_resource_requests{resource="cpu"})
            > 1.5
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} {{ $labels.container }} uses 1.5x its CPU request"
          description: "The request no longer matches usage, so this pod loses CPU when the node gets busy. Raise the request to the 95th percentile of usage at peak, and lower the HPA target to match."
      - alert: NodeCPUSaturated
        expr: |
          1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) > 0.9
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.instance }} CPU above 90% for 15 minutes"
          description: "Pods on this node now compete by CPU request. Check which pods run above their requests with kubectl top pods, and add nodes or raise those requests."

NodeCPUSaturated reads node_cpu_seconds_total from node_exporter. Set the 25% throttling line from your own baseline. A batch job can live with heavy throttling, so scope the rule to your request-serving namespaces with a label matcher.

Roll back

Save the current resources before you change them:

kubectl get deploy api -n shop \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}' \
  > api-resources-before.txt

To put a CPU limit back, set it on the container. The Deployment rolls new pods:

kubectl set resources deploy/api -n shop -c app --limits=cpu=1,memory=1Gi --requests=cpu=500m

If you manage the Deployment with Helm or GitOps, revert the values change there instead, or the next sync undoes your fix.

Roll back now if a container without a limit starves its neighbors: node CPU pinned near 100% and other pods’ latency rising. That points to requests that are too low across the node, so raise them before you try again. For any other reason, revert during a quiet hour, because the change restarts every pod in the Deployment.

More recipes

Further reading