HPA Scaling Behavior That Keeps Up with Peak Traffic

Scale on a metric that leads the load, lower the target and keep spare node room, so the HPA adds pods before peak traffic starts to queue.

Published

Applies to Kubernetes 1.30 and later (autoscaling/v2 with container metrics); the per-HPA tolerance field needs 1.37 or later

HPA scaling behavior decides how fast your Deployment grows when traffic climbs. The defaults react within a minute or two, and a new pod can take minutes more to serve. Peak traffic arrives faster than that. Most fixes sit outside the behavior block: the target, the metric and the time a pod takes to start.

What breaks at peak when HPA scale up is too slow

The HPA controller runs every 15 seconds. Each run it reads the metric for every pod, averages it and computes:

desiredReplicas = ceil(currentReplicas × currentMetricValue / desiredMetricValue)

For CPU and memory, the metric is utilization as a share of the pod’s request, not its limit. A 70% target means “add pods once average usage passes 70% of the request”.

Between the first extra request and a new pod serving traffic, a chain of delays adds up:

  1. Metrics. metrics-server scrapes each kubelet on its --metric-resolution. The upstream manifest sets 15 seconds, while the flag’s own default is one minute. Check what your cluster runs.
  2. The controller. It acts on its next 15-second pass. A ratio within the tolerance, 10% by default, changes nothing.
  3. New pods. For five minutes after a pod starts, the controller can ignore its CPU samples while it isn’t ready. On a scale-up, it assumes not-ready pods use 0% of the target, which damps the next step.
  4. Scheduling. A pod with no room on any node waits in Pending until the cluster autoscaler or Karpenter adds a node. That takes minutes on every cloud.
  5. Startup. Image pull, app boot, cache warm-up and the readiness probe come last.

Meanwhile the existing pods take all the extra load. Latency rises, then timeouts and 5xx follow. CPU limits make it worse: a throttled pod can’t report usage above its limit, so the HPA never sees the real demand. The CPU throttling recipe covers that trap.

The default scale-up policy allows doubling the pod count, or adding 4 pods, every 15 seconds, whichever is more. That’s fast enough for most services, so I tune the target and the start time before I touch the policy.

kubectl describe hpa shows the controller’s view. These messages from the HPA controller tell you why it isn’t scaling:

failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
missing request for cpu in container app of Pod api-7c9f6b5d8-x2k4q
the desired replica count is more than the maximum replica count
the desired replica count is increasing faster than the maximum scale rate

The first means metrics-server is missing or unhealthy, or the pods are too new to report. The second means a container has no CPU request, so utilization has no denominator. The third sits under the ScalingLimited condition with reason TooManyReplicas, so maxReplicas caps you. The fourth carries reason ScaleUpLimit and means your behavior policy holds the HPA back.

The fix

A full HPA object for an API Deployment:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: shop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 6
  maxReplicas: 60
  metrics:
    - type: ContainerResource
      containerResource:
        name: cpu
        container: app
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 8
          periodSeconds: 15
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 20
          periodSeconds: 60

Target from your reaction time. The target leaves headroom for the growth that arrives before new pods serve. As a rule, the target should sit at or below 1 ÷ (1 + growth during reaction time). If traffic can climb 50% in the minutes a new pod takes to serve, the target should be at most 1 ÷ 1.5, about 66%. Take the growth rate from your own traffic graphs on a peak day.

Container metric. ContainerResource scales on the app container alone, so a sidecar’s idle CPU doesn’t drag the average down. The metric type went stable in Kubernetes 1.30.

minReplicas for the first minutes. The HPA can’t help in the first minutes of a spike. Set minReplicas to the count that carries your peak traffic for the length of one reaction chain. For a known daily peak, a CronJob or KEDA’s cron trigger can raise the floor before it starts.

Scale-up policy. The Pods policy matters for small Deployments, where 100% of 3 pods is only 3. Raise it so one step covers a real spike. Leave stabilizationWindowSeconds: 0 on scale-up, which is the default.

Slow scale-down. A 300-second window with 20% per minute keeps the HPA from dropping pods between two bursts. It costs a few minutes of extra pods after each peak.

A metric that leads the load. CPU lags for services that wait on a database or another API. Requests per second, or queue depth for workers, rises before CPU does. For Kafka consumers, the Kafka consumer lag recipe gives you the lag metric to scale on. Expose it through the Prometheus Adapter, or let KEDA create the HPA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api
  namespace: shop
spec:
  scaleTargetRef:
    name: api
  minReplicaCount: 6
  maxReplicaCount: 60
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Pods
              value: 8
              periodSeconds: 15
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        query: sum(rate(http_requests_total{namespace="shop", service="api"}[1m]))
        threshold: "50"

threshold is the rate one pod should handle at your target latency. KEDA divides the query result by it to get the pod count. Find the number with a load test against a single pod, then take 60 to 70% of the rate where its p99 breaks.

Pod start time. Every second of startup adds to the reaction chain. Keep images small, pull from a registry in the same region and use a startupProbe so a slow boot doesn’t trip the liveness probe. A readiness probe should pass only once the pod can serve at full speed.

Spare node room. Node provisioning is the longest delay. Run a few low-priority placeholder pods that hold room on existing nodes. The scheduler evicts them for real pods at once, and the autoscaler replaces the nodes they lose:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning
value: -10
globalDefault: false
description: "Placeholder pods that give way to real workloads"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioning
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: overprovisioning
  template:
    metadata:
      labels:
        app: overprovisioning
    spec:
      priorityClassName: overprovisioning
      terminationGracePeriodSeconds: 0
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.10
          resources:
            requests:
              cpu: "2"
              memory: 4Gi

Size the placeholders to hold the pods your first scale-up step adds.

Database connections. Each new pod opens its own pool. Check that maxReplicas times the pool size fits under the database’s limit, or put PgBouncer in front of PostgreSQL.

Managed control planes. EKS, AKS, GKE and OKE don’t let you change kube-controller-manager flags. The sync period, the default tolerance and the CPU initialization period stay fixed. The behavior block is your lever, plus the per-HPA tolerance field from Kubernetes 1.37. metrics-server ships with AKS and GKE. EKS doesn’t install it, and you add it as a community add-on. OKE offers it as a cluster add-on on enhanced clusters, which needs cert-manager first. For nodes, EKS uses Karpenter, Cluster Autoscaler or Auto Mode. AKS has the cluster autoscaler and node auto-provisioning. GKE has the cluster autoscaler, and Autopilot provisions nodes per pod. OKE offers the Cluster Autoscaler as an add-on.

Test it on your system

Ramp the arrival rate the way your real peak ramps. A flat rate shows whether the HPA settles, but only a ramp shows whether it keeps up. Use k6’s ramping-arrival-rate:

import http from 'k6/http';

export const options = {
  scenarios: {
    morning_ramp: {
      executor: 'ramping-arrival-rate',
      startRate: Number(__ENV.BASE_RATE || 100),
      timeUnit: '1s',
      preAllocatedVUs: 300,
      maxVUs: 3000,
      stages: [
        { target: Number(__ENV.BASE_RATE || 100), duration: '5m' },
        { target: Number(__ENV.PEAK_RATE || 600), duration: __ENV.RAMP || '10m' },
        { target: Number(__ENV.PEAK_RATE || 600), duration: '15m' },
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(99)<500'],
  },
};

export default function () {
  http.get(`${__ENV.BASE_URL}/api/products`);
}
k6 run -e BASE_URL=https://staging.example.com -e BASE_RATE=100 -e PEAK_RATE=600 -e RAMP=10m hpa-ramp.js

Set RAMP to the steepest climb on your traffic graphs, then try a ramp twice as steep. The k6 breakpoint recipe covers how to find the rate where a single pod breaks.

Start staging at the replica count production runs before its peak. While the test runs, watch the HPA and the pods:

kubectl get hpa api -n shop -w
kubectl get pods -n shop -l app=api --field-selector=status.phase=Pending -w

Note the time from the first rise in latency to the first new pod reaching Ready. That is your real reaction time, and it feeds the target formula above.

Write your pass bar down first. For example, p99 stays under budget through the whole ramp, and no pod waits in Pending for more than two minutes. The HPA should never show ScalingLimited with TooManyReplicas. If p99 breaks during the ramp, lower the target or raise minReplicas. If pods sit in Pending, add placeholder room.

Keep it healthy

Scrape kube-state-metrics. These rules catch an HPA at its ceiling, an HPA that lost its metrics and pods that can’t find a node:

groups:
  - name: kubernetes-hpa
    rules:
      - alert: HPAAtMaxReplicas
        expr: |
          kube_horizontalpodautoscaler_status_current_replicas
            >= kube_horizontalpodautoscaler_spec_max_replicas
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} is at maxReplicas"
          description: "The HPA wants more pods than it may add. Check p99 for this service, then raise maxReplicas if the nodes and database connections allow it."
      - alert: HPAMetricsUnavailable
        expr: |
          kube_horizontalpodautoscaler_status_condition{condition="ScalingActive", status="false"} == 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} can't read its metric"
          description: "This service no longer scales. Run kubectl describe hpa for the error, then check metrics-server with kubectl top pods and the CPU requests on every container."
      - alert: HPADesiredAboveCurrent
        expr: |
          kube_horizontalpodautoscaler_status_desired_replicas
            > kube_horizontalpodautoscaler_status_current_replicas
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} has wanted more pods for 10 minutes"
          description: "New pods aren't arriving. Look for Pending pods and node autoscaler events, or a scale-up policy that limits the step size."
      - alert: PodsPendingTooLong
        expr: |
          sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Pods in {{ $labels.namespace }} have been Pending for 5 minutes"
          description: "The scheduler can't place them. Run kubectl describe pod on one and check the node autoscaler's logs for quota or capacity errors."

A few minutes at max at the top of a peak can be normal, so the rule waits 15 minutes. An hour there means the ceiling is too low.

Roll back

Keep the HPA manifest in Git, and save the live object before the change:

kubectl get hpa api -n shop -o yaml > hpa-api-before.yaml

Revert by applying the previous manifest from Git. Remove resourceVersion, uid and status from the saved copy before you apply it.

Roll back now if the new settings flap, with replicas rising and falling every few minutes. Do the same if a scale-up exhausts database connections or node quota. A lower target means more pods at every traffic level, so revert it in a quiet hour if cost is the only reason.

More recipes

Further reading