A pod disruption budget tells Kubernetes how many pods of a service it may take down on purpose at once. Without one, a node drain evicts every replica on that node together. With a budget that’s too strict, the drain never finishes and your cluster upgrade stalls.
What happens when a node dies or drains
Kubernetes splits disruptions into two types, and a PodDisruptionBudget (PDB) covers only one of them.
Voluntary disruptions go through the Eviction API: kubectl drain, node pool upgrades, the cluster autoscaler removing a node and Karpenter consolidation. The API server checks each eviction against the PDBs that select the pod. If the eviction would break the budget, the API returns 429 Too Many Requests, and the caller retries.
Involuntary disruptions skip that check: a node crash, a kernel panic, a spot or preemptible VM reclaim, a zone outage. A PDB can’t stop those. Replica spread across nodes and zones is your only defense.
Without a PDB, a drain evicts every pod on the node at once. The scheduler often packs several replicas of one Deployment onto the same node. Drain that node and the service loses all of them together. Clients get connection errors and 503s until the new pods start, pass readiness and enter the endpoints. That takes from seconds to minutes, depending on image pull and startup.
With a budget that allows no evictions, the drain hangs and prints this every 5 seconds:
error when evicting pods/"api-7c9f6b5d8-x2k4q" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
maxUnavailable: 0, a minAvailable equal to the replica count or a single-replica Deployment with minAvailable: 1 all cause it. A Deployment whose pods are already unready blocks it too, because unready pods don’t count as healthy.
Each managed service handles a blocked drain its own way:
| Platform | What happens to a blocked drain |
|---|---|
| EKS managed node groups | Fails the update after 15 minutes with PodEvictionFailure. The force flag on update-nodegroup-version deletes the pods instead. |
| GKE surge upgrades | Respects the PDB for up to one hour, then evicts the remaining pods by force. |
| AKS | Waits for the drain timeout, 30 minutes by default, then fails with UpgradeFailed. --undrainable-node-behavior Cordon quarantines the node instead. |
| OKE node pools | Waits for the eviction grace duration, 60 minutes by default. A node pool setting decides whether OKE then deletes the node anyway. |
AKS adds one more case. A platform-initiated upgrade of a cluster on an unsupported version can override a PDB that blocks a drain. GKE forces the eviction after an hour, and OKE can be set to, so a stuck budget doesn’t protect you for long.
The failover setup
Three parts work together: a budget, spread and readiness. Changed lines only, in the Deployment’s pod template:
spec:
replicas: 3
template:
metadata:
labels:
app: api
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: api
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: api
And the budget, as its own object:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api
namespace: shop
spec:
maxUnavailable: 1
unhealthyPodEvictionPolicy: AlwaysAllow
selector:
matchLabels:
app: api
maxUnavailable over minAvailable. maxUnavailable: 1 lets one pod go at a time, whatever the replica count. It keeps working as the HPA scales the Deployment up and down. minAvailable: 2 on a Deployment that the HPA shrinks to 2 blocks every drain. Percentages work too, and Kubernetes rounds them up. On my clusters every Deployment with two or more replicas gets maxUnavailable: 1 by default.
unhealthyPodEvictionPolicy. The default, IfHealthyBudget, refuses to evict an unready pod while the service is below its budget. A Deployment stuck in CrashLoopBackOff then blocks every drain on its nodes. AlwaysAllow lets the drain evict pods that aren’t ready, since they serve nothing. The Kubernetes docs recommend it, and it went stable in 1.31.
Replicas. A PDB on one replica can’t keep the service up through a drain. It either blocks the drain or allows an outage. Run at least two replicas, and three for anything on the request path. Set the HPA’s minReplicas to match.
Spread across nodes and zones. The hostname constraint keeps replicas on different nodes, so one drain or one crash takes at most one pod. The zone constraint spreads them across zones. I set it to ScheduleAnyway, so a full zone doesn’t block scheduling. DoNotSchedule on hostname needs at least as many nodes as replicas, or new pods wait in Pending.
Readiness. A PDB counts pods with the Ready condition as healthy. A pod with no readiness probe turns Ready the moment its container starts. The drain then moves on before the replacement can serve. Give every container a readiness probe that checks real serving. The graceful shutdown recipe covers probes and the SIGTERM drain, which every eviction triggers.
Stateful quorum. For etcd, ZooKeeper, Kafka controllers or a Patroni cluster, budget for quorum instead. Five members with a quorum of three get minAvailable: 3. Evict one at a time, and check that each member has rejoined before the next.
Node pool upgrade settings. Give the upgrade room to surge, so new nodes exist before old ones drain. EKS sets parallelism with the node group’s updateConfig. GKE uses max-surge-upgrade and max-unavailable-upgrade. AKS uses --max-surge. OKE replaces nodes through node pool cycling, with its own surge setting. Add a maintenance window on each provider, so drains happen when you’re watching.
Test the failover with kubectl drain and a pod disruption budget
Run this in staging first, under a fixed-rate load test at normal traffic. I kill pods, nodes or a database primary on purpose, in staging first, before I trust any failover setup in production.
Find a node that runs a replica, then drain it:
kubectl get pods -n shop -l app=api -o wide
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data --timeout=15m
--delete-emptydir-data deletes any emptyDir data on the node, so check what lives there first. In a second terminal, watch the budget and the pods:
kubectl get pdb api -n shop -w
kubectl get pods -n shop -l app=api -w
ALLOWED DISRUPTIONS should drop to 0 while the evicted pod restarts elsewhere, then return to 1 once the new pod is ready. Next, drain two nodes that hold replicas at the same time. The second drain should print the Cannot evict pod line and wait until the first replacement is ready. That proves the budget holds.
Then test the disruption a PDB can’t stop. Stop the VM behind one node from your cloud console or CLI, without draining it. The node goes NotReady, and its pods stay listed until the node lifecycle controller evicts them. That takes about five minutes with the default tolerations. Spread decides how much capacity you lose meanwhile.
Bring the drained nodes back with kubectl uncordon NODE_NAME.
Write your pass bar before you start. For example:
- The load test’s error rate stays inside your error budget during each drain.
- No more than one replica is ever unavailable, as shown by the PDB’s
ALLOWED DISRUPTIONS. - Each drain completes well inside your provider’s timeout from the table above.
- With one node stopped, the remaining replicas carry the load within your latency budget.
If the single drain causes errors, look at readiness and the SIGTERM drain. If it hangs, check kubectl get pdb -A for any budget with 0 allowed disruptions.
Keep it healthy
Scrape kube-state-metrics. These rules catch a budget that would block the next drain, a service already below its budget and a drain that stalled:
groups:
- name: kubernetes-disruption
rules:
- alert: PDBBlocksEviction
expr: |
kube_poddisruptionbudget_status_pod_disruptions_allowed == 0
and on (namespace, poddisruptionbudget)
kube_poddisruptionbudget_status_expected_pods > 0
for: 30m
labels:
severity: warning
annotations:
summary: "PDB {{ $labels.namespace }}/{{ $labels.poddisruptionbudget }} allows no disruptions"
description: "The next node drain or upgrade will hang on this service. Check for unready pods with kubectl get pods, or a budget set to maxUnavailable 0 or minAvailable equal to replicas."
- alert: PDBBelowDesiredHealthy
expr: |
kube_poddisruptionbudget_status_current_healthy
< kube_poddisruptionbudget_status_desired_healthy
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.poddisruptionbudget }} has fewer healthy pods than its budget"
description: "The service runs below its minimum. Find the unready or missing pods with kubectl describe pdb and kubectl get pods, and pause any drain or upgrade until it recovers."
- alert: NodeCordonedTooLong
expr: |
kube_node_spec_unschedulable == 1
for: 2h
labels:
severity: warning
annotations:
summary: "Node {{ $labels.node }} has been cordoned for 2 hours"
description: "A drain or upgrade may have stalled. Run kubectl get pdb -A to find a budget with 0 allowed disruptions, or uncordon the node if the work is finished."
PDBBlocksEviction fires for single-replica services that carry a budget, which is the point: either add a replica or drop the budget. Tune NodeCordonedTooLong to the longest maintenance you run on purpose.
Roll back
Save the budget before you change it:
kubectl get pdb api -n shop -o yaml > pdb-api-before.yaml
If a new budget blocks an upgrade or a drain that you need finished now, delete it and the drain continues on its next retry:
kubectl delete pdb api -n shop
Apply it again from Git once the drain completes. Spread constraints apply only to new pods, so reverting them doesn’t move running pods. Roll the Deployment in a quiet hour if you need the old placement back.
More recipes
- Kubernetes CPU Throttling with Spare CPU on the Node
Size CPU requests from real usage and give limits room above them, or drop them, so CFS quota stops pausing pods on a node with CPU to spare.
- Kubernetes Graceful Shutdown Without 502s on Deploy
Hold each old pod in a preStop sleep until the load balancers drop it, then drain open requests on SIGTERM, so a rollout ends with no 502s.
- HPA Scaling Behavior That Keeps Up with Peak Traffic
Scale on a metric that leads the load, lower the target and keep spare node room, so the HPA adds pods before peak traffic starts to queue.
- Fix OOMKilled and Exit Code 137 in Kubernetes
Set Kubernetes memory limits from measured peak usage plus headroom, and cap the runtime below the limit, so pods stop dying with exit code 137.