Kubernetes graceful shutdown needs work from you, and most apps skip it. Each rollout then kills pods that still receive traffic, and a burst of 502s follows every deploy. The fix is a short preStop sleep, a drain on SIGTERM and a grace period that covers both.
What happens when a pod goes away
A rolling update deletes old pods one batch at a time. A node drain and a scale-down do the same. Deleting a pod starts two processes in parallel, and nothing makes one wait for the other:
- The kubelet stops the pod. It runs the
preStophook if there is one, then sends SIGTERM to each container. AfterterminationGracePeriodSeconds, 30 by default, it sends SIGKILL to anything still running. - The control plane removes the pod from traffic. The EndpointSlice marks the endpoint as terminating and not ready. Then kube-proxy on every node, your ingress controller and any cloud load balancer each pick up that change on their own schedule.
The second path takes longer. kube-proxy rewrites rules on every node, and an ingress controller reloads its upstream list. A cloud load balancer deregisters the target through its API. Most runtimes exit on SIGTERM by default. Your app can be gone while load balancers still send it new connections.
The proxy in front of the pod logs the failure. In nginx and ingress-nginx you’ll find lines like these:
connect() failed (111: Connection refused) while connecting to upstream
upstream prematurely closed connection while reading response header from upstream
recv() failed (104: Connection reset by peer) while reading response header from upstream
The first comes from new connections to a pod that stopped listening. The other two come from requests in flight, or from keep-alive connections the app closed under the proxy. An AWS Application Load Balancer counts the same failures in its HTTPCode_ELB_502_Count metric.
A default setup recovers in seconds, once every proxy has the new endpoint list. A deploy rolls through every pod, though, so the error window opens once per batch. Slow apps hit the other edge: an app still draining after 30 seconds gets SIGKILL, and its container exits with code 137. An app that finishes its drain and exits shows 0, or 143 if the signal ended it.
New pods can cause errors too. A pod joins the endpoints as soon as its readiness probe passes. A probe that passes before caches warm, or a pod with no readiness probe at all, takes traffic it can’t serve yet.
The preStop hook and graceful shutdown setup
Changed lines only, in the Deployment spec:
spec:
minReadySeconds: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0
template:
spec:
terminationGracePeriodSeconds: 45
containers:
- name: app
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
failureThreshold: 2
lifecycle:
preStop:
sleep:
seconds: 10
preStop sleep. The kubelet runs the hook before it sends SIGTERM, so your app keeps serving while the proxies catch up. The sleep action needs Kubernetes 1.30 or later, and it runs inside the kubelet. On older clusters use exec with command: ["sleep", "10"], which needs a sleep binary in the image. Distroless and scratch images don’t have one. Set the sleep longer than your slowest proxy takes to drop an endpoint. I start at 10 seconds and raise it if the test below still shows refused connections.
Drain on SIGTERM. Once the sleep ends, the app gets SIGTERM. It should stop accepting connections, finish the requests it holds, close idle keep-alive connections and exit. In Go, http.Server.Shutdown does all of that:
package main
import (
"context"
"errors"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) })
srv := &http.Server{Addr: ":8080", Handler: mux}
go func() {
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatal(err)
}
}()
stop := make(chan os.Signal, 1)
signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
<-stop
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Printf("drain cut short: %v", err)
}
log.Print("shutdown complete")
}
In Node.js, call server.close() to stop new connections and server.closeIdleConnections() to drop idle keep-alive sockets. The Node.js cluster mode recipe shows a full drain across workers. Run your app as PID 1, or behind an init such as tini. A shell wrapper or npm start can swallow SIGTERM.
Grace period. Kubernetes counts the preStop sleep against terminationGracePeriodSeconds. Size it as the sleep, plus your drain timeout, plus a few seconds of margin. The example fits: 10 seconds of sleep, a 25-second drain, 45 seconds in total. If the hook still runs at the deadline, the kubelet grants a one-off 2-second extension, then kills the pod.
Readiness probe. Point it at a path that checks the app can serve, not at a static file. The pod enters the endpoints only after the probe passes, so a slow-starting app needs it most. Kubernetes already marks a terminating endpoint as not ready, so the app doesn’t need to fail its probe on SIGTERM.
Rollout strategy. maxUnavailable: 0 keeps full capacity during the rollout, and maxSurge adds new pods first. minReadySeconds makes each new pod stay ready for 10 seconds before the rollout continues. That catches a pod that passes one probe and then crashes.
Keep-alive timeouts. A proxy that reuses connections must close them before the app does. Set the proxy’s idle timeout below the app’s keep-alive timeout. In ingress-nginx, upstream-keepalive-timeout defaults to 60 seconds. Node.js closes idle sockets after 5 seconds by default, which breaks that rule. The nginx upstream timeouts recipe covers this race outside of deploys.
Cloud load balancers. A load balancer that sends traffic straight to pod IPs has its own deregistration step. Each provider sets it in a different place:
- EKS. With the AWS Load Balancer Controller and
target-type: ip, label the namespaceelbv2.k8s.aws/pod-readiness-gate-inject: enabled. The controller then holds old pods until new ones are healthy in the target group. The ALB’s deregistration delay defaults to 300 seconds. Lower it to a little above your longest request withalb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30. - GKE. With container-native load balancing, set
connectionDraining.drainingTimeoutSecin aBackendConfig. The default of 0 turns draining off. Gateway users set the same timeout in aGCPBackendPolicy. - AKS. With the Application Gateway Ingress Controller, add the annotations
appgw.ingress.kubernetes.io/connection-draining: "true"andappgw.ingress.kubernetes.io/connection-draining-timeout: "30". - OKE. A
LoadBalancerService sends traffic to node ports by default, so the endpoint change happens in kube-proxy and your ingress controller. The preStop sleep covers it.
The drain timeout on the load balancer should fit inside terminationGracePeriodSeconds, or the pod dies while the load balancer still waits on it.
Test the failover
Run this in staging first, under load. Start a fixed-rate k6 test at your normal traffic, then restart the Deployment in the middle of it:
k6 run -e BASE_URL=https://staging.example.com -e RATE=100 rollout-test.js &
sleep 60
kubectl rollout restart deploy/api -n shop
kubectl rollout status deploy/api -n shop --timeout=10m
wait
rollout-test.js can be the constant-arrival-rate script from the CPU throttling recipe, with a 5-minute duration. While it runs, watch the endpoints change in a second terminal:
kubectl get endpointslices -n shop -l kubernetes.io/service-name=api -w
Tail the ingress or proxy log for the three error lines above, and the app logs for shutdown complete. Then repeat with a node drain, since that removes several pods at once. The PodDisruptionBudget recipe covers how to keep a drain to one pod at a time.
A clean rollout, by the bar you set before the test, looks like this:
- k6 reports no failed requests, or fewer than your error budget allows.
- No proxy logs
Connection refusedorprematurely closedduring the rollout. - Every old pod logs
shutdown completebefore its grace period ends. kubectl get events -n shop --field-selector reason=FailedPreStopHookreturns nothing.
If refused connections remain, raise the sleep. If resets remain, check keep-alive timeouts and the drain. If old pods never log completion, your drain runs past the grace period.
Keep it healthy
Scrape ingress-nginx, or whatever proxy fronts the pods, and kube-state-metrics. These rules catch a 5xx burst, a rollout that drags on and one that stops moving:
groups:
- name: kubernetes-rollouts
rules:
- alert: Ingress5xxBurst
expr: |
sum by (ingress) (rate(nginx_ingress_controller_requests{status=~"5.."}[2m]))
/ sum by (ingress) (rate(nginx_ingress_controller_requests[2m]))
> 0.01
for: 1m
labels:
severity: critical
annotations:
summary: "Ingress {{ $labels.ingress }} returns over 1% 5xx"
description: "Run kubectl rollout history on the backing Deployment. If a rollout started in the last few minutes, check the preStop sleep, the SIGTERM drain and terminationGracePeriodSeconds."
- alert: DeploymentRolloutInProgress
expr: |
kube_deployment_status_replicas_updated != kube_deployment_spec_replicas
for: 30m
labels:
severity: warning
annotations:
summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has been mid-rollout for 30 minutes"
description: "Old and new pods are both serving. Run kubectl rollout status, and check for old pods stuck in Terminating past their grace period."
- alert: DeploymentRolloutStuck
expr: |
kube_deployment_status_condition{condition="Progressing", status="false"} == 1
for: 5m
labels:
severity: warning
annotations:
summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} stopped progressing"
description: "New pods aren't becoming ready within progressDeadlineSeconds. Run kubectl rollout status and kubectl describe on a new pod to find the failing probe or crash."
nginx_ingress_controller_requests comes from ingress-nginx. The project was retired in March 2026, but plenty of clusters still run it. On another proxy, swap in its request counter with a status label. Set the 1% line from the error budget of the service. I’d keep Ingress5xxBurst on a short for, because a deploy’s error window lasts only seconds per batch.
Roll back
Save the live Deployment before you change it:
kubectl get deploy api -n shop -o yaml > api-deploy-before.yaml
To revert, apply the previous manifest from Git or Helm. kubectl rollout undo deploy/api -n shop also works, but it returns the whole pod template to the last revision, image included. Use it only if the image hasn’t changed since.
Roll back now if the new probe or hook makes pods fail to start or the rollout stalls. A longer grace period has one cost: a stuck pod takes longer to die. For any other revert, wait for a quiet window, since the change itself triggers a rollout.
More recipes
- Kubernetes CPU Throttling with Spare CPU on the Node
Size CPU requests from real usage and give limits room above them, or drop them, so CFS quota stops pausing pods on a node with CPU to spare.
- HPA Scaling Behavior That Keeps Up with Peak Traffic
Scale on a metric that leads the load, lower the target and keep spare node room, so the HPA adds pods before peak traffic starts to queue.
- Fix OOMKilled and Exit Code 137 in Kubernetes
Set Kubernetes memory limits from measured peak usage plus headroom, and cap the runtime below the limit, so pods stop dying with exit code 137.
- Pod Disruption Budget That Survives Node Drains
Give each service a PodDisruptionBudget and spread its pods across nodes and zones, so drains and upgrades evict one pod at a time.