A GCP load balancer 502 at peak tends to come with one of two statusDetails values in the request log: backend_timeout or backend_connection_closed_before_data_sent_to_client. Both come from settings you control. One is the backend service timeout. The other is how long your web server keeps an idle connection open.
What breaks at peak
The external Application Load Balancer runs two timers that matter here. Google documents both:
- The backend service timeout (
timeoutSec). The load balancer waits this long for the backend’s full response. Except for serverless NEGs, it defaults to 30 seconds. - The backend keepalive timeout. The load balancer holds idle connections to your backends for 600 seconds. You can’t change it on any mode.
Each timer has its own failure, and peak traffic brings both out.
Slow responses hit timeoutSec. At peak your database and downstream calls slow down. A report or checkout that took 20 seconds now takes 35. The load balancer gives up at 30 and logs this in the request’s jsonPayload.statusDetails:
backend_timeout
Google’s description reads “The backend timed out while generating a response.” The client gets a 502 or 503. The global and regional modes also retry a failed GET with no body once by default. Your backend logs can show the same slow request twice.
Your backend closes idle connections first. The load balancer expects a pooled connection to stay open for 600 seconds. Most web servers close idle connections much sooner. nginx defaults keepalive_timeout to 75 seconds, and Node.js closes them after 5. If your server closes the socket at the moment the load balancer sends a request on it, the request fails with:
backend_connection_closed_before_data_sent_to_client
Google describes it as “The backend unexpectedly closed its connection to the load balancer before the response was proxied to the client.” Peak traffic makes it worse, with more pooled connections and more requests racing each close. You see a scatter of 502s with no slow requests behind them, and the app’s own logs show nothing wrong.
Three other values show up in the same log, and each points somewhere else:
statusDetails | What it means | Where to look |
|---|---|---|
failed_to_pick_backend | No healthy backend to send the request to | Health checks, instance group size, autoscaler lag |
failed_to_connect_to_backend | The TCP or TLS connect failed, including a connect timeout | Firewall rules for the health check ranges, backend listen queue |
backend_early_response_with_non_error_status | The backend answered before it read the whole request body | Upload handlers that reply early |
Count them before you change anything:
gcloud logging read \
'resource.type="http_load_balancer" AND httpRequest.status>=500' \
--freshness=1d --limit=5000 \
--format='value(jsonPayload.statusDetails)' | sort | uniq -c | sort -rn
Request logging has to be on for the backend service. Turn it on with gcloud compute backend-services update web-backend --global --enable-logging --logging-sample-rate=1.0. Lower the rate on busy services once you have a baseline.
The Google Cloud load balancer timeout and keepalive config
The fix has two halves. Raise timeoutSec on the load balancer to fit your slowest legitimate request. Then make every backend keep idle connections open longer than the load balancer does.
Size the backend service timeout. Take the p99 latency of your slowest endpoint at peak, add headroom, and stay under what the client will wait. I pull the backend latency distribution from Cloud Monitoring (loadbalancing.googleapis.com/https/backend_latencies) before I pick a number.
gcloud compute backend-services update web-backend --global --timeout=60s
Keep it as short as your slowest real request allows. A long timeout lets a stuck request hold a worker for the whole period, and a retried GET can hold a second one. If one endpoint needs minutes, such as an export, give it its own backend service with its own timeout. Then the rest of the site keeps the tight one.
On the regional external Application Load Balancer, the URL map’s routeAction.timeout replaces the backend service timeout when you set it. Change it there. For WebSocket traffic, timeoutSec caps how long a socket can stay open, so size it for your longest session.
Set the client-side keepalive. This timer covers connections from browsers to the load balancer. It defaults to 610 seconds, and the global and regional modes accept 5 to 1200. The classic mode doesn’t let you change it. I leave it alone unless a client sits behind a proxy with a shorter idle timeout.
gcloud compute target-https-proxies update web-https-proxy \
--http-keep-alive-timeout-sec=610
Keep backend connections open past 600 seconds. Google’s guidance is to set the backend’s HTTP keepalive above 600 seconds, and its examples use 620. The extra 20 seconds means the load balancer always closes first. These are the changed lines only, per server:
# nginx, inside http {} or the server {} block the load balancer hits
keepalive_timeout 620s;
# Apache httpd
KeepAlive On
KeepAliveTimeout 620
// Node.js http.Server, set before server.listen()
server.keepAliveTimeout = 620 * 1000;
server.headersTimeout = 621 * 1000; // keep it above keepAliveTimeout
// Go net/http
srv := &http.Server{
Addr: ":8080",
Handler: mux,
ReadTimeout: 30 * time.Second,
IdleTimeout: 620 * time.Second,
}
# Uvicorn
uvicorn app:app --host 0.0.0.0 --port 8080 --timeout-keep-alive 620
# Gunicorn with gthread or an async worker class
gunicorn app:app --worker-class gthread --threads 8 --keep-alive 620
# Spring Boot, embedded Tomcat
server:
tomcat:
keep-alive-timeout: 620s
Gunicorn’s default sync worker doesn’t hold keepalive connections, so the race doesn’t apply there. Each idle connection costs a file descriptor and a little memory per worker. Check your worker_connections or file limit covers them, as the nginx tuning recipe explains. The same race shows up behind any load balancer, and the Node.js keepalive recipe covers the socket hang up side of it. The OCI load balancer recipe has the Oracle Cloud numbers.
On GKE, the Ingress controller creates the backend service, so set the timeout through a BackendConfig and attach it to the Service:
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: web-backendconfig
spec:
timeoutSec: 60
connectionDraining:
drainingTimeoutSec: 30
---
apiVersion: v1
kind: Service
metadata:
name: web
annotations:
cloud.google.com/backend-config: '{"default": "web-backendconfig"}'
spec:
type: ClusterIP
selector:
app: web
ports:
- port: 80
targetPort: 8080
A gcloud change to a backend service that the Ingress controller manages gets overwritten on its next sync. Make the change in the BackendConfig. drainingTimeoutSec gives in-flight requests time to finish when a pod leaves the backend. The graceful shutdown recipe covers the pod side of that.
Health checks. failed_to_pick_backend at peak often means health checks failed because the app was busy, not down. Point the check at a cheap endpoint that doesn’t touch the database. Give it a timeout above the endpoint’s p99 under load.
Test it on your system
Run the test against a staging load balancer with the same backend type, instance size and timeouts as production. Two failures need two tests.
The keepalive race needs connections that go idle between bursts. A flat load keeps every pooled connection busy and hides it. This k6 script alternates peak and quiet periods at a fixed arrival rate:
import http from 'k6/http';
export const options = {
scenarios: {
bursts: {
executor: 'ramping-arrival-rate',
startRate: 50, timeUnit: '1s',
preAllocatedVUs: 500, maxVUs: 3000,
stages: [
{ target: 800, duration: '2m' }, // your peak rate
{ target: 800, duration: '5m' },
{ target: 20, duration: '30s' }, // let connections idle
{ target: 20, duration: '3m' },
{ target: 800, duration: '30s' }, // burst onto idle connections
{ target: 800, duration: '5m' },
],
},
},
thresholds: { http_req_failed: ['rate<0.001'] },
};
export default function () {
http.get('https://staging.example.com/api/products');
}
Run it once with the old backend keepalive and once with 620 seconds. Set target to your own peak and the threshold to your own error budget.
The backend timeout needs a slow endpoint. Hit your slowest real endpoint at peak rate while the database runs its normal peak load. If you can’t reproduce that, add a staging-only route that sleeps for a set time. Then call it with delays a second under and a second over timeoutSec.
While both run, count statusDetails with the gcloud logging read command above. Watch backend latency in Cloud Monitoring and your web server’s own error log.
Decide the pass criteria before the run. The new config passes when backend_connection_closed_before_data_sent_to_client disappears and backend_timeout appears only for requests you expect to exceed the timeout. The 5xx rate should stay inside your error budget through the burst after the quiet period.
Keep it healthy
Cloud Monitoring holds the load balancer metrics. To alert on them in Prometheus, run stackdriver_exporter with a service account that has roles/monitoring.viewer. The other route is Google Cloud Managed Service for Prometheus, which queries Cloud Monitoring metrics in PromQL under different names. The rules below use the exporter’s names.
The statusDetails value lives in the logs and not in the metrics, so turn it into a log-based metric first. Save this as lb-5xx-metric.yaml:
name: lb_5xx_by_status_details
description: External Application Load Balancer 5xx responses by statusDetails
filter: resource.type="http_load_balancer" AND httpRequest.status>=500
metricDescriptor:
metricKind: DELTA
valueType: INT64
labels:
- key: status_details
valueType: STRING
labelExtractors:
status_details: EXTRACT(jsonPayload.statusDetails)
gcloud logging metrics create lb_5xx_by_status_details --config-from-file=lb-5xx-metric.yaml
stackdriver_exporter \
--google.project-ids=my-project \
--monitoring.metrics-prefixes=loadbalancing.googleapis.com/https/request_count \
--monitoring.metrics-prefixes=loadbalancing.googleapis.com/https/backend_latencies \
--monitoring.metrics-prefixes=logging.googleapis.com/user/lb_5xx_by_status_details \
--monitoring.aggregate-deltas
The exporter names each metric stackdriver_<resource type>_<metric type>. Google files log-based metrics for this load balancer under the l7_lb_rule resource. --monitoring.aggregate-deltas turns Google’s per-minute DELTA values into counters, so rate() and increase() work. Cloud Monitoring publishes these metrics a few minutes late, so the for windows below run longer than they would for a local exporter.
groups:
- name: gcp-load-balancer
rules:
- alert: GcpLbBackendClosedConnection
expr: |
sum by (project_id) (increase(stackdriver_l7_lb_rule_logging_googleapis_com_user_lb_5xx_by_status_details{status_details="backend_connection_closed_before_data_sent_to_client"}[10m])) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "Backends are closing idle connections before the load balancer does"
description: "Set the backend HTTP keepalive above 600 seconds (620 is Google's example) in nginx, Node.js, Gunicorn or your server, then redeploy."
- alert: GcpLbBackendTimeouts
expr: |
sum by (project_id) (increase(stackdriver_l7_lb_rule_logging_googleapis_com_user_lb_5xx_by_status_details{status_details="backend_timeout"}[10m])) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "Requests are hitting the backend service timeout"
description: "Find the slow endpoint in the request logs. Fix its latency, move it to its own backend service, or raise timeoutSec to cover its real p99."
- alert: GcpLb5xxRateHigh
expr: |
sum by (project_id, url_map_name) (rate(stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_request_count{response_code_class="500"}[10m]))
/
sum by (project_id, url_map_name) (rate(stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_request_count[10m]))
> 0.01
for: 10m
labels:
severity: critical
annotations:
summary: "Load balancer {{ $labels.url_map_name }} is serving over 1% 5xx"
description: "Count statusDetails with gcloud logging read for the last 15 minutes. backend_timeout and connection closes have their own fixes in the cookbook recipe."
- alert: GcpLbBackendLatencyNearTimeout
# 48000 ms is 80% of a 60 s timeoutSec; set it from your own timeout
expr: |
histogram_quantile(0.99,
sum by (project_id, backend_target_name, le) (rate(stackdriver_https_lb_rule_loadbalancing_googleapis_com_https_backend_latencies_bucket[10m]))
) > 48000
for: 15m
labels:
severity: warning
annotations:
summary: "p99 backend latency on {{ $labels.backend_target_name }} is near timeoutSec"
description: "Backend latency is in milliseconds. Profile the slow endpoints before peak, or split long requests onto their own backend service."
Set the 1% threshold from your own error budget. The latency rule assumes the 60-second timeout from the fix, so change 48000 when you change timeoutSec.
Roll back
Save the current config before you touch it:
gcloud compute backend-services describe web-backend --global --format=yaml > web-backend-before.yaml
gcloud compute target-https-proxies describe web-https-proxy --format=yaml > web-proxy-before.yaml
To roll back, set the old values from those files:
gcloud compute backend-services update web-backend --global --timeout=30s
gcloud compute target-https-proxies update web-https-proxy --http-keep-alive-timeout-sec=610
On GKE, revert the BackendConfig and let the controller sync it. For the web server, restore its old keepalive value and reload.
I leave the backend keepalive at 620 even when I revert the rest. A higher value costs idle sockets, and the race comes back the moment you lower it. If the new timeoutSec lets slow requests pile up and exhaust your workers, roll it back now and split the slow endpoint out. For any other reason, revert during a low-traffic window, so a mistake hits fewer users.
More recipes
- Cloud NAT Port Exhaustion Alerts in Prometheus
Alert on Cloud NAT port usage per VM and on OUT_OF_RESOURCES dropped packets, so you see port exhaustion before outbound calls start timing out.
- Fix Cloud Run "No Available Instance" Errors
Set Cloud Run concurrency from one instance's real limit and size min and max instances for your peak, so cold starts and scaling stop dropping requests.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.