OCI Load Balancer 502 Bad Gateway and Idle Timeout

Hold backend keepalive above 310 seconds, size the OCI load balancer idle timeout and health checks for peak, so backends stop answering with 502s.

Published

Applies to OCI flexible Load Balancer with HTTP or HTTP2 listeners, including OKE Services of type LoadBalancer; not the Network Load Balancer

An OCI load balancer 502 Bad Gateway at peak tends to trace back to three settings. They are your backend’s keepalive timeout, the listener’s idle timeout and the health check. Oracle fixes some of the load balancer’s own timers, so most of the fix sits on your side of the connection.

What breaks at peak

Oracle documents three connection timers on the Load Balancer service. You can change one of them.

TimerValueCan you change it
Client to load balancer keepalive65 seconds idle, or 10,000 requestsNo
Load balancer to backend keepaliveCloses backend connections idle over 300 secondsNo
Listener idle timeout60 seconds for HTTP listeners, 300 for TCP, 7200 at mostYes

Your backend closes pooled connections first. The load balancer reuses connections to your backends and keeps idle ones for up to 300 seconds. Oracle says the service “doesn’t honor keep-alive settings from backend servers.” Most web servers close idle connections much sooner. nginx closes them after 75 seconds by default, Node.js after 5. If the backend closes a socket as the load balancer sends a request down it, that request fails with a 502. Oracle’s guidance, “To prevent possible 502 errors,” is that backends keep idle connections for at least 310 seconds. Peak traffic makes this race more likely, because there are more pooled connections and more requests to land on a closing one.

A slow response trips the idle timeout. This timer covers the gap between two reads or two writes during a request. A report that runs 90 seconds before it sends its first byte breaks a 60-second timeout. So does an upload where the backend stays silent too long. At peak, slow queries push more requests past the limit. The timer doesn’t cover idle time between requests on a kept-alive connection; the fixed 65-second keepalive does.

Health checks fail under load. By default the check runs every 10 seconds and waits 3 seconds for an answer. Three failures in a row mark a backend unhealthy. A busy backend that answers its health endpoint in 4 seconds drops out of rotation while it still serves traffic. Its load moves to the others, they slow down and drop out next. Once every backend in the set is unhealthy, the load balancer has nowhere to send requests. The backend’s status in the console or API names the reason:

OK
INVALID_STATUS_CODE
TIMED_OUT
REGEX_MISMATCH
CONNECT_FAILED
IO_ERROR
OFFLINE
UNKNOWN

TIMED_OUT at peak means a slow health endpoint. CONNECT_FAILED means a security list or network security group blocks the health checker, or the backend stopped listening. Check one backend from the CLI:

oci lb backend-health get \
  --load-balancer-id "$LB_ID" \
  --backend-set-name app-backends \
  --backend-name 10.0.10.15:8080

Oracle updates health status every three minutes and keeps no history, so read it while the problem is happening.

The OCI load balancer idle timeout and backend config

Keep backend connections open past 310 seconds. I set this before any load balancer change. It removes the 502s that the app logs can’t explain. These are the changed lines only:

# nginx, inside the http {} or server {} block the load balancer reaches
keepalive_timeout 320s;
// Node.js http.Server, before server.listen()
server.keepAliveTimeout = 320 * 1000;
server.headersTimeout = 321 * 1000; // keep it above keepAliveTimeout
// Go net/http
srv := &http.Server{
	Addr:        ":8080",
	Handler:     mux,
	ReadTimeout: 30 * time.Second,
	IdleTimeout: 320 * time.Second,
}
# Spring Boot, embedded Tomcat
server:
  tomcat:
    keep-alive-timeout: 320s

For Gunicorn with the gthread or an async worker, pass --keep-alive 320. For Uvicorn, pass --timeout-keep-alive 320. Each idle connection holds a file descriptor and some memory on the backend. The nginx tuning recipe shows how to size connection limits for that.

Size the listener idle timeout. Find the longest gap with no bytes sent during a real request, at peak. Look at your slowest report or export before its first byte, and at large uploads. Set the timeout a little above it. Keep it tight for API traffic: a long timeout lets hung requests hold backend workers. If one endpoint needs minutes, give it its own listener, so the API listener stays short. WebSocket and server-sent event connections need a timeout above their heartbeat interval.

This Terraform covers the listener, the backend set and its health check:

resource "oci_load_balancer_listener" "https" {
  load_balancer_id         = oci_load_balancer_load_balancer.app.id
  name                     = "https"
  default_backend_set_name = oci_load_balancer_backend_set.app.name
  port                     = 443
  protocol                 = "HTTP"

  connection_configuration {
    idle_timeout_in_seconds = 120
  }

  ssl_configuration {
    certificate_ids         = [var.certificate_id]
    verify_peer_certificate = false
  }
}

resource "oci_load_balancer_backend_set" "app" {
  load_balancer_id = oci_load_balancer_load_balancer.app.id
  name             = "app-backends"
  policy           = "LEAST_CONNECTIONS"

  health_checker {
    protocol          = "HTTP"
    port              = 8080
    url_path          = "/healthz"
    return_code       = 200
    interval_ms       = 10000
    timeout_in_millis = 5000
    retries           = 3
  }
}

Each setting explained:

  • idle_timeout_in_seconds is the listener idle timeout. It’s the one load balancer timer you control. Too low cuts off slow responses and uploads. Too high keeps hung requests open.
  • policy = "LEAST_CONNECTIONS" sends new requests to the backend with the fewest open connections. If request cost varies, it stops a backend stuck on slow requests from getting more of them.
  • url_path should point at a cheap endpoint that answers from memory. A health check that queries the database fails at the moment the database is busy and pulls healthy backends out with it.
  • timeout_in_millis must sit above that endpoint’s p99 at peak. The default is 3000 ms.
  • retries sets how many failures mark a backend unhealthy, and how many passes bring it back. With a 10-second interval and 3 retries, a dead backend takes about 30 seconds to leave rotation.

backend_max_connections caps connections from the load balancer to each backend, with a floor of 256 when set. It protects a backend from overload, but requests over the cap wait or fail at the load balancer. I leave it unset unless a backend falls over under concurrency.

On OKE, the cloud controller creates the load balancer from a Service of type LoadBalancer. Set the idle timeout with an annotation:

apiVersion: v1
kind: Service
metadata:
  name: web
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-connection-idle-timeout: "120"
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
    - port: 443
      targetPort: 8080

The controller can overwrite console edits to a load balancer that OKE manages. Make the change in the Service manifest. The pod side of a clean rollout lives in the graceful shutdown recipe.

Clients in front of the load balancer. The 65-second client keepalive is fixed. A service that calls this load balancer through its own connection pool should close idle connections in under 65 seconds. Otherwise it sends requests on sockets the load balancer has closed, and gets resets. The Node.js keepalive recipe covers that socket hang up case.

Test it on your system

Point the test at a staging load balancer with the same shape, listener settings and backend count as production. Two failures need two tests.

The keepalive race needs pooled connections that go idle, then a burst onto them. A flat load keeps every connection busy and hides it. Hold a fixed arrival rate with k6:

import http from 'k6/http';

export const options = {
  scenarios: {
    bursts: {
      executor: 'ramping-arrival-rate',
      startRate: 50, timeUnit: '1s',
      preAllocatedVUs: 500, maxVUs: 3000,
      stages: [
        { target: 600, duration: '2m' },  // your peak rate
        { target: 600, duration: '5m' },
        { target: 10, duration: '20s' },  // let pooled connections idle
        { target: 10, duration: '2m' },
        { target: 600, duration: '20s' }, // burst onto idle connections
        { target: 600, duration: '5m' },
      ],
    },
  },
  thresholds: { http_req_failed: ['rate<0.001'] },
};

export default function () {
  http.get('https://staging.example.com/api/products');
}

Run it with the old backend keepalive and again with 320 seconds. Set the rate and the threshold to your own peak and error budget.

The idle timeout needs a slow request. Call your slowest real endpoint at peak, or add a staging-only route that waits a set time before its first byte. Try delays below and above the listener’s idle timeout.

While both run, open Metrics Explorer for the oci_lbaas namespace. Watch HttpResponses502, HttpResponses504, BackendTimeouts and UnHealthyBackendServers for the backend set. Run the oci lb backend-health get command above every minute, and tail the backend’s own logs.

Set your pass criteria before the run. The change passes when 502s stay inside your error budget through the burst after the quiet period. No backend should go unhealthy while it serves traffic. Requests longer than the idle timeout should fail where you expect them to, and nowhere else.

Keep it healthy

OCI publishes these metrics to its Monitoring service, under oci_lbaas. I didn’t find a maintained Prometheus exporter for that namespace, so the load balancer alarms below live in OCI Monitoring. Oracle’s metrics reference prints the names in camelCase, while Grafana’s published OCI dashboard queries use names such as HttpResponses502. Copy the exact name from Metrics Explorer in your tenancy before you apply this:

resource "oci_monitoring_alarm" "lb_502" {
  compartment_id        = var.compartment_id
  metric_compartment_id = var.compartment_id
  display_name          = "OCI LB 502 responses"
  namespace             = "oci_lbaas"
  query                 = "HttpResponses502[1m].sum() > 10"
  pending_duration      = "PT5M"
  severity              = "CRITICAL"
  destinations          = [oci_ons_notification_topic.oncall.id]
  is_enabled            = true
  body                  = "502s from the load balancer. Check backend keepalive (must exceed 310 s) and backend health with oci lb backend-health get."
}

resource "oci_monitoring_alarm" "lb_unhealthy" {
  compartment_id        = var.compartment_id
  metric_compartment_id = var.compartment_id
  display_name          = "OCI LB unhealthy backends"
  namespace             = "oci_lbaas"
  query                 = "UnHealthyBackendServers[1m].max() > 0"
  pending_duration      = "PT5M"
  severity              = "WARNING"
  destinations          = [oci_ons_notification_topic.oncall.id]
  is_enabled            = true
  body                  = "A backend failed its health check. Read its status (TIMED_OUT, CONNECT_FAILED) and check the health endpoint's latency under load."
}

Set the > 10 threshold from your own traffic and error budget. Add a third alarm on BackendTimeouts if your listener idle timeout is tight.

In Prometheus, probe through the load balancer from outside with blackbox_exporter. The backend’s own metrics never count a 502 that the load balancer generated, so a probe sees what users see. In blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      preferred_ip_protocol: ip4
groups:
  - name: oci-load-balancer-probe
    rules:
      - alert: OciLbProbeFailing
        expr: avg_over_time(probe_success{job="blackbox-oci-lb"}[5m]) < 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Probes through the OCI load balancer to {{ $labels.instance }} are failing"
          description: "Check probe_http_status_code. A 502 points to backend keepalive or health; run oci lb backend-health get for each backend."
      - alert: OciLbProbe502
        expr: max_over_time(probe_http_status_code{job="blackbox-oci-lb"}[10m]) == 502
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "The OCI load balancer returned 502 to a probe of {{ $labels.instance }}"
          description: "Raise backend keepalive above 310 seconds if it isn't already, then check the backend set's health status in the console."

The rules expect a scrape job named blackbox-oci-lb that probes your public URL through the load balancer. A probe every 15 to 30 seconds catches a sustained problem. It won’t catch a scatter of keepalive 502s, which is why the OCI alarm on HttpResponses502 stays.

Roll back

Save the current load balancer config first:

oci lb load-balancer get --load-balancer-id "$LB_ID" > lb-before.json

With Terraform, revert the commit and apply. On OKE, revert the Service annotation. With the CLI, oci lb listener update takes the full listener definition. Pass every setting from lb-before.json, SSL included:

oci lb listener update \
  --load-balancer-id "$LB_ID" \
  --listener-name https \
  --default-backend-set-name app-backends \
  --port 443 --protocol HTTP \
  --connection-configuration-idle-timeout 60 \
  --ssl-certificate-ids '["ocid1.certificate.oc1..example"]' \
  --force

The backend keepalive stays at 320 seconds even if you revert the rest. Lowering it brings the 502 race back. If a longer idle timeout lets hung requests pile up on your backends, roll it back now. For any other reason, revert when traffic is lowest, so a mistake reaches fewer users.

More recipes

Further reading