An ALB 502 Bad Gateway at peak often comes from a race. A target closes a keep-alive connection while the load balancer is sending a request down it. An ALB 504 means the target took longer than the idle timeout to answer. Both come from one setting on the load balancer and one on your app server, and both are cheap to fix.
What breaks at peak
Your users get this page from the load balancer, with no trace in your app logs:
502 Bad Gateway
504 Gateway Time-out
The load balancer keeps connections to each target open and reuses them. It closes one after it sits idle for the idle timeout, which is 60 seconds by default. Your app server has its own keep-alive timer for the same connection. Node.js closes an idle connection after 5 seconds, Gunicorn after 2 and uvicorn after 5.
At low traffic the gap seldom bites. At peak, the load balancer reuses idle connections all the time. Sooner or later it sends a request on a connection the target is closing that same millisecond. The target answers with a FIN or a TCP RST, and the load balancer returns a 502. AWS lists this cause in its troubleshooting guide: “The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target.”
The other 502 causes on that list are worth a look before you change anything:
- A target that resets new connections, for example a full listen backlog or a crashed process.
- A deregistered target still serving a request when the deregistration delay runs out. Rolling deploys hit this one. The Kubernetes graceful shutdown recipe covers the pod side.
- A Lambda target that errors, times out or gets throttled. The Lambda concurrency recipe covers throttling.
- A response header block over 32 KB, or a malformed response.
A 504 has two common causes. The target accepted the request but didn’t start answering before the idle timeout ran out. Or the load balancer couldn’t open a connection within its 10-second connect timeout. That points at an overloaded target or a network ACL.
Tell the load balancer’s errors from your app’s. CloudWatch splits them. HTTPCode_ELB_502_Count and HTTPCode_ELB_504_Count count errors the load balancer made up. HTTPCode_Target_5XX_Count counts 5xx responses your targets sent. If the ELB counters climb while the target counter stays flat, the fix lies in the timeout settings below. If the target counter climbs, read your app logs first.
ALB access logs record each request with elb_status_code and target_status_code. A 502 with a target status of - means the target never sent a valid response. Turn access logs on before the next peak so you have them when you need them.
The ALB idle timeout and keepalive fix
The rule: every target’s keep-alive timeout must be longer than the ALB idle timeout. Then the load balancer always closes the idle connection first, and it never sends a request down a connection the target is closing. AWS recommends the same order in its attribute docs.
Check the current idle timeout first and save it:
ALB_ARN=arn:aws:elasticloadbalancing:eu-west-1:123456789012:loadbalancer/app/web/50dc6c495c0c9188
aws elbv2 describe-load-balancer-attributes --load-balancer-arn "$ALB_ARN" \
--query "Attributes[?Key=='idle_timeout.timeout_seconds' || Key=='client_keep_alive.seconds']" \
> alb-attributes-before.json
cat alb-attributes-before.json
Leave the idle timeout at 60 seconds unless a real request needs longer. Then set each app server’s keep-alive above it. I use 65 seconds against a 60-second idle timeout, so a few seconds of clock drift can’t reverse the order.
Node.js, changed lines only, in the file that creates the server:
const server = app.listen(3000);
server.keepAliveTimeout = 65_000; // above the ALB idle timeout of 60 s
server.headersTimeout = 66_000; // keep this above keepAliveTimeout
keepAliveTimeout sets how long Node.js holds an idle connection open. Too low, and Node.js closes connections the load balancer still plans to reuse. headersTimeout caps how long Node.js waits for a full request header. Keep it above keepAliveTimeout, so Node.js doesn’t cut off a request the load balancer is still sending. Express, Fastify and NestJS all sit on this server object. The Node.js keep-alive recipe covers the same timer from the client side. On Oracle Cloud, the OCI load balancer recipe covers the same race, with a backend keepalive that Oracle fixes at 300 seconds.
nginx as the target, changed lines only, in the http or server block:
keepalive_timeout 75s;
keepalive_requests 10000;
nginx defaults to 75 seconds, which already beats a 60-second idle timeout. Check that nobody lowered it. keepalive_requests closes a connection after that many requests. A low value forces the load balancer to reconnect often, and each close is another chance for the race.
Gunicorn, in gunicorn.conf.py:
keepalive = 65 # seconds; above the ALB idle timeout
timeout = 30 # worker timeout; see the 504 section below
keepalive sets how long a worker holds an idle keep-alive connection. It only applies to the threaded and async worker classes. The sync worker closes after every response, which avoids the race but costs a new connection per request. uvicorn takes --timeout-keep-alive 65.
Go net/http, changed lines only:
srv := &http.Server{
Addr: ":8080",
Handler: mux,
IdleTimeout: 65 * time.Second,
ReadTimeout: 30 * time.Second,
}
IdleTimeout closes idle keep-alive connections. If you leave it at zero, Go falls back to ReadTimeout, which is often shorter than 60 seconds.
The 504 side. A 504 from an idle timeout means a request ran longer than the idle timeout without sending a byte. Raise the idle timeout only for endpoints that need it, such as exports or uploads. Keep those on their own load balancer if you can. A higher idle timeout on everything means slow requests hold connections longer at peak. For normal requests, find the slow handler. A request that needs more than a minute belongs in a background job.
If you do raise it, raise the targets’ keep-alive with it. The rule holds at any value:
aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" \
--attributes Key=idle_timeout.timeout_seconds,Value=120
The valid range is 1 to 4000 seconds. Deploy the longer target keep-alive first, then change the load balancer.
Client keep-alive. client_keep_alive.seconds is a different setting. It caps how long the load balancer keeps a client’s connection, 3600 seconds by default and never below 60. It doesn’t touch the target side, so leave it alone for this fix.
Deregistration delay. The target group’s deregistration_delay.timeout_seconds defaults to 300 seconds. Set it above your slowest normal request, and make your app finish in-flight requests on SIGTERM. A delay shorter than a slow request turns every deploy into a burst of 502s.
Health checks. A health check that runs on the same busy workers can fail under load and pull healthy targets out. Their traffic then lands on the rest. Point the health check at a cheap path that doesn’t touch the database.
Test it on your system
Run k6 against a staging ALB with the same target type and the same app server settings as production. Use a fixed arrival rate, so a slow target can’t slow the test down and hide errors:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: 300, // requests per second; set this to your peak
timeUnit: '1s',
duration: '15m',
preAllocatedVUs: 200,
maxVUs: 1000,
},
},
thresholds: {
http_req_failed: ['rate<0.001'],
http_req_duration: ['p(99)<800'],
},
};
export default function () {
const res = http.get(`${__ENV.BASE_URL}/api/products`);
check(res, { 'no 502 or 504': (r) => r.status !== 502 && r.status !== 504 });
}
k6 run -e BASE_URL=https://staging.example.com alb-peak.js
The keep-alive race shows up between bursts, when connections sit idle and then get reused. A steady rate can miss it. Run a second test with ramping-arrival-rate that drops to a low rate for longer than your app’s old keep-alive timeout, then climbs back to peak. Repeat that a few times.
While it runs, watch three things:
HTTPCode_ELB_502_Count,HTTPCode_ELB_504_CountandHTTPCode_Target_5XX_Countin CloudWatch, at 1-minute resolution.TargetConnectionErrorCount, which counts connections the load balancer failed to open to a target.- The access log rows with
elb_status_code502 or 504, and theirtarget_processing_time.
Set the thresholds above to your own error budget and p99 target. The test passes when the ELB 502 and 504 counters stay at zero through the idle gaps. The target 5xx count should match what your app logs. Run the same test before the change too, so you know the race was there.
Keep it healthy
CloudWatch metrics reach Prometheus through YACE, the yet-another-cloudwatch-exporter from prometheus-community. This job pulls the ALB counters:
apiVersion: v1alpha1
discovery:
jobs:
- type: AWS/ApplicationELB
regions: [eu-west-1]
period: 60
length: 300
nilToZero: true
metrics:
- name: RequestCount
statistics: [Sum]
- name: HTTPCode_ELB_502_Count
statistics: [Sum]
- name: HTTPCode_ELB_504_Count
statistics: [Sum]
- name: HTTPCode_Target_5XX_Count
statistics: [Sum]
- name: TargetConnectionErrorCount
statistics: [Sum]
ALB only reports a counter when it’s nonzero, so nilToZero turns a missing value into a zero instead of a gap. YACE names each metric aws_<namespace>_<metric>_<statistic> in snake case and puts the load balancer in the dimension_LoadBalancer label.
groups:
- name: alb-gateway-errors
rules:
- alert: AlbElb502Rate
expr: |
sum by (dimension_LoadBalancer) (aws_applicationelb_httpcode_elb_502_count_sum)
/ sum by (dimension_LoadBalancer) (aws_applicationelb_request_count_sum) > 0.001
for: 5m
labels:
severity: critical
annotations:
summary: "ALB {{ $labels.dimension_LoadBalancer }} is returning 502s it generated itself"
description: "Compare each target's keep-alive timeout with idle_timeout.timeout_seconds and check TargetConnectionErrorCount. Read the access log rows with elb_status_code 502."
- alert: AlbElb504Rate
expr: |
sum by (dimension_LoadBalancer) (aws_applicationelb_httpcode_elb_504_count_sum)
/ sum by (dimension_LoadBalancer) (aws_applicationelb_request_count_sum) > 0.001
for: 5m
labels:
severity: critical
annotations:
summary: "ALB {{ $labels.dimension_LoadBalancer }} is timing out requests to its targets"
description: "Find the slow paths in the access log by target_processing_time. Fix the handler or move the work to a background job before you raise the idle timeout."
- alert: AlbTargetConnectionErrors
expr: sum by (dimension_LoadBalancer) (aws_applicationelb_target_connection_error_count_sum) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "ALB {{ $labels.dimension_LoadBalancer }} can't open connections to some targets"
description: "Check target CPU, the app's listen backlog and the security groups and network ACLs between the load balancer subnets and the targets."
The 0.1% thresholds are a starting point. Set them from your own error budget. CloudWatch data reaches YACE a minute or more late, so these alerts fire a few minutes after the errors start.
Roll back
Save the load balancer attributes and each app server’s keep-alive values before you change them. The alb-attributes-before.json file from the fix holds the load balancer side. Keep the app config change in its own commit.
To restore the idle timeout:
aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" \
--attributes Key=idle_timeout.timeout_seconds,Value=60
The keep-alive change on the targets seldom causes trouble. A longer keep-alive holds more idle connections and file descriptors, so check the process’s open file limit if connection counts jump. If the change causes errors, revert the app config and redeploy now. Revert the idle timeout at the same time, so the target timeout stays above it. Leave any other revert until traffic is low.
More recipes
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.
- NAT Gateway ErrorPortAllocation and Port Exhaustion
Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.
- Fix Lambda RDS Too Many Connections with RDS Proxy
Put RDS Proxy between Lambda and RDS and cap concurrency to the proxy's pool, so a spike waits in a short queue instead of exhausting database connections.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.