Fix Socket Hang Up and ECONNRESET Behind a Load Balancer

Keep idle connections open longer on the side that should close them last, so Node.js and the load balancer stop racing to reuse a socket the other has closed.

Published

Applies to Node.js 18 and later; keepAliveTimeoutBuffer from 22.19 and 24.6; Express 4 and 5, Fastify 4 and 5, NestJS 10 and later

Error: socket hang up with code: 'ECONNRESET' means the other end closed a connection while your request was on it. Between Node.js and a load balancer, the usual cause is a keepalive timeout mismatch. One side closes an idle connection at the moment the other side reuses it. The error shows up at random, and more often at peak.

What breaks at peak

Two connections can race, and each one fails a different way.

Load balancer to Node.js. Node’s http.Server closes an idle keepalive connection after keepAliveTimeout, 5 seconds by default on every released line up to Node 26. An AWS Application Load Balancer keeps idle backend connections for 60 seconds by default. The load balancer picks a connection Node closed a moment ago and sends a request down it. The request fails, and the load balancer returns a 502 to your user. Node logs nothing, because the request never reached your handler. Behind nginx you see upstream prematurely closed connection instead, which the nginx upstream timeouts recipe covers.

Node.js to an upstream. Your service calls another API through a pooled agent. The upstream, or a proxy in front of it, closes an idle socket. Your next request picks that socket from the pool and fails with one of these:

Error: socket hang up
    code: 'ECONNRESET'
TypeError: fetch failed
  [cause]: SocketError: other side closed
    code: 'UND_ERR_SOCKET'

The first comes from node:http and libraries built on it, such as axios. The second comes from fetch and undici. Both reach your code as a failed request, and most code turns them into a 500.

Peak makes both worse. More connections sit in the pools, and more of them hit their idle limit at the same instant a new request arrives. Node 19 made http.globalAgent keep connections alive by default, so code that never opted in to keepalive now pools sockets too.

The rule that fixes both: the side that opens connections must close idle ones first. The load balancer opens connections to Node, so Node’s timeout must be longer. Your service opens connections to the upstream, so your agent’s timeout must be shorter.

The keepAliveTimeout fix for your load balancer

Set Node’s keepAliveTimeout a few seconds above the idle timeout of whatever sits in front of it:

In front of Node.jsIdle timeout to beatSet keepAliveTimeout to
AWS Application Load Balancer60 s default, set per load balancer65000, or your setting plus 5 s
Google Cloud external Application Load Balancer600 s, fixed620000, as Google advises
nginx with upstream keepalivethe upstream keepalive_timeout, 60 s by default65000

Plain Node.js and Express. app.listen() returns the http.Server. Set the timeout on it:

const express = require('express');

const app = express();
const server = app.listen(Number(process.env.PORT) || 3000);

// Longer than the load balancer's idle timeout, so the load balancer closes first.
server.keepAliveTimeout = Number(process.env.KEEP_ALIVE_TIMEOUT_MS) || 65000;

keepAliveTimeout sets how long Node keeps an idle connection open after the last response. Too low, and Node closes connections the load balancer still plans to reuse. Too high costs one file descriptor and a little memory per idle connection, which matters little behind a load balancer that holds a bounded pool.

On Node 22.19 and 24.6 or later, Node waits keepAliveTimeout plus keepAliveTimeoutBuffer, 1000 ms by default, before it destroys the socket. Leave the buffer alone and size the timeout itself.

Fastify. Fastify sets the server’s keepAliveTimeout to 72000 ms by default, which already beats an ALB at 60 seconds. It loses to Google Cloud’s 600 seconds. Pass it to the constructor:

const fastify = require('fastify')({ keepAliveTimeout: 620000 });

NestJS. On the Express adapter, set it on the underlying server after listen. On the Fastify adapter, pass it to FastifyAdapter:

// Express adapter
const app = await NestFactory.create(AppModule);
await app.listen(3000);
app.getHttpServer().keepAliveTimeout = 65000;

// Fastify adapter
const fastifyApp = await NestFactory.create<NestFastifyApplication>(
  AppModule,
  new FastifyAdapter({ keepAliveTimeout: 65000 }),
);

Next.js. Run next start --keepAliveTimeout 65000. The rest of the Next.js tuning sits in the Next.js caching recipe.

Two more server timeouts. requestTimeout defaults to 300 seconds since Node 18 and caps the time to receive a whole request. Raise it only for slow uploads. headersTimeout defaults to 60 seconds and caps the time to receive the headers. Neither one closes an idle keepalive connection, so neither causes this race.

On a managed load balancer you change only its side. On AWS, the ALB idle timeout is a load balancer attribute, idle_timeout.timeout_seconds. Google Cloud fixes the backend keepalive at 600 seconds, so Node must exceed it. On Azure Application Gateway, read the backend keepalive and request timeouts for your SKU, and keep Node’s timeout above the longest one.

The agent fix for outbound calls

Your service is the client here, so its pool must drop idle sockets before the upstream does.

node:http and axios. Since Node 19, an agent with keepAlive: true reads the upstream’s Keep-Alive: timeout=N header and closes the socket before that. Node servers send that header. Many proxies and load balancers don’t, so set a timeout below the upstream’s idle timeout yourself. Then retry once when a reused socket fails, which the Node docs suggest for this exact race:

const http = require('node:http');

const agent = new http.Agent({
  keepAlive: true,
  timeout: 4000, // below the upstream's idle timeout
  maxSockets: 100,
});

function get(url, { retried = false } = {}) {
  return new Promise((resolve, reject) => {
    const req = http.get(url, { agent }, (res) => {
      const chunks = [];
      res.on('data', (c) => chunks.push(c));
      res.on('end', () => resolve({ status: res.statusCode, body: Buffer.concat(chunks).toString() }));
      res.on('error', reject);
    });
    req.on('error', (err) => {
      // The upstream closed a pooled socket as we reused it. Retry once, GET only.
      if (err.code === 'ECONNRESET' && req.reusedSocket && !retried) {
        resolve(get(url, { retried: true }));
      } else {
        reject(err);
      }
    });
  });
}

req.reusedSocket is true when the request went out on a pooled socket. A reset on a fresh socket means something else is wrong, so the code doesn’t retry it. Retry only idempotent methods. A POST that failed on a reused socket may have reached the upstream, and a blind retry can charge a card twice. With axios, pass the same agent as httpAgent and httpsAgent.

fetch and undici. Node’s built-in fetch uses undici. Its pool keeps idle sockets for keepAliveTimeout, 4 seconds by default, unless the upstream sends a keepalive hint. keepAliveTimeoutThreshold, 2 seconds on undici 6 and 7, is subtracted from that hint. Set both through a global dispatcher from the undici package:

const { Agent, setGlobalDispatcher } = require('undici');

setGlobalDispatcher(new Agent({
  keepAliveTimeout: 4000,          // below the upstream's idle timeout
  keepAliveMaxTimeout: 60000,      // cap on any hint the upstream sends
  keepAliveTimeoutThreshold: 2000, // margin taken off that hint
}));

I check the upstream’s idle timeout before I touch any agent setting. A curl with -v against the upstream shows whether it sends a Keep-Alive header, and the proxy’s docs give the rest.

If you run several Node.js processes per pod, each one keeps its own pool. The Node.js cluster mode recipe explains how that multiplies connections.

Test it on your system

The race needs idle gaps, so a flat high rate can hide it. Each pooled connection must sit idle close to the timeout, then get reused. A modest rate across many connections does that.

Run two steady phases against staging through the real load balancer. One holds a low rate, so connections idle between requests. The other holds your peak:

import http from 'k6/http';

export const options = {
  scenarios: {
    trickle: {
      executor: 'constant-arrival-rate',
      rate: 5, timeUnit: '1s', duration: '20m',
      preAllocatedVUs: 50, maxVUs: 200,
    },
    peak: {
      executor: 'constant-arrival-rate',
      rate: 400, timeUnit: '1s', duration: '20m', startTime: '20m',
      preAllocatedVUs: 200, maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'], // your error budget
  },
};

export default function () {
  http.get(`${__ENV.TARGET}/api/orders?limit=20`);
}

Run it with k6 run -e TARGET=https://staging.example.com keepalive.js, once on the current settings and once after the change. Point it at an endpoint that calls an upstream, so both connections get exercised.

While it runs, watch these:

  • Load balancer 502s. On AWS, HTTPCode_ELB_502_Count in CloudWatch. A 502 that the load balancer generated, with no matching error in your app logs, is this race.
  • Your app’s upstream errors. Count ECONNRESET and UND_ERR_SOCKET in the logs, and the retry counter from the next section.
  • Open sockets per pod. ss -tn state established | wc -l inside the container. With a longer timeout, the count should stay inside your file descriptor limit.

Pass means no load balancer 502 without an app error behind it, through both phases. Upstream resets that the retry absorbs are fine. Any that reach a user fail the run.

Keep it healthy

Count resets in the client code with prom-client, so you see them before users do. Changed lines only, in the get() function above:

const client = require('prom-client');

const socketResets = new client.Counter({
  name: 'http_client_socket_resets_total',
  help: 'Outbound requests that failed with ECONNRESET on a reused socket',
  labelNames: ['upstream', 'retried'],
});

// inside req.on('error'), before the retry decision
if (err.code === 'ECONNRESET' && req.reusedSocket) {
  socketResets.inc({ upstream: new URL(url).host, retried: String(!retried) });
}

These rules use that counter. Add a CloudWatch alarm on the ALB’s HTTPCode_ELB_502_Count for the inbound side:

groups:
  - name: nodejs-keepalive
    rules:
      - alert: NodeUpstreamSocketResets
        expr: sum by (upstream) (rate(http_client_socket_resets_total{retried="true"}[10m])) > 0.1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Pooled sockets to {{ $labels.upstream }} keep getting reset"
          description: "The upstream closes idle connections before the agent does. Find its idle timeout and set the agent timeout or undici keepAliveTimeout below it."
      - alert: NodeUpstreamResetsReachUsers
        expr: sum by (upstream) (increase(http_client_socket_resets_total{retried="false"}[10m])) > 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Retries to {{ $labels.upstream }} fail on a second reset"
          description: "Requests fail after the retry, so users get errors. Check the upstream's health first, then lower the agent idle timeout."

The retried="true" series counts first failures that the code retried. The retried="false" series counts resets on the retry itself, which reach the caller. Set the warning rate from a week of your own traffic, since a few resets an hour are normal on a busy pool.

Roll back

Note the current values first: the ALB attribute with aws elbv2 describe-load-balancer-attributes, and the KEEP_ALIVE_TIMEOUT_MS or Fastify option in your deploy config.

  • To revert the server side, remove the keepAliveTimeout line or set the env var back. Node returns to 5 seconds and the race comes back.
  • To revert the client side, remove the agent timeout or the global dispatcher. Keep the retry on reused sockets, since it only fires on this race.
  • If open sockets climb toward the file descriptor limit, lower both timeouts together. Keep Node’s keepAliveTimeout above the load balancer’s idle timeout.

If errors climb after the change, roll back now and check which side closes first. For any other reason, wait for a quiet hour.

More recipes