Fix Lambda RDS Too Many Connections with RDS Proxy

Put RDS Proxy between Lambda and RDS and cap concurrency to the proxy's pool, so a spike waits in a short queue instead of exhausting database connections.

Published

Applies to Amazon RDS and Aurora for PostgreSQL and MySQL with RDS Proxy; AWS Lambda; YACE 0.62 and later

Lambda RDS “too many connections” errors come from simple math. Each concurrent Lambda execution environment opens its own database connection, and Lambda can run far more environments than RDS has connection slots. RDS Proxy pools those connections so the database sees a small, steady number. The same fix works when dozens of pods or ECS tasks each hold their own pool.

What breaks at peak

Lambda scales by adding execution environments. Each one runs your handler for one request at a time. If the handler opens a database connection, 400 concurrent requests mean 400 connections. RDS sets max_connections from the instance’s memory by default, so a smaller instance runs out well before Lambda runs out of concurrency.

The database refuses the extra logins. MySQL and Aurora MySQL return:

ERROR 1040 (HY000): Too many connections

PostgreSQL and Aurora PostgreSQL return:

FATAL:  sorry, too many clients already

The PgBouncer recipe lists the other Postgres variants, and the MySQL too many connections recipe covers the MySQL side.

Three Lambda habits make it worse:

  • A connection opened inside the handler. Every invocation pays for a TCP and TLS handshake plus authentication. A crashed invocation can leave the connection open until the database times it out.
  • Idle environments that keep connections. Lambda freezes an environment between requests. Its connection stays open and counts against max_connections while it does nothing.
  • Retries on failure. An async invocation that fails on a connection error goes back to Lambda’s queue and runs again, asking for yet another connection.

RDS Proxy fixes this by holding a pool of database connections and lending one to a client for each transaction. Your functions connect to the proxy, which can accept far more client connections than the database allows.

The fix with RDS Proxy and Lambda concurrency

Create the proxy in the same VPC as the database, with a security group that lets your functions reach it on the database port. Point your functions at the proxy endpoint instead of the instance or cluster endpoint.

Save the current pool settings first, then set your own:

aws rds describe-db-proxy-target-groups --db-proxy-name app-proxy \
  > proxy-target-group-before.json

aws rds modify-db-proxy-target-group \
  --db-proxy-name app-proxy \
  --target-group-name default \
  --connection-pool-config MaxConnectionsPercent=70,MaxIdleConnectionsPercent=35,ConnectionBorrowTimeout=10

In Terraform the same settings live on the default target group:

resource "aws_db_proxy_default_target_group" "app" {
  db_proxy_name = aws_db_proxy.app.name

  connection_pool_config {
    max_connections_percent      = 70
    max_idle_connections_percent = 35
    connection_borrow_timeout    = 10
  }
}

Each setting, and what goes wrong when it’s off:

  • MaxConnectionsPercent caps the proxy’s database connections as a percentage of max_connections. At 100, the default, the proxy can take every slot and lock out migrations, cron jobs and your own psql session. Leave room for them. AWS also asks for at least 30% headroom above your recent peak, because the proxy shifts connection quotas between its own nodes.
  • MaxIdleConnectionsPercent sets how many idle connections the proxy keeps warm. It defaults to half of MaxConnectionsPercent. Too low, and a burst has to open new database connections at the worst moment. For spiky traffic I keep it close to the maximum.
  • ConnectionBorrowTimeout sets how long a client waits for a free database connection before it gets an error. The default is 120 seconds, which is longer than most API requests can wait. Set it below your function timeout so a full pool fails fast. MySQL clients then get ERROR 9501 (HY000): Timed-out waiting to acquire database connection.
  • IdleClientTimeout is set on the proxy itself with aws rds modify-db-proxy. It closes client connections that sit idle, 1,800 seconds by default. Keep any client-side pool’s idle timeout below it, or your clients will reuse connections the proxy already closed.

Cap Lambda to the pool. The proxy queues clients once its pool is full. That beats refused logins, but an uncapped function can still queue thousands of requests that time out together. Set reserved concurrency on each function that uses the database:

aws lambda get-function-concurrency --function-name orders-api > concurrency-before.json
aws lambda put-function-concurrency --function-name orders-api \
  --reserved-concurrent-executions 200

Size it from your own numbers. Measure the share of each invocation spent inside a transaction. Reserved concurrency times that share should fit inside the proxy’s database connections, with headroom. A function that spends a fifth of its time in transactions can run several environments per database connection. A function that spends all of it can’t.

For SQS triggers, set maximum concurrency on the event source mapping instead, so messages wait in the queue rather than failing:

aws lambda update-event-source-mapping --uuid 6d9bce8e-xmpl-4b6f-9a4b-2f2d0c3e8f71 \
  --scaling-config MaximumConcurrency=50

Open the connection once per environment. Create the client outside the handler, so a warm environment reuses it. Node.js with pg:

const { Pool } = require('pg');

// one connection per execution environment, opened on first use
const pool = new Pool({
  host: process.env.PROXY_ENDPOINT,
  database: 'app',
  user: 'app',
  password: process.env.DB_PASSWORD,
  ssl: { rejectUnauthorized: true },
  max: 1,
  idleTimeoutMillis: 120000,
});

exports.handler = async (event) => {
  const { rows } = await pool.query('SELECT id, status FROM orders WHERE id = $1', [event.id]);
  return rows[0] ?? null;
};

Python with PyMySQL:

import os
import pymysql

conn = None

def get_conn():
    global conn
    if conn is None or not conn.open:
        conn = pymysql.connect(
            host=os.environ["PROXY_ENDPOINT"],
            user="app",
            password=os.environ["DB_PASSWORD"],
            database="app",
            ssl={"ca": "/opt/rds-ca.pem"},
            connect_timeout=5,
        )
    conn.ping(reconnect=True)
    return conn

def handler(event, context):
    with get_conn().cursor() as cur:
        cur.execute("SELECT id, status FROM orders WHERE id = %s", (event["id"],))
        return cur.fetchone()

max: 1 keeps one connection per environment, because each environment handles one request at a time. The PyMySQL ping reconnects if the proxy closed an idle connection while the environment was frozen. For IAM authentication, generate a token with the SDK in place of the password and turn on IAM auth on the proxy.

Avoid pinning. The proxy pins a client to one database connection when the session changes state it can’t share. A pinned connection serves one client until that client disconnects, so heavy pinning turns the proxy back into one connection per client. On PostgreSQL, a SET outside a transaction, PREPARE, temporary tables, LISTEN and session advisory locks all pin. On MySQL, SET of most variables, prepared statements, temporary tables, GET_LOCK and table locks pin. Any statement over 16 KB pins on every engine.

Move identical SET statements into the proxy’s initialization query, and use SET LOCAL inside a transaction on PostgreSQL. On MySQL, if you know your app doesn’t depend on session variables carrying over, add SessionPinningFilters=EXCLUDE_VARIABLE_SETS to the connection pool config. The proxy log records each pinned session with its reason, so check it after every release.

Pods and containers. The same pool math applies to ECS tasks and EKS pods. RDS Proxy works for them too. On EKS I often run PgBouncer instead, because it gives me pool modes and an admin console. For Lambda I go straight to RDS Proxy, since there’s nowhere to run a pooler next to a function.

Test it on your system

Point a staging copy of your function at a staging database with production-sized max_connections. Drive it at a fixed arrival rate with k6, through API Gateway or a function URL:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    burst: {
      executor: 'ramping-arrival-rate',
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 300,
      maxVUs: 2000,
      stages: [
        { target: 50, duration: '2m' },
        { target: 600, duration: '1m' },   // set this to your peak
        { target: 600, duration: '10m' },
        { target: 50, duration: '2m' },
      ],
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(99)<1000'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/orders/${Math.floor(Math.random() * 100000)}`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}
k6 run -e BASE_URL=https://staging-api.example.com burst.js

Run it once against the instance endpoint to see the failure, then against the proxy. While it runs, watch:

  • The function’s logs for Too many connections, too many clients and error 9501.
  • ClientConnections and DatabaseConnections on the proxy. Client connections should climb with concurrency while database connections level off.
  • DatabaseConnectionsBorrowLatency and DatabaseConnectionsCurrentlySessionPinned.
  • ConcurrentExecutions and Throttles on the function.

Write your targets before the run. It passes when no connection errors appear, DatabaseConnections stays under MaxDatabaseConnectionsAllowed, borrow latency stays inside your latency budget and the k6 thresholds hold. Throttles at the reserved concurrency are expected if your peak exceeds it. Decide whether that’s the behavior you want.

Keep it healthy

Pull the proxy metrics into Prometheus with YACE. Its AWS/RDS discovery job doesn’t map proxies to the ProxyName dimension CloudWatch uses, so use a static job per proxy:

apiVersion: v1alpha1
static:
  - namespace: AWS/RDS
    name: app-proxy
    regions: [eu-west-1]
    dimensions:
      - name: ProxyName
        value: app-proxy
    period: 60
    length: 300
    metrics:
      - name: DatabaseConnections
        statistics: [Sum]
      - name: MaxDatabaseConnectionsAllowed
        statistics: [Sum]
      - name: DatabaseConnectionsBorrowLatency
        statistics: [Average]
      - name: DatabaseConnectionsCurrentlySessionPinned
        statistics: [Sum]
groups:
  - name: rds-proxy
    rules:
      - alert: RdsProxyPoolNearMax
        expr: |
          aws_rds_database_connections_sum{dimension_ProxyName!=""}
            / aws_rds_max_database_connections_allowed_sum{dimension_ProxyName!=""} > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "RDS Proxy {{ $labels.dimension_ProxyName }} is using over 85% of its database connections"
          description: "Check DatabaseConnectionsCurrentlySessionPinned first. If pinning is low, raise MaxConnectionsPercent or lower the reserved concurrency of the functions behind this proxy."
      - alert: RdsProxyBorrowLatencyHigh
        expr: aws_rds_database_connections_borrow_latency_average{dimension_ProxyName!=""} > 100000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Clients wait over 100 ms for a connection from RDS Proxy {{ $labels.dimension_ProxyName }}"
          description: "The pool is full and requests are queueing. Check the database for long transactions, then compare DatabaseConnections with MaxDatabaseConnectionsAllowed."
      - alert: RdsProxyHeavyPinning
        expr: |
          aws_rds_database_connections_currently_session_pinned_sum{dimension_ProxyName!=""}
            / aws_rds_database_connections_sum{dimension_ProxyName!=""} > 0.5
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Over half the connections on RDS Proxy {{ $labels.dimension_ProxyName }} are pinned"
          description: "Search the proxy log in CloudWatch Logs for pinned sessions and their reasons. Move SET statements into the proxy's initialization query or use SET LOCAL."

Borrow latency is in microseconds, so 100000 is 100 ms. Set it from your own latency budget. Alert on Throttles for each database function too, as the Lambda concurrency recipe shows.

Roll back

You saved proxy-target-group-before.json and concurrency-before.json before the change. To restore the pool settings, run modify-db-proxy-target-group with the values from the file. To remove a reserved concurrency limit:

aws lambda delete-function-concurrency --function-name orders-api

To take the proxy out of the path, point the function’s environment variable back at the database endpoint and publish. Before you do, check that the function’s peak concurrency fits under max_connections, or you’ll swap queueing for refused logins. If the proxy is causing errors, such as broken session state after the switch, roll back now and lower reserved concurrency to fit the database. If you’re reverting for any other reason, pick an off-peak window.

More recipes

Further reading