Fix Lambda TooManyRequestsException Rate Exceeded

Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.

Published

Applies to AWS Lambda, all runtimes; AWS CLI v2; YACE 0.62 and later

Lambda returns TooManyRequestsException with the message Rate Exceeded. when running your function would go past a concurrency limit. The limit is either your account’s total in the region or the function’s own reserved concurrency. Lambda throttling at peak is a sizing problem, and you can fix it before the next spike.

What breaks at peak

A synchronous caller gets an HTTP 429 from the Invoke API. From the AWS CLI it looks like this:

An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 2): Rate Exceeded.

The (reached max retries: 2) part depends on your CLI’s retry settings. The SDK error carries a Reason field that tells you which limit you hit:

  • ConcurrentInvocationLimitExceeded means the account ran out. Every function without reserved concurrency shares the account pool, 1,000 per region by default.
  • ReservedFunctionConcurrentInvocationLimitExceeded means this function hit its own reserved concurrency. Other functions are fine. This one is capped.

The caller decides what reaches your users. An Application Load Balancer with a Lambda target returns a 502 for a throttled function. An async invocation, such as one from S3 or EventBridge, goes back on Lambda’s internal queue. Lambda retries throttled events for up to 6 hours by default, with a growing interval, so the work arrives late. An SQS trigger leaves the message in the queue to retry after its visibility timeout.

Concurrency is requests times duration. AWS gives the formula: concurrency equals average requests per second times average duration in seconds. A function that takes 200 ms at 5,000 requests per second needs 1,000 concurrent environments. That’s the whole default account quota for one function. Duration is half the formula, so a slow downstream call at peak raises concurrency without any change in traffic.

Two more limits catch teams out:

  • Requests per second. Lambda also caps requests per second at 10 times your concurrency quota. A very fast function can hit this limit while concurrency looks low.
  • Scaling rate. Each function can add up to 1,000 execution environments every 10 seconds. A launch or a marketing email that sends traffic from near zero to peak at once can throttle while Lambda catches up.

One function can starve the rest. A runaway batch job or a retry loop can take the whole unreserved pool. Every other function in the account then throttles, including the ones your checkout depends on.

The Lambda concurrency fix

Start by finding which limit you hit. Check the account quota and what’s already reserved, and save both:

aws lambda get-account-settings > account-settings-before.json
aws lambda get-function-concurrency --function-name checkout-api > checkout-concurrency-before.json
aws lambda list-provisioned-concurrency-configs --function-name checkout-api > checkout-pc-before.json

ConcurrentExecutions under AccountLimit is your regional quota. UnreservedConcurrentExecutions is what’s left to reserve. Lambda always keeps 100 units unreserved for functions without their own reservation.

Size it. For each function that matters at peak, take its peak requests per second from the Invocations metric and its p99 duration from Duration. Multiply them, then add headroom for the slow days. I use p99 rather than average duration for this, because the slow requests are the ones that pile up at peak. Add up the functions. If the total is near your account quota, request an increase through Service Quotas well before the launch.

Reserved concurrency sets both a floor and a ceiling for one function. The function always has that many environments available, and it can never use more:

aws lambda put-function-concurrency --function-name checkout-api \
  --reserved-concurrent-executions 300

Use it two ways. Give critical functions, such as checkout or login, a reservation so a noisy neighbor can’t starve them. Some functions call a fragile downstream, such as a database or a partner API with its own rate limit. Give those a reservation that matches what the downstream can take. The RDS Proxy recipe sizes that number for a database. Set it too low and you throttle your own traffic. Every unit you reserve also leaves the shared pool.

Provisioned concurrency keeps environments initialized before traffic arrives. It removes cold starts for that many concurrent requests and sidesteps the scaling rate for them. It only works on a published version or an alias, never on $LATEST:

aws lambda put-provisioned-concurrency-config --function-name checkout-api \
  --qualifier live \
  --provisioned-concurrent-executions 100

It costs money while it’s allocated, whether it serves traffic or not. Above the provisioned amount, the function spills over to on-demand concurrency, with cold starts, until it reaches its reserved concurrency or the account limit. Provisioned concurrency can’t exceed the function’s reserved concurrency.

To follow a daily curve, let Application Auto Scaling manage it. This keeps utilization near 70%:

aws application-autoscaling register-scalable-target --service-namespace lambda \
  --resource-id function:checkout-api:live --min-capacity 50 --max-capacity 400 \
  --scalable-dimension lambda:function:ProvisionedConcurrency

aws application-autoscaling put-scaling-policy --service-namespace lambda \
  --scalable-dimension lambda:function:ProvisionedConcurrency \
  --resource-id function:checkout-api:live \
  --policy-name checkout-pc-utilization \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{ "TargetValue": 0.7, "PredefinedMetricSpecification": { "PredefinedMetricType": "LambdaProvisionedConcurrencyUtilization" }}'

Target tracking reacts in minutes, and AWS notes its alarms use the average by default. Short bursts can slip past it. For a known peak, such as a sale at 9:00, add a scheduled action that raises the minimum an hour before.

Callers that handle throttles. Your own code that calls Invoke should retry TooManyRequestsException with exponential backoff and jitter. The AWS SDKs do this by default with a small retry count. For queues, cap the consumer instead of letting it throttle:

aws lambda update-event-source-mapping --uuid 6d9bce8e-xmpl-4b6f-9a4b-2f2d0c3e8f71 \
  --scaling-config MaximumConcurrency=100

MaximumConcurrency on an SQS event source mapping stops Lambda from polling more messages than that many environments can handle. Messages wait in the queue rather than bouncing off a throttle.

Shorter duration, lower concurrency. Set client timeouts on every downstream call inside the function. A function that waits 30 seconds on a dead dependency holds its environment for all 30. The timeouts and retries recipe covers the client side.

My own default is reserved concurrency on every function that touches a database or a rate-limited API. I add provisioned concurrency only where cold starts break a latency target, and leave the rest on the shared pool.

Test it on your system

Load test a staging copy in an account with the same quotas as production, or with reserved concurrency set to match. Use a fixed arrival rate, then a sudden jump, since the scaling rate only shows up on a jump:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    jump: {
      executor: 'ramping-arrival-rate',
      startRate: 20,
      timeUnit: '1s',
      preAllocatedVUs: 500,
      maxVUs: 3000,
      stages: [
        { target: 20, duration: '2m' },
        { target: 1500, duration: '10s' },  // set this to your peak
        { target: 1500, duration: '10m' },
        { target: 20, duration: '1m' },
      ],
    },
  },
  thresholds: {
    'http_req_failed': ['rate<0.001'],
    'http_req_duration': ['p(99)<600'],
  },
};

export default function () {
  const res = http.post(`${__ENV.BASE_URL}/checkout`, JSON.stringify({ cart: 'test' }), {
    headers: { 'Content-Type': 'application/json' },
  });
  check(res, { 'not throttled': (r) => r.status !== 429 && r.status !== 502 });
}
k6 run -e BASE_URL=https://staging-api.example.com jump.js

While it runs, watch these CloudWatch metrics at 1-minute resolution:

  • Throttles and ConcurrentExecutions for the function.
  • ClaimedAccountConcurrency for the account, which counts unreserved use plus everything reserved or provisioned.
  • ProvisionedConcurrencySpilloverInvocations, if you use provisioned concurrency.
  • The function’s Init Duration in its logs, which tells you how many requests paid for a cold start.

Write your targets down first. The test passes when Throttles stays at zero for every function that must not throttle. The k6 thresholds should hold through the jump, and ClaimedAccountConcurrency should peak with room to spare. A capped function that throttles at its reservation passes only if that’s what you planned for it.

Keep it healthy

YACE, the prometheus-community yet-another-cloudwatch-exporter, brings the per-function metrics into Prometheus:

apiVersion: v1alpha1
discovery:
  jobs:
    - type: AWS/Lambda
      regions: [eu-west-1]
      period: 60
      length: 300
      nilToZero: true
      metrics:
        - name: Invocations
          statistics: [Sum]
        - name: Throttles
          statistics: [Sum]
        - name: ConcurrentExecutions
          statistics: [Maximum]
        - name: ProvisionedConcurrencySpilloverInvocations
          statistics: [Sum]
groups:
  - name: aws-lambda-concurrency
    rules:
      - alert: LambdaThrottling
        expr: |
          sum by (dimension_FunctionName) (aws_lambda_throttles_sum)
            / (
              sum by (dimension_FunctionName) (aws_lambda_invocations_sum)
              + sum by (dimension_FunctionName) (aws_lambda_throttles_sum)
            ) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Lambda {{ $labels.dimension_FunctionName }} is throttling over 1% of requests"
          description: "Run aws lambda get-function-concurrency and get-account-settings to see which limit it hit. Raise reserved concurrency, cut duration or request a quota increase."
      - alert: LambdaProvisionedSpillover
        expr: sum by (dimension_FunctionName) (aws_lambda_provisioned_concurrency_spillover_invocations_sum) > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Lambda {{ $labels.dimension_FunctionName }} is running past its provisioned concurrency"
          description: "Requests above the provisioned amount pay for cold starts. Raise the minimum on the Application Auto Scaling target or add a scheduled action before the daily peak."

Throttles isn’t part of Invocations, so the ratio adds them. The account-wide ClaimedAccountConcurrency metric has no function dimension. Alarm on it in CloudWatch:

aws cloudwatch put-metric-alarm --alarm-name lambda-claimed-concurrency-80pct \
  --namespace AWS/Lambda --metric-name ClaimedAccountConcurrency \
  --statistic Maximum --period 60 --evaluation-periods 5 \
  --threshold 800 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:eu-west-1:123456789012:oncall

Set the threshold to 80% of your own quota. 800 matches the default of 1,000.

Roll back

The three files you saved hold the old settings. To remove a reservation and return the function to the shared pool:

aws lambda delete-function-concurrency --function-name checkout-api

To remove provisioned concurrency and its scaling:

aws application-autoscaling deregister-scalable-target --service-namespace lambda \
  --resource-id function:checkout-api:live \
  --scalable-dimension lambda:function:ProvisionedConcurrency
aws lambda delete-provisioned-concurrency-config --function-name checkout-api --qualifier live

A reservation set too low throttles your own traffic, so raise or delete it right away if throttles start after the change. Removing provisioned concurrency brings cold starts back, so do that one off-peak.

More recipes

Further reading