Lambda returns TooManyRequestsException with the message Rate Exceeded. when running your function would go past a concurrency limit. The limit is either your account’s total in the region or the function’s own reserved concurrency. Lambda throttling at peak is a sizing problem, and you can fix it before the next spike.
What breaks at peak
A synchronous caller gets an HTTP 429 from the Invoke API. From the AWS CLI it looks like this:
An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 2): Rate Exceeded.
The (reached max retries: 2) part depends on your CLI’s retry settings. The SDK error carries a Reason field that tells you which limit you hit:
ConcurrentInvocationLimitExceededmeans the account ran out. Every function without reserved concurrency shares the account pool, 1,000 per region by default.ReservedFunctionConcurrentInvocationLimitExceededmeans this function hit its own reserved concurrency. Other functions are fine. This one is capped.
The caller decides what reaches your users. An Application Load Balancer with a Lambda target returns a 502 for a throttled function. An async invocation, such as one from S3 or EventBridge, goes back on Lambda’s internal queue. Lambda retries throttled events for up to 6 hours by default, with a growing interval, so the work arrives late. An SQS trigger leaves the message in the queue to retry after its visibility timeout.
Concurrency is requests times duration. AWS gives the formula: concurrency equals average requests per second times average duration in seconds. A function that takes 200 ms at 5,000 requests per second needs 1,000 concurrent environments. That’s the whole default account quota for one function. Duration is half the formula, so a slow downstream call at peak raises concurrency without any change in traffic.
Two more limits catch teams out:
- Requests per second. Lambda also caps requests per second at 10 times your concurrency quota. A very fast function can hit this limit while concurrency looks low.
- Scaling rate. Each function can add up to 1,000 execution environments every 10 seconds. A launch or a marketing email that sends traffic from near zero to peak at once can throttle while Lambda catches up.
One function can starve the rest. A runaway batch job or a retry loop can take the whole unreserved pool. Every other function in the account then throttles, including the ones your checkout depends on.
The Lambda concurrency fix
Start by finding which limit you hit. Check the account quota and what’s already reserved, and save both:
aws lambda get-account-settings > account-settings-before.json
aws lambda get-function-concurrency --function-name checkout-api > checkout-concurrency-before.json
aws lambda list-provisioned-concurrency-configs --function-name checkout-api > checkout-pc-before.json
ConcurrentExecutions under AccountLimit is your regional quota. UnreservedConcurrentExecutions is what’s left to reserve. Lambda always keeps 100 units unreserved for functions without their own reservation.
Size it. For each function that matters at peak, take its peak requests per second from the Invocations metric and its p99 duration from Duration. Multiply them, then add headroom for the slow days. I use p99 rather than average duration for this, because the slow requests are the ones that pile up at peak. Add up the functions. If the total is near your account quota, request an increase through Service Quotas well before the launch.
Reserved concurrency sets both a floor and a ceiling for one function. The function always has that many environments available, and it can never use more:
aws lambda put-function-concurrency --function-name checkout-api \
--reserved-concurrent-executions 300
Use it two ways. Give critical functions, such as checkout or login, a reservation so a noisy neighbor can’t starve them. Some functions call a fragile downstream, such as a database or a partner API with its own rate limit. Give those a reservation that matches what the downstream can take. The RDS Proxy recipe sizes that number for a database. Set it too low and you throttle your own traffic. Every unit you reserve also leaves the shared pool.
Provisioned concurrency keeps environments initialized before traffic arrives. It removes cold starts for that many concurrent requests and sidesteps the scaling rate for them. It only works on a published version or an alias, never on $LATEST:
aws lambda put-provisioned-concurrency-config --function-name checkout-api \
--qualifier live \
--provisioned-concurrent-executions 100
It costs money while it’s allocated, whether it serves traffic or not. Above the provisioned amount, the function spills over to on-demand concurrency, with cold starts, until it reaches its reserved concurrency or the account limit. Provisioned concurrency can’t exceed the function’s reserved concurrency.
To follow a daily curve, let Application Auto Scaling manage it. This keeps utilization near 70%:
aws application-autoscaling register-scalable-target --service-namespace lambda \
--resource-id function:checkout-api:live --min-capacity 50 --max-capacity 400 \
--scalable-dimension lambda:function:ProvisionedConcurrency
aws application-autoscaling put-scaling-policy --service-namespace lambda \
--scalable-dimension lambda:function:ProvisionedConcurrency \
--resource-id function:checkout-api:live \
--policy-name checkout-pc-utilization \
--policy-type TargetTrackingScaling \
--target-tracking-scaling-policy-configuration '{ "TargetValue": 0.7, "PredefinedMetricSpecification": { "PredefinedMetricType": "LambdaProvisionedConcurrencyUtilization" }}'
Target tracking reacts in minutes, and AWS notes its alarms use the average by default. Short bursts can slip past it. For a known peak, such as a sale at 9:00, add a scheduled action that raises the minimum an hour before.
Callers that handle throttles. Your own code that calls Invoke should retry TooManyRequestsException with exponential backoff and jitter. The AWS SDKs do this by default with a small retry count. For queues, cap the consumer instead of letting it throttle:
aws lambda update-event-source-mapping --uuid 6d9bce8e-xmpl-4b6f-9a4b-2f2d0c3e8f71 \
--scaling-config MaximumConcurrency=100
MaximumConcurrency on an SQS event source mapping stops Lambda from polling more messages than that many environments can handle. Messages wait in the queue rather than bouncing off a throttle.
Shorter duration, lower concurrency. Set client timeouts on every downstream call inside the function. A function that waits 30 seconds on a dead dependency holds its environment for all 30. The timeouts and retries recipe covers the client side.
My own default is reserved concurrency on every function that touches a database or a rate-limited API. I add provisioned concurrency only where cold starts break a latency target, and leave the rest on the shared pool.
Test it on your system
Load test a staging copy in an account with the same quotas as production, or with reserved concurrency set to match. Use a fixed arrival rate, then a sudden jump, since the scaling rate only shows up on a jump:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
jump: {
executor: 'ramping-arrival-rate',
startRate: 20,
timeUnit: '1s',
preAllocatedVUs: 500,
maxVUs: 3000,
stages: [
{ target: 20, duration: '2m' },
{ target: 1500, duration: '10s' }, // set this to your peak
{ target: 1500, duration: '10m' },
{ target: 20, duration: '1m' },
],
},
},
thresholds: {
'http_req_failed': ['rate<0.001'],
'http_req_duration': ['p(99)<600'],
},
};
export default function () {
const res = http.post(`${__ENV.BASE_URL}/checkout`, JSON.stringify({ cart: 'test' }), {
headers: { 'Content-Type': 'application/json' },
});
check(res, { 'not throttled': (r) => r.status !== 429 && r.status !== 502 });
}
k6 run -e BASE_URL=https://staging-api.example.com jump.js
While it runs, watch these CloudWatch metrics at 1-minute resolution:
ThrottlesandConcurrentExecutionsfor the function.ClaimedAccountConcurrencyfor the account, which counts unreserved use plus everything reserved or provisioned.ProvisionedConcurrencySpilloverInvocations, if you use provisioned concurrency.- The function’s
Init Durationin its logs, which tells you how many requests paid for a cold start.
Write your targets down first. The test passes when Throttles stays at zero for every function that must not throttle. The k6 thresholds should hold through the jump, and ClaimedAccountConcurrency should peak with room to spare. A capped function that throttles at its reservation passes only if that’s what you planned for it.
Keep it healthy
YACE, the prometheus-community yet-another-cloudwatch-exporter, brings the per-function metrics into Prometheus:
apiVersion: v1alpha1
discovery:
jobs:
- type: AWS/Lambda
regions: [eu-west-1]
period: 60
length: 300
nilToZero: true
metrics:
- name: Invocations
statistics: [Sum]
- name: Throttles
statistics: [Sum]
- name: ConcurrentExecutions
statistics: [Maximum]
- name: ProvisionedConcurrencySpilloverInvocations
statistics: [Sum]
groups:
- name: aws-lambda-concurrency
rules:
- alert: LambdaThrottling
expr: |
sum by (dimension_FunctionName) (aws_lambda_throttles_sum)
/ (
sum by (dimension_FunctionName) (aws_lambda_invocations_sum)
+ sum by (dimension_FunctionName) (aws_lambda_throttles_sum)
) > 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "Lambda {{ $labels.dimension_FunctionName }} is throttling over 1% of requests"
description: "Run aws lambda get-function-concurrency and get-account-settings to see which limit it hit. Raise reserved concurrency, cut duration or request a quota increase."
- alert: LambdaProvisionedSpillover
expr: sum by (dimension_FunctionName) (aws_lambda_provisioned_concurrency_spillover_invocations_sum) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Lambda {{ $labels.dimension_FunctionName }} is running past its provisioned concurrency"
description: "Requests above the provisioned amount pay for cold starts. Raise the minimum on the Application Auto Scaling target or add a scheduled action before the daily peak."
Throttles isn’t part of Invocations, so the ratio adds them. The account-wide ClaimedAccountConcurrency metric has no function dimension. Alarm on it in CloudWatch:
aws cloudwatch put-metric-alarm --alarm-name lambda-claimed-concurrency-80pct \
--namespace AWS/Lambda --metric-name ClaimedAccountConcurrency \
--statistic Maximum --period 60 --evaluation-periods 5 \
--threshold 800 --comparison-operator GreaterThanThreshold \
--alarm-actions arn:aws:sns:eu-west-1:123456789012:oncall
Set the threshold to 80% of your own quota. 800 matches the default of 1,000.
Roll back
The three files you saved hold the old settings. To remove a reservation and return the function to the shared pool:
aws lambda delete-function-concurrency --function-name checkout-api
To remove provisioned concurrency and its scaling:
aws application-autoscaling deregister-scalable-target --service-namespace lambda \
--resource-id function:checkout-api:live \
--scalable-dimension lambda:function:ProvisionedConcurrency
aws lambda delete-provisioned-concurrency-config --function-name checkout-api --qualifier live
A reservation set too low throttles your own traffic, so raise or delete it right away if throttles start after the change. Removing provisioned concurrency brings cold starts back, so do that one off-peak.
More recipes
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- NAT Gateway ErrorPortAllocation and Port Exhaustion
Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.
- Fix Lambda RDS Too Many Connections with RDS Proxy
Put RDS Proxy between Lambda and RDS and cap concurrency to the proxy's pool, so a spike waits in a short queue instead of exhausting database connections.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.