k6 breakpoint testing raises the request rate step by step until your system stops meeting its targets. The last step that passed is your capacity. The first metric that went wrong tells you which component to fix. Every load-test recipe in this cookbook uses the method below and keeps only its own script.
Where it breaks
Your breaking point is the arrival rate where p99 latency crosses your budget or the error rate crosses your error budget, whichever comes first. Write both numbers down before you start. Without them, you’ll call any result a pass.
You know you’ve reached it when one of these shows up:
- p99 bends upward. Latency stays flat across several steps, then climbs fast. That bend is the knee, and the step before it is your usable capacity.
- Errors start. 502s and 504s from a proxy, refused connections,
too many clientsfrom a database or timeouts from the app. - Throughput stops following the arrival rate. k6 asks for more requests a second, and completed requests stay flat. k6 then reports
dropped_iterations, because no virtual user was free to start the next one.
Use a staging copy with production instance sizes, production data volume and the same versions. Turn off autoscaling, or cap it at the production maximum. The k6 docs warn that an elastic system keeps growing during the test and you find your cloud bill’s limit instead of the system’s.
Set up the test with ramping-arrival-rate
k6’s default executors are closed-loop: each virtual user waits for its last response before it sends the next request. When the system slows, the load drops with it and the latency looks better than reality. The ramping-arrival-rate executor starts iterations at the rate you set, however slow the responses get. That’s the behavior a breakpoint test needs.
Build the request mix from production logs. Take a day of access log, count requests per path pattern and write a list where each path appears in proportion to its traffic. Keep real query strings and IDs, so caches and indexes see real variety. Save it as paths.json, a JSON array of paths.
// breakpoint.js
import http from 'k6/http';
import { check } from 'k6';
import { SharedArray } from 'k6/data';
const paths = new SharedArray('paths', () => JSON.parse(open('./paths.json')));
const BASE = __ENV.BASE_URL || 'https://staging.example.com';
const STEP = Number(__ENV.STEP || 100); // requests per second added each step
const STEPS = Number(__ENV.STEPS || 10);
const HOLD = __ENV.HOLD || '5m';
const stages = [];
for (let i = 1; i <= STEPS; i++) {
stages.push({ target: STEP * i, duration: '1m' }); // ramp to the next step
stages.push({ target: STEP * i, duration: HOLD }); // hold it
}
export const options = {
discardResponseBodies: true,
scenarios: {
breakpoint: {
executor: 'ramping-arrival-rate',
startRate: STEP,
timeUnit: '1s',
preAllocatedVUs: 200,
maxVUs: 5000,
stages,
},
},
thresholds: {
http_req_failed: [{ threshold: 'rate<0.05', abortOnFail: true, delayAbortEval: '2m' }],
http_req_duration: ['p(99)<800'],
},
};
export default function () {
const path = paths[Math.floor(Math.random() * paths.length)];
const res = http.get(`${BASE}${path}`, { tags: { name: 'mix' } });
check(res, { 'status below 400': (r) => r.status < 400 });
}
Executor settings. startRate and each stage’s target count iterations per timeUnit, here per second. Each iteration sends one request, so the target is your request rate. preAllocatedVUs are the virtual users k6 creates up front. maxVUs caps how many it may add when responses slow down. If k6 hits that cap it logs Insufficient VUs, reached 5000 active VUs and cannot initialize more and drops iterations. That’s a limit on the client side, so raise maxVUs or add load generators before you read anything into the result.
Thresholds. k6 checks thresholds over the whole run, from the first request. A p99 threshold therefore lags behind the step you’re on, because the early quiet steps pull it down. I use the error-rate threshold to stop a run that’s broken. delayAbortEval keeps a single bad sample from ending it. I read the per-step latency from the time series instead, as shown below.
Request names. tags: { name: 'mix' } groups all paths under one name. Without it, every distinct URL becomes its own series in your metrics. Tag by endpoint group if you want a breakdown, such as product and search.
Writes. A read-only mix tells you nothing about the write path. Add POST requests in the same proportion as production, with test accounts and data you can delete after the run.
Run it in steps
Pick the steps from your current peak. Start near half of it, and size each step at about a tenth of the peak, so you get enough points around the knee. I’d rather run held steps than a smooth ramp. Each hold gives pools, caches, garbage collectors and queues time to settle, and you get a clean reading per step. Five minutes is a reasonable hold; lengthen it if a metric is still moving at the end of a step.
Send the results to Prometheus, so you can read each step next to your server metrics:
export K6_PROMETHEUS_RW_SERVER_URL=http://prometheus.internal:9090/api/v1/write
export K6_PROMETHEUS_RW_TREND_STATS="p(95),p(99),max"
k6 run -o experimental-prometheus-rw \
-e BASE_URL=https://staging.example.com -e STEP=100 -e STEPS=12 \
breakpoint.js
Prometheus 2.x needs --web.enable-remote-write-receiver to accept these writes. Run k6 from a machine in the same region as staging, and watch its CPU. A load generator above about 80% CPU slows its own requests, and you’d measure k6 instead of your system.
While each step holds, watch the saturation signals for every component in the request path, in order from the edge:
| Component | What to watch | Recipe |
|---|---|---|
| CDN | Origin request rate, hit ratio | Cache hit ratio at peak |
| HAProxy | Backend queue, sQ in the log | HAProxy maxconn tuning |
| nginx | upstream timed out, worker_connections are not enough | Upstream timeouts and 502, nginx tuning |
| Linux host | Listen overflows, TIME_WAIT, conntrack | Linux network tuning |
| Node.js app | Event loop lag, one busy core | Node.js cluster mode |
| PostgreSQL | Connections, cl_waiting in PgBouncer | PgBouncer pooling |
| Redis | Memory, evictions, command latency | Redis memory and eviction |
Stop the run yourself once p99 sits well past the budget for a full step. Pushing further teaches you little, and it can leave queues and pools in a state that takes a while to clear.
Read the results
Plot the k6 series against the arrival rate. These queries give one line per metric:
# requests per second k6 completed
sum(rate(k6_http_reqs_total[1m]))
# p99 latency as k6 saw it
max(k6_http_req_duration_p99)
# share of failed requests
avg(k6_http_req_failed_rate)
# iterations k6 couldn't start
sum(rate(k6_dropped_iterations_total[1m]))
Check the series names in your Prometheus first. The output adds _total to counters and _rate to rate metrics. The breaking point is the last step where p99 and the error rate both stayed inside budget for the whole hold.
Then find the component that gave out first. Look at the moment the first k6 metric went wrong, and find the server metric that moved before it:
| What you see | Likely cause | Where to look |
|---|---|---|
| Errors with flat latency | A hard limit: connection slots, file descriptors, ports, pool size | Proxy error logs, too many clients, (24: Too many open files), (99: Cannot assign requested address) |
| Latency climbs, CPU near full | The CPU-bound tier ran out | The tier with the highest CPU, then its profile |
| Latency climbs, CPU idle | Waiting on a lock, pool or dependency | Pool wait times, database locks, thread pools |
| 504s from the proxy, slow database | The database is the bottleneck | Slow query log, pg_stat_statements, connection counts |
Throughput flat, dropped_iterations rising, no server errors | k6 ran short of virtual users, or the system queues silently | k6 warnings, server queue metrics |
Fix the first limit only, then run the test again. Each fix moves the bottleneck to the next component, and two changes in one run hide which one helped. Postgres on its own gets a separate method in the pgbench recipe.
Record three numbers from each run: the breaking point, the component that failed and the change you made. That record shows whether your capacity grows ahead of your traffic.
Make it a habit
A breakpoint from last quarter tells you little about this release. I run the test before each launch or campaign that may bring a spike. I also run it after changes to instance sizes, pool sizes or the database. A scheduled run against staging catches the drift in between. This GitHub Actions workflow runs it weekly on a self-hosted runner inside your network:
name: breakpoint
on:
schedule:
- cron: '0 3 * * 1'
workflow_dispatch:
jobs:
k6:
runs-on: [self-hosted, load-test]
steps:
- uses: actions/checkout@v4
- uses: grafana/setup-k6-action@v1
- name: Breakpoint test
env:
K6_PROMETHEUS_RW_SERVER_URL: http://prometheus.internal:9090/api/v1/write
run: k6 run -o experimental-prometheus-rw -e BASE_URL=https://staging.example.com load/breakpoint.js
A failed threshold makes k6 exit with a non-zero code, so the job fails and someone looks. Keep the weekly run below anything production shares, such as a database or a third-party API with rate limits.
Then turn the result into an alert. Production traffic approaching the tested capacity is the warning you want before the next peak. With nginx-prometheus-exporter at the edge:
groups:
- name: capacity
rules:
- alert: TrafficNearTestedCapacity
# 1200 stands for the breaking point from your last test, in requests per second
expr: sum(rate(nginx_http_requests_total[5m])) > 0.7 * 1200
for: 10m
labels:
severity: warning
annotations:
summary: "Traffic is above 70% of the capacity the last breakpoint test found"
description: "Check which component failed first in the last test and scale it now. Rerun the breakpoint test if the system changed since then."
Update the number after each run. With an old breaking point, the alert fires too early after a fix and too late after a regression.
More recipes
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.
- NAT Gateway ErrorPortAllocation and Port Exhaustion
Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.