k6 Breakpoint Testing to Find Your Breaking Point

Ramp a fixed arrival rate with k6 until latency or errors cross your limits, so you know your real peak capacity and which component gives out first.

Published

Applies to k6 v0.45 and later, including v1.x; the Prometheus output needs Prometheus with the remote write receiver on

k6 breakpoint testing raises the request rate step by step until your system stops meeting its targets. The last step that passed is your capacity. The first metric that went wrong tells you which component to fix. Every load-test recipe in this cookbook uses the method below and keeps only its own script.

Where it breaks

Your breaking point is the arrival rate where p99 latency crosses your budget or the error rate crosses your error budget, whichever comes first. Write both numbers down before you start. Without them, you’ll call any result a pass.

You know you’ve reached it when one of these shows up:

  • p99 bends upward. Latency stays flat across several steps, then climbs fast. That bend is the knee, and the step before it is your usable capacity.
  • Errors start. 502s and 504s from a proxy, refused connections, too many clients from a database or timeouts from the app.
  • Throughput stops following the arrival rate. k6 asks for more requests a second, and completed requests stay flat. k6 then reports dropped_iterations, because no virtual user was free to start the next one.

Use a staging copy with production instance sizes, production data volume and the same versions. Turn off autoscaling, or cap it at the production maximum. The k6 docs warn that an elastic system keeps growing during the test and you find your cloud bill’s limit instead of the system’s.

Set up the test with ramping-arrival-rate

k6’s default executors are closed-loop: each virtual user waits for its last response before it sends the next request. When the system slows, the load drops with it and the latency looks better than reality. The ramping-arrival-rate executor starts iterations at the rate you set, however slow the responses get. That’s the behavior a breakpoint test needs.

Build the request mix from production logs. Take a day of access log, count requests per path pattern and write a list where each path appears in proportion to its traffic. Keep real query strings and IDs, so caches and indexes see real variety. Save it as paths.json, a JSON array of paths.

// breakpoint.js
import http from 'k6/http';
import { check } from 'k6';
import { SharedArray } from 'k6/data';

const paths = new SharedArray('paths', () => JSON.parse(open('./paths.json')));
const BASE = __ENV.BASE_URL || 'https://staging.example.com';
const STEP = Number(__ENV.STEP || 100);   // requests per second added each step
const STEPS = Number(__ENV.STEPS || 10);
const HOLD = __ENV.HOLD || '5m';

const stages = [];
for (let i = 1; i <= STEPS; i++) {
  stages.push({ target: STEP * i, duration: '1m' }); // ramp to the next step
  stages.push({ target: STEP * i, duration: HOLD }); // hold it
}

export const options = {
  discardResponseBodies: true,
  scenarios: {
    breakpoint: {
      executor: 'ramping-arrival-rate',
      startRate: STEP,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 5000,
      stages,
    },
  },
  thresholds: {
    http_req_failed: [{ threshold: 'rate<0.05', abortOnFail: true, delayAbortEval: '2m' }],
    http_req_duration: ['p(99)<800'],
  },
};

export default function () {
  const path = paths[Math.floor(Math.random() * paths.length)];
  const res = http.get(`${BASE}${path}`, { tags: { name: 'mix' } });
  check(res, { 'status below 400': (r) => r.status < 400 });
}

Executor settings. startRate and each stage’s target count iterations per timeUnit, here per second. Each iteration sends one request, so the target is your request rate. preAllocatedVUs are the virtual users k6 creates up front. maxVUs caps how many it may add when responses slow down. If k6 hits that cap it logs Insufficient VUs, reached 5000 active VUs and cannot initialize more and drops iterations. That’s a limit on the client side, so raise maxVUs or add load generators before you read anything into the result.

Thresholds. k6 checks thresholds over the whole run, from the first request. A p99 threshold therefore lags behind the step you’re on, because the early quiet steps pull it down. I use the error-rate threshold to stop a run that’s broken. delayAbortEval keeps a single bad sample from ending it. I read the per-step latency from the time series instead, as shown below.

Request names. tags: { name: 'mix' } groups all paths under one name. Without it, every distinct URL becomes its own series in your metrics. Tag by endpoint group if you want a breakdown, such as product and search.

Writes. A read-only mix tells you nothing about the write path. Add POST requests in the same proportion as production, with test accounts and data you can delete after the run.

Run it in steps

Pick the steps from your current peak. Start near half of it, and size each step at about a tenth of the peak, so you get enough points around the knee. I’d rather run held steps than a smooth ramp. Each hold gives pools, caches, garbage collectors and queues time to settle, and you get a clean reading per step. Five minutes is a reasonable hold; lengthen it if a metric is still moving at the end of a step.

Send the results to Prometheus, so you can read each step next to your server metrics:

export K6_PROMETHEUS_RW_SERVER_URL=http://prometheus.internal:9090/api/v1/write
export K6_PROMETHEUS_RW_TREND_STATS="p(95),p(99),max"
k6 run -o experimental-prometheus-rw \
  -e BASE_URL=https://staging.example.com -e STEP=100 -e STEPS=12 \
  breakpoint.js

Prometheus 2.x needs --web.enable-remote-write-receiver to accept these writes. Run k6 from a machine in the same region as staging, and watch its CPU. A load generator above about 80% CPU slows its own requests, and you’d measure k6 instead of your system.

While each step holds, watch the saturation signals for every component in the request path, in order from the edge:

ComponentWhat to watchRecipe
CDNOrigin request rate, hit ratioCache hit ratio at peak
HAProxyBackend queue, sQ in the logHAProxy maxconn tuning
nginxupstream timed out, worker_connections are not enoughUpstream timeouts and 502, nginx tuning
Linux hostListen overflows, TIME_WAIT, conntrackLinux network tuning
Node.js appEvent loop lag, one busy coreNode.js cluster mode
PostgreSQLConnections, cl_waiting in PgBouncerPgBouncer pooling
RedisMemory, evictions, command latencyRedis memory and eviction

Stop the run yourself once p99 sits well past the budget for a full step. Pushing further teaches you little, and it can leave queues and pools in a state that takes a while to clear.

Read the results

Plot the k6 series against the arrival rate. These queries give one line per metric:

# requests per second k6 completed
sum(rate(k6_http_reqs_total[1m]))

# p99 latency as k6 saw it
max(k6_http_req_duration_p99)

# share of failed requests
avg(k6_http_req_failed_rate)

# iterations k6 couldn't start
sum(rate(k6_dropped_iterations_total[1m]))

Check the series names in your Prometheus first. The output adds _total to counters and _rate to rate metrics. The breaking point is the last step where p99 and the error rate both stayed inside budget for the whole hold.

Then find the component that gave out first. Look at the moment the first k6 metric went wrong, and find the server metric that moved before it:

What you seeLikely causeWhere to look
Errors with flat latencyA hard limit: connection slots, file descriptors, ports, pool sizeProxy error logs, too many clients, (24: Too many open files), (99: Cannot assign requested address)
Latency climbs, CPU near fullThe CPU-bound tier ran outThe tier with the highest CPU, then its profile
Latency climbs, CPU idleWaiting on a lock, pool or dependencyPool wait times, database locks, thread pools
504s from the proxy, slow databaseThe database is the bottleneckSlow query log, pg_stat_statements, connection counts
Throughput flat, dropped_iterations rising, no server errorsk6 ran short of virtual users, or the system queues silentlyk6 warnings, server queue metrics

Fix the first limit only, then run the test again. Each fix moves the bottleneck to the next component, and two changes in one run hide which one helped. Postgres on its own gets a separate method in the pgbench recipe.

Record three numbers from each run: the breaking point, the component that failed and the change you made. That record shows whether your capacity grows ahead of your traffic.

Make it a habit

A breakpoint from last quarter tells you little about this release. I run the test before each launch or campaign that may bring a spike. I also run it after changes to instance sizes, pool sizes or the database. A scheduled run against staging catches the drift in between. This GitHub Actions workflow runs it weekly on a self-hosted runner inside your network:

name: breakpoint
on:
  schedule:
    - cron: '0 3 * * 1'
  workflow_dispatch:
jobs:
  k6:
    runs-on: [self-hosted, load-test]
    steps:
      - uses: actions/checkout@v4
      - uses: grafana/setup-k6-action@v1
      - name: Breakpoint test
        env:
          K6_PROMETHEUS_RW_SERVER_URL: http://prometheus.internal:9090/api/v1/write
        run: k6 run -o experimental-prometheus-rw -e BASE_URL=https://staging.example.com load/breakpoint.js

A failed threshold makes k6 exit with a non-zero code, so the job fails and someone looks. Keep the weekly run below anything production shares, such as a database or a third-party API with rate limits.

Then turn the result into an alert. Production traffic approaching the tested capacity is the warning you want before the next peak. With nginx-prometheus-exporter at the edge:

groups:
  - name: capacity
    rules:
      - alert: TrafficNearTestedCapacity
        # 1200 stands for the breaking point from your last test, in requests per second
        expr: sum(rate(nginx_http_requests_total[5m])) > 0.7 * 1200
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Traffic is above 70% of the capacity the last breakpoint test found"
          description: "Check which component failed first in the last test and scale it now. Rerun the breakpoint test if the system changed since then."

Update the number after each run. With an old breaking point, the alert fires too early after a fix and too late after a regression.

More recipes

Further reading