Find Your p99 Latency Limits Before Launch Day

A launch-readiness process for CTOs: set p99 latency budgets, model real traffic with k6, find what breaks first and gate releases on p99.

  • Go to the profile of Safoor Safdar
·13 min read
Title card with a p99 latency line rising above a dashed budget line during a highlighted peak-hours window

You have a launch, a marketing push or a large customer onboarding coming, and your team can’t say how many users the app can handle. At peak, a few requests start taking seconds, and the support queue fills up before the error rate moves. Before that date, I find the p99 limits using latency budgets, realistic traffic models, a check of common causes and a CI gate.

Key Takeaways

  • Averages hide the slowest 1% of requests, so write down a p99 budget for each endpoint before you run any test.
  • Test with an open model (a constant arrival rate), so coordinated omission can’t hide slow periods.
  • Aim for 2x to 3x your forecast peak, because a launch forecast is an estimate made before real users arrive.
  • At peak, connection pools, database IOPS and autoscaling capacity are common first failures, and CPU can look fine while they run out.
  • Put p99 thresholds in CI so a release that breaks the budget fails the pipeline.

Why averages hide the real problem

Average latency hides the experience of your slowest users. p99 latency is the response time that 99% of requests beat. It describes the slowest 1% of requests, which at peak traffic can mean many users each minute.

Latency in web apps tends to have a long tail instead of a neat bell curve. Most requests hit warm caches and fast paths. A small group waits on a lock, a cold container, a garbage collection pause or a slow query. An average of 120 ms can sit on top of a p99 of 3 seconds, and the average moves little when the tail doubles. Your dashboards can show a healthy average on a normal Tuesday while that small group waits.

The tail compounds across a session. One page load can fan out into many API calls, and a user session contains many page loads. Each extra request raises the chance that at least one lands in the slow 1%. Your team can say “the API is fast on average” while customers say the app is slow, and both statements hold.

Sentry’s engineering team documented a case of this. A cold start problem in their workers affected roughly 5% of requests. With 5% of requests affected, the p99 lands inside the slow group and shows the full delay. The average blends it with the fast 95%. Fixing it took p99 latency on their RPC endpoints from 2.4 seconds to 176 milliseconds.

Setting latency budgets per endpoint

A latency budget is a written p99 target for each endpoint that matters, agreed before you test. Without one, you get numbers from a load test and no pass or fail decision.

Set budgets per endpoint type, because a login call, a search query and a PDF export carry different user expectations and costs. I group endpoints by how the user experiences them:

Endpoint typeExampleStarting p99 budget
Interactive readsDashboard data, search300 to 500 ms
Interactive writesCheckout, form submit500 ms to 1 s
Auth and sessionLogin, token refresh300 to 800 ms
Background or asyncExports, webhooksMeasured on queue time, not request time

These numbers are my own starting points for a conversation with product, so adjust them to your users. Write them down next to an error-rate budget for the same endpoints.

If you already run SLOs, the latency budget should line up with your latency SLO. If you don’t, my post on why you should start with SLOs before hiring an SRE covers how to pick targets the business will defend.

Use an arrival rate instead of looping virtual users

A realistic load test reproduces the arrival rate, request mix and geography of your expected peak. If you get this wrong, the test measures a traffic pattern your users won’t produce. An open model sends requests at the rate real users would, even while your system is slow.

Many test scripts start a fixed number of virtual users that loop. Each one sends a request, waits for the response, then sends the next, which makes it a closed model. As the system slows down, each virtual user sends fewer requests, so the load drops at the moment it should rise. Your real users keep clicking while the server struggles.

Coordinated omission happens when the load generator waits on the system under test and skips requests it would have sent during a stall. Gil Tene popularized the term in his talk How NOT to Measure Latency. Your percentiles leave out the slow periods, and your p99 looks healthy because the test stopped measuring when things went wrong.

An open model fixes this, because requests arrive at a fixed rate whether or not earlier ones have finished. In k6 that’s the constant-arrival-rate executor. It starts iterations at the rate you set, as long as VUs are available.

The k6 example below holds 200 iterations per second for 10 minutes. Each iteration makes two requests, so the test sends about 400 requests per second. The run fails if any endpoint’s p99 or error rate goes over budget.

import http from 'k6/http';
import { check } from 'k6';

const BASE_URL = __ENV.BASE_URL || 'https://staging.example.com';

export const options = {
  scenarios: {
    peak_traffic: {
      executor: 'constant-arrival-rate',
      rate: 200,
      timeUnit: '1s',
      duration: '10m',
      preAllocatedVUs: 200,
      maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    'http_req_failed{name:search}': ['rate<0.01'],
    'http_req_failed{name:checkout}': ['rate<0.01'],
    'http_req_duration{name:search}': ['p(99)<500'],
    'http_req_duration{name:checkout}': ['p(99)<1000'],
    dropped_iterations: ['count<1'],
    checks: ['rate>0.99'],
  },
};

export default function () {
  const search = http.get(`${BASE_URL}/api/search?q=shoes`, {
    tags: { name: 'search' },
  });
  check(search, { 'search is 200': (r) => r.status === 200 });

  const payload = JSON.stringify({ sku: 'SKU-123', qty: 1 });
  const checkout = http.post(`${BASE_URL}/api/checkout`, payload, {
    headers: { 'Content-Type': 'application/json' },
    tags: { name: 'checkout' },
  });
  check(checkout, { 'checkout is 200': (r) => r.status === 200 });
}

Durations in k6 thresholds are in milliseconds. The name tag groups requests per endpoint, so each one gets its own latency budget and error threshold. I left out a global duration threshold on purpose, because it would mix both endpoints into one percentile and blur the per-endpoint budgets.

The dropped_iterations threshold catches the case where k6 ran out of VUs and couldn’t hold the target rate. Without it, k6 would fall below the target rate without failing, and your open model would turn into a weaker test. The checks threshold fails the run unless more than 99% of check() calls pass, so failed status checks fail the run too.

Size preAllocatedVUs at the rate multiplied by the expected iteration time, so k6 doesn’t allocate VUs mid-test. If each iteration takes about one second, 200 iterations per second needs 200 VUs, which matches the example. maxVUs gives k6 room to add VUs when iterations slow down under load.

Model the request mix, data and regions

The arrival rate sets how often requests arrive, and the rest of the model sets what those requests do:

  • Mix. Take the ratio of reads to writes and the top endpoints from production logs or your APM.
  • Data shape. Test against a dataset close to production size. A query that’s fast on 10,000 rows can fall over on 50 million.
  • Cache state. Run at least one test with cold caches. A deploy or a cache flush on launch day leaves you with cold caches too.
  • Regions. Generate load from more than one region if your users are spread out. A test run from one region next to your cluster hides TLS handshake costs, CDN misses and cross-region database calls.
  • Shape over time. Add a ramp scenario (k6 has ramping-arrival-rate) next to the flat plateau to see how autoscaling reacts.

Target 2x to 3x your forecast peak. A launch forecast is an estimate made before real users arrive, and the headroom gives you time to react on the day.

Shared resources that break first at peak

Apps that fail at peak hours tend to run out of a shared resource, such as connections or IOPS, before CPU. These are the ones I check first:

CauseWhat you seeWhere to look
Connection pool exhaustionp99 climbs in steps, then timeouts, while the database looks idleApp pool wait time, DB connection count
N+1 queriesLatency grows with data size, query count per request is highAPM traces, query count per request
GC pausesPeriodic spikes across all endpoints at onceRuntime GC logs and pause metrics
Noisy neighborsSpikes that do not match your own trafficNode-level CPU steal, shared DB or cache metrics
Cold startsSlow first requests after scale-out or deployNew pod or function start time, readiness timing
Autoscaling lagLatency spikes for minutes after traffic rises, then recoversTime from metric breach to ready capacity
Database at its IOPS limitWrite latency and queue depth rise togetherDisk queue depth, read and write IOPS against provisioned limit

Autoscaling lag can leave you under-provisioned for several minutes. Your monitoring scrapes metrics on an interval, and the scaler acts on an average of those samples. New nodes take time to join, and new pods need time to warm up. Pre-scale before a known launch instead of trusting the scaler to catch a sudden step change.

On Amazon RDS, the IOPS ceiling comes from the storage type and size you provisioned. The instance class can cap it lower, so check both. On Aurora, watch volume IOPS alongside instance throughput. I wrote about building CloudWatch dashboards for Aurora IOPS for this reason.

If you run Prometheus, you can watch p99 per route during the test with histogram_quantile:

histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

Your instrumentation library decides the metric name and the route label, so adjust both to match your setup. The query returns seconds while the k6 budgets are in milliseconds, and a [1m] window tracks a 10-minute test with less smoothing than [5m].

The result is an estimate interpolated from bucket boundaries. If your buckets jump from 1 s to 5 s, the reported p99 can land far from the real value. Put bucket edges near your budgets.

Gating releases on p99 in CI

A p99 gate in CI makes each release candidate pass your latency budget before it ships. You catch regressions while the change is small and its author still remembers it.

k6 exits with a non-zero code when a threshold fails, so most CI systems fail the job without extra scripting. A workable setup looks like this:

  1. Deploy each release candidate to a staging environment with production-like sizing.
  2. Run a short arrival-rate test (5 to 10 minutes) at your normal peak rate.
  3. Fail the pipeline if p99 or error rate breaks the budget for any tagged endpoint.
  4. Run the full 2x to 3x peak test on a schedule and before every planned launch.

Shared staging environments are noisy, so give the gate some tolerance and judge trends across several runs instead of a single result. Keep the thresholds in the repo next to the test, so a budget change goes through code review like any other change.

My approach before a launch

I start with production data. I pull the real request mix, the busiest endpoints and their current p99 from monitoring. Then I agree budgets with the product owner and write the k6 scenarios last.

At IDWise, an identity verification SaaS, k6 load tests found the bottlenecks that gave 3x traffic headroom. I also helped the team move from a serverless architecture to containers on Kubernetes, and the platform held a 99.95% uptime SLA.

At Niche Academy, an EdTech platform, we raised requests-per-second throughput and cut unplanned outages by 90%. Monitoring matters as much as tuning in work like this, because it shows you the next limit before users hit it.

At Systems Limited, I consulted for Telenor, deploying production Kubernetes, building Prometheus and Grafana monitoring and running chaos experiments.

You can read about my background in cloud reliability or see more of this history on my experience page.

A launch readiness checklist

You’re ready for launch day when you can point to evidence for each row below.

CheckWhat “done” looks like
Latency budgets per endpointA written p99 target and error-rate budget for each endpoint that matters, agreed with product before testing
Arrival-rate test at a stated peak multipleAn open-model test (constant arrival rate) at 2x to 3x your forecast peak, with no dropped iterations
Thresholds wired into CIk6 p99 and error-rate thresholds per tagged endpoint, kept in the repo and failing the pipeline on a breach
Tests from more than one regionLoad generated from more than one region when your users are spread out, not only next to your cluster
Dashboards and alerts on p99p99 per route on a dashboard, histogram bucket edges near your budgets, and an alert when p99 goes over budget

Frequently asked questions

What is p99 latency?

p99 latency is the response time that 99% of requests complete within. The slowest 1% take longer than that value. It’s a better guide to user experience at peak than the average, because it shows the tail where problems first appear.

How many users can my app handle?

A load test gives you the answer as requests per second at a given p99, which is more useful than a user count. Find your peak arrival rate from production data, then test with an open model at 2x to 3x that rate. The highest rate that stays inside your p99 and error budgets is your capacity.

Why does my app go down at peak hours when CPU looks fine?

A shared resource such as database connections, IOPS, a thread pool or cache capacity tends to run out before CPU does. Autoscaling lag and cold starts make it worse, because new capacity arrives minutes after the spike. Check pool wait times and database queue depth before adding more servers.

How do I reduce API latency at p99?

Find the cause before tuning. Trace the slowest requests and look for N+1 queries, pool waits, GC pauses and cold starts. Fix the biggest contributor first. Then add a p99 threshold in CI, so a later release can’t undo the fix without failing the pipeline.