Your product had two outages last month, both at peak hours, and one looked a lot like an incident from the spring. Someone on the leadership team says “we need an SRE,” and someone drafts a job description that week. Deciding when to hire an SRE gets easier once you have a 30-day baseline, your first SLOs and an error budget policy.
Key Takeaways
- Reliability needs an owner once you see two or more signs: peak-hour outages, repeat incidents, on-call fatigue, or churn and sales friction.
- You can adopt SRE practices without an SRE, starting with SLOs set from 30 days of measured data.
- Set your first SLO a notch below the measured baseline, and back it with a one-page error budget policy that roadmap owners sign.
- In my view, many Series A and B teams benefit from a time-boxed reliability engagement (I offer these) before the first SRE hire.
At that point you can’t say what the new SRE should fix first, or how you’d know they fixed it. For a Series A or B SaaS company, the sequence runs baseline, one to three SLOs, then a policy. Once those exist, you can decide between hiring, contracting or waiting from data.
Signs reliability has become a business problem
Reliability becomes a business problem when it starts costing you revenue, customers or engineering time in ways that show up outside engineering. Before that point, a few rough weeks are normal for a growing product.
I look for four signals:
- Outages at peak hours. Incidents land when customers are most active, because that’s when load exposes weak points. A five-minute blip at 3 a.m. costs you less than five minutes down during a customer’s busiest hour.
- Repeat incidents. The same class of failure keeps coming back because the follow-up work keeps losing to features on the roadmap. Common examples are a saturated database connection pool, a noisy neighbor on a shared node, or a bad config rollout.
- On-call fatigue. The same two or three engineers get paged, alerts fire without a clear action, and people start dreading their rotation.
- Churn and sales friction. Customers mention reliability in renewal calls, or enterprise prospects ask for uptime commitments you can’t back with data. Support tickets may also spike after each incident.
If you recognize two or more of these, reliability needs an owner and a plan. At this stage, that owner doesn’t have to be a full-time SRE.
SRE practices without an SRE team
You can adopt the core SRE practices before anyone on your team has SRE in their title. Google’s SRE workbook makes the same point: “Even if you don’t have SREs, you can adopt SRE practices by using SLOs.”
A service level objective (SLO) is a reliability target for a service, such as 99.9% of requests succeeding over 30 days.
The same chapter lists the basic steps. Accept that 100% reliability isn’t the goal, set a reasonable SLO and agree on an error budget policy. Then measure the SLO and stick to the policy. You can do all five without a dedicated team. You do need someone with the authority to get product and engineering to agree, and the technical depth to instrument the service.
Many startups muddle the SRE versus DevOps question at this point. A DevOps or platform engineer owns the build, deploy and runtime path for your code. SRE practice adds a contract about how reliable the service must be and what happens when it falls short. You may already have the first and still lack the second. If your gap is a reliability target nobody has agreed on, you can close it with an SLO before you spend on a hire.
Setting your first SLOs from real data
Your first SLOs should come from 30 days of measured data on one or two user-facing journeys. A number someone picked in a meeting won’t hold up. Start with the API or page your customers would notice first if it broke, and work through four steps:
- Measure a 30-day baseline.
- Pick one to three SLIs.
- Set the target.
- Do the error budget math.
Measure a 30-day baseline. Before you set any target, collect a month of request data from your load balancer, ingress controller or application metrics. You want two numbers per endpoint group: the share of requests that succeeded, and the share that completed under a latency threshold. Thirty days covers four weekly traffic cycles and, for most teams, a deploy or two that went wrong.
Pick one to three SLIs. An SLI (service level indicator) is the measurement. The Google workbook recommends treating the SLI as the ratio of good events to total events. That keeps every SLI on a 0 to 100% scale. For a typical SaaS API, two SLIs cover most of what users feel:
- Availability: good requests divided by total requests, where “good” means the response wasn’t a server error (HTTP 5xx). The rule below counts only 5xx responses as bad. If you return 429s because you’ve run out of capacity, add them to the bad set. Your bad-request selector becomes
code=~"5..|429", and the rule’s numerator becomescode!~"5..|429". Per-client quota rejections can stay good. Your app doesn’t record timeouts that happen at the load balancer, so take those from ingress or load balancer metrics. - Latency: the share of requests that completed under a threshold, for example the share served in under 300 ms. You’ll find it easier to reason about than a raw p99 number, because it turns latency into the same good-over-total ratio. I covered how to choose that threshold in p99 latency and load testing before launch.
This Prometheus recording rule pair covers both SLIs. It assumes the common http_requests_total counter with a code label, plus an http_request_duration_seconds histogram that has a 0.3 bucket. Adjust the metric and label names to match your instrumentation.
groups:
- name: api-slis
rules:
- record: sli:api_availability:ratio_rate5m
expr: |
sum(rate(http_requests_total{job="api", code!~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))
- record: sli:api_latency_under_300ms:ratio_rate5m
expr: |
sum(rate(http_request_duration_seconds_bucket{job="api", le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count{job="api"}[5m]))
The latency rule counts a fast 5xx response as good latency. If your histogram carries a code label, add a code!~"5.." filter to the bucket selector in the numerator.
For the 30-day baseline itself, run an ad hoc query such as sum(increase(http_requests_total{job="api", code!~"5.."}[30d])) / sum(increase(http_requests_total{job="api"}[30d])). The query needs at least 30 days of data. Prometheus keeps 15 days by default, so raise the retention or use a long-term store such as Thanos or Mimir. A 30-day range query is heavy, so run it on demand rather than as a recording rule.
Set the target. The workbook advises that current performance can be a good place to start when you have nothing else. It also warns against letting that number limit you later. I recommend setting the first SLO a notch below the measured baseline. The team then starts inside its budget, and the target reflects what customers already accept. If your baseline availability was 99.94%, a 99.9% SLO is a sensible first target. At a 99.6% baseline, a 99.9% target would put you over budget from day one and freeze the roadmap.
Do the error budget math. An error budget is the amount of unreliability the SLO allows. A 30-day window has 43,200 minutes:
| SLO over 30 days | Allowed bad fraction | Budget in minutes of full outage | Budget per 10M requests |
|---|---|---|---|
| 99.0% | 1% | 432 min (7.2 h) | 100,000 failed requests |
| 99.5% | 0.5% | 216 min (3.6 h) | 50,000 failed requests |
| 99.9% | 0.1% | about 43 min | 10,000 failed requests |
| 99.95% | 0.05% | about 22 min | 5,000 failed requests |
The minutes column assumes a full outage. Your budget can also go on partial failures, such as 2% of requests failing for two hours. An uptime check can pass through that whole window, while a request-based SLI counts each failed request.
An error budget policy people follow
An error budget policy is a short, signed agreement that says what the team does when the budget runs low or runs out. Without one, your SLO is a dashboard people check after an incident.
Google’s example error budget policy makes a good template. Its core rule covers a service that has spent its error budget over the preceding four weeks. In that case the team halts changes and releases until the service is back within its SLO, apart from P0 issues and security fixes. A single incident that consumes more than 20% of the budget over four weeks triggers a postmortem with at least one P0 action item. The CTO settles disagreements about the calculation or the required actions.
For a Series A or B company, I’d keep the first version to one page with five parts:
- Scope. The service and SLOs the policy covers.
- Thresholds. For example, a warning when 50% of the monthly budget is spent, and a feature freeze when 100% is spent.
- Actions. The changes at each threshold: who reviews deploys, which work moves forward, and when the team runs a postmortem.
- Exceptions. Security fixes, incidents caused by an upstream provider, and load tests that knowingly consume budget.
- Owner and escalation. One named person who arbitrates, such as the CTO or VP Engineering.
Get the roadmap owners to sign the policy before the first freeze, rather than in the middle of one. That signature needs a conversation with product, and many reliability efforts stall at that step.
Once the policy exists, page on how fast the budget is burning rather than on raw error spikes. The Google workbook covers this as multi-window, multi-burn-rate alerting.
Hire, contract or wait
With a baseline, SLOs and a policy in place, you can base the hiring decision on workload data. You’ll see how often the budget is at risk, how much follow-up work incidents create, and whether your existing engineers can absorb it.
A recruiting firm that places startup SREs puts the typical threshold for a first SRE hire at around 15 to 40 engineers. The firm pairs that range with substantial production traffic and a handful of paying enterprise customers. It also warns against hiring before reliability problems are real and costly. Use those numbers as a rough guide alongside your own SLO data.
| Option | Fits when | Watch out for |
|---|---|---|
| Wait | SLOs are met most months, incidents are rare and do not repeat, and your current engineers handle on-call without burnout | Waiting without SLOs. Then “wait” just means “not measuring” |
| Time-boxed engagement or consultant | You have reliability pain but no baseline, no SLOs and no policy, or you need someone to set up the practice and hand it over | Engagements with no defined end or no internal owner to take over |
| Full-time SRE hire | SLOs exist, the budget is under pressure most months, follow-up work is steady, and you have the scale the hire will grow into | Hiring one SRE and making them the on-call rotation and the platform team at once |
For many Series A and B teams, I recommend a time-boxed reliability engagement before the first SRE hire. I offer engagements like this, so weigh my view with that in mind. The setup work (instrumentation, the baseline, the first SLOs, the policy, alert cleanup) is front-loaded and finite. After that, you’ll know whether there’s a full-time job’s worth of ongoing work. You’ll also have what you need to write a sharper job description for the person who’ll own it. If you’re comparing part-time senior help more broadly, I broke down what a fractional CTO costs and when to hire one.
Skip the engagement and hire now if you run a multi-region platform with regulatory uptime commitments. The same applies if your incident volume already keeps one engineer busy full time.
How I approach this
I approach a reliability engagement by measuring first and setting targets second, then leaving the team with a practice they can run without me. I’ve spent eight years on cloud infrastructure across AWS, Azure, GCP and Oracle Cloud, and I follow the sequence above.
At Niche Academy, an EdTech platform, I cut unplanned outages by 90% and raised requests per second. At IDWise, an identity verification SaaS, I helped move the platform from serverless to containers on Kubernetes, managed with Terraform. We deployed it on Azure, Oracle Cloud and GCP to meet regional compliance. k6 load tests found the bottlenecks that gave 3x traffic headroom.
Earlier, at Systems Limited, I built observability on EKS with the Prometheus stack for Regeneron. For Telenor I ran production Kubernetes with Prometheus and Grafana, plus chaos experiments.
The output of this work is an agreed definition of “working,” plus the data to show whether the service meets it. At IDWise I ran a platform that held a 99.95% uptime SLA, so I’ve owned a target like this myself. You can read how I work or see the details on my experience page.
Frequently asked questions
When should a startup hire its first SRE?
Hire when you have SLOs, the error budget is under pressure most months, and incident follow-up work is steady enough to fill a role. Before that, a part-time reliability owner on your team or a short engagement can get you further for less.
How do I set my first SLO for a SaaS product?
Measure 30 days of request data for your most important user journey. Then pick an availability SLI and a latency SLI as good-over-total ratios. Set the first target a notch below your measured baseline so you start within budget, and tighten it later if customers need more.
What does a 99.9% SLO mean in practice?
A 99.9% SLO over a 30-day window lets 0.1% of requests fail. That’s about 43 minutes of full outage, or 10,000 failed requests out of every 10 million. You can also spend that budget on partial failures, such as a small error rate that lasts for hours.
Is SRE different from DevOps for a startup?
DevOps and platform work cover how your team builds, ships and runs software. SRE adds explicit reliability targets, error budgets and a policy for what the team does when it misses a target. A startup can practice both with the same engineers before it hires a specialist.






