Provides Site Reliability Engineering best practices for SLOs, SLIs, SLAs, error budgets, toil reduction, reliability reviews, and capacity planning. Use when defining service objectives, measuring reliability, reducing toil, planning capacity, or when user mentions 'SRE', 'SLO', 'SLI', 'SLA', 'error budget', 'toil', 'reliability', 'on-call', 'capacity planning'.
Provides Site Reliability Engineering best practices for SLOs, SLIs, SLAs, error budgets, toil reduction, reliability reviews, and capacity planning. Use when defining service objectives, measuring reliability, reducing toil, planning capacity, or when user mentions 'SRE', 'SLO', 'SLI', 'SLA', 'error budget', 'toil', 'reliability', 'on-call', 'capacity planning'.
type
skill
category
ops
status
stable
origin
tibsfox
modified
false
first_seen
"2026-02-07T00:00:00.000Z"
first_path
examples/sre-patterns/SKILL.md
superseded_by
null
SRE Patterns
Best practices for building and operating reliable systems using Site Reliability Engineering principles.
SLO / SLI / SLA Definitions
These three concepts form the foundation of SRE. They are distinct and frequently confused.
Concept
Definition
Owner
Example
SLI (Service Level Indicator)
A quantitative measurement of a service attribute
Engineering
99.2% of requests completed in < 300ms
SLO (Service Level Objective)
A target value or range for an SLI
Engineering + Product
99.5% of requests must complete in < 300ms
SLA (Service Level Agreement)
A contract with consequences for missing an SLO
Business + Legal
99.9% uptime or customer receives service credits
Relationship
SLI (what you measure)
--> SLO (what you target, always stricter than SLA)
--> SLA (what you promise externally, with penalties)
Key rule: SLO must be stricter than SLA. If your SLA promises 99.9% uptime, your internal SLO should target 99.95%. The gap is your safety margin.
SLI Specification
SLIs must be precise, measurable, and tied to user experience. Vague indicators lead to meaningless objectives.
SLI Types by Service Category
Service Type
SLI Category
Good Event
Valid Event
Request-driven
Availability
Response status < 500
All HTTP requests
Request-driven
Latency
Response time < 300ms
All HTTP requests
Data pipeline
Freshness
Data age < 10 minutes
All data records
Data pipeline
Correctness
Records with no processing errors
All processed records
Storage system
Durability
Objects retrievable after write
All stored objects
Storage system
Throughput
Read operations < 50ms
All read operations
SLI Specification Example
# sli-specification.yamlservice:payment-apislis:-name:availabilitydescription:Proportionofsuccessfulrequestsspecification:good_event:"HTTP response status code is not 5xx"valid_event:"All HTTP requests to /api/v1/payments/*"measurement_source:load_balancer_logsmeasurement_window:rolling_28_daysimplementation:numerator:"count(status < 500)"denominator:"count(all requests)"exclude:-health_check_endpoints-synthetic_monitoring_requests-name:latencydescription:Proportionofrequestsservedwithinthresholdspecification:good_event:"HTTP response completes within 300ms"valid_event:"All non-background HTTP requests"measurement_source:server_side_metricsmeasurement_window:rolling_28_daysimplementation:numerator:"count(duration_ms <= 300)"denominator:"count(all requests)"percentile_targets:p50:100msp95:250msp99:500ms
Error Budget Calculation
The error budget is the inverse of your SLO -- the amount of unreliability you can tolerate.
An error budget policy defines what happens when the budget is consumed. Without a policy, the budget is just a number.
# error-budget-policy.yamlservice:payment-apislo:99.9%availability(rolling28days)policy_owner:payments-team-leadapproved_by:vp-engineeringeffective_date:2025-01-15budget_thresholds:-level:normalcondition:"budget_consumed < 50%"actions:-Continuenormalfeaturedevelopment-Standarddeploymentcadence(daily)-Routinereliabilityimprovementsasscheduled-level:cautioncondition:"budget_consumed >= 50% AND < 75%"actions:-Reviewrecentdeploymentsforreliabilityimpact-Increasemonitoringalertsensitivity-Prioritizeknownreliability-relatedbugs-Reducedeploymentfrequencytotwiceperweek-level:criticalcondition:"budget_consumed >= 75% AND < 100%"actions:-Haltallnon-reliabilityfeaturework-RequireSREapprovalforeverydeployment-Deployonlybugfixesandreliabilityimprovements-Conductfocusedreliabilityreviewwithin48hours-NotifystakeholdersofSLOrisk-level:exhaustedcondition:"budget_consumed >= 100%"actions:-Featurefreezeuntilbudgetreplenishesorrootcauseresolved-Emergencyreliabilityreviewwithin24hours-Postmortemrequiredforeachnewincident-AlldeploymentsrequireSREsign-offandcanaryphase-WeeklystatusreporttoVPEngineeringescalation:budget_dispute:"Escalate to VP Engineering for arbitration"exemptions:"Launch exemptions require VP+ approval with risk acceptance"
Toil Measurement and Reduction
Toil is work that is manual, repetitive, automatable, reactive, and scales linearly with service growth. Toil is the enemy of reliability engineering.
Toil Characteristics
Characteristic
Description
Example
Manual
Requires a human to perform
SSH into server to restart service
Repetitive
Done more than once or twice
Weekly cert rotation by hand
Automatable
A machine could do it
Copying logs to analysis bucket
Reactive
Triggered by an event, not proactive
Responding to disk full alerts
No enduring value
Does not improve the service
Re-running failed batch jobs
Scales with service
More instances = more work
Manually updating configs per server
Toil Measurement Framework
# toil-tracking.yamlteam:platform-sremeasurement_period:2025-Q1target_toil_budget:30%# Max 30% of team time on toilcategories:-name:incident_responsehours_per_week:8toil_percentage:60%# 60% of incident response is toiltoil_hours:4.8examples:-Manuallyrestartingcrashedservices-Clearingstuckqueueitems-Respondingtoknown-causealertswithoutautomation-name:deployment_supporthours_per_week:6toil_percentage:40%toil_hours:2.4examples:-Manualpre-deploychecklistverification-Runningmigrationscriptsbyhand-Monitoringdashboardsduringeverydeploy-name:capacity_managementhours_per_week:4toil_percentage:75%toil_hours:3.0examples:-Manuallyresizinginstances-Trackingdiskusageviaspreadsheets-Filingticketstorequestquotaincreases-name:access_provisioninghours_per_week:3toil_percentage:90%toil_hours:2.7examples:-Creatingaccountsacrossmultiplesystems-Rotatingcredentialsmanually-Revokingaccessfordepartingemployeessummary:total_team_hours_per_week:200# 5 engineers * 40 hourstotal_toil_hours_per_week:12.9toil_percentage:6.45%status:within_budgettop_reduction_targets:-access_provisioning# 90% toil -- automate with IaC/SCIM-capacity_management# 75% toil -- autoscaling policies-incident_response# 60% toil -- self-healing automation
Capacity Planning
Capacity planning ensures services can handle expected and unexpected load without degradation.
Capacity Planning Model
# capacity_model.pyfrom dataclasses import dataclass
from typing importOptional@dataclassclassCapacityPlan:
service: str
current_peak_rps: float
growth_rate_monthly: float# e.g., 0.05 for 5% per month
headroom_target: float# e.g., 0.30 for 30% headroom
max_rps_per_instance: float
current_instances: int
burst_multiplier: float = 2.0# Expected burst over peakdefprojected_peak(self, months_ahead: int) -> float:
"""Project peak RPS N months from now."""returnself.current_peak_rps * (1 + self.growth_rate_monthly) ** months_ahead
defrequired_capacity(self, months_ahead: int) -> float:
"""Required RPS capacity including headroom and burst tolerance."""
projected = self.projected_peak(months_ahead)
with_burst = projected * self.burst_multiplier
with_headroom = with_burst / (1 - self.headroom_target)
return with_headroom
defrequired_instances(self, months_ahead: int) -> int:
"""Number of instances needed."""import math
capacity = self.required_capacity(months_ahead)
return math.ceil(capacity / self.max_rps_per_instance)
defmonths_until_scaling_needed(self) -> Optional[int]:
"""Months until current instance count is insufficient."""
current_max = self.current_instances * self.max_rps_per_instance
for month inrange(1, 37):
ifself.required_capacity(month) > current_max:
return month
returnNone# Sufficient for 3+ yearsdefreport(self, horizon_months: int = 6) -> str:
lines = [f"Capacity Plan: {self.service}", "=" * 40]
lines.append(f"Current peak: {self.current_peak_rps:.0f} RPS")
lines.append(f"Current instances: {self.current_instances}")
lines.append(f"Growth rate: {self.growth_rate_monthly*100:.1f}%/month")
lines.append("")
for m in [1, 3, 6, 12]:
if m <= horizon_months:
needed = self.required_instances(m)
delta = needed - self.current_instances
flag = " ** SCALE NEEDED **"if delta > 0else""
lines.append(f" +{m:2d} months: {needed} instances (delta: {delta:+d}){flag}")
scaling_month = self.months_until_scaling_needed()
if scaling_month:
lines.append(f"\nScaling needed in: {scaling_month} month(s)")
else:
lines.append("\nCapacity sufficient for 3+ years")
return"\n".join(lines)
# Example:
plan = CapacityPlan(
service="payment-api",
current_peak_rps=1200,
growth_rate_monthly=0.08,
headroom_target=0.30,
max_rps_per_instance=500,
current_instances=10,
burst_multiplier=2.0
)
print(plan.report(horizon_months=12))
Reliability Review Process
Reliability reviews are structured evaluations of a service's production readiness and ongoing operational health.
Pre-Launch Review
Review Area
Key Questions
Pass Criteria
SLOs defined
Are SLIs and SLOs documented?
At least availability + latency SLOs
Monitoring
Are dashboards and alerts configured?
SLO-based alerts with multi-window burn rate
Incident response
Is there a runbook?
Documented runbook with escalation paths
Capacity
Can it handle 2x current load?
Load test results proving headroom
Dependencies
Are failure modes mapped?
Dependency map with fallback behavior
Rollback
Can you revert within 5 minutes?
Tested rollback procedure
Data integrity
Are backups tested?
Backup restore tested within last 30 days
Security
Has threat modeling been done?
Threat model documented, critical items resolved
Ongoing Review Cadence
Weekly: Error budget review (automated dashboard)
Monthly: Service health review (SRE + dev team, 30 min)
Quarterly: Full reliability review (cross-functional, 2 hours)
Annually: Architecture review (principal engineers + SRE, half day)
Monthly Service Health Review Template
## Service Health Review: [service-name]
Date: [date]
Attendees: [list]
### SLO Performance- Availability SLO: [target] | Actual: [value] | Budget remaining: [%]
- Latency SLO: [target] | Actual: [value] | Budget remaining: [%]
### Incidents This Period
| Date | Severity | Duration | Budget Impact | Postmortem |
|------|----------|----------|--------------|------------|
### Toil Report- Toil hours this period: [X]
- Top toil sources: [list]
- Automation tickets filed: [count]
### Action Items
| Item | Owner | Due Date | Status |
|------|-------|----------|--------|
### Capacity Outlook- Current utilization: [%]
- Scaling needed by: [date or N/A]
On-Call Best Practices
On-Call Structure
Practice
Recommendation
Rationale
Rotation length
1 week
Long enough for context, short enough to avoid burnout
Team size
Minimum 6-8 engineers
Ensures no one is on-call more than 1 in 6 weeks
Handoff
30-minute overlap meeting
Transfer context on active issues
Escalation
Primary -> Secondary -> Team Lead -> Manager
Clear chain prevents ambiguity
Response time
5 min acknowledge, 15 min start investigation
Documented in on-call agreement
Compensation
Time off in lieu or pay differential
On-call without compensation causes attrition
Alert Quality
Good alert:
- Actionable (something a human must do NOW)
- Tied to an SLO (not a system metric)
- Has a runbook link
- Fires infrequently (< 2 per shift)
Bad alert:
- Informational (log it, don't page)
- Fires often and gets ignored (alert fatigue)
- No runbook (engineer wastes time figuring out what to do)
- Not tied to user impact
Multi-Window Burn Rate Alerting
Alert on error budget burn rate rather than raw error counts. Use multiple windows to balance sensitivity with false positive rate.
Alert Severity
Burn Rate
Long Window
Short Window
Action
Page (urgent)
14.4x
1 hour
5 minutes
Immediate investigation
Page (less urgent)
6x
6 hours
30 minutes
Investigate within 30 min
Ticket
3x
3 days
6 hours
Fix within 1 business day
Log
1x
28 days
3 days
Review at next planning
Incident Management
Severity Levels
Severity
Definition
Response
Example
SEV1
Service down, all users affected
Immediate, all-hands
Total outage of payment processing
SEV2
Significant degradation, many users affected
Immediate, on-call team
Latency 10x normal, 30% errors
SEV3
Partial degradation, some users affected
Within 1 hour
One region experiencing failures
SEV4
Minor issue, few users affected
Next business day
Cosmetic issue in dashboard
Postmortem Structure
Every SEV1 and SEV2 incident gets a blameless postmortem within 72 hours.
## Postmortem: [Incident Title]
Date: [incident date]
Duration: [start time] to [end time] ([total duration])
Severity: [SEV level]
Author: [name]
Reviewers: [names]
### Summary
[1-2 sentence description of what happened and impact]
### Impact- Users affected: [number or percentage]
- Revenue impact: [if applicable]
- Error budget consumed: [percentage]
- SLO status: [still within / breached]
### Timeline
| Time (UTC) | Event |
|-----------|-------|
| HH:MM | First alert fired |
| HH:MM | On-call acknowledged |
| HH:MM | Root cause identified |
| HH:MM | Mitigation applied |
| HH:MM | Full recovery confirmed |
### Root Cause
[Technical description of what caused the incident]
### Contributing Factors- [Factor 1]
- [Factor 2]
### What Went Well- [Thing 1]
- [Thing 2]
### What Could Be Improved- [Thing 1]
- [Thing 2]
### Action Items
| Action | Type | Owner | Bug/Ticket | Due |
|--------|------|-------|-----------|-----|
| [action] | prevent | [name] | [link] | [date] |
| [action] | detect | [name] | [link] | [date] |
| [action] | mitigate | [name] | [link] | [date] |
Anti-Patterns
Anti-Pattern
Problem
Fix
SLOs set by management without engineering input
Unrealistic targets that create constant fire-fighting
SLOs must be data-driven and jointly owned by eng + product
100% availability SLO
Impossible to maintain, blocks all deployments
Highest practical target is 99.999%; most services need 99.9%
