| name | slo-designer |
| description | Designs SLIs, SLOs, and multi-window multi-burn-rate alerts from a service description, then emits the Prometheus recording rules and alerting rules. Invoke when a service needs its first SLO, when an existing threshold-alert is flapping or missing real incidents, or when an SRE team is operationalising error budgets. |
| short_desc | SLI/SLO design + Prometheus burn-rate alert rules |
| keywords | ["SLI","SLO","error budget","burn rate","Prometheus","alerting rules","monitoring setup","design an SLO","set error budgets"] |
| model | opus |
| effort | high |
SLO Designer (Opus)
Purpose: Take a service description (what it does, who uses it, traffic shape) and produce: a chosen SLI, an SLO target with rationale, a 30-day error budget, and a complete set of Prometheus recording + alerting rules using the multi-window multi-burn-rate pattern.
Model: Opus 4.7 at high effort. SLO design involves quantitative reasoning (budget math, burn-rate thresholds) and qualitative reasoning (what users actually feel), benefiting from careful thought.
When to Invoke Autonomously
- A new service is being onboarded and has no monitoring beyond basic up/down
- An existing alert is flapping and the team is tired of being paged on transient blips
- An incident retrospective concludes "we should have caught this earlier" → an SLO would have
- Leadership asks "what's our reliability target for X?"
- A platform team is rolling out org-wide SLO standards and needs per-service tailoring
DO NOT invoke for
- Internal batch/cron jobs (use Prometheus
up and prometheus_rule_evaluation_failures_total; SLOs are for user-facing reliability)
- Services with < 100 RPS where the SLI numerator is too noisy (use synthetic checks instead)
- Pre-prod environments (don't waste budget tracking dev)
Method
Step 1 — Pick the right SLI
The SLI is a ratio of good events to valid events. Three SLI archetypes cover most cases:
| Archetype | Numerator | Denominator | Good for |
|---|
| Availability | non-5xx responses | non-503-during-drain responses | "Did the user get an answer?" |
| Latency | requests served below N ms | total requests | "Did they get it fast enough?" |
| Correctness | requests with correct content | total requests | "Was the answer right?" (hardest to measure; often via async validation) |
For most services, availability + latency is the right pair. Don't try to combine them into one ratio — they answer different questions and need different budgets.
Avoid these SLIs:
- CPU utilization (resource metric, not user experience)
- Pod count (operator metric, not user experience)
- Memory headroom (operator metric)
- Anything that doesn't degrade when users complain
Step 2 — Choose the SLO target
Higher targets are exponentially more expensive. A useful framing — for an "always-on" 30-day service:
| Target | Allowed downtime per 30d | Allowed downtime per 1d | Realism |
|---|
| 99% | 7h 12m | 14m 24s | Easy, often too low for paid services |
| 99.5% | 3h 36m | 7m 12s | Typical internal tool |
| 99.9% (three nines) | 43m 12s | 1m 26s | Default for paid SaaS |
| 99.95% | 21m 36s | 43s | High-tier paid SaaS |
| 99.99% (four nines) | 4m 19s | 8.6s | Requires careful HA across all dependencies |
| 99.999% (five nines) | 26s | < 1s | Telecom / payments core; massive cost |
Rule: don't set the SLO higher than your weakest dependency's published SLA. If you depend on RDS (99.95%), you can't sustain 99.99%.
Step 3 — Define "valid events"
Filter out:
- Health-check endpoints (
/healthz, /readyz)
- Metrics scrape endpoints (
/metrics)
- Synthetic probe traffic (label it explicitly in your ingress)
- Requests during planned maintenance windows (mark with a maintenance label)
- Bot traffic that doesn't represent user value
Step 4 — Generate the rules
Use the multi-window multi-burn-rate pattern from [[knowledge/concepts/slo-error-budget-multi-burn-rate-alerts.md]]:
| Severity | Long window | Short window | Burn-rate threshold | Sustained burn impact (over long window) |
|---|
| Fast page | 1h | 5m | 14.4 | 2% of 30d budget |
| Slow page | 6h | 30m | 6 | 5% of 30d budget |
| Ticket | 3 days | 6h | 1 | 10% of 30d budget |
For an SLO target T, the error budget is 1 - T. A burn-rate threshold of K corresponds to an error rate of K × (1 - T).
Output Format
For a hypothetical service api with availability SLO 99.9% on 30d:
# SLO Design — api availability
## SLI
**Definition**: fraction of HTTP requests that return non-5xx (excluding 503 during pod-shutdown drain) per minute, measured at the ingress controller (NOT the application — measure as close to the user as possible).
**Source**: `nginx_ingress_controller_requests` counter, filtered:
- `host="api.example.com"`
- `path !~ "/(healthz|readyz|metrics).*"`
**Excluded events**:
- 503s with header `X-Maintenance: true` (planned)
- Synthetic checks from `user_agent="prober/*"`
## SLO
- **Target**: 99.9% availability over rolling 30 days
- **Error budget**: 0.1% (43m 12s of downtime over 30 days)
- **Dependency check**: depends on RDS (99.95%), Redis (managed, 99.9%) — 99.9% is achievable but tight; consider 99.5% for budget margin
## Burn Rate Tiers
| Severity | Long window | Short window | Burn rate | Threshold (errors/min) |
|---|---|---|---|---|
| Page (fast) | 1h | 5m | 14.4 | 1.44% error rate |
| Page (slow) | 6h | 30m | 6 | 0.6% error rate |
| Ticket | 3d | 6h | 1 | 0.1% error rate |
## Prometheus Rules
\`\`\`yaml
groups:
- name: slo_api_availability_recording
interval: 30s
rules:
- record: api:slo_errors:ratio_rate5m
expr: |
(
sum(rate(nginx_ingress_controller_requests{host="api.example.com",path!~"/(healthz|readyz|metrics).*",status=~"5.."}[5m]))
-
sum(rate(nginx_ingress_controller_requests{host="api.example.com",status="503",header_x_maintenance="true"}[5m]) or vector(0))
)
/
sum(rate(nginx_ingress_controller_requests{host="api.example.com",path!~"/(healthz|readyz|metrics).*"}[5m]))
- record: api:slo_errors:ratio_rate30m
expr: |
(similar, [30m] window)
- record: api:slo_errors:ratio_rate1h
expr: |
(similar, [1h] window)
- record: api:slo_errors:ratio_rate6h
expr: |
(similar, [6h] window)
- record: api:slo_errors:ratio_rate3d
expr: |
(similar, [3d] window)
- name: slo_api_availability_alerts
rules:
- alert: APIErrorBudgetBurnFast
expr: |
(
api:slo_errors:ratio_rate1h > (14.4 * 0.001)
and
api:slo_errors:ratio_rate5m > (14.4 * 0.001)
)
for: 2m
labels:
severity: page
slo: api-availability
runbook: https://runbooks.example.com/api-availability
annotations:
summary: "api burning 30d availability budget — exhausts in <2d at current rate"
description: |
Current 1h error rate {{ $value | humanizePercentage }} exceeds 14.4x burn threshold.
See dashboard: https://grafana.example.com/d/api-slo
- alert: APIErrorBudgetBurnSlow
expr: |
(
api:slo_errors:ratio_rate6h > (6 * 0.001)
and
api:slo_errors:ratio_rate30m > (6 * 0.001)
)
for: 15m
labels:
severity: page
slo: api-availability
annotations:
summary: "api burning 30d availability budget — exhausts in <5d at current rate"
- alert: APIErrorBudgetBurnTicket
expr: |
api:slo_errors:ratio_rate3d > 0.001
for: 1h
labels:
severity: ticket
slo: api-availability
annotations:
summary: "api consuming budget faster than sustainable; investigate by next business day"
\`\`\`
## Dashboard Panels Recommended
1. Burn rate over time (5m, 30m, 1h, 6h, 3d) on a single panel — easy to spot trend
2. Budget remaining (% of 30d budget unused) as a single big number
3. Error rate by route — find the worst contributor
4. Comparison: SLI from app vs SLI from ingress (catch ingress-layer failures app doesn't see)
## Review Cadence
- Weekly: skim burn rate, note any extended budget burns
- Monthly: SLO review — is target right? Are exclusions still valid? Is dependency SLA still as published?
- Quarterly: target adjustment if appropriate (raising target should be deliberate, lowering can be silent)
Latency SLO Pattern
For "99% of GET /products requests < 250ms":
- record: api:slo_latency_violations:ratio_rate5m
expr: |
1 - (
sum(rate(http_request_duration_seconds_bucket{job="api",method="GET",route="/products",le="0.25"}[5m]))
/
sum(rate(http_request_duration_seconds_count{job="api",method="GET",route="/products"}[5m]))
)
Then the burn-rate alerts compare this ratio against error_budget = 1 - 0.99 = 0.01. The thresholds multiplied:
- Fast page:
14.4 * 0.01 = 0.144 (i.e. 14.4% of requests exceed 250ms)
- Slow page:
6 * 0.01 = 0.06
- Ticket:
1 * 0.01 = 0.01
Quick Workflow Reference
Search KG:
.claude/scripts/kg-search search "SLO error budget" --type concepts
.claude/scripts/kg-search search "prometheus alerting" --type concepts
For deep research: invoke hybrid_search("multi-window multi-burn-rate alerts"). The KG node slo-error-budget-multi-burn-rate-alerts is the primary reference.
Adjacent tools:
sloth — declarative YAML → Prometheus rules generator (https://sloth.dev/)pyrra — Kubernetes-native SLO operator- Grafana SLO plugin / OSS dashboard library
promtool check rules — validate generated rule files
Success Metrics
- ✅ Emitted rules pass
promtool check rules
- ✅ Alerts fire on real incidents, don't fire on transient blips
- ✅ Burn-rate thresholds match (severity, sustainability) — not arbitrary numbers
- ✅ SLI excludes health checks, scrape endpoints, synthetic traffic, maintenance
- ✅ Operator can answer "how much budget have we used this month?" from a single dashboard panel
