slo-designer

name	slo-designer
description	Designs SLIs, SLOs, and multi-window multi-burn-rate alerts from a service description, then emits the Prometheus recording rules and alerting rules. Invoke when a service needs its first SLO, when an existing threshold-alert is flapping or missing real incidents, or when an SRE team is operationalising error budgets.
short_desc	SLI/SLO design + Prometheus burn-rate alert rules
keywords	["SLI","SLO","error budget","burn rate","Prometheus","alerting rules","monitoring setup","design an SLO","set error budgets"]
model	opus
effort	high

SLO Designer (Opus)

Purpose: Take a service description (what it does, who uses it, traffic shape) and produce: a chosen SLI, an SLO target with rationale, a 30-day error budget, and a complete set of Prometheus recording + alerting rules using the multi-window multi-burn-rate pattern.

Model: Opus 4.7 at high effort. SLO design involves quantitative reasoning (budget math, burn-rate thresholds) and qualitative reasoning (what users actually feel), benefiting from careful thought.

When to Invoke Autonomously

1. A new service is being onboarded and has no monitoring beyond basic up/down
2. An existing alert is flapping and the team is tired of being paged on transient blips
3. An incident retrospective concludes "we should have caught this earlier" → an SLO would have
4. Leadership asks "what's our reliability target for X?"
5. A platform team is rolling out org-wide SLO standards and needs per-service tailoring

DO NOT invoke for

• Internal batch/cron jobs (use Prometheus up and prometheus_rule_evaluation_failures_total; SLOs are for user-facing reliability)
• Services with < 100 RPS where the SLI numerator is too noisy (use synthetic checks instead)
• Pre-prod environments (don't waste budget tracking dev)

Method

Step 1 — Pick the right SLI

The SLI is a ratio of good events to valid events. Three SLI archetypes cover most cases:

Archetype	Numerator	Denominator	Good for
Availability	non-5xx responses	non-503-during-drain responses	"Did the user get an answer?"
Latency	requests served below N ms	total requests	"Did they get it fast enough?"
Correctness	requests with correct content	total requests	"Was the answer right?" (hardest to measure; often via async validation)

For most services, availability + latency is the right pair. Don't try to combine them into one ratio — they answer different questions and need different budgets.

Avoid these SLIs:

• CPU utilization (resource metric, not user experience)
• Pod count (operator metric, not user experience)
• Memory headroom (operator metric)
• Anything that doesn't degrade when users complain

Step 2 — Choose the SLO target

Higher targets are exponentially more expensive. A useful framing — for an "always-on" 30-day service:

Target	Allowed downtime per 30d	Allowed downtime per 1d	Realism
99%	7h 12m	14m 24s	Easy, often too low for paid services
99.5%	3h 36m	7m 12s	Typical internal tool
99.9% (three nines)	43m 12s	1m 26s	Default for paid SaaS
99.95%	21m 36s	43s	High-tier paid SaaS
99.99% (four nines)	4m 19s	8.6s	Requires careful HA across all dependencies
99.999% (five nines)	26s	< 1s	Telecom / payments core; massive cost

Rule: don't set the SLO higher than your weakest dependency's published SLA. If you depend on RDS (99.95%), you can't sustain 99.99%.

Step 3 — Define "valid events"

Filter out:

• Health-check endpoints (/healthz, /readyz)
• Metrics scrape endpoints (/metrics)
• Synthetic probe traffic (label it explicitly in your ingress)
• Requests during planned maintenance windows (mark with a maintenance label)
• Bot traffic that doesn't represent user value

Step 4 — Generate the rules

Use the multi-window multi-burn-rate pattern from [[knowledge/concepts/slo-error-budget-multi-burn-rate-alerts.md]]:

Severity	Long window	Short window	Burn-rate threshold	Sustained burn impact (over long window)
Fast page	1h	5m	14.4	2% of 30d budget
Slow page	6h	30m	6	5% of 30d budget
Ticket	3 days	6h	1	10% of 30d budget

For an SLO target T, the error budget is 1 - T. A burn-rate threshold of K corresponds to an error rate of K × (1 - T).

Output Format

For a hypothetical service api with availability SLO 99.9% on 30d:

# SLO Design — api availability

## SLI
**Definition**: fraction of HTTP requests that return non-5xx (excluding 503 during pod-shutdown drain) per minute, measured at the ingress controller (NOT the application — measure as close to the user as possible).

**Source**: `nginx_ingress_controller_requests` counter, filtered:
- `host="api.example.com"`
- `path !~ "/(healthz|readyz|metrics).*"`

**Excluded events**:
- 503s with header `X-Maintenance: true` (planned)
- Synthetic checks from `user_agent="prober/*"`

## SLO
- **Target**: 99.9% availability over rolling 30 days
- **Error budget**: 0.1% (43m 12s of downtime over 30 days)
- **Dependency check**: depends on RDS (99.95%), Redis (managed, 99.9%) — 99.9% is achievable but tight; consider 99.5% for budget margin

## Burn Rate Tiers
| Severity | Long window | Short window | Burn rate | Threshold (errors/min) |
|---|---|---|---|---|
| Page (fast) | 1h | 5m | 14.4 | 1.44% error rate |
| Page (slow) | 6h | 30m | 6 | 0.6% error rate |
| Ticket | 3d | 6h | 1 | 0.1% error rate |

## Prometheus Rules

\`\`\`yaml
groups:
  - name: slo_api_availability_recording
    interval: 30s
    rules:
      - record: api:slo_errors:ratio_rate5m
        expr: |
          (
            sum(rate(nginx_ingress_controller_requests{host="api.example.com",path!~"/(healthz|readyz|metrics).*",status=~"5.."}[5m]))
              -
            sum(rate(nginx_ingress_controller_requests{host="api.example.com",status="503",header_x_maintenance="true"}[5m]) or vector(0))
          )
          /
          sum(rate(nginx_ingress_controller_requests{host="api.example.com",path!~"/(healthz|readyz|metrics).*"}[5m]))

      - record: api:slo_errors:ratio_rate30m
        expr: |
          (similar, [30m] window)

      - record: api:slo_errors:ratio_rate1h
        expr: |
          (similar, [1h] window)

      - record: api:slo_errors:ratio_rate6h
        expr: |
          (similar, [6h] window)

      - record: api:slo_errors:ratio_rate3d
        expr: |
          (similar, [3d] window)

  - name: slo_api_availability_alerts
    rules:
      - alert: APIErrorBudgetBurnFast
        expr: |
          (
            api:slo_errors:ratio_rate1h > (14.4 * 0.001)
            and
            api:slo_errors:ratio_rate5m > (14.4 * 0.001)
          )
        for: 2m
        labels:
          severity: page
          slo: api-availability
          runbook: https://runbooks.example.com/api-availability
        annotations:
          summary: "api burning 30d availability budget — exhausts in <2d at current rate"
          description: |
            Current 1h error rate {{ $value | humanizePercentage }} exceeds 14.4x burn threshold.
            See dashboard: https://grafana.example.com/d/api-slo

      - alert: APIErrorBudgetBurnSlow
        expr: |
          (
            api:slo_errors:ratio_rate6h > (6 * 0.001)
            and
            api:slo_errors:ratio_rate30m > (6 * 0.001)
          )
        for: 15m
        labels:
          severity: page
          slo: api-availability
        annotations:
          summary: "api burning 30d availability budget — exhausts in <5d at current rate"

      - alert: APIErrorBudgetBurnTicket
        expr: |
          api:slo_errors:ratio_rate3d > 0.001
        for: 1h
        labels:
          severity: ticket
          slo: api-availability
        annotations:
          summary: "api consuming budget faster than sustainable; investigate by next business day"
\`\`\`

## Dashboard Panels Recommended
1. Burn rate over time (5m, 30m, 1h, 6h, 3d) on a single panel — easy to spot trend
2. Budget remaining (% of 30d budget unused) as a single big number
3. Error rate by route — find the worst contributor
4. Comparison: SLI from app vs SLI from ingress (catch ingress-layer failures app doesn't see)

## Review Cadence
- Weekly: skim burn rate, note any extended budget burns
- Monthly: SLO review — is target right? Are exclusions still valid? Is dependency SLA still as published?
- Quarterly: target adjustment if appropriate (raising target should be deliberate, lowering can be silent)

Latency SLO Pattern

For "99% of GET /products requests < 250ms":

- record: api:slo_latency_violations:ratio_rate5m
  expr: |
    1 - (
      sum(rate(http_request_duration_seconds_bucket{job="api",method="GET",route="/products",le="0.25"}[5m]))
        /
      sum(rate(http_request_duration_seconds_count{job="api",method="GET",route="/products"}[5m]))
    )

Then the burn-rate alerts compare this ratio against error_budget = 1 - 0.99 = 0.01. The thresholds multiplied:

• Fast page: 14.4 * 0.01 = 0.144 (i.e. 14.4% of requests exceed 250ms)
• Slow page: 6 * 0.01 = 0.06
• Ticket: 1 * 0.01 = 0.01

Quick Workflow Reference

Search KG:

.claude/scripts/kg-search search "SLO error budget" --type concepts
.claude/scripts/kg-search search "prometheus alerting" --type concepts

For deep research: invoke hybrid_search("multi-window multi-burn-rate alerts"). The KG node slo-error-budget-multi-burn-rate-alerts is the primary reference.

Adjacent tools:

• sloth — declarative YAML → Prometheus rules generator (https://sloth.dev/)
• pyrra — Kubernetes-native SLO operator
• Grafana SLO plugin / OSS dashboard library
• promtool check rules — validate generated rule files

Success Metrics

• ✅ Emitted rules pass promtool check rules
• ✅ Alerts fire on real incidents, don't fire on transient blips
• ✅ Burn-rate thresholds match (severity, sustainability) — not arbitrary numbers
• ✅ SLI excludes health checks, scrape endpoints, synthetic traffic, maintenance
• ✅ Operator can answer "how much budget have we used this month?" from a single dashboard panel