slo-reliability-architect

name

slo-reliability-architect

description

Derive SLOs from user journeys, not infrastructure — inventory the journeys users depend on, select symptom-based SLIs per journey (availability, latency percentiles, correctness, freshness — measured where users experience them, blind spots named), set targets with error budgets in user-meaningful units, design burn-rate alerting where PAGES fire on symptoms/budget burn and causes (CPU, restarts, queue depth) go to tickets, analyze failure modes against the targets, and define the error-budget policy (what happens to release velocity when the budget is spent) plus per-tenant/noisy-neighbor views and a review cadence. Produces the SLO catalog and alert spec that observability-operator implements. Use when asked to define SLOs/SLIs/error budgets, decide what should page, set reliability targets, or rationalize a noisy alert inventory. Do NOT use to implement alerts/dashboards (observability-operator), author incident procedures (incident-response-runbook), or debug current failures (systematic-debugger).

SLO Reliability Architect

Purpose

Produce a reliability design the business can reason about and on-call can live with: user-journey-derived SLOs with error budgets, symptom-based SLIs measured where users experience them, burn-rate alerting that pages only on what threatens the budget, failure-mode analysis against the targets, and the budget policy that converts reliability data into release decisions. The discipline is journeys-before-infrastructure: users do not experience CPU; they experience "checkout failed" — so checkout, not CPU, is what gets a target, a budget, and a page.

Use When

• Use when: asked to define SLOs, SLIs, or error budgets for a product, service, or user journey.
• Use when: deciding what should page vs ticket — or rationalizing a noisy alert inventory around symptoms.
• Use when: setting reliability targets for a new surface, or revisiting targets after incidents or architecture changes.
• Use when: reliability vs release-velocity tension needs a policy (the error budget is that policy).
• Use when: a multi-tenant product needs per-tenant reliability visibility (noisy neighbors, enterprise-tier commitments).
• Do NOT use when: implementing the resulting alerts, dashboards, or instrumentation — observability-operator (manual-only) executes this design.
• Do NOT use when: writing what on-call DOES when paged — incident-response-runbook.
• Do NOT use when: drafting contractual SLAs — this skill informs the commercial decision with achievable numbers; it does not own the contract.
• Do NOT use when: diagnosing an active reliability problem — systematic-debugger.

Inputs to Inspect

1. The user journeys that matter: product flows, qa-strategy-architect risk inventory / test-plan-designer critical journeys if present — reliability targets attach to what users depend on.
2. Real traffic and failure history: request volumes, current availability/latency distributions, incident history — targets set without baseline data are aspirations, labeled as such.
3. The architecture and dependency chains (architecture-designer / provider-architect outputs): what each journey traverses, shared dependencies, single points of failure.
4. Current alert inventory and its noise profile (what pages today, what on-call ignores).
5. Commercial commitments: SLAs already sold, tier promises, enterprise contracts — the floor the SLOs must clear with margin.
6. Tenancy shape: pooled/siloed components, tenant size skew — whether one tenant's experience can diverge from the aggregate.
7. Deploy/release cadence and the team's appetite for a budget policy with teeth.

Workflow

1. Inventory the journeys: enumerate the user journeys worth a reliability promise (login, core workflow, checkout/billing, data export, API integration paths), each with its user population, business impact of failure, and traffic volume. Rank; not everything gets an SLO — an SLO catalog nobody can recite protects nothing.
2. Select symptom-based SLIs per journey: availability (good events / valid events, with "good" and "valid" defined precisely — what counts, what is excluded, where measured), latency (percentile thresholds at the user-experienced edge, not the backend hop), correctness/freshness where the journey's value is data (exports complete, dashboards current). Measurement point stated per SLI: load balancer, client, synthetic — each with its blind spots named.
3. Set targets with error budgets: per SLI, a target justified by baseline data, commercial floor, and cost of the next nine — with the error budget stated in user-meaningful units (minutes of full outage per month, failed requests per million). Targets without baseline data are labeled provisional with a review date.
4. Design burn-rate alerting: pages fire on budget burn rates (fast-burn: page now; slow-burn: page/ticket on sustained burn), causes (CPU, GC, restarts, queue depth) go to tickets and dashboards as diagnostic context. Every page maps to a budget threat; every cause-alert that pages today is explicitly demoted or defended in the spec. Alert spec fields: SLI, burn threshold + window, severity, owner, runbook link placeholder — handed to observability-operator to implement and to incident-response-runbook for the procedures.
5. Analyze failure modes against targets: per journey, walk the dependency chain — shared dependencies whose failure burns multiple budgets, retry/timeout behavior that amplifies, saturation points, and the blast radius of each dependency's failure. Findings that require architectural change route to architecture-designer; this skill sizes the reliability impact.
6. Define the error-budget policy: what happens when a budget is spent — release freeze/slowdown criteria, who decides exceptions, how the budget resets, and what "budget spent on planned work vs incidents" means. A budget with no consequences is a chart, not a policy.
7. Add the tenant dimension: per-tenant SLI views where tenancy skew can hide misery in aggregates (one big tenant at 50% error rate inside a 99.9% aggregate), noisy-neighbor detection signals, and tier-differentiated targets where commercial commitments differ.
8. Set the review cadence: SLO review rhythm (post-incident, quarterly target review, alert-precision review — pages that led to action vs noise), and the staleness triggers (architecture change, traffic-shape change) that reopen the design.

Output Format

RELIABILITY DESIGN — <product/scope>
Journey inventory: <journey — users — business impact — volume — ranked;
  explicitly not-covered journeys listed>
SLI definitions: <journey — SLI — good/valid definitions — measurement
  point + blind spots>
SLO targets & budgets: <SLI — target — budget in user-meaningful units —
  justification (baseline/commercial/cost) — provisional flags + review
  dates>
Alert spec (for observability-operator): <SLI — fast/slow burn thresholds
  + windows — severity — owner — page/ticket — runbook link placeholder>
Demotions: <today's cause-alerts that stop paging, each with rationale>
Failure-mode analysis: <journey — dependency chain — shared/single points
  — amplification risks — architectural findings routed>
Error-budget policy: <spent-budget consequences — decider — reset — 
  planned-work accounting>
Tenant dimension: <per-tenant views needed — noisy-neighbor signals —
  tier-differentiated targets>
Review cadence: <rhythms + staleness triggers>
Assumptions & open questions: <each with risk-if-wrong / who answers>

Validation Checklist

• Every SLO traces to a user journey with named business impact — no SLO exists because a metric was available.
• Every SLI defines "good" and "valid" precisely and names its measurement point with blind spots.
• Every target carries a justification (baseline data, commercial floor, cost) or a provisional label with a review date.
• Error budgets are stated in user-meaningful units.
• Every page in the alert spec maps to budget burn; every currently paging cause-alert is explicitly demoted or defended.
• The budget policy has consequences and a decider.
• The tenant dimension is addressed (or explicitly n/a for single-tenant).
• Implementation was handed to observability-operator, procedures to incident-response-runbook — neither restated here.

Gotchas

• Availability measured at the backend misses the CDN, DNS, and client — the user's 99.9% and the server's 99.99% differ by real outages; name the measurement point's blind spots.
• Percentile latency over a low-traffic window is noise: a p99 on 50 requests pages on one slow request. Windows and minimum-volume guards are part of the SLI definition.
• 100%-minus-epsilon targets are budget-free zones: a 99.99% target on a monthly window is ~4 minutes of budget — no room to deploy. The next nine must be priced, not defaulted.
• Aggregate SLOs hide tenant misery in skewed populations; if one tenant is 40% of traffic, per-tenant views are not optional.
• Retries mask failures into latency: a journey "succeeding" after three retries burns latency budget invisibly and amplifies load at the worst time — count retried-success distinctly.
• Error budgets spent by planned maintenance vs incidents are different signals; a policy that cannot tell them apart freezes releases for the wrong reasons.

Stop Conditions

• No baseline data exists and none can be gathered (new product) → provisional targets only, labeled, with instrumentation-first ordering (observability-operator) and a mandatory review date; refuse to present aspirations as commitments.
• The commercially promised SLA exceeds what the architecture can plausibly deliver → surface the gap to the human as a commercial/ architecture decision (human-approval-boundary for the tradeoff); do not design SLOs that paper over it.
• Journey inventory cannot be established (nobody can say what users depend on) → stop; route to product/qa-strategy-architect risk inventory first. SLOs on infrastructure metrics alone are refused.
• The error-budget policy has no owner willing to enforce consequences → deliver the design with the policy gap named as its top risk.

Supporting Files

• references/slo-derivation.md — SLI menu per journey type, burn-rate window/threshold patterns, budget-policy templates.
• evals/evals.json — trigger + behavior cases.
• evals/trigger-evals.json — discrimination within the reliability cluster (observability-operator, incident-response-runbook) and against shipped qa-strategy-architect / saas-cost-architect.