observability-slo

name	observability-slo
description	SLO/SLI — error budgets, alerting, four golden signals.

Four Golden Signals

Signal	Definition	Example Threshold
Latency	Time to serve a request; track p50, p95, p99 separately — errors must not lower the average	p99 < 500ms for checkout API
Traffic	Request rate; baseline for anomaly detection	Requests per second, events per minute
Errors	Rate of failed requests (5xx, timeouts, business logic failures)	< 0.1% error rate
Saturation	How "full" the service is — CPU, memory, queue depth, connection pool	CPU < 80%, queue depth < 1000

Define explicit thresholds for all four signals per service before writing alerts.

---

SLI Definition

An SLI (Service Level Indicator) is a quantitative measure of service behavior:

• Percentage-based — expressed as a ratio of good events to total events
• Directly tied to user experience — not internal metrics like CPU
• Measurable with existing instrumentation

Examples:

• % of /api/checkout requests completing in < 500ms
• % of API requests returning 2xx or 3xx
• % of background jobs completing without error within SLA window

Avoid vanity SLIs (e.g., uptime of internal health-check endpoint that users never call).

---

SLO Targets

Principle	Rule
Realistic, not aspirational	Never set 100% — it is unachievable and creates perverse incentives
Agreed with stakeholders	Product, engineering, and support must align on the target
Documented in the service runbook	Include what the SLO is, how it is measured, and what triggers a response

Common reference points:

SLO	Monthly downtime budget
99.0%	7.3 hours
99.5%	3.6 hours
99.9%	43.8 minutes
99.95%	21.9 minutes

---

Error Budget

• Error budget = 100% - SLO target
• Represents how much unreliability is acceptable per rolling window (typically 30 days)
• When the error budget is burned, freeze new feature work and redirect the team to reliability improvements
• Track error budget burn rate as a primary team health metric

---

Alerting Rules

Alert on SLO burn rate, not on raw metric spikes:

Condition	Action
Burn rate > 2x in the last 1 hour	Page on-call — budget will be exhausted in ~2 days at this rate
Burn rate > 5x in the last 5 minutes	Page on-call immediately — acute incident in progress
Budget < 10% remaining in the window	Notify team lead — feature freeze decision required

Do not alert on:

• CPU > threshold (alert on saturation affecting SLI, not raw CPU)
• Single failed requests (alert on error rate over a window)
• Low traffic periods using absolute thresholds (use relative burn rate)

---

Instrumentation

Three pillars — all three are required for effective observability:

Pillar	Purpose	Tool
Structured logs	Searchable event records with context (user ID, trace ID, error code)	JSON logs → Loki, CloudWatch, Datadog
Distributed traces	Request flow across services; latency breakdown per span	OpenTelemetry → Jaeger, Tempo, X-Ray
Metrics	Aggregated counters and histograms over time	Prometheus, CloudWatch Metrics

Use OpenTelemetry for instrumentation — vendor-neutral, avoids SDK lock-in. Export to any backend.

Minimum required instrumentation per service:

• Incoming request count, latency (histogram), error count
• Outgoing dependency calls (DB, external APIs) — same signals
• Business-level events (order placed, payment processed)

---

Dashboards

One dashboard per service with:

1. Golden signals panel — latency (p50/p95/p99), traffic, error rate, saturation
2. SLO burn rate — current burn rate vs. the 2x and 5x thresholds
3. Error budget remaining — percentage and time left in the rolling window
4. Recent deployments — overlay deploy events on all time-series charts

Link the dashboard URL in the service runbook and on-call rotation doc.

---

On-Call Rotation

• Auto-rotate weekly — no one person should carry on-call indefinitely
• Escalation path must be defined and tested before going on-call
• Post-mortem required for any SEV1 or SEV2 incident (see skills/shared/incident-response/SKILL.md)
• On-call engineer reviews error budget status at the start of each rotation