alerting-rule-design

name	alerting-rule-design
description	Playbook for writing alerts that page on user-visible symptoms using multi-window multi-burn-rate rules — covering threshold selection, alert fatigue reduction, and runbook requirements.

Alerting Rule Design

When to Use This Skill

When a team needs to write or audit alert rules — choosing between symptom-based vs cause-based alerting, setting burn-rate thresholds, and eliminating noise alerts that erode on-call trust.

Core Principle: Alert on Symptoms, Not Causes

Wrong (cause)	Right (symptom)
CPU > 80% for 5 min	Error rate > SLO threshold
Memory pressure	Latency p99 > budget
Pod OOMKilled	Availability SLO burning
Disk I/O wait > threshold	Request success rate < target

A cause alert may fire harmlessly; a symptom alert always means users are experiencing pain.

Multi-Window Multi-Burn-Rate Template

For an SLO with a 30-day error budget, use four alert rules:

Severity	Burn rate	Short window	Long window	Action
P1 (page now)	14x	1 h	5 min	Wake on-call immediately
P2 (page)	6x	6 h	30 min	Page during business hours
P3 (ticket)	3x	1 d	2 h	Create ticket, no page
P4 (info)	1x	3 d	6 h	Dashboard annotation only

Burn rate formula: burn_rate = error_rate / (1 - SLO_target)

A 14x burn rate on a 99.9% SLO means 1.4% errors — budget exhausted in ~2 days.

Prometheus rule example (P1):

- alert: HighErrorBudgetBurnRate
  expr: |
    (
      job:request_errors:rate1h{job="api"} / job:request_count:rate1h{job="api"}
      > 14 * (1 - 0.999)
    )
    and
    (
      job:request_errors:rate5m{job="api"} / job:request_count:rate5m{job="api"}
      > 14 * (1 - 0.999)
    )
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning at 14x rate"
    runbook_url: "https://runbooks.internal/api-error-budget"

Alert Quality Checklist

Every alert rule must answer yes to all five:

• Actionable: Does the on-call engineer know what to do within 5 minutes of receiving this?
• Urgent: Is this worth waking someone at 3 AM? If not, it should not page.
• Symptom-based: Does it measure user-visible impact, not internal cause?
• Has a runbook: The runbook_url annotation links to a document with diagnosis steps and escalation path.
• Has been tested: A test against the alert expression confirmed it fires on a known-bad signal.

Runbook Minimum Requirements

Every pageable alert must have a runbook that includes:

1. What it means — plain-language description of the user impact
2. Diagnosis steps — ordered list of what to check, with links to dashboards
3. Mitigation options — what can be done to stop the bleeding (feature flag, rollback, scaling)
4. Escalation path — who to call if you can't resolve in 30 minutes

Alert Fatigue Audit

Run this audit quarterly on the alerting ruleset:

1. Pull the last 30 days of alert firings.
2. For each alert: count pages, count actionable pages (where on-call took an action), compute the signal ratio.
3. Any alert with signal ratio < 50% is a noise alert — silence, fix the root cause, or delete.
4. Any alert that fired > 20 times without escalating to an incident is likely a false positive.

Pitfalls

• Setting for: 0m on a high-severity alert — spike noise pages. Use at least 2–5 minutes for P1; 15–30 for P2.
• Writing a single alert at a fixed threshold without a short + long window — misses slow burns and misfires on spikes.
• Alerts without runbook_url — on-call stares at a metric with no guidance.
• Alerting on individual pod/instance metrics instead of service-level aggregates — triggers at the wrong scale.

See also

• ../../agents/sre-reliability-engineer.md — SLO/error-budget design and burn-rate alerting
• ../../agents/observability-engineer.md — metrics pipeline and recording rules
• ../../CLAUDE.md — house opinions on symptom-based alerting