| name | flow-code-monitoring |
| description | Use when setting up dashboards, SLOs/SLIs, alerting rules, or on-call runbooks. Covers the monitoring hierarchy from health checks to incident response. |
| tier | 2 |
| user-invocable | true |
Monitoring & Alerting
Overview
Monitoring answers "is the system healthy?" Alerting answers "does someone need to act?" Good monitoring detects problems before users report them. Good alerting wakes people only when necessary.
When to Use
- Setting up monitoring for a new service
- Defining SLOs (Service Level Objectives)
- Creating dashboards
- Configuring alerting rules
- Writing on-call runbooks
SLO/SLI/SLA Framework
| Term | Definition | Example |
|---|
| SLI (Indicator) | Metric that measures service health | 99.2% of requests < 300ms |
| SLO (Objective) | Target for the SLI | 99.5% of requests < 300ms over 30 days |
| SLA (Agreement) | Business contract with consequences | 99.9% uptime or customer gets credits |
| Error Budget | How much failure is allowed | 0.5% = 3.6 hours downtime/month |
Choosing SLIs
Request-driven service (API, web):
- Availability: % of requests returning non-5xx
- Latency: % of requests faster than threshold (p99 < 500ms)
- Correctness: % of requests returning expected result
Pipeline/batch service:
- Freshness: % of data updated within SLO window
- Throughput: records processed per hour
- Completeness: % of expected records processed
Error Budget Policy
Budget remaining > 50%: Ship freely, experiment
Budget remaining 25-50%: Caution — prefer stability over features
Budget remaining < 25%: Freeze deploys, focus on reliability
Budget exhausted (0%): All engineering shifts to reliability
Dashboard Design
The Four Golden Signals (per service)
1. Latency → p50, p95, p99 response time
2. Traffic → Requests per second
3. Errors → Error rate (5xx / total)
4. Saturation → CPU, memory, disk, queue depth
Dashboard Layout
Row 1: SLO status (green/yellow/red) + error budget remaining
Row 2: Golden signals (latency, traffic, errors, saturation)
Row 3: Business metrics (orders/min, signups, revenue)
Row 4: Dependencies (DB latency, cache hit rate, external API status)
Rules:
- Start from user impact (SLO), drill down to infra
- Show rate of change, not just current value (is it getting worse?)
- Use consistent time windows across panels (last 1h, 24h, 7d)
- Every dashboard has an owner (team that maintains it)
Alerting Rules
Alert on Symptoms, Not Causes
Good (symptom): "Error rate > 5% for 5 minutes"
Bad (cause): "CPU > 80%" (might be normal for batch jobs)
Good (symptom): "p99 latency > 2s for 10 minutes"
Bad (cause): "Database connections > 90" (might recover on its own)
Severity Levels
| Severity | Criteria | Response | Example |
|---|
| P1 Critical | SLO violated, users impacted NOW | Page on-call immediately | API returning 500 for all users |
| P2 High | SLO at risk, degraded experience | Notify team, respond within 1h | Latency 3x normal, error rate rising |
| P3 Medium | Anomaly, no user impact yet | Ticket, address within 1 business day | Error budget burning faster than expected |
| P4 Low | Informational, trend alert | Review in next sprint | Disk usage growing 5%/week |
Alert Hygiene
- Every alert has a runbook link (what to do when it fires)
- Review and tune alerts monthly (delete noisy alerts)
- Target < 5 pages per on-call shift per week
- If an alert fires and no action is needed → fix the alert, not the person
Runbook Template
# Alert: API Error Rate > 5%
## Impact
User-facing API returning errors. Orders cannot be placed.
## Quick Check
1. Check dashboard: <link>
2. Check recent deploys: `git log --since="2 hours ago" --oneline`
3. Check dependency status: <status page links>
## Common Causes & Fixes
| Cause | Check | Fix |
|-------|-------|-----|
| Bad deploy | Error started after deploy? | Rollback: `deploy rollback` |
| DB overload | DB dashboard shows high latency? | Scale read replicas |
| Dependency down | External API returning errors? | Enable circuit breaker fallback |
## Escalation
If not resolved in 30 min: page @backend-lead
Common Rationalizations
| Rationalization | Reality |
|---|
| "We'll add monitoring after launch" | Launch without monitoring = flying blind. Add basic golden signals before first deploy. |
| "Alert on everything, we'll tune later" | Alert fatigue kills on-call morale. Start with SLO-based alerts only. |
| "CPU/memory alerts are enough" | Infrastructure metrics don't tell you if users are happy. Alert on symptoms. |
| "We don't need SLOs yet" | Without SLOs, every incident is a fire drill. Define "healthy" before it breaks. |
Red Flags
- No monitoring on a production service
- Alerts without runbooks
- Alerting on infrastructure metrics instead of user-facing symptoms
- More than 10 pages per on-call shift per week (alert fatigue)
- Dashboards showing only "up/down" (no latency, error rate, saturation)
- SLOs defined but never reviewed or enforced
- No error budget tracking
Verification
