| name | metric-alerts |
| description | Design actionable metric alerts for mobile telemetry that deep-link to traces and sessions, avoid paging noise, and page only for symptoms users feel. Use when creating or cleaning up Datadog / Grafana / Sentry alerts. |
Actionable Metric Alerts
Instructions
An alert that cannot be acted on at 3 AM is noise. Design alerts as symptoms, attach diagnostics, and keep the total paging volume under a human budget.
1. Alert on Symptoms, Not Causes
Page on what the user feels:
- Crash-free sessions dropped.
- Cold start p95 doubled.
- Checkout error rate spiked.
- Sign-in success rate dropped.
Do not page on causes unless the cause is not visible elsewhere:
- High CPU on a single device -- not a page.
- A new error class appearing in Sentry -- not a page, create an issue.
- A backend 5xx rate spike already paged by the backend on-call -- do not double-page.
2. Multi-Window Burn-Rate Alerts
For every SLO, define two alerts:
- Fast burn (outage): 5 min window consuming > 2% of the 30-day budget -> page.
- Slow burn (regression): 1 hour window consuming > 5% of the 30-day budget -> ticket, not page.
Example (crash-free sessions, SLO 99.5%, error budget 0.5%):
- alert: crash_free_sessions_fast_burn
expr: (1 - crash_free_sessions_ratio{release="current"}) > 14.4 * 0.005
for: 5m
severity: page
- alert: crash_free_sessions_slow_burn
expr: (1 - crash_free_sessions_ratio{release="current"}) > 3 * 0.005
for: 1h
severity: ticket
14.4x and 3x come from the Google SRE burn-rate table for 2%/1h and 5%/6h budget consumption.
3. Anti-Noise Rules
- Minimum impact threshold: do not page for issues affecting < 1% of sessions and < 100 sessions/hour.
- Deduplicate by release: if the same alert fired for the previous release and is still firing on the new one, annotate rather than re-page.
- OS / device floor: mute alerts that only fire on OS versions older than
N-3 if your policy does not cover those.
- Auto-resolve: configure auto-resolution when the metric returns below threshold for 2x the evaluation window.
- Maintenance windows: suppress during planned staged rollouts, app review submission, or coordinated backend deploys.
4. Required Alert Fields
Every alert must carry:
summary: one sentence, user-visible symptom.impact: "X% of users on release Y on platform Z".runbook_url: link to the markdown runbook (on-call-mobile).deep_links: one per relevant tool -- Crashlytics issue, Sentry dashboard, Datadog RUM session, trace.owner: team slack channel + PagerDuty service.
annotations:
summary: "Crash-free sessions below SLO on my-app-android@4.12.0"
impact: "1.2% of sessions affected over last 1h"
runbook_url: "https://runbooks/crash-free-sessions"
sentry_url: "https://sentry.io/my-org/my-app-android/?release=4.12.0"
datadog_url: "https://app.datadoghq.com/rum/sessions?query=@release:4.12.0"
5. Deep Links Into Traces
Wire every alert into the trace that caused it:
- Include a panel link with the OTel query
{service.name="my-app", error=true} filtered to the alert window.
- Link to one representative session replay (respecting privacy) so the on-call can see the user journey.
- For crash alerts, link to the Crashlytics/Sentry issue URL pre-scoped to the firing release.
6. Alert Ownership and Review
- Every alert has a single owning team. No "everyone" channels.
- Weekly alert review meeting: delete any alert that was acknowledged but did not lead to action in the last 30 days.
- Quarterly blameless audit: count pages per on-call rotation; if > 2 per shift on average, the rotation is overloaded.
7. Testing Alerts
- Wire a chaos job that forces a synthetic crash / slow start on a canary release, so you can validate the alert fires end-to-end at least once per quarter.
- Use
prometheus/promtool test rules or the vendor's test-mode to unit-test alert expressions.
8. Examples of Good Alerts
- name: checkout_error_rate
expr: sum(rate(checkout_payment_submit_count{outcome="error"}[5m]))
/ sum(rate(checkout_payment_submit_count[5m])) > 0.05
for: 10m
severity: page
annotations:
summary: "Checkout error rate > 5% over 10m"
impact: "Users cannot pay"
runbook_url: "https://runbooks/checkout"
- name: cold_start_p95_regression
expr: histogram_quantile(0.95,
sum(rate(app_start_cold_duration_ms_bucket{release="current"}[15m]))
by (le, device_class))
> on(device_class)
1.5 * histogram_quantile(0.95,
sum(rate(app_start_cold_duration_ms_bucket{release="previous"}[1h]))
by (le, device_class))
for: 30m
severity: ticket
9. Examples of Alerts to Delete
- "Any new Sentry issue" -- create an issue auto-triage rule instead.
- "CPU > 80% on > 10 devices" -- not a user symptom.
- "Network latency > 500 ms" without endpoint grouping -- too broad.
Checklist
