| name | prometheus-alert-rule-designer |
| description | Create, review, tune, and validate production-grade Prometheus alerting rules and Kubernetes PrometheusRule resources. Use when writing or repairing alert rule YAML, designing PromQL alert expressions, recommending defensible thresholds and for durations, building SLO/error-budget burn-rate alerts, reducing alert noise, standardizing severity/labels/annotations, adding promtool tests, or reviewing an existing Prometheus rule catalog. 用户要求创建、评审或优化 Prometheus 告警规则、PromQL、告警阈值、告警降噪时使用。 |
Prometheus Alert Rule Designer
Create actionable alerts with correct PromQL, explicit assumptions, justified thresholds, stable labels, and a validation plan. Treat generic thresholds as starting points, never as measured facts.
Workflow
- Establish the target:
- Identify the service, environment, owner, Prometheus version, evaluation interval, scrape interval, rule format, and deployment boundary.
- Determine whether the request is create-only, review-only, or includes deployment. Never deploy without an explicit request and confirmation of the target.
- Prefer the repository's existing format: native
groups, Kubernetes PrometheusRule, or ConfigMap-embedded files.
- Verify the metric contract:
- Search
references/rule-catalog/index.md first. Reuse the closest classified rule instead of generating a duplicate.
- Confirm metric type, unit, labels, reset behavior, and exporter version from live metadata or repository evidence.
- Query representative series and 7-30 days of history when access exists. Do not invent metric or label names.
- If live data is unavailable, state every metric and threshold assumption in the result.
- Choose actionable signals:
- Page on user-visible availability, correctness, sustained latency, or exhausted redundancy.
- Use warning/ticket alerts for capacity, trends, and non-urgent degradation.
- Prefer SLO burn rate or time-to-exhaustion over arbitrary utilization alone.
- Select thresholds in this order:
- SLO/error-budget or contractual limit.
- Product hard limit and required intervention lead time.
- Historical baseline plus headroom and a shadow-evaluation period.
- Conservative defaults from
references/threshold-guide.md, clearly labeled as provisional.
- Write the rule:
- Use
rate() for counters, direct comparisons for gauges, and histogram_quantile() with le preserved for classic histograms.
- Aggregate numerator and denominator over identical dimensions; add a traffic floor to ratio alerts.
- Preserve only labels needed for routing and diagnosis. Never put pod UID, request ID, path, or other unbounded labels into alert identity.
- Set
for with the evaluation interval and signal window in mind. The effective detection delay is the query window plus for.
- Use only
info, warning, and critical unless the target organization already defines another taxonomy.
- Make the alert operable:
- Add stable ownership labels such as
team, service, component, and environment when known.
- Add concise
summary, diagnostic description, and real runbook_url/dashboard_url values when available. Do not fabricate URLs.
- Keep routing IDs, webhook URLs, chat IDs, tokens, and encoded expressions out of rule files.
- Pair warning and critical rules only when they trigger distinct actions; define Alertmanager inhibition so critical suppresses warning.
- Validate and deliver:
- Validate structure and policy with the bundled script.
- Validate PromQL with
promtool, add promtool test rules cases for boundary, sustained firing, recovery, absent data, and counter reset, then execute candidate queries against live Prometheus.
- Return the rule YAML, threshold rationale, assumptions, expected alert labels, validation evidence, and rollout/rollback advice.
Discovery Commands
Use environment variables for endpoints and credentials; never print tokens.
curl -fsS "$PROMETHEUS_URL/api/v1/metadata?metric=<metric>" | jq
curl -fsSG "$PROMETHEUS_URL/api/v1/series" \
--data-urlencode 'match[]=<metric>{service="<service>"}' | jq
curl -fsSG "$PROMETHEUS_URL/api/v1/query" \
--data-urlencode 'query=<candidate-promql>' | jq
Inspect the expression over a range, including quiet and peak periods, before recommending a production threshold.
Quality Gates
- Ensure every alert maps to an owner and a concrete human action.
- Ensure alert names are stable PascalCase identifiers and summaries remain readable.
- Ensure ratios have consistent dimensions, units, and a non-zero denominator.
- Ensure low-traffic services cannot page on one isolated error unless that event is intrinsically critical.
- Ensure missing telemetry is covered separately when silence is unsafe.
- Ensure
for is at least two evaluation intervals for transient signals unless the underlying query already proves sustained failure.
- Ensure warning and critical thresholds cannot be inverted or use mixed percent units such as
0.8 versus 80.
- Ensure expressions do not use empty label matchers, environment-specific instance names, or accidental many-to-many joins.
- Ensure expensive expressions become recording rules when repeated or evaluated over large cardinality.
- Ensure rule evaluation cost, Alertmanager grouping, inhibition, repeat interval, and notification volume are reviewed together.
Validation
python3 scripts/validate_alert_rules.py rules.yaml --evaluation-interval 30s
python3 scripts/validate_alert_rules.py rules.yaml --strict --require-promtool
promtool check rules rules.yaml
promtool test rules rules.test.yaml
For a PrometheusRule, also run schema validation against the exact cluster CRD before deployment. Roll out in shadow or warning-only mode, review firing history for at least one representative business cycle, then enable paging.
References
Load only what the request needs:
references/threshold-guide.md: threshold hierarchy, starting values, SLO burn rates, and calibration.references/promql-patterns.md: production PromQL patterns and common failure modes.references/rule-catalog/index.md: classified entry point for 29 existing rule files; search here before creating rules.references/rule-catalog/kubernetes/pod-production.yaml: curated Pod restart, CrashLoopBackOff, OOM, memory, Pending, and NotReady rules.references/alertrules-review.md: lessons and corrected patterns derived from the repository's alertrules.yaml catalog.
