prometheus-alert-rule-designer

name

prometheus-alert-rule-designer

description

Create, review, tune, and validate production-grade Prometheus alerting rules and Kubernetes PrometheusRule resources. Use when writing or repairing alert rule YAML, designing PromQL alert expressions, recommending defensible thresholds and for durations, building SLO/error-budget burn-rate alerts, reducing alert noise, standardizing severity/labels/annotations, adding promtool tests, or reviewing an existing Prometheus rule catalog. 用户要求创建、评审或优化 Prometheus 告警规则、PromQL、告警阈值、告警降噪时使用。

Prometheus Alert Rule Designer

Create actionable alerts with correct PromQL, explicit assumptions, justified thresholds, stable labels, and a validation plan. Treat generic thresholds as starting points, never as measured facts.

Workflow

1. Establish the target:
   • Identify the service, environment, owner, Prometheus version, evaluation interval, scrape interval, rule format, and deployment boundary.
   • Determine whether the request is create-only, review-only, or includes deployment. Never deploy without an explicit request and confirmation of the target.
   • Prefer the repository's existing format: native groups, Kubernetes PrometheusRule, or ConfigMap-embedded files.
2. Verify the metric contract:
   • Search references/rule-catalog/index.md first. Reuse the closest classified rule instead of generating a duplicate.
   • Confirm metric type, unit, labels, reset behavior, and exporter version from live metadata or repository evidence.
   • Query representative series and 7-30 days of history when access exists. Do not invent metric or label names.
   • If live data is unavailable, state every metric and threshold assumption in the result.
3. Choose actionable signals:
   • Page on user-visible availability, correctness, sustained latency, or exhausted redundancy.
   • Use warning/ticket alerts for capacity, trends, and non-urgent degradation.
   • Prefer SLO burn rate or time-to-exhaustion over arbitrary utilization alone.
4. Select thresholds in this order:
   • SLO/error-budget or contractual limit.
   • Product hard limit and required intervention lead time.
   • Historical baseline plus headroom and a shadow-evaluation period.
   • Conservative defaults from references/threshold-guide.md, clearly labeled as provisional.
5. Write the rule:
   • Use rate() for counters, direct comparisons for gauges, and histogram_quantile() with le preserved for classic histograms.
   • Aggregate numerator and denominator over identical dimensions; add a traffic floor to ratio alerts.
   • Preserve only labels needed for routing and diagnosis. Never put pod UID, request ID, path, or other unbounded labels into alert identity.
   • Set for with the evaluation interval and signal window in mind. The effective detection delay is the query window plus for.
   • Use only info, warning, and critical unless the target organization already defines another taxonomy.
6. Make the alert operable:
   • Add stable ownership labels such as team, service, component, and environment when known.
   • Add concise summary, diagnostic description, and real runbook_url/dashboard_url values when available. Do not fabricate URLs.
   • Keep routing IDs, webhook URLs, chat IDs, tokens, and encoded expressions out of rule files.
   • Pair warning and critical rules only when they trigger distinct actions; define Alertmanager inhibition so critical suppresses warning.
7. Validate and deliver:
   • Validate structure and policy with the bundled script.
   • Validate PromQL with promtool, add promtool test rules cases for boundary, sustained firing, recovery, absent data, and counter reset, then execute candidate queries against live Prometheus.
   • Return the rule YAML, threshold rationale, assumptions, expected alert labels, validation evidence, and rollout/rollback advice.

Discovery Commands

Use environment variables for endpoints and credentials; never print tokens.

curl -fsS "$PROMETHEUS_URL/api/v1/metadata?metric=<metric>" | jq
curl -fsSG "$PROMETHEUS_URL/api/v1/series" \
  --data-urlencode 'match[]=<metric>{service="<service>"}' | jq
curl -fsSG "$PROMETHEUS_URL/api/v1/query" \
  --data-urlencode 'query=<candidate-promql>' | jq

Inspect the expression over a range, including quiet and peak periods, before recommending a production threshold.

Quality Gates

• Ensure every alert maps to an owner and a concrete human action.
• Ensure alert names are stable PascalCase identifiers and summaries remain readable.
• Ensure ratios have consistent dimensions, units, and a non-zero denominator.
• Ensure low-traffic services cannot page on one isolated error unless that event is intrinsically critical.
• Ensure missing telemetry is covered separately when silence is unsafe.
• Ensure for is at least two evaluation intervals for transient signals unless the underlying query already proves sustained failure.
• Ensure warning and critical thresholds cannot be inverted or use mixed percent units such as 0.8 versus 80.
• Ensure expressions do not use empty label matchers, environment-specific instance names, or accidental many-to-many joins.
• Ensure expensive expressions become recording rules when repeated or evaluated over large cardinality.
• Ensure rule evaluation cost, Alertmanager grouping, inhibition, repeat interval, and notification volume are reviewed together.

Validation

python3 scripts/validate_alert_rules.py rules.yaml --evaluation-interval 30s
python3 scripts/validate_alert_rules.py rules.yaml --strict --require-promtool
# Native rule files only; the bundled script extracts CRD/ConfigMap rule groups.
promtool check rules rules.yaml
promtool test rules rules.test.yaml

For a PrometheusRule, also run schema validation against the exact cluster CRD before deployment. Roll out in shadow or warning-only mode, review firing history for at least one representative business cycle, then enable paging.

References

Load only what the request needs:

• references/threshold-guide.md: threshold hierarchy, starting values, SLO burn rates, and calibration.
• references/promql-patterns.md: production PromQL patterns and common failure modes.
• references/rule-catalog/index.md: classified entry point for 29 existing rule files; search here before creating rules.
• references/rule-catalog/kubernetes/pod-production.yaml: curated Pod restart, CrashLoopBackOff, OOM, memory, Pending, and NotReady rules.
• references/alertrules-review.md: lessons and corrected patterns derived from the repository's alertrules.yaml catalog.