loom-prometheus

# Prometheus Monitoring and Alerting Pull-based, multi-dimensional time-series monitoring with PromQL. Scope: PromQL, scrape/SD config, recording & alerting rules, instrumentation, Alertmanager. **Dashboards/panels → loom-grafana; log queries → loom-logging-observability.** ## Overview | Component | Role | | --------- | ---- | | Prometheus server | Scrapes targets, stores TSDB locally, evaluates rules | | Alertmanager | Dedup, group, route, inhibit, silence, notify | | Exporters | Translate third-party systems to the exposition format (node, blackbox, ...) | | Client libraries | In-process instrumentation (Go, Python, Rust, Java, ...) | | Pushgateway | Ephemeral batch jobs push metrics — see gotcha; avoid for services | | Operator / Thanos·Mimir·Cortex | K8s CRD deploy / long-term + global-view remote storage | **Metric types.** `up` is the synthetic per-target health series (1 = scrape ok). | Type | Semantics | Query with | | ---- | --------- | ---------- | | Counter | Monotonic, resets to 0 on restart (`_total` suffix) | `rate()`/`increase()` — **never graph raw** | | Gauge | Up/down (memory, queue depth) | raw, `avg_over_time`, `deriv`, `predict_linear` | | Histogram | Bucketed observations → `_bucket{le}`,`_sum`,`_count` | `histogram_quantile()`; buckets are additive | | Summary | Client-side quantiles → `{quantile}`,`_sum`,`_count` | **not aggregatable** — single-process only | ## Scrape Configuration ```yaml # prometheus.yml global: scrape_interval: 15s # rate() windows must be >= ~4x this evaluation_interval: 15s external_labels: { cluster: production, region: us-east-1 } alerting: alertmanagers: - static_configs: [{ targets: [alertmanager:9093] }] rule_files: ["rules/*.yml", "alerts/*.yml"] scrape_configs: - job_name: prometheus static_configs: [{ targets: [localhost:9090] }] - job_name: application metrics_path: /metrics static_configs: - targets: [app-1:8080, app-2:8080] labels: { env: production, team: backend } # Kubernetes SD → see Service Discovery section (single canonical example) ``` ## Alertmanager ```yaml # alertmanager.yml global: resolve_timeout: 5m route: group_by: [alertname, cluster, service] group_wait: 10s # buffer to batch related alerts in first notification group_interval: 10s # wait before adding new alerts to an existing group repeat_interval: 12h # re-send an unresolved alert receiver: default routes: # matchers/source_matchers/target_matchers (0.27+, UTF-8 aware) REPLACE the # deprecated match/match_re/source_match/target_match (removal pending). - matchers: [severity = critical] receiver: pagerduty continue: true # also fall through to default - matchers: [team = database] receiver: dba-team group_by: [alertname, instance] inhibit_rules: - source_matchers: [severity = critical] # a firing critical suppresses... target_matchers: [severity = warning] # ...matching warnings equal: [alertname, instance] receivers: - name: default slack_configs: - channel: "#alerts" text: "{{ range .Alerts }}{{ .Annotations.description }}{{ end }}" - name: pagerduty pagerduty_configs: # routing_key = Events API v2 (current, richer). service_key = legacy v1; # mutually exclusive. *_file variants read the secret from a mounted file. - routing_key_file: /etc/alertmanager/secrets/pagerduty-routing-key - name: dba-team email_configs: [{ to: dba-team@example.com }] ``` **Time-based routing** — attach `active_time_intervals` (or `mute_time_intervals`) to a route referencing a named `time_intervals` block: ```yaml route: routes: - matchers: [severity = warning] receiver: email active_time_intervals: [business-hours] time_intervals: - name: business-hours time_intervals: - times: [{ start_time: "09:00", end_time: "17:00" }] weekdays: ["monday:friday"] ``` ## Metric Naming & Cardinality Format `<namespace>_<subsystem>_<name>_<unit>`. Base units only: **seconds** (not ms), **bytes** (not KB), **ratio 0.0–1.0** (not 0–100). Counters end `_total`. | Pattern | Good | Bad | | ------- | ---- | --- | | Counter suffix | `http_requests_total` | `http_requests` | | Base unit | `..._duration_seconds` | `..._duration_ms` | | Ratio range | `cache_hit_ratio` (0–1) | `cache_hit_percentage` (0–100) | | Namespace prefix | `myapp_http_requests_total` | `http_requests_total` | | snake_case labels | `{method="GET"}` | `{httpMethod="GET"}` | **Cardinality is the #1 Prometheus killer** — series count = product of label-value counts. Never label with unbounded values. | Cardinality | Examples | Verdict | | ----------- | -------- | ------- | | Low (<10) | method, status class, env | safe anywhere | | Medium (10–100) | endpoint (templated), service, pod | safe with aggregation | | High (100–1k) | container id, hostname | only if necessary | | Unbounded | user id, IP, timestamp, raw URL/path with ids | **never** | Template path labels (`/api/users/:id`, not `/api/users/12345`). Pre-aggregate with recording rules; drop noisy metrics via `metric_relabel_configs`/`write_relabel_configs`. ## Recording Rules Pre-compute expensive/reused queries. **Prefer `without (instance)` over `by (job)`** — `without` names only the label removed and preserves `job` + any future labels; `by` silently drops labels added later (the docs mandate `without`). Keep `le` where `histogram_quantile()` consumes the rule. Naming: `level:metric:operations`. ```yaml # rules/recording_rules.yml groups: - name: performance_rules interval: 30s rules: - record: instance_removed:http_requests:rate5m expr: sum without (instance) (rate(http_requests_total[5m])) # Error ratio: aggregate numerator + denominator SEPARATELY, then divide. # NEVER avg()/sum() a ratio rule downstream — re-aggregate the counts. - record: job:http_request_error_ratio:rate5m expr: | sum without (instance) (rate(http_requests_total{status=~"5.."}[5m])) / sum without (instance) (rate(http_requests_total[5m])) - record: job:http_request_duration_seconds:p95 # le kept for histogram_quantile expr: histogram_quantile(0.95, sum without (instance) (rate(http_request_duration_seconds_bucket[5m]))) - name: aggregation_rules interval: 1m rules: - record: instance:node_cpu_utilization:ratio expr: 1 - avg without (cpu, mode) (rate(node_cpu_seconds_total{mode="idle"}[5m])) - record: cluster:node_cpu_utilization:ratio expr: avg without (instance) (instance:node_cpu_utilization:ratio) ``` ## Alerting Rules **Page on user-facing symptoms, not component causes** (Golden Signals: latency, traffic, errors, saturation). Page at the outermost user-visible boundary — one layer's symptom is another's cause. Vet every page: urgent, user-visible, actionable, non-automatable, not already paged? If not → warning/ticket. ```yaml # alerts/symptom_based.yml groups: - name: symptom_alerts rules: - alert: HighErrorRate expr: | sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05 for: 5m # condition must hold before firing (anti-flap) keep_firing_for: 10m # stay firing after it clears (anti-flap on exit; 2.42+) labels: { severity: critical, team: backend } annotations: summary: "High error rate" description: "Error rate {{ $value | humanizePercentage }} (>5%)" runbook: https://wiki.example.com/runbooks/high-error-rate - alert: HighLatency expr: | histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) > 1 for: 5m labels: { severity: warning, team: backend } annotations: { summary: "P95 latency high on {{ $labels.service }}" } # Detect a missing series (see absent() gotcha for per-instance / flaky cases) - alert: ServiceDown expr: up{job="critical-service"} == 0 for: 2m labels: { severity: critical } ``` ### Multi-window multi-burn-rate SLO (Google SRE Workbook) **No `for:` clause** — duration doesn't scale with severity and resets on data gaps. Each tier ANDs a long detection window with a short confirmation window (~1/12 of the long one, proving the budget is burning *now*); tiers are ORed. For a 99.9% SLO (budget 0.001): ```yaml - alert: SLOBudgetBurnFast # page: ~2% monthly budget in 1h expr: | sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > 14.4 * 0.001 and sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 14.4 * 0.001 labels: { severity: critical, team: sre } - alert: SLOBudgetBurnSlow # ticket: gradual burn expr: | sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) > 6 * 0.001 and sum(rate(http_requests_total{status=~"5.."}[30m])) / sum(rate(http_requests_total[30m])) > 6 * 0.001 labels: { severity: warning, team: sre } # Third tier: 1x over 3d AND 6h (slow ticket). ``` **Alert hygiene:** meaningful `summary`/`description`/`runbook`/`impact` annotations; `team`/`service`/`env` labels for routing; every alert must be actionable. Validate with `promtool check rules`. ## PromQL Value types: instant vector, range vector, scalar, string. ```promql # Selectors / matchers http_requests_total{method="GET", status=~"5.."} # =, !=, =~, !~ http_requests_total{status!=""} # label present http_requests_total[5m] # range vector (window must be >= ~4x scrape) # Rate — ALWAYS rate() the raw counter BEFORE aggregating sum(rate(http_requests_total[5m])) by (service) increase(http_requests_total[1h]) # rate*window; extrapolated float, NOT exact count irate(http_requests_total[5m]) # last 2 samples; dashboards ONLY, never alerts # Error / success ratio (aggregate num & denom separately, then divide) sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) # Histogram percentiles (keep le in by()) histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) # Average latency (NOT a percentile): _sum / _count sum(rate(http_request_duration_seconds_sum[5m])) by (service) / sum(rate(http_request_duration_seconds_count[5m])) by (service) # Aggregation ops: sum avg min max count stddev stdvar quantile group + by()/without() count(up == 1) by (job) # Selection / prediction topk(5, sum(rate(http_requests_total[5m])) by (service)) predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4*3600) < 0 # disk-full in 4h absent(up{job="critical-service"}) # see gotcha # Offsets / time rate(http_requests_total[5m]) / rate(http_requests_total[5m] offset 1h) time() - process_start_time_seconds # uptime (s) ``` **Vector matching.** One-to-one by default (identical label sets). Many-to-one needs `on(labels) group_left(extra)` (right side is "one"); `group_right` mirrors it. Logical: `and`, `or`, `unless`. ```promql sum(rate(http_requests_total[5m])) by (instance, method) / on(instance) group_left