obs-verify

name	obs-verify
description	Audit the observability system itself — every catalog metric present at /metrics with expected labels, bounded label cardinality, Prometheus targets all up, budgets board agreeing with direct PromQL. Use after adding or renaming a metric, when Prometheus/Grafana or the obs-admin board looks wrong, or before flipping an obs gate.

obs-verify — audit the observability system itself

Five checks, in order.

0. Isolation — the app must never NEED the obs stack

pnpm obs:acceptance (needs the API on :3001, the scrubber on :8100 via pnpm scrubber, and Prometheus UP first — the run's last check is "every target is up", and since M6 the scrubber is one): it stops the obs compose project, asserts Prometheus is gone and the API still serves /metrics, runs ui-spec against the obs-less stack, then restores the stack and waits for every target to come back up. This is 06-verification.md §Obs's "automated" clause; it exits 1 on any failure.

1. Stack + targets

• pnpm obs:status — bring up with pnpm obs:up if down.
• After EDITING obs/prometheus/prometheus.yml, restart the stack (pnpm obs:down && pnpm obs:up). It is a single-file bind mount, so an editor that rewrites the file leaves the container on the old inode and POST /-/reload re-reads the stale copy — a new job silently never appears.
• Every Prometheus target healthy:

  curl -s localhost:9090/api/v1/targets | node -e "
    const t = JSON.parse(require('fs').readFileSync(0)).data.activeTargets;
    for (const a of t) console.log(a.labels.job, a.health);
    process.exit(t.length > 0 && t.every((a) => a.health === 'up') ? 0 : 1)"
  

2. Catalog metrics present at /metrics

• Run the enumerating tests (they walk the catalog — the source of truth — never a hand list): pnpm --filter api test src/common/metrics covers catalog, dashboard coverage, and bounded-label units.
• Live e2e against a real app (needs Postgres + Redis up): pnpm --filter api test test/metrics.e2e.test.ts
• The catalog has two owners since M6. scrubber_redactions_total lives on the SCRUBBER's endpoint, and it carries a zero series per detector, so curl -s localhost:8100/metrics | grep scrubber_redactions_total must print one line per type in redact.py's PATTERNS, not zero lines. kafka_consumer_lag / dlt_depth are on the API on purpose — the alarm has to survive the consumer it watches.
• Spot-check the wire: curl -s localhost:3001/metrics and grep any metric you just added or renamed.

3. Cardinality

Route label values are templated paths, never raw URLs. The unit test asserts the bound (< 30 values); also eyeball the live set for UUIDs or query strings that would explode cardinality:

curl -s localhost:3001/metrics | grep -o 'route="[^"]*"' | sort -u

4. Budgets board agrees with direct PromQL (05-obs-admin §5)

• pnpm --filter obs-admin test pins board rendering + query wiring (every budget id must have a query in src/prom/budget-queries.ts — a missing one is a compile error, not a runtime surprise).
• Live diff: for each id in perf/budgets.json, run its PromQL from budget-queries.ts directly — curl -sG localhost:9090/api/v1/query --data-urlencode "query=<expr>" — substituting EVERY {{range}} with the window (e.g. 1h; a half-substituted query is a Prometheus parse error rendered as "no data"). Compute pass/fail against the budget value, then compare with the board at http://localhost:5174: the header "N of M passing" (plus "· N unmeasured" when a promql row has no data) must match your count. web.* rows are "n/a — needs bench run", not failures. Don't regex-parse the queries out of the source — import the module or read the page; regex extraction raised two false alarms in M5.

Report

One line per check, PASS/FAIL, naming the offending metric, label value, or budget id — never just a count.