prometheus-anchored-query

# prometheus-anchored-query ## Purpose Make ad-hoc PromQL queries **safe by default** by wrapping the "Contention And O11y Pattern" discipline (anchor labels in the knowledge base FIRST, then run the query) as a reusable skill. Saves the raw payload to a provenance-bearing bundle so future queries can replay the same shape, and so the perf-lake-contract is honored without operator effort. This skill exists because PromQL is easy to write **wrong** at cluster scale: a missing label selector turns a 100-series query into a 10,000-series query that overloads the Prometheus backend. The bundled server's [`mcp-composition.md`](../../server/docs/mcp-composition.md) Contention And O11y Pattern (step 3) makes the knowledge-base-first call mandatory. This skill enforces it. Use it for ad-hoc questions ("what's the IB traffic on `<node>` right now?" / "is the dataloader being starved?" / "what's the median dcgm_xid rate across a partition?"). ## When to use - Operator has a question that maps to a PromQL query but doesn't know the exact labels yet. - Operator wants to save a Prometheus payload to disk with provenance (for evidence bundles, post-incident write-ups, share-with-team artifacts). - Operator wants to confirm a metric exists + has the expected cardinality before writing a dashboard panel for it. Do **not** use this skill for: - Already-vetted queries - call `mcp__prometheus_mcp__query_prometheus` directly (or run them from whatever dashboarding UI they live in). The anchor pass adds nothing when the query shape is already known. - Log queries - this skill emits PromQL only. The MCP server also fronts a log datasource via `query_loki_logs`, but anchored log queries are out of scope here. ## Example prompts - "What's the IB traffic on these 8 nodes for the last hour?" - "Is dataloader starvation showing up in dcgm_gpu_utilization for the cohort I just ran?" - "Save the dcgm_xid trajectory across the gb300 partition for the last 24h." - "What's the median GPU temp on this rack right now?" - "Anchored PromQL for nccl_perf_bandwidth across the cohort." - `/prometheus-anchored-query --question "IB tx bytes by node for <partition> last 1h" --time-range "now-1h..now"` - `/prometheus-anchored-query --metric dcgm_gpu_temp --labels '{partition="gb300"}' --time-range "now-24h..now"` ### Inference-perf example The [`inference-perf-bench`](../inference-perf-bench/SKILL.md) workflow scrapes four canonical vLLM Prometheus metrics pre/post each benchmark run: | Metric | What it answers | | --- | --- | | `vllm:prefix_cache_hit_rate` | KV cache hit rate during the run (delta pre vs post) | | `vllm:gpu_cache_usage_perc` | GPU cache utilization peak | | `vllm:num_requests_running` | Concurrent requests peak | | `vllm:avg_generation_throughput_toks_per_s` | Generation throughput | Operators replaying a perf-bench run via this skill should anchor against these four metric names first (knowledge-base call confirms cardinality + datasource UID), then run the narrow PromQL with `{model_name="<served-name>", namespace="<ns>"}` selectors. See [`inference-perf-bench`](../inference-perf-bench/SKILL.md) Phase 4 / 7 for the in-pod scrape pattern this skill complements at the Prometheus MCP layer. ## Prerequisites 1. **Prometheus MCP reachable** (`PROMETHEUS_MCP_URL` set). 2. **Operator question or metric+labels** - one of: - Free-form question (skill derives metric + labels via knowledge base). - Explicit `--metric <name> --labels <json>`. 3. **Time range** - `--time-range "now-Nh..now"` or absolute UTC range. Default `now-1h..now`. 4. **`PROFILE_AND_OPTIMIZE_REPO_ROOT`** for the payload bundle path (falls back to `${PWD}/prometheus-bundles/<run-id>/`). ## Interaction style Iterative. The whole point of the skill is the "anchor first, then query" discipline. Pause after the knowledge-base call to surface the proposed PromQL before running it. ## Workflow ### Phase 0: resolve the question If operator provided a free-form question, restate the question + the proposed metric / labels in one sentence. Get confirmation. If operator provided explicit metric + labels, proceed. ### Phase 1: knowledge-base anchor ```text mcp__prometheus_mcp__query_observability_knowledge_base with: query: "<one-line question or metric+context>" user_prompt: "<operator's original prompt>" ``` The knowledge-base response tells us: - Whether the metric exists in the Prometheus MCP's catalog. - The available labels for that metric. - The expected cardinality (so we don't accidentally select 100k series). - The canonical datasource UID for this metric on this cluster (the MCP can front several datasources - e.g. per-cluster Prometheus instances plus a log backend - so every query tool takes an explicit `datasourceUid`). - Any noted caveats (sampling rate, retention, legacy renames). If the knowledge base does NOT cover the metric: stop. Report what's missing. Per [`mcp-composition.md`](../../server/docs/mcp-composition.md) "Contention And O11y Pattern" step 3, running uncovered PromQL is unsafe. ### Phase 2: validate labels exist For each label the proposed query selects on: ```text mcp__prometheus_mcp__list_prometheus_label_values with: datasourceUid: "<from knowledge-base anchor in Phase 1>" labelName: "<label>" matches: ["<metric>{<other-selectors>}"] user_prompt: "<operator's original prompt>" ``` If a label or value doesn't exist, the query would be silently empty. Stop and surface. ### Phase 3: emit the PromQL Construct the narrow PromQL using only knowledge-base-confirmed labels and the operator's `--time-range`. Print it back: ```text Proposed PromQL: <full query> Datasource UID: <from knowledge base> Time range: <start> .. <end> Expected series count: ~<from knowledge base cardinality estimate> ``` Ask: "Run this query?" Do not auto-advance. ### Phase 4: run + save After confirmation: ```text mcp__prometheus_mcp__query_prometheus with: datasourceUid: "<from-anchor>" expr: "<promql>" queryType: "range" startTime: "<start>" endTime: "<end>" stepSeconds: <resolution-seconds> user_prompt: "<operator's original prompt>" ``` (Use `queryType: "instant"` + only `endTime` for a single-point query. Drop `startTime` / `stepSeconds` in that case.) Save the raw payload to: ``` ${PROFILE_AND_OPTIMIZE_REPO_ROOT}/experiments/artifacts/prometheus-bundles/<run-id>/ SOURCE.md # operator + question + cluster + UTC-ts query.promql # the exact PromQL run query.json # the exact request JSON response.json # the raw Prometheus response (saved verbatim) summary.md # human-readable summary: series count, range, top-5 anomalies knowledge-base-anchor.json # the knowledge-base response (provenance for the label discipline) ``` ### Phase 5: report Print: - **Series count** + **time-bucket count** (sanity check: did we get the cardinality the knowledge base predicted?). - **Top 5 anomalies** (highest / lowest values, recent step-deltas). - **Bundle path** for the saved payload. ## Full-context reporting (no bare numbers) Per `docs/METHODOLOGY.md` "Full-context reporting": every number this skill emits (throughput, latency, TPOT/ITL, BW, %SoL, speedup, efficiency, goodput, acceptance rate, scaling efficiency, thermal/failure rate - whatever it reports) MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare number is a defect - it cannot set a default, ship a config, or appear in a report. - **Identity:** model (+HF path), hardware (exact ceiling token `GB300`/`B200`), quant, kv-cache dtype. - **Parallelism:** TP, DP (replicas), PP, EP, parallel_strategy. - **Serving cfg:** max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching. - **Workload:** dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts. - **Regime:** warm vs cold. Latency vs throughput tier. - **Stack:** image/vllm commit, bench backend, serving engine. - **Grounding:** `%SoL` (+ ceiling key from `configs/sol-ceilings.yaml` - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named. (If the metric is not roofline-bound - e.g. accuracy/acceptance - omit `%SoL` but keep the rest of the descriptor.) - **Per-number exact shape (no smoothing):** when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. `c=1 @ ISL1024/OSL256` + `c=64 @ ISL4096/OSL512`, NOT one shared "random"). ## Next lever / BREAKTHROUGH (Grind Mandate) If this skill emits a measured result, its output MUST end by naming the **next perf lever**, its **expected unlock** (direction + rough magnitude), and the **gate** that proves/refutes it, per `docs/METHODOLOGY.md` "Always be grinding". A measured win is the new floor, not the finish -- so **do everything we can to find the next BREAKTHROUGH**: the highest-EV unlock toward Speed-of-Light (a new champion / kernel / router / quant / parallelism / spec-decode win, or an unblocked stack), not just the next micro-lever. Rank the candidate breakthrough levers by value x cost (the GRIND FRONTIER, `perftunereport value_view`), pursue the top, bank the rest with evidence. Record WHY a refuted lever loses, update the standing frontier in the active bundle's `HANDOFF.md`. Never conclude "exhausted/optimal/done" without an explicit next-lever frontier (an empty frontier AND a documented SoL wall only). Delete this section ONLY if the skill produces no measurements. ## Safety - **Knowledge base first is mandatory.** Phase 1 cannot be skipped - fail fast, no silent fallbacks. The whole point of the skill is to enforce that discipline. - **No high-cardinality queries.** If the knowledge base reports the proposed query would return >10k series, the skill refuses to run it. Operator narrows the selector and retries. - **Raw payload preservation.** Per [`perf-lake-contract.md`](../../server/docs/perf-lake-contract.md), every saved query records the datasource UID, the exact PromQL, the time range, the response time, and the source label. - **Read-only.** The skill never mutates state through the Prometheus MCP. Never creates dashboards / annotations / alerts (those would need `create_*` tools which are not in `allowed-tools`). ## Known limitations - **The agent's MCP token may lack `datasources:read`.** If `mcp__prometheus_mcp__query_prometheus` returns `403 datasources:read on <uid>` for every Prometheus datasource UID, Phases 2-4 of this skill are blocked end-to-end. **Workaround:** the operator runs the exact PromQL produced in Phase 3 from an authenticated shell or the Prometheus web UI and pastes the response JSON into `${PROFILE_AND_OPTIMIZE_REPO_ROOT}/experiments/artifacts/prometheus-bundles/<run-id>/response.json`. The skill's Phase 5 summary still works against the operator-pasted payload. ## Source-of-truth references - [`server/docs/mcp-composition.md`](../../server/docs/mcp-composition.md) - "Contention And O11y Pattern" (the workflow this skill generalizes). - [`server/docs/perf-lake-contract.md`](../../server/docs/perf-lake-contract.md) - raw-payload provenance contract. - [`docs/METHODOLOGY.md`](../../../../docs/METHODOLOGY.md) - measurement canon (full-context reporting, verdict rigor).