evidence-bundle-init

# evidence-bundle-init ## Purpose Set up a new evidence bundle directory under `experiments/artifacts/<family>/<run-id>/` with the skeleton reproducibility-grade evidence requires: - `SOURCE.md` - operator identity, cluster, git SHA, UTC timestamp, the original prompt / intent. - `summary.md` - verdict skeleton operator fills in as the experiment progresses. - `commands/` - directory for the four-file `<NN>-<step>.{cmd,stdout,stderr,exit}` tuples one per shell command run during the experiment. - `.gitkeep` markers for the local directory shape. This skill is the operator-facing convenience over `mkdir + cat > SOURCE.md`. Half a minute of friction is enough that people skip the discipline. This skill makes it 5 seconds. ## Experiment isolation & traceability (required for any cluster-touching experiment) The bundle's run-id IS the **experiment-id** - the single join key across the evidence bundle, the cluster objects, and the perf-lake. When the experiment creates cluster resources: - Every Deployment/Pod/PVC/PV/Secret/ConfigMap/Service MUST use an experiment-unique name derived from the id (e.g. `glm51-expt-deepep-ll`, PV `glm51-deepep-expt-pv`) and carry the label `experiment=<id-slug>`. - NEVER reuse a standing/platform/migration name (e.g. a shared `*-inference*` deployment, a standing `*-cache*` PV, or anything labeled `migration=*`). Cluster-scoped PV names are global. A collision silently breaks another owner's PVC. - Tear down by label: `kubectl delete deploy,pod,pvc,secret -l experiment=<id-slug>`. For `Retain` PVs pre-clear the attacher finalizer before delete. - Record the created object names + the perf-lake `campaign=<id>` in `SOURCE.md` (template below has the block). ## Observations vs mechanisms + roofline companions (measurement bundles) A measurement bundle that produces a roofline / SoL analysis MUST separate **observations** (what the instruments reported) from **mechanisms** (the causal "why"), and MUST carry a source-code provenance block. This is the discipline a perf reviewer asks for: "separate the mechanisms from the observations". Scaffold these in any measurement bundle: - `findings/01-observations.md` - **measured tables ONLY** (DCGM SM/tensor/DRAM %, tok/s, TPOT, AA numbers). No interpretation, no "because". Each number is reproducible from `commands/`. - `findings/02-mechanisms.md` - one item per claim, formatted `OBSERVATION -> MECHANISM (causal) -> CONFIDENCE (+ what would raise it)`. A mechanism claim ("decode plateaus at 41% HBM because the sparse-MoE+MLA kernel mix has low DRAM efficiency") needs a profile (DCGM/zymtrace/nsys/ncu) - the rooflines are the observation. The mechanism is the separately-evidenced interpretation. - `findings/00-ANSWERS-*.md` (optional) - the live-sync handout that answers the reviewer's questions directly, each pointing at 01/02. - A ` ```provenance ` block (`experiment_provenance_v1`) in `SOURCE.md` pinning the exact vLLM/SGLang commit + delivery + patch, so the rendered roofline carries a source link (see [`server/tools/perf_tune_report/ROOFLINE-METHODOLOGY.md`](../../server/tools/perf_tune_report/ROOFLINE-METHODOLOGY.md)). Record the REAL `delivery` (`image|overlay|patchedVllm|infr-patch`) the bundle ran -- it is the code-under-test identity: a number from this bundle may be cited only as evidence for THAT delivery, never cross-tier (an `overlay`/offline-prepped run is not evidence for an `infr-patch`, even if the kernels match). The prefill/decode roofline itself (page 7) is captured + always-published via the `inference-perf-tune-report` / `inference-perf-bench` pipeline. This bundle just holds the obs/mechanisms narrative + provenance that the report links to. > This skill is backed by a native MCP verb: `mcp__profile_and_optimize__evidence_init`. The verb does the entire scaffold atomically (mkdir + SOURCE.md + summary.md + commands/README.md + .gitkeep) and returns the bundle path. The Bash-tool path documented below remains supported as a fallback. ## Why a bundle, not a flat directory This repo's reproducibility-grade-evidence convention requires that significant experiments produce a bundle, not loose files: - The bundle's path is the durable handle that future skills (`search_evidence`, `perf-baseline-record`, etc.) reach for. - `SOURCE.md` is the audit trail that says "this evidence was captured by X on cluster Y on date Z from prompt W". - `summary.md` is the human-readable verdict at the bottom of the funnel. - `commands/` is the four-file tuple capture that makes every shell action replayable. ## When to use - Starting any experiment that will produce >1 artifact file. - Reviewer asked "where's the evidence for that claim?" and the answer is a bundle. - Periodic capture (weekly perf-of-record snapshot, monthly drain audit, etc.). - Pairs with every skill that writes artifacts ([`prometheus-anchored-query`](../prometheus-anchored-query/SKILL.md), [`perf-baseline-record`](../perf-baseline-record/SKILL.md)). Do **not** use this skill for: - One-off shell commands whose output you'll throw away - no bundle needed. - Adding to an existing bundle - just `cd` into the bundle and add files. - Benchmark families whose runbooks define their own bundle layout and naming conventions - follow those. This skill is the generic scaffolder. ## Example prompts - "Init a new evidence bundle for the nccl sweep I'm about to run." - "Scaffold a bundle under cluster-health family." - "New evidence bundle for the gpu-burn soak." - "Set up a bundle for the b200 8b regression investigation." - `/evidence-bundle-init --family cluster-health --run-id rack-a-validation` - `/evidence-bundle-init --family perf-baselines --measurement nccl_busbw` ## Prerequisites 1. **`PROFILE_AND_OPTIMIZE_REPO_ROOT`** for the bundle path. The skill writes to `${PROFILE_AND_OPTIMIZE_REPO_ROOT}/experiments/artifacts/<family>/<run-id>/`. 2. **Family** - `--family <name>` (e.g. `cluster-health`, `nccl-tests`, `gpu-burn`, `campaign/llama31_8b`). 3. **Run-id** - `--run-id <slug>` (default: `<UTC-timestamp>` if not supplied). 4. **Operator intent** - `--intent "<one-line description>"` (gets written into `SOURCE.md`). ## Interaction style Fast and autonomous (3-5 seconds). Single optional pause: confirm the family + run-id + intent before write. ## Workflow ### Phase 0: resolve bundle path ```text bundle = ${PROFILE_AND_OPTIMIZE_REPO_ROOT}/experiments/artifacts/<family>/<run-id>/ ``` If the bundle already exists, **stop**. Bundles are immutable. New captures use a new run-id. ### Phase 1: gather provenance In parallel: - `Bash(date -u +%Y-%m-%dT%H:%M:%SZ)` - UTC timestamp. - `Bash(hostname)` - workstation hostname. - `Bash(whoami)` - operator user. - `Bash(git -C ${PROFILE_AND_OPTIMIZE_REPO_ROOT} rev-parse HEAD)` - current SHA of the bundled server tree. - `Bash(git -C ${PROFILE_AND_OPTIMIZE_REPO_ROOT} remote get-url origin)` - repo remote URL for the Provenance section. ### Phase 2: write the skeleton **Preferred (MCP verb):** ```text mcp__profile_and_optimize__evidence_init with: args: ["--family", "<family>", "--intent", "<operator one-line intent>", "--run-id", "<slug>", "--json"] ``` The verb does everything atomically and returns the bundle directory path. Skip to Phase 3. **Fallback (Bash-tool):** ```text mkdir -p ${bundle}/commands Write ${bundle}/SOURCE.md with: # SOURCE **Family:** `<family>` **Run-id:** `<run-id>` **Created at (UTC):** `<ts>` **Created by:** `<USER>` on `<hostname>` **PROFILE_AND_OPTIMIZE SHA (bundled server):** `<git-sha>` ## Intent <operator's --intent text> ## Provenance - Workstation kernel: `<uname -a>` - Repo: `<git -C ${PROFILE_AND_OPTIMIZE_REPO_ROOT} remote get-url origin>` (this plugin marketplace). - Bundle path: `experiments/artifacts/<family>/<run-id>/` ## Experiment isolation & traceability The run-id IS the experiment-id: the single join key across this bundle, the cluster objects, and the perf-lake. (Matches the `mcp__profile_and_optimize__evidence_init` scaffold. Keep these as structured `- key: value` lines so `publish_to_lake` / `experiments_index` can read them.) - experiment_id: <run-id> - family: <e.g. nvfp4-kv | warp-decode | deepep | (blank)> - object label (EVERY cluster object, on metadata AND pod template): `experiment=<run-id>` - cluster resources created (fill in as you create them. Every Deployment/Pod/Job/PVC/PV/Secret/ConfigMap/Service, experiment-unique-named, NEVER a standing/migration name): - - perf-lake campaign: `campaign=<run-id>` (run `perftunereport campaign_init --experiment-id <run-id> --family <family> --evidence-bundle <this-bundle>` so campaign_id == experiment_id. The `s3://perf-lake/...` atlas_v1 + campaign_v1 paths are auto-appended here by `publish_to_lake`). - pre-apply label gate: verify every manifest carries `experiment=<run-id>` before `kubectl apply`. ## Cross-references - `docs/METHODOLOGY.md` - the measurement-methodology canon. Write ${bundle}/summary.md with: # Summary **Status:** in-progress ## Verdict <to-be-filled-in by operator at end of experiment> ## Findings - ## Recommendations - ## Open questions - Write ${bundle}/commands/README.md with: # commands/ Every shell command run during this experiment is captured as a four-file tuple: 00-<step-slug>.cmd # the exact command 00-<step-slug>.stdout # captured stdout 00-<step-slug>.stderr # captured stderr 00-<step-slug>.exit # exit code Filenames are zero-padded sequential (00, 01, 02, ...) so natural sort order matches execution order. From the server root, use the checked-in helper: capture_rc=0 ART_DIR="${bundle}" \ bash tools/shared/capture_cmd.sh ls-image -- ls /mnt/data/images/ \ || capture_rc=$? if [ "$capture_rc" -ne 0 ]; then exit "$capture_rc" fi `tools/shared/capture_cmd.sh` shell-quotes exact argv, writes stdout and stderr separately, writes the exit code before returning, and is tested by `tools/shared/test_capture_cmd.sh`. The `||` branch keeps the tuple complete when the caller uses `set -e` and the wrapped command fails. Touch ${bundle}/commands/.gitkeep to preserve the local directory shape. ``` ### Phase 3: report Print the bundle path and the next-step pointer: - "Add captures from the server root with `tools/shared/capture_cmd.sh`." - "Finalize: edit `summary.md` with verdict + findings before sharing." - "Register a baseline if applicable: `perf-baseline-record --source <bundle>`." ## Output bundle layout ``` ${PROFILE_AND_OPTIMIZE_REPO_ROOT}/experiments/artifacts/<family>/<run-id>/ SOURCE.md summary.md commands/ README.md .gitkeep ``` ## Safety - **Never overwrite an existing bundle.** Bundles are immutable. The skill refuses to init over a populated directory. - **Audit trail.** `SOURCE.md` records the operator's `${USER}` + hostname so future readers know who captured the evidence, where, and from what prompt. - **Local-only by default.** The public repository ignores the full `server/experiments/artifacts/` tree except `.gitkeep` markers. An ordinary `git add` does not stage `SOURCE.md`, command output, or measurements. - **Intentional publication requires review.** Scrub secrets, private paths, operator details, and sensitive workload data. Review the complete bundle, then use `git add -f <bundle>` only when the operator explicitly chose to publish it. - **The provenance hook checks force-staged bundles.** When the gate is enabled, it reads the staged `SOURCE.md` from the Git index. It does not make local evidence public and it does not stage artifacts itself. ## Source-of-truth references - [`docs/METHODOLOGY.md`](../../../../docs/METHODOLOGY.md) - the measurement-methodology canon every bundle feeds. - [`server/AGENTS.md`](../../server/AGENTS.md) - bundled-server discovery contract. - All sibling skills that write artifacts - they all assume the bundle this skill creates.