| name | regen-eval-baseline |
| description | Captures a promptfoo skill-eval baseline JSON for one sumo-qa skill and snapshots it to docs/qa/runs/eval-baselines/, with an automatic delta against the prior snapshot. Use this whenever the user mentions baselining a skill, capturing a before/after eval, running a single-skill eval, or measuring the effect of a SKILL.md edit — common during token-optimisation rounds. The actual work runs through a bundled script that handles path conventions, API-key checks, and diffing in one go. |
| disable-model-invocation | true |
regen-eval-baseline
Captures a promptfoo run for one sumo-qa skill and stores its JSON output in docs/qa/runs/eval-baselines/ (gitignored). The deterministic work lives in scripts/run_baseline.py; this document is the guide for picking inputs and reading the output.
When to use
Trigger this skill when the user wants a per-skill eval snapshot. Common phrasings: "baseline this skill", "snapshot the eval", "run the eval for skill X", "capture before/after for the rewrite I just made". The user invokes it explicitly with /regen-eval-baseline; it doesn't auto-trigger.
This is single-skill on purpose. Full-sweep regeneration belongs on npm run eval:all, which also reads the same tests/evals/promptfoo/skill-*.yaml files but runs them sequentially without snapshotting.
Inputs
Pass exactly one config selector — either --skill (the base config) or --config (a suffixed scenario / .ab.yaml control / explicit path) — plus an optional --label.
--skill <name> — the base config tests/evals/promptfoo/skill-<name>.yaml. Resolves that file exactly; it never cross-matches a longer suffixed sibling (e.g. --skill reviewing-before-merge drives skill-reviewing-before-merge.yaml, NOT skill-reviewing-before-merge-adversarial.yaml). If the named config is absent the script lists the available configs.--config <selector> — for a suffixed scenario config or an .ab.yaml A/B control. Accepts a bare stem (skill-reviewing-before-merge-adversarial), a filename including the double suffix (skill-reviewing-before-merge-adversarial.ab.yaml), or a full/relative path. Resolved exactly — a stem composes one filename and that file must exist; the resolver never falls back to a near-named neighbour, so a base name can't accidentally snapshot a suffixed sibling and vice-versa.--label — short kebab-case tag for the snapshot, defaults to baseline. Past conventions in the dir: baseline, postcut, greenfix, or describe-the-change like removability-gate. The label vocabulary is open-ended and may itself be multi-hyphen, so the snapshot filename separates the slug from the label with a literal __ (double underscore) — <date>-skill-<slug>__<label>.json. Because a validated kebab slug/label can never contain __, the slug/label boundary stays unambiguous even for a multi-hyphen label. The label is what makes two snapshots taken on the same day distinguishable.
The snapshot filename derives from the resolved config, not just --skill: a .ab infix becomes a hyphen in the slug (skill-x-adversarial.ab.yaml → …-skill-x-adversarial-ab__<label>.json), so the base config, its suffixed scenario, and its .ab control each snapshot to a distinct, non-colliding path and diff only against their own prior snapshots.
Wrapper vs raw promptfoo eval -c
Use this wrapper (--skill / --config) whenever you want the repeatable before/after snapshot it exists to provide: it writes the dated JSON to docs/qa/runs/eval-baselines/, prints pass/fail, and diffs against the prior snapshot for the same config. That covers base configs, suffixed scenario configs, and .ab.yaml controls — drive all three through the wrapper so the baseline/postcut delta is preserved (don't hand-roll npx promptfoo eval -c … and lose the snapshot). Reach for raw promptfoo eval -c <path> only for flags the wrapper does not expose (e.g. --repeat N variance runs, -j 1 legible logs, generate dataset) — see tests/evals/promptfoo/README.md. For a normal baseline/postcut capture, the wrapper is the supported path.
Prerequisites the script will check
OPENAI_API_KEY must be set in the environment. The harness reads it from ~/.config/promptfoo-keys.env per tests/evals/promptfoo/README.md; tell the user to source that file if the script reports it missing. Never accept the key pasted in chat — both repo policy and the feedback_never_handle_pasted_secrets memory rule say so.- The selected config must exist:
tests/evals/promptfoo/skill-<name>.yaml for --skill, or the exact suffixed / .ab.yaml config for --config. If it doesn't, the script lists every available config so the user can pick again — it does not guess a near-named sibling.
- A snapshot at the target path already existing will block the run unless
--force is passed. Don't pass --force reflexively — snapshots are evidence of past runs and silently clobbering them loses history.
Run the script
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
--skill <skill-name> \
--label <label>
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
--config skill-reviewing-before-merge-adversarial \
--label baseline
python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py \
--config skill-reviewing-before-merge-adversarial.ab.yaml \
--label postcut
The script:
- Computes the snapshot path:
docs/qa/runs/eval-baselines/<today>-skill-<slug>__<label>.json (the slug comes from the resolved config — see Inputs; __ separates slug from label so a multi-hyphen label stays unambiguous).
- Runs
npx promptfoo eval with --no-cache (so the snapshot reflects fresh judge calls, not stale cache hits) and writes the JSON output to that path.
- Prints pass/fail counts.
- If a prior snapshot for the same config exists, prints a delta — passed and failed counts vs the previous run.
Reading the output
The pass/fail summary tells you the state of the snapshot. The delta tells you whether the most recent SKILL.md edit moved the needle. Three patterns to watch for:
- Passes increased, failures decreased — the edit helped. Keep it.
- Passes decreased, failures increased — the edit hurt. Investigate before reverting; the failure may be informative.
- No change — either the edit was outside the assertions' coverage, or the judge gave the same verdict on different reasoning. Read the JSON's per-test reasons before drawing conclusions.
When FAILs appear
Don't propose loosening the rubric. The standing repo policy (feedback_eval_fixes_target_skill_not_rubric) is to strengthen the SKILL.md so the candidate naturally satisfies the existing rubric. Hand the failing snapshot to the eval-failure-diagnoser subagent — it reads the JSON, locates the failing assertion, and recommends a concrete SKILL.md edit. Re-run this skill after the edit to confirm the failure flipped.
Why the snapshots are gitignored
docs/qa/ is excluded from the repo (.gitignore line 48). Eval snapshots are local evidence — useful for the contributor doing the optimisation work, but not artefact that ships with the package. Per feedback_no_process_artifacts_in_public_repo, results live alongside the QA work, not in the public source tree. Don't propose committing snapshots or removing the gitignore entry.
