| name | skill-compare |
| description | Blind A/B comparison of two skill versions using eval cases — measures relative improvement across correctness, completeness, conciseness, and actionability. |
/skill-compare — Blind A/B testing
Run the same eval prompts against two skill versions and blind-score
which produces better results. Proves a skill change is an improvement,
not just a change.
Usage
/skill-compare <skill-name> # current branch vs main
/skill-compare <skill-name> --file # SKILL.md vs SKILL.md.new
/skill-compare <skill-name> <case-name> # single named case only
If the target has no evals/cases.yaml, stop with: "No eval cases
found for . Run /skill-eval to add cases first."
Workflow
1. Load the two versions
- Default:
git show main:<path-to-SKILL.md> vs working-tree file.
--file: SKILL.md vs SKILL.md.new in the same directory.
Assign random labels — Alpha / Beta or Left / Right, never
"old/new" or "A/B" (those bias scoring). Record the mapping privately;
reveal at the end. If the two versions are identical, stop.
2. Show the diff
Print a unified diff of the two SKILL.md versions before running cases,
so the user sees what changed.
3. For each eval case
a. Generate outputs from both versions. Mentally apply each
version's instructions to the case's scenario + mock_input.
Keep the two outputs separate.
b. Blind-score on four dimensions (1–5 each, scored without knowing
which version produced the output):
| Dimension | What to evaluate |
|---|
| Correctness | Hits expected, avoids anti_expected |
| Completeness | All relevant aspects of the scenario covered |
| Conciseness | Right-sized — neither bloated nor skeletal |
| Actionability | User can act without further clarification |
c. Pick a per-case winner. Total scores within 1 point on all four
dimensions = Tie. Otherwise higher total wins.
d. Also record expected / anti_expected pass/fail as
/skill-eval does — distinguishes "feels better" from "passes
more criteria."
4. Aggregate and report
Summary table with cases won, ties, dimension averages, criteria pass
rate. Per-case breakdown table. Reveal label mapping and declare a
winner (or tie). Append the summary to
<skill-dir>/evals/compare-log.md as a historical record.
5. Interpreting results
| Result | Action |
|---|
| Beta wins ≥70% of cases | Strong improvement — ship it |
| Beta wins 50–70% | Marginal — check for trade-offs (e.g., correctness up but conciseness down) |
| Tie within 1 case | Not worth the diff complexity |
| Alpha wins | Regression — revert or rethink |
Bias guards
- Randomize label assignment every run. Don't always map main → Alpha.
- Score each output independently before comparing — don't look at
one output while scoring the other.
- When in doubt, tie. A genuine tie is more honest than a forced
winner.
- If a comparison is a tie but you believe the change is better, the
eval cases may be too coarse — add cases that target the specific
improvement.
