| name | auto-eval |
| description | Run the AI automation evaluation framework against test fixtures. Verifies agent quality without hitting ADO or LLM APIs. Use after changing prompts, rules, or agent steps. Accepts --all, --agent, --tier2 flags. |
| when_to_use | Use when running the automation evaluation framework against test fixtures, after changing agent prompts or rules, when the user says 'run evals', 'test agent quality', 'verify agent output', or '--all --agent --tier2'. |
| argument-hint | [--all | --agent dor|pr-review|pr-answer|discover | --tier2 | --fixture [name]] |
You run the evaluation framework for the AI automation agents. Eval runs against pre-captured fixtures (no live ADO or LLM calls for tier-1 gates).
0. Prerequisites
Read .ai/automation/infra.json to confirm scaffold exists.
Check Node.js is available: node --version. If missing: STOP with install instructions.
Check eval fixtures exist:
ls .ai/automation/eval/fixtures/ 2>/dev/null | head -5
If no fixtures: "No eval fixtures found. Run /auto-test --capture first to capture fixtures from live ADO, or add fixtures manually to .ai/automation/eval/fixtures/."
1. Parse Arguments
Default: --all if no argument given.
Supported flags (pass through to eval/run.js):
--all — run all fixtures--agent <name> — run only fixtures for a specific agent (dor, pr-review, pr-answer, discover)--tier2 — run tier-2 gates (makes real LLM calls; slower and costs tokens)--fixture <name> — run a single fixture by name
--agent discover — KAI-HUB repo discovery
Scores the dx-discover-repos cascade by alias-set match: a fixture passes
when the discovery output's set of alias values equals the fixture's
expected_aliases (order-independent). Fixtures live in
.ai/automation/eval/fixtures/discover/ (seed copies ship at
dx-automation/skills/auto-eval/fixtures/discover/), one per tier:
| Fixture | Tier | Deterministic? |
|---|
01-explicit | 0 — repos: directive | yes (offline) |
02-simple-block | 1 — ```simple block routing | yes (offline) |
03-crossrepo-table | 2 — ## Cross-Repo Scope table | yes (offline) |
04-llm-inference | 3 — LLM infers from title/description | no — needs --tier2; documents expected set, model-dependent |
Tiers 0–2 run offline in tier-1 (no ADO/LLM) and must pass exactly — they share
logic with dx-hub/skills/dx-discover-repos/tests/run-tests.sh. Tier 3 only runs
under --tier2; treat a miss as a prompt-tuning signal, not a hard gate.
2. Run Eval
cd .ai/automation
node eval/run.js $ARGUMENTS
Report the output as-is.
3. Interpret Results
After eval completes, summarize:
- Pass rate: N/total fixtures passed
- Failed fixtures: list names of failed fixtures
- Gate breakdown: which gates failed (if any)
If any fixtures fail:
⚠️ N fixture(s) failed. Check the output above for details. Do NOT push prompt changes until all fixtures pass.
If all pass:
✓ All fixtures passed. Safe to push changes.
4. Common Eval Workflows
After changing a prompt file:
/auto-eval --all
Before pushing a code change:
/auto-eval --all
Debug a specific agent:
/auto-eval --agent pr-review
Full quality regression (slower, uses LLM):
/auto-eval --tier2
Examples
-
/auto-eval — Runs tier-1 evaluation against all agent test fixtures. Parses agent step outputs, validates against expected results (no ADO or LLM calls). Reports 45/47 assertions passed, 2 failures in the PR Review agent's comment formatting logic.
-
/auto-eval --agent pr-review — Runs tier-1 fixtures for the PR Review agent only. Useful for debugging a specific agent after changing its prompt or step logic. Reports 12/12 assertions passed.
-
/auto-eval --tier2 — Warns about token cost, then runs full quality regression including LLM calls. Tests that agent outputs meet quality thresholds for each scenario. Reports quality scores per agent and overall pass/fail.
Troubleshooting
-
"Fixtures not found for agent"
Cause: No test fixtures exist in the eval directory for the specified agent.
Fix: Create fixtures in .ai/automation/eval/fixtures/<agent-name>/ with input/expected-output pairs. See existing fixtures for the format.
-
Tier-1 passes but tier-2 fails
Cause: Tier-1 tests structural correctness (format, required fields) while tier-2 tests semantic quality (LLM output relevance). A prompt change may produce valid structure but lower quality.
Fix: Review the tier-2 failure details — they show the quality score vs threshold. Adjust the prompt or agent step logic and re-run.
-
Tier-2 costs more tokens than expected
Cause: Each tier-2 test makes LLM API calls. Running all agents' tier-2 tests can consume significant tokens.
Fix: Use --agent <name> to scope tier-2 to a specific agent. Reserve full --tier2 runs for pre-release validation.
Rules
- No ADO or LLM calls in tier-1 — fixtures are self-contained; safe to run anytime
- Tier-2 costs tokens — warn user before running
--tier2
- Never auto-push — eval is advisory; user decides whether to push
