auto-eval

name	auto-eval
description	Run the AI automation evaluation framework against test fixtures. Verifies agent quality without hitting ADO or LLM APIs. Use after changing prompts, rules, or agent steps. Accepts --all, --agent, --tier2 flags.
when_to_use	Use when running the automation evaluation framework against test fixtures, after changing agent prompts or rules, when the user says 'run evals', 'test agent quality', 'verify agent output', or '--all --agent --tier2'.
argument-hint	[--all | --agent dor|pr-review|pr-answer|discover | --tier2 | --fixture [name]]

You run the evaluation framework for the AI automation agents. Eval runs against pre-captured fixtures (no live ADO or LLM calls for tier-1 gates).

0. Prerequisites

Read .ai/automation/infra.json to confirm scaffold exists.

Check Node.js is available: node --version. If missing: STOP with install instructions.

Check eval fixtures exist:

ls .ai/automation/eval/fixtures/ 2>/dev/null | head -5

If no fixtures: "No eval fixtures found. Run /auto-test --capture first to capture fixtures from live ADO, or add fixtures manually to .ai/automation/eval/fixtures/."

1. Parse Arguments

Default: --all if no argument given.

Supported flags (pass through to eval/run.js):

• --all — run all fixtures
• --agent <name> — run only fixtures for a specific agent (dor, pr-review, pr-answer, discover)
• --tier2 — run tier-2 gates (makes real LLM calls; slower and costs tokens)
• --fixture <name> — run a single fixture by name

--agent discover — KAI-HUB repo discovery

Scores the dx-discover-repos cascade by alias-set match: a fixture passes when the discovery output's set of alias values equals the fixture's expected_aliases (order-independent). Fixtures live in .ai/automation/eval/fixtures/discover/ (seed copies ship at dx-automation/skills/auto-eval/fixtures/discover/), one per tier:

Fixture	Tier	Deterministic?
01-explicit	0 — repos: directive	yes (offline)
02-simple-block	1 — ```simple block routing	yes (offline)
03-crossrepo-table	2 — ## Cross-Repo Scope table	yes (offline)
04-llm-inference	3 — LLM infers from title/description	no — needs --tier2; documents expected set, model-dependent

Tiers 0–2 run offline in tier-1 (no ADO/LLM) and must pass exactly — they share logic with dx-hub/skills/dx-discover-repos/tests/run-tests.sh. Tier 3 only runs under --tier2; treat a miss as a prompt-tuning signal, not a hard gate.

2. Run Eval

cd .ai/automation
node eval/run.js $ARGUMENTS

Report the output as-is.

3. Interpret Results

After eval completes, summarize:

• Pass rate: N/total fixtures passed
• Failed fixtures: list names of failed fixtures
• Gate breakdown: which gates failed (if any)

If any fixtures fail:

⚠️ N fixture(s) failed. Check the output above for details. Do NOT push prompt changes until all fixtures pass.

If all pass:

✓ All fixtures passed. Safe to push changes.

4. Common Eval Workflows

After changing a prompt file:

/auto-eval --all

Before pushing a code change:

/auto-eval --all

Debug a specific agent:

/auto-eval --agent pr-review

Full quality regression (slower, uses LLM):

/auto-eval --tier2

Examples

1. /auto-eval — Runs tier-1 evaluation against all agent test fixtures. Parses agent step outputs, validates against expected results (no ADO or LLM calls). Reports 45/47 assertions passed, 2 failures in the PR Review agent's comment formatting logic.

2. /auto-eval --agent pr-review — Runs tier-1 fixtures for the PR Review agent only. Useful for debugging a specific agent after changing its prompt or step logic. Reports 12/12 assertions passed.

3. /auto-eval --tier2 — Warns about token cost, then runs full quality regression including LLM calls. Tests that agent outputs meet quality thresholds for each scenario. Reports quality scores per agent and overall pass/fail.

Troubleshooting

• "Fixtures not found for agent" Cause: No test fixtures exist in the eval directory for the specified agent. Fix: Create fixtures in .ai/automation/eval/fixtures/<agent-name>/ with input/expected-output pairs. See existing fixtures for the format.

• Tier-1 passes but tier-2 fails Cause: Tier-1 tests structural correctness (format, required fields) while tier-2 tests semantic quality (LLM output relevance). A prompt change may produce valid structure but lower quality. Fix: Review the tier-2 failure details — they show the quality score vs threshold. Adjust the prompt or agent step logic and re-run.

• Tier-2 costs more tokens than expected Cause: Each tier-2 test makes LLM API calls. Running all agents' tier-2 tests can consume significant tokens. Fix: Use --agent <name> to scope tier-2 to a specific agent. Reserve full --tier2 runs for pre-release validation.

Rules

• No ADO or LLM calls in tier-1 — fixtures are self-contained; safe to run anytime
• Tier-2 costs tokens — warn user before running --tier2
• Never auto-push — eval is advisory; user decides whether to push