eval-runner

name	eval-runner
description	Use when changing prompts, tools, context packing, provider behavior, permissions, subagents, search, or other agent quality-sensitive logic. Guides focused tests, SanityHarness runs, metric capture, and adoption decisions.

Eval Runner

Use evals to decide, not taste.

Steps

1. Identify the behavior that changed.
2. Run the smallest unit or golden tests first.
3. Run internal task evals when agent behavior changed.
4. Run SanityHarness slice for major changes.
5. Compare against a baseline commit or saved result.
6. Record metrics and decision in eval/results/.

Metrics

Track pass rate, weighted score, tokens, cost, wall time, context misses, unrelated edits, and bug escapes.

Adoption rule

Adopt by default only when quality is flat or better. Token savings alone are not enough.