| name | eval-runner |
| description | Use when changing prompts, tools, context packing, provider behavior, permissions, subagents, search, or other agent quality-sensitive logic. Guides focused tests, SanityHarness runs, metric capture, and adoption decisions. |
Eval Runner
Use evals to decide, not taste.
Steps
- Identify the behavior that changed.
- Run the smallest unit or golden tests first.
- Run internal task evals when agent behavior changed.
- Run SanityHarness slice for major changes.
- Compare against a baseline commit or saved result.
- Record metrics and decision in
eval/results/.
Metrics
Track pass rate, weighted score, tokens, cost, wall time, context misses, unrelated edits, and bug escapes.
Adoption rule
Adopt by default only when quality is flat or better. Token savings alone are not enough.