eval-harness

name	eval-harness
description	Use when non-trivial or AI-assisted behavior needs measurable success criteria and repeatable reliability evidence.

Eval Harness

Apply Eval-Driven Development (EDD): define measurable success before implementation, then evaluate and report after implementation. Announce use before defining criteria.

When to Use

Use for a non-trivial feature, an AI-produced result that is hard to trust, a past regression, modified prompts or agents, or a production-readiness decision. Evals complement unit tests by measuring end-to-end capability and repeatability.

Route

• To choose evals and graders, load eval design.
• To create definition files, reports, and run history, load templates and reporting.
• During planning load the first branch; during reporting load the second. Load both only when creating an eval package end to end.

Invariants

• Follow DEFINE → IMPLEMENT → EVALUATE → REPORT; criteria exist before implementation.
• Include capability evals for intended behavior and regression evals for protected behavior.
• Prefer a deterministic grader based on executable tests, builds, or explicit output checks. Use LLM-as-judge only for open-ended quality and a human gate for security, legal, or subjective UX decisions.
• Record independent attempts rather than retrying until a pass is convenient.
• Report pass@k with pass@1 and pass@3, plus pass^3, accurately. Target pass@3 ≥ 90% for ordinary capability evals and pass^3 = 100% for critical auth, payment, migration, and regression paths.
• Never claim approval when a required grader or human gate is unresolved.

Red Flags

• Success criteria written after seeing the implementation.
• Vague graders, nondeterministic fixtures, changed baselines, cherry-picked attempts, or one run presented as reliability.
• An LLM judge replacing a deterministic check that can be automated.
• Averaging away a failed critical-path run.

Integration

writing-plans defines evals; test-driven-development implements low-level tests; verification-before-completion runs the suite; code-review consumes results; knowledge-compounding records recurring failures. Store active evals under .agent/evals/ and archived reports under docs/eval-results/.