| name | eval-harness |
| description | Use in non-deterministic testing — Mode-2 real-model or Mode-3 real-platform — when exact-string asserts would flake. Supplies the determinism split (mock = assert exact, real-model = assert a predicate over k runs), pass@k vs pass^k reliability metrics, the grader taxonomy (code / model / human), and baseline-SHA regression framing. Do not use for deterministic mock-LLM scenarios (assert exactly) or for scenario generation (your eager QA scenario contract owns that). |
Eval harness
Your scenario list, PASS/FAIL-per-scenario, smallest-repro, and two-artifacts contract live eager in your QA card (qa.md) — this skill does not restate them. It adds the missing piece for non-deterministic runs: how to assert a flaky LLM behaves without manufacturing a false PASS or a false FAIL.
The determinism split (the core rule)
- Mock-LLM (Mode-1, deterministic). Output is fixed → assert exactly: exact strings, exact tool calls, exact counts. One run suffices, and a flaky assert here is a real bug, not noise.
- Real-LLM (Mode-2) / real-platform (Mode-3), non-deterministic. The same input yields different phrasings and tool orderings → never assert exact strings. Assert a behavioral predicate (did it call
{tool}? did the reply contain {required fact}? did it stay within {N} turns? did the ACL hold?) across k runs, and report the rate.
Reliability metrics — report both, with k and n stated
- pass@k = "succeeded at least once in k attempts" → for capability ("can the agent ever do X"); a reasonable target is a high pass@3.
- pass^k = "succeeded in all k attempts" → for anything on a user-facing or safety path (no double-send, no secret/PII egress, ACL isolation, idempotent dispatch); the bar is pass^k = 100%.
- A behavior that is pass@3 but not pass^3 is not shippable on a critical path — say so explicitly. Write
pass^5 = 5/5, never just "passes."
Grader taxonomy — pick the cheapest that's sound
- Code-based (preferred, deterministic): {{TEST_CMD}}, grep over a saved run trajectory, exit codes, DB/state row checks. Use whenever the success condition is mechanically checkable.
- Model-based (only for open-ended output quality): an LLM judges "did the reply answer the question / is the tone right." It is itself non-deterministic → give it a fixed rubric and run it pass@k too; never let a model-grader gate something a code-grader could check.
- Human: flag to the product owner through the gate when the call is taste or irreversible risk and no automatable check is sound.
Regression framing
Pin a baseline SHA (last green commit / deployed baseline). A change is a regression if a scenario that passed at baseline fails now — re-run the same scenario against both and report X/Y passed (was Y/Y at {sha}). This is the {{TEST_CMD}} / {{E2E_HARNESS}} lane the squad already gates on.
Bind to the target repo
Mode-1 mock-LLM = {{E2E_HARNESS}}'s hermetic default; Mode-2 real-model = {{E2E_HARNESS}}'s live/real-dependency variant; Mode-3 real-platform is the human checklist (no mock). Never assert wall-clock latency as PASS/FAIL — record latency and judge the SLO in post-eval (the no-hard-timeout rule). Keep model/API secrets in env; never echo one into evidence.
Defer
Report results in your QA card's PASS/FAIL + two-artifacts + smallest-repro format (qa.md). This skill changes what you assert (a predicate plus a rate) and how confident the result is — not how you report it.
