llm-eval-harness

name	llm-eval-harness
description	Builds an evaluation harness for LLM/agent outputs using golden datasets, code-based scorers, and LLM-as-judge, run as a regression gate when prompts, models, or RAG configs change.
when_to_use	User wants to measure prompt/model/agent quality, stop regressions when changing a prompt or model, set up llm-as-judge or a golden dataset, or go beyond vibes-based testing. NOT for deterministic unit tests of plain code (use write-tests).

When to Use

Reach for this when output quality is non-deterministic and "looks fine" is not good enough:

• Changing a prompt, model version, temperature, tool definitions, or RAG retrieval config and need proof it didn't regress.
• Setting up llm-as-judge, a golden dataset, or any CI gate that scores model/agent output.
• A bug report like "answers got worse after we switched models" — you need a number, not a vibe.

Do NOT use for deterministic code (a parser, a pure function, an API contract) — that is a normal unit test; use write-tests. The line: if the same input can produce different-but-valid outputs, it's an eval; if there's one correct output, it's a test.

Steps

1. Pick the unit of evaluation first. Decide what one "case" is: a single prompt→completion, a full agent trajectory (tool calls + final answer), or a RAG turn (query + retrieved context + answer). Everything below is keyed to this unit. Don't mix units in one suite.

2. Build a lean golden dataset (start at 20–50 cases, not 1000). Pull real cases from production logs/traces, not invented ones — sample across the actual input distribution. Each case is a row with: id, input (+ any variables like retrieved_context), expected (or reference), and tags (e.g. edge, regression, pii). Deliberately seed known edge cases and every past failure. Store as JSONL or CSV under evals/data/ so diffs are reviewable in git.

3. Layer metrics cheapest-first; only escalate to a judge when needed.

   • Code scorers (deterministic, free, run first): exact match, regex/JSON-schema validation, "must contain / must NOT contain" substrings, valid-tool-was-called, refusal-detector, latency/cost budget, output parses as valid JSON. These catch the majority of regressions with zero flakiness.
   • Semantic match: embedding cosine similarity vs. reference, with a calibrated threshold — use for "is this roughly the right answer" when wording varies.
   • LLM-as-judge (last resort, for fuzzy quality): correctness, faithfulness-to-context, helpfulness, tone. Only build a judge for dimensions code can't check.

4. Make the judge rigorous, not vibes-in-a-trenchcoat.

   • Write a rubric with discrete levels (e.g. 1–5 or pass/borderline/fail) where each level has a concrete, observable definition. Force the judge to emit reasoning BEFORE score (CoT lifts agreement).
   • Pin the judge: fixed model id + temperature=0 (or near-0) so the gate is reproducible.
   • Prefer pairwise or reference-based grading over a lonely 1–10 score — absolute scores drift; "is A better than reference B?" is far more stable. Mitigate position bias by randomizing A/B order.
   • For faithfulness/RAG, the judge sees the answer + the retrieved context and rules ONLY on "is every claim supported by context" — not on world knowledge.

5. Calibrate the judge against humans before trusting it. Hand-label 20–30 cases. Run the judge on the same cases and compute agreement (Cohen's κ or simple % match). If agreement is low, the rubric is ambiguous — tighten level definitions and re-run. Do not ship a judge you haven't calibrated; an uncalibrated judge is a random number generator with a PhD voice.

6. Wire variable mapping + per-case pass/fail. Each case template maps dataset columns → prompt variables (e.g. {{question}}, {{context}}). Define per-case pass = all code scorers pass AND judge score ≥ threshold. Aggregate to a suite-level pass rate and per-tag breakdown.

7. Run as a deterministic gate in CI and on every prompt/model/config change. Trigger on changes to prompt files, model id, or RAG config. The runner: load dataset → run candidate system per case → score → compare against the committed baseline scores file (evals/baseline.json). Fail the build if pass-rate drops below threshold OR any case in the regression tag flips from pass→fail. Print a per-case delta table (old score → new score) so the regression is obvious in the PR.

8. Add random-sample probing to surface NEW failures. The golden set only tests known cases. On a schedule, sample fresh production inputs, run them, and have the judge flag low-quality outputs for human review. This finds failure modes the static dataset can't.

9. Close the loop: every confirmed failure becomes a golden case. When probing or production surfaces a bad output, add it to the dataset tagged regression with the corrected expected. The harness gets stronger over time and the same bug can never silently return.

Common Errors

• Judge with temperature > 0 → non-reproducible gate. Two runs of the same diff give different verdicts and the gate becomes noise. Pin model + temp=0, and pin the model version/snapshot (a silently-updated judge model is itself a regression source).
• Same model judges its own output → inflated, biased scores. Use a different (often stronger) model as judge, or at minimum acknowledge the bias. Never let the system grade its own homework on quality dimensions.
• Absolute 1–10 scoring drifts and clusters at 7–8. Switch to pairwise (vs. reference) or a discrete rubric with anchored definitions. "Rate 1–10" without anchors is the #1 cause of a useless judge.
• Position/verbosity bias in pairwise judging. Judges favor the first option and longer answers. Randomize order per case (and optionally swap-and-average); penalize verbosity explicitly in the rubric if needed.
• Tiny or synthetic dataset → green evals, red production. 5 cherry-picked cases prove nothing. Pull from real traffic and cover the actual input distribution, including the boring/edge tails.
• Reaching for the LLM judge when a code scorer would do. If the spec is "output must be valid JSON with field X" or "must not leak the system prompt," that's a regex/schema check — deterministic, free, flake-free. Don't pay a judge to check what assert can.
• No committed baseline → "is this better?" is unanswerable. Without baseline.json under version control you can detect a crash but not a regression. Commit baselines and update them deliberately (in a reviewed PR), never silently.
• Judge marks a faithful RAG answer wrong because it "knows better." A faithfulness judge must rule only on support-by-retrieved-context, not on its own world knowledge. State this explicitly in the judge prompt or you'll get false fails.
• Flaky scorers fail the gate for the wrong reason. Network timeouts / rate limits on judge calls look like quality regressions. Add retries + a clear "harness error" vs. "quality fail" distinction so a 429 doesn't block a ship.

Verify

• Run the full suite twice on the same candidate; the score must be identical (proves the gate is deterministic). If it isn't, a scorer/judge has hidden randomness.
• Confirm the gate actually bites: introduce a deliberately bad prompt and check the suite fails with a clear per-case delta pointing at the regressed cases.
• Confirm a known-good change passes and the baseline-comparison table renders old→new deltas.
• Spot-check 3–5 judge verdicts by hand against the rubric — the judge's reasoning should justify its score. If reasoning and score disagree, the rubric/prompt needs tightening.
• Verify CI is wired: the eval job triggers on prompt/model/RAG-config file changes and blocks merge on failure (not just warns).
• Confirm every regression-tagged case maps to a real past failure, and that the latest surfaced failure has been added back into the golden set.