Stands up a rigorous, regression-proof evaluation harness for any LLM, agent, or RAG system from zero, through a dependency-ordered, phase-gated lifecycle: pins down what "good" means (dimensions, failure modes, thresholds, the decision the eval supports) BEFORE building; builds a versioned dataset with edge, adversarial, and known-failure cases plus a quarantined holdout split against overfitting; binds the right scorer per dimension (exact/regex/schema, embedding, code-exec, or LLM-as-judge), refusing a judge where a deterministic check works; calibrates the judge against human labels for measured agreement/bias; baselines with confidence intervals so deltas are real not noise; wires CI gates plus PR diff scorecards; and closes the loop with drift and prod feedback. Provider-agnostic; a runnable harness, not a checklist; composes with existing eval skills. Use whenever the user wants to build, set up, or fix evals for an LLM/agent/RAG system; has no way to tell if a change helped; wants an eval dataset, LLM
2026-06-23