Menu
Guides validation of evaluators, especially LLM judges, against labeled examples. Use when evaluator quality is uncertain, judge scores seem inconsistent, or you need to check whether the evaluator is biased, noisy, or misaligned.
Guides design of LLM-as-judge prompts for subjective evaluation criteria. Use when deterministic checks are insufficient and you need a judge prompt for quality dimensions like helpfulness, faithfulness, clarity, or tone.
Guides building or improving interfaces for human review of eval traces. Use when humans need to inspect failures, label outputs, compare model behavior, or audit evaluator decisions at scale.
Guides systematic analysis of eval failures using traces. Use when a suite is failing, model outputs seem inconsistent, evaluator behavior is suspect, or you need to classify failures before changing prompts, metrics, or datasets.
Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.
Guides creation of synthetic eval cases that expand coverage without drifting away from real usage. Use when the current eval set is too small, too repetitive, or missing edge cases, and you need more diverse prompts, distractors, or structured scenarios.
Drafts a reviewable judge-template patch from evaluator validation disagreements.
| name | validate-evaluator |
| description | Guides validation of evaluators, especially LLM judges, against labeled examples. Use when evaluator quality is uncertain, judge scores seem inconsistent, or you need to check whether the evaluator is biased, noisy, or misaligned. |
Collect labeled examples or high-confidence gold cases first. Inspect evaluators/index.mjs, especially judge-based paths, and read traces where the evaluator’s decision seems suspicious.
Run:
npm run skill:validate-evaluator
For a specific label file:
node scripts/validate-evaluator.mjs labels/sample-goldens.json
Use repeated runs when validating a judge:
node scripts/validate-evaluator.mjs labels/sample-goldens.json --repeat 5
Treat agreement, false positives, false negatives, disagreement samples, and stability as the primary output. A judge that flips verdicts on repeated calls is not calibrated, even if one run reports high agreement.
The validator exits nonzero when provider calls fail, parse failures occur, agreement or stability are below threshold, or drift exceeds the supplied baseline. Override defaults with --min-agreement, --min-stability, --drift-baseline, --max-agreement-drop, and --max-stability-drop.
The validator also reports Cohen's kappa. Use kappa alongside raw agreement because raw agreement can look strong on imbalanced labels.
Use a judge panel when one model is too noisy:
node scripts/validate-evaluator.mjs labels/sample-goldens.json --judge-panel openai:gpt-5.5,anthropic:claude-haiku-4-5
Run the public synthetic bias checks before trusting pairwise judges:
npm run skill:judge-bias-check
Use:
The validation set should include both passes and fails.
Do not rely on one aggregate accuracy value alone. Check:
Check whether the evaluator:
If the evaluator is unreliable on labeled examples, fix the rubric or the parsing before using it to score more data.
Use suite-specific judge templates from judges/ when possible. Keep generic criteria as a fallback only.
For RAG calibration sets, use rag-quality unless you are intentionally testing a narrower relationship such as rag-faithfulness.
evaluators/index.mjsrun-eval.mjsevals/traces/app.htmllabels/judges/scripts/validate-evaluator.mjs