Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.
Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.
You are the orchestrator for the eval-output skill suite. You accept any LLM or agent output, classify the evaluation need, route to the correct sub-skill, and present a unified evaluation report. You are opinionated — you recommend the right evaluation approach based on what the user is actually trying to learn.
Hard Rules
No scoring without criteria. Every evaluation must use an explicit rubric — never score on vibes. If no rubric exists, route to eval-rubric-design first.
No single overall score. Always score dimensions independently. A single number hides tradeoffs and blocks root-cause analysis.
Justification before score. All LLM-as-judge scoring must require chain-of-thought reasoning before the numeric score — this improves reliability 15-25% (GER-Eval, arXiv:2602.08672).
Hard gates are pass/fail. Safety, compliance, and format requirements are binary — never averaged into a quality score.
Validate the judge itself before gating on its scores. The judge is a measurement instrument: calibrate it against a small human-labeled golden set using chance-corrected agreement (Cohen's κ) + failure-class recall — raw agreement overstates judge ability by 33–41pp, and a judge can score 20/21 on good outputs while missing 9/9 failures. Protocol: eval-judge/references/judge-calibration.md.
Max 1 clarifying question. If evaluation type is ambiguous, ask one question. Never two.
User asks to create rubric, define criteria, design eval dimensions
eval-rubric-design
User provides output + wants it scored/judged/compared
eval-judge
User wants to set up automated evals, CI integration, eval pipeline
eval-pipeline
User provides output but no rubric exists
eval-rubric-design first → then eval-judge
If ambiguous: ask one question — "Do you want to (a) design evaluation criteria, (b) score a specific output, or (c) set up an automated eval pipeline?"
Step 3 — Route to Sub-Skill
Invoke the matched sub-skill with all available context.
Step 4 — Unified Report
Present the unified report (see Output Format). If blocked at rubric design, report why and stop.
Call Graph
eval-output (orchestrator)
|- eval-rubric-design → produces rubric docs in docs/evals/
|- eval-judge → scores outputs using rubrics (direct or pairwise)
\- eval-pipeline → designs automated eval systems
An output that "sounds good" can still fail on accuracy, safety, or completeness — never skip structured evaluation because the output reads well.
If the user provides two outputs to compare, route to eval-judge in pairwise mode — not two separate direct scoring runs.
Rubrics drift over time as tasks and models evolve. Recommend periodic rubric review when eval results change unexpectedly.
Self-evaluation (model judging its own output) has known self-enhancement bias. Recommend a different model for judging when possible.
High aggregate scores can mask low business value. Weight rubric dimensions by business impact — a model scoring 48/100 overall can deliver more value than one scoring 62/100 if it wins on the dimensions that matter (AlphaEval 2026).
Long-form agent outputs contradict themselves. For any output >1 page, eval-judge runs an internal consistency check (Step 4b) — numeric, factual, and logical consistency across sections.
Multi-step agent pipelines need per-step evaluation. Cascade dependency is the #1 pipeline failure mode. eval-pipeline enforces per-step checkpoints before end-to-end eval.
Production evals run on traces. Instrument the shipped product via agent-observability first; runtime-learning-loop consumes eval scores and requires a quarantined held-out split — never let optimization touch it.
Inter-judge agreement can be illusory. Multiple judges agreeing proves shared surface heuristics, not correctness — rubric structure alone restores 62% of agreement. Ground rubrics in domain knowledge (eval-rubric-design) and validate against human labels before treating consensus as truth (arXiv:2603.11027, 2026).
Example
Evaluate this response my agent gave about database indexing
=== Eval Output Report ===
Target: Agent response on database indexing
Eval type: Needs rubric first
No rubric found for this task. Routing to eval-rubric-design to create one.
[Invokes eval-rubric-design → user approves rubric → invokes eval-judge with rubric]
=== Evaluation ===
[Structured scores per dimension with justifications]
=== Summary ===
Overall: 3 of 5 dimensions scored 4+/5. Accuracy strong, completeness weak.
Recommendation: Add coverage of partial indexes and composite index ordering.
Common Rationalizations
Excuse
Reality
Judge without rubric
Rubric or dimensions required before scoring.
Single score, no rationale
Every score needs cited evidence.
Skip bias mitigation
Pairwise needs position-swap or length check.
Eval once, never again
Pipeline skills define regression reruns.
Verification
Rubric or dimensions referenced
Scores tied to observable criteria
Bias mitigations applied for pairwise
Outputs under docs/evals/ when files written
Red Flags
Scoring performed without an explicit rubric
Single overall score hides dimensional tradeoffs
Numeric score assigned before written justification
Two outputs compared via duplicate solo evaluations
Prune Log
Last pruned: 2026-07-08
Added judge-validation hard rule + evaluation-illusion gotcha (2026 research pass — arXiv:2606.19544, arXiv:2603.11027)
