| name | rubric-judge |
| description | To evaluate or compare outputs (candidate answers, PRs, drafts, model results) objectively, define an explicit rubric first and score each criterion with evidence, instead of a vibe "looks good". Use to pick the best of N options, gate quality, or grade generated content. Trigger with /rubric-judge or "score this", "judge against a rubric", "which option is best". |
| version | 0.1.0 |
| user-invocable | true |
| metadata | {"emoji":"⚖️"} |
rubric-judge
"Looks good" is not an evaluation. Define the criteria first, then score each one with evidence. Turns a vague gut call into a repeatable, defensible judgment, and lets you pick the best of several options on the merits.
Why this exists (evidence)
- LLM-as-a-judge (Zheng et al., MT-Bench / Chatbot Arena, arXiv:2306.05685): a model scoring outputs against criteria agrees with human preference ~80%+ of the time, comparable to human-human agreement, when the rubric is explicit. The key is the EXPLICIT rubric; freeform "rate this" is noisy and biased (length, position, self-preference).
- It targets the failure where outputs are accepted on vibe, or N options are compared inconsistently.
When to use
- Pick the best of N candidates (answers, designs, patches, prompts).
- Gate quality on generated content (docs, copy, a PR) before it ships.
- Grade a batch consistently (evals).
- NOT for trivial obvious calls.
The method
- Define the rubric FIRST (before seeing answers if possible): 3-6 criteria that matter for THIS task (e.g. correctness, completeness, clarity, safety, scope-fit), each with a short scale (e.g. 0-2 or pass/partial/fail) and what each level means.
- Score each criterion with EVIDENCE: cite the specific part of the output that earns/loses points. No score without a reason.
- Aggregate: sum/weight; note any hard-fail criterion (e.g. incorrect = reject regardless of style).
- Decide: best option / pass-fail, with the criterion-level breakdown so the call is auditable.
Bias guards: judge on the rubric only; ignore length and order; do not favor your own draft; when comparing, randomize/blind position if you can.
How to run it
- Inline for a few options.
- Workflow: a judge stage scores each candidate against the schema'd rubric in parallel; synthesis picks the winner. Use an INDEPENDENT judge agent (not the author) for less self-preference. Compose with self-consistency (multiple judges, take the consistent verdict) for high-stakes.
Composes with
multi-agent-debate / self-consistency: generate/compare candidates; rubric-judge scores them.adversarial-verify / testsmith: for CODE, prefer executed tests over judged opinion; use rubric-judge for things you cannot run (prose, design, tradeoffs).
Honest limits
- A judge is still a model: it can be wrong, biased, or gamed. For anything executable, a real test beats a judged score. Use rubric-judge where execution is not possible.
- Rubric quality caps judgment quality: vague criteria -> vague scores. Spend the effort on the rubric.
