name	rubric-judge
description	To evaluate or compare outputs (candidate answers, PRs, drafts, model results) objectively, define an explicit rubric first and score each criterion with evidence, instead of a vibe "looks good". Use to pick the best of N options, gate quality, or grade generated content. Trigger with /rubric-judge or "score this", "judge against a rubric", "which option is best".
version	0.1.0
user-invocable	true
metadata	{"emoji":"⚖️"}

rubric-judge

"Looks good" is not an evaluation. Define the criteria first, then score each one with evidence. Turns a vague gut call into a repeatable, defensible judgment, and lets you pick the best of several options on the merits.

Why this exists (evidence)

LLM-as-a-judge (Zheng et al., MT-Bench / Chatbot Arena, arXiv:2306.05685): a model scoring outputs against criteria agrees with human preference ~80%+ of the time, comparable to human-human agreement, when the rubric is explicit. The key is the EXPLICIT rubric; freeform "rate this" is noisy and biased (length, position, self-preference).
It targets the failure where outputs are accepted on vibe, or N options are compared inconsistently.

When to use

Pick the best of N candidates (answers, designs, patches, prompts).
Gate quality on generated content (docs, copy, a PR) before it ships.
Grade a batch consistently (evals).
NOT for trivial obvious calls.

The method

Define the rubric FIRST (before seeing answers if possible): 3-6 criteria that matter for THIS task (e.g. correctness, completeness, clarity, safety, scope-fit), each with a short scale (e.g. 0-2 or pass/partial/fail) and what each level means.
Score each criterion with EVIDENCE: cite the specific part of the output that earns/loses points. No score without a reason.
Aggregate: sum/weight; note any hard-fail criterion (e.g. incorrect = reject regardless of style).
Decide: best option / pass-fail, with the criterion-level breakdown so the call is auditable.

Bias guards: judge on the rubric only; ignore length and order; do not favor your own draft; when comparing, randomize/blind position if you can.

How to run it

Inline for a few options.
Workflow: a judge stage scores each candidate against the schema'd rubric in parallel; synthesis picks the winner. Use an INDEPENDENT judge agent (not the author) for less self-preference. Compose with self-consistency (multiple judges, take the consistent verdict) for high-stakes.

Composes with

multi-agent-debate / self-consistency: generate/compare candidates; rubric-judge scores them.
adversarial-verify / testsmith: for CODE, prefer executed tests over judged opinion; use rubric-judge for things you cannot run (prose, design, tradeoffs).

Honest limits

A judge is still a model: it can be wrong, biased, or gamed. For anything executable, a real test beats a judged score. Use rubric-judge where execution is not possible.
Rubric quality caps judgment quality: vague criteria -> vague scores. Spend the effort on the rubric.

rubric-judge

Why this exists (evidence)

LLM-as-a-judge (Zheng et al., MT-Bench / Chatbot Arena, arXiv:2306.05685): a model scoring outputs against criteria agrees with human preference ~80%+ of the time, comparable to human-human agreement, when the rubric is explicit. The key is the EXPLICIT rubric; freeform "rate this" is noisy and biased (length, position, self-preference).

It targets the failure where outputs are accepted on vibe, or N options are compared inconsistently.

When to use

Pick the best of N candidates (answers, designs, patches, prompts).

Gate quality on generated content (docs, copy, a PR) before it ships.

Grade a batch consistently (evals).

NOT for trivial obvious calls.

The method

Define the rubric FIRST (before seeing answers if possible): 3-6 criteria that matter for THIS task (e.g. correctness, completeness, clarity, safety, scope-fit), each with a short scale (e.g. 0-2 or pass/partial/fail) and what each level means.

Score each criterion with EVIDENCE: cite the specific part of the output that earns/loses points. No score without a reason.

Aggregate: sum/weight; note any hard-fail criterion (e.g. incorrect = reject regardless of style).

Decide: best option / pass-fail, with the criterion-level breakdown so the call is auditable.

Bias guards: judge on the rubric only; ignore length and order; do not favor your own draft; when comparing, randomize/blind position if you can.

How to run it

Inline for a few options.

Workflow: a judge stage scores each candidate against the schema'd rubric in parallel; synthesis picks the winner. Use an INDEPENDENT judge agent (not the author) for less self-preference. Compose with self-consistency (multiple judges, take the consistent verdict) for high-stakes.

Composes with

multi-agent-debate / self-consistency: generate/compare candidates; rubric-judge scores them.

adversarial-verify / testsmith: for CODE, prefer executed tests over judged opinion; use rubric-judge for things you cannot run (prose, design, tradeoffs).

Honest limits

A judge is still a model: it can be wrong, biased, or gamed. For anything executable, a real test beats a judged score. Use rubric-judge where execution is not possible.

Rubric quality caps judgment quality: vague criteria -> vague scores. Spend the effort on the rubric.