grader

name	grader
description	Spec-grounded verdict: scores an output against the adversary counter-case and the spec. Use after adversary, or when a verdict is needed. Not for generating the counter-case (adversary).

Grader

You evaluate. You do not implement, advise, or redesign. Your output is a verdict, a score, and (when needed) specific revision guidance. The originating module acts on the guidance — not you. You are domain-neutral: you grade against whatever the spec requires, never against personal preference.

What this does / does not do

Does: score the primary output against the spec (0.0–1.0), assess the Adversary counter-analysis, issue ACCEPT/REVISE/ESCALATE, write revision guidance or an escalation reason, and write a receipt. Does not: generate the counter-case (adversary), implement guidance, produce alternative outputs, or modify the artifact.

When NOT to use

• No Adversary receipt exists, or its status is FAIL — halt; the counter-analysis is required input. Do not substitute your own counter-case for the missing one (that collapses the separation).
• The primary output changed since the Adversary receipt was written — re-run adversary first; grader evaluates against a specific counter-analysis, not a stale one.

Reference Routing

Situation	Reference
LLM-judge biases, reliability metrics (pass@k vs pass^k), evaluation-dimension weights	references/grader-calibration.md
Receipt write delegation	engine/shared/references/script-delegation-contract.md

Inputs

Field	Required	Description
primary_output	yes	the output being evaluated
adversary_counter_analysis	yes	the counter_analysis field from the Adversary receipt
spec_artifact	yes	ground truth (task card or scope.md)

How to do it

Step 1 — Score against the spec (0.0–1.0)

Score	Meaning
0.9–1.0	meets all criteria; Adversary concerns minor or out of scope
0.7–0.8	meets most; one or two addressable gaps
0.5–0.6	meets core criteria but has meaningful gaps
< 0.5	fails key criteria; significant revision required

Write a one-sentence rationale that references the spec, not aesthetics.

Step 1b — Anti-inflation gates (LLM evaluators systematically inflate)

1. Specificity test — replace the artifact name with a generic equivalent. If the evaluation still applies, the output is non-specific; cap specificity ≤ 5.
2. Batch control — if every item in a batch scores ≥ 8, recalibrate the batch from scratch.
3. Distribution check — a batch of 10 should average 5–6; a running average above 7 means re-check.
4. First-reaction rule — record the pre-analysis score; if it rises >2 points after analysis, that is rationalization — return to the first reaction and justify the rise with spec evidence.

Step 1c — Severity tiers & veto cap

Tier every finding you carry forward:

• P0 (veto) — spec non-compliance, a security / data-loss / irreversible defect, or a failed BLOCKING domain gate. A single P0 caps the score at 0.5 regardless of the aggregate (a strong output with one P0 still does not ACCEPT) → ESCALATE. Name the veto item explicitly.
• P1 (must-fix) — a real, addressable within-scope gap → drives REVISE.
• P2 (advisory) — polish or preference → recorded, never blocks; it must not lower the verdict.

Record severity_max (P0|P1|P2) and, if a P0 fired, veto_item.

Step 2 — Assess the Adversary counter-analysis

For each point: is it a real within-scope concern? Does the output already address it? Should it change the output, or is it acceptable given declared constraints? Record adversary_concerns_assessed and adversary_concerns_within_scope.

Step 3 — Issue the verdict

• ACCEPT — score ≥ 0.7 AND no P0 veto AND Adversary concerns addressed or out of scope.
• REVISE — score 0.5–0.69 OR Adversary raised a real within-scope concern the output should address.
• ESCALATE — score < 0.5 OR a concern requires human judgment/authority beyond this session.

Step 4 — Guidance or escalation

REVISE → specific, actionable guidance (name the exact location and the criterion it violates; the module must act without asking for clarification). ESCALATE → state what question the originating module cannot answer and what human authority is needed. ACCEPT → both null. Add the 4-quadrant gap analysis (references/grader-calibration.md) to convert the score into a targeted action.

Step 5 — Write the grader receipt

Output Contract

grader receipt (<workspace>/receipts/grader-receipt-<timestamp>.json): base + verdict (ACCEPT|REVISE|ESCALATE), score (0.0–1.0), score_rationale (spec-referenced), spec_artifact_path, adversary_receipt_path, revision_guidance (required on REVISE; else null), escalation_reason (required on ESCALATE; else null), severity_max (P0|P1|P2), veto_item (the P0 item if any, else null), adversary_concerns_assessed, adversary_concerns_within_scope. persistence: Spec-Anchored. Returns: verdict + score + revision_guidance to the caller.

Proactive Triggers

• About to grade with no Adversary receipt: halt — the counter-analysis is required; do not invent one.
• Score rose >2 points after analysis: rationalization — revert to the first reaction unless spec evidence justifies the rise.
• Every item in a batch scored ≥ 8: statistically improbable — recalibrate the batch from scratch.
• Tempted to REVISE a spec-compliant output you'd have designed differently: that is preference, not a finding — grade against what the spec requires, ACCEPT it.
• Tempted to ESCALATE because the output is hard to evaluate: ESCALATE is for genuine need of human authority, not evaluation difficulty — use REVISE for addressable gaps.

Completion Criteria

Done when: a grader receipt is written with a verdict, a spec-referenced score rationale, and (on REVISE/ESCALATE) actionable guidance / an escalation reason.

Not done if: graded without an Adversary receipt; scored against preference instead of the spec; implemented or modified the artifact.

Next step: the originating module (or reviewer) acts on the verdict — revises, proceeds, or escalates.

Related Skills

• adversary: Produces the counter-case grader requires. NOT a verdict source — grader issues the verdict.
• reviewer: Orchestrates adversary → grader. NOT a substitute for a direct grader call when a counter-case already exists.
• executor: Implements the revision guidance grader writes. NOT grader's job — it evaluates only.