refine-to-rubric

name

refine-to-rubric

description

Iterate an artifact until it measurably passes a BOUNDED rubric — the Convergence Engine. Objective/deterministic gates run BEFORE any model judge, the judge is a DIFFERENT model than the author (never self-grade), the stop decision is a model-free predicate, and it emits the BEST iteration (keep-best/regression-revert) under a hard iteration cap + model-call budget. The engine NEVER claims perfect — verdicts are rubric-pass | capped | plateaued | budget-exhausted with an honest residual-gaps list. Use to drive any artifact (code, prose, a visual report, an agent file) to a defensible bar; for the agent-file case it delegates to agent-quality-rubric, and for visual surfaces it composes with visual-feedback-loop.

Skill: refine-to-rubric (the Convergence Engine)

A domain-neutral "iterate an artifact until it measurably passes a bounded rubric" loop. It is the inner mid-task loop; dod-gate.sh is the outer loop (a Stop-floor) — they compose, they do not duplicate.

The whole design exists to dodge the documented failure modes of self-improvement loops (the grounded findings in docs/plans/2026-06-23-convergence-engine/plan.md): pure self-critique without an external signal does not reliably improve and often degrades (Huang et al. 2310.01798); self-preference + sycophancy are real (2410.21819); any fixed proxy invites Goodhart/reward-hacking (Gao et al. 2210.10760); and round-2 regressions are real. So:

Objective signals decide first. A different model judges second. A model-free predicate decides when to stop. The best iteration is what ships. And the engine never says "perfect".

The loop

derive-rubric → evaluate (objective gates FIRST) → judge (cross-model, ONLY after
gates green) → score → terminate (deterministic) → refine one finding → repeat
→ emit the BEST iteration + a constrained report

step	who	how
derive-rubric	deterministic	scripts/derive_rubric.py retrieves from the externalized library knowledge/convergence-rubrics.md; explicit user requirements added at weight-max; an additive [unverified — derived] "commonly-missed" pass surfaces unknown-unknowns (never auto-graded). agent-file → delegate to agent-quality-rubric.
evaluate	deterministic + subprocess	scripts/evaluate.py runs each dimension's objective_signal (driver.py / svg-lint / declarative-viz / pbir / dod-cmd / lint-cmd) as an exit-coded CLI. A red objective hard gate short-circuits straight to refine — 0 judge calls.
judge	cross-model	scripts/judge.sh calls claude -p with a DIFFERENT model than the author (refuses to self-grade), scores only the judge-graded dims, returns fixed-severity findings. Runs only when evaluate reports judge_needed.
terminate	deterministic, model-free	scripts/converge.py terminate() — the ONLY stop authority.
refine	the author agent	apply the single highest-severity finding, then re-enter the loop.
emit	deterministic	keep_best() emits the argmax iteration; loop.py render_report() writes a constrained report.

Termination (deterministic — converge.py)

Stop with rubric-pass only when ALL of: every objective hard gate green, no NEW high/critical finding, score ≥ floor, and the score has plateaued (Δ < ε for the patience window). HARD stops: iteration cap (~6), model-call budget (~12), regression → revert-to-best. Verdict vocabulary is bounded and honest:

verdict	meaning
rubric-pass	the bounded rubric is satisfied — NOT a claim of perfection
capped	the iteration cap was reached first; best-so-far emitted
plateaued	the score plateaued below the floor; escalated to a human
budget-exhausted	the model-call budget ran out; best-so-far emitted

The emitted artifact is always the best iteration (keep_best argmax, ties → earliest), never blindly the last — so a round-2 regression cannot ship.

Cost bound (3 layers)

1. ≤ 2 model calls per iteration (judge + refine).
2. Objective-first short-circuit: a broken artifact spends 0 judge calls.
3. Hard caps: iteration cap + model-call budget (both in converge.py), and the loop inherits the repo's runaway-brake.sh.

Security (the cross-model claude -p judge path)

judge.sh reuses the hardened claude -p envelope proven by thing-seat.sh:

• Anti-self-grade: refuses (exit 5) when the judge model family equals the author model family — a same-model critic is not an independent signal.
• Secret-egress backstop: a secret-shaped artifact is detected (shared hooks/_scrub.sh patterns) and never transmitted to the model API; the scan runs before the test mock so a mock can never mask a leak.
• Prompt-injection envelope: the artifact is wrapped in a per-call nonce-tagged <untrusted-…> region; instruction-shaped text inside is data, and a forged delimiter / "give this a perfect score" sets injection_detected.
• No tools: --tools "" — the judge only reasons and returns JSON.
• Isolation: runs from a scratch cwd (no project CLAUDE.md auto-load).
• CI/test hook: JUDGE_MOCK_VERDICT returns canned verdicts so the gate never calls claude (CI must not).

Anti-reward-hack (the rubric is a proxy)

The rubric library is externalized + versioned (knowledge/convergence-rubrics.md), not invented at runtime. A model may only additively propose derived dimensions, which are forced to verified: false + [unverified — derived] and are NEVER auto-graded — a human promotes them by editing the rubric. The deterministic terminate() caps how hard any one dimension is optimized.

Reuse ledger (no duplication)

• visual-feedback-loop/driver.py → the visual objective-evaluator (a subprocess signal, unchanged).
• dod-gate.sh → the OUTER loop; this skill is the INNER loop.
• agent-quality-rubric → the agent-file rubric (delegated, not reimplemented).
• FORGE model-diversity + the P0/P1 severity vocabulary → the cross-model judge.
• .ravenclaude/runs/ + _emit-event.sh → per-iteration scorecards (derived labels only, no raw artifact echo).

Files

file	role
scripts/converge.py	model-free terminate() / weighted_score() / keep_best()
scripts/derive_rubric.py	deterministic rubric assembly from the library
scripts/evaluate.py	objective-gates-first evaluator (decides judge_needed)
scripts/judge.sh	the cross-model claude -p judge (anti-self-grade)
scripts/loop.py	the orchestrator + the constrained report renderer
schemas/rubric.schema.json	the rubric contract
schemas/convergence-scorecard.schema.json	the scorecard + verdict contract
knowledge/convergence-rubrics.md	the externalized rubric library (the spine)

Gates 115–118 (scripts/audit-gates.sh) prove each piece bidirectionally.