| name | judge-panel |
| description | Cross-family cascade-then-jury review. Use when the user says "judge this", "review with a panel", "cross-family review", "jury beats judge", or needs independent verification of an AI-generated artifact (code, spec, patent, copy, decision). Runs ≥2 cheap cross-family small-fish judges first (Gemini-Flash + GPT-nano). If they agree with high confidence, verdict stands. If they disagree or confidence is low, escalates to one expensive cross-family tiebreaker. Returns verdict + confidence + which judges fired + token cost.
|
| metadata | {"version":"3.4.0","author":"Anand Vallamsetla","tier":"core","plugin_number":4} |
BEGIN JUDGE-PANEL
Judge Panel — Cascade-then-Jury Verification
Plugin #4, Core tier. Operationalizes the Defense-in-Depth Part 2 thesis
("jury beats judge"). Constitution anchor: Ground Zero — Independent
Verification Gate + Model-Diversity sub-mandate. Also P0.5 (Boundary
Self-Awareness) and P22 (Boundary-First Qualification).
Why this exists
A single frontier LLM cannot peer-review itself. Same-family reviewers share
training distribution, RLHF, and characteristic failure modes — same-family
review is a closed loop that validates what the author already believes.
Cross-family review crosses the training-distribution boundary and lets new
information enter. Empirically (Defense-in-Depth Part 2 incident,
2026-04-23), a single Gemini-2.5-Flash pass caught a class of drift that
three same-family Claude reviewers had rationalized.
But parallel juries are expensive. Cascade-then-jury (FrugalGPT /
Cascade Routing lineage) is the cheaper default: cheap small-fish first,
escalate only on disagreement or low confidence. The literature:
- Verga et al. 2024 (PoLL) — panel of smaller diverse judges beats single
GPT-4 at ~7× less cost
- Chen/Zaharia/Zou 2023 (FrugalGPT) — cheap→expensive cascade, 30-98%
cost savings
- Dekoninck et al. ICML 2025 (Cascade Routing) — routing + cascading
competitive with either alone
When to activate
Explicit invocation:
judge-panel <artifact> / codi judge <artifact>- "review with a panel"
- "cross-family review this"
- "jury beats judge check"
- "run the cascade on this"
Automatic activation (when another skill delegates):
hallucination-detector (plugin #3) calls judge-panel for post-flight
scoringcalibration-auditor may call judge-panel for sycophancy scoring when
the caller requests an external verdict- Protocol 8 (Auto-Verify by Stakes, co-dialectic v4.1+) calls judge-panel
automatically at T3 and T4 — see Auto-fire trigger section below
Silent mode: When called by another skill or by Protocol 8, emit only the
JSON verdict. Skip the conversational framing.
The cascade
┌─────────────────────────────────────────────────────────────────┐
│ STAGE 1 — SMALL-FISH PANEL (parallel, always fires) │
│ ┌───────────────────┐ ┌────────────────────┐ │
│ │ Gemini Flash Lite │ │ GPT-5.4-nano │ │
│ │ (Google family) │ │ (OpenAI family) │ │
│ └─────────┬─────────┘ └──────────┬─────────┘ │
│ │ │ │
│ └─────────┬──────────────┘ │
│ ▼ │
│ ┌───────────────────────────┐ │
│ │ Aggregate verdicts │ │
│ │ Compute agreement + conf │ │
│ └───────┬───────────────────┘ │
│ │ │
│ ┌───────────┴───────────┐ │
│ ▼ ▼ │
│ AGREE + HIGH CONF DISAGREE or LOW CONF │
│ (skip escalation) (escalate) │
└────┼───────────────────────┼─────────────────────────────────────┘
│ │
│ ▼
│ ┌──────────────────────────────────────────────┐
│ │ STAGE 2 — BIG-FISH TIEBREAKER (only if needed) │
│ │ ┌────────────────────┐ │
│ │ │ GPT-5.4 (default) │ (or Gemini 3.1 Pro) │
│ │ │ cross-family vs. │ │
│ │ │ author (Claude) │ │
│ │ └──────────┬─────────┘ │
│ │ ▼ │
│ │ Final verdict = tiebreaker-weighted │
│ └──────────────────────┬───────────────────────┘
│ │
└──────────┬───────────────────┘
▼
┌──────────────────┐
│ Return JSON │
│ verdict packet │
└──────────────────┘
Persona-driven judges (v3.4.0+)
Judges default to a generic evaluation lens. For rubrics where expert taste
matters more than factual accuracy, the panel now injects a persona line
into each judge's prompt:
"Judge as {persona(s)} — channel the top-0.001% standard in their domain;
scrutinize as they would and catch the minute details they would catch."
Why personas. Apple didn't become Apple without obsession over every
minute UX detail. A generic judge rubber-stamps; a Jobs + Ive judge catches
the misplaced arrow, the inconsistent spacing, the CTA that creates
micro-friction. The persona is a quality FLOOR, not a restriction — it
raises scrutiny to the caliber the domain actually demands.
Persona is LAYERED ON TOP of cross-family diversity. The two small judges
remain Gemini (Google family) + Codex (OpenAI family); each also adopts the
same persona lens. Cross-family guarantee is unchanged.
CLI option
bun run judge_panel.ts --rubric <slug> --artifact "..." --persona "Steve Jobs + Jony Ive"
Also readable from environment:
JUDGE_PANEL_PERSONAS="Steve Jobs + Jony Ive" bun run judge_panel.ts ...
The --persona flag (or JUDGE_PANEL_PERSONAS env var) always overrides
the rubric default. Pass an empty string or omit the flag to use the default.
Default persona map (rubric → persona)
| Rubric | Default persona | Rationale |
|---|
ux | Steve Jobs + Jony Ive | UX demands obsession over minute details; Jobs (product vision) + Ive (tactile/visual craft) together set the highest bar |
visual | Steve Jobs + Jony Ive | Visual design — same Jobs + Ive lens; they catch the misplaced arrow and inconsistent spacing before users do |
product | Steve Jobs + Jony Ive | Product review benefits from Jobs' ruthless simplicity + Ive's craft discipline |
custom-ux | Steve Jobs + Jony Ive | User-defined UX rubric; defaults to the design-excellence lens |
spec-coherence | Jeff Dean | Architecture review needs systems-design rigor at scale; Jeff Dean's lens catches the O(n²) in the happy-path spec |
architecture | Jeff Dean | Same lens as spec-coherence — distributed-systems traps + scale failure modes |
prompt-quality | Shreyas Doshi | Prompt quality = product quality; Doshi's discipline surfaces vague intent and missing success criteria |
prompt-sharpen | Shreyas Doshi | Sharpening a prompt is a product-spec act; same lens as prompt-quality |
hallucination | none | Factual grounding — expert taste doesn't help; a fabricated citation is wrong regardless of domain |
flattery | none | Sycophancy detection is structural — presence of specific marker phrases, not aesthetic judgment |
patent-safety | none | §102 prior-art risk is legal/technical fact; stylistic persona adds noise |
calibration-scan | none | Same as flattery — detecting presence of specific marker phrases |
hallucination-preflight | none | Risk classification — factual, not stylistic |
t0t2-jury | none | Lightweight pass/fail for internal reversible artifacts |
custom | none | User controls the rubric; pass --persona explicitly if a lens is needed |
The persona field in output JSON
The cascade result now includes a top-level persona field showing which
persona (if any) was active:
{
"version": "3.4.0",
"rubric": "spec-coherence",
"persona": "Jeff Dean",
...
}
null when no persona was active (factual rubrics + custom with no flag).
Auth model — OAuth local CLIs (v3.3.0+)
As of v3.3.0, both jurors invoke OAuth-authenticated local CLIs over the
user's paid Pro subscriptions — no API keys required.
| Family | CLI | OAuth source | Pre-condition |
|---|
| Google | agy | Antigravity OAuth | Ultra entitlement |
| OpenAI | codex exec | codex login → ~/.codex/auth.json | ChatGPT Plus / Pro subscription |
The script strips OPENAI_API_KEY, GEMINI_API_KEY, GOOGLE_API_KEY, and
GOOGLE_GENAI_API_KEY from the subprocess env to force the CLIs onto the
OAuth path (otherwise they silently fall back to API billing).
API fallback (v3.2.0+) — opt-in, CLI-not-installed only
The cascade does NOT silently fall back to paid API billing. Fallback is
off by default and gated by a tightened approval semantic:
| CLI binary state | Approval flag set? | Behavior |
|---|
| Not on PATH (binary missing) | No | verdict=error, flags=[CLI_NOT_INSTALLED] |
| Not on PATH (binary missing) | Yes | API fallback fires; flags=[API_FALLBACK_USED, ...] |
| On PATH but auth fails / runtime error / non-zero exit | Either | verdict=error with the specific CLI error. No fallback. |
| On PATH and runs successfully | N/A | CLI used (existing OAuth path) |
The principle: API fallback exists only for the case where the user has
genuinely not installed the CLI yet (e.g., a fresh machine, or running
from CI). CLI installed but failing means the CLI setup is broken — fix
that, do not silently mask via paid API.
Approval can be set via:
- CLI flags on
bun run judge_panel.ts:
--api-fallback-approved (master gate — both lanes)--api-fallback-approved-gemini (Gemini lane only)--api-fallback-approved-openai (OpenAI lane only)
- Environment variables (programmatic / skill activation):
JUDGE_PANEL_API_FALLBACK_APPROVED=1JUDGE_PANEL_API_FALLBACK_APPROVED_GEMINI=1JUDGE_PANEL_API_FALLBACK_APPROVED_OPENAI=1
When approved, the relevant API key must also be set
(GEMINI_API_KEY / GOOGLE_API_KEY / OPENAI_API_KEY); otherwise the
juror returns flags=[API_FALLBACK_NO_KEY]. API fallback bills the
user's pay-per-token API account, not the flat-fee Pro subscription —
this is why approval is explicit and per-lane.
Models used (pinned, read from $CO_DIALECTIC_ENV — default: ~/.co-dialectic/.env)
| Stage | Model | Family | Role | Notes |
|---|
| Small-fish | Gemini 3.5 Flash (Low) | Google | Panel juror 1 | Antigravity OAuth / Ultra |
| Small-fish | gpt-5.4 | OpenAI | Panel juror 2 | See OAuth-tier caveat below |
| Tiebreaker (default) | Gemini 3.1 Pro (High) | Google | Final verdict | Cross-tier vs. Flash |
| Tiebreaker (alt) | gpt-5.4 | OpenAI | Final verdict | Pass via --tiebreaker gpt-5.4 |
OAuth-tier caveat (the small/big cascade collapses on the OpenAI lane).
ChatGPT-account-auth Codex rejects gpt-5.4-nano and other API-only
nano/mini-tier models with: "The 'gpt-5.4-nano' model is not supported when using Codex with a ChatGPT account." So the small-fish OpenAI juror
defaults to gpt-5.4 (the cheapest ChatGPT-Plus-permitted tier). On the
OpenAI lane, small=big in tier — but the cross-FAMILY cascade still holds
(Gemini Flash vs. GPT-5.4 are different training distributions).
The default tiebreaker is therefore Gemini 3.1 Pro (High) — crossing
the tier boundary inside Google AND remaining cross-family vs. both small
judges and the Claude author.
Override the small-OpenAI model via JUDGE_PANEL_OPENAI_OAUTH_MODEL
(in $CO_DIALECTIC_ENV or shell env). When OpenAI ships a ChatGPT-permitted
mini/nano tier, set the pin and the small-OpenAI cost drops automatically.
Cross-family guarantee: the two small judges are from different
families (Google + OpenAI). The tiebreaker is cross-family vs. author
(Claude) AND cross-family vs. whichever small judge triggered escalation.
No two judges in the cascade share a training distribution with the author.
Confidence + agreement rules
Each juror returns: {"verdict": "pass|fail|uncertain", "confidence": 0-100, "flags": [...]}.
Agreement check:
- Both small judges return
pass → AGREE
- Both small judges return
fail → AGREE
- One
pass, one fail → DISAGREE (always escalates)
- One or both
uncertain → DISAGREE (always escalates)
Confidence check (only meaningful when AGREE):
- Both confidences ≥ 80 → HIGH confidence (skip escalation)
- Either confidence < 80 → LOW confidence (escalate)
Escalation condition: DISAGREE OR (AGREE AND LOW_CONF). The
small-fish panel verdict stands only on AGREE AND HIGH_CONF.
Why ≥2 small-fish validation: a single small judge's verdict is one
sample — it could share a blind spot with the author by coincidence.
Requiring two independent cross-family small judges to converge is a
Rastogi-2023-style complementation move: each juror's weakness is paired
with the other juror's orthogonal training distribution, so agreement is
structurally informative.
Invocation — how the main agent runs it
The agent invokes the skill by calling the bundled TypeScript harness:
bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
--rubric "<rubric slug or inline rubric>" \
--artifact-file <path-to-artifact> \
[--persona "<name(s)>"]
Or by passing the artifact inline:
bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
--rubric hallucination \
--artifact "The response text to evaluate..."
With an explicit persona override (overrides the rubric default):
bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
--rubric spec-coherence \
--artifact-file spec.md \
--persona "Linus Torvalds"
Output is a single JSON object on stdout (schema below). No other stdout
writes — errors go to stderr. This makes the skill composable: any other
skill can call judge_panel.ts, parse stdout as JSON, and act on the
verdict without prompting the main LLM again.
Output JSON shape
{
"version": "3.4.0",
"rubric": "hallucination",
"cascade": {
"stage_1_small_fish": [
{
"model": "Gemini 3.5 Flash (Low)",
"family": "google",
"verdict": "pass",
"confidence": 88,
"flags": [],
"tokens_in": 412,
"tokens_out": 48,
"latency_ms": 1840
},
{
"model": "gpt-5.4",
"family": "openai",
"verdict": "pass",
"confidence": 85,
"flags": [],
"tokens_in": 412,
"tokens_out": 52,
"latency_ms": 8200
}
],
"agreement": "agree",
"confidence_tier": "high",
"escalated": false,
"stage_2_tiebreaker": null
},
"final_verdict": "pass",
"final_confidence": 86,
"cross_family": {
"distinct_families_returned": 2,
"degraded": false,
"down_lanes": []
},
"all_flags": [],
"cost_usd_estimate": 0.0041,
"cost_vs_naive_parallel_jury_ratio": 0.32
}
When escalation fires, stage_2_tiebreaker is populated and
final_verdict comes from the tiebreaker, weighted by whichever small
judge agreed with it.
Cross-family health contract
The output includes a top-level cross_family object:
{
"distinct_families_returned": 1,
"degraded": true,
"down_lanes": [
{
"family": "google",
"model": "Gemini 3.5 Flash (Low)",
"reason": "no JSON object in response"
}
]
}
distinct_families_returned counts distinct families that returned a real
pass, fail, or uncertain verdict across every juror that ran
(small-fish plus tiebreaker when fired). down_lanes lists every juror lane
that errored, timed out, or returned empty/no-JSON output. degraded is true
when fewer than two distinct families returned a real verdict.
Consumer contract: gates MUST read cross_family.degraded as the
authoritative machine-checkable signal. NEVER infer degradation from
all_flags[0]; all_flags is for human skim and is not a positional API. A
degraded cross-family T3 result should be surfaced clearly, e.g.
families: openai (google DOWN: no JSON object in response). The consumer
decides policy per stakes, including whether that degraded result is WARN or
BLOCK.
Rubric slugs bundled with the skill
The TypeScript harness ships with named rubrics. A caller passes the slug;
the harness substitutes the artifact into the rubric template and sends
to each judge.
| Slug | Domain | What it evaluates |
|---|
hallucination | Factual risk | Specificity, citation plausibility, confidence calibration, contradiction (mirrors codi-v3.1 spec §5.3) |
flattery | Sycophancy | High/medium/low markers from calibration-auditor spec |
spec-coherence | Artifact coherence | Blast-radius checks, claims-vs-implementation drift (P9) |
patent-safety | Patent disclosure | §102 prior art risk, claim/spec boundary leakage |
prompt-quality | Prompt rubric | Specificity, context, reasoning depth, intent clarity |
custom | Any | Pass --rubric-text <inline> with the full rubric text |
Callers can pass --rubric custom --rubric-text "..." for ad-hoc rubrics.
How to verify the skill works
Trigger command 1 (explicit): Say judge-panel check this: "The Eiffel Tower was built in 1923 and is located in Madrid."
Expected output: Both small judges flag factual errors; AGREE + HIGH
confidence on fail; no escalation; JSON verdict with two flags entries
(wrong year, wrong city).
Trigger command 2 (escalation): Say judge-panel check this: "FrugalGPT (Chen et al. 2023) achieves 30-98% cost savings via a cheap→expensive LLM cascade."
Expected output: Both small judges likely pass with varying
confidence (one may flag the savings range as too wide or the claim as
unsupported without citation). If disagreement → escalates. JSON verdict
with escalated: true.
Trigger command 3 (silent mode, as another skill):
bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
--rubric hallucination --artifact "..." --silent
Expected output: Pure JSON on stdout, nothing on stderr unless an
error fires.
Failure modes:
- Any judge call times out (> 30s) → mark that judge as
timeout, proceed
with remaining judges; if fewer than 2 small-fish returned, force escalate
- Rubric returns non-JSON from a judge → retry once; if second attempt
also non-JSON, mark that judge as
parse_error and force escalate
- All judges fail → return
final_verdict: "error" with stderr log
Cost discipline (P13 + Ground Zero 3D)
Optimal cost, not minimum cost — and on OAuth, "cost" is bounded by
the user's flat subscription fee. The cost_usd_estimate field still
reports what the run WOULD have cost on the pay-per-token API
(useful for cascade-vs-naive-jury comparison), but actual marginal cost
to the user is zero per call up to subscription quota.
OAuth tradeoffs the user is consciously accepting:
- Latency: local-CLI calls add ~3-10s of process startup per call
(codex spins up a session, agy starts an Antigravity run). Wall-clock
per cascade is ~10-20s vs. ~2-5s for the API path. Cross-family
verification is still in P11 / P21 budget.
- Subscription rate-limits: ChatGPT Plus and Gemini Pro have
per-day or per-hour usage caps. A judge-panel campaign that fires
the cascade on hundreds of artifacts in a session can exhaust the
cap. If a juror returns
verdict="error" with rate_limit in the
message, throttle the cascade or fall back to a different rubric.
- Tier collapse on OpenAI lane (see Auth model section above).
If escalation fires on every run, the cost advantage collapses. If
the eval harness reports escalation_rate > 50% over 20 runs, the
confidence threshold (currently 80) is too strict — loosen it, or the
rubrics are ambiguous — sharpen them. This is P14 (self-evolution)
applied to the cascade.
Auto-fire trigger (Protocol 8 — co-dialectic v4.1+)
Protocol 8 (Auto-Verify by Stakes) dispatches to this skill automatically
at T3 and T4. No user command required.
| Tier | Rubric selected | Escalation threshold | What user sees |
|---|
| T3 (important-decision/hard-to-undo) | spec-coherence for architectural/spec artifacts; hallucination for factual artifacts — caller picks based on artifact shape | Standard (agree+high-conf skips escalation) | ✓ reviewed by 2 models on pass; ⚠ review flagged on fail |
| T4 (irreversible/external-facing) | hallucination for general claims; biographical claims routed to career-os.bio-claim-verifier first (fallback to hallucination until that skill ships) | Standard | Result feeds into RED preflight summary; user must type 'send' to confirm |
Integration contract: Protocol 8 invokes judge-panel at T3 via the
fish-swarm dispatcher (same harness; fish-swarm rubric routes the T3 call
through the cascade). At T4, invoked directly for post-flight scoring.
FAIL-HARD (T3 dispatch): If zero fish are reachable when Protocol 8
triggers T3 auto-verify, the response is BLOCKED per the FAIL-HARD
invariant. Protocol 8 surfaces the fish-school remediation block. It does
NOT silently absorb the T3 cascade into Claude's own context — same-model
self-review defeats the cross-family guarantee this skill exists to enforce.
Cost discipline for auto-fire: Protocol 8 only reaches T3 on artifacts
where the classifier reaches HIGH or MEDIUM confidence that the artifact is
architectural/load-bearing/hard-to-undo. The vast majority of conversational
turns are T0/T1 and never reach judge-panel. Expected cascade rate: T3 fires
~5-15% of turns; T4 fires ~1-5%. Token budget is consistent with OPERATIONAL
DISCIPLINE right-sizing.
Agent-teams quality gate (v4.22.0+)
When Claude Code agent teams are enabled (CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1),
the plugin's TaskCompleted hook (hooks/task-completed-judge-gate.ts) runs this
cascade on every teammate task completion — opt-in via CODI_TEAM_JUDGE_GATE=1.
- Rubric:
spec-coherence by default (override: CODI_TEAM_JUDGE_RUBRIC)
fail/uncertain verdict → task completion BLOCKED (hook exit 2); the
jurors' flags are delivered to the teammate as actionable feedback- FAIL-HARD when armed: if the cascade cannot run (CLIs missing/broken), the
task is blocked with remediation — an unreviewable task never completes as
if it were reviewed
- Tasks under 80 chars of content are skipped (nothing substantive to judge)
This makes the Independent Verification Gate structural for multi-agent work:
cross-family review fires on every task boundary, enforced by the harness, not
by agent discipline.
Relationship to other skills
- Upstream callers:
hallucination-detector (plugin #3),
calibration-auditor (plugin #8 — for external-verdict mode),
any skill that needs a cheap cross-family sanity check
- Independent Verification Gate: judge-panel is the default runtime
mechanism for satisfying the Gate on artifacts where full human +
model-diversity review is too slow or expensive
- Anti-pattern: do NOT call judge-panel from inside a Claude subagent
spawned by another Claude. Same-model subagent would pollute the
cross-family guarantee. Always invoke from the main agent or directly
from the Python harness
END JUDGE-PANEL
