judge-panel

name	judge-panel
description	Cross-family cascade-then-jury review. Use when the user says "judge this", "review with a panel", "cross-family review", "jury beats judge", or needs independent verification of an AI-generated artifact (code, spec, patent, copy, decision). Runs ≥2 cheap cross-family small-fish judges first (Gemini-Flash + GPT-nano). If they agree with high confidence, verdict stands. If they disagree or confidence is low, escalates to one expensive cross-family tiebreaker. Returns verdict + confidence + which judges fired + token cost.
metadata	{"version":"3.4.0","author":"Anand Vallamsetla","tier":"core","plugin_number":4}

BEGIN JUDGE-PANEL

Judge Panel — Cascade-then-Jury Verification

Plugin #4, Core tier. Operationalizes the Defense-in-Depth Part 2 thesis ("jury beats judge"). Constitution anchor: Ground Zero — Independent Verification Gate + Model-Diversity sub-mandate. Also P0.5 (Boundary Self-Awareness) and P22 (Boundary-First Qualification).

Why this exists

A single frontier LLM cannot peer-review itself. Same-family reviewers share training distribution, RLHF, and characteristic failure modes — same-family review is a closed loop that validates what the author already believes. Cross-family review crosses the training-distribution boundary and lets new information enter. Empirically (Defense-in-Depth Part 2 incident, 2026-04-23), a single Gemini-2.5-Flash pass caught a class of drift that three same-family Claude reviewers had rationalized.

But parallel juries are expensive. Cascade-then-jury (FrugalGPT / Cascade Routing lineage) is the cheaper default: cheap small-fish first, escalate only on disagreement or low confidence. The literature:

• Verga et al. 2024 (PoLL) — panel of smaller diverse judges beats single GPT-4 at ~7× less cost
• Chen/Zaharia/Zou 2023 (FrugalGPT) — cheap→expensive cascade, 30-98% cost savings
• Dekoninck et al. ICML 2025 (Cascade Routing) — routing + cascading competitive with either alone

When to activate

Explicit invocation:

• judge-panel <artifact> / codi judge <artifact>
• "review with a panel"
• "cross-family review this"
• "jury beats judge check"
• "run the cascade on this"

Automatic activation (when another skill delegates):

• hallucination-detector (plugin #3) calls judge-panel for post-flight scoring
• calibration-auditor may call judge-panel for sycophancy scoring when the caller requests an external verdict
• Protocol 8 (Auto-Verify by Stakes, co-dialectic v4.1+) calls judge-panel automatically at T3 and T4 — see Auto-fire trigger section below

Silent mode: When called by another skill or by Protocol 8, emit only the JSON verdict. Skip the conversational framing.

The cascade

┌─────────────────────────────────────────────────────────────────┐
│  STAGE 1 — SMALL-FISH PANEL (parallel, always fires)             │
│  ┌───────────────────┐   ┌────────────────────┐                  │
│  │ Gemini Flash Lite │   │ GPT-5.4-nano       │                  │
│  │ (Google family)   │   │ (OpenAI family)    │                  │
│  └─────────┬─────────┘   └──────────┬─────────┘                  │
│            │                        │                             │
│            └─────────┬──────────────┘                             │
│                      ▼                                            │
│        ┌───────────────────────────┐                              │
│        │  Aggregate verdicts       │                              │
│        │  Compute agreement + conf │                              │
│        └───────┬───────────────────┘                              │
│                │                                                  │
│    ┌───────────┴───────────┐                                      │
│    ▼                       ▼                                      │
│  AGREE + HIGH CONF     DISAGREE or LOW CONF                       │
│  (skip escalation)     (escalate)                                 │
└────┼───────────────────────┼─────────────────────────────────────┘
     │                       │
     │                       ▼
     │       ┌──────────────────────────────────────────────┐
     │       │  STAGE 2 — BIG-FISH TIEBREAKER (only if needed) │
     │       │  ┌────────────────────┐                        │
     │       │  │ GPT-5.4 (default)  │   (or Gemini 3.1 Pro)  │
     │       │  │ cross-family vs.   │                        │
     │       │  │  author (Claude)   │                        │
     │       │  └──────────┬─────────┘                        │
     │       │             ▼                                  │
     │       │  Final verdict = tiebreaker-weighted           │
     │       └──────────────────────┬───────────────────────┘
     │                              │
     └──────────┬───────────────────┘
                ▼
        ┌──────────────────┐
        │  Return JSON     │
        │  verdict packet  │
        └──────────────────┘

Persona-driven judges (v3.4.0+)

Judges default to a generic evaluation lens. For rubrics where expert taste matters more than factual accuracy, the panel now injects a persona line into each judge's prompt:

"Judge as {persona(s)} — channel the top-0.001% standard in their domain; scrutinize as they would and catch the minute details they would catch."

Why personas. Apple didn't become Apple without obsession over every minute UX detail. A generic judge rubber-stamps; a Jobs + Ive judge catches the misplaced arrow, the inconsistent spacing, the CTA that creates micro-friction. The persona is a quality FLOOR, not a restriction — it raises scrutiny to the caliber the domain actually demands.

Persona is LAYERED ON TOP of cross-family diversity. The two small judges remain Gemini (Google family) + Codex (OpenAI family); each also adopts the same persona lens. Cross-family guarantee is unchanged.

CLI option

bun run judge_panel.ts --rubric <slug> --artifact "..." --persona "Steve Jobs + Jony Ive"

Also readable from environment:

JUDGE_PANEL_PERSONAS="Steve Jobs + Jony Ive" bun run judge_panel.ts ...

The --persona flag (or JUDGE_PANEL_PERSONAS env var) always overrides the rubric default. Pass an empty string or omit the flag to use the default.

Default persona map (rubric → persona)

Rubric	Default persona	Rationale
ux	Steve Jobs + Jony Ive	UX demands obsession over minute details; Jobs (product vision) + Ive (tactile/visual craft) together set the highest bar
visual	Steve Jobs + Jony Ive	Visual design — same Jobs + Ive lens; they catch the misplaced arrow and inconsistent spacing before users do
product	Steve Jobs + Jony Ive	Product review benefits from Jobs' ruthless simplicity + Ive's craft discipline
custom-ux	Steve Jobs + Jony Ive	User-defined UX rubric; defaults to the design-excellence lens
spec-coherence	Jeff Dean	Architecture review needs systems-design rigor at scale; Jeff Dean's lens catches the O(n²) in the happy-path spec
architecture	Jeff Dean	Same lens as spec-coherence — distributed-systems traps + scale failure modes
prompt-quality	Shreyas Doshi	Prompt quality = product quality; Doshi's discipline surfaces vague intent and missing success criteria
prompt-sharpen	Shreyas Doshi	Sharpening a prompt is a product-spec act; same lens as prompt-quality
hallucination	none	Factual grounding — expert taste doesn't help; a fabricated citation is wrong regardless of domain
flattery	none	Sycophancy detection is structural — presence of specific marker phrases, not aesthetic judgment
patent-safety	none	§102 prior-art risk is legal/technical fact; stylistic persona adds noise
calibration-scan	none	Same as flattery — detecting presence of specific marker phrases
hallucination-preflight	none	Risk classification — factual, not stylistic
t0t2-jury	none	Lightweight pass/fail for internal reversible artifacts
custom	none	User controls the rubric; pass --persona explicitly if a lens is needed

The persona field in output JSON

The cascade result now includes a top-level persona field showing which persona (if any) was active:

{
  "version": "3.4.0",
  "rubric": "spec-coherence",
  "persona": "Jeff Dean",
  ...
}

null when no persona was active (factual rubrics + custom with no flag).

Auth model — OAuth local CLIs (v3.3.0+)

As of v3.3.0, both jurors invoke OAuth-authenticated local CLIs over the user's paid Pro subscriptions — no API keys required.

Family	CLI	OAuth source	Pre-condition
Google	agy	Antigravity OAuth	Ultra entitlement
OpenAI	codex exec	codex login → ~/.codex/auth.json	ChatGPT Plus / Pro subscription

The script strips OPENAI_API_KEY, GEMINI_API_KEY, GOOGLE_API_KEY, and GOOGLE_GENAI_API_KEY from the subprocess env to force the CLIs onto the OAuth path (otherwise they silently fall back to API billing).

API fallback (v3.2.0+) — opt-in, CLI-not-installed only

The cascade does NOT silently fall back to paid API billing. Fallback is off by default and gated by a tightened approval semantic:

CLI binary state	Approval flag set?	Behavior
Not on PATH (binary missing)	No	verdict=error, flags=[CLI_NOT_INSTALLED]
Not on PATH (binary missing)	Yes	API fallback fires; flags=[API_FALLBACK_USED, ...]
On PATH but auth fails / runtime error / non-zero exit	Either	verdict=error with the specific CLI error. No fallback.
On PATH and runs successfully	N/A	CLI used (existing OAuth path)

The principle: API fallback exists only for the case where the user has genuinely not installed the CLI yet (e.g., a fresh machine, or running from CI). CLI installed but failing means the CLI setup is broken — fix that, do not silently mask via paid API.

Approval can be set via:

• CLI flags on bun run judge_panel.ts:
  • --api-fallback-approved (master gate — both lanes)
  • --api-fallback-approved-gemini (Gemini lane only)
  • --api-fallback-approved-openai (OpenAI lane only)
• Environment variables (programmatic / skill activation):
  • JUDGE_PANEL_API_FALLBACK_APPROVED=1
  • JUDGE_PANEL_API_FALLBACK_APPROVED_GEMINI=1
  • JUDGE_PANEL_API_FALLBACK_APPROVED_OPENAI=1

When approved, the relevant API key must also be set (GEMINI_API_KEY / GOOGLE_API_KEY / OPENAI_API_KEY); otherwise the juror returns flags=[API_FALLBACK_NO_KEY]. API fallback bills the user's pay-per-token API account, not the flat-fee Pro subscription — this is why approval is explicit and per-lane.

Models used (pinned, read from $CO_DIALECTIC_ENV — default: ~/.co-dialectic/.env)

Stage	Model	Family	Role	Notes
Small-fish	Gemini 3.5 Flash (Low)	Google	Panel juror 1	Antigravity OAuth / Ultra
Small-fish	gpt-5.4	OpenAI	Panel juror 2	See OAuth-tier caveat below
Tiebreaker (default)	Gemini 3.1 Pro (High)	Google	Final verdict	Cross-tier vs. Flash
Tiebreaker (alt)	gpt-5.4	OpenAI	Final verdict	Pass via --tiebreaker gpt-5.4

OAuth-tier caveat (the small/big cascade collapses on the OpenAI lane). ChatGPT-account-auth Codex rejects gpt-5.4-nano and other API-only nano/mini-tier models with: "The 'gpt-5.4-nano' model is not supported when using Codex with a ChatGPT account." So the small-fish OpenAI juror defaults to gpt-5.4 (the cheapest ChatGPT-Plus-permitted tier). On the OpenAI lane, small=big in tier — but the cross-FAMILY cascade still holds (Gemini Flash vs. GPT-5.4 are different training distributions). The default tiebreaker is therefore Gemini 3.1 Pro (High) — crossing the tier boundary inside Google AND remaining cross-family vs. both small judges and the Claude author.

Override the small-OpenAI model via JUDGE_PANEL_OPENAI_OAUTH_MODEL (in $CO_DIALECTIC_ENV or shell env). When OpenAI ships a ChatGPT-permitted mini/nano tier, set the pin and the small-OpenAI cost drops automatically.

Cross-family guarantee: the two small judges are from different families (Google + OpenAI). The tiebreaker is cross-family vs. author (Claude) AND cross-family vs. whichever small judge triggered escalation. No two judges in the cascade share a training distribution with the author.

Confidence + agreement rules

Each juror returns: {"verdict": "pass|fail|uncertain", "confidence": 0-100, "flags": [...]}.

Agreement check:

• Both small judges return pass → AGREE
• Both small judges return fail → AGREE
• One pass, one fail → DISAGREE (always escalates)
• One or both uncertain → DISAGREE (always escalates)

Confidence check (only meaningful when AGREE):

• Both confidences ≥ 80 → HIGH confidence (skip escalation)
• Either confidence < 80 → LOW confidence (escalate)

Escalation condition: DISAGREE OR (AGREE AND LOW_CONF). The small-fish panel verdict stands only on AGREE AND HIGH_CONF.

Why ≥2 small-fish validation: a single small judge's verdict is one sample — it could share a blind spot with the author by coincidence. Requiring two independent cross-family small judges to converge is a Rastogi-2023-style complementation move: each juror's weakness is paired with the other juror's orthogonal training distribution, so agreement is structurally informative.

Invocation — how the main agent runs it

The agent invokes the skill by calling the bundled TypeScript harness:

bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric "<rubric slug or inline rubric>" \
  --artifact-file <path-to-artifact> \
  [--persona "<name(s)>"]

Or by passing the artifact inline:

bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric hallucination \
  --artifact "The response text to evaluate..."

With an explicit persona override (overrides the rubric default):

bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric spec-coherence \
  --artifact-file spec.md \
  --persona "Linus Torvalds"

Output is a single JSON object on stdout (schema below). No other stdout writes — errors go to stderr. This makes the skill composable: any other skill can call judge_panel.ts, parse stdout as JSON, and act on the verdict without prompting the main LLM again.

Output JSON shape

{
  "version": "3.4.0",
  "rubric": "hallucination",
  "cascade": {
    "stage_1_small_fish": [
      {
        "model": "Gemini 3.5 Flash (Low)",
        "family": "google",
        "verdict": "pass",
        "confidence": 88,
        "flags": [],
        "tokens_in": 412,
        "tokens_out": 48,
        "latency_ms": 1840
      },
      {
        "model": "gpt-5.4",
        "family": "openai",
        "verdict": "pass",
        "confidence": 85,
        "flags": [],
        "tokens_in": 412,
        "tokens_out": 52,
        "latency_ms": 8200
      }
    ],
    "agreement": "agree",
    "confidence_tier": "high",
    "escalated": false,
    "stage_2_tiebreaker": null
  },
  "final_verdict": "pass",
  "final_confidence": 86,
  "cross_family": {
    "distinct_families_returned": 2,
    "degraded": false,
    "down_lanes": []
  },
  "all_flags": [],
  "cost_usd_estimate": 0.0041,
  "cost_vs_naive_parallel_jury_ratio": 0.32
}

When escalation fires, stage_2_tiebreaker is populated and final_verdict comes from the tiebreaker, weighted by whichever small judge agreed with it.

Cross-family health contract

The output includes a top-level cross_family object:

{
  "distinct_families_returned": 1,
  "degraded": true,
  "down_lanes": [
    {
      "family": "google",
      "model": "Gemini 3.5 Flash (Low)",
      "reason": "no JSON object in response"
    }
  ]
}

distinct_families_returned counts distinct families that returned a real pass, fail, or uncertain verdict across every juror that ran (small-fish plus tiebreaker when fired). down_lanes lists every juror lane that errored, timed out, or returned empty/no-JSON output. degraded is true when fewer than two distinct families returned a real verdict.

Consumer contract: gates MUST read cross_family.degraded as the authoritative machine-checkable signal. NEVER infer degradation from all_flags[0]; all_flags is for human skim and is not a positional API. A degraded cross-family T3 result should be surfaced clearly, e.g. families: openai (google DOWN: no JSON object in response). The consumer decides policy per stakes, including whether that degraded result is WARN or BLOCK.

Rubric slugs bundled with the skill

The TypeScript harness ships with named rubrics. A caller passes the slug; the harness substitutes the artifact into the rubric template and sends to each judge.

Slug	Domain	What it evaluates
hallucination	Factual risk	Specificity, citation plausibility, confidence calibration, contradiction (mirrors codi-v3.1 spec §5.3)
flattery	Sycophancy	High/medium/low markers from calibration-auditor spec
spec-coherence	Artifact coherence	Blast-radius checks, claims-vs-implementation drift (P9)
patent-safety	Patent disclosure	§102 prior art risk, claim/spec boundary leakage
prompt-quality	Prompt rubric	Specificity, context, reasoning depth, intent clarity
custom	Any	Pass --rubric-text <inline> with the full rubric text

Callers can pass --rubric custom --rubric-text "..." for ad-hoc rubrics.

How to verify the skill works

Trigger command 1 (explicit): Say judge-panel check this: "The Eiffel Tower was built in 1923 and is located in Madrid."

Expected output: Both small judges flag factual errors; AGREE + HIGH confidence on fail; no escalation; JSON verdict with two flags entries (wrong year, wrong city).

Trigger command 2 (escalation): Say judge-panel check this: "FrugalGPT (Chen et al. 2023) achieves 30-98% cost savings via a cheap→expensive LLM cascade."

Expected output: Both small judges likely pass with varying confidence (one may flag the savings range as too wide or the claim as unsupported without citation). If disagreement → escalates. JSON verdict with escalated: true.

Trigger command 3 (silent mode, as another skill):

bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric hallucination --artifact "..." --silent

Expected output: Pure JSON on stdout, nothing on stderr unless an error fires.

Failure modes:

• Any judge call times out (> 30s) → mark that judge as timeout, proceed with remaining judges; if fewer than 2 small-fish returned, force escalate
• Rubric returns non-JSON from a judge → retry once; if second attempt also non-JSON, mark that judge as parse_error and force escalate
• All judges fail → return final_verdict: "error" with stderr log

Cost discipline (P13 + Ground Zero 3D)

Optimal cost, not minimum cost — and on OAuth, "cost" is bounded by the user's flat subscription fee. The cost_usd_estimate field still reports what the run WOULD have cost on the pay-per-token API (useful for cascade-vs-naive-jury comparison), but actual marginal cost to the user is zero per call up to subscription quota.

OAuth tradeoffs the user is consciously accepting:

1. Latency: local-CLI calls add ~3-10s of process startup per call (codex spins up a session, agy starts an Antigravity run). Wall-clock per cascade is ~10-20s vs. ~2-5s for the API path. Cross-family verification is still in P11 / P21 budget.
2. Subscription rate-limits: ChatGPT Plus and Gemini Pro have per-day or per-hour usage caps. A judge-panel campaign that fires the cascade on hundreds of artifacts in a session can exhaust the cap. If a juror returns verdict="error" with rate_limit in the message, throttle the cascade or fall back to a different rubric.
3. Tier collapse on OpenAI lane (see Auth model section above).

If escalation fires on every run, the cost advantage collapses. If the eval harness reports escalation_rate > 50% over 20 runs, the confidence threshold (currently 80) is too strict — loosen it, or the rubrics are ambiguous — sharpen them. This is P14 (self-evolution) applied to the cascade.

Auto-fire trigger (Protocol 8 — co-dialectic v4.1+)

Protocol 8 (Auto-Verify by Stakes) dispatches to this skill automatically at T3 and T4. No user command required.

Tier	Rubric selected	Escalation threshold	What user sees
T3 (important-decision/hard-to-undo)	spec-coherence for architectural/spec artifacts; hallucination for factual artifacts — caller picks based on artifact shape	Standard (agree+high-conf skips escalation)	✓ reviewed by 2 models on pass; ⚠ review flagged on fail
T4 (irreversible/external-facing)	hallucination for general claims; biographical claims routed to career-os.bio-claim-verifier first (fallback to hallucination until that skill ships)	Standard	Result feeds into RED preflight summary; user must type 'send' to confirm

Integration contract: Protocol 8 invokes judge-panel at T3 via the fish-swarm dispatcher (same harness; fish-swarm rubric routes the T3 call through the cascade). At T4, invoked directly for post-flight scoring.

FAIL-HARD (T3 dispatch): If zero fish are reachable when Protocol 8 triggers T3 auto-verify, the response is BLOCKED per the FAIL-HARD invariant. Protocol 8 surfaces the fish-school remediation block. It does NOT silently absorb the T3 cascade into Claude's own context — same-model self-review defeats the cross-family guarantee this skill exists to enforce.

Cost discipline for auto-fire: Protocol 8 only reaches T3 on artifacts where the classifier reaches HIGH or MEDIUM confidence that the artifact is architectural/load-bearing/hard-to-undo. The vast majority of conversational turns are T0/T1 and never reach judge-panel. Expected cascade rate: T3 fires ~5-15% of turns; T4 fires ~1-5%. Token budget is consistent with OPERATIONAL DISCIPLINE right-sizing.

Agent-teams quality gate (v4.22.0+)

When Claude Code agent teams are enabled (CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1), the plugin's TaskCompleted hook (hooks/task-completed-judge-gate.ts) runs this cascade on every teammate task completion — opt-in via CODI_TEAM_JUDGE_GATE=1.

• Rubric: spec-coherence by default (override: CODI_TEAM_JUDGE_RUBRIC)
• fail/uncertain verdict → task completion BLOCKED (hook exit 2); the jurors' flags are delivered to the teammate as actionable feedback
• FAIL-HARD when armed: if the cascade cannot run (CLIs missing/broken), the task is blocked with remediation — an unreviewable task never completes as if it were reviewed
• Tasks under 80 chars of content are skipped (nothing substantive to judge)

This makes the Independent Verification Gate structural for multi-agent work: cross-family review fires on every task boundary, enforced by the harness, not by agent discipline.

Relationship to other skills

• Upstream callers: hallucination-detector (plugin #3), calibration-auditor (plugin #8 — for external-verdict mode), any skill that needs a cheap cross-family sanity check
• Independent Verification Gate: judge-panel is the default runtime mechanism for satisfying the Gate on artifacts where full human + model-diversity review is too slow or expensive
• Anti-pattern: do NOT call judge-panel from inside a Claude subagent spawned by another Claude. Same-model subagent would pollute the cross-family guarantee. Always invoke from the main agent or directly from the Python harness

END JUDGE-PANEL