hallucination-detector

name	hallucination-detector
description	Pre-flight risk-domain classification + post-flight hallucination scoring via the judge-panel cascade. Use when the user says "check for hallucinations", "hall on", "verify this response", "fact-check this", or an AI response makes specific factual / legal / medical / financial claims that must be trusted. Surfaces grounding suggestions before the response arrives and a 0-100 risk score after.
metadata	{"version":"3.2.0","author":"Anand Vallamsetla","tier":"core","plugin_number":3,"depends_on":["judge-panel"]}

BEGIN HALLUCINATION-DETECTOR

Hallucination Detector

Plugin #3, Core tier. Phase-1 MVP from BASE-PLUGINS-V3.md § 5. Constitution anchor: Ground Zero — Data Integrity; P13 (real-world stakes); P0.5 (Boundary Self-Awareness — training-cutoff boundary).

Why this exists

Every frontier LLM hallucinates. Same pattern-completion that produces creative leaps also fabricates citations, dates, URLs, API methods, case law. For HIGH-stakes domains (legal, medical, financial, factual), a silent fabrication is worse than no answer. Detection has two halves:

1. Pre-flight — classify the prompt by risk domain before the response arrives. Inject a grounding suggestion ("cite sources", "say when uncertain") for HIGH-risk domains.
2. Post-flight — score the response for fabrication markers via the cascade (judge-panel plugin #4). Cross-family review is the only way to catch model-specific blind spots — a Claude response scored by Claude is a closed loop (P0.5).

When to activate

Explicit:

• codi hall on / codi hall off — toggle persistent activation
• codi hall check — score the most recent response on demand
• hallucination-detector <text> — score arbitrary text
• fact-check this / verify this claim

Automatic (when active):

• Every prompt the user submits → pre-flight classification
• Every response from the primary model → post-flight scoring (delegated to judge-panel)

Default: OFF for explicit-trigger mode. When Protocol 8 (Auto-Verify) is active (default ON in co-dialectic v4.1+), hallucination-detector fires automatically at T2+ without the user typing codi hall on. See Auto-fire trigger section below.

First explicit codi hall on triggers a one-time confirmation that the user is OK paying the judge-panel token cost (~$0.004/check at small-fish tier).

Pre-flight: risk domain classification

Classify the prompt into one of these domains + risk tiers. Classification is LLM-free — use pattern matching on the prompt text first, fall back to the author model only if patterns don't fire.

Domain	Risk	Trigger patterns (examples)	Grounding suggestion
Factual	HIGH	"what year", "how many", "who invented", dates, statistics	"If uncertain about any fact, say so. Cite sources where possible."
Legal	HIGH	"is this enforceable", "contract", "liable", "sue", "case law"	"Do not invent case law or precedent. State uncertainty. Recommend consulting a lawyer."
Medical	HIGH	"side effects", "diagnose", "symptom", "treatment", "drug interaction"	"Cite medical literature. Flag uncertainty. Recommend professional consultation."
Financial	HIGH	"invest", "stock", "should I buy", "market outcome", "ROI of"	"Factual analysis only. Do not simulate certainty about future market outcomes."
Code	MEDIUM	"write a function", "how do I call", "API for", library + version	"If the API may have changed recently, flag potential staleness."
Citation	HIGH	"cite", "what paper", "author of", "publication year"	"Only cite sources you can verify. Do not fabricate DOIs, URLs, or author names."
Creative	LOW	"write a poem", "story about", "imagine"	No injection — creative tasks have no factual ground truth.
Summarization	LOW	"summarize", "tl;dr of", "key points from"	No injection — the source material is the ground truth.

Observer semantics (codi-v3 discipline): the detector does NOT modify the prompt. It surfaces the grounding suggestion as a sidecar:

[Hallucination Detector: HIGH-risk domain (legal). Consider adding:
 "Do not invent case law. State uncertainty where it exists."
 Add? [y] yes · [n] skip · [e] edit]

In Cruise mode, the suggestion is appended as a silent tip (no prompt mutation). In Drive mode, the user is paused for explicit choice.

Post-flight: cascade scoring via judge-panel

After the primary model responds, pipe the response through judge-panel with the hallucination rubric. Do not re-implement the scoring rubric here — judge-panel owns it.

Invocation:

bun run plugins/co-dialectic/skills/judge-panel/scripts/judge_panel.ts \
  --rubric hallucination \
  --artifact-file /tmp/last_response.txt \
  --silent

Map judge-panel verdict → hallucination score + label:

judge-panel final_verdict	Score range	Status line label	User signal
pass + conf ≥ 80	0-20	✓ Hall	Green — well-grounded
pass + conf < 80 OR uncertain	21-50	~ Hall	Amber — verify
fail (any conf)	51-100	⚠ Hall	Red — fabrication risk
error (all judges failed)	—	? Hall	Judge unavailable — fall back to self-review (weaker, disclose)

Exact score = 100 - final_confidence when final_verdict == "pass"; final_confidence when final_verdict == "fail"; 50 when uncertain or error.

Output behavior

When active, prepend the compact flag to every response:

[Hall: ✓ · 12/100 · 2 judges agree]

or on escalation:

[Hall: ⚠ · 78/100 · escalated to GPT-5.4 · 3 flags]

Fields: label, numeric score, judges fired (counts small-fish + tiebreaker if escalated), flag count.

Expandable detail: If the user says codi hall why or codi hall explain, surface the full flag list from judge-panel.all_flags plus the juror breakdown.

Cost discipline

Post-flight scoring costs tokens every time it fires. Two levers:

1. Small-fish-first cascade (inherited from judge-panel): ~$0.004 per check at the small tier. Escalation fires only ~10-30% of the time on a well-calibrated rubric.
2. Session budget guard: if total post-flight token cost exceeds $JUDGE_PANEL_SESSION_BUDGET (default $0.50/session), surface: "Hall detector has used ~$0.47 this session. Continue (y) / pause (n) / switch to pre-flight-only (p)?"

Auto-fire trigger (Protocol 8 — co-dialectic v4.1+)

Protocol 8 (Auto-Verify by Stakes) dispatches to this skill automatically. No user command required.

Tier	What fires	How
T2 (shared/multi-step-undo)	hallucination-preflight rubric via fish-swarm	Fish-swarm rubric hallucination-preflight runs BEFORE response generation. Classifies risk domain; injects grounding suggestion if HIGH-risk. Result maps to ✓ checked footer on the response.
T4 (irreversible/external-facing)	Full post-flight hallucination scoring	After artifact is drafted (but before T4 explicit-confirm gate), runs full post-flight scoring via judge-panel with hallucination rubric. Verdict feeds into RED preflight summary.

Integration contract: Protocol 8 invokes this skill's pre-flight logic at T2 using the fish-swarm harness (cheap, non-blocking). At T4, it invokes post-flight scoring via the judge-panel harness (full cascade). The skill does not need to know it was invoked by Protocol 8 vs. explicit codi hall on — the invocation path is identical. The only difference is that Protocol 8 routes through fish-swarm for T2 (lightweight path) and directly through judge-panel for T4 (full path).

When career-os.bio-claim-verifier is available (TODO integration point): At T4 with biographical claims, Protocol 8 first dispatches to career-os.bio-claim-verifier before falling back to this skill. This skill handles general factual claims; the bio-claim verifier handles Anand-specific canonical-source diffs. The two are complementary and both can fire on the same T4 artifact.

Interaction with other plugins

• Depends on: judge-panel (plugin #4) — the cascade executor
• Provides input to: beacons-emitter (plugin #5, future) — emits synthesized hallucination-rate-by-topic metrics (never raw text)
• Does NOT interact with: calibration-auditor — flattery and fabrication are orthogonal failure modes; running both in parallel is fine

How to verify

Trigger 1 (pre-flight): Submit the prompt "What year did the Eiffel Tower open in Madrid?".

Expected: Detector classifies as HIGH-risk factual → surfaces grounding suggestion before the response arrives.

Trigger 2 (post-flight, pass): The primary model responds "The Eiffel Tower opened in Paris in 1889.".

Expected: judge-panel returns pass + high confidence → [Hall: ✓ · ~10/100 · 2 judges agree].

Trigger 3 (post-flight, fail): Model responds "The Eiffel Tower opened in Madrid in 1923.".

Expected: judge-panel returns fail + high confidence → [Hall: ⚠ · ~85/100 · 2 judges agree · 2 flags] with flags covering wrong-city

• wrong-year.

Trigger 4 (escalation): Model emits a claim one small-fish judge cannot resolve without literature search → escalated: true, tiebreaker runs, final verdict reflects the bigger model's read.

Failure modes:

• Detector pattern-match fires wrong domain (false HIGH) → user quickly dismisses suggestion; track dismissal rate, tune patterns
• judge-panel unavailable (no gemini/codex CLIs on PATH) → fall back to ? Hall label and disclose "external judges unavailable — falling back to self-review, weaker signal"

END HALLUCINATION-DETECTOR