| name | detect-hallucinations |
| description | Use this to detect when an LLM is making things up, so you can flag or block confident-but-wrong answers before users see them. Trigger on "detect hallucinations", "is the model making this up", "flag unreliable answers", "hallucination check", "confidence scoring for LLM output", or hardening a RAG/QA system. Pick a method that matches whether you have reference context or not. |
| license | CC0-1.0 |
Detect LLM hallucinations
Hallucination detection scores how likely an answer is fabricated, so you can flag, block, or ask-for-review before it reaches a user. There is no single detector; pick by whether you have a ground-truth/context to check against.
Choose a method by what you have
You have retrieved context (RAG / QA): check faithfulness / groundedness - is every claim in the answer supported by the retrieved context?
- NLI/entailment: run each answer sentence against the context with an entailment model; unsupported sentences are suspect.
- LLM-as-a-judge: ask a strong model "is this answer fully supported by the context? list any unsupported claims." Cheap, effective, calibrate it (see
add-llm-evals).
You have no reference (open-ended generation): use consistency across samples - a model that knows the answer says the same thing across re-samples; a hallucinating model varies.
- Self-consistency (SelfCheckGPT-style): sample the answer N times; measure agreement. High disagreement = likely hallucination.
- Semantic entropy: cluster the N samples by meaning (bidirectional entailment), then compute entropy over the semantic clusters. High semantic entropy = the model is uncertain about the fact, not just the wording. This is the current state of the art for reference-free detection.
Cheap signals to combine: token log-probs / low confidence, and "I don't know"-style hedging. Weak alone, useful as features.
See references/methods.md for how each method works, cost/latency tradeoffs, and which libraries implement them.
Wire it into production
- Run the detector inline on high-stakes answers (block/route to human on high hallucination score) or sampled on traffic (dashboard + alert).
- Emit the score as a span attribute so hallucination rate is trackable over time (see
instrument-llm-observability).
- For RAG, log which claims were unsupported so failures are diagnosable (ties to
monitor-rag-quality).
Calibrate (do not skip)
Any detector has false positives/negatives. Label ~50 answers (hallucinated vs not) and tune the threshold to your tolerance - blocking flow favors recall (catch more), UX flow favors precision (fewer false blocks). Re-check after model changes.
Anti-patterns
- Trusting a single LLM-as-judge call as ground truth without calibration.
- Using consistency methods on deterministic (temperature 0) output - you need sampling diversity for them to work.
- Checking only the final answer in RAG, never which retrieved claim was (un)supported.
- Treating low token-probability as proof of hallucination - it is a weak signal, not a verdict.
Grounding
Self-consistency: SelfCheckGPT, Manakul et al. 2023 (arXiv:2303.08896). Semantic entropy: Farquhar et al., Detecting hallucinations in large language models using semantic entropy, Nature 2024 (paper); cheaper variant, Semantic Entropy Probes (arXiv:2406.15927). LLM-as-a-judge: Zheng et al. 2023 (arXiv:2306.05685).
