citation-check

# Citation & Hallucination Checker v2 Verification tool with vision + web search. Validates every claim against authoritative sources or provided documents. Works with content in any language. **Design principle:** Deterministic verification. Same input → Same output. --- ## Two Verification Modes ### Mode 1: Search Verification (Default) - Searches web for authoritative sources - Validates citations actually exist - Checks if cited sources say what's claimed - Finds original data for statistics ### Mode 2: Doc-Only Verification - User provides source document(s) - EVERYTHING must trace to those docs - Flags anything that appears to come from external knowledge - Trigger: "only use this document" / "verify against the PDF only" / "don't search the web" --- ## Two-Pass Architecture **Critical:** Always use two separate passes. Never interleave extraction and verification. ### Pass 1: Extraction Only 1. Read entire document/slides/images 2. Extract ALL claims using the Claim Extraction Rules below 3. Output numbered list: `[claim_id] | [claim_text] | [claim_type] | [location]` 4. **NO verification in this pass** 5. Present extraction to user for confirmation before proceeding ### Pass 2: Verification Only 1. Take Pass 1 output as fixed input 2. Verify each claim_id in sequential order 3. **NO re-extraction allowed** — work only with Pass 1 claims 4. Apply Status Decision Tree to each claim 5. Generate final report This prevents "discovering new claims" mid-verification and ensures consistency. --- ## Claim Extraction Rules (Exhaustive) Extract ONLY these claim types. Apply rules strictly — no judgment calls. ### EXTRACT as claims: | Type | Pattern | Example | | --------------- | ---------------------------------------------------------- | ------------------------------------------------ | | **Statistic** | Any number with unit/context (%, $, count, ratio, decimal) | "92.3% accuracy", "$4.7B market" | | **Comparative** | X is [comparative] than Y | "3x faster than baseline" | | **Temporal** | Time-bound assertion | "In 2024, adoption reached..." | | **Attribution** | Claim tied to source | "According to WHO...", "Smith et al. found..." | | **Causal** | X causes/leads to/results in Y | "This reduces latency by..." | | **Existence** | Asserts something exists/is true | "There are 500M users", "The model supports..." | | **Ranking** | Position claims | "largest", "first", "top 3" | | **Quote** | Direct quotation | Any text in quotation marks attributed to source | ### DO NOT extract as claims: | Type | Example | Reason | | --------------------------------- | -------------------------------------------------- | ------------------------------- | | Definitions | "Machine learning is a subset of AI" | Definitional, not factual claim | | Opinions marked as such | "We believe...", "In our view..." | Explicitly subjective | | Hypotheticals | "If adoption continues...", "Could potentially..." | Speculative | | Questions | "What drives growth?" | Not an assertion | | Future predictions without source | "Will reach $10B by 2030" | Unless citing a forecast report | | Methodology descriptions | "We used PyTorch 2.0" | Process, not factual claim | | Acknowledgments | "Thanks to our collaborators" | Not verifiable | ### Extraction Output Format ``` [C01] | "Model achieves 96.555% accuracy on ImageNet" | Statistic | Slide 3, bullet 2 [C02] | "Outperforms GPT-4 by 12% on reasoning tasks" | Comparative | Slide 3, bullet 3 [C03] | "According to Chen et al. (2024), transformers scale linearly" | Attribution | Slide 5, para 1 [C04] | "Market size reached $4.7B in 2024" | Statistic + Temporal | Slide 7, chart title ``` --- ## Status Decision Tree Apply this tree to EVERY claim. Follow exactly — no shortcuts. ``` START │ ├─ Is this a CITATION claim (references a paper/report/source)? │ ├─ YES → Go to CITATION VALIDATION │ └─ NO → Go to STATISTIC/FACT VALIDATION │ │ CITATION VALIDATION │ ├─ Step 1: Does the cited source exist? │ │ Run ALL mandatory search queries (see Search Templates) │ │ │ ├─ NO → Status: "Citation Not Found" │ │ Issue: "Cannot locate [citation] in any database" │ │ STOP │ │ │ └─ YES → Step 2: Does source contain the claimed topic? │ │ │ ├─ NO → Status: "Misquoted" │ │ Issue: "Source exists but does not discuss [topic]" │ │ STOP │ │ │ └─ YES → Step 3: Does source support the exact claim? │ │ │ ├─ YES (exact match) → Status: "Verified" │ │ Confidence: "exact" │ │ │ ├─ YES (paraphrase, same meaning) → Status: "Verified" │ │ Confidence: "paraphrase" │ │ │ ├─ PARTIALLY (missing context) → Status: "Misleading" │ │ Issue: "Claim omits critical context: [what's missing]" │ │ │ └─ NO (contradicts) → Status: "Hallucination" │ Issue: "Source says [X], claim says [Y]" │ │ STATISTIC/FACT VALIDATION │ ├─ Step 1: Can you find an authoritative source? │ │ Run ALL mandatory search queries (see Search Templates) │ │ │ ├─ NO (no source found) → Status: "Unverified" │ │ Issue: "No authoritative source found" │ │ STOP │ │ │ └─ YES → Step 2: Do values match EXACTLY? │ │ │ ├─ YES → Status: "Verified" │ │ Confidence: "exact" │ │ STOP │ │ │ └─ NO → Status: "Numerical Error" │ Go to NUMERICAL ERROR DETAILS │ │ NUMERICAL ERROR DETAILS (Academic Precision Mode) │ ├─ Record: │ • Source value: [exact number from source] │ • Claimed value: [number in document being checked] │ • Deviation: [calculate exact difference] │ • Source location: [page, table, section] │ ├─ Classification: │ • ANY rounding → Numerical Error │ • ANY truncation → Numerical Error │ • Significant figures mismatch → Numerical Error │ • Unit mismatch → Numerical Error │ • Wrong direction (e.g., increase vs decrease) → Hallucination │ └─ Exception: If source ITSELF provides rounded figure • e.g., Source says "96.555% (approximately 97%)" • Then claiming "97%" → Verified (cite the approximation) ``` --- ## Numerical Precision Rules (Academic Standard) **Default mode: Strict academic precision. Exact numbers only.** | Rule | Source | Claim | Status | | -------------------- | ----------- | ----------- | ----------------- | | Exact match required | 96.555% | 96.555% | ✓ Verified | | Any rounding = error | 96.555% | 97% | ✗ Numerical Error | | Any rounding = error | 96.555% | 96.6% | ✗ Numerical Error | | Truncation = error | 96.555% | 96.5% | ✗ Numerical Error | | Sig figs must match | 0.834 | 0.83 | ✗ Numerical Error | | Units must match | 96.555% | 0.96555 | ✗ Numerical Error | | Direction matters | +12% growth | +15% growth | ✗ Hallucination | | Order of magnitude | $4.7B | $47B | ✗ Hallucination | ### Numerical Error Output Format ```markdown ### Numerical Error: [Claim ID] | Field | Value | |-------|-------| | Claim | "Model achieves 97% accuracy" | | Location | Slide 4, bullet 2 | | Source | Chen et al. (2024), Table 3, p.8 | | Source value | 96.555% | | Claimed value | 97% | | Deviation | +0.445% (rounded up) | | Status | Numerical Error | | Fix | Replace with: "Model achieves 96.555% accuracy" | ``` --- ## Confidence Classification | Level | Criteria | Use when | | ------------------ | ---------------------------------------------------------- | ----------------------------------- | | **exact** | ≥95% word overlap OR identical number with identical units | Direct quote, exact statistic | | **paraphrase** | Same fact, different words, no interpretation added | Restated finding | | **interpretation** | Inference drawn from source data | Calculated from source, synthesized | **Rule:** When uncertain between levels, use the MORE CONSERVATIVE option and flag for review. --- ## Mandatory Search Templates Run ALL applicable templates. Do not stop after first result. ### For Academic Citations ``` Query 1: "[first author last name] [year] [first 3 words of title]" Query 2: "[full paper title]" site:semanticscholar.org OR site:arxiv.org Query 3: "[first author] [year] [venue/journal name]" Query 4: "doi:[DOI]" (if DOI provided) Query 5: "arxiv:[arxiv_id]" (if arXiv ID provided) ``` ### For Statistics (Market size, usage numbers, etc.) ``` Query 1: "[exact number with unit] [topic] [year]" Query 2: "[topic] [year] statistics report site:statista.com" Query 3: "[topic] [year] report site:mckinsey.com OR site:gartner.com" Query 4: "[topic] market size [year] site:gov OR site:edu" Query 5: "[topic] [number] original source" ``` ### For Company/Product Claims ``` Query 1: "[company name] [claim topic] press release [year]" Query 2: site:[company domain] [claim topic] Query 3: "[company name] [metric] official announcement" Query 4: "[company name] [claim] SEC filing" (for public companies) ``` ### For Health/Medical Claims ``` Query 1: "[claim topic] site:who.int OR site:cdc.gov OR site:nih.gov" Query 2: "[claim] systematic review site:cochrane.org" Query 3: "[claim] meta-analysis pubmed" ``` ### For Government/Policy Claims ``` Query 1: "[policy/law name] site:gov" Query 2: "[statistic] official statistics [country]" Query 3: "[claim] [agency name] report" ``` --- ## Source Authority Hierarchy When multiple sources found, prefer in this order: | Rank | Source Type | Examples | | ---- | ----------------------------- | ------------------------------------------------- | | 1 | Primary source | Original study, official report, raw data | | 2 | Government/institutional | WHO, CDC, World Bank, national statistics offices | | 3 | Peer-reviewed publication | Nature, Science, IEEE, ACM | | 4 | Industry reports (named) | Gartner, McKinsey, Statista (with methodology) | | 5 | Reputable news citing primary | NYT, Reuters citing original source | | 6 | Secondary compilations | Wikipedia (check their sources) | **Rule:** If only Rank 5-6 sources found, status = "Unverified" with note "Only secondary sources found" --- ## Multi-Source Verification (Search Mode) A claim achieves "Verified" status only if: | Condition | Sources Required | | ---------------------- | --------------------------------------------------- | | Primary source found | 1 (if authoritative: .gov, peer-reviewed, official) | | Only secondary sources | ≥2 independent sources agreeing | | Sources conflict | Status = "Unverified", note the conflict | --- ## Tie-Breaker Rules When uncertain, apply these rules. No judgment calls. | Situation | Rule |