citation-audit

# Citation Audit > **Codex assurance:** base audit artifacts record > `review_independence: same-family` and `acceptance_status: provisional`. > Cross-family overlays or deterministic metadata checks may record accepted; > unavailable semantic review emits BLOCKED. Verify every `\cite{...}` in a paper against three independent layers: 1. **Existence** — the cited paper actually exists at the claimed arXiv ID / DOI / venue. 2. **Metadata correctness** — author names, year, venue, and title match canonical sources (DBLP, arXiv, ACL Anthology, Nature, OpenReview, etc.). 3. **Context appropriateness** — the cited paper actually supports the claim it is being used to support in the manuscript. This skill is the fourth layer of \aris{}'s evidence-and-claim assurance, complementing `experiment-audit` (code), `result-to-claim` (science verdict), and `paper-claim-audit` (numerical claims). Together they form a bottom-up integrity stack from raw evaluation code to manuscript bibliography. ## When to Use This Skill **Run before submission.** The right gating point is: - After `paper-write` has produced the LaTeX draft and bib file - After `paper-claim-audit` has verified numerical claims - Before final `paper-compile` for submission **Do not** run this on a half-written draft — most of the work is in cross-checking each `\cite` against context, which is wasted on placeholder text. ## What This Skill Catches The dangerous citation problems are **not** wildly fake citations — those are easy to spot. The dangerous ones are: - **Wrong-context citations**: real paper, but the cited claim is not what that paper actually establishes (e.g., citing Self-Refine to support "self-feedback produces correlated errors" — Self-Refine actually argues the opposite). - **Author hallucinations**: anonymous-author placeholders that slipped through, missing co-authors, wrong order. - **Title drift**: arXiv v1 vs v3 with different titles silently merged. - **Venue confusion**: arXiv preprint cited but the official venue is now CVPR/ICML/NeurIPS — using the wrong record. - **Year mismatch**: arXiv 2023 preprint with 2024 conference acceptance, year reported inconsistently. - **Phantom DOIs**: DOI looks real but does not resolve. - **Self-citation drift**: your own prior work cited with year off by one. ## Constants - **REVIEWER_MODEL = `gpt-6-astra`** — Fresh Codex reviewer with web access; same-family provisional in the base mirror. - **CONTEXT_POLICY = `fresh`** — Each audit run uses a new reviewer thread (REVIEWER_BIAS_GUARD). Continue only with `send_input` when explicitly resuming the same audit. - **WEB_SEARCH = required** — The reviewer must perform real web/DBLP/arXiv lookups, not pattern-match from memory. - **OUTPUT = `CITATION_AUDIT.md`** — Human-readable per-entry verdict report. - **STATE = `CITATION_AUDIT.json`** — Machine-readable verdict ledger consumable by downstream tools. - **SOFT_ONLY = `false`** — When true (set via `— soft-only` / `— soft_only` flag), the audit runs all three layers normally but **forbids any `.bib` file mutation**. Findings that would otherwise mutate the bib (FIX / REPLACE / REMOVE) are translated into per-occurrence sentence-rewrite proposals against the citing `*.tex` files. Used by `/resubmit-pipeline` Phase 1 to honor the user's hard "freeze the bib" constraint. - **RENDER_HTML = true** — When `true` (default), auto-render `CITATION_AUDIT.md` to HTML after writing the report. Uses **full review gate** (audit-class artifact). Set `false` to skip, or pass `— render html: false`. **Non-blocking**: failures don't invalidate the audit verdict. ## Workflow ### Step 1: Discover bib file and section files Locate: - `references.bib` (or `paper.bib` / similar) under the paper directory - All `*.tex` files containing `\cite{...}` calls (typically `sec/` or `sections/`) If multiple bib files exist, audit each separately. ### Step 2: Extract all (cite-key, context) pairs For each `\cite{key1,key2,...}` invocation in the paper: - Record the cite key - Record the file + line number - Record the surrounding sentence (≥ 1 full sentence around the cite, for context check) Output a flat list of `(key, file, line, surrounding_sentence)` tuples. Also build the inverse: for each bib entry, the list of all places it is cited. Define two protocol sets used throughout the rest of the workflow: `cited_keys` is the set of unique cite keys appearing in any `\cite{...}` invocation across the audited `*.tex` files (de-duplicated), and `bib_keys` is the set of keys parsed from the audited bib file(s). `cited_keys` drives Step 3 (audit only cited entries); `bib_keys \ cited_keys` is the uncited residual surfaced by the `--uncited` opt-in. If the user passed `--uncited`, also compute the set difference `bib_keys \ cited_keys` here and stash it for use in Steps 5 and the JSON aggregation; see "Uncited Entry Detection (opt-in)" below for the protocol. The set-diff is a string operation only and does not consume reviewer budget. Save the extracted contexts to `paper/.aris/citation-audit/contexts.txt` so the reviewer can read it directly. Use the paper-dir-relative path `.aris/citation-audit/contexts.txt` when recording the file in `audited_input_hashes`; do not stage under `/tmp` or other transient locations that the verifier cannot rehash later. ### Step 3: Send each entry to a fresh reviewer (same-family provisional by default) For each **cited** bib entry — i.e., each key in `cited_keys` with at least one extracted citation context — launch a fresh Codex reviewer agent. Do not reuse the same reviewer across entries. Do **not** spawn an agent for entries in `bib_keys \ cited_keys`; those are detect-only and surface only when `--uncited` is explicitly enabled (see "Uncited Entry Detection" below). ``` spawn_agent: model: gpt-6-astra reasoning_effort: xhigh message: | You are auditing a bibliographic entry. Use web/DBLP/arXiv search. ## Bib entry @article{key2024example, author = {...}, title = {...}, journal = {...}, year = {...}, ... } ## Where this entry is cited in the paper [paste extracted contexts] For this entry, verify: 1. EXISTENCE: does this paper exist at the claimed arXiv ID / DOI / venue? Output: YES / NO / UNCERTAIN, with the verifying URL. 2. METADATA: are author names, year, venue, title correct? For each, output: correct / wrong: should be ... / typo: ... 3. CONTEXT: for each use, does the cited paper actually support the surrounding claim? Output per-use: SUPPORTS / WEAK / WRONG, with one-sentence reasoning. VERDICT: KEEP / FIX / REPLACE / REMOVE - KEEP: entry is clean, all uses are appropriate - FIX: metadata needs correction; uses are appropriate - REPLACE: cite is wrong-context, find a different paper that actually supports the claim - REMOVE: entry is hallucinated or unsupportable Be honest. If you cannot verify online, say UNCERTAIN; do not guess. ``` Save the response to `.aris/traces/citation-audit/<date>_runNN/<key>.md` per the review-tracing protocol. ### Step 4: Aggregate verdicts Build `CITATION_AUDIT.json` following the schema defined in **"Submission Artifact Emission"** below (single authoritative schema for this file). Per-entry ledger data goes under `details.per_entry`, not under a top-level `entries` field. The top-level `verdict` is a single overall value (PASS / WARN / FAIL / NOT_APPLICABLE / BLOCKED / ERROR) derived from per-entry verdicts per the decision table in "Submission Artifact Emission"; the top-level `summary` is a one-line human-readable string. Concretely, `details` carries the per-entry ledger: ```json "details": { "total_entries": 29, "counts": { "KEEP": 11, "FIX": 14, "REPLACE": 3, "REMOVE": 1 }, "per_entry": [ { "key": "lu2024aiscientist", "verdict": "KEEP", "axis_failures": [], "uses": [ {"file": "sections/1.intro.tex", "line": 11, "verdict": "SUPPORTS"}, {"file": "sections/6.related.tex", "line": 8, "verdict": "SUPPORTS"} ] }, { "key": "madaan2023selfrefine", "verdict": "FIX", "axis_failures": ["CONTEXT"], "uses": [ {"file": "sections/2.overview.tex", "line": 42, "verdict": "WRONG", "note": "Self-Refine demonstrates iterative improvement, not correlated errors"}, {"file": "sections/6.related.tex", "line": 13, "verdict": "SUPPORTS"} ] } ] } ``` See "Submission Artifact Emission" for the full artifact (top-level fields `audit_skill`, `verdict`, `reason_code`, `summary`, `audited_input_hashes`, `trace_path`, `thread_id`, `reviewer_model`, `reviewer_reasoning`, `generated_at`, `details`). ### Step 5: Generate human-readable report Write `CITATION_AUDIT.md`: ```markdown # Citation Audit Report **Date**: 2026-04-19 **Bib file(s)**: references.bib **Total entries**: 29 ## Summary | Verdict | Count | |---------|------| | KEEP | 11 | | FIX | 14 | | REPLACE | 3 | | REMOVE | 1 | ## Priority Fixes (CRITICAL — apply before submission) ### REMOVE: anon2025placeholder - Author listed as "Anonymous" — canonical record exists with real authors and full title - Title is incomplete - ACTION: Replace key with the canonical citekey, update authors and title ### REPLACE-CONTEXT: example2023priorwork in sec/2.overview.tex:42 - Cited to support a specific technical claim - The cited paper actually demonstrates a different (related but distinct) phenomenon - ACTION: Rewrite the sentence; cite the prior work for what it actually establishes [... continues for each entry ...] ## All-Clean Entries (no action needed) [list of KEEP keys] ``` When `--uncited` is set, append the following section after "All-Clean Entries": ```markdown ## Uncited Entries (opt-in) The following bib entries are present in the audited bib file(s) but are not referenced by any `\cite{...}` in the paper body: - `author2010example` — suggestion: prune (uncited; no local evidence of intent) - `someone2015othercite` — suggestion: prune (uncited; no local evidence of intent) - `third2024todo` — suggestion: check (a `% TODO: cite third2024todo` comment was found in `sections/3.related.tex`) This section is detect-only; it does not change the top-level verdict. ``` ### Step 6: Apply fixes (interactive) For each FIX/REPLACE/REMOVE verdict, prompt the user: ``` Fix [key]? Change: <description of change> Files affected: references.bib + sec/X.tex:Y [Apply / Skip / Defer] ``` If `AUTO_APPLY = true`, apply all FIX-level changes (metadata corrections only). REPLACE and REMOVE always require human approval — they involve content changes. ### Step 7: Recompile and verify ```bash latexmk -C && latexmk -pdf -interaction=nonstopmode main.tex ``` Confirm: - No new `Citation undefined` warnings - No `Reference undefined` warnings - Page count unchanged or only minimally affected by metadata fixes ## Uncited Entry Detection (opt-in) **Default**: disabled. Existing users see no behavior change — only `\cite{...}` keys are audited, and bib entries with no `\cite` reference in the manuscript are silently ignored. **Opt-in**: pass `--uncited` on invocation. The skill then performs a set-diff after Step 2 and reports bib entries that appear in any audited bib file(s) but are not cited anywhere in the paper. Detect-only — uncited entries are **not** sent to the reviewer agent, so there is no extra reviewer/web-lookup cost. ### Why opt-in This skill's headline output is the three-axis audit on cited entries. Surfacing uncited bib entries by default would (a) change long-form output for every existing run, and (b) noise up the verdict for users who intentionally maintain a superset bib file (e.g., shared lab bib, in-progress section reorder where the cite has been removed but the entry intentionally retained). The flag preserves zero behavior change for existing callers. ### Effect when enabled When `--uncited` is set: - `CITATION_AUDIT.md` gains a `## Uncited Entries (opt-in)` section listing the keys with a one-line suggestion each: `prune` (entry is dead weight; recommend deleting) or `check` (entry might be intentional; flag for user review). Default suggestion is `prune`; only emit `check` when there is concrete local evidence (e.g., a TODO comment in a `.tex` file mentioning the key, or a recently removed `\cite` visible in `git diff`). Do not infer intent from the bib key string alone. - `CITATION_AUDIT.json` `details` gains an `uncited_entries` array; see "Submission Artifact Emission" below for the schema. - The top-level `verdict` is **unchanged**: uncited entries do not upgrade or downgrade the PASS / WARN / FAIL / etc. classification. The `reason_code` and `summary` are likewise unchanged in shape; only the `details.uncited_entries` field appears. - Verifier gates and downstream skills (`paper-writing` Phase 6, `verify_paper_audits.sh`) MUST NOT treat the presence of `uncited_entries` as a blocking signal. ### When opt-in is appropriate - Pre-submission cleanup (drop dead bib entries before sharing camera-ready ZIP). - Shared lab bib file where the paper uses a subset and the user wants to confirm what is in scope. - Recurring audits where the user has previously seen the uncited count and wants to track whether it changed. ### Fallback when bib enumeration fails If `--uncited` is enabled but full bib-key enumeration fails (e.g., malformed bib syntax that the parser cannot recover), the cited-entry audit must still proceed if at all possible. In that case: - Do **not** alter the top-level `verdict`, `reason_code`, or `summary`. - Emit `details.uncited_entries` as an empty array `[]`. - Add `details.uncited_entries_status: "unavailable"` plus a one-line note explaining why (e.g., `"bib parser could not enumerate keys; cited-entry audit completed normally"`). - Verifier gates and downstream skills MUST treat `unavailable` the same as the field being absent: not blocking. If the bib file cannot be read well enough to audit even the cited entries, fall back to the existing `BLOCKED` / `bib_unreadable` path defined in the verdict decision table; this is the same behavior as the no-flag default. ## Key Rules - **Fresh reviewer thread per audit run** — never reuse prior review context - **Web access required** — the reviewer must do real lookups, not memory pattern-match - **Wrong-context > metadata** — a real paper used to support a wrong claim is more dangerous than a typo in author name - **REPLACE/REMOVE require human approval** — never auto-modify content claims - **Always emit, never block** — this skill always writes `CITATION_AUDIT.json` with a verdict; the decision to block finalization lives in `paper-writing` Phase 6 + `verify_paper_audits.sh`, driven by the `assurance` level. See "Submission Artifact Emission" below. - **Run once per submission** — the audit is wall-clock expensive (web lookups for each entry); not for every save - **Uncited detection is opt-in only** — never auto-enable; never block on uncited entries; existing callers must observe identical output if they do not pass `--uncited` - **Under `--soft-only`, citation-audit emits text-rewrite proposals only; bib files are never mutated regardless of finding severity.** The audit semantics (existence + metadata + context) and the per-entry KEEP/FIX/REPLACE/REMOVE ledger are preserved verbatim; only the action layer is translated to per-occurrence sentence rewrites in the citing `*.tex` files. Refuse any downstream-proposed bib edit while `--soft-only` is set. ## Comparison with Other Audit Skills | Skill | What it audits | What it catches | |-------|---------------|-----------------| | `/experiment-audit` | Evaluation code | Fake ground truth, self-normalized scores, phantom results | | `/result-to-claim` | Result-to-claim mapping | Claims unsupported by evidence | | `/paper-claim-audit` | Numerical claims in manuscript | Number inflation, best-seed cherry-pick, config mismatch |