check-paper-citations

name

check-paper-citations

description

Verify every citation in a paper is real, accurate, and correctly used. Downloads PDFs, reads abstracts, checks claims against source text, flags AI-hallucinated references, identifies uncited claims, and maintains structured annotations in the .bib file. Use when the user says "check citations", "verify references", "are these citations real", or when iter-refine-writing finds unannotated bib entries. Also use before submission as a final citation gate.

Citation Verification

Systematically verify every citation in a paper. This skill reads cited papers, confirms they exist, checks that claims match sources, and maintains a structured annotation system in the .bib file.

Do not perform any Git operation. Return citation and report changes to the caller.

First Step

Read the paper's .tex file and .bib file. Identify the docs/reference/ directory (or create it) for storing downloaded PDFs.

Mechanical pre-check (mandatory). Before any manual verification, run scripts/verify_bib.py (relative to this skill's directory) on the .bib file:

python3 /path/to/check-paper-citations/scripts/verify_bib.py references.bib

This script checks every bib entry through two paths:

1. API verification: queries arXiv, Crossref, DBLP, and Semantic Scholar to compare title, author count, year, and venue against external metadata. DBLP is preferred over arXiv metadata for published computer-science conference and journal entries.
2. URL verification (fallback): if no API returns a match, the script checks an explicit url or the https://doi.org/... resolver and verifies that the resource is reachable. Reachability establishes existence only, not metadata correctness; the manual passes below still verify the citation's claims and fields.

Every entry must pass at least one path. The script exits non-zero if:

• Any VERIFIED entry has a metadata mismatch (ERROR)
• Any entry has neither API match nor reachable URL (ERROR — add a url or doi field)
• Any entry has a broken/unreachable URL (ERROR)

The script distinguishes two severity levels. Errors make it exit non-zero; warnings are reported without blocking the run:

• ERROR (must fix, blocks exit): metadata mismatch, missing/broken URL, missing required fields, hallucinated entries, "and others" with <3 named authors
• WARN (should review): suspicious author names, workshop/conference venue mismatch, arXiv preprint that may have a published version, possible AI-generated title patterns

If a title API is rate-limited or unavailable and the entry has no stable URL/DOI fallback, the script reports an inconclusive ERROR rather than claiming that the citation is hallucinated. Retry later or add a stable identifier; do not downgrade a real citation based only on transient API failure.

Fix all errors before proceeding to manual passes. Never mark an entry as % VERIFIED: <date> without this script passing for that entry.

All citations must live in a standalone .bib file (e.g., references.bib), never as inline \bibitem{} in the .tex file. If the paper uses \bibitem{}, migrate all entries to a .bib file first and switch to \bibliography{references}.

Then run the three passes below in order. Do not skip passes.

Pass 1: Existence and Authenticity

For every entry in the .bib file:

1. Check the bib annotation block. Look for a comment block above each entry with these fields:

   % VERIFIED: <date>
   % REAL: yes|no|unverified
   % PDF: <path or "not available">
   % ABSTRACT: <1-2 sentence summary of what the paper actually says>
   % USED_FOR: <what this paper uses it for, in our context>
   

   If this block is missing or incomplete, the entry needs full verification.

2. Verify the paper exists and metadata is correct. For each unverified entry:

   • Search for the paper by title on the web (arxiv, ACM DL, IEEE, USENIX, Google Scholar)
   • Confirm ALL metadata fields: title, authors (full list, correct spelling), year, venue/booktitle, pages, DOI
   • If any metadata is wrong, fix it in the .bib entry
   • If the paper does not exist at all, flag as HALLUCINATED (must-fix)
   • If found, download the PDF to docs/reference/ (naming is not this skill's concern; keep whatever name is natural)
   • Do NOT remove unused bib entries. Mark them with % STATUS: unused but keep them. Only flag hallucinated entries for removal. The author may intend to use them later.

3. Read the abstract and introduction of each downloaded PDF. Write a 1-2 sentence summary of what the paper actually says (its main contribution, not what we want it to say).

4. Update the .bib annotation block with all fields filled in.

Pass 2: Claim-Citation Alignment

For every \cite{} in the .tex file:

1. Extract the surrounding claim. Read the sentence containing the citation and 1-2 surrounding sentences for context.

2. Check: does the cited paper support this claim?

   • Read the cited paper's abstract/intro (from PDF or annotation)
   • Does the paper actually say what we claim it says?
   • Common problems:
     • Mischaracterization: "Paper X shows Y" but Paper X actually shows Z
     • Over-attribution: claiming the paper says more than it does
     • Context mismatch: using a finding from one context to support a claim in a different context

3. If a claim is inaccurate:

   • Report as Must-fix with: the claim text, what the paper actually says, and a suggested rewrite
   • Edit the .tex to fix the claim (with Chinese translation if the paper uses bilingual format)
   • Update the USED_FOR field in the .bib annotation

Pass 2.5: Academic Integrity Checks

For every citation, check for these additional risks:

1. Preprint vs published version. If a bib entry cites an arXiv preprint, check whether a published (conference/journal) version now exists. If yes, update the bib entry to cite the published version with correct venue, pages, and DOI.

2. Secondary citations. If the paper attributes a claim to source A, but source A itself attributes it to source B, the paper should cite B directly (or cite both). Check key claims against their cited sources.

3. String citations in claims. \cite{a,b,c} chains are fine in related work surveys. Only flag when a specific factual claim in the paper's own argument rests on a chain without explaining which source supports it.

4. Self-citation and double-blind. Self-citation is encouraged — it shows research continuity. But under double-blind review:

   • Self-citations must use third person ("X et al." not "our prior work")
   • Flag if the .tex text says "our" or "we" when referring to a cited paper that shares authors
   • Do NOT flag self-citations as a problem — flag only identity leaks

5. Retracted papers. For each academic paper citation, check if it has been retracted (search for retraction notices).

6. Venue accuracy. Confirm that the venue/booktitle in the bib matches where the paper was actually published. Common errors: citing a workshop paper as a main conference paper, or vice versa.

7. Ghost citations. If a bib entry is cited (\cite{X} appears) but the surrounding text never explains WHY it is cited, flag as Should-fix. Every citation should earn its place.

Pass 3: Missing Citations

Scan the paper for citation gaps. This pass both checks existing text AND actively searches for missing citations.

3a. Factual claims without support

Statements like "X% of Y", "research shows that Z", or "agents invoke dozens of calls per minute" without a citation. Every factual claim that is not the paper's own measurement or common knowledge needs a cite.

3b. System, technique, dataset, and benchmark mentions

Every named system (Docker, gVisor, Firecracker, etc.) and technique (dual-LLM, lethal trifecta, least privilege) should cite the original source at first mention.

Datasets, benchmark suites, and traces are citations too: every dataset, benchmark, workload suite, or trace source used in the evaluation must cite its origin (paper, official page, or artifact repository) at first mention. An evaluation that names a benchmark or public trace without a cite is a Must-fix.

3c. Citation density by section

Check citation density per section. Flag sections where the ratio of claims to citations is low:

• Introduction — should have a cite every 2-3 sentences on average
• Background/Motivation — should be the MOST densely cited section. Nearly every factual sentence needs a cite. Flag if any paragraph has fewer than 1 cite per 2 sentences.
• Design — lower density OK (it's the paper's own contribution), but borrowed concepts still need cites
• Related Work — every paragraph should cite multiple papers

3d. Actively search and add missing citations

For each gap found, do not just flag — actively search for the right paper to cite:

• Use WebSearch / WebFetch to find papers
• Download the PDF to docs/reference/
• Add the bib entry with full annotation block (VERIFIED, REAL, PDF, ABSTRACT, USED_FOR)
• Edit the .tex to add the \cite{} at the right location (with Chinese translation if the paper uses bilingual format)

Common citations that background sections often need:

• Classic principles: least privilege (Saltzer & Schroeder 1975), capability-based security (Dennis & Van Horn 1966)
• Agent frameworks: Claude Code, OpenHands, SWE-Agent, Devin — cite the relevant ones
• Measurement claims: if you say "dozens of calls per minute", cite AgentCgroup or your own measurement
• Kernel mechanisms: cite original papers or documentation for Landlock, seccomp, namespaces, cgroups

Note: For a comprehensive related-work coverage audit (are there important papers in the field that the paper doesn't discuss at all?), use research-literature-novelty separately. This pass focuses on citation gaps in the existing text.

Bib Annotation Format

Every bib entry must have this comment block immediately above it:

% VERIFIED: 2026-07-07
% REAL: yes
% PDF: docs/reference/agentcgroup.pdf
% ABSTRACT: Characterizes OS-level resource dynamics of AI coding agents
%   across 144 tasks. Finds container-level resource controls are
%   mismatched with tool-call-level dynamics (granularity, responsiveness,
%   adaptability mismatches).
% USED_FOR: Motivation for per-tool-call granularity (resource dimension)
@misc{agentcgroup,
  title={...},
  ...
}

Fields:

• VERIFIED: Date of last verification (YYYY-MM-DD)
• REAL: yes (confirmed real), no (hallucinated, must remove), unverified
• PDF: Path to downloaded PDF, or "not available" if not downloadable
• ABSTRACT: Factual summary of what the paper says (2-3 lines max)
• USED_FOR: How our paper uses this citation (1 line)

No Standalone Verification Ledger

The bib annotation blocks are the single source of truth for verification state. Do NOT create or maintain docs/citation-verification.md or any other standalone citation ledger, database, or report file — it duplicates the annotations and drifts. When a quick overview is needed, generate it on the fly from the bib file (e.g. grep -c '% REAL: yes' references.bib, or list entries missing annotation blocks) and report it inline.

Output

After all three passes, report:

• Number of citations verified
• Number of hallucinated citations (must-fix)
• Number of inaccurate claims fixed
• Number of missing citations added
• Any entries that could not be verified (PDF not available, etc.)

When Called from iter-refine-writing

When triggered as a gate check (not a full run):

• Only check whether all .bib entries have complete annotation blocks
• If any annotation is missing or has REAL: unverified, trigger a full run of this skill
• If all annotations are present and REAL: yes, just check for missing citations in the .tex (Pass 3 only)