verify-bibliography

name

verify-bibliography

description

Verifies that bibliography references exist and have accurate metadata. Use when the user asks to "check references", "verify a bibliography", "validate citations", "check for hallucinated citations", "check a .bib file", or mentions auditing a BibTeX file for accuracy. Looks up each entry against Crossref and arXiv, then spawns web-search subagents for non-indexed sources, flagging missing DOIs, wrong years, title mismatches, venue mismatches, and broken URLs.

Verify Bibliography

Audit a bibliography file by looking up each entry against Crossref and arXiv, then dispatching web-search subagents for @misc, software, and preprint entries that aren't in either index. Useful for catching fabricated citations (common in LLM-generated drafts), transcription errors, and dead URLs.

When to use

Trigger when the user wants to verify the accuracy of references in a BibTeX (.bib) file. Typical phrasings: "check my references", "is this bibliography real", "verify these citations", "audit my .bib file".

Inputs

• A path to a .bib file. If the user does not provide one, ask before proceeding.
• Optionally, an email address for the Crossref "polite pool" User-Agent. If not provided, use a generic placeholder and mention it in the report.

Procedure

1. Set up

Confirm the .bib file path exists. If multiple .bib files are present in the project and the user did not specify, list them and ask which to check.

Ensure dependencies are installed using uv, not pip.

uv venv && VIRTUAL_ENV=.venv uv pip install bibtexparser requests

2. Primary checker (Crossref + arXiv)

Run the structured checker, which handles most entries with one API-based pass.

.venv/bin/python scripts/check_bib.py <path-to-bib> \
    --email [e-mail] --json report.json

The script does the following.

• Hits Crossref by DOI first, falls back to Crossref title+author search.
• For arXiv DOIs (10.48550/arxiv.*) that miss on Crossref, falls back to the arXiv API. Also does arXiv title search for preprints with no DOI.
• Compares the matched canonical record against the bib title (default 0.85 similarity), year (exact), venue (default 0.70 similarity), and first-author surname (default 0.60 similarity).
• Detects duplicates by DOI and by lowercased title.
• Uses per-host circuit breakers so a flaky upstream downgrades affected results to unverifiable instead of hanging or false-flagging.

Findings come back in five severity buckets.

• fabricated for entries verified not-found in any source.
• metadata for entries found but disagreeing on a field.
• missing_doi for entries matched by title only. Suggest adding the canonical DOI.
• unverifiable for entries not expected to be in Crossref or arXiv (such as @misc, @software, @techreport), or an arXiv lookup that the circuit breaker skipped.
• duplicate for entries with the same DOI or same title across two citekeys.

3. ArXiv recheck (if the API was rate-limited)

If the run summary says "arXiv API was unreachable" or "circuit-breaker tripped", the rate-limited search API was unavailable. Run the recheck script. It sidesteps the API by fetching public abstract pages (https://arxiv.org/abs/<id>), which are not rate-limited.

.venv/bin/python scripts/recheck_arxiv.py <path-to-bib> report.json \
    --out recheck_arxiv.json

This only handles entries with a parseable arXiv ID in their DOI field. Entries with no DOI need the next step.

4. Web-search subagents for the residue

After steps 2 and 3, the remaining unverifiable and any still-fabricated entries fall into a few buckets that web search handles well.

• @misc and @software entries with a url= field (CLIs, frameworks, GitHub repos, blog posts).
• @techreport entries from labs or organizations.
• Preprints with a title but no DOI or arXiv ID.

For each such entry, spawn a Sonnet subagent via the Agent tool, in parallel, to do an independent web verification. Use the general-purpose agent and pass model: "sonnet" to keep cost low for fan-out. Send all subagent calls in a single message so they run concurrently.

The prompt for each subagent must be self-contained (the agent has no context from this session). Include the full bib entry verbatim, what "verify" means for this entry type, and an instruction to report concisely in 120 words or fewer.

Subagent prompt template

You are verifying one BibTeX entry by web search. The entry has not been
found in Crossref or arXiv, so it's likely a software tool, technical report,
or non-indexed preprint.

Entry (verbatim):
<paste the @misc{...}/@techreport{...}/etc. block exactly as it appears>

Tasks:
1. Search the web for the title, author/org, and URL.
2. Confirm whether the cited work actually exists.
3. If a URL is in the entry, fetch it and confirm it resolves to the
   claimed work. Note if it 404s or redirects to something unrelated.
4. Check that the year and author/organization match what you find.
5. If the entry cites a software tool or product, confirm the canonical
   project URL/repo and note the latest version or release year.

Report in 120 words or fewer, with this structure.
- VERDICT: one of "confirmed", "metadata-mismatch", "broken-url",
           "not-found", or "ambiguous"
- EVIDENCE: 1-3 URLs you found.
- ISSUES (if any): specific field-by-field bib-vs-canonical differences.

Do not write to any files. Do not edit the bib. Just report.

Choosing what to send

• Send subagents only for entries that steps 2 and 3 couldn't classify. Don't re-verify entries the structured checker already confirmed.
• Skip subagents for entries that are obviously third-party software with a known canonical URL (such as https://github.com/anthropics/claude-code for anthropicClaudeCode2025). Just spot-check the URL with WebFetch.
• Cap fan-out at around 10 subagents per message. If there are more, batch.

After all subagents return, fold their verdicts into the final report under a new section called "Web-verified entries", one row per entry, with verdict and evidence link.

5. Present the consolidated report

Group findings by severity, fabricated first. For each entry surface the citekey, a one-line issue description, bib-value vs. canonical-value side by side for metadata mismatches, and a verification link (DOI, arXiv abstract URL, or web evidence URL).

For anything still flagged fabricated after web search, do not silently delete. Surface it and ask the user how to handle it.

If the user asks to fix the metadata errors, edit the .bib file in place using the canonical values. Preserve the original entry keys.

Notes on limitations

• Crossref coverage is strong for journal articles and conference papers, weaker for books, theses, preprints, and humanities sources. An entry not found in Crossref alone is not proof of fabrication. The arXiv and web-search passes exist precisely for that reason.
• The title-based fallback can produce false matches for short or generic titles. Treat title-only matches as suggestive rather than authoritative, and surface the matched DOI so the user can verify.
• Default thresholds are 0.85 for title, 0.70 for venue, 0.60 for first-author surname. Tune via --title-threshold, --venue-threshold, and --author-threshold.
• The arXiv API rate-limits aggressively (HTTP 429 after about 30s under load). The recheck script (recheck_arxiv.py) works around this by scraping abstract pages.
• Web-search subagents are non-deterministic. Re-running may yield slightly different evidence URLs. Treat their VERDICT as a strong signal but not as a 100% guarantee, and ask the user before deleting an entry on subagent evidence alone.

Files

• scripts/check_bib.py is the primary Crossref and arXiv checker. JSON output via --json. Exits non-zero if any fabricated, metadata, or duplicate finding remains (CI-friendly).
• scripts/recheck_arxiv.py re-verifies arXiv-DOI entries by scraping arxiv.org/abs/<id> abstract pages. Use when the arXiv API was rate-limited or unreachable.
• examples/references-atomisticskills.bib is a sample bib (131 entries).
• examples/REPORT.md is the sample full audit output.