derived-bibliography

name

derived-bibliography

description

Build, refresh, or promote rows from `docs/initialDocs/derivedBibliography.md` — the auto-generated catalogue of references cited by the papers under `literature/papers/`. Use when the user asks to "extract references", "build the derived bibliography", "refresh derived refs", "what does paper X cite", "promote this derived ref to the main bibliography and download it", or to scan/aggregate citations from the local PDFs. Also: every time a new paper folder lands on disk, this refresh has to run so the derived list stays current.

This skill captures the end-to-end workflow for the derived bibliography: references that the papers we already track cite, resolved into structured entries that share the exact column schema of docs/initialDocs/revisedBibliography.md so they can be promoted into the main inventory whenever you decide one is worth tracking.

Authoritative paths

• Main inventory (input — used for dedup): docs/initialDocs/revisedBibliography.md.
• Derived (output of this skill): docs/initialDocs/derivedBibliography.md.
• Per-paper extraction artefacts: literature/papers/<slug>/references_ids.json, literature/papers/<slug>/references_raw.txt, literature/papers/<slug>/references.bib.
• Resolution cache: scripts/bib_refs_cache.json.
• Folder layout / slug / notes.md rules: docs/initialDocs/bibliographySpec.md.

What the derived bibliography is

For every PDF under literature/papers/<slug>/paper.pdf:

1. We extract its References section (pdftotext + heuristic heading detection, including LaTeX R EFERENCES ligatures).
2. We pull every DOI and arXiv ID we can find in that section into <slug>/references_ids.json and the raw section text into <slug>/references_raw.txt.
3. We resolve those identifiers via Crossref (DOI → BibTeX + metadata) and arXiv (arXiv ID → BibTeX + metadata) into scripts/bib_refs_cache.json.
4. We write <slug>/references.bib — a per-paper BibTeX file with one @misc{...} entry per resolved citation.
5. We aggregate everything into docs/initialDocs/derivedBibliography.md, grouped by topic, sorted by citation count inside our corpus, and filtered to drop entries that are already in docs/initialDocs/revisedBibliography.md.

docs/initialDocs/derivedBibliography.md uses the same columns as the main inventory plus a leading # column with the citation count inside our corpus:

| # | Status | Title | Organization | Main link | GitHub | Literature dir |

To promote a row to the main inventory, drop the # cell and paste the rest of the row into the appropriate section of docs/initialDocs/revisedBibliography.md. Without that leading column, every other cell is identical to a main-inventory row and is processed by the rest of the bib_* pipeline as-is.

Tooling

• python3 scripts/bib_extract_refs.py [--pass1] [--pass2] [--refresh]
  • pass 1 — local extraction (no network). Writes references_ids.json and references_raw.txt per paper.
  • pass 2 — global Crossref + arXiv resolution. Updates scripts/bib_refs_cache.json and writes per-paper references.bib. Throttles: 50 ms / Crossref request, batched arXiv queries (HTTPS id_list=id1,id2,..., up to 100 per call).
  • --refresh ignores caches and re-does everything.
• python3 scripts/bib_derived.py
  • Aggregates the cache into docs/initialDocs/derivedBibliography.md.
  • Drops references whose DOI / arXiv ID / title overlaps with what is already tracked in docs/initialDocs/revisedBibliography.md or any literature/papers/<slug>/notes.md.
  • Topic taxonomy (A. DETR core lineage … Z. off-topic) lives at the top of the script — adjust there when the paper proposal changes.
  • Path caveat: the script currently hardcodes docs/bibliography.md and docs/derivedBibliography.md. When run against the current layout, ensure those paths point at docs/initialDocs/... (patch the script or symlink) before relying on the output.

How to operate

For "refresh the derived bibliography" requests (or as the trailing step of any workflow that creates a new paper folder):

1. python3 scripts/bib_extract_refs.py --pass1 — quick (~5 s); only touches papers without a references_ids.json yet. Idempotent.
2. python3 scripts/bib_extract_refs.py --pass2 — slower the first time (Crossref + arXiv lookups), but idempotent thanks to the cache. arXiv batching keeps a fresh full sweep under ~30 s.
3. python3 scripts/bib_derived.py — regenerates docs/initialDocs/derivedBibliography.md.
4. Report counts to the user: total derived rows, breakdown by topic bucket, top 10 most-cited derived references.

For "what does paper X cite" requests:

1. Read literature/papers/<slug>/references_ids.json for the structured list of DOIs / arXiv IDs.
2. For human-readable BibTeX, read literature/papers/<slug>/references.bib (one @misc per resolved citation; references that lack a DOI or arXiv ID stay in references_raw.txt).

For "promote this derived ref into the main bibliography and download it" requests:

1. Locate the target row in docs/initialDocs/derivedBibliography.md. It carries 7 cells (# | Status | Title | Organization | Main link | GitHub | Literature dir).
2. Drop the leading # cell. Paste the remaining 6 cells into the appropriate section of docs/initialDocs/revisedBibliography.md, leaving the Literature dir cell empty.
3. Pipe just that row's JSON through the standard download pipeline:

   python3 scripts/bib_parse.py docs/initialDocs/revisedBibliography.md \
     | jq '[.[] | select(.title == "<exact title>")][0]' \
     | python3 scripts/bib_init_paper.py
   

   bib_init_paper.py will fetch the PDF, clone any GitHub URL, write notes.md + citation.bib, and on success the caller updates the row's Literature dir cell + status to ✅.
4. After committing the new row, regenerate the derived bibliography (bib_extract_refs.py --pass1 → --pass2 → bib_derived.py) so the now-tracked reference is dropped from docs/initialDocs/derivedBibliography.md and the new paper's own citations enter the corpus.

For "extract refs from a brand new paper" requests:

1. Make sure the paper has been imported (it should already have a literature/papers/<slug>/paper.pdf).
2. Run python3 scripts/bib_extract_refs.py --pass1 — it skips papers already extracted, so this only touches the new one.
3. Run python3 scripts/bib_extract_refs.py --pass2 — incremental: only the new IDs hit the network.
4. Regenerate the derived bibliography with bib_derived.py.

Always run this skill after creating a new paper folder

Whenever the add-paper skill, the paper-organizer agent, the manual import path, or the bulk runner produce a new literature/papers/<slug>/paper.pdf, run the three-step refresh (pass1 → pass2 → bib_derived.py) before reporting back to the user.

The user has explicitly flagged this as a recurring requirement: the derived bibliography is a live view of the corpus and must not be allowed to drift while the corpus grows.

Skip the refresh only when nothing was actually committed (every row was a committed: false) — there are no new references to harvest.

Things to NOT do

• Do not edit docs/initialDocs/derivedBibliography.md by hand — it is regenerated from the cache + per-paper references_ids.json. Any manual edit will be overwritten on the next bib_derived.py run.
• Do not drop or invent BibTeX entries inside per-paper references.bib files — they are also auto-regenerated by bib_extract_refs.py's pass 2.
• Do not loosen the topic-keyword lists in bib_derived.py to chase off-topic citations into the main buckets. Off-topic refs (LLM-finance papers, plant-pest detection, EEG, …) are correctly grouped under Z. Off-topic / unclassified.
• Do not over-throttle arXiv. Single-ID lookups must respect the 3-second rate limit, but batched queries (id_list=id1,id2,..., up to 100 per request) finish a full corpus in a few calls.
• Do not try to scrape MDPI for citation metadata from this server's IP — it is Akamai-blocked. Crossref already provides DOI metadata for every MDPI article in the cache.

Reporting back

When a refresh completes, give the user:

• the size of scripts/bib_refs_cache.json (<doi-resolved>/<doi-total>, same for arXiv);
• the size of each topic bucket in docs/initialDocs/derivedBibliography.md;
• the top 10 most-cited derived references (read from the leading # column);
• which per-paper references.bib files are still empty so the user knows where the extractor failed (typically Chinese-language papers, references using only In <venue>, <year> style, or scanned PDFs).