Search, download, convert, organize, and audit academic literature collections. Use when asked to find papers, build a literature library, add papers to references, download PDFs, convert papers to markdown, organize references by category, audit a reference collection, or collect code/dataset links for tools mentioned in papers.
Search, download, convert, organize, and audit academic literature collections. Use when asked to find papers, build a literature library, add papers to references, download PDFs, convert papers to markdown, organize references by category, audit a reference collection, or collect code/dataset links for tools mentioned in papers.
Literature Manager
Manage academic literature collections: search → download → convert → organize → verify.
Dependencies
pdftotext (poppler-utils) — PDF text extraction
curl — downloading
python3 — JSON processing in audit
file (coreutils) — PDF validation
uvx markitdown[pdf] (optional) — fallback PDF→MD converter (note: plain uvx markitdown does NOT work for PDFs — must use uvx markitdown[pdf])
Quick Start
# Download a single paper by DOI
bash scripts/download.sh "10.1038/s41592-024-02200-1" output_dir/
# Convert PDF to markdown
bash scripts/convert.sh paper.pdf output.md
# Verify a single PDF+MD pair
bash scripts/verify.sh paper.pdf paper.md
# Full audit of a references/ folder
bash scripts/audit.sh /path/to/references/
⚠️ Legal note: Sci-Hub may violate publisher terms of service or copyright law in some jurisdictions. Use only if you understand and accept the legal implications in your context.
If all sources fail (including Sci-Hub), flag as permanent paywall. Provide the user with the DOI and ask for manual download.
3. Convert
Run scripts/convert.sh <input.pdf> <output.md>. Uses pdftotext (reliable) with uvx markitdown[pdf] as fallback.
# Correct markitdown command for PDFs:
uvx markitdown[pdf] input.pdf > output.md
# ⚠️ The following will NOT work for PDFs (missing [pdf] extra):# uvx markitdown input.pdf
Prefer uvx markitdown[pdf] over pdftotext when full fidelity (tables, figures captions) matters.
4. Organize
Standard folder structure:
references/
├── README.md # Human index (summaries per category)
├── index.json # Machine index (structured metadata)
├── RESOURCES.md # Code repos + datasets
├── resources.json # Structured version
├── <category-1>/
│ ├── papers/ # PDFs
│ └── markdown/ # Converted text
└── <category-N>/
├── papers/
└── markdown/
Categories are user-defined. Number-prefix for sort order (e.g., 01-theoretical-frameworks/).
To build this mapping: cross-reference each paper's DOI in index.json against actual files on disk. Use find + Python to automate.
index.json Known Pitfalls
id: null corruption: If many entries have id=null and share the same pdf_path, the index was likely corrupted during a batch write. Rebuild from actual files on disk.
DOI errors: Verify DOIs resolve correctly — typos in DOI fields are common (e.g., wrong suffix digits). Always cross-check with publisher page.
Dead markdown_path: After restructuring folders, markdown_path in index.json often points to old locations. Use the mapping file above as the source of truth.
5. Verify
Run scripts/audit.sh <references_dir/> for full verification:
Every PDF is valid (file -b = PDF)
Every PDF title matches filename (pdftotext | head)
Every PDF has matching markdown (and vice versa)
index.json is valid, complete, paths exist, no duplicate IDs
README.md stats match actual counts
6. Collect Resources
For tool/method papers, find GitHub repos and public datasets. Store in RESOURCES.md + resources.json.
Sub-agent Strategy
For large batches, parallelize:
Download: 1 sub-agent per batch of ~5-8 papers
Organize: 1 sub-agent to build indexes
Verify: 1 independent sub-agent (never the same as organizer)
Always use a separate sub-agent for verification (QC should not self-grade).
⚠️ Sub-agent Rules (Learned from Practice)
One batch at a time — do not spawn multiple note-writing batches simultaneously; LLM rate limits will cause silent failures
Set a cron monitor whenever spawning long-running agents — agents can fail silently without triggering auto-announce; cron catches this
Cron monitor pattern:
1. Spawn agent(s)
2. Immediately set a cron job (every 10-15 min, isolated agentTurn)
→ Check if expected output files exist
→ Re-spawn failed agents
→ When all complete: announce + delete cron
3. After task finishes, confirm cron was removed
Adding Papers Incrementally
To add papers to an existing collection:
Download + convert new papers into correct category folder
