zotero-smart-ingest

name

zotero-smart-ingest

description

Smartly ingest journal articles into Zotero with PDF attachments. Searches arXiv + Crossref + Unpaywall + Google Scholar (careful scraping with cluster-based version enumeration and a free Semantic Scholar fallback when blocked), groups duplicate versions of the same article (arXiv preprint vs published vs OA copy), makes a human pick the best version, then creates the Zotero item and attaches the PDF via the Zotero API. Use when the user wants to add a paper to Zotero, find the best version of an article, or save a paper found on Google Scholar/arXiv into their library with its PDF.

Zotero Smart Ingest

Turn "I want this paper in Zotero" into a verified library entry with PDF, while a human chooses the best version when the article exists in several forms (arXiv preprint, published journal article, open-access copy).

When to use

• User gives a paper title, DOI, or arXiv ID and wants it in Zotero.
• User asks to find the best/published version of a paper they have as a preprint.
• User wants a paper they saw on Google Scholar or arXiv saved with its PDF.

Sources (all free, no API keys)

• arXiv API (export.arxiv.org) — preprints, by title or ID.
• Crossref (api.crossref.org) — published journal versions, DOI resolution.
• Google Scholar (scholar.py) — no public API, so: (1) scrape the search page with a real Chrome User-Agent + CONSENT cookie; (2) expand each result's cluster id to enumerate EVERY version (journal, preprint, working paper, author copy) — this is what makes "pick the best version" actually see all options; (3) if Scholar serves a block page, fall back to the free Semantic Scholar API (no key), which also merges versions and gives open-access PDFs. If both fail, capture Scholar in a real browser session and inject the results with --from-json (CDP path below).
• Unpaywall (api.unpaywall.org) — open-access PDF locations for DOIs. Requires an email (UNPAYWALL_EMAIL).
• Semantic Scholar (api.semanticscholar.org) — free fallback, no key needed; set S2_API_KEY (free key from semanticscholar.org/product/api) for higher rate limits. Retries once with backoff on 429.

Google Scholar browser/CDP path

Scholar blocks scripted scrapes from datacenter IPs. The pipeline escalates automatically: HTTP scrape blocked → Chrome CDP (if reachable) → Semantic Scholar API. You can also force it with --cdp.

The CDP mode drives a real Chrome — ideally the user's own (logged-in state, real fingerprint) — and gets the same candidates plus the [PDF] links Scholar shows (author copies, course sites) plus BibTeX per result via the cite popup. This is the strongest "try hard to get a PDF" path.

macOS launch recipe (Chrome 149+ refuses CDP on the default profile dir — must use a non-default --user-data-dir):

# Kill Chrome first, then launch the copy-of-profile approach:
cp -R "$HOME/Library/Application Support/Google/Chrome/Default" "$HOME/.hermes/chrome-data/Default"
cp "$HOME/Library/Application Support/Google/Chrome/Local State" "$HOME/.hermes/chrome-data/"
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --remote-debugging-port=9222 --user-data-dir="$HOME/.hermes/chrome-data" \
  --profile-directory="Default" --no-first-run
# Verify: curl http://127.0.0.1:9222/json/version

Then:

# Force CDP search (auto-fallback to Semantic Scholar if no Chrome is up)
uv run scripts/scholar.py "Time series momentum" --cdp

# Include BibTeX for the first 3 results (cite popup -> scholar.bib)
uv run scripts/scholar.py "Time series momentum" --cdp --bibtex 3

# The full ingest pipeline in CDP mode:
uv run scripts/smart_ingest.py "Time series momentum" --cdp

CDP endpoint: default http://127.0.0.1:9222, override with --cdp-url or the CDP_URL env var. The WebSocket/HTTP client is stdlib-only (no deps).

Fallback chain recap: scrape → CDP → Semantic Scholar → --from-json (an agent can dump a browser page and inject it as JSON).

Google PSE (Programmable Search Engine) does NOT work for Scholar — its index excludes scholar.google.com, so a CSE query returns nothing useful.

Human selection is the core step

The same article often has several versions. The skill groups candidates by DOI (fallback: normalized title) and prints a numbered menu — never auto-pick a version and ingest silently. In agent mode, present the menu to the user and pass their choice with --pick N.

Library dedup (default on)

Before the menu is shown, the pipeline fetches the Zotero library once and marks any candidate that already exists (matched by DOI, else normalized title — case-insensitive; attachments ignored). Existing items are reported with their zotero.org link, and if every found version is already in the library the run stops with "nothing to ingest" — no duplicates are ever created. Disable with --no-dedup. (Needs the API key; silently skips with a note when the key is missing.)

Setup

export ZOTERO_API_KEY=...        # web API: https://www.zotero.org/settings/keys
export ZOTERO_USER_ID=...        # numeric user ID (URL of your library)
export UNPAYWALL_EMAIL=[e-mail]   # needed for OA PDF lookup

Per convention, check 1Password for stored keys before assuming they're unavailable. Alternative backend: run with --local to use the Zotero desktop app's local API (port 23119, app must be running, user ID 0).

Usage

# Interactive (human picks from the menu):
uv run scripts/smart_ingest.py "Attention is all you need"
uv run scripts/smart_ingest.py 10.1016/j.jfineco.2011.11.003
uv run scripts/smart_ingest.py 1706.03762

# Agent-driven (inspect JSON, show menu to user, then pick):
uv run scripts/candidates.py --json "Time series momentum"
uv run scripts/smart_ingest.py "Time series momentum" --pick 1

# Safe checks before touching Zotero:
uv run scripts/smart_ingest.py 1706.03762 --pick 1 --dry-run

# Standalone pieces:
uv run scripts/candidates.py "query"          # menu output
uv run scripts/candidates.py --json "query"   # machine-readable
uv run scripts/ingest.py --title "..." --pdf paper.pdf --doi 10.xxxx/...

Scripts are stdlib-only Python — no dependencies, uv run or python3 both work.

Pipeline

1. Search arXiv + Crossref + Scholar (with S2 fallback) for the query (title / DOI / arXiv ID).
2. Enrich each candidate with an Unpaywall OA PDF URL (when DOI + email set).
3. Group candidates into distinct articles (versions of the same paper together).
4. Print the numbered menu; human picks the best version.
5. Download the PDF, verify %PDF- magic bytes.
6. Create the Zotero item (journalArticle or arXiv preprint) via the API.
7. Upload the PDF via the v3 three-step flow: create child attachment item → request upload authorization (md5/filename/filesize/mtime) → POST prefix+file+suffix to the storage URL → register the upload. (Local API fallback: legacy multipart with file + params.)
8. Read the item back and report the title + zotero:// link.

PDF hunting (when Unpaywall has no OA copy)

"Try hard for the PDF" playbook, in order:

1. Semantic Scholar openAccessPdf — https://api.semanticscholar.org/graph/v1/paper/DOI:<doi>?fields=openAccessPdf (worked for the Hansen JBES paper: green SSRN copy).
2. Scholar result PDF links (gs_or_ggsm / [PDF]) — author copies, course sites; best via --cdp.
3. Author homepage / working-paper series (RePEc/IDEAS pages list other versions).
4. Institutional repositories — e.g. UNC Carolina Digital Repository hosts postprints of highly-cited papers (live downloads sit behind a JS bot challenge; the Wayback snapshot of the /downloads/<id> URL bypasses it).
5. Wayback Machine — snapshot the known PDF URL or CDX-search web.stanford.edu/~peterhansen* style paths.
6. SSRN delivery (papers.ssrn.com/sol3/Delivery.cfm?abstractid=<id>) needs a session cookie from first visiting the abstract page — curl alone returns HTML; use a real browser (CDP) if needed.

JSTOR stable URLs map to the published DOI via Crossref (query.bibliographic + filter=container-title:); the DOI is what the pipeline wants as input.

Verification checklist

• Menu shows grouped versions, not a flat pile of duplicates.
• Chosen candidate's PDF downloaded and passes the %PDF- check.
• Zotero API returned successful with an item key.
• Read-back confirms the title; attachment key returned.
• Report both web URL and zotero://select/items/<key>.

Pitfalls

• No email → no OA PDFs. Unpaywall silently returns nothing without UNPAYWALL_EMAIL.
• Scholar blocks scripted scrapes from datacenter IPs. The pipeline detects block pages ("unusual traffic", captcha) and escalates: CDP browser if reachable, else Semantic Scholar — but version-cluster enumeration is then lost, so use --cdp for full version lists + PDF links + BibTeX. Keep scrape volume low (polite 1s pacing, expand at most 3 clusters) and never retry in a tight loop.
• CDP needs a running Chrome. --cdp with no Chrome on the endpoint prints a hint and falls back to Semantic Scholar. Chrome 149+ requires a non-default --user-data-dir (see launch recipe above).
• Google PSE is not a Scholar workaround. Programmable Search Engine's index excludes scholar.google.com; a CSE query returns nothing useful.
• Crossref rate limits. Always include mailto= in queries; keep --max modest.
• arXiv DOIs are DataCite (10.48550/arXiv.xxxx) — Unpaywall usually has no OA copy for those; the arXiv PDF is the attachment.
• Zotero v3 write API wants a bare JSON array ([{...}]), not {"items": [...]} — the wrapper is rejected with "Uploaded data must be a JSON array".
• Web API PDF upload is three-step (file_upload docs): attachment item → authorize (md5&filename&filesize&mtime, If-None-Match: *) → POST prefix+bytes+suffix to the returned URL (201) → register (upload=<uploadKey> → 204). The old single multipart POST fails with "POST data not provided" on the web API; it still works on the local API.
• Local API needs the app running (port 23119). Connection refused → use the web API instead.
• Never echo API keys; keep them in env vars or 1Password.
• Interactive prompt fails in non-tty — agents must pass --pick N (menu already printed to stdout).
• Zotero write token: plain item creation needs no write token; file uploads to the web API use the multipart form (no separate token needed).

Extending

• Add sources by writing a search_<source>(query, max_results) returning candidate dicts (title, authors, doi, pdf_url, version_hint, ...).
• Grouping key logic lives in group_candidates() — DOI first, title fallback.