zotero-bridge

name	zotero-bridge
description	Sync and search a Zotero library locally. Multi-level paper processing (metadata → abstract → full-text → LLM summary). Agent prioritizes Zotero literature in scientific discussions while supplementing with web search. Use when discussing scientific topics, looking up papers, managing references, or needing to consult your personal research library.

Zotero Bridge

Sync your Zotero library into a local searchable knowledge base, with multi-level paper processing.

Setup

1. Get your Zotero API key at https://www.zotero.org/settings/keys
2. Find your user ID at same page (Your userID for use in API calls)
3. Copy references/config-template.yaml to config.yaml and fill in credentials
4. Install deps: pip install -r requirements.txt

Usage

Sync Library

# Quick sync (metadata + abstracts)
python scripts/sync.py

# Deep sync (full-text + summaries)
python scripts/sync.py --depth 3

# Force full sync (ignore incremental state)
python scripts/sync.py --full

Search & Retrieve

# Keyword search across all processed content
python scripts/search.py "quantum error correction"

# RAG mode — returns context for injection into agent prompt
python scripts/search.py "surface codes" --mode rag --top-k 5

Process Papers

# Download PDF and convert to markdown (L2) — single paper
python scripts/ingest.py --key ABC123

# Batch: acquire PDFs for all papers without PDF
python scripts/ingest.py --batch
python scripts/ingest.py --batch --limit 10     # first 10 only

# Generate LLM summary (L3)
python scripts/summarize.py --key ABC123

# Batch summarize all L2 papers
python scripts/summarize.py --batch

PDF acquisition chain (configurable via pdf.sources in config.yaml):

1. Unpaywall — queries the Unpaywall API for legal open-access versions (no API key needed, email for rate limits)
2. Sci-Hub — tries mirrors in rotation (sci-hub.se, .st, .ru)

Configure pdf.unpaywall.email in config.yaml to avoid rate limiting.

Deep Search (Web + Zotero)

# Research a topic — combines Zotero + web search
python scripts/deep_search.py "CRISPR off-target prediction"

Processing Levels

Level	Flag	Content	Storage
L0	always	Metadata (title, authors, DOI, etc.)	SQLite
L1	default	Abstract text	SQLite + abstracts/
L2	--depth 2	Full text PDF → Markdown	fulltext/
L3	--depth 3	LLM summary + citation graph	summaries/ + SQLite

Fulltext Verification (L2+ quality control)

After PDF→Markdown conversion, ingest.py automatically runs verification + artifact cleaning:

Detector	What it catches	Example
Vertical-text artifact	Rotated PDF headers extracted as 1-char-per-line blocks	"Physics of Plasmas" → "P\nh\ny\ns\ni\nc\ns"
Orphaned page numbers	Isolated "42", "–42–" lines from PDF pagination	Page breaks leave bare numbers
Repeated headers/footers	Journal name, DOI, "© 2024" appearing every Nth line	AIP journal headers
Boilerplate blocks	Disclaimer, conflict-of-interest, data-availability statements	Author contributions, funding
Garbled symbol lines	High ratio of non-alphanumeric chars (OCR/encoding errors)	"▯▯▯ ◆◆◆"
Reference-section bleed	Bibliography content appearing before References heading	"[1] Smith, J. Phys..." in text body

Manual verification only:

python scripts/verify_fulltext.py --all --check          # report all papers
python scripts/verify_fulltext.py --key ABC123 --clean    # save cleaned copy
python scripts/verify_fulltext.py --key ABC123 --clean-in-place  # overwrite + update DB

Ingest automatically cleans in place after conversion. Use --no-verify to skip.

Agent Interaction Rules

🔴 IRON RULE: Processing Level Gate

Level	What you KNOW	What you may DISCUSS
L0	Title, authors, year, DOI, journal. Nothing else.	You may mention the paper EXISTS, but you MUST NOT discuss its content, methodology, data, or conclusions. You have NOT read this paper.
L1	L0 + abstract text.	You may reference the abstract's content. Clearly label it as "the abstract states…". Do NOT infer methods/results beyond what the abstract says.
L2	L1 + full text extracted as Markdown.	You may discuss the paper's content. Check <data_dir>/fulltext/<key>.md before discussing.
L3	L2 + LLM-generated structured summary.	Full discussion supported. AI Summary is LLM-generated — flag potential errors.

VIOLATION: Presenting your own physical derivation or general knowledge as if it came from an L0 paper is strictly forbidden. If you catch yourself doing this, stop and explicitly separate "what the paper says" from "what I derived independently."

Search now defaults to --level-min 1 (L1+ only). Use --include-l0 to see L0 papers — they carry a prominent ⚠️ warning.

Literature Retrieval Strategy

RAG-first, full-text on demand. Default to search.py --mode rag --top-k 5 for discussion — only retrieve the most relevant paragraphs, not the whole paper. This saves context tokens and is faster.

User intent	Retrieval method
"这个论文里XX怎么处理的" / specific mechanism question	search.py --mode rag (default)
"精读全文" / "逐段分析" / "给我完整摘要" / explicit deep-read request	read_file on the local fulltext .md
Multi-paper comparison	RAG each paper first, then full-text only on key sections

Rules

1. When user asks a scientific question, first run search.py <query> against local Zotero DB. RAG mode by default (--mode rag --top-k 5). L0 papers are excluded by default — use --include-l0 only when the user explicitly asks to see all papers including unparsed ones.
2. ALWAYS check the Processing Level in search results before discussing any paper. The level is shown prominently in RAG output. L0 papers carry a ⚠️ warning.
3. IRON RULE: Never present content as "from a paper" if the paper is L0. If you only have title/authors for an L0 paper, your physical analysis is YOUR OWN DERIVATION. Say so explicitly. Example: "Morrow (2014) exists on this topic but I have not parsed it (L0). The following analysis is my own derivation based on first principles."
4. When re-discussing a previously processed paper (L2+), check <data_dir>/fulltext/<key>.md and <data_dir>/storage/<key>.md — if L2 fulltext exists, prioritize this local record.
5. If relevant L1+ papers found: cite them as primary references. Use RAG-injected context for discussion.
6. Always assess whether additional web search is needed — Zotero is priority, not exclusive.
7. Suggest adding interesting web-found papers to user's Zotero library.
8. When PDF acquisition fails (ingest returns non-zero, or output shows "All sources exhausted"):
   • Stop immediately — do NOT silently fall back to abstract-only or web search
   • Ask the user to provide the PDF file directly, listing the paper title, first author, year, DOI, and Zotero key
   • Once the user provides the PDF, save it to <data_dir>/pdfs/<key>.pdf and re-run ingest.py --key <key> to complete L2 conversion

Dependencies

• pyzotero (Zotero Web API)
• markitdown (PDF → Markdown)
• requests (PDF downloads)
• Python stdlib sqlite3 + yaml