bip-lit

name	bip-lit
description	Unified guidance for using the bipartite reference library CLI. Use when searching for papers, managing the library, or exploring literature via S2/ASTA.

Bip Reference Library

A CLI tool for managing academic references with local storage and external paper search.

Repository: Configured via nexus_path in ~/.config/bip/config.yml

Issues: https://github.com/matsen/bipartite/issues

⚠️ CRITICAL: Local-First, Paper-First Policy

ALWAYS search locally before using external APIs. NEVER call ASTA without explicit user permission.

When answering questions about papers, READ THE ACTUAL PAPER PDF. Do not rely on abstracts, S2 metadata, or ASTA when the paper is in the local library. Reach for the right reader for the job:

• Text and search → pdf-navigator MCP (MuPDF). Use search_pdf_text to jump to the relevant pages, then read_pdf_text / read_pdf_page. MuPDF keeps reading order across columns and inline math intact, and the text is searchable. This is the default.
• Figures, panels, rendered equations, tables → built-in Read with a narrow pages range. The built-in reader renders pages as images (token-heavy, not searchable), so use it only for the 1–3 pages that hold the visual you need — never the whole paper.

The nexus library has ~6000 papers. Most relevant papers are already there.

Required Search Order

1. Local search FIRST (always do this):

   bip search -a "LastName" "keyword" --human
   

   Always use --human — the default JSON output is verbose and easy to mis-scan.

2. If bip search fails (e.g., schema error), rebuild the database:

   bip rebuild
   

   Then retry the search.

3. If found locally, READ THE PAPER to answer the question:

   bip get <id> --human   # Get PDF path
   

   Then use mcp__pdf-navigator__search_pdf_text (jump to the page) and mcp__pdf-navigator__read_pdf_text to find the answer directly in the paper. For a figure or equation you need to see, use the built-in Read tool on a narrow pages range instead. The PDF base path is /Users/matsen/Google Drive/My Drive/Paperpile.

4. Only if not found locally AND user confirms, use ASTA:

   "I couldn't find that paper in the local library. Would you like me to search Semantic Scholar (ASTA)?"
   

DO NOT call bip asta, mcp__asta__*, or any external API without asking first.

DO NOT rely on abstracts or S2 metadata when you have access to the actual paper PDF.

Argument Handling

When invoked with arguments like /bip-lit find <query> or /bip-lit <query>:

1. Always search local library first with bip search "<query>" --human
2. If local search fails with an error, rebuild the database and retry
3. If found locally and answering a question, read the paper PDF using pdf-navigator tools
4. Only after exhausting local options, ask user if they want to search externally
5. For title searches, use the full title; for topic searches, use key terms

Proactive Concept Discovery

When discussing papers, always look for opportunities to create concept nodes:

• Papers that introduce new methods, models, or techniques (e.g., "categorical Jacobian")
• Papers that apply existing concepts in novel ways
• Connections between papers through shared concepts

Suggest creating concepts when you notice:

• A named method or algorithm being introduced
• A technique being reused across multiple papers
• A bridge between the user's work and external literature

Quick Reference

Task	Command
Search local library	bip search "query" --human (searches title, abstract, authors, notes)
Search by author	bip search -a "LastName" --human
Search by title	bip search -t "keywords" --human
Search by year	bip search --year 2024 --human
Search by note	bip search "AlphaSeq" --human (user notes from Paperpile are indexed)
Search by venue	bip search --venue "Nature" --human
Lookup by DOI	bip search --doi "10.1234/..." --human
Combined search	bip search "topic" -a "Author" --year 2020: --human
Semantic search	bip semantic "query"
Get paper details	bip get <id>
Export to BibTeX	bip export --bibtex <id>...
Append to .bib file	bip export --bibtex --append main.bib <id>...
Add paper to collection	bip s2 add DOI:10.1234/...
Find literature gaps	bip s2 gaps
Backfill missing PMCIDs from NCBI	bip ncbi backfill --dry-run
One-off PMCID lookup	bip ncbi pmcid DOI:10.1234/...
Fast paper search (external)	bip asta search "query"
Find text snippets	bip asta snippet "query"
Create concept	bip concept add <id> --name "Name"
Link paper to concept	bip edge add -s <paper> -t concept:<concept> -r <type> -m "summary"
Papers for concept	bip concept papers <concept-id>
Concepts for paper	bip paper concepts <paper-id>
Import projects from config	bip project import <file>
Import with concept edges	bip project import <file> --link-concepts

Search Strategy

Field-Specific Search Flags

Use --author and --year flags for precise filtering:

# Search by author (exact last name matching to avoid false positives)
bip search --author "Yu" --author "Bloom"    # Last names only
bip search -a "Tim Yu" -a "Bloom"            # First + last name
bip search -a "Yu, Timothy"                  # Last, First format

# Filter by year
bip search --year 2024           # exact year
bip search --year 2020:2024      # range (inclusive)
bip search --year 2022:          # 2022 and later
bip search --year :2020          # 2020 and earlier

# Combine keyword + filters
bip search "deep mutational scanning" --author "Bloom" --year 2023:

Multiple authors use AND logic - all must appear in the paper.

Author matching rules:

• Single word (e.g., -a "Yu") → exact last name match (won't match "Yujia")
• Two+ words (e.g., -a "Tim Yu") → exact last name + first name prefix
• Comma format (e.g., -a "Yu, Tim") → same as above

Query Formulation Tips

Keep queries short and specific - Long conceptual queries perform poorly:

• Bad: "correlation between BME criterion and Felsenstein likelihood around correct tree"
• Good: "BME Felsenstein likelihood phylogeny" or "Bruno WEIGHBOR likelihood"

Use --author flag instead of embedding names in query - Precise last name matching:

• Good: bip search -a "Yu" -a "Bloom" --year 2022: (exact last name match)
• Good: bip search -a "Tim Yu" -a "Bloom" (first prefix + exact last name)
• Bad: bip search "Tim Yu Bloom" (keyword search is substring-based)

Use specific method/algorithm names:

• "WEIGHBOR", "FASTME", "neighbor joining" rather than general descriptions

Systematic Search Workflow

For finding a specific paper or result:

1. Local library first (fastest, already curated):

   # Use flags for author/year filtering (most reliable)
   bip search -a "AuthorName" --year 2020: --human
   bip search "topic" -a "Author" --human
   
   # Or plain keyword search (use -a for authors when possible)
   bip search "distinctive title words" --human
   bip semantic "conceptual description"  # for topic-heavy queries
   

2. If found, read the paper to get authoritative answers:

   bip get <id> --human  # Get PDF path
   # Then use pdf-navigator to search/read the PDF
   

3. External keyword search (only if not found locally, with permission):

   bip asta search "AuthorName keyword1 keyword2" --limit 20 --human
   

4. Broaden if needed - remove author, try synonyms:

   bip asta search "minimum evolution likelihood" --human
   bip asta search "distance method maximum likelihood phylogeny" --human
   

5. Citation tracing - if you find a related paper, check what cites it:

   bip asta citations DOI:10.xxxx/yyyy --limit 50 --human
   

6. MCP tools directly - for more control over fields and filters:

   mcp__asta__search_papers_by_relevance with specific date ranges
   mcp__asta__get_citations with publication_date_range filter
   

Snippet Search Caveats

The bip asta snippet command can be slow and unreliable (timeouts are common). Alternatives:

• Use keyword search first to find candidate papers
• Use MCP mcp__asta__snippet_search directly with smaller limits
• If snippet times out, fall back to bip asta search

S2 vs ASTA vs NCBI: When to Use Which

S2 and ASTA both access Semantic Scholar; NCBI is a separate ID-resolution service:

Use Case	Command	Why
Add paper to collection	bip s2 add	Only S2 can modify local library
Find literature gaps	bip s2 gaps	Analyzes your collection
Explore without adding	bip asta *	Faster, read-only
Find text snippets in papers	bip asta snippet	Unique to ASTA
Fast paper search	bip asta search	10x faster rate limit
Get citations/references	Either S2 or ASTA	ASTA is faster
Backfill PMCIDs (e.g., for NIH RPPR)	bip ncbi backfill	NCBI is the canonical source; S2/ASTA do not return PMCIDs reliably

Rule of thumb: Use bip asta for exploration, bip s2 when you want to modify your library, bip ncbi for authoritative PMCID resolution. NCBI only knows PMCIDs for papers actually in PMC — absence is not a signal that the paper is missing.

See api-guide.md for detailed comparison.

Common Workflows

Find a Paper / Answer a Question About a Paper

1. Search local library first:

   bip search "Schmidler phylogenetics" --human
   # or for topic-heavy queries:
   bip semantic "importance sampling MCMC"
   

2. Get PDF path for a result:

   bip get <id> --human
   # pdf_path field + "/Users/matsen/Google Drive/My Drive/Paperpile"
   

3. Read the actual paper to answer questions:

   # Text: search to the relevant page (MuPDF — searchable, clean reading order)
   mcp__pdf-navigator__search_pdf_text(file_path, "phage display")
   mcp__pdf-navigator__read_pdf_text(file_path, 2, 3)
   

   For a figure, panel, or rendered equation you need to see, use the built-in Read tool on a narrow page range instead (it renders pages as images — token-heavy, so read only the page(s) with the visual):

   Read(file_path, pages="4")
   

   Always prefer reading the paper over relying on abstracts or external metadata.

4. Only if not in library, search externally (with user permission):

   bip asta search "phylogenetic inference"
   

Update Library from Paperpile

1. Export from Paperpile (JSON format) to ~/Downloads
2. Find the export file:

   ls -t ~/Downloads/Paperpile*.json | head -1
   

3. Import:

   bip import --format paperpile "<path>"
   

4. Rebuild the search index:

   bip rebuild
   

5. Optionally delete the export file after confirming success

Explore Literature

1. Search by topic:

   bip asta search "variational inference phylogenetics" --limit 20
   

2. Find specific text passages:

   bip asta snippet "Bayesian phylogenetic inference"
   

3. Trace citations:

   bip asta citations DOI:10.1093/sysbio/syy032
   bip asta references DOI:10.1093/sysbio/syy032
   

4. Add interesting papers to your collection:

   bip s2 add DOI:10.1093/sysbio/syy032
   

See workflows.md for detailed workflow instructions.

Output Format

All commands output JSON by default. Add --human for readable format:

bip asta search "phylogenetics" --human
bip s2 lookup DOI:10.1234/example --human

Paper ID Formats

Both S2 and ASTA accept these identifier formats:

• DOI:10.1093/sysbio/syy032
• ARXIV:2106.15928
• PMID:19872477
• CorpusId:215416146
• Raw Semantic Scholar ID (40-char hex)

Opening a paper page in the browser

The reliable way to open an S2 paper page in Chrome is to resolve the identifier to the 40-char paper ID, then open the website URL. The https://api.semanticscholar.org/... redirect form is NOT reliable — Chrome often gets a JSON/non-navigable response instead of the rendered page.

# From a DOI: resolve to paperId, then open the website page
doi="10.1093/sysbio/syy032"
pid=$(curl -s "https://api.semanticscholar.org/graph/v1/paper/DOI:$doi?fields=title" \
  | sed -n 's/.*"paperId": *"\([^"]*\)".*/\1/p')
open -a "Google Chrome" "https://www.semanticscholar.org/paper/$pid"

# If you already have the 40-char SHA paper ID, open it directly:
open -a "Google Chrome" "https://www.semanticscholar.org/paper/<sha>"

Note: the bare https://www.semanticscholar.org/paper/CorpusID:... form does NOT resolve (404s) — you must use the SHA paper ID. To resolve a CorpusId instead of a DOI, swap DOI:$doi above for CorpusId:236964352.

Concept Nodes (Knowledge Graph)

Build a knowledge graph by creating concepts and linking papers to them.

Create Concepts

# Add a concept with name, aliases, and description
bip concept add somatic-hypermutation \
  --name "Somatic Hypermutation" \
  --aliases "SHM,shm" \
  --description "Process by which B cells diversify antibody genes"

# List all concepts
bip concept list --human

# Get a specific concept
bip concept get somatic-hypermutation --human

Link Papers to Concepts

# Use flags: -s (source paper), -t (target concept with concept: prefix), -r (relationship type), -m (summary)
bip edge add -s Halpern1998-yc -t concept:mutation-selection-model -r introduces \
  -m "Foundational paper defining the mutation-selection model"

bip edge add -s Yaari2013-dg -t concept:somatic-hypermutation -r models \
  -m "Introduces S5F model for SHM targeting"

Note: Use concept: prefix for concept targets, project: for project targets.

Standard Relationship Types

Type	When to Use
introduces	Paper first presents or defines this concept
applies	Paper uses concept as a tool or method
models	Paper creates computational/mathematical model
evaluates-with	Paper uses concept for evaluation/benchmarking
critiques	Paper identifies limitations or problems
extends	Paper builds upon or extends the concept

Query the Knowledge Graph

# Find all papers linked to a concept
bip concept papers somatic-hypermutation --human

# Filter by relationship type
bip concept papers somatic-hypermutation --type introduces

# Find what concepts a paper relates to
bip paper concepts Halpern1998-yc --human

Manage Concepts

# Update a concept
bip concept update somatic-hypermutation --description "Updated description"

# Delete a concept (warns if papers linked)
bip concept delete unused-concept

# Force delete (removes linked edges too)
bip concept delete old-concept --force

# Merge duplicate concepts
bip concept merge shm somatic-hypermutation --human

Troubleshooting

Snippet Search Timeouts

bip asta snippet frequently times out with "context deadline exceeded". Workarounds:

1. Reduce limit: --limit 5 instead of default
2. Use MCP directly: mcp__asta__snippet_search with small limit
3. Fall back to keyword search: bip asta search is more reliable
4. Retry once - sometimes it's transient

No Results Found

If searches return nothing relevant:

1. Check spelling of author names and technical terms
2. Simplify query - fewer terms, more common synonyms
3. Try both local and external:

   bip search "topic" --human # local
   bip semantic "topic"      # local semantic
   bip asta search "topic"   # external
   

4. Check date filters - paper may be too old/new for range

Paper Not Found by ID

If bip get <id> or bip asta paper <id> fails:

1. Verify ID format: DOI:10.xxxx/yyyy (include prefix)
2. Try alternate IDs: Same paper may have DOI, PMID, arXiv ID
3. Search by title instead: bip asta search "exact paper title"

SQL Schema Errors

If you see errors like no such column: pmid or similar schema mismatches:

bip rebuild

The SQLite database is ephemeral and rebuilt from the JSONL source of truth. Schema changes require deleting and rebuilding.

Slow Performance

• bip s2 commands are rate-limited to 1 req/sec
• Use bip asta for bulk exploration (10 req/sec)
• Run searches in parallel when independent