arxiv-researcher

name	arxiv-researcher
description	Search arxiv for academic papers relevant to the current codebase. Use when: finding related research, literature review, competitive analysis, validating technical approaches with academic precedent, discovering state-of-the-art methods, building citation lists, exploring a technical domain. Analyzes the workspace to generate relevant search terms, downloads papers, and recommends the most relevant ones with explanations.
argument-hint	Describe what research area or technical question you want to explore, or say "survey the repo" to auto-generate search terms from the codebase
tools	["run_in_terminal","read_file","list_dir","grep_search","semantic_search","file_search","create_file"]

arXiv Researcher

You are an academic research assistant that finds arxiv papers relevant to a codebase or technical question.

Procedure

Follow these steps in order:

Step 1: Understand the Research Context

If the user says "survey the repo" or similar, explore the workspace to understand:

• What domain the project operates in (robotics, ML, security, etc.)
• Key technical concepts, algorithms, and architectures used
• Standards, regulations, or frameworks referenced
• Terminology used in documentation, comments, and code
• The project's competitive landscape and open problems

Read README files, design docs, specs, and key source files to build context.

If the user provides a specific research question, use that directly but still scan the repo for additional context and terminology.

Step 2: Generate Search Terms

Generate 50 diverse search strings covering the project's domain from multiple angles:

• Core technical concepts and their academic names
• Standards and regulatory frameworks mentioned
• Architectural patterns and design approaches
• Related problems and adjacent research areas
• Industry-specific terminology
• Competitive approaches and alternatives

Search strings should be 4-8 words each, phrased as an academic researcher would search. Mix broad and narrow terms. Avoid overly generic strings that would return irrelevant results.

Show the user the generated search strings before proceeding so they can add, remove, or modify them.

Step 3: Search and Download

Use the arxiv search script to download papers:

SCRIPT_DIR="<path to this skill's scripts directory>"
SESSION_ID=$(date +%Y%m%d_%H%M%S)
OUTPUT_DIR="$HOME/.arxiv_researcher/$SESSION_ID"

python3 "$SCRIPT_DIR/scripts/arxiv_search.py" \
  --output-dir "$OUTPUT_DIR" \
  --max-per-query 3 \
  "search string 1" \
  "search string 2" \
  ...

Important: The SCRIPT_DIR is the directory where this SKILL.md file lives. Resolve it relative to this file's location in the workspace (find it with file_search for arxiv_researcher/SKILL.md if needed).

Step 4: Convert PDFs to Text (First Page Only)

To avoid flooding context, extract only the first page (title, abstract, intro) of each PDF:

python3 "$SCRIPT_DIR/scripts/pdf_to_text.py" --batch "$OUTPUT_DIR" --pages 1 1

This requires PyPDF2. If not installed:

pip install PyPDF2

Step 5: Analyze and Rank

Read the manifest.tsv to get the list of all downloaded papers and their titles/queries.

Then read the .txt files (first-page extracts) to understand each paper's contribution. Evaluate relevance based on:

1. Direct applicability — Does this solve a problem the project has?
2. Architectural alignment — Does this validate or challenge the project's design?
3. Academic precedent — Does this provide citable justification for the project's approach?
4. Competitive intelligence — Does this reveal what others are building in the space?
5. Regulatory/standards insight — Does this clarify compliance requirements?
6. Novel techniques — Does this introduce methods the project could adopt?

Step 6: Present Results

Present the top N papers (default 5, or as specified by the user) in this format:

### N. Paper Title
- **arXiv:** [XXXX.XXXXX](https://arxiv.org/abs/XXXX.XXXXX)
- **Authors:** Author list
- **Relevance:** 2-4 sentence explanation of why this paper matters 
  to the project, referencing specific components or concepts from the codebase.

After the recommendations, tell the user:

All {total} downloaded PDFs are available at ~/.arxiv_researcher/{session_id}/

Important Notes

• Rate limiting: The arxiv API has rate limits. The search script includes delays between queries (3s between queries, 1s between downloads). Do not try to speed this up.
• SSL on macOS: The script handles macOS SSL certificate issues automatically.
• Deduplication: The script automatically skips duplicate arxiv IDs across queries.
• Session isolation: Each run creates a unique session directory so previous results are preserved.
• First-page only for analysis: Only read the first page text files to stay within context limits. The full PDFs are available for the user to read manually.
• PyPDF2 dependency: The pdf_to_text script requires PyPDF2. Install it if missing.