arxiv-research-synths

name	arxiv-research-synths
description	Gap-driven arXiv research: identify gaps in project docs, search arXiv API for papers, extract key sections via ar5iv HTML fallback, write structured synthesis md, convert best paper to Python. Trigger when user asks to 'find papers', 'research papers for X', '补强文档', or '搜论文并转成py'.
tags	["arxiv","research","paper","synthesis","gap-analysis"]

arxiv-research-synths

Research pipeline: project gap analysis → arXiv paper discovery → section extraction → synthesis md → paper2code.

When to use

• User wants to strengthen project docs with external research
• User asks to search arXiv for papers on specific topics
• User wants "find papers and convert to code"
• Evaluating a project's theoretical foundations

Pipeline

Stage 1 — Gap analysis

1. Read all project docs (README, architecture, plans, research notes)
2. Identify missing theoretical foundations, unvalidated assumptions, missing risk controls
3. Rank gaps by severity (critical / medium / low)
4. Output gap list with: gap description, why it matters, what kind of paper would fix it

Stage 2 — arXiv search

Use arXiv API directly (not web_search for arXiv — API is faster and structured):

curl -s "https://export.arxiv.org/api/query?search_query=id:{ARXIV_ID}" | grep -E '<id>http|<title>|<summary>'

Search by topic (use specific terms, avoid generic words):

# Good: specific, returns relevant results
curl -s "https://export.arxiv.org/api/query?search_query=all:ornstein-uhlenbeck+AND+all:microstructure&start=0&max_results=5&sortBy=submittedDate&sortOrder=descending"

# Bad: too generic, returns physics/astro noise
curl -s "https://export.arxiv.org/api/query?search_query=all:regime+OR+all:switching"

Key rules:

• Use AND (not OR) between terms for precision
• Search by exact arXiv ID when you already have a reference: id:2602.19419
• cat:q-fin.TR / cat:q-fin.ST / cat:q-fin.MF for finance-specific filtering
• sortBy=submittedDate&sortOrder=descending for latest papers
• If API search returns noise, switch to id: lookups from reference lists

Multiple topic search: run 3-5 API queries in parallel for different gap areas.

Stage 3 — Paper extraction (ar5iv fallback)

When pip install pymupdf4llm fails (common in sandboxed environments):

curl -sL "https://ar5iv.labs.arxiv.org/html/{ARXIV_ID}v{VERSION}" -o /tmp/{slug}.html

Then extract with Python html.parser (see paper2code skill for full TextExtractor class).

Section map: after extraction, scan for section titles and print line numbers:

for i, line in enumerate(lines):
    if any(k in line.lower() for k in ['abstract','introduction','method','experiment','result','conclusion']):
        if len(line) < 100:
            print(f"L{i}: {line}")

Must-read sections for each paper:

• Abstract + Intro → one-sentence contribution
• Method/Model → what to implement
• Experiment/Results → key numbers to cite
• Appendix → hidden implementation details

Rate limiting: ar5iv may throttle. Add 1-2s delay between fetches if hitting many papers.

Stage 4 — Synthesis writing

Write docs/{project}-research-synthesis.md with this structure:

# Research Gaps & Paper Synthesis

> 评估日期：YYYY-MM-DD

## 一、文档已做到位的
(brief positive assessment)

## 二、核心不足（按影响排序）
### A. Gap title [severity]
- What's missing
- Why it matters
- Root cause

## 三、论文补强矩阵
### 3.1 Paper Title — relevance tag [relevance stars]
- arXiv ID + PDF link
- Core contribution (2-3 sentences)
- Direct application to project (specific modules/features)
- Actionable insights

## 四、发散性跨领域思路
(cross-domain inspirations from the papers)

## 五、论文 PDF 汇总
| # | 论文 | arXiv ID | PDF 链接 | 相关度 |

## 六、建议阅读顺序
1. Paper with highest relevance
2. ...

## 七、建议行动
### 立即可做（1-2 天）
### Sprint N 内做
### 远期做

Stage 5 — Paper2code conversion

For the highest-relevance paper, use the paper2code skill to convert to Python:

1. Write contribution.md — one-sentence summary, paper type, implementation scope
2. Write ambiguity_audit.md — specified vs unspecified items
3. Write configs/base.yaml — all hyperparameters with paper citations
4. Write core src/*.py modules — one module per paper concept
5. Write REPRODUCTION_NOTES.md — explain all unspecified choices
6. Test OU/statistics modules without torch (sandbox constraint)
7. Write README.md with ict-engine integration example

Output directory: paper2code/{paper_slug}/

Stage 6 — Integration bridge

For each paper implementation, write a bridge function showing how it maps to the project:

def state_to_execution_features(state) -> dict:
    """Map paper's state vector to project's domain types."""
    return {
        "ou_theta": state.theta,
        "regime_laziness_score": state.theta * state.active_frac,
        ...
    }

Key learnings from past runs

• arXiv API OR operator is broken — returns results from any field, not just specified ones. Always use AND.
• ar5iv HTML has formula rendering issues — but section structure is preserved. Good enough for contribution analysis and code skeleton.
• Read specific line ranges, not full papers — papers are 20-40 pages. Use the section map to read only what you need.
• Python tests without torch — if torch not installed, test statistics/estimation modules standalone. DDQN agent can be verified structurally (imports, shapes) without running training.
• 3 papers in parallel — optimal batch size for research. More than 3 overwhelms context, fewer misses connections.
• Multi-paper session (7 papers) — feasible if you do: fetch all HTML → extract sections → read key sections → write all implementations → batch test. Don't interleave read-implement-read across papers.
• Paper priority: implement papers that are directly mappable to existing project modules first (e.g., OU estimator → existing ou.rs), then defensive/safety modules, then speculative/upgrade-path papers.
• "Defensive" papers are high value: papers describing system failures (Red Queen's Trap) produce safety modules that are immediately usable. They require less interpretation than novel-method papers.
• arXiv search query patterns for finance: all:ornstein-uhlenbeck+AND+all:microstructure, all:ising+AND+all:financial+AND+cat:q-fin.ST, id:{EXACT_ID} for known papers. Avoid broad terms like "regime" without finance qualifier — returns physics/astro noise.
• Finance market-regime / strategy source scans: when the user wants papers plus repos/forums/scripts for profitable factors, options, VRP, or market shape classification, use references/finance-regime-strategy-source-scan.md for source mix, query patterns, output shape, and validation gates.

Stage 6 — Architecture bridge (per paper)

For each paper implementation, write bridge functions showing how paper concepts map to project domain types:

def state_to_execution_features(state) -> dict:
    """Map paper's state vector to project's domain types."""
    return {
        "ou_theta": state.theta,
        "regime_laziness_score": state.theta * state.active_frac,
        ...
    }

Pattern: each paper module should expose one apply_to_ict_*() or state_to_*() function that translates paper-specific types into the project's existing type system. This makes integration testable independently of the paper's full implementation.

Stage 7 — Batch test & summary

After all papers implemented, run all tests in one pass:

cd paper2code && for d in */; do python3 $d/src/*.py 2>&1 | tail -1; done

Then produce final summary table:

| 论文 | arXiv | 实现 | 行数 | 测试 | 核心迁移点 |

Fallback: no arXiv access

If ar5iv and arXiv API both fail:

1. Use abstract page (arxiv.org/abs/{ID}) for contribution summary
2. Search for paper title on Google Scholar → "All versions" → find free PDF
3. Check SSRN for working paper versions
4. Ask user to provide PDF path