paper-search-pro

# paper-search-pro Multi-source literature search with adjustable depth. Four tiers, five data sources orchestrated by you (the main agent). Python helpers handle deterministic work; LLM classification is delegated to parallel Inline SubAgents — no external API key required. ## When to use this skill - User wants to find academic papers / 找文献 / 论文搜索 - User is preparing a literature review, systematic review (SR), scoping review, or meta-analysis - User wants to scope research on a topic for a thesis / proposal / coursework / news story - User asks "what research exists on X" / "find me papers about Y" - User uploads a query that suggests literature gathering (PICO, SPIDER, MeSH, RCT, etc.) ## When NOT to use - User wants to **read** a specific paper (use PDF reader / download tool) - User wants to **summarize** a single known paper (use a summarizer) - User wants to **download** PDFs given DOIs (use `paper-downloader-portable`) - User already has a literature set and wants to write a review (use `literature-set-review` / `factor-outcome-review`) - User wants concept explanation, not papers ("what is prospect theory" → just answer) --- ## 🤖 Called by another agent / headless mode **If you are an agent driving this Skill for your own reasoning (not for a human who wants an HTML report)**, do NOT hand-run the 14-STEP recipe below. There is a single structured-data channel built for you: ```bash PYTHONPATH=$PSP_HOME python3 -m scripts.agent_search "<query>" > result.json ``` One command runs the whole deterministic core — multi-strategy retrieve → dedup → heuristic relevance score (computed for every paper) → saturation signal → quota snapshot → per-paper journal metric — and prints **one JSON envelope** (no HTML, no PRISMA, no LLM classification SubAgent). The human path below is unaffected. That command gives you a deterministic **floor**, not the finished job — agent mode is *not* meant to stop at the machine output; `references/agent_mode.md` is where you layer your own semantic judgement on top to reach human-recipe quality (the command guarantees the floor; you supply the quality). 📖 **Read `references/agent_mode.md`** for the full envelope schema, every flag (`--verify`, `--min-relevance`, `--quartile`, `--min-impact`, …), the relevance formula, error codes / exit codes, and source selection. This is the SSOT for agent callers — everything else in this section is just the pointer to it. Everything from here down is the **human-facing 14-STEP recipe** (HTML report + exports). Use it when the consumer is a person. --- ## 🔥 Execution discipline (read before running anything) Four invariants govern every step — ignoring them is the dominant failure mode in real sessions: - **A — NEVER `cd` into the Skill directory.** `cd $PSP_HOME` rebinds `./` to the Skill asset folder, so `./paper-search-results/...` lands inside the Skill instead of the user's workspace (and a re-install wipes it). Run every helper from the user's PWD: `PYTHONPATH=$PSP_HOME python3 -m scripts.<name> … > "$SEARCH_DIR/..."`. `$PSP_HOME` (STEP 0) is the install dir; `$SEARCH_DIR` (STEP 0) is an absolute path under the user's PWD. - **B — Dispatch classifier SubAgents in parallel.** STEP 6 puts up to 5 `Task` blocks in one assistant message; serial dispatch inflates Standard tier from ~10 to ~17 min. The worked example lives in STEP 6 — it is not repeated elsewhere. - **C — Announce every skip.** If you skip a STEP (budget / empty data / user choice), say *what* you skipped, *why*, *what's lost*, and *how to recover* (e.g. "re-run at `--tier deep`"). Skipping is fine; surprising the user is not. - **D — Read a step's cited reference when that step is non-trivial for this case.** Each STEP names a `references/<file>.md` carrying edge cases not duplicated here. You won't read all of them every run, nor should you — but skipping the reference for a step you are *actually about to run* is where boundary knowledge (dict-vs-list shapes, enrich-not-search, DOI casing) gets lost. Read the one in front of you. --- ## Architecture at a glance ``` You (main agent) drive the workflow per this SKILL.md. Python helpers do deterministic work — NO LLM inside, NO external API key. L1 OpenAlex (primary) → deep top-100 multi-strategy L2 PubMed (medical) → MeSH enricher (mostly; Audit-tier can search independently) L2 arXiv (CS/AI) → last-year recent layer (+ 4-day sentinel on freshness words) L2 OpenReview (AI) → accepted-venue labels (ICLR / NeurIPS / ICML …) L3 Semantic Scholar → influentialCitationCount + abstract fallback L3 CrossRef → funder / license / clinical-trial-number Classification → Inline SubAgents (parallel, file-IPC, 5 per message) Output → HTML (Shadcn) + MD + BibTeX/RIS/CSV + PRISMA-S log ``` --- ## The 4 tiers — pick first | Tier | Wall-clock | Papers | When to pick | |------|------------|--------|--------------| | Quick | ~5-8 min | 20-60 | "查一下" / "几篇" / "before tomorrow" / fast scope | | **Standard** (default) | ~10-17 min | 60-180 | Scope a topic / write background / general lit search | | Deep | ~30-45 min | 180-400 | "thorough" / writing a review article / 综述写作 | | Audit | ~2-3 hr | 400-1000+ | "systematic review" / "PRISMA" / "Cochrane" / "meta-analysis" | 📖 **BEFORE picking, read `references/tier_decision.md`.** Tell the user your choice and why. For Audit, show limitations warning + get explicit confirmation before starting. --- ## The recipe For every literature search, follow these steps in order. Each step references a `references/` file for details. Skip files only when the step is obviously trivial for the case at hand — and announce the skip per Rule C. ### STEP 0 — Setup ($PSP_HOME + working directory) 📖 BEFORE THIS STEP, read: `references/setup.md`. **Resolve the Skill install path into `$PSP_HOME`** (every later step uses `PYTHONPATH=$PSP_HOME`). Prefer explicit injection / agent env var; otherwise scan the known cross-agent install locations. If your harness already exposes this SKILL.md's absolute path, just `export PSP_HOME="<that dir>"` and skip the scan. 📖 Full rationale, why this can't be a script, and the complete path list: `references/runtime_bootstrap.md`. ```bash PSP_HOME="${PSP_HOME:-${CLAUDE_SKILL_DIR:-${CODEBUDDY_SKILL_DIR:-}}}" # explicit / agent-injected if [ -z "$PSP_HOME" ]; then # else scan known installs for base in "$HOME/.claude" "$HOME/.codex" "$HOME/.agents" "$HOME/.config/opencode" \ "$HOME/.codeium/windsurf" "$HOME/.config/goose" "$HOME/.cline" "$HOME/.roo" \ "$HOME/.copilot" ./.claude ./.codex ./.agents ./.cursor ./.opencode ./.windsurf; do [ -f "$base/skills/paper-search-pro/SKILL.md" ] && PSP_HOME="$base/skills/paper-search-pro" && break done fi [ -z "$PSP_HOME" ] && { echo "ERROR: paper-search-pro install not found. Set PSP_HOME to the dir containing SKILL.md."; exit 1; } export PSP_HOME; echo "Using Skill install: $PSP_HOME" ``` **Verify config keys** (executed from any cwd, never `cd` into the Skill dir): ```bash PYTHONPATH=$PSP_HOME python3 -c \ "from scripts.config import load_config; c = load_config(); print('OK' if c.openalex_api_key and c.ncbi_email else 'MISSING — see references/setup.md')" ``` If "MISSING", point the user to `references/setup.md` (5 keys, all free, ~15 min total) and halt. **Set up the working directory variable** — every subsequent step uses `$SEARCH_DIR`: ```bash SEARCH_ID="<topic_slug>_<tier>_$(date +%Y%m%d_%H%M%S)" # e.g. clt_education_quick_20260522_103045 SEARCH_DIR="$(pwd)/paper-search-results/$SEARCH_ID" mkdir -p "$SEARCH_DIR/raw" "$SEARCH_DIR/batches" "$SEARCH_DIR/classifications" echo "Outputs will land in: $SEARCH_DIR" ``` `$SEARCH_DIR` is now an **absolute path under the user's PWD**. Use `"$SEARCH_DIR/..."` (quoted, with the variable) in every helper command below — not `./paper-search-results/...`. ### STEP 1 — Plan the query (MANDATORY for all tiers) 📖 BEFORE THIS STEP, read: `references/query_planner.md`. **Detect the report UI language** — `UI_LANG` (`zh` for Chinese queries, `en` for everything else) selects which UI language the final HTML report renders in. Paper titles / abstracts / authors / venues are NEVER translated — only the report's UI chrome. Pass `--language $UI_LANG` to STEP 12b. ```bash UI_LANG=$(PYTHONPATH=$PSP_HOME python3 -m scripts.detect_language "$USER_QUERY") ``` The detector routes Japanese / Korean / European queries to **English** (the bundle ships only EN + ZH dictionaries; English is the international academic default). 📖 The exact Unicode rule and why kana is checked before Han live in `references/runtime_bootstrap.md`. **Determine the search language space** (`search_language`, axis 2 — *which literature ocean*, distinct from `UI_LANG` above which is only *report chrome*). 📖 The parsing SSOT is `references/source_routing.md` §"Language scope"; resolve the space here, before phrasing the query, because it changes how STEP 3's query is built. This is **additive and opt-in — a pure English query resolves to the `en` space with zero new prompts or behavior (R-19)**; everything below fires only for Chinese queries or explicit signals. - Read the persisted default `config.search_language` (auto | en | zh | both) and apply the priority ladder **flags > in-query markers > config > auto**. CJK presence is the mechanical fact from `detect_language` above; markers (`CSSCI`, `中文文献`, `SSCI`, `知网`, …) and non-signals (`中科院一区`, topic-about-China) are your semantic judgment per the §"Language scope" tables. - **`auto` + a Chinese (CJK) query + no language marker + no persisted value → ask ONE question before retrieving** (this is the human path's job; the CLI/agent path passes through instead). Two sentences, offer to persist, and don't re-ask later this session: > 你用中文提问——文献要英文、中文,还是都要?顺便可以说"以后都这样",我就记成默认、下次不再问。 - "中文" / "都要" → enter that space (STEP 2 discipline routing takes over; report one line there). - "英文" → v2.2 behavior (Chinese topic planned as an English query), report one line. - "无所谓 / 都行" → **this run uses `both`** (Recall > Precision), not persisted; if the same user answers "无所谓" a second time, add one light offer to set `both` as default, then never ask again. - Only an explicit "以后都…" persists to `config.search_language` (single answers never auto-persist). - **If a rank ambiguity (bare "Q1") also fired this run, merge both questions into ONE message** — ask language + platform together, never in two rounds (over-asking is a red line). - Once the space is known, phrase the query per `references/query_planner.md` §"Cross-language query handling": **`zh` keeps Chinese search terms (no translation)**, `en` uses the English terms (v2.2 behavior), `both` builds two sets. **Write each retrieval query as concept blocks, not a word list.** OpenAlex requires every bare word, so synonyms written as words exclude every paper that uses only one of them (a user's 11-word query matched one paper by title and abstract over five years). Synonyms go in OR groups, 2–3 blocks joined by AND, phrases in quotes — `("short video" OR TikTok OR Douyin) AND (attention OR concentration)` — and keep the population block out of at least one strategy. 📖 `references/query_planner.md` §"How OpenAlex reads a query". Apply PICO / SPIDER / PEO depending on domain: - Medical/clinical → PICO (Population/Intervention/Comparator/Outcome) - Qualitative → SPIDER - Scoping → PEO (Population/Exposure/Outcome) - Open-ended → just extract 2-4 concept blocks + 2-5 synonyms each **Journal-rank intent recognition (additive — only acts when the query mentions a partition).** Before you extract concept blocks, check whether the user's query carries a journal-rank/partition phrase — "中科院一区", "Q1", "JCR Q1", "SJR Q2", "顶刊 / top journal". If so, that phrase is a **filter condition, not a search term**, and it MUST be **stripped from the topic** before retrieval. This roots out the failure that motivated the whole feature: "中科院一区 情绪调节" used to send "中科院一区" to the search engine as a topic word, so it searched for papers *about* 中科院一区 instead of papers *on* 情绪调节 *filtered to* CAS tier 1. The deterministic parser does both jobs (extract + strip) for you: ```bash PYTHONPATH=$PSP_HOME python3 -c " from scripts.rank_intent import parse_rank_intent i = parse_rank_intent('''<original user query>''') import json; print(json.dumps({ 'platform': i.platform, 'tiers': i.tiers, 'quartiles': i.quartiles, 'top': i.top, 'ambiguous': i.ambiguous, 'candidates': i.candidate_platforms, 'cleaned_query': i.cleaned_query, 'stripped': i.matched}, ensure_ascii=False)) " ``` Then act on the parse: - **`cleaned_query`** is the rank-stripped retrieval input — use it (NOT the raw query) when building STEP 3 queries and the query plan. It is not automatically `SEARCH_TOPIC` or the report title: the semantic planning step below still removes conversational and operational text. When the query had no rank phrasing, `cleaned_query == query` and retrieval behavior remains unchanged (R-19). - **`platform` + `tiers`/`quartiles`/`top`** are the filter you will apply right after STEP 5 — remember them (and record them as `rank_filter` in `query_plan.json`); do not filter here. - **`ambiguous == True`** (a tier or quartile with no platform word, e.g. a bare "Q1" or "SSCI 一区" — the recogniser never guesses a platform): **ask the user one short question inline** before going further, offering the platforms in `candidates` — *"按 JCR 还是 SJR 的 Q1 筛?"* / *"按中科院分区还是 JCR 分区?"*, plus *"顺便要不要设为以后的默认?"* The CLI/headless path cannot ask, so this inline question is specifically the human path's job. - If the query mentions no partition at all, skip this entirely — STEP 1 proceeds exactly as before. **Time scope — decide the year window before retrieving.** 📖 `references/query_planner.md` §"Year filter heuristics". Sort the request into one of four cases: - **Explicit range** ("2020–2024", "近两年", "2023 年以后", "last 3 years", "只要 2026 年的") → set `year_min` / `year_max`; relative ranges count back from the current year ("近两年" = current year − 1 … current year). No question. One line: *"本次限定 2025–2026。"* - **"New" without a number** ("最新", "最近", "近年", "前沿", "recent", "latest", "cutting-edge") → how recent is genuinely ambiguous and changes the result set. If `config.recent_years` is set, use it and say so in one line. Otherwise **ask one question** — in the same message as any language or rank question from above, never a second round: > 「最新」想要多近？近一年 / 近两年 / 近三年 / 近五年，或者直接给区间（如 2023–2026）。说「以后都这样」我就记成默认。 Only an explicit "以后都…" persists (`recent_years: <N>` in `~/.paper-search-pro/config.yaml`, same mechanism as `search_language`). - **Classics** ("经典", "奠基", "起源", "seminal", "classic") → no year window; add `seminal` in STEP 3. - **Nothing said** (most requests) → no year window, and **do not ask**: STEP 3's recent leg still brings the most relevant papers of the last year and of the two years before. One line: *"没说时间范围：本次不限年份，并单独补一层近一年的新文献。只要近几年的，说「只要近两年」即可。"* If a language or rank question is being asked anyway, you may offer the time choices in that same message. - **Audit tier**: date limits belong to the protocol (PRISMA-S reports them) — confirm the window as part of the Audit confirmation you already ask for. Where the host has a structured-choice tool (Claude Code's AskUserQuestion; Codex's `request_user_input` when it is listed), ask with these options plus free input; otherwise ask in text. The agent/headless path never asks — it takes `--year-min` / `--year-max` as given.