research-genealogy

name

research-genealogy

description

Given a research direction (e.g. "generated image detection", "对比学习", "neural machine translation"), research the literature and produce a NON-LINEAR development genealogy: a terminal ASCII tree (built from `--` connectors and text) plus a narrative tracing who proposed what, against which problem, what built on what, and which lines ran in parallel — from the founding work up to the latest. Use when the user wants to understand how a field evolved, not just a flat list or a theme-organized survey.

Research Genealogy

Turn a research direction into a readable, non-linear development genealogy.

The value is NOT "another paper search". It is the lineage: founding work → the problem it tackled → who built on it → which parallel branches emerged → which approaches were superseded → today's frontier.

The deliverable is a complete report (saved as a markdown file AND shown in the conversation): the genealogy tree plus a narrative that walks through the whole development, era by era and branch by branch, with every claim tied to a real paper. A bare tree or a bare list is NOT done.

Hard rules (avoid citation hallucination)

• Propose freely, but ground everything. You SHOULD use your own knowledge and WebSearch to name the field's landmark and newest papers (that recall is the point — it beats a blind keyword search). But a proposed paper only becomes a node after it is resolved to a real record by the scripts (papers.py resolve / --seed-titles, OpenAlex / arXiv / Semantic Scholar). A title that resolves to nothing stays in _unresolved — never invent its authors, year, venue, or citations to make it a node. You organize and narrate; the scripts certify.
• Ground every summary. Write each node's problem / contribution from the paper's real abstract (the _abstract field in the draft, or papers.py search "<title>" --abstract), not from memory.
• Edges must be real citations. The draft's builds-on edges are derived from OpenAlex reference lists, so they are real by construction. If you add or redirect an edge by hand, confirm it with scripts/verify.py (or papers.py expand); leave honest ⚠ marks on edges you cannot confirm.
• If a paper you believe is important does not appear in the fetched data, run a targeted title search for it rather than asserting it from memory.
• Keep the trunk small: the 承上启下 (load-bearing) nodes, not every paper. 10–16 nodes is a good tree.

Step 0 — turn the user's direction into search phrasings

The user's wording ("AI4Reaction", "生成图像检测", a lab's nickname for a field) is usually NOT what the literature is indexed under. Before searching, derive:

1. one primary English phrasing — the name the papers themselves use (e.g. AI4Reaction → "machine learning chemical reaction prediction");
2. 2–3 alias phrasings covering the direction's main sub-tasks and synonyms (→ "retrosynthesis prediction deep learning", "reaction yield prediction machine learning", "large language models chemistry") — a field is always named several ways, and one query never covers all branches. Include one phrasing aimed at the newest wave (the LLM/ foundation-model angle exists in almost every field since 2023).

Not sure how a field is phrased in the literature? Mine candidates first:

python3 scripts/genealogy.py "<your best primary phrasing>" --suggest-aliases

It runs one seed search and prints the most frequent title phrases (real sub-topic names) as ready-to-paste --alias lines — pick the ones that name a genuine branch.

The final field label shown to the user stays in their language — edit it in the JSON afterwards.

Step 1 — research the field yourself (Claude proposes)

Before running the pipeline, act like the expert being asked "调研一下 X 方向". Using your own knowledge and WebSearch, name the papers that actually carry the field's story — 15–25 candidate titles spanning:

• the founding / landmark works (the ones any survey opens with);
• the high-impact middle (the methods everyone builds on);
• the 2026 frontier — and this is where WebSearch matters most, because OpenAlex lags on the newest work. Search the live web: "<field> 2025 2026 survey", "best <field> papers 2026 arXiv", the relevant benchmark/leaderboard or lab pages, recent arXiv listings.

Write the exact titles (resolution matches on the title) to seeds.txt, one per line. Don't worry about getting metadata right — the next step grounds every title to a real record and quarantines anything it can't find. You are supplying recall; the scripts supply proof.

python3 scripts/papers.py resolve --file seeds.txt   # optional: preview what grounds

Step 2 — generate the draft (one command)

python3 scripts/genealogy.py "<primary phrasing>" \
    --alias "<alias 1>" --alias "<alias 2>" --alias "<alias 3>" \
    --seed-titles seeds.txt \
    --nodes 14 --out lineage.json

--seed-titles resolves each title you proposed to real metadata and injects it as a trusted node (and as a snowball hub, so its real ancestors/heirs join the pool too). Titles that resolve nowhere land in the draft's _unresolved list — re-find or drop them in Step 3; never invent them. Omit --seed-titles to fall back to pure keyword search.

Read the diagnostics it prints. Each phrasing reports its precise hits, and the pool reports its core size. If a phrasing got ~0 precise hits it contributed nothing — replace it. If the core is thin (< 5) the relevance gate is weak — re-phrase and re-run before investing in refinement. One re-run with better phrasings beats an hour of manual repair.

What the command does (so you know what you can trust):

1. seed grounding (--seed-titles) — resolves the titles you proposed in Step 1 to real records and injects them as trusted nodes + snowball hubs; unresolved titles go to _unresolved, never invented;
2. multi-pass keyword search (broad + precise + frontier);
3. arXiv frontier pass — searches arXiv for the newest preprints (OpenAlex often lags months behind), back-resolves each to OpenAlex by title to recover real references; genuinely-unindexed ones are surfaced in _frontier_candidates (marked source: arXiv …) for you to verify and wire in by hand — they are never auto-added as trunk nodes;
4. snowball expansion — pulls references + citing works of the field's core papers (and your seeds), so landmarks the keywords missed still enter the pool;
5. relevance gating anchored on the core (the largest mutually-citing cluster of precise matches) — off-topic keyword twins and generic mega-cited backbones are dropped (your seeds are never gated out);
6. in-field scoring — nodes are ranked by citations within the pool, so the field's true landmarks beat globally-famous tangents;
7. era-stratified selection (your seeds reserved first, then founders + hubs + frontier), then edges straight from the real citation graph with transitive reduction (A→C is dropped when A→B→C exists) and parallel detection (same-era pairs that share references but don't cite each other).

Every builds-on edge in the draft is a real citation and arrives pre-marked "verified": "verified". The draft also carries:

• _stats — orphans (nodes with no edges) and roots;
• _frontier_candidates — strong recent (last ~2 years) papers that did NOT make the cut, with abstracts;
• _alternates — other high-scoring papers that just missed selection;
• _unresolved — seed titles that resolved to no real record (Step 3 handles).

Use the two candidate pools to swap in better nodes without re-searching.

Step 3 — refine the draft (this is your real job)

Work through ALL of these, editing lineage.json directly:

1. Prune & replace. Drop nodes that are off-topic, redundant (two surveys covering the same ground), or pure infrastructure (a dataset paper that isn't a turning point). If a known landmark is missing, fetch it by title and wire it in:

   python3 scripts/papers.py search "<exact paper title>" --abstract --limit 1
   python3 scripts/papers.py expand <paperId> --limit 30   # ground its edges
   

2. Fix orphans (_stats.orphans): either find the node's real citation link via expand and add the edge, or delete the node. An orphan box helps nobody. 2b. Resolve the _unresolved list (seed titles that grounded to nothing): for each, try a corrected/exact title (papers.py resolve "<title>", or WebSearch for the real title), and if it now resolves, wire it in via papers.py search "<title>" --abstract + expand. If it still resolves to nothing, drop it — never promote an unresolved title to a node. Delete the _unresolved key when the list is handled.

3. Rewrite every problem / contribution as one crisp line each, in the user's language, from the node's _abstract — the seeded text is just the abstract's first sentence. problem = what was broken/missing before this paper; contribution = what it introduced. Delete the _abstract keys when done (keeps the file clean).

4. Confirm / fix relation labels. First let the data settle what it can — run verify.py --fix to auto-correct the edges real citations decide:

   python3 scripts/verify.py lineage.json --fix
   

   It swaps backwards builds-on arrows (↺ reversed) and converts any parallel edge that is really a citation (‼ cross-cite) into a directed builds-on — reconciled across duplicate OpenAlex records, so you don't hand-resolve these or apologize for them in the narrative. Honest ⚠ gaps and true parallel pairs are left untouched.

   Then handle what the data can't: edges are builds-on by default, but the draft also guesses some inspired-by / supersedes labels with a heuristic and flags each with "_label_hint": "auto" (rendered as a yellow ?). Confirm or correct each hinted edge from the abstracts — they are guesses, not facts — and add the ones the heuristic missed:

   • inspired-by — conceptual influence rather than direct extension. Typical case: a generation milestone (e.g. DALL·E 2, Stable Diffusion) that triggered a detection/analysis wave — keep it as a root/context node but mark its outgoing edges inspired-by.
   • supersedes — the later method made the earlier one obsolete on the same problem.
   • parallel — independent same-era attacks on the same problem (the draft detects some; add ones you can argue from the abstracts).

   Delete the _label_hint key once you've confirmed (or fixed) an edge.

5. Name the branches. Identify the 2–4 lines of attack the tree contains (e.g. 频域路线 / CLIP特征路线 / 重建误差路线) — you'll use these names in the narrative, and the tree should visibly hang each branch off its own ancestor. If two branches are tangled, redirect primary edges so each branch's papers chain together.

6. Secure the frontier. The last ~2 years must contribute ≥ 3 nodes that represent distinct new directions (e.g. an LLM/foundation-model take, a new benchmark, a new paradigm) — not three lookalike surveys. Shop in _frontier_candidates and swap weak picks (0-citation book chapters, redundant reviews) for papers that show where the field is going.

7. (Only if you hand-edited edges) re-check them:

   python3 scripts/verify.py lineage.json --write
   

Quality bar — enforced by lint.py

Run the gate before rendering — a non-zero exit means the genealogy is NOT done:

python3 scripts/lint.py lineage.json

It hard-checks exactly the rules below and fails on any unmet one:

• Refinement finished: no blank problem/contribution, no leftover _abstract keys, draft scaffolding (_stats, _unresolved, etc.) removed, summaries no longer raw abstract seeds.
• No star topology: the deepest chain is ≥ 3 edges and no node has 6+ direct children (reroute children to their real, nearest predecessor).
• Reaches the present with a real frontier: max year ≥ now−2 AND ≥ 3 nodes from the last ~2 years (a genealogy that stops 2–3 years ago is the #1 failure mode; a frontier that is all surveys warns).
• Structure is visible: ≥ 1 non-builds-on relation survives (parallel/inspired-by/supersedes) and the trunk branches; zero orphans.

Fix what it flags and re-run until it passes. (Curated/frozen example files use --curated to relax the now-relative frontier check.)

Step 4 — render

python3 scripts/render_tree.py lineage.json

Prints a colored view with a left year-axis, role markers (● founder / ◉ hub / ★ frontier), relation-coded branches (├── builds-on, ├┈┈ inspired-by), citation bars, dim ∥ parallel cross-links, and per-edge ✓/⚠ verification marks. --no-color for files/README, --width N to fix.

Other output formats via --format:

python3 scripts/render_tree.py lineage.json --format markdown   # report: tree + table + edges
python3 scripts/render_tree.py lineage.json --format figure-prompt  # image-model prompt

figure-prompt builds a ready-to-feed prompt for an image model (GPT-image / Midjourney / DALL·E) that draws the genealogy as a publication figure. It has a prose body (subject, swimlanes, color semantics, style + negative constraints) and a hard structure checklist — every node and every edge listed verbatim, with ✓/⚠ marks — so the model can only draw the real connections, never invent them. The swimlanes come out as 路线①〈待命名〉; rename each to its real route name (the branch names from Step 3 #5) before feeding it. --lang en for an English brief.

Step 5 — deliver the report (the actual product)

Write the report to <field-slug>-genealogy.md AND present its narrative + tree in the conversation. Everything in the user's language. Template:

# <方向名> 发展历程

> 一句话定位这个方向 · <起始年>–<最新年> · N 篇关键论文 · 引用边均经 OpenAlex 验证

## 全景谱系树
<render_tree.py --no-color output, in a code block>

> 引用数取自 OpenAlex 单条记录，会因预印本/正式版被拆分而**低估**（经典论文
> 尤甚），仅作量级参考；谱系的价值在「谁建立在谁之上」的关系，不在这些数字。
> （只此一行说明，不要在正文里反复解释。）

## 发展历程

### 奠基（<years>）：<这一段在解决什么>
<谁最先把问题立起来，当时的背景，最初的方法是什么、为什么不够>

### <路线A名称>（<years>）
<这条路线为什么从主干分出来（前一代方法做不到什么）→ 代表工作依次
解决了什么 → 它后来被什么取代/吸收 → **明确点出它与相邻路线的 parallel/
supersedes 关系**（这条边就是故事，不能省）>

### <路线B名称>（<years>）
<同上；路线之间的 parallel/对比关系必须点明>

### 转折：<外部冲击是什么>（<year>）
<指向 inspired-by 节点：什么事件改写了问题本身，领域如何转向>

### 近两年前沿（<NOW-1>–<NOW>）
<≥3 个新方向，每个：它接着哪条路线、新在哪里、目前到什么程度>

### 开放问题
<3-5 条，从前沿论文的 limitation/未来工作中来，不要泛泛而谈>

## 论文清单
<render_tree.py --format markdown 的 Papers 表>

Hard rules for the narrative:

• Every node appears in the narrative at least once, cited as "作者 (年份)"; never introduce a paper that is not a node in the tree.
• Explain causality, not chronology: "B 在 A 的基础上解决了 X" beats "然后 B 出现了". The builds-on/inspired-by edges ARE the story.
• Each route states its relationship to the neighboring route(s) in one sentence — parallel（同期之争）/ supersedes（取代）/ 分叉自哪个祖先. The edges are the genealogy's point; a route described in isolation wastes them.
• The 近两年前沿 section is mandatory and must name concrete new directions — a genealogy that stops 2–3 years ago is the #1 failure mode.

Final gate before delivery — run the report linter; it mechanically checks that every node is actually discussed, the required sections exist, and the tree block survived (a non-zero exit means the report is NOT done):

python3 scripts/lint.py lineage.json --report <field-slug>-genealogy.md

Optional — a publication figure. If the user wants a diagram, generate the prompt with render_tree.py lineage.json --format figure-prompt, rename its 〈待命名〉 swimlanes to your Step 3 branch names, and feed the whole output (prose body and structure checklist) to an image model. See examples/diffusion-models-figure-prompt.md. To render the image automatically, pipe it through an OpenAI-compatible image relay (中转站) running gpt-image-2 with scripts/gen_figure.py — default relay is ZenMux, --relay wegoo for the alternate (key/setup in image-relay.md):

python3 scripts/gen_figure.py lineage.json --out figure.png            # data → prompt → image (ZenMux)
python3 scripts/gen_figure.py <renamed-prompt>.md --out figure.png     # from a refined prompt
python3 scripts/gen_figure.py lineage.json --relay wegoo --out figure.png  # alternate relay

If no relay key is configured, gen_figure.py does not error — it saves the prompt to <out>.prompt.txt and prints two ways to proceed: supply a key (--api-key / $ZENMUX_API_KEY) and re-run, or paste that prompt into ChatGPT / a GPT image page (Midjourney / DALL·E) to draw it by hand.

Manual search passes (when the draft missed something)

python3 scripts/papers.py search "<direction>" --limit 30            # anchor
python3 scripts/papers.py search "<direction>" --precise --from-year 2024 \
    --sort citations --limit 15                                      # frontier
python3 scripts/papers.py --source arxiv search "<direction>" \
    --from-year 2025 --limit 15                                      # newest preprints
python3 scripts/papers.py search "<exact title>" --abstract --limit 1 # landmark
python3 scripts/papers.py resolve "<title A>" "<title B>" …          # ground a list
python3 scripts/papers.py expand <paperId> --limit 40                # lineage

--precise requires every term in title/abstract — use it whenever a relevance search returns off-topic giants. --from-year/--to-year window, --sort relevance|citations|recent. --source arxiv hits the newest preprints directly; resolve grounds free-text titles you (or WebSearch) found.

Robustness (so you fight the tool less)

• Duplicate-record-aware verification. OpenAlex often stores a paper under several work-ids, so "B cites A" can fail on an exact-id check even when the citation is real. verify.py now reconciles by normalized title / DOI before marking ⚠, and falls back to Semantic Scholar when OpenAlex's reference list is empty — so genuine citations stop showing up as false ⚠. A surviving ⚠ now much more likely means a real gap worth a look.
• Polluted/empty abstracts (e.g. an OpenAlex record whose abstract is some unrelated repo's README) are detected and repaired from Semantic Scholar, so your grounded summaries don't inherit garbage.
• Orphan auto-repair. genealogy.py now links orphan nodes to their real parents/children automatically (including the duplicate-record case) before writing the draft; --prune-orphans drops any that truly can't be linked. You should see far fewer orphans to fix by hand.
• arXiv recall for the frontier. Beyond OpenAlex/S2, a dedicated arXiv pass pulls the very newest preprints (which OpenAlex can lag months behind) so the "近两年前沿" section has fresh material. arXiv has no citation graph, so these hits enrich recall only: each is back-resolved to OpenAlex by title to recover real references, and any that are genuinely unindexed stay as ref-less _frontier_candidates you wire in by hand — the "edges are real citations" guarantee is never weakened.
• Tighter topic gate. Application/review papers from an adjacent domain (e.g. diffusion-for-materials when the field is image generation) that merely cite the core are now filtered as domain drift, so the pool stays on-topic.

Notes

• Scripts use only the Python stdlib. Default backend is OpenAlex (no key); set OPENALEX_MAILTO for its faster polite pool. Semantic Scholar is used automatically as a fallback for missing reference lists / abstracts; set S2_API_KEY to avoid its throttled keyless pool.
• Disk cache: API responses are cached under ~/.cache/research-genealogy (so draft → verify → re-run is fast and reproducible). RG_NO_CACHE=1 to disable, RG_CACHE_TTL=<seconds> to tune, RG_CACHE_DIR to relocate.
• genealogy.py --from-year/--to-year scopes the whole genealogy (e.g. only the last decade); --no-expand skips the snowball for a quick draft; --prune-orphans drops unlinkable orphan nodes.
• Regression guard: python3 scripts/selftest.py checks the examples' invariants offline; --online also re-verifies them and runs a live draft.
• Quality over coverage: a tight, correct trunk beats an exhaustive mess.