oa-literature-harvester

name

oa-literature-harvester

description

Use when the user wants Codex to search, download, audit, or resume a corpus of open-access academic literature by topic, field, keywords, year range, or target size. Trigger this skill for requests such as “帮我搜这个领域的 OA 文献”, “按关键词下载 PDF”, “从 2000 到现在持续采集”, “帮我统计真实 PDF 数量”, “帮我定向找 NCS 和领域顶刊文献”, or when Codex should operate the local `oa-literature-downloader` project and return corpus-quality metrics.

OA Literature Harvester

Wrap the local project at /Users/linyong/vscode/oa-literature-downloader into a repeatable harvesting workflow. Use the bundled scripts to launch or resume harvesting, then audit the output and report actual corpus quality instead of only quoting downloader logs.

This skill now supports two distinct modes:

1. broad OA recall for corpus building
2. high-precision claim-targeted search for review writing and citation mapping

It also supports an arXiv-first collection path for fast-moving CS topics where recent preprints are more useful than strict OA journal recall.

Source Routing Rule

Choose sources by field instead of using one source pack for everything.

Default routing:

1. computer science / AI / ML / LLM / agent / reasoning topics:
   • arXiv-first
   • then add selective venue filtering or handpicked top-conference papers when needed
   • use OpenAlex or Crossref only as backfill
2. natural science / earth science / remote sensing / biology / medicine / interdisciplinary domain science:
   • NCS / top-tier first
   • then add high-citation field journals
   • then use broader OA recall only to fill real gaps

Operational interpretation:

• CS topics move fastest on preprints, so freshness and method recall matter more than journal OA coverage
• non-CS domain reviews usually need stronger venue control, so start from Nature / Cell / Science families or top field journals before broadening
• do not mix these defaults unless the user explicitly asks for both

The high-precision path is the first-stage literature landing workflow. Its job is to produce a defendable, reusable paper set on disk before deeper writing work starts.

This skill also supports the full downstream review pipeline after landing:

1. literature collection
2. literature analysis
3. literature briefing / PPT production

In practice, this should be treated as one end-to-end chain:

1. collect papers
2. land PDFs
3. analyze papers
4. build references
5. produce review prose or briefing slides

Quick Start

Set these once:

export CODEX_HOME="${CODEX_HOME:-$HOME/.codex}"
export OA_SKILL="$CODEX_HOME/skills/oa-literature-harvester"
export OA_PROJECT="/Users/linyong/vscode/oa-literature-downloader"

Run a harvest:

python3 "$OA_SKILL/scripts/run_harvest.py" \
  --topic "glacier remote sensing" \
  --start-year 2000 \
  --end-year 2025 \
  --target-dir "/absolute/output/dir"

Run an arXiv-first harvest:

python3 "$OA_SKILL/scripts/run_arxiv_harvest.py" \
  --topic "code augmented math reasoning" \
  --query "program aided language models math reasoning" \
  --query "mathcoder mathematical reasoning code execution" \
  --start-year 2021 \
  --end-year 2026 \
  --target-dir "/absolute/output/dir"

Recommended source choice examples:

• 代码增强数学推理:
  • arXiv-first
• 遥感冰川近五年顶刊:
  • local NCS cache first
  • then Crossref / top field journals
• 某交叉学科新方向综述:
  • if method-heavy CS contribution dominates, start with arXiv
  • if domain-evidence quality dominates, start with NCS + field journals

Audit an existing output directory:

python3 "$OA_SKILL/scripts/audit_output.py" \
  --output-dir "/absolute/output/dir"

High-precision search for review writing:

python3 /Users/linyong/vscode/oa_review_workflow/scripts/precision_harvest.py \
  --outline "/absolute/path/to/query_plan.md" \
  --start-year 2020 \
  --end-year 2026 \
  --max-results 8 \
  --pages 2 \
  --source-pack ncs \
  --source-pack cryosphere_core \
  --output-dir "/absolute/output/dir"

This is project-agnostic. You can point --outline and --output-dir to any other workspace.

Local NCS Cache First

For the glacier remote sensing continuation workflow in /Users/linyong/vscode/oa_review_workflow, do not start from OpenAlex or a fresh Crossref crawl if the request is about:

• NCS supplementation
• Nature / Science / Cell family papers
• glacier remote sensing top-tier screening
• “近五年子刊以上” style expansion

Check the local pool first:

1. full Crossref pool: /Users/linyong/vscode/oa_review_workflow/artifacts/harvest_runs/ncs_crossref_pool_20260401/crossref_ncs_merged/crossref_ncs_pool_merged_20210101_to_20261231.json
2. glacier subset: /Users/linyong/vscode/oa_review_workflow/artifacts/harvest_runs/ncs_crossref_pool_20260401/crossref_ncs_merged/crossref_ncs_pool_glacier_20210101_to_20261231.json
3. glacier remote sensing coarse subset: /Users/linyong/vscode/oa_review_workflow/artifacts/harvest_runs/ncs_crossref_pool_20260401/crossref_ncs_merged/crossref_ncs_pool_glacier_remote_sensing_20210101_to_20261231.json
4. refined shortlist: /Users/linyong/vscode/oa_review_workflow/artifacts/harvest_runs/ncs_crossref_pool_20260401/crossref_ncs_refined/crossref_ncs_glacier_remote_sensing_refined_20210101_to_20261231.json
5. grouped reading outputs: /Users/linyong/vscode/oa_review_workflow/artifacts/process_notes/ncs_glacier_extension_20260401/glacier_direction_groups/
6. local PDFs: /Users/linyong/vscode/oa_review_workflow/papers_pdf/ncs_glacier_remote_sensing_refined_20260401/

Operational rule:

• local metadata lookup first
• local PDF reconciliation second
• external precision harvest third
• fresh Crossref backfill only when the local pool is clearly insufficient

For this project line, prefer the local pool as the reusable source of truth for top-tier screening, then use external APIs only to fill real gaps.

Chinese digest from high-precision results:

export OPENAI_API_KEY="..."
python3 /Users/linyong/vscode/oa_review_workflow/scripts/build_chinese_digest.py \
  --input "/absolute/path/to/precision_harvest.json" \
  --output "/absolute/path/to/digest.md" \
  --top-per-query 8 \
  --top-total 88 \
  --use-openai \
  --model openai/gpt-5.4-mini \
  --base-url https://openrouter.ai/api/v1

This is also project-agnostic. The only required inputs are a precision_harvest.json file and an output markdown path.

Research-problem matrix from the digest:

export OPENAI_API_KEY="..."
python3 /Users/linyong/vscode/oa_review_workflow/scripts/build_research_matrix.py \
  --input "/absolute/path/to/digest.md" \
  --output "/absolute/path/to/research_matrix.md" \
  --use-openai \
  --model openai/gpt-5.4-mini \
  --base-url https://openrouter.ai/api/v1

This turns the landed literature set into a structured reading layer:

• paper
• scientific question
• difficulty
• technical route
• required data / conditions

Problem-grouped reading file from the matrix:

• group by scientific problem, not by journal or year
• for each problem group:
  • explain what this class of papers is broadly trying to solve
  • then list each paper with:
    • problem
    • difficulty
    • route
    • required data / conditions
    • representative figure if available
    • citation / reference line if available

This grouped file is the bridge from “reading” to “presentation”.

Paper-level deep reading for a focused landed batch:

• first normalize the batch into canonical and duplicates
• build a manifest for the canonical set
• if local PDFs already exist, parse them first instead of staying at abstract level
• prefer MinerU when available; otherwise use the local figure extraction path
• extract representative figures
• then generate:
  • a quick-reading markdown pack
  • per-paper deep summaries
  • multimodal figure backfill for the 配图讲解 section

Recommended local commands in oa_review_workflow:

export OPENAI_API_KEY="..."
export OPENAI_BASE_URL="https://openrouter.ai/api/v1"
export OPENAI_MODEL="openai/gpt-4o"
python /Users/linyong/vscode/oa_review_workflow/scripts/build_deep_paper_summaries.py
python /Users/linyong/vscode/oa_review_workflow/scripts/augment_figure_analysis.py

This is appropriate when the user already has a small, concrete PDF pack and wants “逐篇精读 + 图也讲解”.

Hard rule for landed local PDFs:

1. normalize canonical PDFs
2. run MinerU / PDF-to-Markdown parsing if configured
3. extract representative figures
4. run multimodal figure interpretation if the user wants figure explanation
5. only then write grouped reading notes or slides

Do not downgrade to abstract-only slides when local PDFs and usable figures already exist. Abstract/citation cards are fallback only for papers that are strategically important but still missing local PDF/figure assets.

Ongoing organization rule

For this workflow, do not wait until the end to clean up materials. During collection, reading, and PPT production, keep materials continuously organized, registered, and reusable.

Required behavior:

1. as soon as a paper becomes part of the working set, record where its assets live:
   • source PDF path
   • parsed markdown path if available
   • figure directory if available
   • whether it entered the deck, notes, or proposal draft
2. keep filenames and directories predictable:
   • avoid scattering the same deck or paper assets across unrelated working directories
   • prefer keeping compile products beside the deck source unless the user explicitly requests another location
3. when a reading note, grouped synthesis file, or proposal draft becomes part of the real workflow, place it in a stable project location instead of leaving it as an ad hoc scratch file
4. after meaningful work, update the local progress or process-notes log if the workspace already has one:
   • record what was changed
   • record which files are now the source of truth
   • record any missing PDFs or unresolved gaps
5. for future reuse, think in terms of handoff packages:
   • deck assets
   • paper PDFs
   • grouped reading notes
   • proposal / problem-derivation notes

The point is that the workflow should stay auditably organized while work is happening, not only after the user asks for cleanup.

PPT Guardrails

When this skill is used to produce literature briefing slides, treat the deck as an audience-facing talk, not as an internal working note.

Required defaults:

1. keep the four main directions explicitly visible near the start:
   • 冰川识别与制图
   • 冰川变化监测
   • 冰川变化机理
   • 冰川变化影响与风险
2. the second content page should normally be a directory / orientation page:
   • list the four directions
   • explain them in plain-language scientific-question form
3. each paper slide should default to this structure:
   • 问题
   • 方案
   • 数据
   • 结果
4. do not write presenter-facing meta language into slides:
   • avoid phrases like “这一页图的意义是告诉听众”
   • avoid “这里要强调” or similar speaker notes in the slide body
   • use neutral audience-facing wording such as “图示：区域整体加速退缩与空间分异”
5. prefer figures, maps, profiles, charts, and model outputs from the paper itself:
   • do not default to lab photos, field photos, or author portraits when a scientific figure is available
6. portrait / narrow figures should be given enough space:
   • reduce wasted top whitespace
   • enlarge the figure region rather than shrinking the panel into a small corner
7. if a deck is for external briefing, optimize for readability by others:
   • fewer internal workflow notes
   • cleaner captions
   • clearer section logic
8. after major slide revisions, save a dated, versioned copy instead of only overwriting the main file:
   • include date and version in the filename
   • for example: deck_20260401_v2.tex and deck_20260401_v2.pdf
   • keep the stable main filename only when the user explicitly wants in-place overwrite

Beamer layout defaults for literature briefing decks

For the glacier remote sensing briefing line, keep these concrete Beamer lessons in mind:

1. do not use a fixed-height frametitle box when titles can wrap to two lines:
   • avoid templates like ht=... for the title row
   • use an auto-height title block so two-line titles do not collide with the top navigation bar
2. reserve explicit horizontal space for the university logo:
   • place the logo in its own right-side box or minipage
   • do not let the logo float via \hfill after the title text, or it will drift when titles wrap
3. keep the title close to the navigation bar, but not touching it:
   • prefer a small positive top offset
   • then fine-tune logo position independently by nudging the logo box upward a little
4. logo sizing should be conservative:
   • default to a moderate size rather than a dominant badge
   • if the logo feels visually heavy, reduce logo width before shrinking title text
5. when 问题 / 方案 / 数据 / 结果 figure cards feel crowded, do not solve it by globally shrinking everything first:
   • first reduce wasted top whitespace
   • then slightly rebalance text and figure heights
   • only one or two genuinely dense transition slides should drop to a smaller local font
6. prefer this order of adjustment for crowded figure cards:
   • move the title band upward a little
   • shrink the logo a little
   • slightly reduce figure height
   • then enlarge the left text block by one step if space allows
7. readability target:
   • figcard text should usually stay at \scriptsize or \footnotesize
   • avoid \tiny as a global fix unless the user explicitly accepts a dense handout style
8. compile in the source deck directory, not an unrelated working directory:
   • keep .aux, .nav, .log, .xdv, and .pdf beside the source deck unless the user asks otherwise
   • do not leave duplicate compile artifacts in another workspace
9. if the user wants archival outputs, save a dated and versioned PDF in the same source directory:
   • follow a pattern like 20260402_v05_ncs_glacier_remote_sensing_slides.pdf
   • keep the source tree tidy and predictable

Summary / transition slides

When adding a slide that bridges from reviewed papers to a new scientific question or proposal, do not write it as a chatty recap.

Preferred structure:

1. use two columns, but not necessarily symmetric
2. if the right column is the real payload, use an asymmetric split such as:
   • left 0.34
   • right 0.62
3. left column should be short:
   • label each paper by a short title phrase from the paper title
   • do not waste space on author names
   • do not repeat citation numbering in both columns
   • only state what that paper already accomplished
4. right column should carry the analytical weight:
   • write the limitation, uncertainty, missing process, missing constraint, or future need that is explicit in the paper or close reading
   • this side is usually more important and can be longer than the left
5. bottom line should be a single concise synthesis sentence:
   • “据此，我们的问题是……”
   • keep it short enough to avoid clipping at the page bottom
6. remove low-value slide clutter:
   • do not add “后面引用顺序”
   • do not repeat author-year keys when the page is about synthesis rather than bibliography
7. if space is tight, prefer:
   • narrower left column
   • smaller local font on that slide
   • reduced item spacing
   • shorter synthesis sentence before deleting important analytical content from the right column

Writing rule:

• the problem statement on these transition slides should come from what the original paper explicitly says is still limited, uncertain, missing, under-observed, under-modelled, or still needed
• do not build the slide mainly from Codex-side synthetic “gaps between papers” unless the user explicitly wants a higher-level synthesis

Post-PPT study-pack handoff

After a literature briefing deck is finished, assume the user may want a reusable personal study pack. When appropriate, prepare or be ready to prepare a package that bundles the deck with the materials that produced it.

Default study-pack contents:

1. deck source and exported PDF:
   • current .tex
   • current main .pdf
   • dated / versioned .pdf if one was produced
2. local source PDFs for the papers actually used in the deck:
   • copy them into one clearly named folder
   • rename them into readable filenames if the repository names are hash-heavy or downloader-style
3. reasoning / synthesis markdown files that explain how the deck was built:
   • grouped reading notes
   • direction summaries
   • “what these papers already did”
   • “extension space / evidence”
   • “how we get to our proposal / plan”
4. a short README for the package:
   • explain the folder structure
   • note any paper that appears in the deck but still lacks a confirmed local PDF
   • point to the current source-of-truth deck file

Packaging rules:

1. create a clearly named folder in the project tree, typically date-prefixed and purpose-specific
2. separate subfolders for:
   • PPT
   • PDFs
   • plan / evidence markdown
3. do not just copy everything blindly:
   • include the files actually used
   • call out missing assets explicitly
4. if the workspace has a progress log, record that the study pack was produced and where it lives

This handoff step matters because the deck is not the only deliverable. The user may later want to read the papers, inspect the evidence chain, or reuse the proposal logic without reconstructing the entire workflow.

Workflow

Three-Stage SOP

For review-oriented projects, the default large-grain workflow should be:

1. literature collection
2. literature analysis
3. literature briefing

Concretely:

1. Collection

• high-precision retrieval
• PDF landing
• PDF recovery

2. Analysis

• digest
• research matrix
• grouped-by-problem reading file
• MinerU / PDF-to-Markdown parsing whenever local PDFs already exist
• figure extraction
• deep per-paper reading pack
• multimodal figure interpretation backfill
• refs normalization

3. Briefing / Writing

• citation mapping
• prose revision
• slide production
• PDF export

1. Translate the user's request into:

• topic
• start_year / end_year
• max_results per year; use 0 for no cap
• target_dir
• whether to use --loop-years and --resume

If the user already has a review outline or draft, also translate it into:

• claim blocks
• query plans
• target journal tier
• whether the next step is corpus expansion or citation mapping
• where the first-stage landed literature should be stored on disk

2. Prefer the existing downloader backend instead of reimplementing logic.

• Main project: /Users/linyong/vscode/oa-literature-downloader
• Use main.py for harvesting
• Use this skill's audit_output.py to verify real PDF counts

3. Default to resumable year-by-year collection for anything larger than a quick sample.

• Use --loop-years
• Use --resume
• Keep downloader concurrency on
• Report both downloader totals and audited valid-PDF totals

4. Never present raw downloader “success” as final truth.

• Always audit the output directory afterward
• Distinguish:
  • discovered papers
  • unique papers after dedupe
  • files on disk
  • valid PDFs
  • invalid PDFs
  • failed/missing items

5. Separate broad recall from high-precision retrieval.

• Broad recall is for corpus building.
• High-precision retrieval is for first-stage landed literature and later claim support.
• Do not use broad noisy search results directly as paragraph evidence in a review.

6. Separate reading layers after the first-stage literature is landed.

• The landed literature set on disk is the source of truth.
• The Chinese digest is the fast-reading layer.
• The research-problem matrix is the structured synthesis layer.
• Formal references must still be checked against DOI or official publisher metadata before entering refs.bib.

7. Separate review production from reading production.

• Reading production is for understanding the corpus.
• Review production is for strengthening prose and citations.
• Presentation production is for speaking and visual briefing.
• Do not mix these outputs into one file by default.

8. For user-supplied PDF batches, normalize the filesystem first.

• Create a canonical set before running deep-reading steps.
• Keep duplicates in a separate directory.
• Build later reading products from a manifest, not by trusting whatever filenames happen to be present.

Decision Rules

• For a small exploratory request, start with a narrow run:
  • 1 to 3 years
  • 50 to 200 max results per year
  • confirm quality before scaling up
• For a large corpus request, use:
  • --loop-years
  • --resume
  • --max-results 0
  • --browser-fallback
  • nontrivial --download-rounds
• If the user asks for “几万篇”, do not promise the number from a single query.
  • Run
  • audit
  • diagnose bottlenecks
  • iterate query strategy or downloader strategy
• If the user wants “高质量”“顶刊”“NCS”“可直接补到综述里”, switch to the high-precision path first.
• If the user is working inside /Users/linyong/vscode/oa_review_workflow and asks for NCS supplementation, inspect the local NCS Crossref pool before launching a new search.
• If the user already has draft text, prioritize finding evidence for the first weak paragraph before expanding the whole corpus again.
• If the user is in a different domain, keep the workflow but replace the query plan and source packs accordingly.

Query Strategy

• Start with the user's own wording as the base query.
• If the field is broad, consider multiple passes with adjacent queries instead of one oversized query.
• Merge by OpenAlex ID / DOI when comparing runs.
• Expect OpenAlex relevance to drift; audit topicality separately from download validity.

For high-precision review writing:

• derive queries from paragraph claims, not only from section titles
• prefer English technical query strings even if the draft is in Chinese
• target high-value sources first
• treat quantity as secondary to claim fit
• save the first-stage result set as a stable artifact that later sessions can reopen directly

For structured literature reading after landing:

• first build a digest so the user can skim authors, journals, abstracts, and keywords
• then build a research matrix that extracts:
  • scientific question
  • difficulty
  • technical route
  • required data / conditions
• if the user wants “按科学问题整理”, group papers by research problem instead of by journal or year
• within each problem group, write:
  • what this class of papers is broadly trying to solve
  • then list each paper with its question, difficulty, route, and data dependence
• this grouped matrix is for reading and strategy, not yet for direct citation insertion

For figure-heavy deep reading:

• do not claim “图像分析” if you only used figure captions or nearby text
• if the user explicitly wants image interpretation, send the image files themselves to a multimodal model
• if local PDFs already exist, prefer MinerU or equivalent PDF-to-Markdown parsing before any grouped writing
• if local PDFs already exist, representative figures should be extracted before slide writing begins
• the safest upgrade path is:
  • generate the main per-paper summary first
  • then rewrite only the figure-analysis section

For presentation production after analysis:

• first define the stable scientific-problem buckets
• for each bucket, prepare:
  • one overview slide:
    • what problem this direction studies
    • which 3 to 4 representative papers matter most
    • what journal tier those papers belong to
  • then one detail slide per representative paper:
    • what problem it solves
    • what method / route it uses
    • what data or mathematical setup it relies on
    • what it accomplishes
    • what the representative figure shows
    • the full reference line at the bottom
• for beamer / LaTeX slide production:
  • prefer one stable slide template and reuse it
  • do not let multiple papers share one figure
  • do not let placeholder labels like paper 1 survive into the final deck
  • if a title collides with the beamer navigation bar, move the frametitle block downward instead of shortening the science content first
  • for title pages, treat the cover as a separate layout problem; do not blindly inherit the same navigation/header treatment
  • if a local PDF and figure exist, the default detail-slide layout should be:
    • left: problem / method / data / completed work
    • right: representative figure + what it shows
  • only use abstract evidence cards on the right when the paper still has no local PDF or no recoverable figure

Preferred source packs:

• ncs
  • Nature
  • Science
  • Cell
  • Nature Communications
  • Nature Geoscience
  • Nature Reviews Earth & Environment
  • Nature Climate Change
  • Communications Earth & Environment
  • Science Advances
  • Proceedings of the National Academy of Sciences
• cryosphere_core
  • Remote Sensing of Environment
  • ISPRS Journal of Photogrammetry and Remote Sensing
  • The Cryosphere
  • Earth System Science Data
  • Scientific Data
  • Geophysical Research Letters
  • Journal of Glaciology
  • Annals of Glaciology
  • IEEE Transactions on Geoscience and Remote Sensing
  • Journal of Geophysical Research: Earth Surface

Important implementation detail:

• OpenAlex /sources resolution may time out under unstable network conditions
• prefer built-in source-id packs when possible
• use --allow-source-search only when adding new journals not already covered by the packs
• if /sources times out, this is not proof the proxy is missing; it is often an OpenAlex handshake or route instability issue

Generalization rule:

• do not hardcode project-specific institutions, teams, or field clues into the skill logic
• if a project has domain-specific institutions or datasets, treat them as query-plan content in that project, not as universal skill behavior

See workflow.md for parameter mapping and reporting format.

PDF Recovery

Do not assume that a missing local PDF means the paper is not open access. In practice, a common failure mode is:

• OpenAlex marks the work as OA
• the stored pdf_url points to a publisher direct link
• the direct link returns 403, HTML, or a non-PDF placeholder

This showed up repeatedly with:

• Wiley / AGU pdfdirect
• PNAS direct PDF routes
• repository landing pages that require one extra hop before the actual PDF

When this happens, run a recovery pass instead of stopping at the first failed download.

Recovery Order

1. Reconcile manifest against disk first.

• The manifest may be stale while the PDF is already on disk.
• Match by:
  • exact filename
  • DOI-hash suffix
  • normalized year + source + title prefix

2. Query OpenAlex locations for each missing DOI.

• Do not rely only on best_oa_location.
• Inspect all locations.
• Prefer in this order:
  • direct repository PDF
  • direct publisher PDF that actually returns %PDF-
  • repository landing page that can be scraped for a PDF link
  • DOI landing page only as a last resort

3. Scrape repository landing pages for hidden PDF links.

• Check for:
  • citation_pdf_url
  • og:pdf
  • href="...pdf"
  • site-specific /download routes
• Normalize HTML entities and relative URLs.

4. Validate every recovered file.

• A successful HTTP response is not enough.
• Reject files that:
  • are too small
  • do not start with %PDF-
  • are clearly HTML error pages

5. Persist progress incrementally.

• Update manifest JSON and CSV after each recovered paper.
• Do not wait until the end of a long recovery run.
• Otherwise a slow repository page can waste the whole pass.

Practical Rules

• Treat publisher 403 as a recovery trigger, not as final proof that the PDF is unavailable.
• For AGU / Wiley papers, check OpenAlex locations for:
  • institutional repositories
  • accepted manuscripts
  • mirrored OA copies
• For Elsevier papers with empty pdf_url, OpenAlex may still list a repository PDF under another location.
• For PNAS, PMC may expose a usable article page and PDF path even when the publisher PDF blocks scripted access.
• Expect some repository hosts to fail because of network or proxy quirks; mark these as environment-limited, not definitively unavailable.

Reporting

After recovery, separate:

• landed local PDFs
• still missing but likely recoverable
• currently blocked by publisher or repository access
• no working fallback found yet

Do not flatten these into one generic “failed” bucket.

Reference Implementations

Useful scripts from this workspace:

• /Users/linyong/vscode/oa_review_workflow/scripts/download_precision_pdfs.py
• /Users/linyong/vscode/oa_review_workflow/scripts/reconcile_pdf_manifest.py
• /Users/linyong/vscode/oa_review_workflow/scripts/recover_missing_pdfs.py

In other projects, reuse the workflow even if script paths differ:

1. land first-pass PDFs
2. reconcile manifest against disk
3. recover missing PDFs from OpenAlex locations
4. validate PDF headers
5. only then report the true landed corpus

Output Contract

When using this skill, always report:

• query / year range / output dir
• total discovered papers
• unique papers after dedupe
• downloader-reported success/fail counts
• audited valid-PDF count
• audited invalid-PDF count
• whether the result quality is good enough to scale further

When using the high-precision path, also report:

• which claim block or paragraph the search is serving
• which source pack or source ids were used
• how many results were kept after reranking and dedupe
• where the landed literature files were saved
• which items are strongest candidates for citation mapping
• whether the result set is sufficient to revise the target paragraph

If you additionally build the research matrix, report:

• which digest file was used as input
• how many rows were extracted into the matrix
• whether the matrix is grouped by paper list or by research problem
• where the grouped reading file was saved
• whether any placeholder or non-paper entries were excluded before grouping

If you continue into grouped analysis and presentation production, also report:

• where the grouped-by-problem reading file was saved
• whether representative figures were linked for each key paper
• where the slide source file was saved
• where the rendered PDF deck was saved
• whether the deck is a reading deck, a briefing deck, or a condensed talk deck

Guardrails

• Treat this as OA-only harvesting. Do not claim coverage of paywalled literature.
• Use absolute output paths.
• Prefer incremental scaling over one giant blind run.
• Preserve existing output directories; do not delete prior corpora unless the user explicitly asks.
• If a large run underperforms, keep iterating on query and download strategy instead of stopping at the first result.
• Do not confuse “many hits” with “strong evidence”.
• Do not let the model rewrite large parts of a review before evidence has been anchored.
• Treat the first-stage landed literature set as a reusable artifact for the current or any later project step.
• Prefer this execution order for review work:
  1. identify the first weak paragraph
  2. derive targeted queries
  3. retrieve high-quality evidence
  4. build a Chinese digest if needed
  5. perform citation mapping
  6. revise the paragraph
• Only after one paragraph is demonstrably stronger should the same process be scaled to later sections.
• For PPT work, do not jump straight from raw harvest results to slides. Required order is:
  1. land literature
  2. build digest
  3. build matrix
  4. group by scientific problem
  5. if PDFs exist, run MinerU / local PDF parsing first
  6. choose representative papers and figures
  7. build the slide deck

Resources

• Run wrapper: scripts/run_harvest.py
• Output auditor: scripts/audit_output.py
• Workflow notes: workflow.md
• High-precision search script: /Users/linyong/vscode/oa_review_workflow/scripts/precision_harvest.py
• Chinese digest script: /Users/linyong/vscode/oa_review_workflow/scripts/build_chinese_digest.py
• Capability notes: /Users/linyong/vscode/oa_review_workflow/docs/high_precision_capability.md
• Research matrix script: /Users/linyong/vscode/oa_review_workflow/scripts/build_research_matrix.py
• Figure extraction output example: /Users/linyong/vscode/oa_review_workflow/me/主图提取总览_main_65.md
• Grouped analysis example: /Users/linyong/vscode/oa_review_workflow/me/顶刊问题-难点-路线矩阵_按科学问题逐篇整理版.md
• Slide source example: /Users/linyong/vscode/oa_review_workflow/me/glacier_review_slides.tex
• Slide PDF example: /Users/linyong/vscode/oa_review_workflow/me/glacier_review_slides.pdf

Reuse In Other Projects

To use this skill in another project:

1. prepare a new outline or query-plan markdown in that project
2. choose a new project-specific output directory
3. run precision_harvest.py against that outline
4. optionally run build_chinese_digest.py on the resulting JSON
5. optionally run build_research_matrix.py
6. group the matrix by scientific problem
7. use the landed literature set for:
   • citation mapping
   • review drafting
   • presentation slide production

Only these inputs should usually change between projects:

• outline / query-plan file
• year range
• output directory
• source packs or extra journals
• digest output path
• grouped-analysis output path
• slide source / slide output path