keyword-research-harvest

name	keyword-research-harvest
description	Use when the user wants a reusable local workflow to search scholarly APIs for any topic keywords, build a candidate table, download accessible PDFs or HTML/XML full texts, run a second-pass HTML-to-PDF attempt, and deduplicate the downloaded files. Best for keyword-driven research-article harvesting that should be reproducible and saved into a project folder.

Keyword Research Harvest

Use this skill when the user gives a topic or keyword set and wants a local literature-harvesting workflow, not a one-off manual search.

What this skill does

• Bundles its own literature_harvest/scripts stack, including search_pubmed.py, search_europepmc.py, search_crossref.py, search_openalex.py, merge_and_deduplicate.py, download_fulltexts.py, and harvest_utils.py.
• Searches PubMed/PMC, Europe PMC, Crossref, and OpenAlex through the bundled pipeline.
• Builds a no-dedup candidate table first.
• Downloads all legally accessible files to a new run folder.
• Saves HTML/XML when PDF is not directly available.
• Runs a second pass to chase PDF links from saved HTML pages.
• Produces a deduplicated file folder and manifest after downloading.

Dependency model

This skill is self-contained. The target project does not need to already contain:

• literature_harvest/
• search_pubmed.py
• download_fulltexts.py

The bundled copies live under:

• literature_harvest/scripts/

Files in this skill

• scripts/run_keyword_harvest_no_dedup.py Use this to launch a new broad keyword harvest run into a new run folder.
• scripts/continue_download_and_dedup.py Use this to resume pending downloads, try HTML-to-PDF second pass, and build a deduplicated download set.
• literature_harvest/scripts/ Bundled search and download dependencies copied from a working local harvest stack so the skill is portable.
• references/config_template.json Copy and edit this for the topic-specific query set and filtering terms.
• references/prompt_template.md Reusable prompt for another AI/agent.

Recommended workflow

1. Choose any output parent folder where the new harvest run folder should be created.
2. Copy references/config_template.json and edit it for the user's topic keywords.
3. Run scripts/run_keyword_harvest_no_dedup.py with:
   • --output-root
   • --config
   • --run-name
4. If the job is large, rerun with the same --run-name and --skip-search to avoid repeating API search.
5. Run scripts/continue_download_and_dedup.py --run-root <run-folder> --retry-failed.
6. Report:
   • candidate count
   • downloaded file count
   • true PDF count
   • HTML/XML count
   • remaining pending
   • deduplicated keep count

Constraints

• Do not silently drop failed downloads.
• Keep metadata even when download fails.
• Prefer original research when the config asks for it, but do not hard-code one domain such as Aspergillus.
• Do not claim HTML/XML is PDF.
• Treat second-pass cluster-like or support-like annotations as auxiliary only; downloading remains the primary task.

When to read references

• Read references/config_template.json before preparing a new run config.
• Read references/prompt_template.md when the user wants to hand this skill to another AI/agent.