Route web-scraping work into the lightest workable Scrapling mode instead of defaulting to a browser. Use when the user needs HTML extraction, JS-rendered page retrieval, protected-target escalation, quick CLI scraping, agent-facing MCP access, or a larger crawl with Scrapling spiders. Triggers on: scrapling, scrape website, crawl site, adaptive scraping, selector drift, stealthy fetch, browser scraping, scrape to markdown, scrapling mcp, scrapling spider, research harvesting, literature scraping, paper metadata.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Route web-scraping work into the lightest workable Scrapling mode instead of defaulting to a browser. Use when the user needs HTML extraction, JS-rendered page retrieval, protected-target escalation, quick CLI scraping, agent-facing MCP access, or a larger crawl with Scrapling spiders. Triggers on: scrapling, scrape website, crawl site, adaptive scraping, selector drift, stealthy fetch, browser scraping, scrape to markdown, scrapling mcp, scrapling spider, research harvesting, literature scraping, paper metadata.
allowed-tools
Bash Read Write Edit Glob Grep WebFetch
compatibility
Requires Python 3.10+ and works best in a virtual environment. Browser-backed fetchers, CLI shell, MCP, and spiders require extras plus `scrapling install` for browser dependencies.
Use a virtual environment for the Python package unless the user explicitly wants a system install.
bash scripts/install.sh --profile parser
bash scripts/install.sh --profile fetchers
bash scripts/install.sh --profile shell
bash scripts/install.sh --profile ai
bash scripts/install.sh --profile all
scripts/install.sh performs both steps together by default; use SKIP_SKILL=1 to install only the Python package, or SKIP_PACKAGE=1 to register only the plugin (see the script header for GLOBAL/AGENTS knobs).
Installation guidance:
parser — local HTML parsing only
fetchers — HTTP + browser-backed fetchers
shell — interactive CLI shell workflows
ai — MCP server workflows
all — only when the user truly needs the full surface
Browser-backed flows require scrapling install. Parser-only workflows do not.
Step 3: Choose the lightest extraction path that fits
A. Parser-only path
Use this when HTML is already available or the user mainly needs selector logic.
Treat anti-bot handling as a documented capability, not a guarantee
If a target still fails or requires stronger infra, route honestly to proxies or managed unblockers instead of pretending one more flag will always fix it
Step 7: Run the research-harvesting gate for literature/dataset targets
Use this step whenever the target is scientific literature, citation metadata,
or a dataset — not general web content — following the deterministic,
script-backed skill pattern from
ai4s-research/open-science.
The gate is stdlib-only and prints one ```review fenced JSON block
flagging a robots.txt disallow/crawl-delay and a known-paywalled-publisher
heuristic. It never claims a scrape is authorized — resolve or explicitly
accept every finding before harvesting. Full pattern and hand-off guidance
live in references/research-harvesting.md.
Examples
Example 1: I already have HTML and just need selectors
Use Selector, not a browser.
Example 2: A marketing page loads over plain HTTP
Start with Fetcher or scrapling extract get, then escalate only if evidence says the content is missing.
Example 3: A React dashboard is empty until JS runs
Move to DynamicFetcher and use network_idle or wait_selector.
Example 4: A protected target returns a challenge page
Try StealthyFetcher, but make it explicit that stronger proxy/unblocker infrastructure may still be required.
Example 5: An agent needs bounded web extraction
Expose Scrapling through MCP instead of forcing a handwritten scraper first.
Example 6: The task has become a real crawl
Route from repeated fetches into spiders with checkpointing and session control.
Example 7: The target is a paper, dataset, or citation source
Run scripts/research_harvest_gate.py first, resolve/accept its findings, then
prefer a metadata API (Crossref/OpenAlex/Semantic Scholar) over scraping the
rendered abstract page.
Best practices
Start with the smallest viable mode and escalate only on evidence.
Prefer direct API/JSON access when that is the real source of truth; do not force DOM scraping for browser-heavy apps.
Reuse sessions for repeated requests instead of relaunching browsers on every page.
Use adaptive scraping for selector drift, not as a blanket fix for every failure mode.
Prefer CSS selectors plus .md or .txt outputs to keep model context smaller.
Treat CLI, MCP, and spiders as operator surfaces layered on top of the same routing logic, not three unrelated features.
Route honestly to Playwright/Crawlee-style automation, external unblockers, or another extraction service when Scrapling is no longer the best fit.
Run the research-harvesting gate before scraping scientific literature, datasets, or citation sources — never treat a clean gate result as proof of authorization.
