| name | web-fetch-extract |
| description | Generic web-data primitive. Fetches a URL via scripts/scrape.py and extracts structured data inline using the model's own reasoning over the fetched HTML. Use when any other skill or task needs data from a public web page - bios, leadership teams, prices, OpenGraph tags, page titles, recent posts, or "what does this page say about X". Triggered by natural-language page fetch asks and by other skills as a sub-step. Falls back to the WebFetch tool if scrape.py is missing or fails.
|
| why | Pulls a page into context and extracts the answer without an LLM API call - extraction is the model's reasoning over the fetched text, so it works on a free plan and never sends the page to a paid endpoint. |
| enhance | Pick the lightest extract mode for the goal (meta, og, links, text) instead of dumping full HTML - it keeps the payload small and the extraction sharp, and only escalate to render mode when a first attempt comes back near-empty. |
| summary | Fetch a public web page and extract structured data from it inline. |
| allowed-tools | ["Bash","Read","WebFetch"] |
| mcp_requirements | [] |
Web Fetch + Extract
Runs on: local-exec - runs a local script; on a cloud surface I read the results, I do not run it.
Generic skill for pulling a URL into context and extracting whatever the calling skill or user asked for. No LLM API call - extraction is the model's reasoning over the fetched HTML or text. Works on a free plan.
Dependencies
scripts/scrape.py needs three Python packages for the default path, and one optional package for JS-rendered pages:
pip install httpx selectolax tenacity
Optional, only for --render (JS-walled pages):
pip install playwright
python -m playwright install chromium
The script imports Playwright lazily, so the three core packages are enough unless you actually pass --render. If a required package is missing, the script errors clearly and this skill falls back to the WebFetch tool.
Pre-flight
If scripts/scrape.py does not exist, fall back to the WebFetch tool for the whole flow and tell the user the script is missing.
Inputs
The skill receives two things:
- URL - a single fully-qualified URL.
- Extraction goal - natural language describing what to pull. Examples: "bio + role + recent posts", "leadership team names and titles", "title + meta description", "the three pricing tiers and what is in each".
Procedure
-
Decide render mode.
Default to no JS render. Add --render only if the user explicitly says the page is JS-heavy or a first attempt returns near-empty HTML or text. Playwright is not installed by default. If --render is requested and Playwright is missing, the script errors clearly - fall back to WebFetch and note the gap.
-
Pick the lightest extraction mode for the goal.
- Title + description / OG tags only:
python scripts/scrape.py <url> --extract meta
- OG-only payload:
python scripts/scrape.py <url> --extract og
- All outbound links (for example discovering a sitemap or About page):
python scripts/scrape.py <url> --extract links
- Text body (cleaned):
python scripts/scrape.py <url> --extract text (optionally --selector "main" or similar)
- Full HTML for messy parsing:
