web-fetch-extract

name	web-fetch-extract
description	Generic web-data primitive. Fetches a URL via scripts/scrape.py and extracts structured data inline using the model's own reasoning over the fetched HTML. Use when any other skill or task needs data from a public web page - bios, leadership teams, prices, OpenGraph tags, page titles, recent posts, or "what does this page say about X". Triggered by natural-language page fetch asks and by other skills as a sub-step. Falls back to the WebFetch tool if scrape.py is missing or fails.
why	Pulls a page into context and extracts the answer without an LLM API call - extraction is the model's reasoning over the fetched text, so it works on a free plan and never sends the page to a paid endpoint.
enhance	Pick the lightest extract mode for the goal (meta, og, links, text) instead of dumping full HTML - it keeps the payload small and the extraction sharp, and only escalate to render mode when a first attempt comes back near-empty.
summary	Fetch a public web page and extract structured data from it inline.
allowed-tools	["Bash","Read","WebFetch"]
mcp_requirements	[]

Web Fetch + Extract

Runs on: local-exec - runs a local script; on a cloud surface I read the results, I do not run it.

Generic skill for pulling a URL into context and extracting whatever the calling skill or user asked for. No LLM API call - extraction is the model's reasoning over the fetched HTML or text. Works on a free plan.

Dependencies

scripts/scrape.py needs three Python packages for the default path, and one optional package for JS-rendered pages:

pip install httpx selectolax tenacity

Optional, only for --render (JS-walled pages):

pip install playwright
python -m playwright install chromium

The script imports Playwright lazily, so the three core packages are enough unless you actually pass --render. If a required package is missing, the script errors clearly and this skill falls back to the WebFetch tool.

Pre-flight

If scripts/scrape.py does not exist, fall back to the WebFetch tool for the whole flow and tell the user the script is missing.

Inputs

The skill receives two things:

1. URL - a single fully-qualified URL.
2. Extraction goal - natural language describing what to pull. Examples: "bio + role + recent posts", "leadership team names and titles", "title + meta description", "the three pricing tiers and what is in each".

Procedure

1. Decide render mode. Default to no JS render. Add --render only if the user explicitly says the page is JS-heavy or a first attempt returns near-empty HTML or text. Playwright is not installed by default. If --render is requested and Playwright is missing, the script errors clearly - fall back to WebFetch and note the gap.

2. Pick the lightest extraction mode for the goal.

   • Title + description / OG tags only: python scripts/scrape.py <url> --extract meta
   • OG-only payload: python scripts/scrape.py <url> --extract og
   • All outbound links (for example discovering a sitemap or About page): python scripts/scrape.py <url> --extract links
   • Text body (cleaned): python scripts/scrape.py <url> --extract text (optionally --selector "main" or similar)
   • Full HTML for messy parsing: