webclaw

# webclaw Web extraction for AI agents, powered by a local Rust engine. It turns any URL into clean Markdown, text, or JSON through the webclaw MCP server, and it works with no API key. ## Install ``` npx create-webclaw ``` This writes the `npx @webclaw/mcp` MCP config into your agent (Claude Code, Cursor, Windsurf, Codex, Antigravity, and more) — nothing to install; the server is fetched and cached on first launch. Restart the agent to load the tools below. - **No key needed.** Extraction runs on your machine: free, private, and unlimited for the common case. Static sites, docs, blogs, server-rendered pages, and product pages all work locally. - **Optional upgrade.** Set `WEBCLAW_API_KEY` (from https://webclaw.io) and webclaw escalates to the hosted engine for the pages local extraction can't finish: bot-protected sites and JavaScript-rendered SPAs. Without a key, those pages return a clear message that tells you how to unlock them. ## When to use this skill - You need reliable web content with clean, structured output. - `web_fetch` returns empty, truncated, or blocked content. - You need structured data (pricing tables, product specs, contact info) as JSON. - You need to crawl a whole site or discover every URL. - You need LLM-optimized content, cleaner than raw markdown. - You need a site-specific extractor (GitHub, Reddit, YouTube, npm, PyPI, Amazon). - You need to summarize, diff, or search the web. ## Tools All tools run locally with no key unless noted. Output formats: `markdown` (default), `text`, `llm` (adds a title and URL header with clean link references, best for feeding to a model), and `json` (full metadata). ### `scrape`: extract a single URL `url` (required), `format`, `include_selectors`, `exclude_selectors`, `only_main_content`, `browser` (`chrome` | `firefox` | `random`), `cookies`. YouTube `watch`, `shorts`, and `youtu.be` URLs also return a transcript and a video-metadata block alongside the content. ### `crawl`: scrape an entire site `url`, `depth` (default 2), `max_pages` (default 50), `concurrency` (default 5), `use_sitemap`, `format`. ### `map`: discover URLs `url`. Sitemap-first discovery, with a bounded same-origin crawl fallback when the sitemap is thin. ### `batch`: many URLs in parallel `urls` (array), `format`, `concurrency` (default 5). ### `extract`: structured data via LLM `url`, plus either `prompt` (natural language) or `schema` (a JSON schema). See **LLM setup** below. ### `summarize`: quick summary `url`, `max_sentences` (default 3). See **LLM setup** below. ### `diff`: detect content changes `url`, `previous_snapshot` (a prior extraction as JSON). Compares at the extracted-content level rather than raw HTML. ### `brand`: visual identity `url`. Returns colors, fonts, logo, and favicon. ### `search`: web search `query`, `num_results` (≤10), `country`, `lang`, `scrape` (also fetch and extract each result page). Uses **your own** `SERPER_API_KEY` (free at serper.dev) locally, and falls back to the hosted API when unset. ### `vertical_scrape`: typed JSON for a specific site `name` (extractor name), `url`. Returns typed fields (title, price, author, rating) instead of generic markdown, or a clear "URL mismatch" error when the URL doesn't fit the extractor. Verticals for protected sites (Amazon, eBay, Etsy, Trustpilot) need `WEBCLAW_API_KEY`; without one they return a message asking you to set it. ### `list_extractors`: list all 28 site extractors No params. Returns each extractor's name and URL shape: `reddit`, `hackernews`, `github_repo`, `github_pr`, `github_issue`, `github_release`, `pypi`, `npm`, `crates_io`, `huggingface_model`, `huggingface_dataset`, `arxiv`, `docker_hub`, `dev_to`, `stackoverflow`, `substack_post`, `youtube_video`, `linkedin_post`, `instagram_post`, `instagram_profile`, `shopify_product`, `shopify_collection`, `ecommerce_product`, `woocommerce_product`, `amazon_product`, `ebay_listing`, `etsy_listing`, `trustpilot_reviews`. ### `research`: deep multi-source research *(requires `WEBCLAW_API_KEY`)* `query`, `deep`, `topic`. Runs a search, read, and synthesize loop on the hosted engine and returns a cited report. ## Which tools need a key? | Works with no key (runs locally) | Needs `WEBCLAW_API_KEY` (hosted) | |---|---| | `scrape`, `crawl`, `map`, `batch`, `extract`, `summarize`, `diff`, `brand`, `vertical_scrape`, `list_extractors` | `research` | | `search` (uses your own `SERPER_API_KEY`; hosted fallback if unset) | escalation for bot-protected and JavaScript-rendered pages | ## LLM setup (for `extract` and `summarize`) These two tools use an LLM provider chain: local **Ollama** first (free and private; install from ollama.com), then your own `OPENAI_API_KEY`, `GEMINI_API_KEY`, or `ANTHROPIC_API_KEY` if set. No webclaw key needed. ## Tips - For GitHub, Reddit, YouTube, npm, PyPI, or Amazon, use `vertical_scrape` (or plain `scrape`, which auto-detects most verticals) to get typed fields in one call. - Set `only_main_content: true` to strip navigation, sidebars, and footers. - Use the `llm` format when passing content to a model. - Run `map` before `crawl` to scope a site, then crawl the section you need. - If a page is bot-protected or JS-only, set `WEBCLAW_API_KEY` to escalate; without one you get a clear note that the page needs it. ## vs `web_fetch` | | webclaw | `web_fetch` | |---|---|---| | Output quality | Multi-step extraction pipeline; clean markdown and `llm` format | Basic HTML parsing | | Structured extraction | LLM- and schema-based, 28 typed extractors | None | | Crawling and mapping | Whole-site crawl and URL discovery | Single page | | Bot-protected and JS pages | Handled (local best-effort; automatic with a key) | Fails or readability-only | | Cost | Free and local by default | Free | Use `web_fetch` for a quick one-off lookup. Reach for webclaw when you need reliability, clean structure, structured data, or whole-site coverage.