web-scraping

name

web-scraping

description

Use for known URLs/content to scrape/read with fast or browser mode, summarize, map robots/sitemaps/llms, crawl links, batch URLs, diff snapshots, extract JSON/regex/verticals/selector, get YouTube transcripts/captions, get responseId/jobId, drive/operate a live page across steps with web_browser (navigate/click/fill/select/read/screenshot/evaluate, persistent logged-in sessions), not search/research. Verticals: github/gitlab/stackoverflow/wikipedia/npm/pypi/crates/docker/youtube/reddit/hn/arxiv/hf/deepwiki/ossinsight/docsite.

How to choose

1. URL from a known site? → use web_extract action=vertical extractor=<name> with the matching vertical (table below). Hits APIs directly — no HTML scraping. (GitHub, YouTube transcripts, npm, Reddit, Stack Overflow, PyPI, arXiv, Hugging Face, etc.) If the vertical returns URL metadata only, 404, empty data, or not found, check the vertical's reference page for fallback guidance.
2. Need the raw page content? → web_scrape. Read a single URL. Add mode=fingerprint if bot-protected.
3. Need JS rendering, bot mitigation, or logged-in pages? → use mode=browser (CloakBrowser default; browserBackend=playwright opt-out).
4. Need a summary? → web_extract action=summarize. Need structured data? → web_extract with pattern (sections/regex/excerpts), selector (CSS class/ID/attribute/XPath), css-extract/xpath-extract (field-mapped JSON), regex-extract (capture groups), cosine (relevance scoring), or adhoc (LLM). For images/files, extract the URL then use web_scrape saveToFile=true.
5. Need to explore a site? → web_crawl to follow links and read pages (BFS/DFS/best-first). Or web_map for URL inventory only.
6. Multiple independent URLs? → web_batch for parallel scraping.
7. Compare page changes? → web_scrape({ url, diff }) against stored snapshots.
8. Get a previous result back? → web_get_result by responseId, jobId, or snapshot.
9. Operate a page (click/fill/submit, multi-step)? → web_browser — stateful driving via @eN refs. Read a page once → mode=browser; drive it over steps → web_browser.

If a specialized tool is not active, call web_tools query="<capability>" first. It searches crawl, map, batch, retrieval, and interactive-browser tools and activates matching tools for the session.

Tools

Each tool has a reference with full args, examples, and rules.

Tool	When	Ref
web_tools	Find and activate specialized web tools on demand	—
web_browser	Drive a live page: navigate/click/fill/select/inspect + read/screenshot/evaluate/exportCookies (stateful, sessionId required)	ref
web_scrape	Read a single URL or inline content (check verticals table first for known sites)	ref
web_scrape format=raw	Verify raw server output (no transformations)	ref
web_scrape format=text	Get plain text with all HTML stripped	ref
web_extract action=summarize	Summarize via LLM (single source)	ref
web_map	Inventory URLs from robots/sitemaps/llms (no bodies)	ref
web_crawl	Follow links, read pages, build context	ref
web_batch	Scrape many independent URLs in parallel	ref
web_scrape + diff	Diff current content against stored snapshot	ref
web_extract	Vertical/pattern/selector/strategy/adhoc extraction	ref
web_get_result	Retrieve stored result by responseId/jobId/snapshot	ref

Proxy quick rules

• Omit proxy to use standard environment variables: HTTPS_PROXY/HTTP_PROXY/ALL_PROXY; NO_PROXY bypasses env-derived proxies.
• Explicit proxy always wins over env vars.
• Static fetch modes (fast, readable) support http://, https://, socks5://, socks://, and socks4:// proxy URLs. SOCKS targets are resolved locally before CONNECT for SSRF validation.
• Reject socks5h:///socks4a://; proxy-side DNS would bypass local DNS/SSRF checks.
• web_crawl proxy=[...] rotates per page (a,b,c,a,b). web_scrape proxy=[...] resolves one proxy for that single scrape call.
• mode=fingerprint works with HTTP(S) proxies; SOCKS proxies are only safe for literal-IP targets, so hostname targets fail closed with guidance.

Vertical extractors

Use web_extract action=vertical extractor=<name> url=<url> — bypasses HTML scraping via site APIs.

URL matches →	Use extractor	Tips	Ref
github.com/:owner/:repo	github_repo		ref
github.com/:owner/:repo/issues/:number	github_issue		ref
github.com/:owner/:repo/pull/:number	github_pr		ref
github.com/:owner/:repo/releases/tag/:tag	github_release		ref
gitlab.com/:owner/:repo or self-hosted :host/:owner/:repo	gitlab		ref
docs sites (Docusaurus, ReadTheDocs, GitBook, MDN)	docsite		ref
raw .ts/.tsx/.js/.jsx/.py/.rs source files	docstrings	Content-based — pass raw source text via content: param, NOT a URL	ref
npmjs.com/package/:name	npm		ref
pypi.org/project/:name	pypi		ref
crates.io/crates/:name	crates_io		ref
hub.docker.com/r/:ns/:repo or _/:repo	docker_hub		ref
YouTube video (watch?v=, youtu.be, shorts) — metadata, comments, transcripts/captions	youtube		ref
reddit post (/r/:sub/comments/:id, redd.it/:id)	reddit		ref
subreddit feed (/r/:sub + sort)	reddit_listing		ref
news.ycombinator.com/item?id=:id	hackernews		ref
stackoverflow.com/questions/:id or /:id/:slug	stackoverflow		ref
en.wikipedia.org/wiki/:title or :lang.wikipedia.org/wiki/:title	wikipedia		ref
deepwiki.com/:owner/:repo	deepwiki	Uses deepwiki.com (auto-generated code wikis), NOT en.wikipedia.org	ref
arxiv.org/abs/:id or arxiv.org/pdf/:id	arxiv		ref
huggingface.co/:owner/:model or legacy /:model	huggingface_model		ref
huggingface.co/datasets/:owner/:dataset or legacy /datasets/:dataset	huggingface_dataset		ref
ossinsight.io/collections	ossinsight_collections	Use ossinsight.io URLs, NOT github.com	ref
ossinsight.io/collections/:slug	ossinsight_collection_ranking	Use ossinsight.io URLs; supports ?metric= & ?period= params	ref
ossinsight.io/trending[/:language]	ossinsight_trending_repos	Use ossinsight.io URLs	ref
ossinsight.io/analyze/:owner/:repo	ossinsight_repo_analytics	Use ossinsight.io/analyze/:owner/:repo, NOT github.com/:owner/:repo	ref

Custom vertical manifests

Need a missing known-site vertical or a local override? Add a project/user YAML manifest; see custom vertical manifest ref.

Modes

Set mode=<name> on scrape/crawl tools when fast isn't enough.

If this happens →	Use mode	Ref
Default (no issues)	fast	ref
403 / Cloudflare / empty shell on fast	fingerprint	ref
Need clean article text without chrome	readable	ref
JS-rendered SPA or fingerprint also blocked	browser	ref
Let it escalate automatically	auto	ref

mode=browser vs web_browser: mode=browser renders and reads one URL (a stateless scrape). web_browser drives a page over multiple steps (stateful session, clickable @eN refs). Different tools — read once vs operate.

Sessions across tools

sessionId carries one persistent browser context (cookies/localStorage/sessionStorage) across every browser-backed tool. Authenticate once with web_browser, then scrape/extract the gated pages with the same sessionId + mode=browser:

web_browser action=navigate sessionId="s1" url=".../login"   # drive the login
web_browser action=fill ... ; web_browser action=click ...
web_scrape  url=".../dashboard" mode=browser sessionId="s1"  # same authed context
web_extract url=".../data" action=adhoc prompt="…" mode=browser sessionId="s1"

saveSession=true persists to disk (resume after a restart). To wipe a session, pass clearSession=true on a scrape-family call (web_scrape/web_extract/web_crawl/web_batch) — web_browser itself has no clearSession. Continuity holds only within mode=browser — fast/fingerprint use a separate cookie jar keyed by the same id. The handoff shares the session (cookies), not the live page's current view; the other tools open a fresh tab and re-navigate.