site-crawl

name

site-crawl

description

Bulk-extract clean content from many pages on a site via the Tavily MCP. Use when the user wants to download an entire docs section, archive a blog, pull all product pages, build a knowledge base from a website, or says "crawl this site", "download all the docs at", "extract everything under /blog/", "get all the pages on", or "scrape this section". Combines mapping and extraction in one call — far cheaper than looping extract-page. For URL discovery without content, use site-map. For a single URL, use extract-page.

Site Crawl

Call tavily_crawl to walk a site and pull clean content from every page it discovers. One call replaces dozens of tavily_map + tavily_extract round-trips.

When to use this skill

• The user wants every page in a site section (docs, blog, pricing).
• Building a one-shot RAG corpus from a website.
• Archiving a documentation set as local markdown.

For URL-only discovery, use site-map. For a single URL, use extract-page — crawl is overkill for one page.

How to call the tool

Pass these arguments to tavily_crawl:

Argument	When to set it
url	Required. Root URL to crawl from.
max_depth	Default 1. Bump to 2–3 for nested docs trees. Each level multiplies pages.
max_breadth	Pages per level. Default 20.
limit	Total page cap. Default 50. Raise for big sites; this is the main cost lever.
instructions	Natural-language guide. E.g. "only product feature pages", "skip release notes".
select_paths	Regex include for URLs.
exclude_paths	Regex exclude. Always exclude /tag/, /author/, /page/\\d+/ for blog crawls.
extract_depth	basic or advanced. Use advanced for SPAs / heavy-JS docs sites.
format	markdown (default) or text.
categories	Optional content category hints.
allow_external	Bool. Off by default.

Cost discipline

Crawls are the most expensive Tavily operation. Before kicking one off:

1. Estimate page count with site-map first if you're unsure.
2. Always set limit explicitly — never rely on the default unless you know the site is small.
3. Tell the user how many pages you'll crawl and how long it will take.
4. Use instructions and exclude_paths aggressively to skip noise.

Patterns

• Docs corpus: url: "https://docs.example.com", max_depth: 3, limit: 200, exclude_paths: ["/blog/", "/changelog/"].
• Blog archive: url: "https://example.com/blog", max_depth: 2, exclude_paths: ["/tag/", "/author/", "/page/\\d+/"].
• Competitor product pages: url: "https://competitor.com/product", max_depth: 2, instructions: "product feature and pricing pages only".

Saving outputs

Crawled content can be large. For Cowork users:

1. Write each page as a separate .md in ~~document store under a folder named after the source domain.
2. If the document store doesn't support folders or isn't connected, save as <domain>.md in the workspace folder with a table of contents at the top.
3. Always include the source URL as the first line of each file.

Failure modes

• Crawl returns nothing: check that url is reachable. SPAs may need extract_depth: "advanced".
• Crawl returns way too much: tighten select_paths, exclude_paths, lower limit, or use site-map first to scope.
• Hits limit before reaching target pages: raise limit (warn user about cost) or narrow scope to start from a deeper root URL.