context-crawl

name	context-crawl
description	Discover or read multiple pages from a website with Context.dev. Use when the user asks to map a site, find URLs in a sitemap, crawl documentation or a site section, gather several linked pages, or locate the most relevant pages within one domain.

Map and crawl websites

Choose between URL discovery and content collection:

• Use web-scrape-sitemap to discover and rank URLs without reading every page.
• Use web-crawl to retrieve content from a bounded set of linked pages.

Workflow

1. Confirm the target domain or starting URL and the section the user cares about.
2. Use sitemap search when the user wants particular pages rather than the whole site.
3. Apply path, subdomain, and link limits that match the request.
4. Keep synchronous crawls focused; do not expand scope beyond the requested site or section.
5. Return page URLs alongside the relevant content so results remain traceable.

Use web-scrape-markdown for one known page. For a large crawl or thousands of URLs, use submit-batch instead of forcing the work through a synchronous crawl.