| name | context-crawl |
| description | Discover or read multiple pages from a website with Context.dev. Use when the user asks to map a site, find URLs in a sitemap, crawl documentation or a site section, gather several linked pages, or locate the most relevant pages within one domain. |
Map and crawl websites
Choose between URL discovery and content collection:
- Use
web-scrape-sitemap to discover and rank URLs without reading every page.
- Use
web-crawl to retrieve content from a bounded set of linked pages.
Workflow
- Confirm the target domain or starting URL and the section the user cares about.
- Use sitemap search when the user wants particular pages rather than the whole site.
- Apply path, subdomain, and link limits that match the request.
- Keep synchronous crawls focused; do not expand scope beyond the requested site or section.
- Return page URLs alongside the relevant content so results remain traceable.
Use web-scrape-markdown for one known page. For a large crawl or thousands of URLs, use submit-batch instead of forcing the work through a synchronous crawl.
