crw

Reference skill for building production-ready crw integrations. Covers verb selection, call surfaces (CLI/MCP/REST), post-filtering strategies, context-window hygiene, Hybrid RAG patterns, common pitfalls, and crw-specific operational considerations (search backend limits, renderer pool, proxy rotation). Load this when writing application code that embeds crw, designing a multi-step agent workflow, or debugging an integration that isn't behaving as expected.

Programmatic web search and scrape with context isolation. Use for any research task where you need to search the web, filter results, and extract specific information — without flooding your context window with raw HTML and boilerplate. This is the single biggest token-saver in the crw skill set. Triggered by "search for", "look up", "find", "research", "what's the latest on", or any query that requires current web information. Also use when asked to "search and filter", "find the important parts", or any task where you suspect the raw output will be large (multi-page scrapes, news aggregation, competitive research).

Coming from Firecrawl? Switch to fastCRW in one line. Use when the user has existing Firecrawl SDK code (firecrawl-py, firecrawl-js, REST calls, or an MCP config) and wants to point it at fastCRW — managed or self-hosted. Covers the exact base_url swap, which endpoints are drop-in, which have gaps, and how to verify the switch worked.

Search the web with fastCRW and get titles, URLs, and descriptions. Use when you have a question or topic but not a URL — "search for", "find pages about", "look up", "what is", "who is", "latest news on", "find docs for". Own search backend: self-hosted, no API key, no per-query cost, high recall via meta-search aggregation. Step 1 of the crw workflow ladder.

Stand up your own fastCRW API server — single binary, Docker, or docker-compose with a bundled search-backend sidecar. Use when the user wants to run crw locally or on their own infra, configure renderers/proxies/ auth/LLM extraction, or understand the embedded vs proxy MCP modes.

Scrape, crawl, map, search, parse, and extract web data with fastCRW — the open-source, self-hostable Firecrawl alternative (single Rust binary, ~6 MB RAM, Firecrawl-compatible /v1 + /v2 API). Use whenever the user needs page content, site-wide extraction, URL discovery, web search, PDF parsing, structured JSON from pages, or change tracking. Also use when the user mentions Firecrawl, Tavily, Crawl4AI, or "scrape/crawl/map/fetch/get the page/read this site/search the web" — crw is a drop-in for the Firecrawl SDKs.

Scrape, crawl, map, and search the web using fastCRW's native /v1 API. Use when the user needs web page content, site-wide extraction, URL discovery, or web search results. Single binary, 6 MB RAM; /v2 exists separately for Firecrawl migration.

Scrape, crawl, map, and search the web using fastCRW's native /v1 API. Use when the user needs web page content, site-wide extraction, URL discovery, or web search results. Single binary, 6 MB RAM; /v2 exists separately for Firecrawl migration.

Find ALL the arXiv papers that answer a research question, using fastCRW's Firecrawl-compatible Research API. Use when the ask is to survey a literature, enumerate papers on a topic, find what a paper compares against or builds on, list the best models on a benchmark, or recover a paper from a vague description — "papers that do X", "what does X benchmark against", "best open model on Y", "find the paper that ...". Reaches 61.0% recall on the ArXivQA benchmark vs Firecrawl's Research Index 53.3%.

Crawl an entire website or section and extract content from every page. Use when you need content from many pages under a common URL prefix: "crawl the whole site", "get all docs pages", "scrape every blog post", "download the full docs for RAG", "extract all pages under /api". Async BFS — starts a job and polls for results. Step 4 of the crw workflow ladder.

Extract a typed JSON object from one or more web pages against a JSON Schema with fastCRW. Use when you need structured data — "get the price and stock status", "extract all job listings as JSON", "pull structured fields from this page". Step 6 of the crw workflow ladder.

Discover all URLs on a website without fetching content — fast, low-cost URL inventory via sitemap.xml + link extraction BFS. Use when you need to know which pages exist before deciding what to scrape or crawl: "list all pages", "find URLs on this site", "discover links", "what pages does this site have", "map the site". Step 3 of the crw workflow ladder.

Parse a local or remote FILE (PDF) into markdown or structured JSON with fastCRW. Use when the source is a file on disk — "parse this PDF", "extract text from this document", "read this report", "convert PDF to markdown". Routing rule: URL → use crw-scrape; file on disk → use crw-parse. Step 5 of the crw workflow ladder.

Scrape a single known URL into clean markdown / HTML / links / structured JSON with fastCRW. Use when you already have the URL and want the page content — "scrape", "grab", "fetch", "pull", "read this page", "get the content of". Handles JavaScript-rendered SPAs automatically. Step 2 of the crw workflow ladder.

Detect what changed between two page snapshots with fastCRW — stateless diff as a REST primitive. Use when you need to track content changes, monitor a page for updates, or build a cron-based alert system: "has this page changed?", "alert me when pricing changes", "diff this week's scrape against last week's". Step 7 of the crw workflow ladder.