web-scraping

# Web Scraping for Development Practical scraping patterns for building applications — extracting content, data pipelines, and content migration. ## Choose the Right Approach | Need | Tool | When | |------|------|------| | Single page content | `read_url` (Jina) | Fast markdown extraction, no JS needed | | Single page with JS | `firecrawl_scrape` | SPA, dynamic content, needs rendering | | Multiple pages | `read_url` with a `url` array (Jina, up to 5 per call) | Batch reading known URLs | | One answer from a long page | `read_url` with `question` (Jina) | Returns only the relevant passages — cheap | | Entire site | `firecrawl_crawl` | Full site crawl with depth control | | Site URL discovery | `firecrawl_map` | Find all URLs before selective scraping | | Structured data, one URL | `firecrawl_scrape` with `formats: ["json"]` | Extract to a JSON schema (LLM-powered) | | Structured data, unknown URLs | `firecrawl_agent` + `firecrawl_agent_status` | Research across sites, returns JSON | | Interactive pages | `firecrawl_interact` | Login, click, scroll before extraction | | Screenshots | `capture_screenshot_url` (Jina) | Visual snapshots | ## Pattern 1: Scrape Single Page ### Simple Content Extraction ``` Tool: read_url Input: { "url": "https://example.com/article" } ``` Fast, returns clean markdown. Use for blog posts, articles, documentation. ### JavaScript-Heavy Pages ``` Tool: firecrawl_scrape Input: { "url": "https://example.com/spa-page", "formats": ["markdown"], "waitFor": 3000, "onlyMainContent": true } ``` Use `waitFor` for SPAs and pages that load content dynamically. ## Pattern 2: Extract Structured Data Turn unstructured web pages into structured JSON for your app. For one known URL use `firecrawl_scrape` with the `json` format — the prompt and schema go into `jsonOptions`: ``` Tool: firecrawl_scrape Input: { "url": "https://example.com/products/item-1", "formats": ["json"], "jsonOptions": { "prompt": "Extract the product details", "schema": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "description": { "type": "string" }, "specs": { "type": "object" }, "images": { "type": "array", "items": { "type": "string" } } } } } } ``` Use for: product catalogs, pricing pages, directory listings, event data. Run one call per URL (loop over a mapped URL list). When the URLs are **not known** or the data is spread over several sites, use the research agent instead — it returns only a job ID, so poll it: ``` Tool: firecrawl_agent Input: { "prompt": "Find the pricing plans of the top 5 headless CMS vendors", "schema": { "type": "object", "properties": { "vendors": { "type": "array", "items": { "type": "object" } } } }, "maxCredits": 500 } Tool: firecrawl_agent_status Input: { "id": "<job id from the previous call>" } # repeat every 15-30 s until completed ``` ## Pattern 3: Batch Scrape Multiple Pages ### Known URL List `read_url` accepts up to 5 URLs per call and reads them concurrently. Add `question` to get only the relevant passages instead of the full pages. ``` Tool: read_url Input: { "url": [ "https://docs.example.com/page-1", "https://docs.example.com/page-2", "https://docs.example.com/page-3" ]} ``` For more than 5 URLs, split them into chunks of 5. ### Discover Then Scrape ``` Step 1 — Map the site: Tool: firecrawl_map Input: { "url": "https://example.com", "limit": 100 } Step 2 — Read selected URLs (5 per call): Tool: read_url Input: { "url": [<up to 5 selected URLs from step 1>] } ``` ## Pattern 4: Full Site Crawl Crawl an entire site for content migration or archival: ``` Tool: firecrawl_crawl Input: { "url": "https://old-site.example.com", "limit": 200, "maxDiscoveryDepth": 4, "includePaths": ["/blog/*", "/docs/*"], "excludePaths": ["/admin/*", "/login/*"], "scrapeOptions": { "formats": ["markdown", "links"], "onlyMainContent": true } } ``` `firecrawl_crawl` polls the job itself and returns the final status with the collected pages, so keep `limit` conservative — large crawls produce large responses. Use `firecrawl_check_crawl_status` with a crawl ID only to pick up a job that was started earlier or cut off. ## Pattern 5: Interactive Scraping (Login Required) For pages behind authentication, use `firecrawl_interact` — a single call drives a live browser session with a natural-language prompt (or code): ``` Step 1 — Interact: Tool: firecrawl_interact Input: { "url": "https://example.com/login", "prompt": "Log in with email user@example.com and password from my credentials, then navigate to the dashboard and return its content" } Step 2 — Clean up (use the scrapeId returned by step 1): Tool: firecrawl_interact_stop Input: { "scrapeId": "<scrapeId from step 1>" } ``` To continue on the same page, call `firecrawl_interact` again with `scrapeId` instead of `url`. For lighter interactions (click, type, scroll before extraction), the `actions` array on `firecrawl_scrape` is a simpler alternative — no session to manage. ## Pattern 6: Content Pipeline Script Generate a script that scrapes and imports content into your app: 1. **Identify source URLs** — use `firecrawl_map` or manual list 2. **Extract structured data** — use `firecrawl_scrape` with `formats: ["json"]` and an app-specific schema 3. **Transform data** — map scraped fields to your app's data model 4. **Upload** — use your app's API/SDK to create records Example workflow: ``` Map site → Filter URLs → Extract structured data → Transform → Upload to Directus/NocoDB/API ``` ## Pattern 7: Generate Standalone Scraper Script When the user needs a reusable script they can run independently (not just MCP tool calls): ### TypeScript (Node.js) ```typescript import { Firecrawl } from 'firecrawl'; const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY }); async function scrapeProducts(baseUrl: string) { // 1. Discover URLs — map() returns { links: [{ url, title, description }] } const map = await firecrawl.map(baseUrl, { limit: 200 }); const productUrls = map.links.map(l => l.url).filter(u => u.includes('/product')); // 2. Extract structured data — one scrape per URL with the json format const schema = { type: 'object', properties: { name: { type: 'string' }, price: { type: 'number' }, description: { type: 'string' } }}; const results = []; for (const url of productUrls) { const doc = await firecrawl.scrape(url, { formats: [{ type: 'json', prompt: 'Extract the product details', schema }], }); results.push(doc.json); await sleep(1000); // Rate limit respect } return results; } ``` ### Python ```python import requests, json, time JINA_KEY = os.environ["JINA_API_KEY"] headers = {"Authorization": f"Bearer {JINA_KEY}", "Accept": "application/json"} def scrape_urls(urls: list[str]) -> list[str]: """Read multiple URLs via Jina Reader API.""" results = [] for url in urls: resp = requests.get(f"https://r.jina.ai/{url}", headers=headers) results.append(resp.text) time.sleep(0.5) return results ``` Use this pattern when the user says "build me a scraper," "create a script," or "I need something I can run on a schedule." For one-time extraction, MCP tool patterns (Patterns 1-6) are simpler. ## Best Practices - Start with `read_url` (fastest; pass `question` when you need one answer, not the whole page), escalate to `firecrawl_scrape` only if needed - Use `firecrawl_map` before crawling to estimate scope - Set reasonable `limit` and `maxDiscoveryDepth` to avoid excessive crawling - Use `includePaths`/`excludePaths` to focus on relevant content - Always use `onlyMainContent: true` to skip navigation, footers, ads - For large sites, process in batches of 10-25 URLs - Respect rate limits — add delays between requests in scripts - Handle errors gracefully — some pages will fail, continue with the rest