scraping-websites

name	scraping-websites
description	Scrapes websites and extracts content using Firecrawl API. Single-page scraping, full site crawling, URL mapping, and structured data extraction. Handles JavaScript-rendered content. Triggers include "scrape this URL", "crawl the documentation", "get all links from", "extract data from website", "map this site", or /scraping-websites.

Firecrawl - Web Scraping Tool

Overview

Firecrawl provides powerful web scraping capabilities including single-page scraping, full site crawling, URL mapping, and LLM-powered structured data extraction. It handles JavaScript-rendered content and converts pages to clean markdown.

Workflow Philosophy: Sequence is Everything

Web scraping tasks fail when you act before understanding. The disciplined approach follows a precise sequence:

MAP → SCRAPE (sample) → PLAN → EXECUTE → EXTRACT

The Sequence

1. MAP First (Reconnaissance) Before touching content, understand the terrain. What URLs exist? What's the structure?

• Exception: Skip only when you have one specific URL already
• Use --search to filter by keyword when exploring large sites

2. SCRAPE a Sample (Understand the Material) Never crawl or extract blind. Scrape 1-2 representative pages to understand:

• What does the markdown output look like?
• Is the content in main body or scattered in sidebars/footers?
• Does it need --wait-for for JavaScript rendering?
• What structure would an extraction schema require?

3. PLAN Before Executing Based on reconnaissance, decide:

• Do I need to crawl (broad collection) or can I scrape specific URLs?
• What --include-paths or --exclude-paths patterns apply?
• What --limit makes sense?
• What output format serves the goal?

4. EXECUTE with Discipline

• Always set --limit on crawls (never unbounded)
• Always save to files (-o) for non-trivial operations
• Use --async for large crawls, check status periodically

5. EXTRACT Only When Ready Extraction requires knowing:

• The exact JSON schema that matches the actual content structure
• That the URL genuinely contains the data you expect (verified by sampling)

Anti-Patterns to Avoid

Anti-Pattern	Why It Fails
Crawl first, ask questions later	Wastes API calls, returns irrelevant pages
Extract without sampling	Schema mismatches, empty results
No output files	Can't inspect, debug, or resume
No limits on crawls	Runaway operations, context bloat
Scraping JS-heavy sites without --wait-for	Empty or incomplete content

Pre-Flight Checklist

Before any Firecrawl operation, answer:

1. What specific information am I trying to get?
2. Do I know where it lives, or do I need to discover that first (MAP)?
3. Have I seen a sample of the actual content (SCRAPE sample)?
4. What's the minimum operation that gets what I need?

Example: The Disciplined Workflow

User: "Get me the API documentation from docs.example.com"

Step 1 - MAP (discover):
  python3 firecrawl_tool.py map https://docs.example.com --search "api"
  → See 47 URLs, notice /api/v2/* structure

Step 2 - SCRAPE (sample):
  python3 firecrawl_tool.py scrape https://docs.example.com/api/v2/overview
  → Inspect markdown structure, confirm content quality

Step 3 - PLAN:
  Need /api/v2/* paths only, estimate ~20 pages, want main content

Step 4 - EXECUTE:
  python3 firecrawl_tool.py crawl https://docs.example.com \
    --include-paths "/api/v2/*" --limit 30 --only-main-content true \
    -o api_docs.json

---

Tool Location

The Firecrawl CLI tool is located at:

/Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py

API key is configured in ~/.env as FIRECRAWL_API_KEY.

Commands Reference

Scrape - Single URL

Extract content from a single URL as markdown (default) or other formats.

python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py scrape <url> [options]

Options:

• -o, --output FILE - Save output to file
• --json - Output as JSON instead of markdown
• --formats - Comma-separated: markdown,html,rawHtml,links,screenshot
• --include-tags - Only include specific HTML tags (comma-separated)
• --exclude-tags - Exclude specific HTML tags (comma-separated)
• --only-main-content true/false - Extract only main content
• --wait-for MS - Wait N milliseconds before scraping (for JS)
• --timeout MS - Request timeout in milliseconds
• --mobile - Use mobile viewport
• --skip-tls - Skip TLS certificate verification

Examples:

# Basic scrape to markdown
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py scrape https://example.com

# Save to file
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py scrape https://docs.example.com -o page.md

# Get JSON with multiple formats
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py scrape https://example.com --json --formats markdown,links

# Wait for JavaScript to render
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py scrape https://spa-app.com --wait-for 3000

Crawl - Full Website

Crawl an entire website starting from a URL.

python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py crawl <url> [options]

Options:

• -o, --output FILE - Save results to JSON file
• --limit N - Maximum pages to crawl
• --max-depth N - Maximum crawl depth
• --include-paths - Only crawl matching paths (comma-separated patterns)
• --exclude-paths - Skip matching paths (comma-separated patterns)
• --allow-subdomains - Allow crawling subdomains
• --allow-external - Allow crawling external links
• --ignore-sitemap - Ignore sitemap.xml
• --only-main-content true/false - Extract only main content
• --async - Start async crawl (returns job ID)

Examples:

# Crawl documentation site (limit 50 pages)
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py crawl https://docs.example.com --limit 50 -o docs.json

# Crawl only /api/ paths
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py crawl https://docs.example.com --include-paths "/api/*,/reference/*"

# Start async crawl for large sites
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py crawl https://large-site.com --async --limit 1000

Map - Site URL Discovery

Get a list of all URLs on a website without scraping content.

python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py map <url> [options]

Options:

• -o, --output FILE - Save URLs to file (one per line)
• --include-subdomains - Include subdomain URLs
• --limit N - Maximum URLs to return
• --search QUERY - Filter URLs by search query

Examples:

# Get all URLs
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py map https://example.com

# Search for specific content
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py map https://docs.example.com --search "api reference"

# Save URL list
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py map https://example.com -o sitemap.txt

Extract - Structured Data Extraction

Use LLM to extract structured data from a URL based on a JSON schema.

python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py extract <url> --schema <json_or_@file> [options]

Options:

• --schema - JSON schema string or @filepath to load from file
• --prompt - Additional extraction prompt
• --system-prompt - System prompt for extraction
• -o, --output FILE - Save extracted data to file

Examples:

# Extract product info with inline schema
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py extract https://store.com/product \
  --schema '{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"},"description":{"type":"string"}}}'

# Extract using schema file
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py extract https://example.com/page --schema @schema.json

# Extract with custom prompt
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py extract https://blog.com/article \
  --schema '{"type":"object","properties":{"title":{"type":"string"},"author":{"type":"string"},"summary":{"type":"string"}}}' \
  --prompt "Extract the main article metadata"

Status - Check Async Job

Check the status of an async crawl job.

python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py status <job_id>

Common Workflows

Scrape Documentation for Context

# Map the docs site first
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py map https://docs.example.com --search "getting started"

# Scrape specific pages
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py scrape https://docs.example.com/quickstart -o quickstart.md

Build a Knowledge Base

# Crawl entire documentation
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py crawl https://docs.example.com \
  --limit 100 --only-main-content true -o knowledge_base.json

Extract Structured Data from Multiple Pages

# First map to find relevant pages
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py map https://store.com --search "products"

# Then extract from each (iterate in bash or Python)
python3 /Users/tyler/Dev/tools/firecrawl/firecrawl_tool.py extract https://store.com/product/1 \
  --schema @product_schema.json -o product1.json

Output Formats

• markdown (default): Clean markdown text, ideal for LLM consumption
• html: Cleaned HTML content
• rawHtml: Original unprocessed HTML
• links: List of all links on the page
• screenshot: Screenshot of the page (returns URL)
• extract: Structured JSON data (with extract command)

Specialized Extractors

Probate Code Extractor

Extract estate planning and probate code for US states and territories using Firecrawl's Agent API.

python3 /Users/tyler/Dev/tools/firecrawl/probate_extractor.py <state> <base_url> [options]

Usage:

# Basic usage
python3 /Users/tyler/Dev/tools/firecrawl/probate_extractor.py "California" "https://leginfo.legislature.ca.gov"

# With flags and output file
python3 /Users/tyler/Dev/tools/firecrawl/probate_extractor.py \
  --state "Texas" \
  --url "https://statutes.capitol.texas.gov" \
  -o texas_probate.json

# With credit limit and verbose output
python3 /Users/tyler/Dev/tools/firecrawl/probate_extractor.py \
  -s "New York" \
  -u "https://www.nysenate.gov/legislation/laws" \
  -m 500 -v

Options:

• --state, -s - State or territory name (required)
• --url, -u - Base URL for state legal code website (required)
• --output, -o - Output JSON file path
• --max-credits, -m - Maximum credits to spend (controls cost)
• --verbose, -v - Print progress information

Output Schema:

{
  "state": "California",
  "probate_code_sections": [
    {
      "section_number": "15000",
      "title": "Trust Law",
      "full_text": "...",
      "code_name": "Probate Code",
      "effective_date": "2024-01-01",
      "source_url": "https://..."
    }
  ],
  "related_code_sections": [...],
  "extraction_notes": "..."
}

Coverage: The extractor looks for:

• Probate Code / Estates Code / Succession Code
• Property Code (inheritance, trusts)
• Tax Code (estate/inheritance taxes)
• Trust Code
• Family Code (inheritance rights)
• Wills, intestate succession, powers of attorney, guardianship, conservatorship

Note: Uses Firecrawl Agent (research preview). The agent autonomously navigates the specified domain to find all relevant statutory sections.

---

Error Handling

• API key missing: Ensure FIRECRAWL_API_KEY is set in ~/.env
• Rate limits: Firecrawl has rate limits; for large crawls use --async
• JavaScript content: Use --wait-for to allow JS to render
• TLS errors: Use --skip-tls for self-signed certificates