Scrape web pages to clean markdown using Firecrawl v2 — handles JS-heavy pages, site crawls, URL mapping, document parsing (PDF/DOCX/XLSX), LLM-powered extraction, autonomous agent scraping, and post-scrape browser interaction (Interact API). Prefer over WebFetch for quality and completeness. Triggers on scrape URL, fetch page, crawl site, extract content, parse document, web to markdown, DeepWiki, Firecrawl.
Scrape web pages to clean markdown using Firecrawl v2 — handles JS-heavy pages, site crawls, URL mapping, document parsing (PDF/DOCX/XLSX), LLM-powered extraction, autonomous agent scraping, and post-scrape browser interaction (Interact API). Prefer over WebFetch for quality and completeness. Triggers on scrape URL, fetch page, crawl site, extract content, parse document, web to markdown, DeepWiki, Firecrawl.
Firecrawl & Jina Web Scraping
Firecrawl vs WebFetch
Prefer firecrawl scrape URL --only-main-content over the WebFetch tool—it produces cleaner markdown, handles JavaScript-heavy pages, and avoids content truncation (>80% benchmark coverage). WebFetch is acceptable as a fallback when Firecrawl is unavailable.
Inspired by Anthropic's dynamic filtering—always filter before reasoning. This reduced input tokens by ~24% and improved accuracy by ~11% in their benchmarks.
Use --only-main-content to strip navigation and footer boilerplate, reducing token consumption. Omit only when nav/footer content is specifically needed.
Use --only-clean-content (Python API script) for aggressive cleaning—strips nav, ads, and cookie banners. Stronger than --only-main-content; use when the page is still noisy after main-content filtering.
Use firecrawl map URL --search "topic" first to find relevant subpages before scraping
Use --format links first to get URL list, evaluate, then scrape selectively
Use --max-chars with exa_contents.py to cap extraction length
Use --formats summary (Python API script) over full text when you need the gist, not raw content
Claude API Native Tools (for API Agent Builders)
Anthropic's API now offers built-in dynamic filtering tools:
These have built-in dynamic filtering via code execution. Use them when building Claude API agents directly. Use Firecrawl/Exa when you need: autonomous agents, batch scraping, structured extraction, domain-specific crawling, or when not on the Claude API.
firecrawl search "machine learning best practices 2026" --scrape --scrape-formats markdown
Document Parsing (Local Files)
Parse local documents into clean Markdown. Use parse for local or non-public files; use scrape for public URLs pointing to documents—both use the same Rust-based parser.
# PDF to markdown
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse report.pdf
# Excel spreadsheet with main content only
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse data.xlsx --only-main-content
# Word doc with zero data retention, save to file
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse contract.docx --zero-data-retention -o contract.md
# Raw JSON output for programmatic use
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse invoice.pdf --json
Scrape a page, then take actions on it—click buttons, fill forms, extract dynamic content. Two modes: AI prompts (natural language) and code execution (Node.js/Python/Bash).
When to Use Interact vs. Actions
Need
Use
Why
Click/wait before a single scrape
--actions on scrape
Fire-and-forget, no session overhead
Multiple interactions with same page
interact
Persistent session, back-and-forth
Fill forms, log in, navigate
interact
Stateful, multi-step
Simple "wait for JS to load"
--actions with wait
Cheaper, no session
Basic Interact (AI Prompt Mode)
# Step 1: Scrape and note the Scrape ID from output
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://example.com/pricing"# Step 2: Interact using natural language
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
--prompt "Click the Enterprise pricing tab"# Step 3: More interactions on same session
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
--prompt "What is the monthly price for the Enterprise plan?"# Step 4: Stop when done
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact-stop SCRAPE_ID
# Scrape with a named profile — browser state persists across sessions
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://app.example.com/login" \
--profile my-app --json
# Interact to log in
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
--code "await page.fill('#email', 'user@example.com'); await page.fill('#password', 'pass'); await page.click('button[type=submit]');"# Later: scrape another page with same profile — cookies restored
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://app.example.com/dashboard" \
--profile my-app
Important: Interact does NOT return page markdown. To get updated content after interaction, use code mode to extract specific elements, or issue a follow-up scrape.
Full interact reference:references/interact-reference.md
Billing Notes
Credit/token unification (v2.9): Credits and tokens are now unified—15 tokens = 1 credit. All pricing is expressed in credits.
Default cache TTL: Results are cached for 2 days. Use --max-age 0 (or maxAge: 0 in API) to force a fresh scrape regardless of cache.
query format: Pass formats=["query"] (Python API) to get a direct answer (data.answer) instead of full markdown. Use for factual lookups where you don't need the full page content.
audio format:formats=["audio"] returns an MP3 of the page read aloud. Useful for accessibility pipelines or voice interfaces.
wikimedia engine: Pass engine="wikimedia" in search options to route queries through Wikimedia. Useful for encyclopedic lookups.
Troubleshooting
# Check status and credits
firecrawl --status && firecrawl credit-usage
# Re-authenticate
firecrawl logout && firecrawl login --api-key $FIRECRAWL_API_KEY# Check API keyecho$FIRECRAWL_API_KEY
Scrape fails: Try jina URL, or add --wait-for 3000 for JS-heavy sites
Async job stuck: Check with crawl-status/batch-status, cancel with crawl-cancel/batch-cancel
python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py --quick # Quick validation
python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py # Full suite
python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py --test scrape # Specific test
