| name | flarecrawl-ops |
| description | Use flarecrawl CLI for web scraping via Cloudflare Browser Rendering. Triggers: flarecrawl, scrape, crawl, browser rendering, cloudflare scrape, web content, tech-detect, anti-bot, mcp server |
| version | 1.1.0 |
| category | operations |
| tool | flarecrawl |
| requires | {"bins":["flarecrawl"],"skills":[]} |
| allowed-tools | Read Bash Grep |
Flarecrawl Operations
Cloudflare Browser Rendering CLI — Firecrawl-compatible, cost-efficient at scale.
Renders JavaScript, extracts markdown/HTML, crawls sites, takes screenshots,
generates PDFs, AI-extracts structured data, fingerprints tech stacks, and
cracks anti-bot walls. Also ships an MCP server and a CDP mode for
persistent, authenticated browser sessions.
Orient first. flarecrawl guide prints the packaged agent reference
(when/why each command, JSON shapes, exit codes, footguns). flarecrawl guide <topic> drills in (aliases: hard-targets, json, errors, rules,
auth). flarecrawl guide --list shows topics. Treat guide as the source
of truth; this skill is the quick map.
Auth Check
flarecrawl auth status --json
flarecrawl usage --json
MCP Server (for agents)
Flarecrawl exposes a Model Context Protocol stdio server — 36 tools in a
three-tier surface — so an agent can call the toolkit directly instead of
shelling out.
uv pip install 'flarecrawl[mcp]'
flarecrawl mcp
flarecrawl mcp --read-only
Wire into .mcp.json:
{"mcpServers": {"flarecrawl": {"command": "flarecrawl", "args": ["mcp"]}}}
| Tier | Count | When | Examples |
|---|
| Orientation | 5 | session start / capability check | capabilities(), guide(), diagnostics() |
| T1 Composite | 5 | ~80% of tasks | read_page(), research_web(), site_overview() |
| T2 Curated | 17 | single-resource ops | web_search(), tech_detect(), crawl_start() |
| T3 Raw | 9 | full CLI fidelity | scrape_raw(), p6_raw(), spider_raw() |
Always call capabilities() first — one call returns the full catalogue,
permissions, coverage gaps, and worked recipes. Content tools default to
--agent-safe. Errors return {"ok": false, "error": {"code", "next_steps"}};
meta.blocked verdicts auto-suggest recovery (Akamai → stealth/p6; CF-1020 →
terminal). Coverage gaps (CLI only): videos, authcrawl, frontier, batch,
auth login/logout, cache clear, rules *, cdp *, webmcp *, some
session subcommands, --interactive/--headed.
Core Operations
flarecrawl scrape https://example.com --json
flarecrawl scrape URL --format html --json
flarecrawl scrape URL --js
flarecrawl scrape URL --only-main-content
flarecrawl crawl https://docs.example.com --wait --limit 50 --json
flarecrawl crawl JOB_ID --ndjson --fields url,markdown
flarecrawl map URL --json
flarecrawl discover URL --json
flarecrawl spider URL --limit 500
flarecrawl fetch URL --json
flarecrawl fetch URL -o file.pdf
flarecrawl extract "Get product names and prices" --urls URL --json
flarecrawl download URL --limit 50
flarecrawl screenshot URL -o page.png --full-page
flarecrawl pdf URL -o page.pdf
flarecrawl favicon URL --all --json
Tech Detection (Wappalyzer)
flarecrawl tech-detect URL --json
flarecrawl tech-detect URL --json --exclude-categories "Miscellaneous,Security,Tag managers,RUM"
flarecrawl tech-detect -i urls.txt --workers 10 --json
flarecrawl tech-detect URL --render --json
flarecrawl scrape URL --tech-detect --json
Hard Targets & Anti-Bot
For sites that fingerprint TLS, return stubs to CF, or hide data in JS state:
flarecrawl scrape URL --paywall --stealth --json
flarecrawl scrape URL --js --browser local --headed
flarecrawl scrape URL --js --capture-pattern "*.csv,*.json" --capture-dir ./out/
flarecrawl scrape URL --then-fetch-from manifest.csv --then-fetch-column "Link" --then-fetch-output ./files/
flarecrawl p6 https://site/ --jar jar.json --target https://site/api --output-dir ./out
flarecrawl session inspect @site
flarecrawl recipe flow.yml --dry-run
flarecrawl recipe flow.yml --resume
meta.blocked {blocked, vendor, kind, terminal, signal} is the bot-wall source
of truth (scrape CDP, fetch --json, recipe). terminal: true (CF-1020) =
non-bypassable, don't burn a re-mint. Don't string-match block pages yourself.
CDP — Persistent / Authenticated Sessions
uv pip install websockets
flarecrawl scrape URL --interactive --json
flarecrawl scrape URL --cdp --keep-alive 60 --save-cookies session.json --har traffic.har --json
flarecrawl scrape URL --load-cookies session.json --cdp --json
flarecrawl cdp sessions --json && flarecrawl cdp close
CDP flags (--interactive, --live-view, --record, --save/load-cookies,
--keep-alive, --browser-cookies) all auto-promote to --cdp.
Agent-Safe Mode
flarecrawl scrape URL --agent-safe --json
13 sanitisers, two phases (HTML + text). metadata.agentSafety reports findings.
Prompt injection removed (short-line bias <200 chars); semantic manipulation
flagged, not removed. Works on scrape/crawl/download/extract and --stdin.
Batch & Parallel
scrape and extract support --batch/-b and --workers/-w:
flarecrawl scrape --batch urls.txt --workers 5
flarecrawl map URL --json | jq -r '.data[]' > urls.txt && flarecrawl scrape --batch urls.txt --workers 5
- Input: plain text (one/line,
# comments), JSON array, or NDJSON (auto-detected)
- Output: NDJSON
{index, status, data/error}, sorted by index
- Workers: default 3, max 50 (CF raised concurrency to 120). Override with
FLARECRAWL_MAX_WORKERS
- Partial failures don't stop processing; batch fails fast on auth/permission errors
Output Interpretation
{"data": {"url": "...", "content": "# Markdown...", "elapsed": 2.9}, "meta": {"format": "markdown"}}
| Field | Where | Description |
|---|
data.content | scrape | Markdown or HTML |
data.elapsed | scrape | Seconds to fetch |
data.records[] | crawl | Crawled pages (.url, .markdown, .status) |
data.job_id / data.total / data.browser_seconds | crawl | Job id, page count, browser time |
metadata.source | scrape | content-negotiation (0 browser time) vs browser-rendering |
meta.blocked | scrape/fetch/recipe | Bot-wall verdict |
Gotchas
- Free tier: 10 min/day, 3 concurrent browsers →
--workers 3. Paid: up to --workers 50
- Crawl is async — use
--wait (or poll --status); jobs persist 14 days on CF
- Markdown content negotiation is automatic —
scrape tries Accept: text/markdown first (0 browser time); --no-negotiate to force browser
--input is legacy — use --batch- Binary outputs (screenshot/pdf) save to files;
--json for base64
- Auto-retry on 429/502/503 with backoff (3 attempts)
--cdp, --stealth, --paywall, --browser local, recipes need optional extras (websockets / curl_cffi / playwright / PyYAML)- Exit codes: 0 ok · 1 error · 2 auth · 3 not-found · 4 validation · 5 forbidden · 7 rate-limited
Pipe Patterns
| Chain | Command |
|---|
| → content | flarecrawl scrape URL --json | jq '.data.content' |
| → map then scrape | flarecrawl map URL --json | jq -r '.data[]' > urls.txt && flarecrawl scrape --batch urls.txt |
| → filter batch | flarecrawl scrape --batch urls.txt | jq 'select(.status=="ok") | .data.content' |
| → stream crawl | flarecrawl crawl JOB_ID --ndjson --fields url,markdown | jq -r '.markdown' |
| → tech-detect feed | flarecrawl tech-detect -i urls.txt --workers 10 --json | jq -r '.data[].technologies[].name' |
For full flag reference and JSON shapes, run flarecrawl guide <topic> or read the repo AGENTS.md.
