メニュー
Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact <url> --actions` with the real action schema, result shape, limits, and external-CDP options.
Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.
Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `crawlberg map <url>` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.
Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `crawlberg scrape <url>`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.
Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.
| name | headless-fallback |
| description | Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost. |
Some pages are unscrapable without a real browser — SPA shells, infinite scroll, Cloudflare interstitials, JS-rendered article bodies. Crawlberg ships with an optional headless-Chrome backend driven by chromiumoxide.
--browser-mode auto # default — try static first, fall back to browser on JS/WAF
--browser-mode always # skip static, go straight to browser
--browser-mode never # static only, fail closed
auto (default)The engine fetches statically, then inspects the response. It launches headless Chrome and re-fetches when it sees:
<noscript> warnings, near-empty <body> with heavy JS.This is the right default. The browser only spins up when needed.
alwaysSkip the static probe entirely. Use when:
<script>-emitted state that never lands in static HTML.crawlberg scrape https://spa.example.com --browser-mode always --format markdown
neverStatic only — the browser path is disabled. Use when:
In never mode, JS-only pages return empty/stub content. Inspect
markdown.content and markdown.warnings before treating the result as
final.
In --browser-mode never or when you suspect the auto detector missed a
signal:
markdown.content is short, nav-only, or just a loading message.status_code is 200 but metadata.headings is empty on a page that
clearly has headings.markdown.warnings mentions JS-render-required or WAF detection.server: cloudflare,
cf-mitigated, x-amz-cf-id, set-cookie: __cf_bm=…).Re-run with --browser-mode always. If that succeeds, leave it set for
that host.
Point at an already-running Chrome (Browserless, Steel, your own) instead of launching locally:
crawlberg scrape https://example.com \
--browser-mode always \
--browser-endpoint ws://browser.internal:9222/devtools/browser/<id> \
--format markdown
The endpoint must be a WebSocket URL — ws:// or wss://. The CLI
rejects anything else with a clear error.
Use external CDP when:
Headless Chrome is expensive relative to a static fetch:
NetworkIdle wait, plus the page's own
JS load time.--concurrent.Mitigations:
--browser-mode auto — the engine only pays the cost when it
needs to.--browser-endpoint to share one warm browser across jobs.--concurrent when you know the crawl will route through Chrome.Pass via --config JSON when you need control:
crawlberg scrape https://example.com --browser-mode always \
--config '{"browser":{"wait_strategy":{"type":"Selector","selector":".article-body"}}}'
Supported strategies:
NetworkIdle (default) — wait until the network goes quiet.Selector — wait until a CSS selector resolves.Fixed — wait a fixed duration.extra_wait adds milliseconds on top of the wait strategy if the page
keeps loading content after the primary signal.
crawlberg scrape https://app.example.com --browser-mode always \
--config '{"browser_profile":"prod","save_browser_profile":true}'
Profile names are path-traversal-validated. Use them to keep cookies, localStorage, and login state across runs without re-authenticating.