菜单
Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/xberg-io/crawlberg --skill headless-fallback命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact URL --actions` with the real action schema, result shape, limits, and external-CDP options.
Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.
Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
基于 SOC 职业分类
正在显示 SKILL.md
| name | headless-fallback |
| description | Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost. |
Some pages are unscrapable without a real browser — SPA shells, infinite scroll, Cloudflare interstitials, JS-rendered article bodies. Crawlberg ships with an optional headless-Chrome backend driven by chromiumoxide.
--browser-mode auto # default — try static first, fall back to browser on JS/WAF
--browser-mode always # skip static, go straight to browser
--browser-mode never # static only, fail closed
auto (default)The engine fetches statically, then inspects the response. It launches headless Chrome and re-fetches when it sees:
<noscript> warnings, near-empty <body> with heavy JS.This is the right default. The browser only spins up when needed.
alwaysSkip the static probe entirely. Use when:
<script>-emitted state that never lands in static HTML.crawlberg scrape https://spa.example.com --browser-mode always --format markdown
neverStatic only — the browser path is disabled. Use when:
In never mode, JS-only pages return empty/stub content. Inspect
markdown.content and markdown.warnings before treating the result as
final.
In --browser-mode never or when you suspect the auto detector missed a
signal:
markdown.content is short, nav-only, or just a loading message.status_code is 200 but metadata.headings is empty on a page that
clearly has headings.markdown.warnings mentions JS-render-required or WAF detection.server: cloudflare,
cf-mitigated, x-amz-cf-id, set-cookie: __cf_bm=…).Re-run with --browser-mode always. If that succeeds, leave it set for
that host.
Point at an already-running Chrome (Browserless, Steel, your own) instead of launching locally:
crawlberg scrape https://example.com \
--browser-mode always \
--browser-endpoint ws://browser.internal:9222/devtools/browser/<id> \
--format markdown
The endpoint must be a WebSocket URL — ws:// or wss://. The CLI
rejects anything else with a clear error.
Use external CDP when:
Headless Chrome is expensive relative to a static fetch:
NetworkIdle wait, plus the page's own
JS load time.--concurrent.Mitigations:
--browser-mode auto — the engine only pays the cost when it
needs to.--browser-endpoint to share one warm browser across jobs.--concurrent when you know the crawl will route through Chrome.Pass via --config JSON when you need control:
crawlberg scrape https://example.com --browser-mode always \
--config '{"browser":{"wait":"selector","wait_selector":".article-body"}}'
Supported strategies (the browser.wait field is a string enum; pair
"selector" with a sibling wait_selector):
network_idle (default) — wait until the network goes quiet.selector — wait until the CSS selector in wait_selector resolves.fixed — wait a fixed duration.extra_wait adds milliseconds on top of the wait strategy if the page
keeps loading content after the primary signal.
crawlberg scrape https://app.example.com --browser-mode always \
--config '{"browser_profile":"prod","save_browser_profile":true}'
Profile names are path-traversal-validated. Use them to keep cookies, localStorage, and login state across runs without re-authenticating.