cli

Use crawlberg cli reference documentation.

Jump to install Review files

Repository: xberg-io/crawlberg
Last source activity: October 5, 2026 at 05:25
Detected SKILL.md language: English
Stars: 182
Forks: 28

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install only the Agent Skill at https://github.com/xberg-io/crawlberg/tree/main/.agents/skills/cli.

A direct command skips the review prompt. Inspect the source before running it.

npx skills add https://github.com/xberg-io/crawlberg --skill cli

The command stays on one line. Scroll horizontally to inspect it before copying.

Prefer a local copy? Download the files currently available to SkillsMP.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

SKILL.md

Source instructions · Read-only preview

name: cli
description: Use crawlberg cli reference documentation.

# crawlberg CLI Reference Use this skill when invoking, documenting, or changing the crawlberg command-line interface. - `crawlberg`: High-performance web crawler and scraper - `crawlberg batch-crawl`: Crawl multiple websites concurrently - `crawlberg batch-scrape`: Scrape multiple URLs concurrently - `crawlberg citations`: Convert markdown links into numbered citations - `crawlberg crawl`: Crawl a website following links - `crawlberg download`: Download a document from a URL and report its metadata - `crawlberg interact`: Execute browser actions on a single page - `crawlberg map`: Discover all URLs on a website via sitemaps and link extraction - `crawlberg mcp`: Start the MCP server (stdio transport by default). - `crawlberg scrape`: Scrape a single URL and extract metadata - `crawlberg serve`: Start the REST API server. - `crawlberg version`: Print the crawlberg version as JSON

View on GitHub

Back to search Back to installation

automating-the-browser

Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact URL --actions` with the real action schema, result shape, limits, and external-CDP options.

2026-10-06

crawlberg

Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.

2026-10-06

crawling-a-site

Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.

2026-10-06

View all from this repository

cli

Source facts

Install options

Install with an AI assistant

Review the source files

More from this repository

Install with an AI assistant

More from this repository