crawling-a-site

name	crawling-a-site
description	Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.

Crawling a site

Reach for crawlberg crawl when one URL is not enough — the user wants the docs site, the blog, the marketing pages, or the whole domain.

Quick recipe

crawlberg crawl https://example.com \
  --depth 3 \
  --max-pages 200 \
  --concurrent 8 \
  --rate-limit 250 \
  --stay-on-domain \
  --respect-robots-txt \
  --format markdown

Defaults you should usually override:

• --depth 2 is shallow — set it explicitly.
• --max-pages is unbounded by default; cap it for any unknown site.
• --concurrent 10 is aggressive for small hosts; drop to 4-8 for third-party sites.

Flag surface

Flag	Default	Purpose
--depth, -d	2	Maximum hop count from the seed URL.
--max-pages, -n	—	Hard cap on pages fetched. Set this on any unknown site.
--concurrent, -c	10	Parallel in-flight requests.
--rate-limit	200	Milliseconds between requests to the same origin.