crwl

name	crwl
description	Crawl and extract web content using the Crawl4AI CLI (crwl).

crwl - Web Crawling & Extraction

Use the crwl CLI to fetch web pages, extract structured data, and answer questions about web content. Always run via Bash tool.

Triggers

• crawl this page
• extract data from this URL
• read this web page
• scrape this site
• what does this page say about

Quick Reference

Task	Command
Fetch as markdown	crwl URL -o md-fit -B ~/.dotfiles/crwl/browser.yml -C ~/.dotfiles/crwl/crawler.yml -f ~/.dotfiles/crwl/filter_pruning.yml
Ask a question	crwl URL -q "your question"
LLM extraction	crwl URL -j "describe what to extract" -o json
CSS extraction	crwl URL -e extract.yml -s schema.json -o json
Deep crawl (BFS)	crwl URL --deep-crawl bfs --max-pages 10 -o md-fit
Save to file	crwl URL -o md-fit -O output.md

Config Files

User config files live at ~/.dotfiles/crwl/:

File	Purpose
browser.yml	Headless, random user-agent, 1280x720 viewport
crawler.yml	Bypass cache, networkidle, magic mode, excludes nav/footer/header/aside
filter_pruning.yml	Pruning content filter at 0.48 threshold

Always include these flags unless the user specifies otherwise:

-B ~/.dotfiles/crwl/browser.yml \
-C ~/.dotfiles/crwl/crawler.yml \
-f ~/.dotfiles/crwl/filter_pruning.yml

Output Formats

Flag	Description
-o md	Full markdown
-o md-fit	Filtered/pruned markdown (recommended for LLM consumption)
-o json	JSON with all metadata
-o all	Everything (html, markdown, metadata)

Default to -o md-fit for reading content. Use -o json for structured extraction.

Usage Patterns

1. Read a web page

crwl "https://example.com/article" \
  -o md-fit \
  -B ~/.dotfiles/crwl/browser.yml \
  -C ~/.dotfiles/crwl/crawler.yml \
  -f ~/.dotfiles/crwl/filter_pruning.yml

2. Ask a question about a page

crwl "https://example.com" \
  -q "What are the main features listed?" \
  -B ~/.dotfiles/crwl/browser.yml \
  -C ~/.dotfiles/crwl/crawler.yml

3. Extract structured data with LLM

crwl "https://example.com/products" \
  -j "Extract all products with name, price, and description" \
  -o json \
  -B ~/.dotfiles/crwl/browser.yml \
  -C ~/.dotfiles/crwl/crawler.yml

For schema-based extraction, create a JSON schema file and extraction config:

crwl "https://example.com/products" \
  -e extract_llm.yml \
  -s schema.json \
  -o json \
  -B ~/.dotfiles/crwl/browser.yml \
  -C ~/.dotfiles/crwl/crawler.yml

4. Deep crawl a site

crwl "https://docs.example.com" \
  --deep-crawl bfs \
  --max-pages 20 \
  -o md-fit \
  -B ~/.dotfiles/crwl/browser.yml \
  -C ~/.dotfiles/crwl/crawler.yml \
  -f ~/.dotfiles/crwl/filter_pruning.yml

Strategies: bfs (breadth-first), dfs (depth-first), best-first (relevance-based).

5. Crawl with authentication (browser profile)

# First, create a profile interactively (user must do this manually)
crwl profiles

# Then use it
crwl "https://private-site.com/dashboard" \
  -p my-profile \
  -o md-fit \
  -B ~/.dotfiles/crwl/browser.yml \
  -C ~/.dotfiles/crwl/crawler.yml

6. Target specific content with CSS

crwl "https://example.com" \
  -c "css_selector=article.main-content" \
  -o md-fit \
  -B ~/.dotfiles/crwl/browser.yml \
  -C ~/.dotfiles/crwl/crawler.yml \
  -f ~/.dotfiles/crwl/filter_pruning.yml

Inline Overrides

Override config file settings with -b (browser) and -c (crawler) flags:

# Show browser window
crwl URL -b "headless=false" -B ~/.dotfiles/crwl/browser.yml -C ~/.dotfiles/crwl/crawler.yml

# Target specific element and wait longer
crwl URL -c "css_selector=#content,page_timeout=60000" -B ~/.dotfiles/crwl/browser.yml -C ~/.dotfiles/crwl/crawler.yml

LLM Configuration

The default LLM provider for -j and -q is configured globally:

crwl config list                                          # View current settings
crwl config set DEFAULT_LLM_PROVIDER "openai/gpt-4o"     # Set provider
crwl config set DEFAULT_LLM_PROVIDER_TOKEN "your-key"    # Set API key

Anti-Patterns

Avoid	Instead
Omitting config flags	Always pass -B, -C, -f from ~/.dotfiles/crwl/
Using -o md for LLM input	Use -o md-fit — it's pruned and cleaner
Deep crawl without --max-pages	Always set a limit to avoid runaway crawls
Fetching full sites for one page	Use single URL, reserve --deep-crawl for multi-page needs
Running crwl profiles in Bash	Profile creation is interactive — tell the user to run it manually

GitHub URLs

Don't use crwl for GitHub file URLs — use curl with raw URLs instead (faster, no browser needed).

Convert blob URLs to raw:

https://github.com/{owner}/{repo}/blob/{branch}/{path}
→ https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{path}

Example:

# Instead of: crwl "https://github.com/user/repo/blob/main/src/app.py" ...
curl -sSL "https://raw.githubusercontent.com/user/repo/main/src/app.py"

Also works directly for raw.githubusercontent.com and gist.githubusercontent.com/*/raw/ URLs.