fetcher

name	fetcher
description	Fetch web pages, PDFs, and documents with automatic fallbacks and content extraction. Use when user says "fetch this URL", "download this page", "crawl this website", "extract content from", "get the PDF", or provides URLs needing retrieval.
allowed-tools	Bash, Read
triggers	["fetch this URL","download page","crawl website","extract content from","get the PDF","scrape this site","retrieve document"]
metadata	{"short-description":"Web crawling and document fetching CLI"}
provides	["web-fetch"]
composes	["extractor","memory","task-monitor"]
taxonomy	["web","ingestion"]

Fetcher - Web Crawling

Fetch web pages and documents with automatic fallbacks, proxy rotation, and content extraction.

Self-contained skill - auto-installs via uvx from git (no pre-installation needed).

Fully automatic - Playwright browsers are installed on first run for SPA/JS page support.

Simplest Usage

# Via wrapper (recommended - auto-installs)
.pi/skills/fetcher/run.sh get https://example.com

# Or directly if fetcher is installed
fetcher get https://example.com

Common Commands

./run.sh get https://example.com                   # Fetch single URL
./run.sh get-manifest urls.txt                     # Fetch list of URLs
./run.sh get-manifest - < urls.txt                 # Fetch from stdin

Common Patterns

Fetch a single URL

fetcher get https://www.nasa.gov --out run/nasa

Outputs to run/nasa/:

• consumer_summary.json - structured result
• Walkthrough.md - human-readable summary
• downloads/ - raw content files

Fetch multiple URLs

# From file (one URL per line)
fetcher get-manifest urls.txt --out run/batch

# From stdin
echo -e "https://example.com\nhttps://nasa.gov" | fetcher get-manifest -

ETL mode (full control)

fetcher-etl --inventory urls.jsonl --out run/etl_batch
fetcher-etl --manifest urls.txt --out run/demo

Check environment

fetcher doctor                    # Check dependencies and config
fetcher get --dry-run <url>       # Validate without fetching
fetcher-etl --help-full           # All options
fetcher-etl --find metrics        # Search options

Output Structure

run/artifacts/<run-id>/
├── results.jsonl              # Fetch results per URL
├── consumer_summary.json      # Summary stats
├── Walkthrough.md             # Human-readable summary
├── downloads/                 # Raw files (HTML, PDF, etc.)
├── text_blobs/                # Extracted text
├── markdown/                  # LLM-friendly markdown
├── fit_markdown/              # Pruned markdown for LLM input
├── junk_results.jsonl         # Failed/junk URLs
└── junk_table.md              # Quick triage table