scrape

name	scrape
description	Scrape any website with auto-escalating engines (httpx -> Playwright -> Cloudflare bypass), AI extraction, link discovery, screenshots, proxy support, PDF parsing. Use when the user wants to scrape, crawl, or extract data from web pages.
argument-hint	<url> [--mode auto|fast|browser|stealth] [--extract '<schema>'] [--discover] [--screenshot]
allowed-tools	Bash(python *) Bash(pip *) Read Write Edit

/scraper:scrape — Universal Web Scraper

You have access to a powerful, product-agnostic web scraper at ${CLAUDE_SKILL_DIR}/scraper.py.

Setup (first run only)

pip install httpx                                                    # Required
pip install playwright && python -m playwright install chromium      # For browser/stealth modes
pip install anthropic                                                # For --extract (Claude extraction)

Quick Start

# Single URL (auto-detects best engine)
python ${CLAUDE_SKILL_DIR}/scraper.py $ARGUMENTS

# Force a specific mode
python ${CLAUDE_SKILL_DIR}/scraper.py <url> --mode stealth

# Extract structured data via Claude
python ${CLAUDE_SKILL_DIR}/scraper.py <url> --extract '{"title": "", "price": 0}'

Engines (Modes)

Mode	Engine	Best For	Concurrency
auto	httpx -> Playwright -> stealth	General use (default)	10
fast	httpx only	Static HTML, APIs, high throughput	20+
browser	Playwright headless	SPAs, JS-rendered pages	5
stealth	Playwright non-headless + anti-detect	Cloudflare, bot protection	3

Auto mode tries httpx first, detects JS-heavy pages or Cloudflare challenges, and escalates automatically.

How to Use This Skill

When the user asks you to scrape, follow these steps:

1. Determine the right approach

• Single URL, just read content -> run scraper.py directly
• Multiple URLs -> use --file urls.txt or pass multiple URLs
• Need structured data -> use --extract with a JSON schema
• Cloudflare/protected site -> use --mode stealth
• Crawl subpages -> use --follow '/path1,/path2'

2. Ensure dependencies are installed

Check if httpx is available. If the user needs browser/stealth mode, ensure Playwright is installed too:

pip install httpx
# Only if browser/stealth needed:
pip install playwright && python -m playwright install chromium

3. Run the scraper

# Basic scrape
python ${CLAUDE_SKILL_DIR}/scraper.py https://example.com --format text

# Save to directory
python ${CLAUDE_SKILL_DIR}/scraper.py https://example.com -o ./scraped_data

# Batch from file
python ${CLAUDE_SKILL_DIR}/scraper.py --file urls.txt --mode fast -c 20 -o ./results

# With Claude extraction
python ${CLAUDE_SKILL_DIR}/scraper.py https://example.com \
  --extract '{"company_name": "", "pricing_tiers": [{"name": "", "price": 0}]}'

# Crawl base + subpages
python ${CLAUDE_SKILL_DIR}/scraper.py https://example.com \
  --follow '/about,/pricing,/docs,/team' \
  --format markdown -o ./site_content

4. Post-process results

After scraping, help the user with what they actually need:

• Summarize the content
• Extract specific data points
• Compare information across pages
• Save in their desired format

CLI Reference

python scraper.py <urls...> [options]

Positional:
  urls                    URLs to scrape (space-separated)

Options:
  --file, -f FILE         Read URLs from file (one per line)
  --mode, -m MODE         Engine: auto|fast|browser|stealth (default: auto)
  --format FORMAT         Output: text|html|markdown|json (default: text)
  --output-dir, -o DIR    Save results to directory
  --extract, -e SCHEMA    JSON schema for Claude extraction
  --follow PATHS          Comma-separated paths to crawl (e.g. '/about,/pricing')
  --discover              Auto-discover and follow internal links
  --depth N               Max crawl depth for --discover (default: 1)
  --screenshot            Capture page screenshots as PNG (requires -o)
  --proxy URL             HTTP/SOCKS5 proxy (e.g. http://user:pass@host:port)
  --retries, -r N         Retries with exponential backoff per URL (default: 2)
  --concurrency, -c N     Max parallel requests (default: 10)
  --delay, -d SECS        Delay between requests (default: 1.5)
  --timeout, -t MS        Per-page timeout in ms (default: 30000)
  --verbose, -v           Debug logging

Anti-Bot Techniques

The scraper uses these techniques (escalating with mode):

fast mode: Realistic User-Agent, Accept headers, SSL verification disabled for broken certs

browser mode: All of fast, plus full JS execution, networkidle wait, realistic viewport (1440x900), Sec-Fetch headers, random delays

stealth mode: All of browser, plus non-headless Chromium, AutomationControlled blink feature disabled, navigator.webdriver patched to undefined, fresh browser instance per request

Programmatic Usage

The scraper can also be imported as a Python library:

import asyncio
from scraper import fetch_auto, fetch_stealth, scrape_urls, extract_with_claude

# Single URL
result = asyncio.run(fetch_auto("https://example.com"))
print(result["text"])

# Batch scrape
results = asyncio.run(scrape_urls(
    ["https://a.com", "https://b.com"],
    mode="auto",
    concurrency=5,
))

# Extract structured data
data = extract_with_claude(
    result["text"],
    '{"title": "", "price": 0}',
    url=result["url"],
)