crawler

name	crawler
description	Advanced web crawler skill for bypassing anti-bot protections and scraping data. Use when the user wants to crawl, scrape, or extract data from websites, especially those with anti-bot mechanisms like Cloudflare, DataDome, PerimeterX, reCAPTCHA, or JavaScript rendering requirements. Triggers include: 'crawl this site', 'scrape data from', 'bypass Cloudflare', 'extract data', 'spider this website', 'write a scraper', or any task requiring programmatic web data extraction with anti-bot bypass.
allowed-tools	Bash(python3:*), Bash(pip3:*), Bash(node:*), Bash(npx:*), Bash(agent-browser:*), Bash(chmod:*), Bash(mkdir:*), Bash(cat:*), Bash(ls:*)

Web Crawler Skill

Agent vs Script Responsibilities

Step	Who	How
Detect anti-bot	🔧 Script	python3 scripts/detect.py <URL>
Choose tool	🔧 Script	crawl.py --mode auto does this automatically
Fetch page	🔧 Script	crawl.py handles Fetcher/StealthyFetcher/DynamicFetcher
Decide WHAT to extract	🤖 Agent	You understand user intent, choose selectors
Write custom extraction logic	🤖 Agent	You write the Python code for specific data
Debug failures	🤖 Agent	You interpret errors and adjust strategy
Handle login/CAPTCHA	🤖 Agent	You decide auth approach, use login_session.py or agent-browser

Key principle: Scripts handle the mechanics (detect, fetch, parse). You handle the intelligence (what to extract, how to handle failures).

---

Two Commands You Need

1. crawl.py — Auto-Detect + Fetch + Extract

The primary tool. One command detects protection, chooses the right fetcher, and returns data.

# Auto-detect everything and extract all links:
python3 scripts/crawl.py <URL>

# Extract specific elements (you provide CSS selectors):
python3 scripts/crawl.py <URL> --selector "h2.title a" --fields "title:text href:url"

# Force a specific fetch mode:
python3 scripts/crawl.py <URL> --mode stealth
python3 scripts/crawl.py <URL> --mode dynamic

# JSON output (for programmatic parsing):
python3 scripts/crawl.py <URL> --output json

# Save results to file:
python3 scripts/crawl.py <URL> --output json --output-file results.json

What it returns:

• title: Page title
• links: All links on the page (text + href, resolved to absolute URLs)
• text_preview: First 2000 chars of page text
• text_length: Total text length
• mode: Which fetcher was used (static/stealth/dynamic)
• detection: Full anti-bot detection results
• extracted: Custom elements if --selector was used

When it's enough: You just need to see what's on a page, get all links, or extract data with known selectors.

2. detect.py — Just Detection (when you need diagnosis only)

python3 scripts/detect.py <URL> --verbose
python3 scripts/detect.py <URL> --json    # structured output

When to use: When crawl.py fails and you need to understand why, or when deciding which custom script to write.

---

When crawl.py Isn't Enough → You Write Code

Scenario Decision Tree

Can crawl.py get the data you need?
├── Yes → ✅ Done. Use crawl.py.
└── No → Why not?
    ├── Need specific structured data (products, tables, etc.)
    │   → Write a Scrapling script with custom selectors (see examples below)
    ├── Need pagination/crawling multiple pages
    │   → Write a loop script (see examples below)
    ├── Site requires login
    │   → Use login_session.py template or interactive login
    ├── Chinese platform detected by detect.py
    │   → Use MediaCrawler (see references/chinese-platforms.md)
    ├── Need to intercept API calls
    │   → Use Playwright network interception (see templates/api_interception.py)
    └── CAPTCHA that can't be auto-solved
        → Use agent-browser for manual solving

Headless vs. Headed Mode

Critical: Login flows (QR code, phone verification, OAuth) require headed mode (headless=False). In headless mode, the browser UI is invisible — the user cannot scan QR codes or see login forms.

Scenario	Mode	Why
Simple scraping (no login)	headless=True	Faster, no UI needed
Login required (QR code/phone)	headless=False	User must see browser to authenticate
Debugging selector issues	headless=False	Visual inspection of page elements
Cloudflare challenge	headless=True	Automated solving works headless

---

Writing Custom Extraction Code

Quick Scrapling Examples (Agent writes these)

Extract article titles from a page:

from scrapling import Fetcher
page = Fetcher(auto_match=False).get('https://example.com', impersonate='chrome131')
titles = page.css('h2.article-title::text').getall()
for t in titles[:10]:
    print(t)

Extract structured items (name + price + link):

from scrapling import Fetcher
from urllib.parse import urljoin
page = Fetcher(auto_match=False).get('https://shop.example.com', impersonate='chrome131')
base = page.url
for item in page.css('div.product'):
    name = item.css('h3::text').get()
    price = item.css('span.price::text').get()
    link = urljoin(base, item.css('a::attr(href)').get())
    print(f"{name} | {price} | {link}")

Countermeasures (StealthyFetcher for protected sites):

from scrapling import StealthyFetcher
page = StealthyFetcher.fetch('https://protected-site.com', headless=True, solve_cloudflare=True)
# Same extraction API as above
titles = page.css('h2::text').getall()

Chinese platforms (use MediaCrawler, NOT Scrapling):

# Install once:
git clone https://github.com/NanmiCoder/MediaCrawler.git
cd MediaCrawler && pip install -r requirements.txt

# Crawl with QR code login:
python main.py --platform xhs --lt qrcode --type search --keyword "AI"

DrissionPage Examples (for Chinese sites with heavy anti-bot)

CRITICAL: DrissionPage requires these settings to avoid hanging:

• Always use opts.set_load_mode("eager") — default "normal" can hang on slow resources
• Always pass timeout to page.get(url, timeout=15) — otherwise it may hang forever
• For login flows, MUST use headless=False

from DrissionPage import ChromiumPage, ChromiumOptions

opts = ChromiumOptions()
opts.headless(True)
opts.set_load_mode("eager")  # CRITICAL: avoids hanging
opts.set_argument("--disable-blink-features=AutomationControlled")
opts.set_argument("--no-sandbox")

page = ChromiumPage(opts)
page.get('https://example.com', timeout=15)  # CRITICAL: always set timeout

# Close login popup if present
close_btn = page.ele('css:.close-circle', timeout=2)
if close_btn:
    close_btn.click()

# Extract items
for card in page.eles('css:section.note-item'):
    title = card.ele('css:.title', timeout=1)
    if title:
        print(title.text)

---

Tool Selection Quick Reference

Site Type	Tool	Command
Static HTML (no anti-bot)	Scrapling Fetcher	crawl.py <URL> --mode static
Cloudflare-protected	Scrapling StealthyFetcher	crawl.py <URL> --mode stealth --solve-cloudflare
JS-rendered SPA	Scrapling DynamicFetcher	crawl.py <URL> --mode dynamic
Custom WAF	Scrapling StealthyFetcher	crawl.py <URL> --mode stealth
Chinese platform	MediaCrawler	See references/chinese-platforms.md
Login required	login_session.py template	See templates/login_session.py
Need specific data	Agent writes Scrapling/DrissionPage code	See examples above
API interception needed	Playwright network intercept	See templates/api_interception.py
CAPTCHA	agent-browser (manual)	agent-browser open <URL>

---

Template Files

Template	Use Case	Primary Tool
templates/scrapling_crawler.py	General scraping, auto-detect	Scrapling
templates/basic_requests.py	Lightweight HTTP	curl_cffi
templates/playwright_stealth.py	JS rendering + stealth	Playwright
templates/selenium_undetected.py	DataDome, PerimeterX	undetected-chromedriver
templates/cloudflare_bypass.py	Cloudflare Turnstile/Challenge	Playwright
templates/dynamic_content.py	Infinite scroll, lazy load	Playwright
templates/api_interception.py	Intercept API calls	Playwright
templates/login_session.py	Authenticated crawling	Playwright
templates/drissionpage_crawler.py	Chinese sites, no WebDriver	DrissionPage

Reference Documentation

Reference	Content
references/anti-detection.md	Anti-bot techniques & bypass methods
references/frameworks.md	Framework comparison & selection guide
references/strategies.md	Strategy decision flowchart & patterns
references/chinese-platforms.md	Chinese platform crawling guide

Important Rules

1. Start with crawl.py — It auto-detects and auto-selects the right fetcher.
2. For Chinese platforms detected by detect.py, skip crawl.py and use MediaCrawler or DrissionPage directly — crawl.py cannot handle platform-specific anti-bot signatures.
3. Only write custom code when crawl.py can't extract what you need — usually you just need different selectors.
4. For Chinese platforms, always recommend MediaCrawler first — don't write custom scrapers when MediaCrawler already handles them.
5. Use detect.py for diagnosis only — crawl.py already runs detection internally.
6. The agent's job is understanding intent and writing selectors — the scripts handle detection and fetching.
7. All crawler scripts must pass a full test-debug-fix-test cycle before delivery — unverified scripts must not be delivered.
8. If a page requires login, the agent must write a login module in the script — handle cookies/sessions, or bypass the auth wall programmatically. If automation is infeasible (QR code, CAPTCHA, etc.), the script may pause and prompt the user to log in manually, then resume crawling with the authenticated session.
9. Always use timeout parameter with DrissionPage's page.get() — without it, the browser may hang indefinitely. Also set opts.set_load_mode("eager").