scrape

name

scrape

description

Web scraping and data extraction using Scrapling. Use this skill whenever the user wants to scrape a website, crawl pages, extract data from URLs, search for products/prices online, compare prices across sites, or pull structured data from web pages. Also use when the user mentions Coupang, Amazon, shopping search, price comparison, or any task involving fetching and parsing web content — even if they don't explicitly say "scrape".

Scrape — Web Scraping with Scrapling

Extract data from any website using Scrapling, a Python scraping framework with TLS fingerprint spoofing and anti-bot bypass.

Environment

• Python: ~/.scrapling-venv/bin/python3
• Chromium libs: ~/.local/lib/chromium-deps/usr/lib/x86_64-linux-gnu/
• Temp scripts: write to /tmp/scrapling_task.py, execute from there

When using StealthyFetcher or DynamicFetcher (headless Chromium), prefix the command with:

LD_LIBRARY_PATH="$HOME/.local/lib/chromium-deps/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH"

Choosing a Fetcher

Pick the lightest fetcher that works. Escalate only on failure.

Situation	Fetcher	Speed	Chromium?
Static HTML, no bot protection	Fetcher	⚡ fast	No
Akamai, Cloudflare, bot walls	StealthyFetcher	🐢 slow	Yes (LD_LIBRARY_PATH)
JS-rendered SPA (React, Vue)	DynamicFetcher	🐢 slow	Yes (LD_LIBRARY_PATH)
Multi-page crawl	Spider	⚡ parallel	No
Stateful (cookies/login)	StealthySession	🐢 slow	Yes (LD_LIBRARY_PATH)

Escalation order: Try Fetcher first. If 403/418/captcha → switch to StealthyFetcher. If content still missing (JS render) → DynamicFetcher.

Writing a Scraping Script

Every scraping task follows this pattern:

1. Write a Python script to /tmp/scrapling_task.py
2. Execute with the venv Python
3. Parse results and present to user

Fetcher (fast, no browser)

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://example.com', stealthy_headers=True, impersonate="chrome131")
print(f"Status: {page.status}")

# CSS selectors (Scrapy-style)
titles = page.css('h2::text').getall()
links = page.css('a::attr(href)').getall()

# XPath
items = page.xpath('//div[@class="item"]/text()').getall()

Key parameters: impersonate (browser TLS fingerprint), stealthy_headers (real browser headers), follow_redirects, timeout, headers, cookies.

Available impersonate values: chrome131, chrome136, firefox135, safari184, chrome_android, etc.

StealthyFetcher (beats Akamai/Cloudflare)

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch('https://protected-site.com',
    headless=True,
    network_idle=True,        # wait for all network requests to finish
    solve_cloudflare=True,    # auto-solve Turnstile
)

Session (cookie persistence)

from scrapling.fetchers import StealthySession

with StealthySession(headless=True) as session:
    page1 = session.fetch('https://site.com/login')
    page2 = session.fetch('https://site.com/data')  # cookies carry over

Spider (multi-page)

from scrapling.spiders import Spider, Response

class MySpider(Spider):
    name = "crawler"
    start_urls = ["https://example.com/page/1"]
    concurrent_requests = 5

    async def parse(self, response: Response):
        for item in response.css('.product'):
            yield {
                "name": item.css('.name::text').get(),
                "price": item.css('.price::text').get(),
            }
        next_page = response.css('.next a::attr(href)').get()
        if next_page:
            yield response.follow(next_page)

result = MySpider().start()
result.items.to_json("/tmp/output.json")

Execution

# Fetcher (no Chromium needed)
~/.scrapling-venv/bin/python3 /tmp/scrapling_task.py

# StealthyFetcher / DynamicFetcher (Chromium needed)
LD_LIBRARY_PATH="$HOME/.local/lib/chromium-deps/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH" \
  ~/.scrapling-venv/bin/python3 /tmp/scrapling_task.py

Site-Specific Playbooks

Read references/sites.md for site-specific parsing patterns (Coupang, Amazon, etc.). These change over time — when a known pattern fails, investigate the HTML and update the reference.

Error Recovery

Error	Fix
403 / Access Denied	Escalate to StealthyFetcher
418 / bot detection	StealthyFetcher + solve_cloudflare=True
Empty content (JS)	DynamicFetcher + network_idle=True
libnspr4.so not found	Add LD_LIBRARY_PATH prefix
Import error	~/.scrapling-venv/bin/pip install "scrapling[all]"
Timeout	Increase timeout parameter

Output

• Present extracted data in a clean table or summary
• For large datasets, save to /tmp/ as JSON and tell the user the path
• Always include source URLs/links in results