菜单
Web scraping with Scrapling: MCP tool wrappers for quick fetching, plus direct Python API for advanced scraping with selectors, sessions, and spiders
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Workspace and research management — dispatch analyses, monitor running agents, manage workspaces and threads.
Draw price lines, trendlines, zones, and event markers directly on a stock's price chart — reach for it whenever you'd otherwise describe a level, pattern, or event in prose. Renders live on MarketView and as a clickable preview card in any other chat.
Event tracker: earnings dates, economic releases, conferences, regulatory events
Financial model audit: structural checks, formula validation, integrity testing
Comparable company analysis: operating metrics, valuation multiples, peer benchmarking
DCF valuation: free cash flow projections, WACC, terminal value, sensitivity analysis
| name | web-scraping |
| description | Web scraping with Scrapling: MCP tool wrappers for quick fetching, plus direct Python API for advanced scraping with selectors, sessions, and spiders |
| license | MIT |
Two ways to scrape in the sandbox:
get(), fetch(), stealthy_fetch() directly. Synchronous, returns dicts.Auto-registered as top-level functions in the sandbox. No imports needed. Synchronous — no await.
Quick fetches can run inline via ExecuteCode. For spiders, multi-URL crawls, or anything you'll iterate on, write the scraper to work/<task_name>/scraper.py and run it via Bash — edit-and-rerun beats resubmitting code.
# Fast HTTP fetch → markdown
result = get(url="https://example.com", extraction_type="markdown")
print(result["status"]) # 200
print(result["url"]) # "https://example.com"
print(result["content"][0]) # markdown string (first element of list)
# Browser fetch for JS-rendered pages
result = fetch(url="https://spa-site.com", extraction_type="markdown", network_idle=True)
# Anti-bot bypass (Cloudflare, etc.)
result = stealthy_fetch(url="https://protected-site.com", extraction_type="markdown", solve_cloudflare=True)
All MCP tools return a dict (not a Page object):
{
"status": 200,
"url": "https://example.com",
"content": ["<markdown or html text>", ""] # list, use [0] for content
}
.css(), .xpath(), .find_all() methods — use BeautifulSoup to parse if needed.body, .headers, .cookies — only status, url, contentcontent is always a list; the actual text is content[0]The css_selector param returns raw HTML of matched elements, not parsed text:
# Returns HTML of matched elements — must parse manually
result = get(url="https://example.com", css_selector="h1", extraction_type="HTML")
html_fragment = result["content"][0]
# Parse with BeautifulSoup if you need text/attributes
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_fragment, "html.parser")
titles = [h1.get_text() for h1 in soup.find_all("h1")]
| Function | Use case | Key params |
|---|---|---|
get(url, ...) | Static pages, APIs | impersonate, stealthy_headers, timeout (seconds) |
fetch(url, ...) | JS-rendered SPAs | headless, network_idle, wait_selector, disable_resources, timeout (ms) |
stealthy_fetch(url, ...) | Anti-bot sites | All fetch params + solve_cloudflare, hide_canvas |
bulk_get(urls, ...) | Parallel HTTP | urls: list[str], same params as get |
bulk_fetch(urls, ...) | Parallel browser | urls: list[str], same params as fetch |
bulk_stealthy_fetch(urls, ...) | Parallel stealth | urls: list[str], same params as stealthy_fetch |
| Param | Default | Notes |
|---|---|---|
extraction_type | "markdown" | "markdown", "HTML", or "text" |
css_selector | None | Returns raw HTML of matched elements |
main_content_only | True | Extract <body> only |
proxy | None | Proxy URL |
For selectors, sessions, spiders, or when you need the full Page object. Requires imports. Async.
from scrapling.fetchers import AsyncFetcher
page = await AsyncFetcher.get("https://example.com", stealthy_headers=True)
print(page.status) # 200
print(page.body) # Raw bytes
print(page.headers) # Response headers
# CSS selectors (Scrapy-style pseudo-elements)
titles = page.css("h1::text").getall()
links = page.css("a::attr(href)").getall()
# XPath
items = page.xpath("//div[@class='item']/text()").getall()
# BeautifulSoup-style
divs = page.find_all("div", class_="content")
from scrapling.fetchers import DynamicFetcher
page = await DynamicFetcher.async_fetch(
"https://spa-website.com",
headless=True,
network_idle=True,
disable_resources=True,
timeout=30000,
wait_selector=".data-table",
)
rows = page.css("table.data-table tr")
for row in rows:
cells = row.css("td::text").getall()
from scrapling.fetchers import StealthyFetcher
page = await StealthyFetcher.async_fetch(
"https://protected-site.com",
headless=True,
solve_cloudflare=True,
network_idle=True,
)
from scrapling.fetchers import FetcherSession
with FetcherSession(impersonate="chrome") as session:
login_page = session.post("https://site.com/login", data={...})
dashboard = session.get("https://site.com/dashboard")
data = dashboard.css(".user-data::text").getall()
from scrapling.spiders import Spider, Request, Response
class PriceScraper(Spider):
name = "prices"
start_urls = ["https://example.com/products"]
concurrent_requests = 5
async def parse(self, response: Response):
for product in response.css(".product"):
yield {
"name": product.css(".name::text").get(),
"price": product.css(".price::text").get(),
}
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield Request(next_page)
spider = PriceScraper()
result = spider.start()
result.items.to_json("results/prices.json")
import html2text
converter = html2text.HTML2Text()
converter.body_width = 0 # No line wrapping
markdown = converter.handle(html_string)
| Need | Use |
|---|---|
| Quick page content as markdown | MCP get() or fetch() |
| Extract specific elements (CSS/XPath) | Direct Python API with selectors |
| Login + scrape authenticated pages | Direct Python API with sessions |
| Crawl many pages with pagination | Direct Python API with Spider |
| Bypass Cloudflare | MCP stealthy_fetch() or direct StealthyFetcher |
| Save results to file | Direct Python API (spider .to_json()) |