Menü
Scrape sites with stealth browsing and Cloudflare bypass.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Basierend auf der SOC-Berufsklassifikation
Query and edit a SiYuan knowledge base via its API.
Create, read, edit Excel .xlsx spreadsheets and CSVs.
Create, read, edit Excel .xlsx spreadsheets and CSVs.
Curate LLM training data: dedupe, filter, PII redaction.
Clean training loops with built-in distributed support.
Roleplay a hostile user to find and triage UX pain points.
| name | scrapling |
| description | Scrape sites with stealth browsing and Cloudflare bypass. |
| version | 1.0.0 |
| author | FEUAZUR |
| license | MIT |
| platforms | ["linux","macos","windows"] |
| metadata | {"hermes":{"tags":["Web Scraping","Browser","Cloudflare","Stealth","Crawling","Spider"],"related_skills":["duckduckgo-search","domain-intel"],"homepage":"https://github.com/D4Vinci/Scrapling"}} |
| prerequisites | {"commands":["scrapling","python"]} |
Scrapling 是一个具备反机器人检测绕过、隐身式浏览器自动化以及爬虫框架功能的网页抓取工具。它提供了三种数据获取策略（HTTP、动态JS处理、隐身/Cloudflare模式），并配有完整的命令行界面。
本功能仅用于教育与研究目的。 用户必须遵守当地及国际上的数据爬取法规，并尊重相关网站的服务条款。
web_extract 工具无法获取所需数据时pip install "scrapling[all]"
scrapling install
最小化安装版本（仅支持 HTTP，无需浏览器）：
pip install scrapling
仅支持浏览器自动化功能：
pip install "scrapling[fetchers]"
scrapling install
| 方式 | 类别 | 适用场景 |
|---|---|---|
| HTTP | Fetcher / FetcherSession | 静态页面、API以及大量数据的快速请求 |
| 动态渲染 | DynamicFetcher / DynamicSession | 通过JS动态生成的内容、单页应用 |
| 隐蔽访问 | StealthyFetcher / StealthySession | 面临Cloudflare防护或反爬机制的网站 |
| 爬虫模式 | Spider | 基于链接追踪进行多页面爬取 |
scrapling extract get 'https://example.com' output.md
支持使用 CSS 选择器及浏览器模拟功能：
scrapling extract get 'https://example.com' output.md \
--css-selector '.content' \
--impersonate 'chrome'
scrapling extract fetch 'https://example.com' output.md \
--css-selector '.dynamic-content' \
--disable-resources \
--network-idle
scrapling extract stealthy-fetch 'https://protected-site.com' output.html \
--solve-cloudflare \
--block-webrtc \
--hide-canvas
scrapling extract post 'https://example.com/api' output.json \
--json '{"query": "search term"}'
输出格式由文件扩展名决定：
.html -- 原始 HTML 格式.md -- 转换为 Markdown 格式.txt -- 纯文本格式.json / .jsonl -- JSON 格式from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
for q in quotes:
print(q)
from scrapling.fetchers import FetcherSession
with FetcherSession(impersonate='chrome') as session:
page = session.get('https://example.com/', stealthy_headers=True)
links = page.css('a::attr(href)').getall()
for link in links[:5]:
sub = session.get(link)
print(sub.css('h1::text').get())
page = Fetcher.post('https://api.example.com/data', json={"key": "value"})
page = Fetcher.put('https://api.example.com/item/1', data={"name": "updated"})
page = Fetcher.delete('https://api.example.com/item/1')
page = Fetcher.get('https://example.com', proxy='http://user:pass@proxy:8080')
适用于需要执行JavaScript的页面（单页应用、延迟加载内容）：
from scrapling.fetchers import DynamicFetcher
page = DynamicFetcher.fetch('https://example.com', headless=True)
data = page.css('.js-loaded-content::text').getall()
page = DynamicFetcher.fetch(
'https://example.com',
wait_selector=('.results', 'visible'),
network_idle=True,
)
关闭字体、图片、媒体文件及样式表（可提升约25%的速度）：
from scrapling.fetchers import DynamicSession
with DynamicSession(headless=True, disable_resources=True, network_idle=True) as session:
page = session.fetch('https://example.com')
items = page.css('.item::text').getall()
from playwright.sync_api import Page
from scrapling.fetchers import DynamicFetcher
def scroll_and_click(page: Page):
page.mouse.wheel(0, 3000)
page.wait_for_timeout(1000)
page.click('button.load-more')
page.wait_for_selector('.extra-results')
page = DynamicFetcher.fetch('https://example.com', page_action=scroll_and_click)
results = page.css('.extra-results .item::text').getall()
适用于受 Cloudflare 保护或存在严重指纹特征的网站：
from scrapling.fetchers import StealthyFetcher
page = StealthyFetcher.fetch(
'https://protected-site.com',
headless=True,
solve_cloudflare=True,
block_webrtc=True,
hide_canvas=True,
)
content = page.css('.protected-content::text').getall()
from scrapling.fetchers import StealthySession
with StealthySession(headless=True, solve_cloudflare=True) as session:
page1 = session.fetch('https://protected-site.com/page1')
page2 = session.fetch('https://protected-site.com/page2')
所有数据获取器都会返回一个包含以下方法的 Selector 对象：
page.css('h1::text').get() # First h1 text
page.css('a::attr(href)').getall() # All link hrefs
page.css('.quote .text::text').getall() # Nested selection
page.xpath('//div[@class="content"]/text()').getall()
page.xpath('//a/@href').getall()
page.find_all('div', class_='quote') # By tag + attribute
page.find_by_text('Read more', tag='a') # By text content
page.find_by_regex(r'\$\d+\.\d{2}') # By regex pattern
查找结构相似的元素（适用于产品列表等场景）：
first_product = page.css('.product')[0]
all_similar = first_product.find_similar()
el = page.css('.target')[0]
el.parent # Parent element
el.children # Child elements
el.next_sibling # Next sibling
el.prev_sibling # Previous sibling
用于通过链接追踪进行多页面爬取：
from scrapling.spiders import Spider, Request, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com/"]
concurrent_requests = 10
download_delay = 1
async def parse(self, response: Response):
for quote in response.css('.quote'):
yield {
"text": quote.css('.text::text').get(),
"author": quote.css('.author::text').get(),
"tags": quote.css('.tag::text').getall(),
}
next_page = response.css('.next a::attr(href)').get()
if next_page:
yield response.follow(next_page)
result = QuotesSpider().start()
print(f"Scraped {len(result.items)} quotes")
result.items.to_json("quotes.json")
将请求路由至不同的获取器类型：
from scrapling.fetchers import FetcherSession, AsyncStealthySession
class SmartSpider(Spider):
name = "smart"
start_urls = ["https://example.com/"]
def configure_sessions(self, manager):
manager.add("fast", FetcherSession(impersonate="chrome"))
manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
async def parse(self, response: Response):
for link in response.css('a::attr(href)').getall():
if "protected" in link:
yield Request(link, sid="stealth")
else:
yield Request(link, sid="fast", callback=self.parse)
spider = QuotesSpider(crawldir="./crawl_checkpoint")
spider.start() # Ctrl+C to pause, re-run to resume from checkpoint
pip install 安装后，必须运行 scraping install——若未执行此步骤，DynamicFetcher 和 StealthyFetcher 将无法正常工作。DynamicFetcher/StealthyFetcher 的超时时间以毫秒为单位（默认值为 30000），而 Fetcher 的超时时间则以秒为单位。solve_cloudflare=True 会使数据获取时间增加 5 至 15 秒——请仅在必要时启用该选项。StealthyFetcher 需要运行真实的浏览器，因此请控制其并发使用数量。robots.txt 文件及服务条款。本库仅用于教育和研究目的。