Menu
增强版网页抓取功能。支持会话管理、多模式提取（CSS/XPath/正则）、并发抓取、反爬虫增强。提供完整的命令行接口。
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Generate images using Qwen Image API (Alibaba Cloud DashScope). Use when users request image generation with Chinese prompts or need high-quality AI-generated images from text descriptions.
Transform agents from task-followers into proactive partners.
Orchestrate multi-agent teams with defined roles, task lifecycles, handoff protocols, and review workflows. Use when: (1) Setting up a team of 2+ agents with different specializations, (2) Defining task routing and lifecycle (inbox → spec → build → review → done), (3) Creating handoff protocols between agents, (4) Establishing review and quality gates, (5) Managing async communication and artifact sharing between agents.
自动生成短视频。支持图文轮播、文字动画、卡片风格和 AI 视频大模型生成等多种视频类型。用户需要生成视频时使用此技能。
Process multiple items with progress tracking, checkpointing, and failure recovery.
增强版浏览器自动化。基于 Playwright 实现完整浏览器操作，包括基础导航、截图、点击、表单填写、内容提取、文件上传/下载、多标签页管理。支持反检测和高级自动化。
| name | web-scraper |
| triggers | null |
| version | 1.0.0 |
| status | enhanced |
| description | 增强版网页抓取功能。支持会话管理、多模式提取（CSS/XPath/正则）、并发抓取、反爬虫增强。提供完整的命令行接口。 |
| provides | null |
| os | null |
| clawdbot | null |
| emoji | 🕷️ |
| category | data |
| priority | high |
| updated | "2026-03-11T00:00:00.000Z" |
✅ 状态：增强版已上线，功能完整。
~/.xyvaclaw/output/scraping/~/.xyvaclaw/sessions/web-scraper/用户说"抓取网页"或"提取数据"时自动触发。
# 健康检查
python3 scripts/scraper_cli.py --check
# 基础抓取
python3 scripts/scraper_cli.py --url "https://example.com" --css '{"title": "h1"}'
# 使用会话
python3 scripts/scraper_cli.py --url "https://example.com" --session "my_session"
# XPath 提取
python3 scripts/scraper_cli.py --url "https://example.com" --xpath '{"title": "//h1/text()"}'
# 正则提取
python3 scripts/scraper_cli.py --url "https://example.com" --regex '{"emails": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"}'
# 并发抓取
python3 scripts/scraper_cli.py --urls "https://example1.com" "https://example2.com" --concurrent
# 指定输出文件
python3 scripts/scraper_cli.py --url "https://example.com" --output "result.json"
# 会话管理
python3 scripts/scraper_cli.py --list-sessions
python3 scripts/scraper_cli.py --clear-session --session "old_session"
from scraper_enhanced import scrape_enhanced, SessionManager, DataExtractor
# 简单抓取
result = scrape_enhanced(
url="https://example.com",
selectors={"title": "h1", "content": ".article"},
session_name="my_session"
)
# 使用会话管理器
session = SessionManager("test_session")
fetch_result = session.fetch("https://example.com")
# 使用数据提取器
extractor = DataExtractor(fetch_result["content"])
css_data = extractor.extract_css({"title": "h1"})
xpath_data = extractor.extract_xpath({"title": "//h1/text()"})
基于 requests + BeautifulSoup4 + lxml 实现，提供完整的抓取解决方案。