Menu
智能网页内容抓取工具，根据URL自动选择最佳抓取策略
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
火山引擎语音识别 - 将音频/视频转文字，支持长音频分段识别。
Use when publishing an Obsidian plugin to the community plugin store, creating GitHub Releases for Obsidian plugins, or submitting PRs to obsidianmd/obsidian-releases
一键部署静态网站到 Vercel 和 Cloudflare Pages。支持 HTML/React/Vue 等任意静态项目。
将 Expo/React Native 项目通过 EAS Build 构建 iOS 包并提交到 TestFlight。支持从零配置 EAS、构建、提交全流程。
综合画图工具 - AI 生图 (Nanobanana/Gemini) + 图片后处理（拼接/抠图/圆形裁剪/合成）。支持中英文自然语言画图。
统一搜索与抓取能力层 - 整合所有外部信息获取接口（搜索、抓取、下载）
| name | webfetch-plus |
| shortname | 🌐 webfetch+ |
| description | 智能网页内容抓取工具，根据URL自动选择最佳抓取策略 |
| version | 1.0.0 |
| author | Claude |
| tags | ["fetch","scraper","twitter","social-media","firecrawl","api"] |
WebFetchPlus 是一个强大的智能网页抓取工具，能够根据URL类型自动选择最佳的抓取策略：
# 自动识别并抓取
skill webfetch-plus "https://x.com/elonmusk/status/123456789"
# 抓取抖音视频
skill webfetch-plus "https://v.douyin.com/xxxxx"
# 搜索Twitter
skill webfetch-plus --search "AI news"
# 批量抓取
skill webfetch-plus --batch urls.txt
# 指定策略
skill webfetch-plus "https://example.com" --strategy firecrawl
# JSON输出
skill webfetch-plus "https://x.com/OpenAI" --json
from webfetch_plus import WebFetchPlus
# 初始化
fetcher = WebFetchPlus()
# 抓取推文
result = fetcher.fetch("https://x.com/elonmusk/status/123456789")
print(result["text"])
print(result["author"]["username"])
# 搜索Twitter
results = fetcher.search_twitter("GPT-4", max_results=20)
for tweet in results["results"]:
print(f"@{tweet['author']['username']}: {tweet['text']}")
# 批量抓取
urls = ["url1", "url2", "url3"]
results = fetcher.batch_fetch(urls)
# 获取用户信息
user = fetcher.fetch("https://x.com/OpenAI")
print(f"{user['name']} - {user['metrics']['followers_count']} followers")
{
"id": "123456789",
"text": "推文内容",
"created_at": "2024-01-28T10:00:00Z",
"author": {
"name": "Elon Musk",
"username": "elonmusk",
"verified": true
},
"metrics": {
"like_count": 10000,
"retweet_count": 5000,
"reply_count": 1000
},
"media": [...],
"_metadata": {
"url": "原始URL",
"strategy": "twitter_api",
"timestamp": "抓取时间",
"success": true
}
}
{
"title": "视频标题",
"author": "作者名",
"video_url": "视频地址",
"cover_url": "封面地址",
"stats": {
"likes": 10000,
"comments": 500,
"shares": 200
},
"_metadata": {...}
}
配置文件位置：~/.claude/skills/webfetch-plus/config.json
{
"twitter_api_key": "your_key",
"firecrawl_api_key": "your_key",
"proxy": null,
"timeout": 30,
"max_retries": 3,
"save_path": "~/webfetch_downloads",
"debug": false,
"strategies": {
"twitter": {
"enabled": true,
"api_base": "https://api.twitterapi.io"
},
"firecrawl": {
"enabled": false,
"api_base": "https://api.firecrawl.dev"
},
"douyin": {
"enabled": true,
"use_cookies": true
}
}
}
支持通过环境变量配置：
export TWITTER_API_KEY="your_twitter_api_key"
export FIRECRAWL_API_KEY="your_firecrawl_api_key"
export WEBFETCH_PROXY="http://proxy:port"
# 添加自定义抓取策略
from webfetch_plus import WebFetchPlus, FetchStrategy
fetcher = WebFetchPlus()
fetcher.add_strategy(
FetchStrategy.CUSTOM,
patterns=[r'custom\.site\.com'],
handler=my_custom_handler
)
# 添加请求中间件
def auth_middleware(request):
request.headers["Authorization"] = "Bearer token"
return request
fetcher.add_middleware(auth_middleware)
# 启用缓存
fetcher.enable_cache(ttl=3600) # 1小时缓存
# 清除缓存
fetcher.clear_cache()
所有错误都会在返回的JSON中包含error字段：
{
"error": "错误描述",
"_metadata": {
"url": "请求的URL",
"strategy": "使用的策略",
"timestamp": "时间戳",
"success": false
}
}
获取抓取统计：
skill webfetch-plus --stats
输出：
{
"total_fetches": 100,
"success": 95,
"failed": 5,
"success_rate": "95.0%"
}