智能网页内容抓取工具，根据URL自动选择最佳抓取策略
npx skills add https://github.com/Leoyishou/personal-ai-company --skill webfetch-plusCopy and paste this command into Claude Code to install the skill
火山引擎语音识别 - 将音频/视频转文字，支持长音频分段识别。
Use when publishing an Obsidian plugin to the community plugin store, creating GitHub Releases for Obsidian plugins, or submitting PRs to obsidianmd/obsidian-releases
一键部署静态网站到 Vercel 和 Cloudflare Pages。支持 HTML/React/Vue 等任意静态项目。
将 Expo/React Native 项目通过 EAS Build 构建 iOS 包并提交到 TestFlight。支持从零配置 EAS、构建、提交全流程。
综合画图工具 - AI 生图 (Nanobanana/Gemini) + 图片后处理（拼接/抠图/圆形裁剪/合成）。支持中英文自然语言画图。
统一搜索与抓取能力层 - 整合所有外部信息获取接口（搜索、抓取、下载）
| name | webfetch-plus |
| shortname | 🌐 webfetch+ |
| description | 智能网页内容抓取工具，根据URL自动选择最佳抓取策略 |
| version | 1.0.0 |
| author | Claude |
| tags | ["fetch","scraper","twitter","social-media","firecrawl","api"] |
WebFetchPlus 是一个强大的智能网页抓取工具，能够根据URL类型自动选择最佳的抓取策略：
# 自动识别并抓取
skill webfetch-plus "https://x.com/elonmusk/status/123456789"
# 抓取抖音视频
skill webfetch-plus "https://v.douyin.com/xxxxx"
# 搜索Twitter
skill webfetch-plus --search "AI news"
# 批量抓取
skill webfetch-plus --batch urls.txt
# 指定策略
skill webfetch-plus "https://example.com" --strategy firecrawl
# JSON输出
skill webfetch-plus "https://x.com/OpenAI" --json
from webfetch_plus import WebFetchPlus
# 初始化
fetcher = WebFetchPlus()
# 抓取推文
result = fetcher.fetch("https://x.com/elonmusk/status/123456789")
print(result["text"])
print(result["author"]["username"])
# 搜索Twitter
results = fetcher.search_twitter("GPT-4", max_results=20)
for tweet in results["results"]:
print(f"@{tweet['author']['username']}: {tweet['text']}")
# 批量抓取
urls = ["url1", "url2", "url3"]
results = fetcher.batch_fetch(urls)
# 获取用户信息
user = fetcher.fetch("https://x.com/OpenAI")
print(f"{user['name']} - {user['metrics']['followers_count']} followers")
{
"id": "123456789",
"text": "推文内容",
"created_at": "2024-01-28T10:00:00Z",
"author": {
"name": "Elon Musk",
"username": "elonmusk",
"verified": true
},
"metrics": {
"like_count": 10000,
"retweet_count": 5000,
"reply_count": 1000
},
"media": [...],
"_metadata": {
"url": "原始URL",
"strategy": "twitter_api",
"timestamp": "抓取时间",
"success": true
}
}
{
"title": "视频标题",
"author": "作者名",
"video_url": "视频地址",
"cover_url": "封面地址",
"stats": {
"likes": 10000,
"comments": 500,
"shares": 200
},
"_metadata": {...}
}
配置文件位置：~/.claude/skills/webfetch-plus/config.json
{
"twitter_api_key": "your_key",
"firecrawl_api_key": "your_key",
"proxy": null,
"timeout": 30,
"max_retries": 3,
"save_path": "~/webfetch_downloads",
"debug": false,
"strategies": {
"twitter": {
"enabled": true,
"api_base": "https://api.twitterapi.io"
},
"firecrawl": {
"enabled": false,
"api_base": "https://api.firecrawl.dev"
},
"douyin": {
"enabled": true,
"use_cookies": true
}
}
}
支持通过环境变量配置：
export TWITTER_API_KEY="your_twitter_api_key"
export FIRECRAWL_API_KEY="your_firecrawl_api_key"
export WEBFETCH_PROXY="http://proxy:port"
# 添加自定义抓取策略
from webfetch_plus import WebFetchPlus, FetchStrategy
fetcher = WebFetchPlus()
fetcher.add_strategy(
FetchStrategy.CUSTOM,
patterns=[r'custom\.site\.com'],
handler=my_custom_handler
)
# 添加请求中间件
def auth_middleware(request):
request.headers["Authorization"] = "Bearer token"
return request
fetcher.add_middleware(auth_middleware)
# 启用缓存
fetcher.enable_cache(ttl=3600) # 1小时缓存
# 清除缓存
fetcher.clear_cache()
所有错误都会在返回的JSON中包含error字段：
{
"error": "错误描述",
"_metadata": {
"url": "请求的URL",
"strategy": "使用的策略",
"timestamp": "时间戳",
"success": false
}
}
获取抓取统计：
skill webfetch-plus --stats
输出：
{
"total_fetches": 100,
"success": 95,
"failed": 5,
"success_rate": "95.0%"
}