Menu
统一采集框架 - 支持 RSS/Web/API，207+ 采集源，AI 评分/分类/摘要
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Inspect and manage guarded Codex App-native or launchd heartbeats for Codex main control threads. Use when the user wants Codex to periodically continue a specific thread only when it is idle, avoid interrupting active work, keep App-visible interactions, inspect thread status, or enable/disable project autopilot heartbeats.
Coordinate Lark/Feishu CLI tasks from Codex. Use when a request involves Feishu/Lark docs, wiki, Drive files, Sheets, Base, Calendar, Minutes, IM, Mail, Tasks, Contacts, Slides, whiteboards, approvals, attendance, or when a Lark URL/token must be read, edited, summarized, routed, or turned into a real work artifact.
Use when the user asks Codex to research, find learning materials, process "素材：" links, "请你读" / "精读" a material, build a material radar, or use SenSight-like broad information retrieval for career learning and Agent infra tracking.
Organize tasting menus, menu photos, and visit notes into the right gourmet note. Use when the user shares a restaurant menu, wants a tasting write-up, or asks to merge a new visit into Notes/Gourmet.md without overwriting earlier visits.
Track and synthesize current AI hotspots into a bilingual HTML daily report. Use when the user says "AI热点", asks for AI日报/AI日报 HTML/AI hot topics, or wants the best recent AI products, papers, viewpoints, and cognition-shifting items from AI/news aggregators.
Extract a Markdown heading tree with line numbers before editing or integrating notes. Use when you need to inspect a long Markdown file quickly and choose the best insertion point instead of editing blind.
| name | common-fetcher |
| version | 1.0.0 |
| description | 统一采集框架 - 支持 RSS/Web/API，207+ 采集源，AI 评分/分类/摘要 |
| metadata | {"openclaw":{"emoji":"🕸️","category":"data","requires":{"bins":["node","npm"],"env":[],"config":[]},"primaryEnv":"","install":[{"id":"npm-install","kind":"node","package":"common-fetcher","bins":["common-fetcher"],"label":"Install common-fetcher (npm)"}]}} |
统一采集框架，为 AI Agent 提供强大的信息采集能力。
# 抓取煤炭行业数据
common-fetcher --industry coal --output daily.md
# 抓取房地产行业数据
common-fetcher --industry realestate --output daily.md
# 抓取 AI 技术数据
common-fetcher --industry ai --output daily.md
# 自定义采集源
common-fetcher --config custom-sources.json --output daily.md
import { CommonFetcher } from 'common-fetcher';
const fetcher = new CommonFetcher({
industry: 'coal',
maxArticles: 50,
timeout: 15000,
});
const result = await fetcher.fetch();
console.log(`成功抓取 ${result.totalArticles} 篇文章`);
在 openclaw.json 中配置：
{
"skills": {
"common-fetcher": {
"enabled": true,
"industry": "coal",
"schedule": "0 8 * * *"
}
}
}
┌─────────────────────────────────────────┐
│ Common-Fetcher │
├─────────────────────────────────────────┤
│ Source Layer (采集源层) │
│ ├─ RSS 源 │
│ ├─ 网页源 │
│ └─ API 源 │
├─────────────────────────────────────────┤
│ Fetcher Layer (抓取层) │
│ ├─ RSS Fetcher (并发 + 超时) │
│ ├─ Web Scraper (cheerio) │
│ └─ Cache Manager │
├─────────────────────────────────────────┤
│ Processor Layer (处理层) │
│ ├─ 去重 (标题/URL 哈希) │
│ ├─ 时间过滤 │
│ ├─ AI 评分/分类 │
│ └─ AI 摘要 │
├─────────────────────────────────────────┤
│ Output Layer (输出层) │
│ ├─ Markdown 报告 │
│ ├─ JSON 数据 │
│ └─ 多渠道推送 │
└─────────────────────────────────────────┘
| 解析器 | 文章数/次 | 耗时 | 成功率 |
|---|---|---|---|
| 观点地产网 | 30 篇 | 605ms | 100% |
| 煤炭资源网 | 30 篇 | 455ms | 100% |
| 房天下 | 17 篇 | 579ms | 100% |
| MIT Tech Review | 9 篇 | 393ms | 100% |
| 总计 | 86 篇/次 | ~2s | 100% |
在 config/ 目录下管理采集源：
coal-sources.json - 煤炭行业采集源realestate-sources.json - 房地产行业采集源ai-sources.json - AI 技术采集源自定义解析器参考 src/parsers/ 目录：
export function parseGuandian(html: string, baseUrl: string): Article[] {
// 解析逻辑
}
欢迎提交 Issue 和 PR！
MIT License
Common-Fetcher - 为 AI Agent 提供强大的信息采集能力 🕸️