huangruiteng
huangruiteng/CS-Notes
相关职业SOC
基于 SOC 职业分类
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/huangruiteng/CS-Notes --skill common-fetcher命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
Qualify the exact final diff for a LoopX-managed goal. Use when goal policy enables change_quality_qualification, before a non-trivial delivery or merge, and when producing or repairing an exact-scope quality receipt. The workflow is language-neutral, permits at most one policy-authorized safe-fix pass, and never grants merge or repository authority.
Operate an explicitly activated LoopX Material Lifecycle for a connected project. Use for material-store inventory, lossless migration, candidate/archive transitions, exact-read-backed ranking, ranked-entry rebuilds, bounded Explore intake, owner-gated apply, rollback, and audit. Do not use for ordinary one-off reading or research when the project has not activated Material Lifecycle.
Use when the user asks Codex to research, find learning materials, process "素材：" links, "请你读" / "精读" a material, build a material radar, or use SenSight-like broad information retrieval for career learning and Agent infra tracking. Do not use the career/Agent-infra routing bias for user-directed `整理笔记` into a named note.
正在显示 SKILL.md
基于 SOC 职业分类
| name | common-fetcher |
| version | 1.0.0 |
| description | 统一采集框架 - 支持 RSS/Web/API，207+ 采集源，AI 评分/分类/摘要 |
| metadata | {"openclaw":{"emoji":"🕸️","category":"data","requires":{"bins":["node","npm"],"env":[],"config":[]},"primaryEnv":"","install":[{"id":"npm-install","kind":"node","package":"common-fetcher","bins":["common-fetcher"],"label":"Install common-fetcher (npm)"}]}} |
统一采集框架，为 AI Agent 提供强大的信息采集能力。
# 抓取煤炭行业数据
common-fetcher --industry coal --output daily.md
# 抓取房地产行业数据
common-fetcher --industry realestate --output daily.md
# 抓取 AI 技术数据
common-fetcher --industry ai --output daily.md
# 自定义采集源
common-fetcher --config custom-sources.json --output daily.md
import { CommonFetcher } from 'common-fetcher';
const fetcher = new CommonFetcher({
industry: 'coal',
maxArticles: 50,
timeout: 15000,
});
const result = await fetcher.fetch();
console.log(`成功抓取 ${result.totalArticles} 篇文章`);
在 openclaw.json 中配置：
{
"skills": {
"common-fetcher": {
"enabled": true,
"industry": "coal",
"schedule": "0 8 * * *"
}
}
}
┌─────────────────────────────────────────┐
│ Common-Fetcher │
├─────────────────────────────────────────┤
│ Source Layer (采集源层) │
│ ├─ RSS 源 │
│ ├─ 网页源 │
│ └─ API 源 │
├─────────────────────────────────────────┤
│ Fetcher Layer (抓取层) │
│ ├─ RSS Fetcher (并发 + 超时) │
│ ├─ Web Scraper (cheerio) │
│ └─ Cache Manager │
├─────────────────────────────────────────┤
│ Processor Layer (处理层) │
│ ├─ 去重 (标题/URL 哈希) │
│ ├─ 时间过滤 │
│ ├─ AI 评分/分类 │
│ └─ AI 摘要 │
├─────────────────────────────────────────┤
│ Output Layer (输出层) │
│ ├─ Markdown 报告 │
│ ├─ JSON 数据 │
│ └─ 多渠道推送 │
└─────────────────────────────────────────┘
| 解析器 | 文章数/次 | 耗时 | 成功率 |
|---|---|---|---|
| 观点地产网 | 30 篇 | 605ms | 100% |
| 煤炭资源网 | 30 篇 | 455ms | 100% |
| 房天下 | 17 篇 | 579ms | 100% |
| MIT Tech Review | 9 篇 | 393ms | 100% |
| 总计 | 86 篇/次 | ~2s | 100% |
在 config/ 目录下管理采集源：
coal-sources.json - 煤炭行业采集源realestate-sources.json - 房地产行业采集源ai-sources.json - AI 技术采集源自定义解析器参考 src/parsers/ 目录：
export function parseGuandian(html: string, baseUrl: string): Article[] {
// 解析逻辑
}
欢迎提交 Issue 和 PR！
MIT License
Common-Fetcher - 为 AI Agent 提供强大的信息采集能力 🕸️