qiaomu-markdown-proxy

# Markdown Proxy - URL to Markdown 将任意 URL 转为干净的 Markdown。支持需要登录的页面、PDF、专有平台。 ## Trigger Priority（先触发，再分流） - 用户给出 URL 并要求“读取、抓取、提取、解析、转 Markdown”时，优先使用本 Skill。 - 用户要求“读取这篇链接后再写稿、总结、翻译、做播客或分析”时，先用本 Skill 取回原文，再把 Markdown 交给下游 Skill；不要因为最终任务是写作而跳过抓取。 - `mp.weixin.qq.com` 和飞书文档属于专用路由。不要先试普通网页打开或通用内容解析器。 - 用户已经贴出全文时不触发。YouTube 交给 `qiaomu-youtube-download`，普通搜索问题交给搜索工具。 ## URL Routing (先判断再执行) 收到 URL 后，先判断类型，不同类型走不同通道： | URL Pattern | Route To | Reason | |-------------|----------|--------| | `mp.weixin.qq.com` | `scripts/fetch_weixin.sh` | 先代理，验证码页自动回退 Playwright | | `feishu.cn/docx/` `feishu.cn/wiki/` `larksuite.com/docx/` | `scripts/fetch_feishu.py` | 需飞书 API 认证 | | `youtube.com` `youtu.be` | `qiaomu-youtube-download` skill | YouTube 有专用工具链 | | `huggingface.co/papers/` | 提取 arXiv ID → `scripts/extract_tex.py` | HuggingFace 论文页实际是 arXiv 镜像，先找到 arXiv 链接再走 LaTeX 提取 | | `arxiv.org/abs/` `arxiv.org/pdf/` | `scripts/extract_tex.py` | 从 LaTeX 源码提取结构化内容 (章节/图表/公式) | | `.pdf` (URL or local path) | `scripts/extract_pdf.sh` | PDF 专用提取 | | All other URLs | `scripts/fetch.sh` | 代理级联自动 fallback | ## Workflow ### Step 1: Route by URL Type ``` if URL contains "mp.weixin.qq.com": → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_weixin.sh "URL" → Done if URL contains "feishu.cn/docx/" or "feishu.cn/wiki/" or "larksuite.com/docx/": → python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_feishu.py "URL" → Done if URL contains "huggingface.co/papers/": → First fetch the page (WebFetch) to find the arXiv URL → Then python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py "{arxiv_url}" → Done if URL contains "arxiv.org/abs/" or "arxiv.org/pdf/": → python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py "URL" → Done if URL contains "youtube.com" or "youtu.be": → Call qiaomu-youtube-download skill → Done if URL ends with ".pdf" or is local PDF path: if remote URL: → Try: curl -sL "https://r.jina.ai/{url}" → If fails: download + extract_pdf.sh if local path: → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_pdf.sh "PATH" → Done else: → bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh "URL" → Done ``` ### Step 2: Display Content After fetching, show to user: ``` Title: {title} Author: {author} (if available) Source: {platform} (公众号 / 飞书文档 / 网页 / PDF) URL: {original_url} Summary {3-5 sentence summary} Content {full Markdown, truncated at 200 lines if long} ``` ### Step 3: Continue or Save - **Composite request**（“读取后写稿/总结/分析”）：把提取结果直接交给下游任务，在同一轮继续；除非用户要求，不必额外保存源文件。 - **Extraction-only request**（“只读取/转 Markdown”）：保存到 `~/Downloads/{title}.md`，使用 YAML frontmatter。 - Filename: use article title, remove special characters. - Format: YAML frontmatter (title, author, date, url, source) + Markdown body. - Tell the user the saved path. - Skip saving if the user says “just preview” or “don't save”. Only stop after extraction when extraction was the complete request. If the user asked for a downstream deliverable, continue to that deliverable. ## Examples ### General URL ```bash bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh "https://example.com/article" ``` ### X/Twitter Post ```bash bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh "https://x.com/username/status/1234567890" ``` ### WeChat Article ```bash bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_weixin.sh "https://mp.weixin.qq.com/s/abc123" ``` ### Feishu Document ```bash python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch_feishu.py "https://xxx.feishu.cn/docx/xxxxxxxx" ``` ### arXiv LaTeX Source ```bash python3 ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_tex.py "https://arxiv.org/abs/1706.03762" ``` ### PDF (Remote) ```bash curl -sL "https://r.jina.ai/https://example.com/paper.pdf" ``` ### PDF (Local) ```bash bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/extract_pdf.sh "/path/to/paper.pdf" ``` ### With Custom Proxy ```bash bash ~/.agents/skills/qiaomu-markdown-proxy/scripts/fetch.sh "https://example.com" "http://127.0.0.1:7890" ``` ## Notes - r.jina.ai and defuddle.md require no API key - `fetch.sh` handles proxy cascade with automatic fallback - Content validation: filters error, login-wall, and WeChat verification pages; requires >5 lines - WeChat wrapper prefers dependency-free proxies, then uses local Playwright; when Python packages are missing and `uv` is available, it runs them in an isolated environment - Playwright fallback requires a Chromium runtime; install once with `python3 -m playwright install chromium` if absent - Feishu script requires: `FEISHU_APP_ID` + `FEISHU_APP_SECRET` env vars - PDF extraction tries: marker-pdf → pdftotext → pypdf - For detailed method documentation, see `references/methods.md`