feedgrab

name

feedgrab

description

Universal content grabber — fetch any URL and return structured Markdown. Supports X/Twitter, WeChat, Xiaohongshu, YouTube, GitHub, LinuxDo/IDCFlare/Discourse, Feishu/Lark, KDocs, Youdao Note, Zhihu, Bilibili (with subtitle transcription), Xiaoyuzhou podcasts, Ximalaya podcasts, Telegram, HackerNews, Medium, Reddit, Weibo, Douyin, Zsxq (Knowledge Planet), RSS, 300+ paywall news sites, and any web page. Use when user provides a URL and wants its content extracted.

feedgrab — Universal Content Grabber

Give it a URL, get back structured Markdown. Supports 17+ platforms with deep extraction.

Trigger

Activate when user provides a URL and wants content fetched/extracted/read:

• /feedgrab <URL>
• "Grab this article"
• "Read this tweet/post"
• "抓取这个链接"
• Any URL from supported platforms

Prerequisites Check

Before fetching, verify feedgrab is installed:

which feedgrab 2>/dev/null || command -v feedgrab 2>/dev/null

If NOT installed, tell the user:

feedgrab is not installed. Run `/feedgrab-setup` or manually:
  pip install feedgrab[all]
  feedgrab setup

Then stop — do not proceed without feedgrab.

Supported Platforms

Platform	URL Pattern	Method
X/Twitter	x.com/*/status/*, twitter.com/*	GraphQL → FxTwitter → Syndication → oEmbed → Jina → Playwright
WeChat (微信公众号)	mp.weixin.qq.com/*	Playwright JS evaluate → Jina
Xiaohongshu (小红书)	xiaohongshu.com/explore/*, xhslink.com/*	API (xhshow) → Pinia Store injection → Jina → Playwright
YouTube	youtube.com/watch?v=*, youtu.be/*, Shorts	InnerTube API → yt-dlp subtitles → Groq Whisper
GitHub	github.com/*/*	REST API (Chinese README priority + subdirectory scan)
LinuxDo / Discourse	linux.do/t/*	Discourse topic JSON API → CDP → Playwright in-page fetch → Jina
IDCFlare / Discourse	idcflare.com/t/*	Discourse topic JSON API → CDP → Playwright in-page fetch → Jina
Feishu/Lark (飞书)	feishu.cn/docx/*, feishu.cn/wiki/*	Open API → CDP → Playwright PageMain → Jina
KDocs (金山文档)	kdocs.cn/l/*	Playwright ProseMirror DOM (virtual scroll + CDP)
Youdao Note (有道云笔记)	share.note.youdao.com/*	JSON API → Playwright iframe → Jina
Zhihu (知乎)	zhihu.com/question/*/answer/*, zhuanlan.zhihu.com/p/*	API v4 → Playwright CDP/DOM → Jina
Bilibili (B站)	bilibili.com/video/*, b23.tv/*	API metadata + 3-tier subtitle fallback (v2 → WBI v2 → Whisper)
Xiaoyuzhou (小宇宙)	xiaoyuzhoufm.com/episode/*	SSR __NEXT_DATA__ + Groq Whisper transcription
Ximalaya (喜马拉雅)	ximalaya.com/sound/*, m.ximalaya.com/sound/*	Web Revision API + canPlay degradation + Groq Whisper
Telegram	t.me/*	Telethon
HackerNews	news.ycombinator.com/item?id=*	Firebase API v0 (item.json + first-layer comments + hn top/new/best/ask/show/jobs list batch)
Medium	medium.com/*, *.medium.com/*	Jina Reader → JSON-LD articleBody → Stealth Browser; user/publication batch via RSS
Reddit	reddit.com/r/*/comments/*, redd.it/*	old.reddit.com .json + self-UA → CDP → Stealth Playwright + saved session → Jina (with Top 50 comments + reddit-sub batch)
Weibo	weibo.com/*, weibo.cn/*, m.weibo.cn/status/*	m.weibo.cn /statuses/show + container/getIndex + SSR $render_data fallback (SUB cookie optional)
Douyin (抖音)	douyin.com/video/*, v.douyin.com/* (short link)	CDP → Stealth Playwright + saved session → SSR RENDER_DATA → Jina (browser-side signing, no algorithm break)
Zsxq (知识星球)	articles.zsxq.com/id_*.html, wx.zsxq.com/group/*/topic/*, t.zsxq.com/* (short)	HTTP cookie (articles SSR HTML / api.zsxq.com topic JSON) → CDP → Stealth Playwright → Jina (auth-walled, login required)
RSS	RSS/Atom feed URLs	feedparser
Paywall news (300+)	NYT/WSJ/FT/Economist/Bloomberg...	JSON-LD → Googlebot/Bingbot UA → AMP → EU IP → archive.today → Google Cache → Jina
Any web page	Any other URL	JSON-LD pre-scan → Jina Reader fallback

Pipeline

Step 1: Fetch Content

feedgrab "$ARGUMENTS"

The CLI auto-detects the platform and routes to the appropriate fetcher.

Step 2: Locate Output File

feedgrab saves output to OUTPUT_DIR (default: ./output/). Check the CLI output for the saved file path, typically:

• output/X/author_date：title.md
• output/mpweixin/author_date：title.md
• output/XHS/author_date：title.md
• output/YouTube/author_date：title.md
• output/GitHub/author_date：title.md
• output/LinuxDo/author_date：title.md
• output/IDCFlare/author_date：title.md
• output/Feishu/author_date：title.md
• output/KDocs/author_date：title.md
• output/NoteYouDao/author_date：title.md
• output/Zhihu/author_date：title.md
• output/Bilibili/author_date：title.md
• output/Xiaoyuzhou/author_date：title.md
• output/Ximalaya/author_date：title.md
• output/Web/author_date：title.md (paywall / generic pages)

Step 3: Read and Present

Read the output .md file and present the content to the user. The file includes:

• YAML front matter (title, source, author, published, likes, tags, etc.)
• Full article/tweet/post content in Markdown
• Images (as remote URLs or local paths if media download is enabled)

Clipboard Mode

If the user says "grab from clipboard" or the URL contains & (which breaks PowerShell):

feedgrab clip

This reads the URL from the system clipboard.

Error Handling

Error	Solution
feedgrab: command not found	Run /feedgrab-setup
Cookie expired / 401 / 403	feedgrab login <platform> to refresh
Jina timeout (30s)	feedgrab auto-retries with Playwright
Rate limit (429)	feedgrab auto-rotates cookies if configured
OUTPUT_DIR not set	feedgrab setup to configure

Tips

• For Twitter deep extraction (views, bookmarks, threads): configure cookies via feedgrab login twitter
• For WeChat articles: no login needed for single articles
• For Xiaohongshu: pip install xhshow for API mode (faster, no browser needed)
• For GitHub: set GITHUB_TOKEN for higher rate limits (5000/hr vs 60/hr)
• For LinuxDo / Discourse: feedgrab login linuxdo if the topic is private or the site requires a browser session / Cloudflare cookie
• For IDCFlare / Discourse: feedgrab login idcflare if the topic is private or the site requires a browser session / Cloudflare cookie
• For Discourse forums: default reply mode is OP + topic-author follow-up replies only; switch with LINUXDO_REPLY_MODE / IDCFLARE_REPLY_MODE (author / all / none)
• For Feishu: set FEISHU_APP_ID + FEISHU_APP_SECRET for Open API access
• For KDocs: feedgrab login kdocs to save session (or enable KDOCS_CDP_ENABLED=true to reuse running Chrome)
• For Zhihu: feedgrab login zhihu to save session (enables full answer content)
• For Bilibili subtitles: free tier (player/v2 + WBI) works out of the box; set BILIBILI_SUBTITLE_WHISPER=true for Whisper fallback on videos without subtitles
• For Xiaoyuzhou / Ximalaya: set GROQ_API_KEY for Whisper transcription (free tier = metadata + shownotes only)
• For paywall sites: PAYWALL_ENABLED=true by default, no extra config
• Run feedgrab doctor to diagnose issues