| name | read-url |
| description | Extract clean, complete markdown from any web page โ articles, docs, READMEs, blog/social posts, academic papers. Also use as a fallback when curl returns noisy HTML or WebFetch returns truncated, summarized, or refused results.
|
| allowed-tools | ["Bash(npx defuddle*:*)","Bash(pnpx defuddle*:*)","Bash(defuddle*:*)","Bash(curl:*)","Bash(jq:*)","Bash(rg:*)","Bash(npm view:*)","Bash(pnpm view:*)","Bash(gh repo view:*)","Bash(gh gist view:*)","Bash(gh issue view:*)","Bash(gh pr view:*)","Bash(yt-dlp:*)","Bash(instaloader:*)","Bash(*html-select*:*)","Bash(*html-unescape*:*)","WebFetch"] |
Read URL
Work down this fallback ladder in order. Each step is only tried when prior steps don't apply or fail.
Fallback ladder
- Raw
.md / .txt / plain-text URL โ curl -sL <url> (already clean, no HTML to strip)
- Known site โ use the dedicated CLI/API from the routing table below
- Docs page โ try
curl -sL <url>.md. Mintlify and other docs platforms serve clean markdown on the .md route โ if the response is text/markdown, you're done; otherwise fall through
- Blog / newsletter / multi-post index โ try RSS first:
curl -sL <url>/feed (also /rss, /feed.xml, /atom.xml, /index.xml). Most static-site generators and CMS platforms expose one; RSS gives you clean <content:encoded> or <summary> bodies without chrome
- Generic site (articles, docs, tech blogs, unknown) โ
npx defuddle parse <url> --markdown โ see references/defuddle.md
- JS-rendered page (defuddle returns empty / skeleton-only content) โ
/agent-browser skill
- Cloudflare / anti-bot protection (Turnstile, blocked responses, 403/503) โ
/scrapling skill
- Still blocked and genuinely need this page โ ask the user to open it and paste the content, or offer the
/chrome-cdp skill (requires explicit user approval first). Otherwise, give up and report the failure.
Routing table
Step 2 โ URLs matching a known domain:
| Domain / Pattern | Preferred path |
|---|
github.com / gist.github.com | File via raw.githubusercontent.com; issue/PR via gh issue view / gh pr view; search via anonymous api.github.com/search/issues โ see references/github.md |
x.com / twitter.com / t.co | curl -sL https://api.fxtwitter.com/<user>/status/<id> | jq |
bilibili.com | /bilibili-api skill โ fetches video title, description, comments |
youtube.com / youtu.be | yt-dlp --dump-json --skip-download for title/description/metadata; yt-dlp --write-auto-sub --sub-lang en --skip-download for transcript |
arxiv.org / ssrn.com | /jina-ai skill |
mp.weixin.qq.com (ๅพฎไฟกๅ
ฌไผๅท) | /scrapling skill โ scrapling extract get <url> works without a browser |
www.cnblogs.com (ๅๅฎขๅญ) | Plain defuddle works โ server-rendered HTML with the article body inline. For a user's post index: curl -sL 'https://www.cnblogs.com/<user>/rss' (Atom feed) |
blog.csdn.net (CSDN) | /scrapling skill โ plain curl returns a JS-skeleton (content is JS-loaded) and defuddle hits 404 anti-bot. For a summary-only index: curl -sL 'https://blog.csdn.net/<user>/rss/list' returns RSS with ๆ่ฆ (not full bodies) |
zhihu.com / zhuanlan.zhihu.com (็ฅไน) | Hard โ plain curl returns a bot-challenge page and even scrapling's stealthy-fetch gets 403 with empty content. Try /chrome-cdp with the user's logged-in session, or ask them to paste the content |
juejin.cn (ๆ้) | /scrapling skill โ Nuxt SPA; escalate to if stealthy-fetch returns only shell |
Bulk discovery
For whole-site ingestion, probe <site>/llms.txt (URL index) and /llms-full.txt (full corpus). Convention adopted by Mintlify, Cloudflare, Stripe, Next.js, and others.
vs. WebFetch
This skill returns full page text (markdown), parsed locally โ no summarization, no information loss. WebFetch routes through a remote small model that may summarize, refuse, or truncate; reach for it only when you want an AI summary, not the content itself.
When to bypass the ladder
- Need a quick AI summary โ built-in WebFetch
- No specific URL yet, need to search โ built-in WebSearch or
/jina-ai skill
