| name | read-url |
| description | Extract clean, complete markdown from any web page — articles, docs, READMEs, blog/social posts, academic papers. Also use as a fallback when curl returns noisy HTML or WebFetch returns truncated, summarized, or refused results.
|
| allowed-tools | ["Bash(npx defuddle*:*)","Bash(pnpx defuddle*:*)","Bash(defuddle*:*)","Bash(curl:*)","Bash(jq:*)","Bash(rg:*)","Bash(npm view:*)","Bash(pnpm view:*)","Bash(gh repo view:*)","Bash(gh gist view:*)","Bash(gh issue view:*)","Bash(gh pr view:*)","Bash(yt-dlp:*)","Bash(instaloader:*)","Bash(*html-select*:*)","Bash(*html-unescape*:*)","WebFetch"] |
Read URL
Work down this fallback ladder in order. Each step is only tried when prior steps don't apply or fail.
Fallback ladder
- Raw
.md / .txt / plain-text URL → curl -sL <url> (already clean, no HTML to strip)
- Known site → use the dedicated CLI/API from the routing table below
- Docs page → try
curl -sL <url>.md. Mintlify and other docs platforms serve clean markdown on the .md route — if the response is text/markdown, you're done; otherwise fall through
- Blog / newsletter / multi-post index → try RSS first:
curl -sL <url>/feed (also /rss, /feed.xml, /atom.xml, /index.xml). Most static-site generators and CMS platforms expose one; RSS gives you clean <content:encoded> or <summary> bodies without chrome
- Generic site (articles, docs, tech blogs, unknown) →
npx defuddle parse <url> --markdown — see references/defuddle.md
- JS-rendered page (defuddle returns empty / skeleton-only content) →
/agent-browser skill
- Cloudflare / anti-bot protection (Turnstile, blocked responses, 403/503) →
/scrapling skill
- Still blocked and genuinely need this page → ask the user to open it and paste the content, or offer the
/chrome-cdp skill (requires explicit user approval first). Otherwise, give up and report the failure.
Routing table
Step 2 — URLs matching a known domain:
| Domain / Pattern | Preferred path |
|---|
github.com / gist.github.com | File via raw.githubusercontent.com; issue/PR via gh issue view / gh pr view; search: api.github.com/search/{code,issues,repositories}?q= (anonymous) — see references/github.md |
x.com / twitter.com / t.co | curl -sL https://api.fxtwitter.com/<user>/status/<id> | jq |
bilibili.com | /bilibili-api skill — fetches video title, description, comments |
youtube.com / youtu.be | yt-dlp --dump-json --skip-download for title/description/metadata; yt-dlp --write-auto-sub --sub-lang en --skip-download for transcript |
arxiv.org / ssrn.com | /jina-ai skill |
mp.weixin.qq.com (微信公众号) | /scrapling skill — scrapling extract get <url> works without a browser |
www.cnblogs.com (博客园) | Plain defuddle works — server-rendered HTML with the article body inline. For a user's post index: curl -sL 'https://www.cnblogs.com/<user>/rss' (Atom feed) |
blog.csdn.net (CSDN) | /scrapling skill — plain curl returns a JS-skeleton (content is JS-loaded) and defuddle hits 404 anti-bot. For a summary-only index: curl -sL 'https://blog.csdn.net/<user>/rss/list' returns RSS with 摘要 (not full bodies) |
zhihu.com / zhuanlan.zhihu.com (知乎) | scripts/fetch_zhihu.py <url> — see references/zhihu.md |
juejin.cn (掘金) | /scrapling skill — Nuxt SPA; escalate to /chrome-cdp if stealthy-fetch returns only shell |
| (思否) |
Rows tagged search: expose a dedicated search API for when you have a topic, not a URL. Otherwise run WebSearch or /jina-ai skill with a site: filter, then fetch the result URL via this ladder.
Bulk discovery
For whole-site ingestion, probe <site>/llms.txt (URL index) and /llms-full.txt (full corpus). Convention adopted by Mintlify, Cloudflare, Stripe, Next.js, and others. On 404, fetch the index page <site>/ instead.
vs. WebFetch
This skill returns full page text (markdown), parsed locally — no summarization, no information loss. WebFetch routes through a remote small model that may summarize, refuse, or truncate; reach for it only when you want an AI summary, not the content itself.
When to bypass the ladder
- Need a quick AI summary → built-in WebFetch
- No specific URL yet, need to search → built-in WebSearch or
/jina-ai skill
