read-url

name	read-url
description	Extract clean, complete markdown from any web page — articles, docs, READMEs, blog/social posts, academic papers. Also use as a fallback when curl returns noisy HTML or WebFetch returns truncated, summarized, or refused results.
allowed-tools	["Bash(npx defuddle*:*)","Bash(pnpx defuddle*:*)","Bash(defuddle*:*)","Bash(curl:*)","Bash(jq:*)","Bash(rg:*)","Bash(npm view:*)","Bash(pnpm view:*)","Bash(gh repo view:*)","Bash(gh gist view:*)","Bash(gh issue view:*)","Bash(gh pr view:*)","Bash(yt-dlp:*)","Bash(instaloader:*)","Bash(*html-select*:*)","Bash(*html-unescape*:*)","WebFetch"]

Read URL

Work down this fallback ladder in order. Each step is only tried when prior steps don't apply or fail.

Fallback ladder

1. Raw .md / .txt / plain-text URL → curl -sL <url> (already clean, no HTML to strip)
2. Known site → use the dedicated CLI/API from the routing table below
3. Docs page → try curl -sL <url>.md. Mintlify and other docs platforms serve clean markdown on the .md route — if the response is text/markdown, you're done; otherwise fall through
4. Blog / newsletter / multi-post index → try RSS first: curl -sL <url>/feed (also /rss, /feed.xml, /atom.xml, /index.xml). Most static-site generators and CMS platforms expose one; RSS gives you clean <content:encoded> or <summary> bodies without chrome
5. Generic site (articles, docs, tech blogs, unknown) → npx defuddle parse <url> --markdown — see references/defuddle.md
6. JS-rendered page (defuddle returns empty / skeleton-only content) → /agent-browser skill
7. Cloudflare / anti-bot protection (Turnstile, blocked responses, 403/503) → /scrapling skill
8. Still blocked and genuinely need this page → ask the user to open it and paste the content, or offer the /chrome-cdp skill (requires explicit user approval first). Otherwise, give up and report the failure.

Routing table

Step 2 — URLs matching a known domain:

Domain / Pattern	Preferred path
github.com / gist.github.com	File via raw.githubusercontent.com; issue/PR via gh issue view / gh pr view; search: api.github.com/search/{code,issues,repositories}?q= (anonymous) — see references/github.md
x.com / twitter.com / t.co	curl -sL https://api.fxtwitter.com/<user>/status/<id> | jq
bilibili.com	/bilibili-api skill — fetches video title, description, comments
youtube.com / youtu.be	yt-dlp --dump-json --skip-download for title/description/metadata; yt-dlp --write-auto-sub --sub-lang en --skip-download for transcript
arxiv.org / ssrn.com	/jina-ai skill
mp.weixin.qq.com (微信公众号)	/scrapling skill — scrapling extract get <url> works without a browser
www.cnblogs.com (博客园)	Plain defuddle works — server-rendered HTML with the article body inline. For a user's post index: curl -sL 'https://www.cnblogs.com/<user>/rss' (Atom feed)
blog.csdn.net (CSDN)	/scrapling skill — plain curl returns a JS-skeleton (content is JS-loaded) and defuddle hits 404 anti-bot. For a summary-only index: curl -sL 'https://blog.csdn.net/<user>/rss/list' returns RSS with 摘要 (not full bodies)
zhihu.com / zhuanlan.zhihu.com (知乎)	scripts/fetch_zhihu.py <url> — see references/zhihu.md
juejin.cn (掘金)	/scrapling skill — Nuxt SPA; escalate to /chrome-cdp if stealthy-fetch returns only shell
(思否)

Rows tagged search: expose a dedicated search API for when you have a topic, not a URL. Otherwise run WebSearch or /jina-ai skill with a site: filter, then fetch the result URL via this ladder.

Bulk discovery

For whole-site ingestion, probe <site>/llms.txt (URL index) and /llms-full.txt (full corpus). Convention adopted by Mintlify, Cloudflare, Stripe, Next.js, and others. On 404, fetch the index page <site>/ instead.

vs. WebFetch

This skill returns full page text (markdown), parsed locally — no summarization, no information loss. WebFetch routes through a remote small model that may summarize, refuse, or truncate; reach for it only when you want an AI summary, not the content itself.

When to bypass the ladder

• Need a quick AI summary → built-in WebFetch
• No specific URL yet, need to search → built-in WebSearch or /jina-ai skill