content-fetching

name	content-fetching
description	Fetch and clean web content for the wiki — RSS discovery, HTML-to-markdown extraction, login-session browser (Playwright Profile + iPhone UA), Playwright Stealth headless (init_script patches), and cloud anti-detection browser (TinyFish). Five techniques unified under one entry point; load the umbrella first, then read the relevant reference for the specific scenario.
tags	["content-fetching","rss","web","scraper","ocr","extraction","accessibility","login","captcha","js-rendering"]
category	wiki
related_skills	["wiki/manual-article-ingestion","wiki/web-content-reviewer","wiki/inbox-screener","research/web-scraper-comparison"]

Content Fetching & Cleaning

The wiki pipeline needs to get clean article content from many different source types (newsletter links, WeChat subscriptions, paywalled articles, JS-rendered SPAs, captcha-protected pages). Each requires a different technique.

Five Techniques

Technique	When to use	Reference
RSS feed discovery	Find RSS/Atom feeds for a new source you want to subscribe to	references/rss-feed-discovery.md
HTML-to-markdown extraction (Readability)	Already have HTML; need to clean to semantic markdown	references/readability-extractor.md
Login-session browser (Playwright Profile + iPhone UA)	Sites requiring cookies, OAuth, or persistent browser profile; also doubles as anti-bot bypass for heavy Chinese sites (Zhihu, XHS, CSDN). WeChat: use CDP Chrome (wechat-cdp-fetch.py) instead — Playwright Profile is fallback only.	references/playwright-profile-fetcher.md + references/playwright-mobile-ua-bypass.md
Playwright Stealth headless (init_script patches)	Heavy-bot sites that need headless/cron/SSH execution with no profile/API key (Zhihu zse-ck, CSDN, Xiaohongshu)	references/playwright-stealth-headless.md
Cloud anti-detection browser (TinyFish)	Cloudflare/CAPTCHA sites that block even stealth Playwright; cloud-based, requires API key	references/tinyfish-web-agent.md
AutoCLI (Rust CLI + Chrome Extension)	Browser-mode sites (Xiaohongshu, Zhihu, Bilibili, Weibo, Douban, etc.) that need login session + anti-bot — pip install-free, single 6MB binary, 55+ built-in site commands	references/autocli-cli.md
Zhihu-specific (zhuanlan.zhihu.com)	Thin wrapper linking to the general anti-bot techniques above	references/zhihu-and-cnsite-fetcher.md

Quick Decision Tree

Need web content?
│
├── Has RSS/Atom feed? ─────────► rss-feed-discovery.md
│
├── mp.weixin.qq.com (WeChat article)?
│   ├── RSS pipeline (auto) ──► CDP Chrome (Plan A): wechat-cdp-fetch.py
│   │   • 连接 Chrome CDP (port 9222), 导航到文章页, 提取完整正文
│   │   • 真实登录的 Chrome, 无反爬拦截, 成功率 ≈100%
│   │   • 要求: Chrome 后台运行 (launchd com.hermes.wechat-chrome)
│   │   • fallback: Playwright Profile (Plan B)
│   │   See: ~/wiki/drafts/wechat-pipeline-architecture.md
│   │
│   ├── User-pasted URL ──► CDP Chrome FIRST: wechat-cdp-fetch.py
│   │   • 直连 Chrome CDP, 跳过 curl/browser_navigate/Jina
│   │   • 速度: 3-5 秒/篇, 无反爬
│   │   • fallback: Playwright Profile (fetch-with-profile.js)
│   │   See: ~/wiki/scripts/wechat-cdp-fetch.py
│   │
├── Source URL, public site, no login?
│   ├── Simple blog/news ──────► Jina Reader (default)
│   ├── JS-rendered / CAPTCHA ─► tinyfish-web-agent.md
│   ├── Heavy-bot (headless/cron) ► playwright-stealth-headless.md
│   ├── Chinese heavy-bot (Zhihu/CSDN) ► zhihu-and-cnsite-fetcher.md
│   ├── Xiaohongshu (xhs.com)
│   ├── AutoCLI (autocli xiaohongshu search/user/feed) — best for search + user feeds
│   ├── Cloud browser (browser_navigate) — WILL be IP-blocked with "IP存在风险" error
│   │   └── Fallback: extract note ID from the blocked redirect URL (explore/<note_id>),
│   │       then search GitHub / web for the topic name — many小红书 posts reference
│   │       open-source projects the agent can read via raw.githubusercontent.com.
│   │       See references/cloud-browser-ip-block-fallback.md.
│   ├── xhs SDK + vision_analyze
│   └── computer_use (local Chrome with login session)
│   └── Need login / cookies ──► playwright-profile-fetcher.md
│
└── Already have raw HTML? ────► readability-extractor.md

Anti-patterns:

• Declaring "URL unreachable" after only trying curl + browser_navigate + Jina. The umbrella has at least 5 documented layers. Exhaust them before reporting failure.
• Cloud browser IP block — browser_navigate returning "IP存在风险" is not a dead end. The content ID is still in the redirect URL. Extract it and search GitHub/web for the topic. See references/cloud-browser-ip-block-fallback.md.
• WeChat curl-as-canary: trying curl first on mp.weixin.qq.com URLs to "save time." curl succeeds ~50% but the other 50% returns a 17KB verify.html CAPTCHA wall with empty metadata. The retry loop (curl→browser→Jina→finally Playwright) wastes 30-60s per URL. Go straight to CDP Chrome (wechat-cdp-fetch.py) — every time, no exceptions, no canary. CDP is faster (3-5s vs 10-15s), never hits CAPTCHA, and works with the existing logged-in Chrome profile. Use Playwright Profile only as fallback if CDP is unavailable.

Common Layered Strategy (validated pattern)

Layer 1: Jina Reader (r.jina.ai)  → covers 80%, fast, clean
  ↓ fail
Layer 2: Platform-specific (AWS RSS / WeChat we-mp-rss SQLite)
  ↓ fail
Layer 3: Playwright Profile with iPhone 14 Pro UA (persistent local Chrome)
          → also handles login-required sites AND heavy-bot Chinese sites (Zhihu zse-ck bypass)
  ↓ fail
Layer 4: TinyFish 兜底 (anti-detection cloud, requires TINYFISH_API_KEY)
  ↓ fail
Layer 4.5: Playwright Stealth (headless=True + init_script + desktop Chrome UA) — 2026-06-10
          → runs in cron/SSH, no profile warm-up, no API key. Slower per-fetch
            but ZERO setup. Use when Layers 1-4 unavailable/failed AND you need
            headless/cron execution. See `references/zhihu-and-cnsite-fetcher.md` Layer 8.
  ↓ fail
Flag failure + ask user for alternative

Key 2026-06-10 update: Playwright Profile (Layer 3 here, Layer 7 in zhihu-and-cnsite-fetcher.md) is now the preferred last-resort for Chinese heavy-bot sites because it has no per-fetch API cost and uses the user's existing local Chrome. The iPhone 14 Pro UA + persistent profile combination bypasses Zhihu's zse-ck challenge (mechanism: iOS Safari TLS fingerprint + returning-visitor cookie state).

2026-06-10 follow-up: Playwright Stealth (Layer 4.5 here, Layer 8 in zhihu-and-cnsite-fetcher.md) — headless=True + add_init_script + desktop Chrome UA — also bypasses Zhihu zse-ck (verified same URL). Use this when you need headless / cron / SSH execution and don't have Playwright Profile set up. Reference implementation: ~/wiki/scripts/zhihu-stealth/fetch-zhihu-stealth.py. Trade-off: ~10-15s per fetch (no warm-up) vs Profile's ~5-10s after warm-up; runs anywhere Python+playwright can run.

See research/web-scraper-comparison for the empirical comparison data.

When This Skill Should NOT Be Loaded

• For the decision of whether to ingest content (use wiki/web-content-reviewer)
• For the ingestion workflow once you have clean content (use wiki/manual-article-ingestion)
• For the scheduling of cron jobs that drive these fetchers (use cron-management)

See Also

• research/web-scraper-comparison — Jina vs Playwright vs Obscura vs TinyFish empirical comparison
• wiki/manual-article-ingestion — orchestrates these fetchers in a single-URL workflow
• wiki/web-content-reviewer — what to do with fetched content
• wiki/rss-to-wiki-pipeline — how RSS feeds are scanned and processed at scale
• wiki/wechat-mp-rss-extractor — WeChat-specific fetcher
• wiki/blogwatcher — blogwatcher-cli wrapper