content-fetching

name	content-fetching
description	Fetch and clean web content for the wiki — RSS discovery, HTML-to-markdown extraction, login-session browser (Playwright Profile + iPhone UA), Playwright Stealth headless (init_script patches), and cloud anti-detection browser (TinyFish). Five techniques unified under one entry point; load the umbrella first, then read the relevant reference for the specific scenario.
tags	["content-fetching","rss","web","scraper","ocr","extraction","accessibility","login","captcha","js-rendering"]
category	wiki
related_skills	["wiki/manual-article-ingestion","wiki/web-content-reviewer","wiki/inbox-screener","research/web-scraper-comparison"]

Content Fetching & Cleaning

The wiki pipeline needs to get clean article content from many different source types (newsletter links, WeChat subscriptions, paywalled articles, JS-rendered SPAs, captcha-protected pages). Each requires a different technique.

Five Techniques

Technique	When to use	Reference
RSS feed discovery	Find RSS/Atom feeds for a new source you want to subscribe to	references/rss-feed-discovery.md
HTML-to-markdown extraction (Readability)	Already have HTML; need to clean to semantic markdown	references/readability-extractor.md
Login-session browser (Playwright Profile + iPhone UA)	Sites requiring cookies, OAuth, or persistent browser profile; also doubles as anti-bot bypass for heavy Chinese sites (Zhihu, XHS, CSDN); MANDATORY for user-pasted mp.weixin.qq.com URLs — see references/wechat-user-pasted-fast-path.md	references/playwright-profile-fetcher.md + references/playwright-mobile-ua-bypass.md + references/wechat-user-pasted-fast-path.md
Playwright Stealth headless (init_script patches)	Heavy-bot sites that need headless/cron/SSH execution with no profile/API key (Zhihu zse-ck, CSDN, Xiaohongshu)	references/playwright-stealth-headless.md
Cloud anti-detection browser (TinyFish)	Cloudflare/CAPTCHA sites that block even stealth Playwright; cloud-based, requires API key	references/tinyfish-web-agent.md
Zhihu-specific (zhuanlan.zhihu.com)	Thin wrapper linking to the general anti-bot techniques above	references/zhihu-and-cnsite-fetcher.md

Quick Decision Tree

Need web content?
│
├── Has RSS/Atom feed? ─────────► rss-feed-discovery.md
│
├── mp.weixin.qq.com (WeChat article)?
│   └── Playwright Profile FIRST — skip curl/browser/Jina entirely
│       • curl returns 17KB verify.html (CAPTCHA wall, metadata empty)
│       • browser_navigate → same CAPTCHA wall
│       • Jina Reader → times out or returns HTML shell
│       • Playwright Profile: local Chrome + real WeChat login cookies
│         reliably bypasses. Command:
│         cd ~/wiki/scripts/playwright-profile
│         node fetch-with-profile.js "URL" --text > /tmp/wx.txt
│       • Session-level: after first CAPTCHA hit, skip curl for ALL
│         subsequent WeChat URLs in same session
│       • Fallback: TinyFish (if Playwright Profile not set up)
│       See: references/playwright-profile-fetcher.md
│
├── Source URL, public site, no login?
│   ├── Simple blog/news ──────► Jina Reader (default)
│   ├── JS-rendered / CAPTCHA ─► tinyfish-web-agent.md
│   ├── Heavy-bot (headless/cron) ► playwright-stealth-headless.md
│   ├── Chinese heavy-bot (Zhihu/CSDN/XHS) ► zhihu-and-cnsite-fetcher.md
│   └── Need login / cookies ──► playwright-profile-fetcher.md
│
└── Already have raw HTML? ────► readability-extractor.md

Anti-patterns:

• Declaring "URL unreachable" after only trying curl + browser_navigate + Jina. The umbrella has at least 5 documented layers. Exhaust them before reporting failure.
• WeChat curl-as-canary: trying curl first on mp.weixin.qq.com URLs to "save time." curl succeeds ~50% but the other 50% returns a 17KB verify.html CAPTCHA wall with empty metadata. The retry loop (curl→browser→Jina→finally Playwright) wastes 30-60s per URL. Go straight to Playwright Profile — every time, no exceptions, no canary. User confirmed this preference directly (2026-06-17: "使用 playwright 方案" after the assistant did curl-then-fallback). The skill already designates Playwright Profile as the preferred path; treat it as mandatory, not just preferred.

Common Layered Strategy (validated pattern)

Layer 1: Jina Reader (r.jina.ai)  → covers 80%, fast, clean
  ↓ fail
Layer 2: Platform-specific (AWS RSS / WeChat we-mp-rss SQLite)
  ↓ fail
Layer 3: Playwright Profile with iPhone 14 Pro UA (persistent local Chrome)
          → also handles login-required sites AND heavy-bot Chinese sites (Zhihu zse-ck bypass)
  ↓ fail
Layer 4: TinyFish 兜底 (anti-detection cloud, requires TINYFISH_API_KEY)
  ↓ fail
Layer 4.5: Playwright Stealth (headless=True + init_script + desktop Chrome UA) — 2026-06-10
          → runs in cron/SSH, no profile warm-up, no API key. Slower per-fetch
            but ZERO setup. Use when Layers 1-4 unavailable/failed AND you need
            headless/cron execution. See `references/zhihu-and-cnsite-fetcher.md` Layer 8.
  ↓ fail
Flag failure + ask user for alternative

Key 2026-06-10 update: Playwright Profile (Layer 3 here, Layer 7 in zhihu-and-cnsite-fetcher.md) is now the preferred last-resort for Chinese heavy-bot sites because it has no per-fetch API cost and uses the user's existing local Chrome. The iPhone 14 Pro UA + persistent profile combination bypasses Zhihu's zse-ck challenge (mechanism: iOS Safari TLS fingerprint + returning-visitor cookie state).

2026-06-10 follow-up: Playwright Stealth (Layer 4.5 here, Layer 8 in zhihu-and-cnsite-fetcher.md) — headless=True + add_init_script + desktop Chrome UA — also bypasses Zhihu zse-ck (verified same URL). Use this when you need headless / cron / SSH execution and don't have Playwright Profile set up. Reference implementation: ~/wiki/scripts/zhihu-stealth/fetch-zhihu-stealth.py. Trade-off: ~10-15s per fetch (no warm-up) vs Profile's ~5-10s after warm-up; runs anywhere Python+playwright can run.

See research/web-scraper-comparison for the empirical comparison data.

When This Skill Should NOT Be Loaded

• For the decision of whether to ingest content (use wiki/web-content-reviewer)
• For the ingestion workflow once you have clean content (use wiki/manual-article-ingestion)
• For the scheduling of cron jobs that drive these fetchers (use cron-management)

See Also

• research/web-scraper-comparison — Jina vs Playwright vs Obscura vs TinyFish empirical comparison
• wiki/manual-article-ingestion — orchestrates these fetchers in a single-URL workflow
• wiki/web-content-reviewer — what to do with fetched content
• wiki/rss-to-wiki-pipeline — how RSS feeds are scanned and processed at scale
• wiki/wechat-mp-rss-extractor — WeChat-specific fetcher
• wiki/blogwatcher — blogwatcher-cli wrapper