| name | crawl-wechat |
| description | Crawl and extract WeChat public account (微信公众号) articles into structured data and clean markdown. Use this skill whenever the user wants to scrape, crawl, read, extract, or fetch content from a WeChat article URL (mp.weixin.qq.com). Also trigger when the user mentions 微信公众号文章、抓取微信文章、 爬取公众号、or provides a WeChat article link and wants its content extracted. This skill handles the tricky parts: spoofing the WeChat in-app browser user-agent, waiting for dynamic content to load, fixing lazy-loaded images, extracting structured metadata (title, author, publish time), generating clean markdown with inline images, and downloading images locally to bypass hotlink protection.
|
Crawl WeChat Articles
This skill extracts content from WeChat public account articles using the crawl4ai library. WeChat articles require special handling because they check the User-Agent header, render content dynamically, and use lazy-loading for images.
When to use
- User provides a
mp.weixin.qq.com/s/... URL and wants its content
- User asks to scrape/crawl/extract/read a WeChat (微信) article
- User wants to batch-process multiple WeChat article links
- User needs the article in markdown or structured format
Setup (run once before first use)
Before running the script, ensure dependencies are installed:
pip install crawl4ai aiohttp && crawl4ai-setup
If crawl4ai is already importable and the browser backend is ready, skip this step. When the script fails with ModuleNotFoundError or browser-related errors, run the commands above to fix it.
How it works
Run the bundled script to crawl a WeChat article:
python <skill-dir>/scripts/crawl_wechat.py <URL> [--download-images] [--save-html] [--save-markdown] [--output-dir DIR]
The script outputs a JSON summary to stdout and optionally saves the full HTML and/or markdown to files.
Key technical details
-
User-Agent spoofing: The script sets MicroMessenger/8.0.43 in the UA string so WeChat serves the full article instead of a "please open in WeChat" block.
-
Dynamic wait: Uses wait_for="css:#js_content" to ensure the article body has fully rendered before scraping.
-
Lazy-image fix: WeChat uses data-src for lazy-loaded images. The script injects JS to copy data-src → src so the markdown generator can pick up real image URLs.
-
Structured extraction: Uses JsonCssExtractionStrategy with a schema targeting WeChat's DOM structure (#activity-name for title, #js_name for author, #publish_time for date, #js_content for body).
-
Clean markdown with images: Uses DefaultMarkdownGenerator to produce readable markdown. SVG placeholder images and data-URI artifacts are cleaned out, preserving only real article images inline with the text.
-
Image hotlink protection: WeChat images on mmbiz.qpic.cn block requests with non-QQ referrers. Use --download-images to download all images locally with the correct Referer header, automatically replacing remote URLs with local paths in both HTML and markdown output.
Extracted fields
| Field | Description |
|---|
title | Article title |
author | Public account name |
publish_time | Publication timestamp |
account_desc | Account description/bio |
markdown | Clean markdown of article body |
html | Raw HTML of article body |
url | Final URL after any redirects |
Example usage
Single article with images downloaded locally:
python <skill-dir>/scripts/crawl_wechat.py "https://mp.weixin.qq.com/s/xxx" --download-images --save-markdown --output-dir ./output
For programmatic use in Python:
from crawl_wechat import crawl_wechat_article
import asyncio
article = asyncio.run(crawl_wechat_article(
"https://mp.weixin.qq.com/s/...",
images_dir="./output/images",
))
print(article["title"])
print(article["markdown"])
Limitations
- Requires a valid, non-expired WeChat article URL — cannot search or list articles from an account
- High-frequency crawling may trigger WeChat's anti-bot measures (CAPTCHAs, IP blocks)
- Some temporary share links expire after a period
