docai-web2md

name	docai-web2md
description	Convert any web URL to Markdown. Triggers on "转成Markdown/转换/网页转Markdown/convert to Markdown + URL". Handles static sites, dynamic SPAs, WeChat, arXiv, Twitter/X.

docai:web2md

When to Trigger

User wants to convert a web page to Markdown. Common patterns:

• "把这个链接转成 Markdown"、"网页转 Markdown"、"提取网页内容"
• "convert this URL to Markdown"、"get the content of this page"
• Any URL + intent to extract/read content (without summarization)

If user wants summary, use docai-web2summary instead.

How to Execute

python skills/docai-web2md/tools/convert.py <URL> [--use-python] [-o <file>]

入口会在导入第三方依赖前自动探测运行环境：优先使用当前解释器中已有的依赖， 其次复用项目内的 .venv/venv/env，然后使用项目 pyproject.toml 配合 uv run 创建或同步隔离环境，再尝试 Poetry、Pipenv、pyenv 环境；全部不可用时 会输出明确的依赖安装命令。通常不需要手动激活虚拟环境或把依赖安装到系统 Python。

Parameters

Parameter	Required	Description
url	Yes	Web page URL
--use-python	No	Force Python method (skip Jina/Firecrawl)
-o / --output	No	Save to file instead of stdout

Examples

# Basic conversion (parallel: Jina / Firecrawl / Python)
python skills/docai-web2md/tools/convert.py https://example.com/article

# arXiv paper (auto HTML priority, PDF fallback)
python skills/docai-web2md/tools/convert.py https://arxiv.org/abs/2601.04500v1

# Save to file
python skills/docai-web2md/tools/convert.py https://mp.weixin.qq.com/s/... -o article.md

# Force Python method
python skills/docai-web2md/tools/convert.py https://example.com --use-python

What It Does

Four methods run in parallel, returning the first successful result:

1. Jina Reader API (fastest, zero install)
2. Firecrawl API (if key configured)
3. Python fallback (requests + BeautifulSoup)
4. Playwright (headless browser for JS-rendered pages)

Special cases handled automatically: WeChat → Playwright first (mobile UA, JS rendering), arXiv → HTML priority, X Article → structured content with inline images, ordinary Twitter/X → proxy/Playwright fallback.

Troubleshooting

• arXiv PDF garbled: Requires pymupdf — pip install pymupdf
• Dynamic page empty: Script auto-detects SPAs and uses Playwright
• X Article images missing: the converter restores images from the Article structured blocks before using the proxy fallback
• 依赖缺失: 直接用上面的 python 命令运行即可，入口会自动尝试项目虚拟环境和 uv；若环境管理器不可用，错误信息会给出 python3 -m pip install ... 回退命令
• All methods fail: Try --use-python to bypass API methods