huawei-docs-scraper

name

huawei-docs-scraper

description

Scrape HarmonyOS developer documentation from developer.huawei.com as structured Markdown. Use when the user wants to download, scrape, or archive Huawei HarmonyOS API references, guides, or any documentation from the Huawei developer portal. Also use when the user says 抓取鸿蒙资料, arkts代码生成所需资料, arkts代码审查所需资料, arkts代码静态检查规则, arkts单元测试所需资料, or similar phrases about collecting HarmonyOS/ArkTS documentation. Handles JavaScript-rendered sidebar via Playwright, intercepts internal JSON APIs, converts HTML to Markdown, automatically fixes Markdown syntax errors, and supports selective category scraping, C API exclusion, and resume.

Huawei HarmonyOS Documentation Scraper

Prerequisites

• Node.js 18+ — required to run the scraper
• Python 3.8+ — required for the Markdown auto-fixer (fix_markdown.py)
• Everything else (Playwright npm package, Chromium browser) is installed automatically on first run

The script checks dependencies at startup and installs any that are missing:

• playwright npm package → npm install playwright (run once in the skill directory)
• Bundled Chromium browser → npx playwright install chromium (one-time ~170 MB download)
• python3 → cannot be auto-installed; a warning is shown if absent

How It Works

The Huawei developer site (developer.huawei.com) renders its sidebar navigation via JavaScript (Angular + NG-ZORRO virtual-scroll tree). Standard HTTP fetches return empty content. This skill bypasses that by:

1. Intercepting the getCatalogTree API — Playwright loads the page and captures the internal POST request that returns the full navigation tree as JSON (~1.9 MB for API references).
2. Calling getDocumentById API — For each page in the tree, fetches the HTML content directly via the internal API (10x faster than browser rendering).
3. Converting HTML to Markdown — Strips tags, preserves tables/code/headings.
4. Organizing by directory — Creates directories per category/Kit.
5. Auto-fixing Markdown — Runs fix_markdown.py on the output directory to repair common HTML→Markdown conversion artifacts (see below).

Usage

Interactive mode (recommended)

Run without arguments to select a preset interactively:

node scripts/scrape.js

You will be prompted to choose:

请选择需要抓取的资料方向:

  [1] 代码生成所需资料
      · API 参考文档 (应用框架 / 系统 / 媒体 / 图形 / 应用服务 / AI)
      · ArkTS 语言指南 (ArkTS语言介绍 + ArkTS编程规范)
      · 应用权限列表 → app-permissions/permissions_list/
      · 申请应用权限 → app-permissions/requesting_permission/
      · 错误码信息 (ArkTS方舟编程语言) → errcode/
      → 输出至: arkts_code_lookup_references/

  [2] 代码审查所需资料
      · ArkTS 语言指南 (ArkTS语言介绍 + ArkTS编程规范)
      · ArkTS 开发指南 (ArkTS方舟编程语言 + ArkUI方舟UI框架)
      → 输出至: arkts_code_review_references/

  [3] 代码静态检查规则
      · Code Linter 代码检查规则 (harmonyos-guides/ide-codelinter-rule)
      → 输出至: arkts_code_linter_references/

  [4] 单元测试所需资料
      · 自动化测试框架使用指导 (单元测试/UI测试/白盒性能测试)
      · 开发自测试 (代码自测/测试用例编写)
      → 输出至: arkts_code_test_references/

Non-interactive preset mode

node scripts/scrape.js --preset 1   # 代码生成所需资料
node scripts/scrape.js --preset 2   # 代码审查所需资料
node scripts/scrape.js --preset 3   # 代码静态检查规则
node scripts/scrape.js --preset 4   # 单元测试所需资料

Legacy single-catalog mode

node scripts/scrape.js --direct --catalog harmonyos-references --entry ability-arkts \
  --categories "应用框架,系统" --output ./my_docs

Options

Flag	Default	Description
--preset <1|2|3>	—	Use a predefined scraping configuration
--direct	—	Enable legacy single-catalog mode
--catalog <name>	harmonyos-references	Catalog name (used with --direct)
--entry <slug>	ability-arkts	Entry page slug to trigger catalog tree load
--categories <names>	all	Comma-separated top-level category names
--output <dir>	preset default	Override output directory
--exclude-capi / --no-exclude-capi	true	Include or exclude C API branches
--delay <ms>	200	Delay between API requests
--resume	—	Resume from .scrape_progress.json
--browser <channel>	chromium	Browser: chromium (bundled), msedge, or chrome
--no-fix	—	Skip running fix_markdown.py after scraping

Examples

Resume an interrupted scrape:

node scripts/scrape.js --preset 1 --resume

Scrape to a custom output directory:

node scripts/scrape.js --preset 2 --output ./my_review_docs

Scrape without running the Markdown fixer:

node scripts/scrape.js --preset 1 --no-fix

Run the Markdown fixer standalone on an existing output directory:

python3 scripts/fix_markdown.py ./harmonyos_docs                          # deprecated 内容默认移除
python3 scripts/fix_markdown.py ./harmonyos_docs --dry-run                # preview only
python3 scripts/fix_markdown.py ./harmonyos_docs --no-remove-deprecated   # 保留 deprecated 内容

Preset Definitions

Preset 1 — 代码生成所需资料

Output: arkts_code_lookup_references/

Task	Catalog	Categories
API 参考文档	harmonyos-references	应用框架, 系统, 媒体, 图形, 应用服务, AI (C API excluded)
ArkTS 语言指南	harmonyos-guides	ArkTS语言介绍, ArkTS编程规范
应用权限列表	harmonyos-guides	系统 → 安全 → 应用权限列表
申请应用权限	harmonyos-guides	系统 → 安全 → 申请应用权限
错误码信息	harmonyos-references	应用框架 → ArkTS（方舟编程语言） → 错误码

Preset 2 — 代码审查所需资料

Output: arkts_code_review_references/

Task	Catalog	Categories / Kits
ArkTS 语言指南	harmonyos-guides	ArkTS语言介绍, ArkTS编程规范
ArkTS 开发指南	harmonyos-guides	应用框架 → ArkTS（方舟编程语言）, ArkUI（方舟UI框架）only

Preset 3 — 代码静态检查规则

Output: arkts_code_linter_references/

Task	Catalog	Categories / Kits
Code Linter 代码检查规则	harmonyos-guides	编写与调试应用 → 代码检查

Preset 4 — 单元测试所需资料

Output: arkts_code_test_references/

Task	Catalog	Categories / Kits
自动化测试框架使用指导	harmonyos-guides	应用测试
开发自测试	harmonyos-guides	编写与调试应用

Output Structure

Preset 1 — arkts_code_lookup_references/

arkts_code_lookup_references/
├── harmonyos_references/
│   ├── 应用框架/
│   │   ├── Ability Kit（程序框架服务）/
│   │   │   ├── @ohos.app.ability.UIAbility (带界面的应用组件).md
│   │   │   └── ...
│   │   └── ...
│   ├── 系统/
│   ├── 媒体/
│   ├── 图形/
│   ├── 应用服务/
│   └── AI/
├── learning-arkts/
│   ├── <ArkTS语言介绍页面>.md
│   └── <ArkTS编程规范页面>.md
├── app-permissions/
│   ├── permissions_list/
│   │   └── *.md    # 开放权限（系统授权）/ 受限开放权限 等
│   └── requesting_permission/
│       └── *.md    # 申请权限流程相关页面
└── errcode/
    └── ArkTS（方舟编程语言）/
        └── 错误码/
            └── *.md    # ArkTS 错误码文档

Preset 2 — arkts_code_review_references/

arkts_code_review_references/
├── learning-arkts/
│   ├── <ArkTS语言介绍页面>.md
│   └── <ArkTS编程规范页面>.md
└── 应用框架/
    ├── ArkTS（方舟编程语言）/
    │   └── *.md
    └── ArkUI（方舟UI框架）/
        └── *.md

Preset 3 — arkts_code_linter_references/

arkts_code_linter_references/
└── Code Linter代码检查规则/
    └── 通用规则@typescript-eslint/
        └── *.md    # Code Linter 各规则页面（跳过 代码检查/ 层级）

Preset 4 — arkts_code_test_references/

arkts_code_test_references/
├── app-testing/                # 自动化测试框架使用指导
│   └── *.md    # 单元测试/UI测试/白盒性能测试相关页面
└── ide-app-test/               # 开发自测试
    └── *.md    # 代码自测/测试用例编写相关页面

Each .md file contains:

• # Title heading
• > 来源: <url> source link
• Full document content in Markdown

Automatic Markdown Fixes (fix_markdown.py)

After scraping, fix_markdown.py is automatically invoked to correct artifacts produced by the HTML→Markdown conversion:

Fix	Description
Duplicate h1 title	Removes the second # Title that the page HTML adds after the scraper-prepended title
Orphaned ** lines	Deletes lines containing only ** (stray bold markers from empty <strong> tags)
Asymmetric bold markers	系统能力：** foo → **系统能力：** foo (missing opening **)
[h2] tag artifacts	#### [h2]method → #### method
Bare code fences	``` → ```typescript (adds language tag for syntax highlighting)
Broken list items	Merges lone - marker with content stranded on the next line
Version superscripts	funcName12+ → funcName (strips <sup>12+</sup> appended to identifiers)
Table spacing	Inserts blank line before tables not preceded by one (required by CommonMark)
Lone backticks in tables	Escapes isolated ` in table cells as \` to prevent unclosed code spans
Trailing whitespace	Strips all line-ending spaces
Excessive blank lines	Collapses 3+ consecutive blank lines to 2
Deprecated sections	Removes (deprecated) headings and their body content (default: on, --no-remove-deprecated to disable)
Deprecated table rows	Removes table rows where the first column contains deprecated

Handling Duplicate Filenames

When multiple pages share the same title within a Kit (common in ArkUI), the script appends the page slug as a suffix to disambiguate: PageTitle (slug-suffix).md

Adapting to Other Catalogs

The same API pattern works for other Huawei doc catalogs. Use --direct with custom --catalog and --entry to target different documentation sets:

# HarmonyOS Guides (any catalog)
node scripts/scrape.js --direct --catalog harmonyos-guides --entry learning-arkts --output ./guides