| name | huawei-docs-scraper |
| description | Scrape HarmonyOS developer documentation from developer.huawei.com as structured Markdown. Use when the user wants to download, scrape, or archive Huawei HarmonyOS API references, guides, or any documentation from the Huawei developer portal. Also use when the user says 抓取鸿蒙资料, arkts代码生成所需资料, arkts代码审查所需资料, arkts代码静态检查规则, arkts单元测试所需资料, or similar phrases about collecting HarmonyOS/ArkTS documentation. Handles JavaScript-rendered sidebar via Playwright, intercepts internal JSON APIs, converts HTML to Markdown, automatically fixes Markdown syntax errors, and supports selective category scraping, C API exclusion, and resume. |
Huawei HarmonyOS Documentation Scraper
Prerequisites
- Node.js 18+ — required to run the scraper
- Python 3.8+ — required for the Markdown auto-fixer (
fix_markdown.py)
- Everything else (Playwright npm package, Chromium browser) is installed automatically on first run
The script checks dependencies at startup and installs any that are missing:
playwright npm package → npm install playwright (run once in the skill directory)- Bundled Chromium browser →
npx playwright install chromium (one-time ~170 MB download)
python3 → cannot be auto-installed; a warning is shown if absent
How It Works
The Huawei developer site (developer.huawei.com) renders its sidebar navigation
via JavaScript (Angular + NG-ZORRO virtual-scroll tree). Standard HTTP fetches
return empty content. This skill bypasses that by:
- Intercepting the
getCatalogTree API — Playwright loads the page and
captures the internal POST request that returns the full navigation tree as
JSON (~1.9 MB for API references).
- Calling
getDocumentById API — For each page in the tree, fetches the
HTML content directly via the internal API (10x faster than browser rendering).
- Converting HTML to Markdown — Strips tags, preserves tables/code/headings.
- Organizing by directory — Creates directories per category/Kit.
- Auto-fixing Markdown — Runs
fix_markdown.py on the output directory to
repair common HTML→Markdown conversion artifacts (see below).
Usage
Interactive mode (recommended)
Run without arguments to select a preset interactively:
node scripts/scrape.js
You will be prompted to choose:
请选择需要抓取的资料方向:
[1] 代码生成所需资料
· API 参考文档 (应用框架 / 系统 / 媒体 / 图形 / 应用服务 / AI)
· ArkTS 语言指南 (ArkTS语言介绍 + ArkTS编程规范)
· 应用权限列表 → app-permissions/permissions_list/
· 申请应用权限 → app-permissions/requesting_permission/
· 错误码信息 (ArkTS方舟编程语言) → errcode/
→ 输出至: arkts_code_lookup_references/
[2] 代码审查所需资料
· ArkTS 语言指南 (ArkTS语言介绍 + ArkTS编程规范)
· ArkTS 开发指南 (ArkTS方舟编程语言 + ArkUI方舟UI框架)
→ 输出至: arkts_code_review_references/
[3] 代码静态检查规则
· Code Linter 代码检查规则 (harmonyos-guides/ide-codelinter-rule)
→ 输出至: arkts_code_linter_references/
[4] 单元测试所需资料
· 自动化测试框架使用指导 (单元测试/UI测试/白盒性能测试)
· 开发自测试 (代码自测/测试用例编写)
→ 输出至: arkts_code_test_references/
Non-interactive preset mode
node scripts/scrape.js --preset 1
node scripts/scrape.js --preset 2
node scripts/scrape.js --preset 3
node scripts/scrape.js --preset 4
Legacy single-catalog mode
node scripts/scrape.js --direct --catalog harmonyos-references --entry ability-arkts \
--categories "应用框架,系统" --output ./my_docs
Options
| Flag | Default | Description |
|---|
--preset <1|2|3> | — | Use a predefined scraping configuration |
--direct | — | Enable legacy single-catalog mode |
--catalog <name> | harmonyos-references | Catalog name (used with --direct) |
--entry <slug> | ability-arkts | Entry page slug to trigger catalog tree load |
--categories <names> | all | Comma-separated top-level category names |
--output <dir> | preset default | Override output directory |
--exclude-capi / --no-exclude-capi | true | Include or exclude C API branches |
--delay <ms> | 200 | Delay between API requests |
--resume | — | Resume from .scrape_progress.json |
--browser <channel> | chromium | Browser: chromium (bundled), msedge, or chrome |
--no-fix | — | Skip running fix_markdown.py after scraping |
Examples
Resume an interrupted scrape:
node scripts/scrape.js --preset 1 --resume
Scrape to a custom output directory:
node scripts/scrape.js --preset 2 --output ./my_review_docs
Scrape without running the Markdown fixer:
node scripts/scrape.js --preset 1 --no-fix
Run the Markdown fixer standalone on an existing output directory:
python3 scripts/fix_markdown.py ./harmonyos_docs
python3 scripts/fix_markdown.py ./harmonyos_docs --dry-run
python3 scripts/fix_markdown.py ./harmonyos_docs --no-remove-deprecated
Preset Definitions
Preset 1 — 代码生成所需资料
Output: arkts_code_lookup_references/
| Task | Catalog | Categories |
|---|
| API 参考文档 | harmonyos-references | 应用框架, 系统, 媒体, 图形, 应用服务, AI (C API excluded) |
| ArkTS 语言指南 | harmonyos-guides | ArkTS语言介绍, ArkTS编程规范 |
| 应用权限列表 | harmonyos-guides | 系统 → 安全 → 应用权限列表 |
| 申请应用权限 | harmonyos-guides | 系统 → 安全 → 申请应用权限 |
| 错误码信息 | harmonyos-references | 应用框架 → ArkTS（方舟编程语言） → 错误码 |
Preset 2 — 代码审查所需资料
Output: arkts_code_review_references/
| Task | Catalog | Categories / Kits |
|---|
| ArkTS 语言指南 | harmonyos-guides | ArkTS语言介绍, ArkTS编程规范 |
| ArkTS 开发指南 | harmonyos-guides | 应用框架 → ArkTS（方舟编程语言）, ArkUI（方舟UI框架）only |
Preset 3 — 代码静态检查规则
Output: arkts_code_linter_references/
| Task | Catalog | Categories / Kits |
|---|
| Code Linter 代码检查规则 | harmonyos-guides | 编写与调试应用 → 代码检查 |
Preset 4 — 单元测试所需资料
Output: arkts_code_test_references/
| Task | Catalog | Categories / Kits |
|---|
| 自动化测试框架使用指导 | harmonyos-guides | 应用测试 |
| 开发自测试 | harmonyos-guides | 编写与调试应用 |
Output Structure
Preset 1 — arkts_code_lookup_references/
arkts_code_lookup_references/
├── harmonyos_references/
│ ├── 应用框架/
│ │ ├── Ability Kit（程序框架服务）/
│ │ │ ├── @ohos.app.ability.UIAbility (带界面的应用组件).md
│ │ │ └── ...
│ │ └── ...
│ ├── 系统/
│ ├── 媒体/
│ ├── 图形/
│ ├── 应用服务/
│ └── AI/
├── learning-arkts/
│ ├── <ArkTS语言介绍页面>.md
│ └── <ArkTS编程规范页面>.md
├── app-permissions/
│ ├── permissions_list/
│ │ └── *.md # 开放权限（系统授权）/ 受限开放权限 等
│ └── requesting_permission/
│ └── *.md # 申请权限流程相关页面
└── errcode/
└── ArkTS（方舟编程语言）/
└── 错误码/
└── *.md # ArkTS 错误码文档
Preset 2 — arkts_code_review_references/
arkts_code_review_references/
├── learning-arkts/
│ ├── <ArkTS语言介绍页面>.md
│ └── <ArkTS编程规范页面>.md
└── 应用框架/
├── ArkTS（方舟编程语言）/
│ └── *.md
└── ArkUI（方舟UI框架）/
└── *.md
Preset 3 — arkts_code_linter_references/
arkts_code_linter_references/
└── Code Linter代码检查规则/
└── 通用规则@typescript-eslint/
└── *.md # Code Linter 各规则页面（跳过 代码检查/ 层级）
Preset 4 — arkts_code_test_references/
arkts_code_test_references/
├── app-testing/ # 自动化测试框架使用指导
│ └── *.md # 单元测试/UI测试/白盒性能测试相关页面
└── ide-app-test/ # 开发自测试
└── *.md # 代码自测/测试用例编写相关页面
Each .md file contains:
# Title heading> 来源: <url> source link- Full document content in Markdown
Automatic Markdown Fixes (fix_markdown.py)
After scraping, fix_markdown.py is automatically invoked to correct artifacts
produced by the HTML→Markdown conversion:
| Fix | Description |
|---|
| Duplicate h1 title | Removes the second # Title that the page HTML adds after the scraper-prepended title |
Orphaned ** lines | Deletes lines containing only ** (stray bold markers from empty <strong> tags) |
| Asymmetric bold markers | 系统能力：** foo → **系统能力：** foo (missing opening **) |
[h2] tag artifacts | #### [h2]method → #### method |
| Bare code fences | ``` → ```typescript (adds language tag for syntax highlighting) |
| Broken list items | Merges lone - marker with content stranded on the next line |
| Version superscripts | funcName12+ → funcName (strips <sup>12+</sup> appended to identifiers) |
| Table spacing | Inserts blank line before tables not preceded by one (required by CommonMark) |
| Lone backticks in tables | Escapes isolated ` in table cells as \` to prevent unclosed code spans |
| Trailing whitespace | Strips all line-ending spaces |
| Excessive blank lines | Collapses 3+ consecutive blank lines to 2 |
| Deprecated sections | Removes (deprecated) headings and their body content (default: on, --no-remove-deprecated to disable) |
| Deprecated table rows | Removes table rows where the first column contains deprecated |
Handling Duplicate Filenames
When multiple pages share the same title within a Kit (common in ArkUI),
the script appends the page slug as a suffix to disambiguate:
PageTitle (slug-suffix).md
Adapting to Other Catalogs
The same API pattern works for other Huawei doc catalogs. Use --direct with
custom --catalog and --entry to target different documentation sets:
node scripts/scrape.js --direct --catalog harmonyos-guides --entry learning-arkts --output ./guides
