菜单
Build the URL link graph from a previously captured docs corpus, emitting Mermaid and JSON forms. Use after scrape-firecrawl when a topic needs sitemap visualisation or hub-and-leaf analysis of a docs site.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
Canonical spec for how research entries, sources, tools, topics, and pipelines are structured in this repo. Use whenever adding to INDEX/, when writing under topics/, when authoring or modifying a pipeline skill, or when validating a contribution against the standard. Defines frontmatter schema, identifier rules, linking conventions, status and confidence lifecycles, and provenance requirements.
Re-classify rows in INDEX/sources.md and INDEX/tools.md that landed in the `unsorted` bucket — either with `category=unsorted` or `kind=unsorted`. Re-applies the categories.md vocabulary and the kind heuristic with a fresh attempt, using GitHub topics, description, README excerpts. Use on demand or on a slow weekly schedule to drain the unsorted backlog.
Diff your current GitHub stars against INDEX/tools.md, emit a PR adding newly-starred repos and flagging unstarred ones. Also re-checks staleness (last_pushed > 3 years or archived) and moves stale repos to _archived/tools.md. Use daily via the star-sweep GitHub Action.
Add a new article, video, talk, paper, book, podcast, or repo to the corresponding INDEX/ table with correctly-shaped frontmatter, derived labels, validated URL, and provenance. Use whenever the user wants to add reading material or a new tool to the awesome list. Wraps the row shapes defined in `.claude/templates/{source,repo}.md`.
Sweep INDEX/sources.md (and optionally INDEX/tools.md) for dead links, moving entries with confirmed-dead URLs to _archived/ with tombstones (last_seen, reason, optional wayback link). Use weekly via the link-check GitHub Action, or on demand when the user wants a manual sweep. Never hard-deletes.
Parse a captured docs corpus into structured concept, API, tool, and entity extracts with frontmatter, plus relation edges, and emit the knowledge graph in Mermaid and JSON forms. Use after scrape-firecrawl when a topic needs structured semantic extraction beyond the URL link graph.
基于 SOC 职业分类
| name | sitemap-graph |
| description | Build the URL link graph from a previously captured docs corpus, emitting Mermaid and JSON forms. Use after scrape-firecrawl when a topic needs sitemap visualisation or hub-and-leaf analysis of a docs site. |
| id | sitemap-graph |
| title | Build the sitemap graph |
| type | pipeline |
| status | stable |
| confidence | high |
| created | "2026-05-15T00:00:00.000Z" |
| updated | "2026-05-17T00:00:00.000Z" |
| related | ["scrape-firecrawl","extract-knowledge-graph"] |
Build the URL-level link graph from a captured docs corpus. The graph reveals how information flows and consolidates across pages — which pages are hubs, which are leaves, which are isolated, where there are redirect chains, where the navigation diverges from the link graph.
| name | type | notes |
|---|---|---|
topic_id | slug | the topic whose 20-sources/ to read |
sub_corpus | slug | optional; if multiple corpora live in 20-sources/, restrict to one |
Under topics/<topic_id>/40-graphs/:
sitemap.json — { "nodes": [...], "edges": [...] }.sitemap.mmd — Mermaid flowchart LR rendering for visual inspection.sitemap-analysis.md — derived metrics: hub pages (top-N in-degree), leaf pages, orphan pages, redirect chains, link-asymmetry.Collect nodes. For every _meta.md in 20-sources/<sub_corpus>/, emit a node:
{ "id": "src_…", "title": "<title>", "url": "<url>", "type": "source", "depth": <int> }
Collect edges. For every _meta.md, for every URL in links_out:
type: external and add an external flag.{ from: <src-id>, to: <target-id>, kind: "links-to" }.Detect redirect chains from _meta.md's canonical_url. Emit redirects-to edges separately.
Detect nav-of edges. If a page lives in the site's navigation tree (heuristic: links from the site's index pages, plus a sustained pattern of mutual linking from siblings), tag the edges nav-of instead of links-to. This is heuristic; mark the analysis confidence accordingly.
Emit sitemap.json in the schema declared in CONVENTIONS/diagrams.md.
Render sitemap.mmd. Mermaid flowchart LR with one node per source, one edge per links-to. For corpora over ~150 nodes, generate a faceted set instead: sitemap-by-section.mmd per top-level URL prefix, since a 200+ node Mermaid graph is unreadable in every renderer.
Compute sitemap-analysis.md. Metrics:
sitemap.json corresponds to an existing _meta.md (for type: source nodes) or is flagged external.from and to resolve to a node in the graph.mmdc or visually).| symptom | cause | fix |
|---|---|---|
many external nodes | URL canonicalisation missed redirects | re-run with stricter canonicalisation |
| Mermaid file too large to render | corpus too big | switch to faceted output (-by-section) |
| orphans dominate | crawl did not start from the right root | inspect _manifest.md, re-crawl with corrected base_url |
The sitemap graph is the navigation skeleton over which the knowledge graph hangs. Hub pages frequently coincide with concept-definition pages; leaf pages are often API reference. The downstream extract-knowledge-graph pipeline uses sitemap hub-rank as a hint for which pages to extract first.