Menu
Build the URL link graph from a previously captured docs corpus, emitting Mermaid and JSON forms. Use after scrape-firecrawl when a topic needs sitemap visualisation or hub-and-leaf analysis of a docs site.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Canonical spec for how research entries, sources, tools, topics, and pipelines are structured in this repo. Use whenever adding to INDEX/, when writing under topics/, when authoring or modifying a pipeline skill, or when validating a contribution against the standard. Defines frontmatter schema, identifier rules, linking conventions, status and confidence lifecycles, and provenance requirements.
Re-classify rows in INDEX/sources.md and INDEX/tools.md that landed in the `unsorted` bucket — either with `category=unsorted` or `kind=unsorted`. Re-applies the categories.md vocabulary and the kind heuristic with a fresh attempt, using GitHub topics, description, README excerpts. Use on demand or on a slow weekly schedule to drain the unsorted backlog.
Diff your current GitHub stars against INDEX/tools.md, emit a PR adding newly-starred repos and flagging unstarred ones. Also re-checks staleness (last_pushed > 3 years or archived) and moves stale repos to _archived/tools.md. Use daily via the star-sweep GitHub Action.
Add a new article, video, talk, paper, book, podcast, or repo to the corresponding INDEX/ table with correctly-shaped frontmatter, derived labels, validated URL, and provenance. Use whenever the user wants to add reading material or a new tool to the awesome list. Wraps the row shapes defined in `.claude/templates/{source,repo}.md`.
Sweep INDEX/sources.md (and optionally INDEX/tools.md) for dead links, moving entries with confirmed-dead URLs to _archived/ with tombstones (last_seen, reason, optional wayback link). Use weekly via the link-check GitHub Action, or on demand when the user wants a manual sweep. Never hard-deletes.
Parse a captured docs corpus into structured concept, API, tool, and entity extracts with frontmatter, plus relation edges, and emit the knowledge graph in Mermaid and JSON forms. Use after scrape-firecrawl when a topic needs structured semantic extraction beyond the URL link graph.
| name | sitemap-graph |
| description | Build the URL link graph from a previously captured docs corpus, emitting Mermaid and JSON forms. Use after scrape-firecrawl when a topic needs sitemap visualisation or hub-and-leaf analysis of a docs site. |
| id | sitemap-graph |
| title | Build the sitemap graph |
| type | pipeline |
| status | stable |
| confidence | high |
| created | "2026-05-15T00:00:00.000Z" |
| updated | "2026-05-17T00:00:00.000Z" |
| related | ["scrape-firecrawl","extract-knowledge-graph"] |
Build the URL-level link graph from a captured docs corpus. The graph reveals how information flows and consolidates across pages — which pages are hubs, which are leaves, which are isolated, where there are redirect chains, where the navigation diverges from the link graph.
| name | type | notes |
|---|---|---|
topic_id | slug | the topic whose 20-sources/ to read |
sub_corpus | slug | optional; if multiple corpora live in 20-sources/, restrict to one |
Under topics/<topic_id>/40-graphs/:
sitemap.json — { "nodes": [...], "edges": [...] }.sitemap.mmd — Mermaid flowchart LR rendering for visual inspection.sitemap-analysis.md — derived metrics: hub pages (top-N in-degree), leaf pages, orphan pages, redirect chains, link-asymmetry.Collect nodes. For every _meta.md in 20-sources/<sub_corpus>/, emit a node:
{ "id": "src_…", "title": "<title>", "url": "<url>", "type": "source", "depth": <int> }
Collect edges. For every _meta.md, for every URL in links_out:
type: external and add an external flag.{ from: <src-id>, to: <target-id>, kind: "links-to" }.Detect redirect chains from _meta.md's canonical_url. Emit redirects-to edges separately.
Detect nav-of edges. If a page lives in the site's navigation tree (heuristic: links from the site's index pages, plus a sustained pattern of mutual linking from siblings), tag the edges nav-of instead of links-to. This is heuristic; mark the analysis confidence accordingly.
Emit sitemap.json in the schema declared in CONVENTIONS/diagrams.md.
Render sitemap.mmd. Mermaid flowchart LR with one node per source, one edge per links-to. For corpora over ~150 nodes, generate a faceted set instead: sitemap-by-section.mmd per top-level URL prefix, since a 200+ node Mermaid graph is unreadable in every renderer.
Compute sitemap-analysis.md. Metrics:
sitemap.json corresponds to an existing _meta.md (for type: source nodes) or is flagged external.from and to resolve to a node in the graph.mmdc or visually).| symptom | cause | fix |
|---|---|---|
many external nodes | URL canonicalisation missed redirects | re-run with stricter canonicalisation |
| Mermaid file too large to render | corpus too big | switch to faceted output (-by-section) |
| orphans dominate | crawl did not start from the right root | inspect _manifest.md, re-crawl with corrected base_url |
The sitemap graph is the navigation skeleton over which the knowledge graph hangs. Hub pages frequently coincide with concept-definition pages; leaf pages are often API reference. The downstream extract-knowledge-graph pipeline uses sitemap hub-rank as a hint for which pages to extract first.