Menu
Capture every page of a target docs site via Firecrawl into a topic's 20-sources/ directory as cleaned markdown plus link metadata. Use when starting a new topic that needs a docs corpus as its source-of-truth.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Canonical spec for how research entries, sources, tools, topics, and pipelines are structured in this repo. Use whenever adding to INDEX/, when writing under topics/, when authoring or modifying a pipeline skill, or when validating a contribution against the standard. Defines frontmatter schema, identifier rules, linking conventions, status and confidence lifecycles, and provenance requirements.
Re-classify rows in INDEX/sources.md and INDEX/tools.md that landed in the `unsorted` bucket — either with `category=unsorted` or `kind=unsorted`. Re-applies the categories.md vocabulary and the kind heuristic with a fresh attempt, using GitHub topics, description, README excerpts. Use on demand or on a slow weekly schedule to drain the unsorted backlog.
Diff your current GitHub stars against INDEX/tools.md, emit a PR adding newly-starred repos and flagging unstarred ones. Also re-checks staleness (last_pushed > 3 years or archived) and moves stale repos to _archived/tools.md. Use daily via the star-sweep GitHub Action.
Add a new article, video, talk, paper, book, podcast, or repo to the corresponding INDEX/ table with correctly-shaped frontmatter, derived labels, validated URL, and provenance. Use whenever the user wants to add reading material or a new tool to the awesome list. Wraps the row shapes defined in `.claude/templates/{source,repo}.md`.
Sweep INDEX/sources.md (and optionally INDEX/tools.md) for dead links, moving entries with confirmed-dead URLs to _archived/ with tombstones (last_seen, reason, optional wayback link). Use weekly via the link-check GitHub Action, or on demand when the user wants a manual sweep. Never hard-deletes.
Parse a captured docs corpus into structured concept, API, tool, and entity extracts with frontmatter, plus relation edges, and emit the knowledge graph in Mermaid and JSON forms. Use after scrape-firecrawl when a topic needs structured semantic extraction beyond the URL link graph.
| name | scrape-firecrawl |
| description | Capture every page of a target docs site via Firecrawl into a topic's 20-sources/ directory as cleaned markdown plus link metadata. Use when starting a new topic that needs a docs corpus as its source-of-truth. |
| id | scrape-firecrawl |
| title | Scrape via Firecrawl |
| type | pipeline |
| status | stable |
| confidence | high |
| created | "2026-05-15T00:00:00.000Z" |
| updated | "2026-05-17T00:00:00.000Z" |
| related | ["sitemap-graph","extract-knowledge-graph"] |
Capture every page of a target site via Firecrawl, persisting each page as cleaned markdown plus link metadata. The captured corpus is the source-of-truth for downstream pipelines — sitemap-graph and extract-knowledge-graph both read from it and never re-fetch.
| name | type | notes |
|---|---|---|
base_url | URL | the root of the docs site to crawl |
topic_id | slug | the topic this corpus belongs to |
include_paths | list[regex] | optional; only URLs matching at least one regex are crawled |
exclude_paths | list[regex] | optional; URLs matching are skipped |
max_depth | int | crawl depth from base_url |
max_pages | int | safety cap |
firecrawl_endpoint | enum | /scrape (one URL) | /crawl (recursive) | /map (URL discovery only) |
formats | list | typically ['markdown']; add 'links' to get explicit link metadata |
Under topics/<topic_id>/20-sources/:
_manifest.md — appended row per captured page (linking table).<src-id>/_meta.md — provenance file (frontmatter only).<src-id>/content.md — cleaned markdown.firecrawl-map first against base_url with the include/exclude regexes. Record the URL list. Estimate cost: n_pages × per-page-rate. Compare against the rate-limit budget; if it exceeds 15% of monthly budget, surface the cost-rethinker decision.formats: ['markdown', 'links']. Set onlyMainContent: true to drop nav and footer noise.src_<ULID> for the page.content_hash = sha256(markdown)._meta.md with the universal frontmatter plus the source-specific extensions (url, retrieved_at, retriever: firecrawl-crawl, content_hash, mime: text/markdown, crawl_depth, links_out, canonical_url if redirected). The body of _meta.md is optional; the frontmatter carries the metadata.content.md with the markdown returned by Firecrawl, no transformation.20-sources/_manifest.md with one row per page (id, title, source_type, status, url, retrieved_at, crawl_depth, content_hash).INDEX/sources.md (or regenerate).status: seed and a failure_reason frontmatter field for later retry._manifest.md equals the number of _meta.md files in 20-sources/._meta.md has a corresponding content.md (or is annotated with failure_reason).content_hash in a _meta.md matches the SHA-256 of its content.md.links_out arrays in _meta.md are populated for at least 80% of pages — if fewer, the links format was likely not requested.| symptom | likely cause | fix |
|---|---|---|
| pages return empty markdown | site is heavily JS-rendered, /scrape insufficient | switch to /scrape with actions or use the firecrawl-interact build-side skill |
| crawl exhausts budget early | depth or include too permissive | tighten include_paths, lower max_depth |
pages return 403 / 429 | rate-limit or anti-bot | reduce concurrency, set politeness_delay, ask user before retrying |
| same content captured under multiple URLs | URL canonicalisation not applied | follow canonical_url, deduplicate by content_hash |
topics/docs-knowledge-graph-pipeline/ uses this pipeline twice:
base_url: https://docs.firecrawl.dev (dogfood — capture the scraper's own docs)base_url: https://moonrepo.dev/docs (the user's stated pain point)Each becomes its own sub-corpus under 20-sources/firecrawl-docs/ and 20-sources/moon-docs/ respectively, with separate manifests.
Firecrawl exposes /scrape, /crawl, /map, /extract, and /interact endpoints. For docs-site capture, /crawl plus formats: ['markdown', 'links'] is the workhorse. /extract (LLM-driven schema extraction) is not used in this pipeline — extraction is a separate, deterministic step in extract-knowledge-graph so we can re-derive the KG without re-spending LLM tokens.