Menu
Capture every page of a target docs site via Firecrawl into a topic's 20-sources/ directory as cleaned markdown plus link metadata. Use when starting a new topic that needs a docs corpus as its source-of-truth.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Baseado na classificação ocupacional SOC
Canonical spec for how research entries, sources, tools, topics, and pipelines are structured in this repo. Use whenever adding to INDEX/, when writing under topics/, when authoring or modifying a pipeline skill, or when validating a contribution against the standard. Defines frontmatter schema, identifier rules, linking conventions, status and confidence lifecycles, and provenance requirements.
Re-classify rows in INDEX/sources.md and INDEX/tools.md that landed in the `unsorted` bucket — either with `category=unsorted` or `kind=unsorted`. Re-applies the categories.md vocabulary and the kind heuristic with a fresh attempt, using GitHub topics, description, README excerpts. Use on demand or on a slow weekly schedule to drain the unsorted backlog.
Diff your current GitHub stars against INDEX/tools.md, emit a PR adding newly-starred repos and flagging unstarred ones. Also re-checks staleness (last_pushed > 3 years or archived) and moves stale repos to _archived/tools.md. Use daily via the star-sweep GitHub Action.
Add a new article, video, talk, paper, book, podcast, or repo to the corresponding INDEX/ table with correctly-shaped frontmatter, derived labels, validated URL, and provenance. Use whenever the user wants to add reading material or a new tool to the awesome list. Wraps the row shapes defined in `.claude/templates/{source,repo}.md`.
Sweep INDEX/sources.md (and optionally INDEX/tools.md) for dead links, moving entries with confirmed-dead URLs to _archived/ with tombstones (last_seen, reason, optional wayback link). Use weekly via the link-check GitHub Action, or on demand when the user wants a manual sweep. Never hard-deletes.
Parse a captured docs corpus into structured concept, API, tool, and entity extracts with frontmatter, plus relation edges, and emit the knowledge graph in Mermaid and JSON forms. Use after scrape-firecrawl when a topic needs structured semantic extraction beyond the URL link graph.
| name | scrape-firecrawl |
| description | Capture every page of a target docs site via Firecrawl into a topic's 20-sources/ directory as cleaned markdown plus link metadata. Use when starting a new topic that needs a docs corpus as its source-of-truth. |
| id | scrape-firecrawl |
| title | Scrape via Firecrawl |
| type | pipeline |
| status | stable |
| confidence | high |
| created | "2026-05-15T00:00:00.000Z" |
| updated | "2026-05-17T00:00:00.000Z" |
| related | ["sitemap-graph","extract-knowledge-graph"] |
Capture every page of a target site via Firecrawl, persisting each page as cleaned markdown plus link metadata. The captured corpus is the source-of-truth for downstream pipelines — sitemap-graph and extract-knowledge-graph both read from it and never re-fetch.
| name | type | notes |
|---|---|---|
base_url | URL | the root of the docs site to crawl |
topic_id | slug | the topic this corpus belongs to |
include_paths | list[regex] | optional; only URLs matching at least one regex are crawled |
exclude_paths | list[regex] | optional; URLs matching are skipped |
max_depth | int | crawl depth from base_url |
max_pages | int | safety cap |
firecrawl_endpoint | enum | /scrape (one URL) | /crawl (recursive) | /map (URL discovery only) |
formats | list | typically ['markdown']; add 'links' to get explicit link metadata |
Under topics/<topic_id>/20-sources/:
_manifest.md — appended row per captured page (linking table).<src-id>/_meta.md — provenance file (frontmatter only).<src-id>/content.md — cleaned markdown.firecrawl-map first against base_url with the include/exclude regexes. Record the URL list. Estimate cost: n_pages × per-page-rate. Compare against the rate-limit budget; if it exceeds 15% of monthly budget, surface the cost-rethinker decision.formats: ['markdown', 'links']. Set onlyMainContent: true to drop nav and footer noise.src_<ULID> for the page.content_hash = sha256(markdown)._meta.md with the universal frontmatter plus the source-specific extensions (url, retrieved_at, retriever: firecrawl-crawl, content_hash, mime: text/markdown, crawl_depth, links_out, canonical_url if redirected). The body of _meta.md is optional; the frontmatter carries the metadata.content.md with the markdown returned by Firecrawl, no transformation.20-sources/_manifest.md with one row per page (id, title, source_type, status, url, retrieved_at, crawl_depth, content_hash).INDEX/sources.md (or regenerate).status: seed and a failure_reason frontmatter field for later retry._manifest.md equals the number of _meta.md files in 20-sources/._meta.md has a corresponding content.md (or is annotated with failure_reason).content_hash in a _meta.md matches the SHA-256 of its content.md.links_out arrays in _meta.md are populated for at least 80% of pages — if fewer, the links format was likely not requested.| symptom | likely cause | fix |
|---|---|---|
| pages return empty markdown | site is heavily JS-rendered, /scrape insufficient | switch to /scrape with actions or use the firecrawl-interact build-side skill |
| crawl exhausts budget early | depth or include too permissive | tighten include_paths, lower max_depth |
pages return 403 / 429 | rate-limit or anti-bot | reduce concurrency, set politeness_delay, ask user before retrying |
| same content captured under multiple URLs | URL canonicalisation not applied | follow canonical_url, deduplicate by content_hash |
topics/docs-knowledge-graph-pipeline/ uses this pipeline twice:
base_url: https://docs.firecrawl.dev (dogfood — capture the scraper's own docs)base_url: https://moonrepo.dev/docs (the user's stated pain point)Each becomes its own sub-corpus under 20-sources/firecrawl-docs/ and 20-sources/moon-docs/ respectively, with separate manifests.
Firecrawl exposes /scrape, /crawl, /map, /extract, and /interact endpoints. For docs-site capture, /crawl plus formats: ['markdown', 'links'] is the workhorse. /extract (LLM-driven schema extraction) is not used in this pipeline — extraction is a separate, deterministic step in extract-knowledge-graph so we can re-derive the KG without re-spending LLM tokens.