learn-site-structure

name

learn-site-structure

description

Authoritative reference for how docs in this repo (and 5 other Netdata-org repos) become published pages on `learn.netdata.cloud`. Covers the `<repo>/docs/.map/map.yaml` source-of-truth (the actual lever -- filesystem path is irrelevant for routing), the live `ingest/ingest.py` orchestrator in the learn repo (NOT the legacy `ingest.js`), frontmatter injection, slug rules, sidebar autogeneration, MDX escape rules, versioning, the 4-mechanism redirect stack, the 6 source repositories, the every-3-hours CI ingest, Netlify deploy, and the `part_of_learn=True` opt-in for files hand-authored in the learn repo. Use when adding/moving/renaming/deleting a docs page; when a page on Learn looks wrong; when wondering whether to edit a doc here or in the learn repo; when reading `ingest.py`, `sidebars.js`, `docusaurus.config.js`, `static.toml`, `LegacyLearnCorrelateLinksWithGHURLs.json`, `netlify.toml`, the `<!--startmeta` blocks in `.mdx` files, or the workflows `ingest.yml` and `daily-learn-link-check.yml`.

learn-site-structure

This skill is the single place to learn how content in this repo (and adjacent Netdata-org repos) becomes published pages on the Netdata learn site (learn.netdata.cloud). It documents:

• the <repo>/docs/.map/map.yaml source-of-truth that every published page must appear in;
• the source-path-to-Learn-URL computation (frontmatter is injected from map.yaml; the source filesystem path is irrelevant for routing);
• the 16-step ingest/ingest.py orchestrator in the learn repo (the live one -- NOT the legacy ingest.js);
• the 6 source repositories the pipeline pulls from;
• sidebar autogeneration via Docusaurus autogenerated mode;
• the MDX escape rules every transformation enforces;
• the 4-mechanism redirect stack and how moves auto-redirect;
• the every-3-hours CI ingest cadence and Netlify deploy contract;
• the part_of_learn: True opt-in for files hand-authored in the learn repo;
• every surprise, dead artifact, and gotcha worth knowing.

After reading SKILL.md plus the per-domain guides linked below, an assistant should never need to ask "how does a page get added to learn?", "how is the URL computed?", "should I edit this here or in the learn repo?", or "what runs in CI?".

Key concepts (read first)

1. <repo>/docs/.map/map.yaml is the lever. Every page published on Learn appears in this file. To add a page, add a node. To move a page, move its node. To unpublish a page, remove its node and (manually) update the redirect catalog. The schema lives at <repo>/docs/.map/map.schema.json; the authoring guide lives at <repo>/docs/.map/README.md.

2. Source filesystem path is IRRELEVANT for routing. A doc's destination URL on Learn is computed from sidebar_label + learn_rel_path injected from map.yaml, NOT from where the file lives in this repo. Renaming a source file does not break the URL; renaming the meta.label in map.yaml DOES (and triggers an auto-redirect).

3. The live orchestrator is ingest/ingest.py. Ignore ingest.js and ingest.md. Both are legacy. The README at the learn repo root and docs/.map/README.md document the live Python flow. Don't edit ingest.js; don't trust ingest.md.

4. Sidebar is autogenerated. ${NETDATA_REPOS_DIR}/learn/sidebars.js is one line: {type: "autogenerated", dirName: "."}. Order is driven by sidebar_position frontmatter that ingest assigns from map.yaml traversal order. To reorder the sidebar, reorder rows in map.yaml.

5. docs/ (in the learn repo) is OWNED by ingest. Anything under ${NETDATA_REPOS_DIR}/learn/docs/ is cleaned and regenerated each ingest run, EXCEPT files marked part_of_learn: True (currently only docs/ask-nedi.mdx). Do NOT hand-edit docs/ in the learn repo unless you set that flag and accept that this is a hand-authored exception.

6. 6 source repositories feed Learn. This repo (netdata/netdata) is the bulk; also pulled: netdata-cloud-onprem, .github, agent-service-discovery, netdata-grafana-datasource-plugin, helmchart. Each can be overridden locally with --repos for testing.

7. Auto-redirects on move/rename, manual on delete. Moving or renaming a page in map.yaml produces an automatic Netlify redirect from the old URL to the new target. Deleting a page requires manual surgery on ${NETDATA_REPOS_DIR}/learn/LegacyLearnCorrelateLinksWithGHURLs.json (documented in <repo>/docs/.map/README.md).

Table of contents

Guide	Purpose
mapping.md	The map.yaml schema; frontmatter injection; source-path-to-URL computation; slug rules; edge cases.
pipeline.md	The 16-step ingest.py flow; source repos; CI workflow; Netlify deploy.
sidebars.md	Docusaurus autogenerated sidebar; ordering rules; category overview pages; auto-grid pages.
mdx-rules.md	Every transformation in _escape_mdx_braces and sanitize_page; what breaks MDX 3 parsing.
redirects.md	The 4-mechanism redirect stack; auto-redirect on move/rename; manual unpublish surgery.
pitfalls-and-gotchas.md	Every surprise, dead artifact, edge case, silent failure mode, undocumented behavior.
authoring-boundary.md	What is owned by ingest (do not edit) vs hand-authored in learn repo; part_of_learn: True; cross-repo authoring.
recipes/INDEX.md	Step-by-step recipes for add / move / rename / delete a doc page.
how-tos/INDEX.md	Live catalog: every analysis question becomes a how-to entry.

Live how-to rule (mandatory)

If an assistant is asked a concrete question about how a Learn page is produced that is NOT already documented under how-tos/ or one of the per-domain guides above, AND answering it requires non-trivial analysis (reading ingest.py or related scripts, running ingest locally, checking cross-references between this repo's map.yaml and the learn-repo output), the assistant MUST author a new how-to under how-tos/<slug>.md and add a one-line entry to how-tos/INDEX.md BEFORE completing the task. This rule is durable.

Path discipline

This skill follows <repo>/.agents/sensitive-data-discipline.md:

• This repo's files: repo-relative (<repo>/docs/.map/map.yaml, <repo>/docs/...).
• Learn repo files: ${NETDATA_REPOS_DIR}/learn/... (env-key from .env).
• Other Netdata-org sibling repos: ${NETDATA_REPOS_DIR}/<repo>/....
• No literal home-directory or workstation-root paths anywhere.

Sources of truth referenced by this skill

• <repo>/docs/.map/map.yaml -- the publication source of truth.
• <repo>/docs/.map/map.schema.json -- the schema.
• <repo>/docs/.map/README.md -- the authoring guide.
• ${NETDATA_REPOS_DIR}/learn/ingest/ingest.py -- live orchestrator.
• ${NETDATA_REPOS_DIR}/learn/ingest/autogenerateRedirects.py -- redirect generator.
• ${NETDATA_REPOS_DIR}/learn/sidebars.js -- sidebar config.
• ${NETDATA_REPOS_DIR}/learn/docusaurus.config.js -- site config.
• ${NETDATA_REPOS_DIR}/learn/static.toml -- hand-curated static redirects.
• ${NETDATA_REPOS_DIR}/learn/test_escape_mdx_braces.py -- MDX escape test suite.
• ${NETDATA_REPOS_DIR}/learn/.github/workflows/ingest.yml -- ingest CI workflow.
• ${NETDATA_REPOS_DIR}/learn/.github/workflows/daily-learn-link-check.yml -- daily link checker.

Related skills

• integrations-lifecycle -- the integrations side (metadata.yaml -> per-integration .md). Integration pages are produced before ingest runs and inserted into map.yaml by ingest's populate_integrations step (see mapping.md).
• project-writing-collectors -- the broader collector authoring context.