myco-myco-sh-docs-pipeline

name	myco:myco-sh-docs-pipeline
description	Apply this skill when working on the myco.sh documentation site — including adding or updating guides, modifying the build pipeline, maintaining the GitHub Actions Pages deploy workflow, auditing SEO/AI-crawler discovery, or fixing rendering bugs in the static HTML output — even if the user doesn't explicitly ask about build infrastructure or deployment. Covers four interlocking procedures: (1) dual-surface architecture — keeping raw .md files for AI crawlers while generating parallel .html in docs/_site via docs/build.mjs; (2) build pipeline configuration using markdown-it + Shiki with the linkify fuzzyLink guard in docs/lib/render.mjs; (3) discovery layer maintenance — robots.txt, sitemap.xml, llms.txt, JSON-LD, and canonical tags; and (4) GitHub Pages workflow — matching upload-pages-artifact and deploy-pages version pairs to prevent silent deploy failures.
managed_by	myco
user-invocable	true
allowed-tools	Read, Edit, Write, Bash, Grep, Glob

myco.sh Docs Site — Dual-Surface Build Pipeline, Pages Deployment, and Content Safety

The myco.sh docs site serves two distinct audiences from one content base: raw .md files for AI crawlers and LLM contexts (indexed via llms.txt, sitemap.xml, and robots.txt), and statically-rendered HTML pages for human readers. The build step (npm run build inside docs/) compiles everything into docs/_site/ — both surfaces live there: copied raw .md files and generated .html files. That _site/ directory is what gets deployed to GitHub Pages.

Changes to any of these areas are coupled — a new guide needs an entry in docs/lib/nav.mjs (the sidebar manifest), and a link in llms.txt. The sitemap.xml is auto-generated from the nav manifest. Understand the full picture before editing any one part.

---

Prerequisites

• Node.js available; build runs as npm run build from inside the docs/ directory (executes docs/build.mjs).
• Familiarity with docs/ directory structure:
  • docs/*.md — source markdown guides (do not move or rename without updating llms.txt and the nav manifest)
  • docs/lib/nav.mjs — sidebar nav manifest and single source of truth for guide ordering; add new guides here
  • docs/build.mjs — build orchestrator; reads nav manifest, renders HTML, copies static files, generates sitemap
  • docs/lib/render.mjs — markdown-it renderer (Shiki theme + linkify guard)
  • docs/lib/template.mjs — HTML template with left-sidebar nav
  • docs/lib/links.mjs — link rewriter (.md → clean URL; out-of-tree → GitHub blob)
  • docs/_site/ — build output directory (deployed to Pages); not committed to git
  • .github/workflows/docs.yml — Pages deploy on push to main
• Read docs/lib/nav.mjs before adding guides — guides not listed in the NAV array get no inbound navigation links and their source files are not picked up by the build.

---

Procedure 1: Dual-Surface Architecture — Adding or Restructuring Guides

The fundamental constraint: raw .md source files must stay at their paths under docs/. The build copies them verbatim into docs/_site/, preserving the AI-crawler surface. Deleting or moving .md source files breaks llms.txt links and the LLM-facing surface even if the human-facing HTML is intact.

How the dual surface works:

docs/quickstart.md            ← Source (not deployed directly)
                                 ↓ build step
docs/_site/quickstart.md      ← AI crawlers, llms.txt links, sitemap (raw Markdown copy)
docs/_site/quickstart.html    ← Human readers (generated by build step)
/quickstart                   ← Clean URL served by GitHub Pages (maps to .html)

Adding a new guide:

1. Write the guide as docs/<name>.md. Follow the write-myco-user-documentation skill for content style.
2. Open docs/lib/nav.mjs and add a { slug: '<name>', title: '<Display Title>' } entry to the appropriate group in the NAV array. The NAV array is an ordered list of { group, items } objects; each item uses slug (filename without .md) and title (sidebar label). Omitting this step means the build ignores the guide entirely — no HTML generated, no sitemap entry, no inbound nav links.
3. Add a link to llms.txt under the appropriate section so LLM crawlers discover it. Use the .md URL (e.g., https://myco.sh/<name>.md), not the .html URL.
4. Run npm run build from inside docs/ and verify docs/_site/<name>.html is generated without errors.
5. Smoke-test with a Pages-faithful local server (see Procedure 4) — confirm internal links resolve and no bare filenames were linkified (see Procedure 2).

Note — sitemap is auto-generated. The build script generates docs/_site/sitemap.xml from the NAV manifest on every build using today's date (or DOCS_BUILD_DATE env var). Do not manually edit a static sitemap.xml — it will be overwritten on the next build.

Link rewriting rules (enforced by docs/lib/links.mjs via docs/build.mjs):

Link type in .md source	Rewritten to in .html output
[text](other-guide.md)	/other-guide (clean URL)
[text](../../AGENTS.md)	GitHub blob URL (out-of-tree)
https://example.com	Unchanged

Do not write .html links in markdown source — the renderer handles the conversion.

---

Procedure 2: Build Pipeline Configuration (docs/lib/render.mjs)

The renderer uses markdown-it with Shiki syntax highlighting. There are two non-obvious configuration decisions that must be preserved when modifying render.mjs.

Shiki theme: The dark theme is applied at build time (no client-side JS required). If you change the theme, rebuild and visually verify code blocks render correctly — Shiki theme names are not validated at import time.

The linkify fuzzyLink guard (critical — do not remove):

markdown-it with linkify: true auto-linkifies bare strings that look like domains. Because .md, .sh, .ts, .io are real ccTLDs, filenames like CLAUDE.md, AGENTS.md, and install.sh get converted to broken external links (http://CLAUDE.md) in rendered prose. Technical docs mention filenames constantly, so this produces broken links throughout the site.

The fix is already in place in docs/lib/render.mjs at lines 9–13 — do not remove or change these lines:

const md = MarkdownIt({ html: true, linkify: true, typographer: true });
// Required: disables fuzzy (schemeless) linkification while keeping
// explicit-scheme URLs (https://...) auto-linked.
md.linkify.set({ fuzzyLink: false, fuzzyEmail: false, fuzzyIP: false });

This preserves https://example.com auto-linking while suppressing CLAUDE.md → http://CLAUDE.md conversion. Removing fuzzyLink: false re-introduces the bug across all rendered guides.

This bug is not caught by unit tests on the render function — it only surfaces when real prose with filenames is rendered and crawled. Always smoke-test with the link crawler (see Procedure 4) after render changes.

---

Procedure 3: Discovery Layer Maintenance

The discovery layer (robots.txt / sitemap.xml / llms.txt / JSON-LD) is independently important from the HTML rendering. It must be kept current whenever guides are added, renamed, or removed.

Component inventory and update triggers:

File	Purpose	Update when
docs/robots.txt	Explicit allow rules for crawlers + sitemap pointer	Rarely — only if adding new crawler allow rules
docs/_site/sitemap.xml	Auto-generated from NAV manifest on every build	Automatic — just keep docs/lib/nav.mjs current
docs/llms.txt	Curated LLM-readable entry point (Markdown)	Adding or significantly revising a guide
docs/index.html (head)	Canonical URL, robots meta, OG/Twitter tags, JSON-LD	Changing site name, URL, or description

docs/robots.txt — allowed crawler entries:

• GPTBot (OpenAI training)
• ClaudeBot (Anthropic)
• PerplexityBot
• OAI-SearchBot (ChatGPT search)

Add new crawlers here as they emerge with stable user-agent strings. The sitemap directive must point to the live https://myco.sh/sitemap.xml.

docs/llms.txt — keep concise. It is not a full index — it is a curated entry point. Include the guide name, a one-line description, and the .md URL (not the .html URL — LLMs benefit from the raw Markdown). Do not link to GitHub URLs for docs that live on myco.sh; local links preserve topical authority. The build script verifies that all myco.sh/*.md links in llms.txt resolve to actual files in _site/ — broken links will abort the build.

docs/index.html JSON-LD schemas — three schemas are present: Organization, WebSite, SoftwareApplication. Update the SoftwareApplication version string and datePublished/dateModified when releasing a new myco version. These fields feed structured-data cards in search results.

SEO baseline for guides: The build injects <link rel="canonical">, <title>, and <meta name="description"> into each generated HTML page using the guide's slug, first H1, and opening paragraph. Verify these are populated correctly in the rendered output after adding a new guide.

---

Procedure 4: GitHub Actions Pages Deployment Workflow

The deploy workflow (.github/workflows/docs.yml) triggers on push to main (when docs/** changes) and runs npm run build from inside the docs/ directory before uploading docs/_site as the artifact and deploying to Pages.

Version pairing rule — do not mix major versions:

GitHub Pages actions must use compatible major versions. A mismatched upload-pages-artifact / deploy-pages pair can fail silently or with an opaque error at the deploy step (the upload step reports success, making the failure hard to diagnose).

Always use a matched pair. The current workflow uses v4:

# Current (both @v4)
- uses: actions/upload-pages-artifact@v4
  with:
    path: docs/_site
- uses: actions/deploy-pages@v4

# If bumping to v5, bump BOTH at the same time:
- uses: actions/upload-pages-artifact@v5
  with:
    path: docs/_site
- uses: actions/deploy-pages@v5

When bumping versions: bump both actions in the same commit. A partial bump restores the mismatch problem. Check the GitHub Actions changelog for deprecation notices before merging workflow PRs.

Workflow validation checklist before merging any .github/workflows/docs.yml change:

1. upload-pages-artifact and deploy-pages are the same major version.
2. The path: passed to upload-pages-artifact is docs/_site (the build output directory — contains both .md and .html files for both surfaces).
3. The build step (npm run build from working-directory: docs) runs before the upload step.
4. Permissions block includes pages: write and id-token: write.

Local smoke-test before pushing (Pages-faithful routing):

GitHub Pages serves clean URLs by trying path → path.html → path/index.html → 404.html. A plain python3 -m http.server does NOT replicate this — .html extension stripping won't work and internal clean-URL links will 404, giving false negatives.

Use a Pages-faithful preview server pointed at docs/_site:

// /tmp/myco-docs-preview.mjs  (Node, run with: node /tmp/myco-docs-preview.mjs)
import http from 'http';
import fs from 'fs';
import path from 'path';

const ROOT = './docs/_site';
http.createServer((req, res) => {
  const tries = [
    path.join(ROOT, req.url),
    path.join(ROOT, req.url + '.html'),
    path.join(ROOT, req.url, 'index.html'),
    path.join(ROOT, '404.html'),
  ];
  for (const f of tries) {
    if (fs.existsSync(f) && fs.statSync(f).isFile()) {
      res.writeHead(200);
      return fs.createReadStream(f).pipe(res);
    }
  }
  res.writeHead(404); res.end('Not found');
}).listen(8080, () => console.log('http://localhost:8080'));

Then run a BFS link crawler against http://localhost:8080 and assert all internal links return 200. Flag any external links that look like bare filenames (e.g., http://CLAUDE.md) — those indicate the linkify fuzzyLink guard was removed.

---

Cross-Cutting Gotchas

Adding a guide without updating docs/lib/nav.mjs means the build ignores it. The guide won't be rendered to HTML, won't get a sitemap entry, and won't have inbound navigation links. The build throws an error if a NAV slug has no corresponding .md source file — but the reverse is silent: a .md file with no NAV entry is silently skipped.

Do not serve docs with python3 -m http.server for testing. It does not strip .html extensions, so clean-URL internal links (/quickstart, /install) will 404. Always test against docs/_site using the Pages-faithful preview server in Procedure 4.

sitemap.xml is generated, not committed. It lives in docs/_site/sitemap.xml after a build — do not create or maintain a static docs/sitemap.xml. Keeping guide URLs current in sitemap.xml is automatic once the NAV manifest is accurate.

Link rewriting is path-relative. If you restructure the docs/ directory hierarchy (e.g., add subdirectories like architecture/), re-verify the out-of-tree link detection logic in docs/lib/links.mjs. Links classified as out-of-tree (rewritten to GitHub blob URLs) may become in-tree or vice versa.

llms.txt links must point to .md URLs, not .html URLs. LLMs benefit from the raw Markdown. The rendered HTML is for humans. Mixing this up undermines the dual-surface strategy. The build verifies llms.txt .md links resolve — broken ones abort the build.

Both surfaces must be present in docs/_site. The build copies raw .md files alongside generated .html files into _site/ and runs a post-build guard to verify the copy succeeded. If the artifact path in the workflow is narrowed to only .html files, the raw .md files disappear from myco.sh and break the AI-crawler surface.