sync-docs

# /sync-docs — Paperclip docs sync skill This skill keeps **paperclip-docs** in lockstep with the parent code repo **paperclipai/paperclip**, rewriting changes in our friendly-tutorial voice for an audience that spans end users, operators, and developers. > Our docs are not a translation of the parent's docs. The **parent's code** is the source of truth. The parent's release notes and inline docs are reference data for understanding intent. > For the visual model of the branch flow and per-run phases, see [`maintenance/maintenance.md`](../../maintenance/maintenance.md) → Sync workflow → Architecture (two mermaid diagrams). This file is the operational playbook; that file is the picture. ## Mental model | | `nightly` mode | `release` mode | |---|---|---| | Tracks | parent `<parent-default>` HEAD (currently `master`) | parent's latest **release tag** | | Targets | `nightly` branch of this repo | `main` branch of this repo (via PR from `nightly`) | | Deploys to | Cloudflare Pages branch preview | docs.paperclip.ing | | Audience | early adopters, contributors | everyone — end users to devs | | Run cadence | daily (cron / `/loop 24h`) | when a new parent tag appears | | Touches live? | No | Yes | The branch model is non-negotiable: end users on the latest *released* paperclip must never see docs for features that aren't out yet. ## Files this skill reads / writes - `scripts/sync/anchor-map.json` — read-only config: which parent paths to watch, which docs paths they map to, which tier (auto-merge vs PR). - `.sync-state.json` — read/write: where this branch last left off and what was last applied. Schema: `branch_mode`, `base_release_tag`, `base_release_sha`, `last_seen_parent_sha`, `last_applied_manifest_hash`, `last_run_at`, `last_run_outcome`. - `docs/user-guides/screenshots/registry.json` — read/write: screenshot dependency tracking. - `PENDING.md` (on `nightly` branch only) — **regenerated** from scratch each run (not appended) so it always reflects the current cumulative manifest. Stale entries from reverted commits never linger. - `SCREENSHOTS_PENDING.md` (committed) — regenerated each run, lists screenshots whose `depends_on` paths changed in the diff window. - `docs/reference/changelog.md` — **release mode only**: append one accordion entry per release (Phase 6.5). User-facing and linked from the nav; nothing else writes it. Writing the entry is only half the job — it is not done until `docs.paperclip.ing` is serving it (Phase 9). **Helper scripts called during the run:** - `scripts/sync/compare-window.mjs` — Phase 2 cumulative diff (handles GitHub compare API truncation via midpoint bisection). - `scripts/sync/check-drift.mjs` — Phase 1.5 drift detection (finds documented surfaces missing from parent). - `scripts/sync/detect-renames.mjs` — Phase 3 directory rename detection (distinguishes a renamed surface from a brand-new one). - `scripts/sync/verify-edit.mjs` — Phase 5.5 post-edit verification (checks that authored claims still match parent code). - `scripts/sync/check-tag-accuracy.mjs` — **release mode only** (Phase 5.7): the tag-accuracy gate. Verifies every doc page changed on `nightly` vs `main` against the **stable release tag** and flags post-tag leaks — content nightly drafted from `master` that is not in the release. See `maintenance/release-channels-plan.md` for why this exists. - `scripts/sync/realign-nightly.mjs` — post-release realign (Phase 7): re-anchors `nightly` onto the merged release squash commit (restoring ancestry) **without discarding** nightly's post-tag `master` drafts. **Build gates that must pass before commit** (Phase 7; all also run in CI via `.github/workflows/docs-checks.yml`): - `npm run docs:test:seo-metadata` — every page has a unique, hand-written `seo_title` and `seo_description`. See "Authored SEO metadata". - `npm run docs:test:crawlable-links` — no raw `.md` hrefs, no links to missing or redirecting targets, sidebar server-rendered, sitemap dates not uniform. - `npm run docs:test:skill-source-blocks` — per-skill pages embed their authoritative `SKILL.md`. - `npm run docs:test:static-routes`, `npm run docs:test:asset-fingerprints`, `npm run docs:test:hierarchical-skills-nav`. `npm run docs:test` runs all six. ## Invocation ``` /sync-docs # auto: nightly if no new parent tag, release if new tag /sync-docs --nightly # force nightly mode /sync-docs --release # force release mode /sync-docs --dry-run # triage only, no edits, no commits, no PRs /sync-docs --since v2026.318.0 # override starting point /sync-docs --batched # release mode only: one PR per release tag (use for big gaps) ``` ## Preconditions to verify before doing anything 1. `gh auth status` is logged in (we need GitHub API for parent repo). 2. Working tree is clean (`git status`). If dirty, abort and tell the user — never stash their work. 3. `.sync-state.json` exists and parses. If missing, abort and tell the user to seed it from `scripts/sync/state.example.json`. 4. `scripts/sync/anchor-map.json` exists and parses. 5. **Resolve parent default branch.** Call `gh api repos/paperclipai/paperclip -q '.default_branch'` once at the start of the run and store the result in a local variable referenced below as `<parent-default>`. As of this writing it returns `master`, but capturing it dynamically means the skill keeps working if the parent ever renames to `main`. All subsequent references to "parent's main HEAD" / "parent main HEAD" in this skill resolve to `<parent-default>`. 6. **Preflight watcher paths.** For every concrete (non-glob) path in `anchor-map.json` watchers' `parent_paths`, call `gh api repos/paperclipai/paperclip/contents/<path> --silent` and **warn** (do not fail) on 404. This catches stale path entries like the `server/src/env.ts` case found in the design dry-run — surface them in the run summary so the human can update `anchor-map.json`. ## Step-by-step ### Phase 1 — Decide mode and target branch > **Release channels — read this first.** Paperclip publishes on four lanes: > `canary` (every merge to `<parent-default>`) → `nightly` (a green `master` build, > smoke-tested nightly) → `beta` (a promoted nightly, soaks ≥3 days) → `stable` > (the manually cut release). **Only `stable` has git tags and GitHub Releases** > (clean CalVer `vYYYY.MDD.P`); beta/nightly/canary are npm/Docker artifacts with > no git ref we can diff against. **Docs ship on `stable`.** Note the naming > collision: *our* `nightly` branch tracks parent `<parent-default>` HEAD (≈ the > **canary** lane), which runs far ahead of the stable tag — so a release branch > cut from `nightly` inherits post-tag `master` drafts. Phase 5.7 is the gate that > catches those. See `maintenance/release-channels-plan.md`. 1. Read `.sync-state.json`. Note `branch_mode` and `base_release_tag`. 2. Fetch parent's latest **stable** release: `gh api repos/paperclipai/paperclip/releases/latest -q '.tag_name'`. `releases/latest` already excludes prereleases, but **guard defensively**: the release target MUST match `^v?\d{4}\.\d{1,4}\.\d+$` (clean CalVer). Any `-beta`/`-nightly`/`-canary` tag is never a release target — if the API ever returns one, treat the run as nightly mode and note it in the summary. 3. Auto-detect mode (unless overridden): - If a new stable release tag exists AND `base_release_tag` is older → **release mode**. - Otherwise → **nightly mode**. 4. Check out the right branch: - Release mode: ensure on `main`. Release mode is **self-sufficient** — it does not require the `nightly` branch to exist or to have drafts. If `nightly` exists with relevant drafts, they're used as a starting point; if not, the release run computes everything from scratch. - Nightly mode: ensure on `nightly`. If `nightly` doesn't exist, create it from `main`. 5. Nightly mode only: **merge `main` into `nightly` first** to absorb any hot-fix typos that landed on released docs. Resolve trivially or abort if conflicts need human attention. - **Ancestry guard (squash-only repo).** Before merging, check `git merge-base --is-ancestor <last release squash commit> nightly` — practically: does the merge produce add/add conflicts on pages both branches created? This repo squash-merges release PRs, which severs the ancestry between `main` and `nightly`; if the post-release realign was skipped, this merge explodes into add/add conflicts on every page nightly drafted that cycle. If that happens, do NOT resolve by hand — abort and run `node scripts/sync/realign-nightly.mjs <release-branch>` (see Phase 8 and the special case below), then retry. ### Phase 2 — Resolve the diff window (cumulative) Both modes use **cumulative diffs** — always from a stable base, never incrementally from yesterday. This makes reverts auto-cancel (they're net-zero in the cumulative diff) and lets `nightly` be safely regenerated. - **Release mode**: `prev = state.base_release_tag`, `next = latest release tag`. Build a list of intermediate tags so `--batched` can produce one PR per tag. - **Nightly mode**: `prev = state.base_release_tag` (NOT yesterday's SHA), `next = parent `<parent-default>` HEAD`. Then apply **quarantine**: ignore any commits younger than `quarantine_hours` (default 24) so reverts have time to land before we process the original. For each window: ``` gh api repos/paperclipai/paperclip/compare/$PREV...$NEXT \ -q '.files[] | {filename, status, additions, deletions, patch}' ``` Cache result under `/tmp/paperclip-sync/<sha>/` so we don't refetch within a run. > **Pagination & truncation.** The GitHub `compare` endpoint caps responses at 300 files / 250 commits. Use the helper that handles recursive midpoint bisection by SHA: > > ``` > node scripts/sync/compare-window.mjs $PREV $NEXT --json > ``` > > Returns `{ from, to, total_commits_seen, leaves, truncated_leaves, files: [...] }`. `truncated_leaves` MUST be 0 in a successful run; non-zero means a leaf still hit the cap (should be impossible) and the run should abort with that fact in the summary. > > The script applies these status-merge rules when unioning leaf file lists: `added`+`removed` → drop; `added`+`modified` → `modified`; `modified`+`removed` → `removed`; `renamed` wins over `modified`. Latest-seen status wins otherwise. > > Leaf responses are cached under `/tmp/paperclip-sync/` so re-runs within a day are cheap. > **Why cumulative, not incremental?** If we diffed `yesterday → today`, a revert commit landing today would need to be processed to undo yesterday's doc edit — and filtering revert commits by message regex would lose that signal. With cumulative diffs from the last release, reverts simply aren't in the diff at all. The original commit and its revert cancel out before we ever see them. ### Phase 1.5 — Drift check (independent of diff window) Drift is the inverse of the cumulative diff: it's the set of things **we already document** that have since vanished or moved upstream. It exists regardless of when the last sync happened — a parent surface can disappear between two sync runs even if our diff window is empty. The wet-run that motivated this phase found `POST /api/companies/{companyId}/logo` documented but absent from current `server/src/routes/companies.ts`, with no sign of it in any diff window the sync had ever processed. Run the drift checker against the reference for this mode. **Nightly mode → `<parent-default>`** (drift is about the live upstream). **Release mode → the stable release tag** — checking release-mode drift against `master` is blind to the tag divergence and will pass surfaces that exist on `master` but not in the release (that is how the Kimi leak slipped past a `0`-drift run). Use `<release-tag>` in release mode: ``` # nightly mode node scripts/sync/check-drift.mjs --json --against <parent-default> # release mode node scripts/sync/check-drift.mjs --json --against <release-tag> ``` The script scans `docs/**` for four reference classes and verifies each one still exists in parent: | Class | What we scan | Confidence | |---|---|---| | parent-path-missing | `cli/src/...`, `server/src/...`, `packages/<name>/...`, `skills/paperclip/...` references with `.ts`/`.mjs`/`.js` extensions | high | | cli-command-missing | `paperclipai <subcommand>` invocations under `docs/reference/cli/**` | high | | env-var-missing | Rows in `docs/reference/deploy/environment-variables.md` | high | | rest-route-missing | `GET/POST/PUT/PATCH/DELETE /api/...` headers under `docs/reference/api/**` | medium | Output is structured JSON with a `drift` array of records: `kind`, `doc` (file:line), `documented`, `parent_searched`, `confidence`, `suggest`. The script always exits 0 — drift is a warning, never a hard failure. Results are cached under `/tmp/paperclip-sync/drift-<ref>/` so re-runs within a day are cheap. **Where drift records go.** They join the change manifest as a separate **drift tier** — not `auto-merge`, not `pr`, not `context-only`. They have their own category because they're driven by what's missing from parent, not by what changed in a window. **Never auto-resolve drift.** Even high-confidence drift requires human judgement: a missing route may have been moved (update the doc) or removed (delete the section). Always surface drift to the human: - In the run summary (Phase 4 / Phase 8). - In `PENDING.md` under a "⚠ Drift" section (nightly mode). - In the PR body under a "⚠ Drift" heading (release mode). **Confidence handling in the summary.** High-confidence findings (parent paths, CLI commands, env vars) get prominent placement at the top of the drift section. Medium-confidence findings (REST routes — route prefixing and dynamic registration can hide real matches) are prefixed with `Verify:` so reviewers know to spot-check before acting. The medium tier is intentionally noisy on the side of caution. If `--dry-run` is set, drift candidates are printed alongside the rest of the manifest summary and the run stops there. ### Phase 3 — Surface diff (the change manifest) For each watcher in `anchor-map.json`: 1. Intersect the changed-files list with the watcher's `parent_paths` globs. 2. For matching files, apply the watcher's detection rule (descriptions in `anchor-map.json`'s `detect` field — interpret semantically, you're not running grep blindly). 3. **Rename pass.** Before finalising entries that look like "new doc page for a brand-new top-level dir" (especially under the `adapters`, `agent-skills`, `server-adapters`, `plugin-sdk` watchers), run: ``` node scripts/sync/detect-renames.mjs /tmp/diff.json --json ``` Apply the result: - **Detected rename** (`renamed_from → renamed_to`): do NOT emit a "new doc page" entry for the `renamed_to` dir. Instead route the entry to the **existing** doc page for the `renamed_from` dir with `change_kind: renamed` and an `evidence` line that includes the helper's `confidence` and `signal` fields. - **`added_dirs_genuinely_new`**: proceed as the existing rules say — emit a "new page" entry mirroring a neighbour. - **`removed_dirs_no_match`**: surface as a ⚠ Reconcile-style flag (a documented surface vanished upstream with no rename target — the human decides whether to remove or archive the doc page). 4. Output a structured entry: ```yaml - watcher: cli-commands parent_files: [cli/src/commands/worktree.ts] change_kind: added # added | modified | renamed | removed surface: "paperclipai worktree prune" evidence: "New program.command('prune') call at line 84" docs_targets: [docs/reference/cli/worktree.md] tier: pr parent_commits: [a1b2c3d, e4f5g6h] release_context: "v2026.319.0 — Highlights: 'Worktree pruning for orphaned dirs'" ``` Filter the change manifest: - Drop watcher-`context-only` entries (they only feed framing into other entries). - Sort by tier (auto-merge first, then PR). > **We do not filter commits by message regex.** No `skip_patterns`. The cumulative diff already filters by *outcome* (a reverted commit's net change is zero, so it's not in the diff). Filtering by commit-message regex would dangerously hide undo signals — see the warning in Phase 2. Write the manifest to `/tmp/paperclip-sync/manifest.yaml` and, in nightly mode, **regenerate** `PENDING.md` at repo root from this manifest (overwrite — do not append). Compute `manifest_hash = sha256(canonical manifest yaml)` for use in the reconciliation step below. ### Phase 3.5 — Reconciliation (catches reverts of previously-applied changes) Cumulative diffs prevent us from making *new* wrong edits, but they don't automatically undo edits we already committed in a prior run for a feature that has since been reverted.