rollout

# stardust:rollout — whole site → AEM (Edge Delivery Services) `deploy` converts **one** page to AEM. `rollout` delivers the **whole site**: it inventories the agnostic output of `migrate`, then drives `deploy` across every page, tracking delivery coverage so you always know what's done and what's left. `rollout` is **delivery-only** — it does not redesign. The page-by-page redesign (`extract → direct → prototype → migrate`) and `deploy` itself are **unchanged**; `rollout` is the across-pages layer on top. Design rationale, coverage model, and phasing are in [`notes/rollout/PLAN.md`](../../notes/rollout/PLAN.md). The flow runs **A→I** below. ## When to use **Full mode** — the user has a fully migrated site at `stardust/migrated/` (per-page HTML + `_meta.json` from `stardust migrate`), an EDS/AEM project + DA destination (the same target `deploy` needs), and wants the **entire** site delivered, incrementally and resumably. **Archetypes-only mode** — the user has one migrated archetype per template plus a full page inventory in `stardust/state.json` (with `type` per page), and wants to ship all block code immediately without waiting for every page to be migrated. Sibling pages register as `content-pending` and get their content later via a separate track. If there is no `stardust/migrated/` tree at all, recommend `stardust migrate` on at least the archetype pages first. For a single page, use `stardust deploy` directly. ## Setup 1. Run the master skill's setup (`skills/stardust/SKILL.md` § Setup). **Flow guard:** `stardust/state.json` without `flow` on a migration ask → do not roll out; print the master's two-flow table and hand back to its routing (`skills/stardust/reference/state-machine.md` § Flow keys). 2. Verify `stardust/migrated/` exists with at least one `*.html` page (full mode: all pages; archetypes-only: the archetypes + a `state.json` with `type` populated). If not, recommend `stardust migrate` on the archetypes and stop. **Gated-archetype precondition (`flow: replica`).** Read `stardust/replica/progress.json`: a page type may ship only when its archetype has a gate result at every configured breakpoint that is `pass: true`, or over the bar with every residual carrying a `cause` (`skills/replica/reference/source-fidelity-gate.md` § Residual logging format — a documented residual is a pass with an asterisk). A page type whose archetype was never gated, or is over the bar with no residual entries, is **blocked**: list it with its archetype slug and the command to gate it (`$stardust replica <archetype>`), and neither fan out its siblings nor `POST /live/` any of them. Accepting logged residuals under hands-off is not a bypass for an ungated archetype. Thresholds are the gate's, unchanged. (Recorded: 2,207 pages published at 24–28 % diff from an archetype that never passed; a 3,366-page re-import after a random review found what a gate would have.) 3. Verify the EDS/AEM target is ready exactly as `deploy` requires (project scaffolding, `DA_TOKEN`, code branch pushable). `rollout` adds no new transport. 4. If `state.json.handsOff` is true (`skills/stardust/SKILL.md` § Hands-off mode), run full-auto: no per-phase pauses. Every gate and verify step below runs unchanged — hands-off removes waiting, not validation. ## Procedure ### Phase A — Inventory (build the coverage) ```bash node skills/rollout/scripts/inventory.mjs --site-url <source-url> # defaults: --migrated stardust/migrated --out stardust/rollout # archetypes-only mode: add the full page roster from state.json node skills/rollout/scripts/inventory.mjs --site-url <source-url> --state stardust/state.json ``` Writes `coverage/pages.json` (one row per page: slug, delivered `path`, `templateId`, `blocks`, `sourceHash`, `delivery` status), `coverage/templates.json` (pages grouped by template), and `rollout.json` (target + DA config + `lastRun`). **Archetypes-only mode** (`--state`): pages with a `_meta.json` are seeded as in full mode; pages present only in `state.json` are seeded with `templateId` from `type`, `blocks` from the archetype sidecar, and `delivery.status: content-pending`. Inventory is **idempotent and incremental**: delivery status is preserved; a page whose migrated HTML changed after delivery is re-flagged `stale`. Fill in the DA coordinates in `rollout.json` (`site.da.org`, `site.site`, `site.da.ref`, `site.liveHost`) if not inferred. ### Phase B — Block dedup plan (FIRST-CLASS, before any conversion) ```bash node skills/rollout/scripts/blocks.mjs # → coverage/blocks.json (the dedup unit) node skills/rollout/scripts/plan.mjs # → plan.json + a readable conversion plan ``` - `blocks.mjs` collapses every block instance (per-page `modules` + chrome) into the **distinct** set, assigns each a canonical `edsBlockName` (kebab, reserved-class-guarded per deploy #15), and records `usedByPages` / `instanceCount`. Chrome (`header`/`nav`/`footer`) is `kind: chrome` → site-wide authored documents (`/nav`, `/footer`) fed to the header/footer blocks. In archetypes-only mode the archetype sidecars fully determine the block set; `content-pending` pages add none. A module that maps to EDS **default content** (title, text, image, button, separator — deploy's D1) needs no block: record it `update-coverage.mjs --block <id> --status converted --eds-name default-content`; such a row is never counted pending, whatever its status. - `plan.mjs` orders pages **representative-first per template** and gives each distinct block a **single conversion point**: the first page that uses it CONVERTS it, every later page REUSES it by name. The per-page `convert`/`reuse` lists are exactly `deploy`'s Step-7 brief input, so each block converts once **without changing deploy**. `content-pending` pages are always `convert: []`. > Extending an already-delivered site? A "new template" is almost always a new > COMPOSITION of the existing block library, not new block code — audit `blocks/` > first. See `reference/operational-learnings.md`. ### Phase B2 — Dynamic surface (PRE-IMPORT GATE — verify the inventory) **Before Phase C.** `stardust/dynamic-features.md` (from prepare-migration 4.5 or replica Phase 2) must exist with a disposition on every row; verify it against fresh evidence here — `dynamics-detect.mjs --from-state … --reach stardust/current` and `dynamics-plan.mjs --target-origin <live host> --migrated stardust/migrated` (host-bound APIs, rows the capture already delivered). New evidence → new rows. The listings contract (per-type `<meta>` fields + `helix-query.yaml`) is emitted by Phase C's `deploy` brief per page: retrofitting metadata across published pages is a second migration. Missing inventory → run the stardust `dynamics` skill Phases 1–3 now. Contract: `skills/dynamics/reference/triage.md`, `reference/listings.md`. ### Phase C — Deliver the site (drive `deploy` per page, per the plan) **Blocked on Phase B2** — author each page's metadata contract into its metadata block during delivery, so the indexes are rich at import time. Walk `plan.json.steps` in order (representative pages first). For each page: 1. **Convert + push** the migrated HTML (`source.migratedHtml`) to AEM via the `deploy` methodology. **Pass the plan step into deploy's brief**: create only the blocks in `convert`; for each block in `reuse`, REUSE the existing block by its `edsBlockName` (do not recreate). **The brief MUST carry the Experience Workspace editability contract** (deploy SKILL.md § 8, EW1–EW10): every converted block moves authored elements into wrappers (never rebuilds from text) and passes the EW gate (`block-roundtrip --ew`) before it counts as delivered — a brief without it skipped the contract on 27/27 blocks of a real site. **Deploy first, judge on the preview origin.** The local harness (`build-harness.mjs` → `qa-gate.mjs`, `block-roundtrip.mjs --ew`) serves the structural asserts only; every pixel or visual judgment — replica's source-fidelity gate, the header/footer `crop-compare` bands, the deployed eyeball — runs against the page's preview URL after `PUT → preview`, never against the harness before the first PUT (deploy SKILL.md § Local QA before deploy, § Step 10). A recorded delivery session iterated CSS against a local harness pixel diff for its whole budget and delivered no page. **Chrome and fragment documents carry `Robots | noindex`.** `content/nav.html`, `content/footer.html`, every per-locale `nav-*` / `footer-*` document and any locale shell that is not a page get one more metadata block row at write time — `<div><div>Robots</div><div>noindex</div></div>` (renders `<meta name="robots" content="noindex">`). Without it the platform's index of published documents, and the `/sitemap.xml` it serves, list them as pages: a recorded hands-off run served 58 sitemap urls for a 36-page site (every page plus 22 chrome documents). Phase D verifies the served sitemap. **`content-pending` pages** (archetypes-only): no migrated HTML — skip the document push entirely (no shell/placeholder), record `content-pending`, surface as "awaiting content track." Their block code is already deployed via the archetype. 2. **Static contract lint (pre-PUT, deterministic).** Before the push, run the delivery-contract linter — it catches the cheap, deterministic failures (wrapper, one-CTA-per-`<p>`, trailing-slash, path-safety, `/img/` src, `about:error`) offline so a broken page never reaches preview. Mechanics in `reference/delivery-lint.md`. **A P0/P1 blocks the PUT.** ```bash node skills/rollout/scripts/delivery-lint.mjs --file <html> --path </da/path> node skills/rollout/scripts/media-reconcile.mjs --file <html> --deploy-host <branch>--<repo>--<owner>.aem.live [--media-ledger <file>] [--apply] ``` `media-reconcile` resolves every non-hosted image on the network and decides optimize/keep/rewrite/omit (`skills/migrate/reference/media-reconciliation.md`) — the authoritative form of the image-fidelity gate below. A content-host URL (`content.da.live`) is the `hosted` decision instead: checked offline against the media ledger (`--media-ledger <file>`, auto-detected), never fetched anonymously (`401` by design); missing from the ledger fails the gate. 3. **Run the delivery gates** before flipping a page to `deployed`. Each is a one-line rule here; mechanics + helpers in `reference/delivery-gates.md`: - **Source-fidelity** — don't add sections the source lacks; never fabricate facts. `node skills/rollout/scripts/section-fidelity.mjs --file <html> --source <url>` (a static outline check on the authored file — not replica's pixel source-fidelity gate, which runs on the published origin after `deployed`) - **Image-fidelity** — every authored `<img>` src must return 200 or be omitted; never ship `<img src="about:error">`. Run `media-reconcile.mjs` (step 2). - **Path-safety** — normalize source paths to AEM-Edge-safe form (lowercase, no trailing `-`/`_`, no `--` segment); record original→normalized in `stardust/redirects.tsv`. (delivery-lint flags violations.) - **Source-content hygiene** — skip dead source URLs; author bodyless/PDF-only sources thin and faithful (tier `thin`, `skills/migrate/reference/fidelity-tiers.md`), don't pad with invented prose. - **Fidelity tier declared** — record each page's `fidelityTier` (archetype/sibling/thin) so coverage shows what was craft-gated vs cloned (`skills/migrate/reference/fidelity-tiers.md`). 4. **Record outcomes** with the state-writer (never hand-edit the ledger): ```bash node skills/rollout/scripts/update-coverage.mjs <slug> --status converting node skills/rollout/scripts/update-coverage.mjs --block <id> --status converted --eds-name <name> node skills/rollout/scripts/update-coverage.mjs --block <id> --status converted --eds-name default-content # maps to default content, no block node skills/rollout/scripts/update-coverage.mjs <slug> --status deployed --url <branch-preview-url> node skills/rollout/scripts/update-coverage.mjs <slug> --status content-pending # no document push ``` **Publish in the loop (`PUT → preview → live`), don't stop at preview** — any query-index (Phase D2) builds from the **live** tree, so a preview-only delivery leaves indexes empty. On failure: `--status failed --error "<reason>"` and continue (one page's failure never aborts the rollout). **Foundation-first gate (hard block, once per rollout).** When the FIRST archetype page flips to `deployed`, stop and prove the foundation before authoring any second page: run the stardust `diff` skill (both probes) against its prototype, **plus computed-style invariants in a headless render** — grid containers compute `display: grid` (not stacked single-column), sections are full-bleed where the design says so, and the CTA/button classes are actually styled (per `stardust/runtime-contract.json`, `skills/deploy/SKILL.md` § Runtime-detection probe). `deployed` means after PUT + preview: the gate's probes run against the page's preview URL, never a local harness — a pre-deploy harness pixel diff is NOT this gate. A wrong runtime assumption (block wrapper class, button classes) is silent and sitewide — typography still looks fine while every grid stacks. This one gate is the difference between fixing one page and rebuilding every template. **Archetype first, per template (#126):** the template's archetype is deployed and passes its full gate row on the preview URL (`gate-all.mjs --only`, cap 3 fix rounds) before any of its siblings is rendered or converted — the recorded unit `C-archetype` (handoff contract § 3, row C). **Execution model: waves.** Deliver in waves of parallel **author-only** agents — each agent curls its source pages and writes files only, never deploys or edits blocks — template clusters concurrently (non-overlapping pages), representative-first so blocks exist to be reused; then a **central deploy** per page; then background batches with a per-page OK/FAIL ledger, re-driving FAILs only. For clusters of 6–20+ siblings, the full flow is `reference/delivery-gates.md` § Batched delivery (it also names the one variant where cluster agents deploy themselves — replica's Phase 5 fan-out, per-cluster deploy ledgers, lock-safe shared ledgers). The central deploy step should run the bundled, resumable driver rather than a serial loop: `node skills/deploy/scripts/deploy-batch.mjs --org <org> --repo <repo> --branch <branch> --content <dir>` (concurrency pool, persistent ledger that skips already-live pages, retry/backoff, append-only log, delivered-`.plain.html` check). The driver and every batch run in the background; its log and ledger are the progress file. Check them at most every 4 minutes and never with a fixed `sleep` of 5 minutes or more (the prompt-cache window) — the master skill's wait discipline; recorded batch waits of 9–10 minutes re-wrote a ~650k prefix each time. After a transient blip, re-run the same command — it re-drives only the FAILs. Then reconcile the ledger into coverage with `update-coverage.mjs`. **Foundation first:** lint AND commit `styles.css`, header/footer CSS and the contract file BEFORE the first agent spawns; no token or custom-property rename after fan-out (a recorded rename under three running agents cost seven coordination messages). **Disjoint clusters spawn concurrently** — a wave waits only on a real dependency (a recorded second wave idled 14 min behind an unrelated first). **Every tool call stays under 4 minutes, the main agent's included** (the prompt cache holds 5; calls of 5.2 and 6.6 min re-wrote the whole context, and so did one 338 s foreground turn on the main agent that ran a pixel loop beside a deploy-batch start + wait): a long instrument (gate rounds, pixel loops, Playwright captures, deploy batches) goes through `run-bg.mjs start`, and `wait` is the NEXT tool call — it returns within its `--max` (100 s by default); never two long instruments as parallel tool calls in one turn, never two `wait`s in one command. ### Phase D — Site assembly (whole-site artifacts) ```bash node skills/rollout/scripts/assemble.mjs # → rollout/site/{sitemap.xml,robots.txt,manifest.json} — the EXPECTED set node skills/rollout/scripts/assemble.mjs --verify-origin https://<branch>--<repo>--<owner>.aem.live # vs the SERVED /sitemap.xml; exit 1 on a mismatch ``` Generates site-wide artifacts: `sitemap.xml` + `robots.txt` from delivered paths, and a fragments manifest mapping chrome blocks to the authored chrome documents (`content/nav.html`, `content/footer.html`) with their `canon/*.html` source (`deploy` authors + deploys the documents through the normal content chain — they MUST be published or the chrome 404s sitewide). **The assembled sitemap is never the served one.** `stardust/rollout/` is in `.hlxignore`, so `site/sitemap.xml` is a local artifact — the EXPECTED url set. The platform serves its own `/sitemap.xml`, built from its index of every published document, chrome documents included unless each carries `Robots |