cro-build

# CRO Build & Ship ## Configuration Read `.warp/skills/cro/references/config.md` for Notion IDs, data access, and status labels. Read `.warp/skills/cro/references/config-slack.md` for Slack posting config (token, channels, Thread TS protocol). Skill-specific config: - **Stats library**: `.warp/skills/cro/build/scripts/experiment.py` - **AB framework**: `your-marketing-site/src/lib/ab/` (`types.ts`, `experiments/index.ts`, `evaluator.ts`, `signals.ts`) - **Middleware**: `your-marketing-site/src/middleware.ts` - **AB testing reference**: `your-marketing-site/docs/ab-testing-and-tracking.md` — dual-source experiments (code + Sanity), evaluation flow, cookie scheme, file locations - **Page content**: Sanity `page` documents are the runtime source of truth. Markdown files at `your-marketing-site/src/content/pages/<slug>.md` are versioned fallback artifacts only and should be mirrored after Sanity query-back. Vercel previews and production fail closed unless `SANITY_PAGE_MARKDOWN_FALLBACK_ENABLED=true` is explicitly set. - **Default branch**: `main`. GitHub repo: `your-org/your-marketing-site`. ## Input bud_server passes these fields from the Notion webhook: ``` NOTION_PAGE_ID: <hypothesis-notion-page-id> EXPERIMENT_DESC: Hero copy test — AI messaging vs terminal speed PAGE_PATH: /example-page CHANGE_TYPE: copy CHANGE_DESCRIPTION: Change hero headline from X to Y ``` ## Procedure ### Step 0: Fetch hypothesis and page context Fetch the full hypothesis from Notion using `NOTION_PAGE_ID`. Extract all properties: Hypothesis, Change Description, Evidence, RICE scores, Page, Change Type. **SEO-only scope guard (fail-fast, silent).** Experiment Bud is CRO-only. Read `references/config.md` → Experiment Scope and inspect Change Type, Hypothesis, Change Description, and Evidence. If the entry is clearly SEO-only, set Status → `0-archived`, log `Skipping SEO-only hypothesis {NOTION_PAGE_ID}: out of Experiment Bud scope.` to the agent session, and exit without creating a PR, Sanity document, dashboard, or Slack post. Fetch the **Page Registry** entry for this page to get: - **Primary Metric** and **Secondary Metric(s)** — use for stats planning - **Daily Traffic** — use for duration estimation - **Owner** — used to resolve the Slack `{owner_mention}` for Step 7 (see `references/config.md` → Page Owners). If `Owner` is empty, the build PR Slack post goes out without an @-mention. ### Step 1: Create statistical experiment plan **Get baseline metrics** from BigQuery using the Primary Metric from the Page Registry: ```sql SELECT COUNT(*) AS pageviews, COUNT(*) / COUNT(DISTINCT pageview_date) AS daily_pageviews, ROUND(SAFE_DIVIDE(COUNTIF({{PRIMARY_METRIC}} = TRUE), COUNT(*)) * 100, 2) AS primary_cvr_pct FROM `your-gcp-project.analytics.website_conversion` WHERE path = '{{PAGE_PATH}}' AND pageview_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) ``` If `Primary Metric` is empty, **do not post to Slack**. Instead: 1. Set the Page Registry entry's `Research Status` to `needs-primary-metric` (the human-facing signal on the registry — see `page_registry_sync` Step 7): ```bash curl -s -X PATCH "https://api.notion.com/v1/pages/{{PAGE_REGISTRY_ENTRY_ID}}" \ -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \ -H "Notion-Version: 2022-06-28" \ -H "Content-Type: application/json" \ -d '{"properties": {"Research Status": {"select": {"name": "needs-primary-metric"}}}}' ``` 2. Revert the backlog entry back to `2-prioritized` so backlog_monitor does not immediately re-dispatch this build; the hypothesis will naturally be picked up again once the Primary Metric is filled in and the page's Research Status moves off `needs-primary-metric`. 3. Exit silently. No Slack. **Choose methodology** using the 7-day rule: ```bash python3 .warp/skills/cro/build/scripts/experiment.py duration \ --baseline-cvr {{BASELINE_CVR}} \ --mde 0.05 \ --daily-traffic {{DAILY_PAGEVIEWS}} ``` - `estimated_days` ≤ 7 → **frequentist** (fixed sample, p-value) - `estimated_days` > 7 → **Bayesian** (7-day timebox, early stopping) Store the chosen `Method` in the Notion entry. **Document the plan:** - Hypothesis, method, primary metric, variant split (50/50) - For frequentist: target sample size, estimated duration, significance threshold (95%) - For Bayesian: max 7 days, ship threshold (P ≥ 90% + uplift ≥ 5%), kill threshold (P ≤ 20%), min 200 visitors/variant before any decision ### Step 2: Create Sanity-first variant page Marketing-3 runtime content comes from Sanity. Markdown under `src/content/pages/` is a fallback and versioned artifact only; it is not the canonical creation source for experiments. Build the variant by cloning the control Sanity `page` document into the `development` dataset first, patching only intended structured fields, then mirroring markdown after query-back. If the marketing repo does not yet expose the guardrail scripts below, stop and report the dependency on `your-org/your-marketing-site#324` (or its successor) instead of falling back to the old markdown-first flow. 1. **Fetch latest marketing main and create the experiment branch** in the `your-marketing-site` repo or a dedicated worktree: ```bash git fetch origin main git checkout -b experiment/{{EXPERIMENT_ID}} origin/main ``` 2. **Choose the variant route/slug** using the next available suffix for the control route (for example, `/example-page-v2`). Check both Sanity development documents and `src/content/pages/` so the route is unique. 3. **Clone the control page in Sanity development** using the guardrail CLI. This command copies the current Sanity control document, applies dev-sync protection, keeps media references intact, and writes only to `development`: ```bash npm run experiment:create-sanity-variant -- --control-route={{PAGE_PATH}} --variant-route={{VARIANT_PATH}} --title="{{VARIANT_TITLE}}" --apply ``` Record the command output in the PR body as Sanity clone evidence, including the source document ID, variant document ID, dataset, protected document IDs, and whether the command was a dry run or apply. 4. **Patch only the intended structured Sanity fields** in the development variant document. Use exact Sanity paths such as `blocks[_key=="hero"].headline`; do not edit the legacy `data` JSON blob as the only copy of important content. Keep all non-targeted fields identical to control so the experiment is clean. 5. **Set and verify `seoNoindex: true`** on the development variant page. Variants are experiment artifacts and must not be crawlable. 6. **Run control-vs-variant parity/media checks with an explicit allowlist** for every intended content delta: ```bash npm run experiment:variant-parity -- --control-route={{PAGE_PATH}} --variant-route={{VARIANT_PATH}} --allow-path='{{INTENDED_SANITY_PATH}}' ``` Add one `--allow-path` per intended delta. Treat unallowlisted content drift, missing Sanity image refs, raw `/img/...` fallbacks, missing bento/carousel media, or control/variant structural drift as blockers. 7. **Sync assets only if parity reports missing assets.** If asset sync is needed, run the development-safe asset sync, then rerun parity with the same allowlist: ```bash npm run sanity:sync-assets -- --apply npm run experiment:variant-parity -- --control-route={{PAGE_PATH}} --variant-route={{VARIANT_PATH}} --allow-path='{{INTENDED_SANITY_PATH}}' ``` If parity reports no missing assets, do not run asset sync; record `not needed` in the PR body. 8. **Query back the development variant document** and verify the intended fields, `seoNoindex: true`, and image references. Never print token values; load the token into an environment variable and use it only in the request header. 9. **Mirror markdown only after Sanity query-back.** Update or create `src/content/pages/{{VARIANT_SLUG}}.md` as a fallback/versioned artifact that matches the queried Sanity variant. Do not use markdown to overwrite Sanity or treat it as canonical. Do not touch the control page's markdown file or Sanity document during build. The middleware does an internal rewrite — control content stays at the original URL. ### Step 3: Add experiment config Experiments use a **per-file pattern** to avoid merge conflicts. Each experiment lives in its own file under `your-marketing-site/src/lib/ab/experiments/`. 1. **Create a new experiment file** at `src/lib/ab/experiments/<experiment-id>.experiment.ts`: ```ts import type { Experiment } from '@/lib/ab/types' export const experiment: Experiment = { id: '<experiment-id>', path: '<page-path>', targeting: [], variants: [ { id: 'control', url: '<page-path>', weight: 50 }, { id: '<descriptive-id>', url: '<variant-path>', weight: 50 }, ], } ``` - `id`: experiment ID matching the hypothesis (kebab-case slug) - `path`: the page path the experiment intercepts (e.g., `/example-page`) - `targeting`: `[]` (all traffic) unless the hypothesis specifies otherwise - `variants`: **50/50 split**, two entries. Note the field name is `url` (not `pagePath`). All variant weights must sum to 100. 2. **Regenerate the experiments index** instead of editing `src/lib/ab/experiments/index.ts` by hand: ```bash cd your-marketing-site && npm run generate:experiments ``` The generated index imports every `*.experiment.ts` file and keeps the array sorted, which avoids merge conflicts across concurrent experiment PRs. 3. **Do not rely on `npm run sanity:sync-experiments` during build.** The code-defined experiment in the PR is enough for local dev and Vercel preview routing. If a Studio-visible experiment document is needed later, create/sync it only after confirming the sync script supports the current per-file experiment layout, and never create an active production Sanity experiment before merge. ### Step 4: Validate the build From `your-marketing-site/`: 1. `npm run lint` — must pass, or note existing warnings separately from new errors. 2. `npm run build` — runs TypeScript checks, build-gating Vitest suites, and `next build`. Must pass with no errors. 3. `npm run experiment:variant-parity -- --control-route={{PAGE_PATH}} --variant-route={{VARIANT_PATH}} --allow-path='{{INTENDED_SANITY_PATH}}'` — must pass after the intended Sanity edits. If asset sync was needed, include both the before-sync failure and after-sync pass in the PR body. 4. `npm run experiment:preview-check -- --runtime-dataset=development` — must pass before the PR is marked ready for human review. 5. Verify the experiment config: weights sum to 100, IDs are unique across the generated index and experiment file, every `variants[].url` corresponds to the control route or the Sanity-backed variant route. 6. Manually open the variant URL on the local dev server (`npm run dev`) to confirm it renders without errors. 7. Verify the Vercel preview can render both the control and variant URL with runtime dataset `development` when the launch preview is configured for staged content. A missing development Sanity page or missing media reference is build-blocking. ### Step 5: Create Metabase dashboard ```bash python3 .warp/skills/cro/experiment_monitor/scripts/create_dashboard.py \ --experiment-id {{EXPERIMENT_ID}} \ --page-path {{PAGE_PATH}} \ --experiment-title "{{EXPERIMENT_DESC}}" \ --days {{ESTIMATED_DURATION_DAYS}} ``` Store the dashboard URL in the Notion entry's `Dash` field. The dashboard won't have data yet but will populate once the experiment goes live. ### Step 6: Open PR, assign reviewer, and update Notion 1. Create branch: `experiment/{{EXPERIMENT_ID}}` in `your-marketing-site`. 2. Commit with message: `[Experiment] {{EXPERIMENT_DESC}}` and include `Co-Authored-By: Oz <oz-agent@warp.dev>`. 3. **Write the PR body** following the template in `.warp/skills/cro/build/pr-body-template.md`. The body MUST contain all five human-review sections in order: **Hypothesis**, **Change**, **Plan**, **How to verify the changes**, **How to make further edits**. It must also include Sanity clone evidence, intended delta paths, parity/media output, asset sync status, development Sanity query-back, preview readiness output, files changed, links, human production promotion notes, Conversation/Run links, the Oz attribution line, and `Co-Authored-By: Oz <oz-agent@warp.dev>`. Write the completed body to a temp file (e.g. `/tmp/pr-body.md`). 4. **Open a draft PR** until all guardrail evidence is present and a human decides it is ready for review: ```bash gh pr create --repo your-org/your-marketing-site --base main \ --head "experiment/{{EXPERIMENT_ID}}" \ --title "[Experiment] {{EXPERIMENT_DESC}}" \ --body-file /tmp/pr-body.md \ --draft ``` Capture the PR number from the output (e.g. `https://github.com/your-org/your-marketing-site/pull/NNN` → `NNN`). 5. **Assign the page Owner as PR reviewer** after the draft has the required Sanity evidence. Look up the Owner's GitHub username from `owners.py`. If the owner has no GitHub account, skip this step silently (same pattern as Slack mentions). ```bash GH_USER=$(python3 .warp/skills/cro/references/owners.py --github "{{OWNER}}") if [ -n "$GH_USER" ]; then gh pr edit {{PR_NUMBER}} --repo your-org/your-marketing-site --add-reviewer "$GH_USER" fi ``` 6. **Do not write to production Sanity.** Production promotion is human-reviewed. The PR body must explain the required human production steps, including reviewing the Sanity diff/release workflow, promoting only the intended variant page/content through the approved production path, verifying `seoNoindex: true`, and revalidating affected routes. Do not run production seed scripts, do not overwrite production Sanity from markdown, do not create an active production Sanity experiment before the code deploy is ready, and do not enable auto-merge on behalf of the human reviewer. 7. **Update Notion:** - Status → `4-ready for review` only after the draft PR contains the required guardrail evidence - `Build PR` → PR URL - `Start Date` → today - `Dash` → dashboard URL ### Step 7: Post to Slack Post a **new top-level message** to `#your-notifications-channel`. Do NOT reuse any existing `Slack Thread TS` value from the Notion entry — the build skill always starts a fresh thread for the experiment. After posting, write the returned `ts` back to the entry's `Slack Thread TS` field so subsequent skills (monitor, analyze) can reply to it. **Token: use ONLY `$EXPERIMENT_BUD_SLACK_BOT_TOKEN`.** If unset, abort and do not post — do NOT fall back to `$SLACK_BOT_TOKEN`, `$OTHER_APP_*`, or any other Slack credential. ```python import json, os, sys, urllib.request token = os.environ.get('EXPERIMENT_BUD_SLACK_BOT_TOKEN', '').strip() if not token: print("FATAL: EXPERIMENT_BUD_SLACK_BOT_TOKEN is unset. Refusing to post.", file=sys.stderr) sys.exit(1) payload = {'channel': 'SLACK_NOTIFICATIONS_CHANNEL_ID', 'text': message_text} # Build skill always posts a new top-level message — do NOT set thread_ts here. req = urllib.request.Request( 'https://slack.com/api/chat.postMessage', data=json.dumps(payload).encode('utf-8'),