cro-build

name	cro-build
description	Build and ship an A/B test experiment from a prioritized hypothesis at 50/50 split. Triggered by bud_server webhook when a Roadmap entry's Status changes to 3-building.

CRO Build & Ship

Configuration

Read .warp/skills/cro/references/config.md for Notion IDs, data access, and status labels. Read .warp/skills/cro/references/config-slack.md for Slack posting config (token, channels, Thread TS protocol).

Skill-specific config:

• Stats library: .warp/skills/cro/build/scripts/experiment.py
• AB framework: your-marketing-site/src/lib/ab/ (types.ts, experiments/index.ts, evaluator.ts, signals.ts)
• Middleware: your-marketing-site/src/middleware.ts
• AB testing reference: your-marketing-site/docs/ab-testing-and-tracking.md — dual-source experiments (code + Sanity), evaluation flow, cookie scheme, file locations
• Page content: Sanity page documents are the runtime source of truth. Markdown files at your-marketing-site/src/content/pages/<slug>.md are versioned fallback artifacts only and should be mirrored after Sanity query-back. Vercel previews and production fail closed unless SANITY_PAGE_MARKDOWN_FALLBACK_ENABLED=true is explicitly set.
• Default branch: main. GitHub repo: your-org/your-marketing-site.

Input

bud_server passes these fields from the Notion webhook:

NOTION_PAGE_ID: <hypothesis-notion-page-id>
EXPERIMENT_DESC: Hero copy test — AI messaging vs terminal speed
PAGE_PATH: /example-page
CHANGE_TYPE: copy
CHANGE_DESCRIPTION: Change hero headline from X to Y

Procedure

Step 0: Fetch hypothesis and page context

Fetch the full hypothesis from Notion using NOTION_PAGE_ID. Extract all properties: Hypothesis, Change Description, Evidence, RICE scores, Page, Change Type.

SEO-only scope guard (fail-fast, silent). Experiment Bud is CRO-only. Read references/config.md → Experiment Scope and inspect Change Type, Hypothesis, Change Description, and Evidence. If the entry is clearly SEO-only, set Status → 0-archived, log Skipping SEO-only hypothesis {NOTION_PAGE_ID}: out of Experiment Bud scope. to the agent session, and exit without creating a PR, Sanity document, dashboard, or Slack post.

Fetch the Page Registry entry for this page to get:

• Primary Metric and Secondary Metric(s) — use for stats planning
• Daily Traffic — use for duration estimation
• Owner — used to resolve the Slack {owner_mention} for Step 7 (see references/config.md → Page Owners). If Owner is empty, the build PR Slack post goes out without an @-mention.

Step 1: Create statistical experiment plan

Get baseline metrics from BigQuery using the Primary Metric from the Page Registry:

SELECT
  COUNT(*) AS pageviews,
  COUNT(*) / COUNT(DISTINCT pageview_date) AS daily_pageviews,
  ROUND(SAFE_DIVIDE(COUNTIF({{PRIMARY_METRIC}} = TRUE), COUNT(*)) * 100, 2) AS primary_cvr_pct
FROM `your-gcp-project.analytics.website_conversion`
WHERE path = '{{PAGE_PATH}}'
  AND pageview_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)

If Primary Metric is empty, do not post to Slack. Instead:

1. Set the Page Registry entry's Research Status to needs-primary-metric (the human-facing signal on the registry — see page_registry_sync Step 7):

   curl -s -X PATCH "https://api.notion.com/v1/pages/{{PAGE_REGISTRY_ENTRY_ID}}" \
     -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \
     -H "Notion-Version: 2022-06-28" \
     -H "Content-Type: application/json" \
     -d '{"properties": {"Research Status": {"select": {"name": "needs-primary-metric"}}}}'
   

2. Revert the backlog entry back to 2-prioritized so backlog_monitor does not immediately re-dispatch this build; the hypothesis will naturally be picked up again once the Primary Metric is filled in and the page's Research Status moves off needs-primary-metric.
3. Exit silently. No Slack.

Choose methodology using the 7-day rule:

python3 .warp/skills/cro/build/scripts/experiment.py duration \
  --baseline-cvr {{BASELINE_CVR}} \
  --mde 0.05 \
  --daily-traffic {{DAILY_PAGEVIEWS}}

• estimated_days ≤ 7 → frequentist (fixed sample, p-value)
• estimated_days > 7 → Bayesian (7-day timebox, early stopping)

Store the chosen Method in the Notion entry.

Document the plan:

• Hypothesis, method, primary metric, variant split (50/50)
• For frequentist: target sample size, estimated duration, significance threshold (95%)
• For Bayesian: max 7 days, ship threshold (P ≥ 90% + uplift ≥ 5%), kill threshold (P ≤ 20%), min 200 visitors/variant before any decision

Step 2: Create Sanity-first variant page

Marketing-3 runtime content comes from Sanity. Markdown under src/content/pages/ is a fallback and versioned artifact only; it is not the canonical creation source for experiments. Build the variant by cloning the control Sanity page document into the development dataset first, patching only intended structured fields, then mirroring markdown after query-back.

If the marketing repo does not yet expose the guardrail scripts below, stop and report the dependency on your-org/your-marketing-site#324 (or its successor) instead of falling back to the old markdown-first flow.

1. Fetch latest marketing main and create the experiment branch in the your-marketing-site repo or a dedicated worktree:

   git fetch origin main
   git checkout -b experiment/{{EXPERIMENT_ID}} origin/main
   

2. Choose the variant route/slug using the next available suffix for the control route (for example, /example-page-v2). Check both Sanity development documents and src/content/pages/ so the route is unique.

3. Clone the control page in Sanity development using the guardrail CLI. This command copies the current Sanity control document, applies dev-sync protection, keeps media references intact, and writes only to development:

   npm run experiment:create-sanity-variant -- --control-route={{PAGE_PATH}} --variant-route={{VARIANT_PATH}} --title="{{VARIANT_TITLE}}" --apply
   

   Record the command output in the PR body as Sanity clone evidence, including the source document ID, variant document ID, dataset, protected document IDs, and whether the command was a dry run or apply.

4. Patch only the intended structured Sanity fields in the development variant document. Use exact Sanity paths such as blocks[_key=="hero"].headline; do not edit the legacy data JSON blob as the only copy of important content. Keep all non-targeted fields identical to control so the experiment is clean.

5. Set and verify seoNoindex: true on the development variant page. Variants are experiment artifacts and must not be crawlable.

6. Run control-vs-variant parity/media checks with an explicit allowlist for every intended content delta:

   npm run experiment:variant-parity -- --control-route={{PAGE_PATH}} --variant-route={{VARIANT_PATH}} --allow-path='{{INTENDED_SANITY_PATH}}'
   

   Add one --allow-path per intended delta. Treat unallowlisted content drift, missing Sanity image refs, raw /img/... fallbacks, missing bento/carousel media, or control/variant structural drift as blockers.

7. Sync assets only if parity reports missing assets. If asset sync is needed, run the development-safe asset sync, then rerun parity with the same allowlist:

   npm run sanity:sync-assets -- --apply
   npm run experiment:variant-parity -- --control-route={{PAGE_PATH}} --variant-route={{VARIANT_PATH}} --allow-path=
   

Do not touch the control page's markdown file or Sanity document during build. The middleware does an internal rewrite — control content stays at the original URL.

Step 3: Add experiment config

Experiments use a per-file pattern to avoid merge conflicts. Each experiment lives in its own file under your-marketing-site/src/lib/ab/experiments/.

1. Create a new experiment file at src/lib/ab/experiments/<experiment-id>.experiment.ts:

   import type { Experiment } from '@/lib/ab/types'
   
   export const experiment: Experiment = {
     id: '<experiment-id>',
     path: '<page-path>',
     targeting: [],
     variants: [
       { id: 'control', url: '<page-path>', weight: 50 },
       { id: '<descriptive-id>', url: '<variant-path>', weight: 50 },
     ],
   }
   

   • id: experiment ID matching the hypothesis (kebab-case slug)
   • path: the page path the experiment intercepts (e.g., /example-page)
   • targeting: [] (all traffic) unless the hypothesis specifies otherwise
   • variants: 50/50 split, two entries. Note the field name is url (not pagePath). All variant weights must sum to 100.

2. Regenerate the experiments index instead of editing src/lib/ab/experiments/index.ts by hand:

   cd your-marketing-site && npm run generate:experiments
   

   The generated index imports every *.experiment.ts file and keeps the array sorted, which avoids merge conflicts across concurrent experiment PRs.

3. Do not rely on npm run sanity:sync-experiments during build. The code-defined experiment in the PR is enough for local dev and Vercel preview routing. If a Studio-visible experiment document is needed later, create/sync it only after confirming the sync script supports the current per-file experiment layout, and never create an active production Sanity experiment before merge.

Step 4: Validate the build

From your-marketing-site/:

1. npm run lint — must pass, or note existing warnings separately from new errors.
2. npm run build — runs TypeScript checks, build-gating Vitest suites, and next build. Must pass with no errors.
3. npm run experiment:variant-parity -- --control-route={{PAGE_PATH}} --variant-route={{VARIANT_PATH}} --allow-path='{{INTENDED_SANITY_PATH}}' — must pass after the intended Sanity edits. If asset sync was needed, include both the before-sync failure and after-sync pass in the PR body.
4. npm run experiment:preview-check -- --runtime-dataset=development — must pass before the PR is marked ready for human review.
5. Verify the experiment config: weights sum to 100, IDs are unique across the generated index and experiment file, every variants[].url corresponds to the control route or the Sanity-backed variant route.
6. Manually open the variant URL on the local dev server (npm run dev) to confirm it renders without errors.
7. Verify the Vercel preview can render both the control and variant URL with runtime dataset development when the launch preview is configured for staged content. A missing development Sanity page or missing media reference is build-blocking.

Step 5: Create Metabase dashboard

python3 .warp/skills/cro/experiment_monitor/scripts/create_dashboard.py \
  --experiment-id {{EXPERIMENT_ID}} \
  --page-path {{PAGE_PATH}} \
  --experiment-title "{{EXPERIMENT_DESC}}" \
  --days {{ESTIMATED_DURATION_DAYS}}

Store the dashboard URL in the Notion entry's Dash field. The dashboard won't have data yet but will populate once the experiment goes live.

Step 6: Open PR, assign reviewer, and update Notion

1. Create branch: experiment/{{EXPERIMENT_ID}} in your-marketing-site.

2. Commit with message: [Experiment] {{EXPERIMENT_DESC}} and include Co-Authored-By: Oz <[e-mail]>.

3. Write the PR body following the template in .warp/skills/cro/build/pr-body-template.md. The body MUST contain all five human-review sections in order: Hypothesis, Change, Plan, How to verify the changes, How to make further edits. It must also include Sanity clone evidence, intended delta paths, parity/media output, asset sync status, development Sanity query-back, preview readiness output, files changed, links, human production promotion notes, Conversation/Run links, the Oz attribution line, and Co-Authored-By: Oz <[e-mail]>. Write the completed body to a temp file (e.g. /tmp/pr-body.md).

4. Open a draft PR until all guardrail evidence is present and a human decides it is ready for review:

   gh pr create --repo your-org/your-marketing-site --base main \
     --head "experiment/{{EXPERIMENT_ID}}" \
     --title "[Experiment] {{EXPERIMENT_DESC}}" \
     --body-file /tmp/pr-body.md \
     --draft
   

   Capture the PR number from the output (e.g. https://github.com/your-org/your-marketing-site/pull/NNN → NNN).

5. Assign the page Owner as PR reviewer after the draft has the required Sanity evidence. Look up the Owner's GitHub username from owners.py. If the owner has no GitHub account, skip this step silently (same pattern as Slack mentions).

   GH_USER=$(python3 .warp/skills/cro/references/owners.py --github "{{OWNER}}")
   if [ -n "$GH_USER" ]; then
     gh pr edit {{PR_NUMBER}} --repo your-org/your-marketing-site --add-reviewer "$GH_USER"
   fi
   

6. Production promotion is human-reviewed. The PR body must explain the required human production steps, including reviewing the Sanity diff/release workflow, promoting only the intended variant page/content through the approved production path, verifying , and revalidating affected routes. Do not run production seed scripts, do not overwrite production Sanity from markdown, do not create an active production Sanity experiment before the code deploy is ready, and do not enable auto-merge on behalf of the human reviewer.

Step 7: Post to Slack

Post a new top-level message to #your-notifications-channel. Do NOT reuse any existing Slack Thread TS value from the Notion entry — the build skill always starts a fresh thread for the experiment. After posting, write the returned ts back to the entry's Slack Thread TS field so subsequent skills (monitor, analyze) can reply to it.

Token: use ONLY $EXPERIMENT_BUD_SLACK_BOT_TOKEN. If unset, abort and do not post — do NOT fall back to $SLACK_BOT_TOKEN, $OTHER_APP_*, or any other Slack credential.

import json, os, sys, urllib.request

token = os.environ.get('EXPERIMENT_BUD_SLACK_BOT_TOKEN', '').strip()
if not token:
    print("FATAL: EXPERIMENT_BUD_SLACK_BOT_TOKEN is unset. Refusing to post.", file=sys.stderr)
    sys.exit(1)

payload = {'channel': 'SLACK_NOTIFICATIONS_CHANNEL_ID', 'text': message_text}
# Build skill always posts a new top-level message — do NOT set thread_ts here.

req = urllib.request.Request(
    'https://slack.com/api/chat.postMessage',
    data=json.dumps(payload).encode('utf-8'),
    headers={'Authorization': f'Bearer {token}', 'Content-Type': 'application/json; charset=utf-8'},
)
with urllib.request.urlopen(req) as resp:
    result = json.loads(resp.read().decode('utf-8'))
    if not result.get('ok'):
        print(f"Slack post failed: {result.get('error')}", file=sys.stderr)
        sys.exit(1)

Message template (omit the final line entirely if Owner is empty):

🧪 Experiment ready for review — {experiment_desc}
{page_path} · {change_type} · est. {duration} days · 50/50 split
📊 <{dashboard_url}|Dashboard> · 🔗 <{pr_url}|PR ready for review>
{owner_mention}

Important Notes

• 50/50 from the start. Experiments go live at 50/50 only after the code deploy and required human production content promotion are complete.
• CRITICAL: Never push directly to main on your-org/your-marketing-site. All changes go through PRs.
• Variant pages MUST set seoNoindex: true in structured Sanity fields and in the markdown fallback artifact. Variants are experiment artifacts, not crawlable pages.
• Sanity is the runtime source of truth. Clone and edit the variant in the development dataset first. Markdown is a fallback/versioned artifact only, created after Sanity query-back.
• Agents must not write production Sanity. Production launch requires human review and the approved production promotion workflow. Do not seed production from markdown, do not use broad production overwrite flags, and do not create an active production Sanity experiment before the deployment/content sequence is approved.
• Use the Sanity-first guardrail CLI contract. Required marketing commands are npm run experiment:create-sanity-variant, npm run experiment:variant-parity, and npm run experiment:preview-check -- --runtime-dataset=development. If those scripts are not on main, report the dependency instead of reverting to a markdown-first workaround.
• Field name is url, not pagePath. The marketing-3 Variant schema uses id, url, weight. Don't carry over old field names from your-legacy-marketing-site.
• Experiments run on ALL traffic (paid + organic). The winning variant ships to everyone.
• SEO-only changes are out of scope. Do not build experiments for title tags, meta descriptions, canonical tags, robots/noindex, sitemap, structured data, schema.org, JSON-LD, crawler routing, keyword targeting, SERP snippets, or indexing/crawlability work.
• The experiment_monitor skill handles everything post-merge: validation, running checks, conclusion detection.