cro-research

name	cro-research
description	Research website pages and generate A/B test hypotheses. Runs when triggered or on-demand for a single page. Discovers traffic sources, analyzes on-page behavior and visual design, then writes findings and hypotheses back to Notion.

CRO Research & Hypothesize

Configuration

Read .warp/skills/cro/references/config.md for shared CRO settings (data access, Notion database IDs, secrets).

Skill-specific config:

• Query templates: .warp/skills/cro/research/scripts/research_queries.sql
• Priors: .warp/skills/cro/references/priors.yaml
• Team Experiment Principles (Notion): NOTION_TEAM_PRINCIPLES_PAGE_ID — rubric for hypothesis synthesis and Confidence nudging.
• Team Experiment Backlog (Notion): NOTION_TEAM_BACKLOG_PAGE_ID — specific tb-* ideas the team has queued; don't re-propose duplicates.
• Default lookback: 30 days

Procedure

Step 0: Read inputs and page context

The task input will include:

PAGE_PATH: /pricing
RESEARCH_STATUS: never-researched

• PAGE_PATH (required): The page to research. If missing, stop and report the error.
• RESEARCH_STATUS: Either never-researched or generate-more-hypotheses or examine-hit-rate.
  • never-researched: Full research — all steps below.
  • generate-more-hypotheses: Page was researched before. Run all data-gathering steps, but focus synthesis on net-new hypotheses. Compare findings to existing Knowledge Base insights and past reports, and call out what's changed.
  • ** examine-hit-rate**: Page was researched before, but the hit rate on experiments so far is low. Review past experiment findings and focus synthesis on net-new hypotheses in different areas.

Variant-slug guard (fail-fast, silent). If PAGE_PATH matches the A/B variant pattern -v<digits>$ (e.g. /example-page-v2, /example-page-v3), this invocation was triggered in error — variant pages are experiment artifacts, not pages that should ever be researched. Do NOT fetch the Page Registry for analysis, do NOT do any research. Instead:

1. If an entry exists in Page Registry for this path, archive it silently:

   curl -s -X PATCH "https://api.notion.com/v1/pages/{{PAGE_REGISTRY_ENTRY_ID}}" \
     -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \
     -H "Notion-Version: 2022-06-28" \
     -H "Content-Type: application/json" \
     -d '{"archived": true}'
   

2. Log the early exit to the agent session output and return.

Fetch the Page Registry entry for this page:

curl -s -X POST "https://api.notion.com/v1/databases/NOTION_PAGE_REGISTRY_DB_ID/query" \
  -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \
  -H "Notion-Version: 2022-06-28" \
  -H "Content-Type: application/json" \
  -d '{"filter": {"property": "Page Path", "title": {"equals": "{{PAGE_PATH}}"}}}'

CRO Excluded guard (fail-fast, silent). If the entry exists and CRO Excluded = true, exit silently — do not do any work. A human has opted this page out of CRO; respect that without additional noise.

SEO-only scope guard. Experiment Bud is CRO-only. Read references/config.md → Experiment Scope before synthesizing hypotheses. You may use SEO/organic query data as evidence of visitor intent, but do not create Roadmap & Backlog hypotheses for SEO-only changes such as title tags, meta descriptions, canonical tags, robots/noindex, sitemap, structured data, schema.org, JSON-LD, keyword targeting, SERP snippets, crawler routing, or indexing/crawlability work. If you find an SEO-only opportunity, write it as a Knowledge Base insight and do not add it to the experiment backlog.

Low-traffic guard (fail-fast, silent). Check the Daily Traffic value from the Page Registry entry. If Daily Traffic < 50, exit silently. Do not research, do not write any outputs. Log the early exit to the agent session output: "Skipping research on {PAGE_PATH}: Daily Traffic ({traffic}) below minimum threshold (50)."

Extract:

• Primary Metric and Secondary Metric(s) — use these as the conversion metrics throughout the analysis. If Primary Metric is empty, set the Page Registry entry's Research Status to needs-primary-metric and exit silently. The status becomes the human-facing signal — anyone reviewing the Page Registry will see which pages are blocked on a missing metric.

  curl -s -X PATCH "https://api.notion.com/v1/pages/{{PAGE_REGISTRY_ENTRY_ID}}" \
    -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \
    -H "Notion-Version: 2022-06-28" \
    -H "Content-Type: application/json" \
    -d '{"properties": {"Research Status": {"select": {"name": "needs-primary-metric"}}}}'
  

• Daily Traffic, CVR — for context on primary metric performance over last 7 days.

Step 1: On-page behavior analysis

Run queries B1 through B7 from research_queries.sql, substituting {{LP_PATH}} with the page path and {{DAYS}} with 30.

Use the Primary Metric from Step 0 as the main conversion metric when interpreting results. The B queries use did_download_same_day and did_on_page_download_cta_click — if the Page Registry specifies a different primary metric, adapt the queries accordingly.

Analyze:

• B1 (funnel): pageview → CTA click → conversion. Where's the biggest drop-off?
• B2 (paid vs organic): Do paid and organic traffic convert at different rates? A large gap suggests mismatched ad-to-LP messaging.
• B3 (OS/device): Any segment with significantly lower conversion?
• B4 (CTA locations): Which CTAs get the most clicks? Hero vs header vs nav?
• B5 (LP benchmark): How does this LP compare to others?
• B6 (path analysis): Where do visitors go after the LP? High homepage % = visitors didn't find what they needed.
• B7 (geo): Any countries with significantly different conversion?

Step 2: Visual audit

Screenshot the page using computer_use:

1. Navigate to https://www.example.com{{PAGE_PATH}}
2. Screenshot at desktop (~1440px) and mobile (~390px), scrolling through all sections

Analyze:

• Intent-headline match: Does the hero match what visitors expect based on how they arrive?
• CTA prominence: Primary CTAs visible above fold?
• Value prop clarity: Clear in under 5 seconds?
• Feature relevance: Do the sections address visitor needs?
• Mobile experience: CTAs clear and tappable? Focus here in particular if mobile is a sizeable % of page traffic.

Step 3: Traffic source evaluation

Determine where traffic primarily comes from by breaking down visitors by UTM parameters:

SELECT
  COALESCE(utm_source, '(direct)') AS source,
  COALESCE(utm_medium, '(none)') AS medium,
  utm_campaign AS campaign,
  channel,
  COUNT(*) AS pageviews,
  ROUND(SAFE_DIVIDE(COUNT(*), SUM(COUNT(*)) OVER()) * 100, 1) AS pct_of_total
FROM `your-gcp-project.analytics.website_conversion`
WHERE path = '{{LP_PATH}}'
  AND pageview_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
GROUP BY source, medium, campaign, channel
ORDER BY pageviews DESC
LIMIT 20;

Only investigate sources that are a sizable portion of traffic (roughly >10% of total or >50 pageviews in the lookback). For each significant source, proceed to Step 4.

Step 4: Source-specific deep dive

Run the appropriate analysis for each significant traffic source identified in Step 3. Use the on-page behavior (Step 1) and visual audit (Step 2) findings to inform your analysis of ad↔page alignment.

4a. Paid SEM

Campaign & keyword analysis:

1. Run query A0 from research_queries.sql to discover which Google Ads campaigns drive traffic.
2. For each campaign, run A1 (daily trends), A2 (keyword performance), A3 (actual search queries).
3. Run F1 to get responsive search ad headlines and descriptions.

Analyze:

• Are top search queries aligned with the LP headline? If not → intent-message mismatch.
• High-volume + low-CVR keywords → messaging gap opportunity.
• Best-performing ad headlines vs LP hero copy — any ad↔LP messaging gap?
• Intent signals in queries ("best", "free", "vs", "alternative to") the LP doesn't address?

Ad copy hypotheses are valid even though they're implemented in Google Ads, not on the LP. Tag with change_type: ad-copy and note they require human implementation.

Organic search comparison: Run queries D1 and D2 from research_queries.sql using the LP slug.

• How do organic queries differ from paid keywords?
• If organic converts better (from B2), are organic queries more aligned with LP messaging?
• High-volume organic queries the paid LP doesn't address?

Keyword discovery: Use the Google Ads Keyword Planner API (credentials in config.md) with top keywords from A2 as seeds:

ACCESS_TOKEN=$(curl -s -X POST "https://oauth2.googleapis.com/token" \
  -d "client_id=$GOOGLE_ADS_CLIENT_ID" \
  -d "client_secret=$GOOGLE_ADS_CLIENT_SECRET" \
  -d "refresh_token=$GOOGLE_ADS_REFRESH_TOKEN" \
  -d "grant_type=refresh_token" | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")

curl -s -X POST "https://googleads.googleapis.com/v20/customers/$GOOGLE_ADS_CUSTOMER_ID:generateKeywordIdeas" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "developer-token: $GOOGLE_ADS_DEVELOPER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "keywordSeed": {"keywords": ["<top keywords from A2>"]},
    "language": "languageConstants/1000",
    "geoTargetConstants": ["geoTargetConstants/2840"],
    "keywordPlanNetwork": "GOOGLE_SEARCH"
  }'

Look for:

• High-volume keywords with strong intent we don't bid on
• Keywords suggesting messaging angles we haven't considered
• Low-competition keywords with meaningful volume

Tag keyword hypotheses with change_type: keyword-opportunity and flag for human action.

4b. Paid Social (and other non-SEM paid channels)

Use ad_dimensions to look up ad creative details. This unified dbt model covers Meta, Google, LinkedIn, and Reddit — one row per ad with UTMs, headlines, body, and CTA.

Match UTMs from the traffic source breakdown (Step 3) to ads:

SELECT
  ad_id, ad_name, platform, campaign_name,
  headline, body, call_to_action, creative_type,
  utm_source, utm_medium, utm_campaign,
  destination_url, ad_format, optimization_goal,
  persona_function, persona_level, geo_countries
FROM `your-gcp-project.analytics.ad_dimensions`
WHERE utm_source = '{{UTM_SOURCE}}'
  AND utm_medium = '{{UTM_MEDIUM}}'
  AND utm_campaign = '{{UTM_CAMPAIGN}}'
  AND status = 'ACTIVE'
ORDER BY campaign_name;

Analyze:

• What headlines and creative themes are used in the ads driving traffic here?
• Do the ad messages match the LP hero/value prop?
• What audience targeting (persona, geo) is used — does the LP address that audience's needs?
• For video/social ads: does the creative set expectations the LP doesn't meet?

If creative data isn't available for certain ads (e.g., headline is null for LinkedIn/Reddit), note it as a data gap and move on.

Tag ad-copy hypotheses with change_type: ad-copy and note they require human implementation.

4c. SEO/Organic

Run queries D1 and D2 from research_queries.sql using the LP slug.

• What queries drive organic traffic?
• Are high-volume queries aligned with page messaging?
• CRO-safe content or copy opportunities based on search intent?

Do not propose SEO-only experiments from this section. Organic search findings may support landing-page conversion tests (headline, CTA, section ordering, proof, feature emphasis), but title/meta/canonical/schema/sitemap/robots/indexing/keyword-targeting work must be recorded only as insights, not Roadmap & Backlog hypotheses.

4d. Other sources

For other campaign types: note what's available about creative/targeting and flag obvious ad↔LP mismatches. If creative data isn't accessible, note the gap and move on.

Step 5: SERP + competitor analysis

Only run this step if SEM or SEO are significant traffic sources from Step 3.

Using the top 3 search queries from Step 4a or 4c, use computer_use to:

1. Google each query and screenshot the SERP
2. Note what ads appear, AI overview content, and top organic results
3. Click the top 1-2 non-Warp results and screenshot their pages
4. For each competitor: note headline, CTA, page structure, key messaging

Analyze:

• What messaging themes do competitors use?
• How do competitor CTAs compare?
• Messaging angles competitors use that we don't address?

Include key screenshots in the Notion report.

Step 6: Synthesize hypotheses and RICE score

Before synthesizing, load team context from Notion:

1. Fetch the Team Experiment Principles page (NOTION_TEAM_PRINCIPLES_PAGE_ID). This is the team's working set of beliefs (P1–P23) about what makes a good experiment — opinions/priors, not gospel. Use them as a rubric.
2. Fetch the Team Experiment Backlog page (NOTION_TEAM_BACKLOG_PAGE_ID) to see which specific tb-* experiment ideas the team has already queued. Do not re-propose a hypothesis that already exists there; if your finding reinforces an existing idea, cite its tb-* ID in the Evidence field instead.

Cross-reference all data sources. Look for patterns like:

1. Intent-message mismatch: Top queries/ads suggest X, LP headline says Y
2. Ad↔LP messaging gap: Best-performing ad headlines emphasize different value props than the LP
3. Paid vs organic gap: Different conversion rates suggest mismatched messaging
4. CTA friction: Users click CTAs but don't convert
5. Segment underperformance: A specific OS/device/geo converts much lower
6. Feature irrelevance: Features shown don't address visitor needs
7. Below-fold waste: Path analysis shows visitors leaving without engaging
8. Untapped keyword opportunity: High-volume, high-intent keywords we don't target
9. Competitor messaging advantage/inspiration: Competitors use messaging/CTAs that we lack or have not thought of
10. Team-principle application: A finding that maps onto one or more principles (e.g. duplicate CTAs → P1; generic H1 → P6; buried differentiator → P9).

These are examples, not an exhaustive checklist. Use your judgment to identify any other opportunities the data reveals. Be creative.

For each hypothesis, RICE score it. Load priors from .warp/skills/cro/references/priors.yaml.

• Reach (0-1): Fraction of visitors affected. Hero ≈ 1.0, below-fold ≈ 0.3-0.6.
• Impact (1-3): Expected lift. Use change_type_priors as baseline.
• Confidence (0-1): Start from avg_success_rate in priors. Adjust based on evidence strength. Nudge +0.05 to +0.15 when the hypothesis maps cleanly onto a high-conviction principle (marked with 🔥 on the Principles Notion page); record the nudge in Evidence.
• Effort (1-5): 1=copy or CTA change, 2=minor UI or layout change, 3=new frontend element, 5=structural/backend.

RICE score = (Reach × Impact × Confidence) / Effort. Sort descending.

Always cite specific numbers in hypothesis evidence. Never make vague claims. When a hypothesis aligns with a team principle, explicitly name it in Evidence (e.g. "Aligns with P3 — CTA topic should match page topic"). When a hypothesis contradicts a principle, call it out so the human reviewer can weigh the challenge.

Step 7: Write outputs to Notion

Before writing, load existing context to avoid duplicates:

• Knowledge Base: Read existing insights for this page. If not the first analysis, compare to past insights and note what's changed.
• Roadmap & Backlog: Read existing hypotheses for this page. Don't re-propose duplicates. Fetch the database schema to resolve the live Status labels.

1. Create the report as a subpage under Reports (NOTION_REPORTS_PAGE_ID):

Title: YYYY-MM-DD {{PAGE_PATH}} Research Analysis

The report should contain:

• Executive summary: Bullets on current performance and biggest opportunities
• Deep Dive: Findings from each data source examined

curl -s -X POST "https://api.notion.com/v1/pages" \
  -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \
  -H "Notion-Version: 2022-06-28" \
  -H "Content-Type: application/json" \
  -d '{
    "parent": { "page_id": "NOTION_REPORTS_PAGE_ID" },
    "properties": {
      "title": [{ "text": { "content": "YYYY-MM-DD /page Research Analysis" } }]
    },
    "children": [
      ... report content as Notion blocks ...
    ]
  }'

Save the report page URL — it will be linked from all other database entries.

2. Create Analysis Log entry (NOTION_ANALYSIS_LOG_DB_ID):

• Title: same as report title
• Status: completed
• Date: today
• Pages Analyzed: {{PAGE_PATH}}
• Full Report: link to report subpage
• Key Takeaways: 2-3 sentence summary

3. Add insights to Knowledge Base (NOTION_KNOWLEDGE_BASE_DB_ID): For significant, durable findings. Each insight should:

• Have a specific, quantified title (always cite numbers)
• Link to the report via the Analysis Link field
• Have appropriate Tags and Insight Source
• Be tagged to {{PAGE_PATH}} via the Page field

4. Add hypotheses to Roadmap & Backlog (NOTION_ROADMAP_DB_ID): For each hypothesis, add a row with:

• Status: 1-proposed (use the exact label resolved from the database schema)
• All RICE fields: Reach, Impact, Confidence, Effort, RICE Score
• Page: {{PAGE_PATH}}
• Owner: copy the Owner select value from the Page Registry entry (fetched in Step 0). If the page has no Owner, leave this field empty.
• Hypothesis, Change Type, Change Description, Evidence
• Analysis Link: report URL
• Buildable: true

Before creating each row, apply the SEO-only scope guard from references/config.md. If the proposed change is SEO-only, do not create a Roadmap row; include it only in the report/Knowledge Base as an insight.

Tag hypotheses requiring non-LP changes (ad copy, keyword targeting) with the appropriate change_type and note they require human implementation.

5. Update Page Registry (NOTION_PAGE_REGISTRY_DB_ID):

• Set Last Researched to today
• Set Research Status to current
• Set Latest Report to the report URL

6. Print the full report to the agent session output as well.