experiment-platform

name	experiment-platform
description	Build an internal experimentation platform for running A/B tests at scale. Outputs experiment service architecture, assignment engine, metric pipeline, and statistical analysis framework.
argument-hint	["traffic volume","number of concurrent experiments","team size","existing analytics stack"]
allowed-tools	Read, Write

Experiment Platform

An experiment platform standardises how A/B tests are run across the organisation — experiment definition, user assignment, metric collection, and statistical analysis. Without a platform, every team implements its own experiment logic inconsistently. With one, experiments are faster, more reliable, and more trustworthy.

Architecture

Experiment Service (define + assign)
  ├── Experiment Registry (what experiments exist, who is in them)
  ├── Assignment Engine (deterministic bucketing by user_id)
  └── Exposure Logging (record when user saw the experiment)

Metric Pipeline
  ├── Event collection (existing analytics pipeline)
  ├── Metric computation (joins exposure log with events)
  └── Results store (pre-computed stats per experiment)

Analysis Service
  ├── Statistical tests (t-test, z-test, sequential)
  ├── Segment breakdowns
  └── Results API (for dashboard)

Assignment Engine

import hashlib
from dataclasses import dataclass

@dataclass
class Experiment:
    id: str
    name: str
    variants: list[dict]  # [{"name": "control", "weight": 50}, {"name": "treatment", "weight": 50}]
    status: str           # "active" | "paused" | "concluded"
    targeting: dict       # Who qualifies: {"plans": ["pro", "enterprise"], "countries": ["US"]}

class AssignmentEngine:
    def assign(self, user_id: str, experiment: Experiment) -> str | None:
        """
        Deterministic, sticky assignment.
        Same user always gets same variant.
        Returns variant name or None if user not in experiment.
        """
        if experiment.status != "active":
            return None

        # Hash user_id + experiment_id for consistent bucketing
        hash_input = f"{user_id}:{experiment.id}"
        bucket = int(hashlib.md5(hash_input.encode()).hexdigest(), 16) % 100

        # Walk through variants by weight
        cumulative = 0
        for variant in experiment.variants:
            cumulative += variant["weight"]
            if bucket < cumulative:
                return variant["name"]

        return None  # User not bucketed (weights sum < 100)

    def is_eligible(self, user: dict, experiment: Experiment) -> bool:
        """Check targeting rules before assignment."""
        targeting = experiment.targeting
        if "plans" in targeting and user.get("plan") not in targeting["plans"]:
            return False
        if "countries" in targeting and user.get("country") not in targeting["countries"]:
            return False
        return True

Results Computation (SQL)

-- Compute experiment results
WITH exposures AS (
    SELECT DISTINCT
        user_id,
        experiment_id,
        variant_name,
        MIN(exposed_at) AS first_exposed_at
    FROM experiment_exposures
    WHERE experiment_id = 'exp-checkout-v2'
      AND exposed_at >= '2024-03-01'
    GROUP BY user_id, experiment_id, variant_name
),

conversions AS (
    SELECT
        e.user_id,
        e.variant_name,
        COUNT(o.order_id) AS orders,
        SUM(o.total_amount) AS revenue
    FROM exposures e
    LEFT JOIN orders o ON e.user_id = o.customer_id
        AND o.created_at > e.first_exposed_at
        AND o.created_at <= e.first_exposed_at + INTERVAL '7 days'
    GROUP BY e.user_id, e.variant_name
)

SELECT
    variant_name,
    COUNT(*) AS users,
    AVG(orders > 0) AS conversion_rate,
    AVG(revenue) AS avg_revenue_per_user,
    STDDEV(revenue) AS revenue_stddev
FROM conversions
  variant_name;

Anti-Patterns to Avoid

Anti-Pattern	Problem	Fix
Multiple experiment systems	Inconsistent methodology; conflicting results	Single platform, shared assignment engine
Non-deterministic assignment	Users switch variants between sessions	Hash-based deterministic bucketing
Novelty effect ignored	First-week lift disappears	Require 2+ week run; check week-over-week consistency
Too many concurrent experiments	Interaction effects corrupt results	Mutual exclusion or layered experiment design
Peeking and stopping early	Inflated false positives	Pre-registered sample size; sequential testing

10 Rules

1. Assignment is deterministic — same user always gets same variant.
2. Exposure is logged at the moment of assignment — not at conversion.
3. Analysis uses only users who were exposed — not all users.
4. Experiments run for minimum 2 weeks — novelty effects bias early results.
5. Sample size is calculated upfront — experiments are not stopped early.
6. Primary metric is pre-registered — changing metrics post-hoc is p-hacking.
7. Guardrail metrics are checked — a conversion win that harms NPS is not a win.
8. Segment results by major dimensions — aggregates hide important patterns.
9. Mutual exclusion prevents experiments from interacting unless explicitly layered.
10. The platform is owned — ungoverned experiment proliferation creates false results.

Deep dive: applying this in practice

The sections above describe what to produce. This section describes how practitioners actually run this in the field, including the conversations, artefacts, and review loops that turn a one-page recommendation into a sustained outcome.

The 30/60/90 cadence

A recommendation that is never revisited is a recommendation that quietly fails. Bake review checkpoints in from day one:

• Day 0 — Decision committed. Owner, scope, success metrics, and the first-checkpoint date are recorded in the decision log. The artefact is linked from the team's working space so it is discoverable without asking.
• Day 30 — Early-signal review. Look at the leading indicators, not the lagging ones. Has the team actually started? Are the assumed dependencies real? Have any of the named risks materialised? Adjust scope, not the goal.
• Day 60 — Course-correction window. This is the last cheap moment to change direction. If the leading indicators are flat or negative, escalate. Silence at day 60 is the most expensive form of optimism.
• Day 90 — Outcome review. Measure against the success criteria captured on day 0, not against the story the team is telling now. Write the post-mortem (or pre-mortem-confirmed) in the same artefact so the rationale, the outcome, and the lessons live together.

Stakeholder choreography

Decisions stall not because the analysis is wrong but because the choreography is wrong. Use a lightweight RACI on every recommendation:

Role	Meaning	Anti-pattern
Responsible	Does the work	More than two people listed
Accountable	Owns the outcome, signs off	Shared accountability (always becomes no accountability)
Consulted	Two-way input before the decision	Consulted after the decision is made — purely performative
Informed	One-way notification after the decision	Informed people are asked to approve — wastes their time and yours

If you cannot name a single Accountable person in one minute, the recommendation is not ready to ship.

Writing for senior readers

Senior readers scan first, read second, and only re-read the parts they disagree with. Optimise for that pattern:

1. Lead with the recommendation, not the analysis. The reader should know what you want them to do before they finish the first paragraph.
2. One screen, one page, one decision. If the artefact needs scrolling on a laptop, it is too long for the audience it is written for.
3. Tables beat paragraphs for comparing options. Prose hides the trade-off; a table forces it into the open.
4. Numbers beat adjectives. Replace "significant" with the actual number. Replace "soon" with a date. Replace "improved" with a baseline and a target.
5. Name the disconfirming evidence. A recommendation that lists what would change the author's mind is read as honest; one that does not is read as advocacy.

Common failure modes

Failure mode	Symptom	Counter-move
Analysis paralysis	Weeks of investigation, no decision	Time-box the analysis. State the decision quality you can defend in the time available.
HiPPO override	Highest-paid person's opinion wins regardless of evidence	Force the trade-off table into the room before opinions are voiced
Sunk-cost gravity	Team defends the current path because of prior investment	Re-frame: what would we choose today with no prior investment?
Scope creep at the checkpoint	Review becomes a re-planning session	Separate "did this work?" from "what next?" Run them as two meetings.
Stealth de-scoping	Success metrics quietly soften between day 0 and day 90	Lock the day-0 metrics into the artefact; require an explicit amendment to change them.
Owner drift	Accountable person leaves, no one re-assigns	Owner reassignment is a mandatory step in onboarding/offboarding the role

A worked example

A product line is debating whether to invest in a major rewrite of a legacy service that has been failing under peak load.

A weak response: "We should rewrite it because the code is old."

A response that uses this skill:

Recommendation. Do not rewrite. Invest one quarter in targeted performance work on the existing service and a parallel strangler-fig migration of the top two failing endpoints. Confidence: medium. Would change my mind if peak-load incidents continue at the current rate for two consecutive months after the performance work ships.

Options considered. (1) Full rewrite — 9–12 months, ~$1.4M, high risk of partial delivery. (2) Performance fix in place — 6 weeks, ~$120K, addresses 80% of incident volume per last-quarter analysis. (3) Strangler-fig migration — 6 months for the two hottest endpoints, ~$400K, preserves optionality.

Plan. Owner: Platform tech lead. Day 30: performance fix in staging with load test results. Day 60: production rollout and a 30-day incident-rate comparison. Day 90: decision on whether to expand the strangler-fig scope.

Risks. (1) Performance fix masks a deeper architectural issue — mitigated by capturing flame graphs before and after. (2) Strangler-fig endpoints are not in fact the hottest ones — mitigated by re-running the traffic analysis at day 0. (3) Team capacity collides with a separate compliance deadline — escalated to the portfolio review on the next planning cycle.

That is the shape of output this skill should produce: a defensible, time-bound, owner-attached recommendation that respects the reader's time and survives turnover.

Quick reference card

• One paragraph of context, three options with trade-offs, one recommendation with confidence, one plan with an owner and a date.
• If you cannot name the owner, the metric, and the checkpoint date in one breath, the artefact is not done.
• A decision without a written rationale is a rumour. A rationale without a checkpoint is a wish. A checkpoint without a metric is theatre.
• Reversibility matters more than people admit: one-way doors deserve the slow lane, two-way doors deserve the fast lane.
• The best artefacts in this category are short, dated, signed, and easy to find six months later.