| name | product-experiment |
| description | Design the smallest reliable experiment or assumption test capable of changing a product decision. Use when the user has a hypothesis, wants an A/B test or prototype test, needs causal evidence, must choose an evidence standard, or wants success, guardrail, stop, and interpretation rules. |
Product Experiment
Reduce a named uncertainty with evidence strong enough for the decision and no stronger or costlier than necessary.
Inputs
- Decision and decision owner
- Hypothesis and causal mechanism
- Target population
- Current evidence and baseline
- Expected effect and minimum meaningful effect
- Traffic, access, time, ethics, and instrumentation constraints
Workflow
- Express the assumption as a falsifiable claim tied to a decision.
- Classify the uncertainty: desirability, usability, feasibility, viability, causal impact, or AI quality.
- Choose the evidence strength based on reversibility and cost of error.
- Select a method: interview or artifact review, concierge test, prototype, fake door, usability test, technical spike, pricing offer, controlled experiment, staged rollout, or observational analysis.
- Define primary outcome, guardrails, population, exposure, and analysis window.
- For randomized tests, specify assignment unit, control, treatment, contamination risks, sample-ratio checks, and stopping rule.
- Precommit success, failure, inconclusive, and harm interpretations.
- Audit instrumentation and operational readiness before exposure.
- Define the next decision for every possible result.
Output contract
Return the decision, hypothesis, mechanism, method and rationale, participants or sample, primary and guardrail metrics, test procedure, bias and instrumentation risks, duration or stopping rule, interpretation table, and next actions.
Quality gate
- The test can disconfirm the claim.
- The method matches the uncertainty.
- Practical importance is separate from statistical significance.
- Peeking, novelty, selection, and instrumentation risks are addressed.
- An inconclusive result has an explicit next step.
Avoid
- Building the complete product as the first test
- Using an A/B test with insufficient traffic or unstable assignment
- Changing the hypothesis after seeing results
- Declaring success from one positive metric while guardrails worsen
- Treating absence of significance as proof of no effect
Source grounding
Operational synthesis informed by assumption testing in Continuous Discovery Habits, prototype and risk-reduction practices in Inspired, and measurement discipline in Lean Analytics. Controlled-experiment safeguards are framework-informed; this release does not claim fidelity to Kohavi, Tang, and Xu's full text.
