experiment-design

name	experiment-design
description	Test spec + metric + gate 조건 + 기간 + variant 정의. data-analyst 의 Confidence Score 모델 입력 가설을 받아 manifest.yaml 생성.
schema_version	2
tier	leader
team	_skill
category	planning
used_by	["product-manager"]
dev_capability	false
triggers	{"keyword":["실험","experiment","a/b test","ab test","test design"]}
pm_conventions	{"anti_sycophancy":true,"hard_gate":true,"post_labeling":true,"minimum_approaches":2}

Experiment Design Skill

Hypothesis-design 의 후속. recommended hypothesis 를 검증 가능한 실험으로 변환.

입력

• recommended hypothesis (XYZ format) from skills/hypothesis-design
• team OKR (어떤 KR 에 영향?)

산출 — <org>/experiments/<id>/manifest.yaml

experiment_id: exp-YYYY-MM-DD-<slug>
hypothesis:
  xyz: "[X%] of [Y] will [Z] within [T], because [R]"
  source_pr: "..."

variants:
  - id: control
    description: "current behavior"
    traffic_pct: 50
  - id: treatment
    description: "..."
    traffic_pct: 50

metrics:
  primary:
    name: "..."
    formula: "..."
    direction: increase | decrease
    threshold: "..."
  secondary: [...]
  guardrail: [...]

design:
  type: ab_test | feature_flag | concierge | wizard_of_oz | fake_door
  duration_days: 14
  min_sample: 1000
  power: 0.8
  alpha: 0.05

gates:
  start:
    - "instrumentation verified"
    - "guardrail metrics baselines stable"
  stop_early:
    - "guardrail breach"
    - "primary metric Δ p<0.01 (positive)"
  end:
    - "min_sample reached"
    - "duration reached"

evidence_refs:
  - "<org>/workflows/.../PRD.md"
  - "<org>/memory/open-questions/<task>.json"

≥ 2 Designs Rule

experiment design 도 항상 ≥2 후보:

• AB test vs concierge MVP
• feature flag vs cohort
• 다른 traffic split

가장 cheap 한 valid design 우선 권고.

HARD GATE

- [ ] primary metric 1개 + threshold + window
- [ ] sample size 충분성 (power=0.8, α=0.05)
- [ ] guardrail metric ≥1 (regression 방지)
- [ ] start/stop/end gate 모두 정의
- [ ] product-designer 검토 (사용자 segment 차별 시)

Reference

• phuryn/pm-skills/pm-execution/brainstorm-experiments
• v1.1 PRD §12.3 (Experiment 인프라)