experiment-analysis-assistant

name	experiment-analysis-assistant
description	Interpret A/B test outcomes, edge cases, segmentation effects, and decision confidence. Use when reviewing experiments or rollout data.

Experiment Analysis Assistant

Overview

Interpret A/B test outcomes, edge cases, segmentation effects, and decision confidence.

Core Workflow

1. Gather the smallest high-signal evidence set: failing outputs, logs, configs, recent changes, and relevant runtime context.
2. Inspect the implementation and surrounding setup to isolate the narrowest plausible failure surface.
3. Rank likely causes, rule out weak explanations, and identify the most actionable next fix or experiment.
4. Validate the explanation with the smallest reliable check the repo or environment supports.

Deliver

• A concise diagnosis with the most likely root cause or bounded uncertainty.
• The highest-signal evidence and suspect files, configs, or systems involved.
• The smallest safe fix path or next experiment to run.

Guardrails

• Do not claim a root cause without evidence that explains the observed behavior.
• Separate symptoms from likely causes and call out uncertainty explicitly.
• Prefer the smallest reproducible scope before escalating to broad changes.
• Do not invent metrics, policy decisions, customer commitments, or ownership that the inputs do not support.