| name | experiment-analyzer |
| description | Analyze completed growth experiment results, validate hypotheses, generate insights, and suggest follow-up experiments. Use when experiments are completed, when the user asks about results or learnings, or when discussing what to do next based on experiment outcomes. |
| allowed-tools | ["Read","Write","Grep","Glob"] |
Experiment Analyzer Skill
Analyze completed growth experiments, extract insights, and drive continuous learning.
When to Activate
This skill should activate when:
- User marks experiment as "completed"
- User asks "what did we learn?"
- User mentions "results", "outcomes", or "analysis"
- User asks "what should we do next?"
- User wants to compare multiple experiments
- User asks about experiment success rates
Analysis Framework
1. Result Classification
Win (Positive + Significant)
- Result is better than baseline
- Statistical significance ≥ 95%
- Change is meaningful (usually ≥5%)
Loss (Negative + Significant)
- Result is worse than baseline
- Statistical significance ≥ 95%
- Change is meaningful
Inconclusive
- Statistical significance < 95%
- Not enough data to make decision
- Sample size may be insufficient
Neutral
- Minimal change (< ±2%)
- No meaningful impact either way
- May indicate hypothesis was off
2. Hypothesis Validation
Compare original hypothesis to results:
Hypothesis Components:
- Proposed change → Was it implemented as planned?
- Target audience → Did we reach the right users?
- Expected outcome → Did we hit the target?
- Rationale → Was our reasoning correct?
Validation Questions:
- Did we achieve the expected outcome? (Yes/No/Partially)
- Was the underlying assumption correct?
- What surprised us?
- What would we do differently?
3. ICE Score Retrospective
Compare predicted vs actual:
Impact Score Validation:
- Predicted Impact: [original score]
- Actual Impact: [calculate based on results]
- Delta: [difference]
- Learning: Was our impact prediction accurate?
Confidence Score Validation:
- Predicted Confidence: [original score]
- Outcome: [win/loss/inconclusive]
- Learning: Was our confidence justified?
Ease Score Validation:
- Predicted Ease: [original score]
- Actual Time: [if tracked]
- Learning: Was implementation as easy as expected?
4. Insight Generation
Key Questions:
- What worked? Specific elements that drove success
- What didn't work? Elements that failed or harmed metrics
- What was surprising? Unexpected findings
- What patterns emerge? Connections to other experiments
- What new questions arise? Areas to investigate further
Secondary Metrics:
- Review all secondary metrics tracked
- Look for unintended positive effects
- Watch for negative side effects
- Consider holistic impact
5. Follow-up Experiment Suggestions
Based on the outcome, suggest 2-3 follow-up experiments:
For Wins:
- Scale: Roll out to 100% of users
- Amplify: Make the winning element more prominent
- Extend: Apply pattern to related areas
- Optimize: Test variations to improve further
For Losses:
- Pivot: Try alternative approach to same problem
- Investigate: Run research to understand why
- Revert: Document and move on
- Learn: Apply learnings to future experiments
For Inconclusive:
- Re-run: Increase sample size or duration
- Simplify: Test smaller version to isolate variable
- Segment: Test with specific user segments
- Refine: Adjust hypothesis based on early signals
Analysis Process
Step 1: Load and Validate
1. Read experiment JSON from completed/archived folder
2. Verify results data exists:
- Primary metric
- Baseline value
- Result value
- Statistical significance
- Sample size
- Duration
3. Check if hypothesis is documented
4. Review ICE scores
Step 2: Calculate Key Metrics
Change Percentage = ((Result - Baseline) / Baseline) × 100
Result Classification:
- IF change% > 2% AND significance >= 95% → Win
- IF change% < -2% AND significance >= 95% → Loss
- IF significance < 95% → Inconclusive
- IF abs(change%) < 2% → Neutral
Step 3: Generate Insights
1. Classify result (Win/Loss/Inconclusive/Neutral)
2. Validate hypothesis against results
3. Review ICE score predictions
4. Extract key learnings
5. Identify surprising findings
6. Check secondary metrics
7. Look for patterns across related experiments
Step 4: Create Follow-up Ideas
1. Based on result type, brainstorm 2-3 follow-ups
2. For each follow-up:
- Draft hypothesis
- Explain rationale (reference current learnings)
- Suggest category
- Provide preliminary ICE estimate
3. Prioritize follow-ups by potential impact
Step 5: Generate Report
1. Create markdown analysis report
2. Include:
- Summary (result classification, key numbers)
- Hypothesis validation
- ICE score retrospective
- Key insights (bulleted list)
- Secondary metrics review
- Recommendations
- Follow-up experiment ideas
3. Save to experiments/archive/[id]_analysis.md
4. Update experiment JSON with learnings
Analysis Output Template
# Experiment Analysis: [Title]
**Date:** [Analysis date]
**Experiment ID:** [id]
**Status:** [Win/Loss/Inconclusive/Neutral] ✓/✗/?/○
## Summary
- **Primary Metric:** [metric name]
- **Baseline:** [baseline value]
- **Result:** [result value]
- **Change:** [+/-X%]
- **Statistical Significance:** [XX%]
- **Sample Size:** [count]
- **Duration:** [days]
## Hypothesis Validation
### Original Hypothesis
[Full hypothesis statement]
### Validation
- **Expected Outcome:** [what we expected]
- **Actual Outcome:** [what happened]
- **Hypothesis Validated:** [Yes/No/Partially]
**Analysis:**
[Explanation of whether and why hypothesis was validated]
## ICE Score Retrospective
| Component | Predicted | Actual/Assessment | Accuracy |
|-----------|-----------|------------------|----------|
| Impact | [score] | [calculate from results] | [good/overestimated/underestimated] |
| Confidence | [score] | [based on outcome] | [justified/overconfident/underconfident] |
| Ease | [score] | [based on actual effort] | [accurate/harder/easier] |
**Learnings for Future Scoring:**
- [What we learned about predicting impact]
- [What we learned about confidence]
- [What we learned about ease]
## Key Insights
1. **[Primary insight]** - [Explanation with data]
2. **[Secondary insight]** - [Explanation]
3. **[Surprising finding]** - [What we didn't expect]
## Secondary Metrics
| Metric | Change | Interpretation |
|--------|--------|----------------|
| [metric 1] | [+/-X%] | [Good/Bad/Neutral] |
| [metric 2] | [+/-X%] | [Good/Bad/Neutral] |
**Side Effects:**
- Positive: [Any unexpected positive impacts]
- Negative: [Any unexpected negative impacts]
## Recommendations
### Immediate Actions
- [ ] [Action item 1]
- [ ] [Action item 2]
### Strategic Implications
[Broader implications for product/growth strategy]
## Follow-up Experiment Ideas
### 1. [Experiment Title]
**Category:** [category]
**Hypothesis:**
[Full hypothesis following template]
**Rationale:**
[Why this follow-up based on current learnings]
**Preliminary ICE:**
- Impact: [score] - [reasoning]
- Confidence: [score] - [reasoning]
- Ease: [score] - [reasoning]
- **Total: [score]**
---
### 2. [Experiment Title]
[Repeat format]
---
### 3. [Experiment Title]
[Repeat format]
## Related Experiments
[List any related experiments and their outcomes for pattern recognition]
## Notes
[Any additional context, edge cases, or considerations]
Cross-Experiment Analysis
When user asks to analyze multiple experiments:
Metrics to Calculate:
- Success Rate: % of wins out of completed experiments
- Category Performance: Which funnel stages have best win rate?
- ICE Score Accuracy: How well do high-ICE experiments perform?
- Average Impact: What's the typical metric improvement?
- Cycle Time: Average days from backlog → completed
Pattern Recognition:
- Which types of experiments succeed most?
- Which audience segments respond best?
- Which testing methods are most reliable?
- What confidence levels actually predict success?
Portfolio View:
# Experiment Portfolio Analysis
## Overview
- Total Experiments: [count]
- Completed: [count]
- Win Rate: [X%]
- Average Change: [+X%]
## By Category
| Category | Experiments | Win Rate | Avg Impact |
|----------|-------------|----------|------------|
| Acquisition | [count] | [X%] | [+X%] |
| Activation | [count] | [X%] | [+X%] |
| Retention | [count] | [X%] | [+X%] |
| Revenue | [count] | [X%] | [+X%] |
| Referral | [count] | [X%] | [+X%] |
## ICE Score Performance
- Experiments with ICE > 500: [X% win rate]
- Experiments with ICE 300-500: [X% win rate]
- Experiments with ICE < 300: [X% win rate]
**Learning:** [Are high ICE scores actually better predictors?]
## Top Performers
1. [Experiment] - [+X%] change
2. [Experiment] - [+X%] change
3. [Experiment] - [+X%] change
## Key Patterns
- [Pattern 1 discovered across experiments]
- [Pattern 2]
- [Pattern 3]
## Recommendations
[Strategic recommendations based on portfolio analysis]
Integration Points
- Automatically trigger when
/experiment-update sets status to "completed"
- Work with ICE scorer skill to validate predictions
- Inform hypothesis generator with learnings
- Feed into metrics calculator for portfolio analysis
Continuous Improvement
After each analysis:
- Store learnings in a knowledge base
- Update ICE scoring calibration
- Refine hypothesis templates
- Build pattern library
- Improve follow-up suggestions
