| name | experiment-results |
| description | Closes an experiment: reads results against the original spec, renders a Win / No Signal / Loss / Confounded verdict, extracts the learning (why, not what), and specifies the next action. Required when any experiment window closes. Connects back to /growth-experiment for reformulated experiments, or forward to scaling decisions when the verdict is Win. |
| triggers | ["/experiment-results","experiment window has closed","user has data from a completed or in-progress growth experiment","need to decide whether to scale, iterate, or kill a test"] |
| role | workflow |
| version | 1.0.0 |
| sources | ["RICE / ICE prioritization frameworks","growth-experiment-results (growth-skills v1.0, score 8/10)"] |
| feeds | ["growth/growth-experiment (for reformulation on Loss or Confounded)"] |
Experiment Results
Before starting
Confirm before reading results:
- Original experiment spec — hypothesis, success metric, baseline, and window must be available. Reading results without a pre-registered spec is HARKing.
- Window status — has the full experiment window closed? If not, only directional reads are permitted.
- Confounder events — any outages, concurrent experiments, external events, or tracking changes during the window?
- Sample size — is the sample sufficient to detect the expected effect? (stated in spec; re-evaluate if actual sample differs significantly)
Contract
This skill guarantees:
- A BLOCK is issued if the original spec is unavailable — results are never read without a pre-registered hypothesis
- "No Signal ≠ Loss" rule is enforced in every ambiguous result
- Learning extraction answers WHY (mechanism), not just WHAT (outcome)
- Every analysis produces an archive entry — no experiment closes without a record
- Confounder scan is run before scoring any result
Role: Impartial Analyst. You read results against the original spec — not against hope or pessimism. You enforce the "No Signal ≠ Loss" rule. You extract learnings about mechanism, not just outcome. You require a next action before closing every analysis.
Inputs
Required before proceeding:
- Original experiment spec (hypothesis + success metric + baseline)
- Results data (metric outcomes for the measured window)
If original spec is unavailable:
BLOCK. Do not reconstruct. Return:
"The original experiment spec is not available. Reading results without a
pre-specified hypothesis is HARKing (Hypothesizing After Results are Known)
— it produces post-hoc justification, not learning.
A reconstructed spec is not a valid spec. Anyone who saw the results before
writing the hypothesis cannot produce an unbiased success criterion.
Action: locate the original spec document. If it cannot be found:
→ Archive this experiment as UNREADABLE
→ Log the hypothesis and metric used (if known) for future reference
→ Do NOT render a WIN/LOSS verdict
→ The only permitted output is: 'Results exist but cannot be evaluated
without the pre-registered spec. Archived as UNREADABLE.'"
Pre-read checks
Before scoring, apply these checks:
Check 1 — Window status
IF experiment window has NOT fully closed:
→ Directional read only. Label output clearly:
"DIRECTIONAL READ — window not closed. Do not make scaling decisions on this data."
IF window has closed (or user indicates it has):
→ Full verdict.
Check 2 — Confounder scan
Before reading the data, check for events that could have compromised the result:
| Confounder type | Signal |
|---|
| Platform outage | Traffic drop, increased errors, partial service during experiment window |
| Concurrent experiment | Another test on the same audience during the same window |
| External event | Competitor launch, industry news, seasonal effect |
| Measurement error | UTM break, tracking event misfired, analytics gap |
| Audience contamination | Test and control audiences overlapped |
IF confounder detected → note it before scoring.
A confounded result must be labeled Confounded, not Win or Loss.
Verdict framework
Four verdicts. Apply the one that matches, in order.
VERDICT 1 — WIN
Conditions:
Primary metric meets or exceeds the stated threshold.
No critical confounders that invalidate the measurement.
Sample size is sufficient to distinguish signal from noise (use practical judgment).
Example: "Hypothesis: ≥35% email open rate in 7 days vs 22% baseline.
Result: 41% open rate over 7 days across n=3,400 sends.
Verdict: WIN. Threshold met; sample is substantial."
---
VERDICT 2 — NO SIGNAL
Conditions:
Result is in the right direction but below threshold, OR
Sample size is insufficient to detect the effect with reasonable confidence, OR
Result is within noise range of the baseline.
CRITICAL RULE: No Signal ≠ Loss.
"We didn't prove it works" is not the same as "it doesn't work."
Before recording No Signal, state:
- What sample size or run time would be needed to detect the effect if it exists?
- Is the cost of running a larger experiment worth the information?
Example: "Hypothesis: ≥35% open rate. Result: 26% over 7 days on n=180 sends.
Verdict: NO SIGNAL. Direction is positive but sample too small to
distinguish from noise. Need ~500 sends to have confidence. Worth rerunning."
---
VERDICT 3 — LOSS
Conditions:
Primary metric is below threshold AND in the wrong direction (worse than baseline), OR
Primary metric is substantially below threshold with sufficient sample.
Example: "Hypothesis: ≥35% open rate. Result: 18% over 7 days on n=2,200 sends.
Verdict: LOSS. Significantly below baseline (22%). Moving in the wrong
direction with sufficient sample."
---
VERDICT 4 — CONFOUNDED
Conditions:
A confounder was identified (from Check 2) that makes the result uninterpretable.
Example: "Hypothesis: conversion rate improvement. Result: 52% drop in traffic
during day 4-6 of a 7-day window due to a platform outage.
Verdict: CONFOUNDED. Traffic drop invalidates conversion comparison.
Rerun with controls is required."
Learning extraction
Required. One paragraph. This is the most important output of the analysis.
Not: "The experiment didn't work." (That's a result, not a learning.)
Not: "Open rates increased." (That's what happened, not why.)
Yes: what do you now know about the mechanism that you did not know before?
Learning extraction must answer:
(a) WHY did the result happen? (mechanism, not just outcome)
(b) What does this change about the next experiment you'd run?
(c) What assumption was confirmed or invalidated?
Examples:
STRONG LEARNING:
"We learned that subject lines naming a specific framework (e.g. 'the dual-write
pattern') outperform general benefit claims (e.g. 'migrate without downtime').
This suggests our ICP is optimization-seeking (looking for a technique to steal)
rather than outcome-seeking. The next experiment should test a different technique
name framing."
WEAK LEARNING (not acceptable):
"We learned that subject line matters."
(Too vague. What about it matters? For which audience? How will you apply it?)
Next step by verdict
WIN:
→ Scale or extend.
→ Identify: what must be true to hold this result at 3× the audience/volume?
Are there resource constraints? Audience exhaustion risks? Quality degradation risks?
→ If repeatable: operationalize (add to playbook, document the winning tactic)
NO SIGNAL:
→ Option A: Rerun with larger sample, tighter audience, or longer window — if
cost of certainty < value of the information.
→ Option B: Kill if the cost of running a definitive test exceeds the value of
knowing the answer. Not every question is worth answering precisely.
→ Return to /growth-experiment with updated scope.
LOSS:
→ Kill the specific tactic.
→ Decide: was the hypothesis wrong (wrong channel / wrong message / wrong audience)
or was the tactic wrong (right hypothesis, wrong execution)?
→ Return to /growth-experiment with the updated hypothesis.
CONFOUNDED:
→ Rerun with controls to prevent the confounder.
→ Note the confounder in the experiment archive so it is not overlooked next time.
→ If the confounder is structural (e.g. seasonal effect), adjust the experiment window.
Archive entry (required — close every analysis with this)
## Experiment Archive Entry
[Date] | [Experiment name]
Verdict: [Win / No Signal / Loss / Confounded]
One-line learning: [One sentence — what you know now that you didn't before]
Next action: [Specific action]
This entry feeds the experiment history that is referenced in future sprints. Without it, learnings are lost.
Output format
## Experiment Results
**Experiment:** [Name]
**Hypothesis:** [Original hypothesis — copy from spec]
**Success metric:** [Metric + threshold + window]
**Baseline:** [Pre-experiment value]
### Pre-read checks
Window status: [Closed / Open — directional only]
Confounders: [None detected / List with descriptions]
### Verdict
[WIN / NO SIGNAL / LOSS / CONFOUNDED]
Evidence:
Result: [Actual metric value]
vs Baseline: [Baseline value]
vs Threshold: [Threshold — was it met?]
Sample: [N = ?]
Directional: [Positive / Negative / Mixed]
### Learning
[One paragraph: why did this happen? What assumption was confirmed or invalidated?
What does this change about the next experiment?]
### Next action
[Specific action: scale / rerun with [change] / kill / rerun with controls]
[For No Signal: state the sample needed for a definitive answer]
### Archive entry
[Date] | [Experiment name] | Verdict: [verdict] | Learning: [one sentence] | Next: [action]
Connects to
Upstream: /growth-experiment — the spec this analysis reads against.
Downstream: /growth-experiment (Loop: Loss or Confounded → reformulate) or scaling/operationalization (Win).
Brain reads / writes
If a companion aether-growth-brain repo is connected:
Before reading:
- Read
experiments/experiment-log.md — locate the original experiment entry; confirm hypothesis and success metric as pre-registered
Brain write (required — close every analysis):
- Update the experiment entry in
experiments/experiment-log.md with:
verdict: WIN / NO-SIGNAL / LOSS / CONFOUNDEDone_line_learning: the mechanism learned (not the outcome)next_action: scale / rerun with [change] / kill / rerun with controlsdate_closed: today's date
- For WIN verdicts: also write to
playbooks/messaging.md if the winning tactic is a messaging insight worth operationalizing
Brain not connected: archive entry is the minimum equivalent; recommend maintaining a local experiment log.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|
| Reading results without original spec (HARKing) | Post-hoc justification is not learning; cannot distinguish "it worked" from "it happened to go up" | Block until original spec is located; archive as UNREADABLE if not found |
| Declaring Loss when the signal window was too short | A 7-day test on a 30-day metric is No Signal, not Loss | Check window against activation/retention cycle before rendering verdict |
| Declaring Win on insufficient sample | n=15 cannot be a Win | State sample needed for confidence; render No Signal if sample is inadequate |
| Scaling a Confounded result | A confounded Win might be a real Loss | Always rerun with controls before scaling |
| Learning "the experiment didn't work" | That's a result, not a learning | Learning must answer: WHY? What mechanism was confirmed or invalidated? |
| Not writing the archive entry | Learnings are lost; same experiments get re-run in later sprints | Archive entry is mandatory — it is the primary deliverable of the analysis |
| Treating concurrent experiment contamination as valid data | Two simultaneous tests on the same audience make both unreadable | Label as Confounded; rerun sequentially |
Related skills
| Skill | When to use |
|---|
growth/growth-experiment/SKILL.md | After LOSS or CONFOUNDED: reformulate the experiment |
growth/funnel-audit/SKILL.md | If LOSS verdict reveals the hypothesis was wrong: re-audit the funnel |
growth/retention-analysis/SKILL.md | If experiment targeted retention churn type: compare result against curve shape prediction |
Validation criteria
References & Sources
Tier 1:
- RICE / ICE prioritization: practical vs statistical significance framework
- growth-experiment-results (growth-skills v1.0, score 8/10): Win/No Signal/Loss/Confounded framework, learning extraction method, archive entry format
