ab-test-power-calc

name	ab-test-power-calc
description	Calculate sample size and runtime for an A/B test given variance and MDE.
argument-hint	["baseline","MDE","traffic","variance"]
allowed-tools	Read, Write, Bash

Ab Test Power Calc

When to use

Calculate sample size and runtime for an A/B test given variance and MDE.

Process

1. Clarify scope, constraints, and decision owners.
2. Gather current-state inputs and benchmark references.
3. Draft an initial design and surface trade-offs explicitly.
4. Pressure-test against failure modes and reversibility.
5. Define rollout, metrics, and rollback triggers.
6. Document the decision and update relevant runbooks.

Output template

• Context — why this matters now
• Decision — what we will do
• Trade-offs — what we are giving up
• Rollout plan — phased steps and gates
• Metrics & alerts — what success and failure look like
• Rollback — explicit reverse path

Anti-patterns

• Optimizing without a measurable target.
• Skipping a written record of the decision.
• Treating reversible bets like irreversible ones (or vice-versa).

Rules

1. Name the customer or system the change serves.
2. State assumptions before recommendations.
3. Quantify expected impact and confidence.
4. Pre-commit kill / rollback criteria.
5. Identify the single owner.
6. List dependencies and unblock paths.
7. Define observability before launch.
8. Plan the smallest reversible first step.
9. Schedule a follow-up review date.
10. Archive the artifact in the team's source of truth.

Deep dive: applying this in practice

The sections above describe what to produce. This section describes how practitioners actually run this in the field, including the conversations, artefacts, and review loops that turn a one-page recommendation into a sustained outcome.

The 30/60/90 cadence

A recommendation that is never revisited is a recommendation that quietly fails. Bake review checkpoints in from day one:

• Day 0 — Decision committed. Owner, scope, success metrics, and the first-checkpoint date are recorded in the decision log. The artefact is linked from the team's working space so it is discoverable without asking.
• Day 30 — Early-signal review. Look at the leading indicators, not the lagging ones. Has the team actually started? Are the assumed dependencies real? Have any of the named risks materialised? Adjust scope, not the goal.
• Day 60 — Course-correction window. This is the last cheap moment to change direction. If the leading indicators are flat or negative, escalate. Silence at day 60 is the most expensive form of optimism.
• Day 90 — Outcome review. Measure against the success criteria captured on day 0, not against the story the team is telling now. Write the post-mortem (or pre-mortem-confirmed) in the same artefact so the rationale, the outcome, and the lessons live together.

Stakeholder choreography

Decisions stall not because the analysis is wrong but because the choreography is wrong. Use a lightweight RACI on every recommendation:

Role	Meaning	Anti-pattern
Responsible	Does the work	More than two people listed
Accountable	Owns the outcome, signs off	Shared accountability (always becomes no accountability)
Consulted	Two-way input before the decision	Consulted after the decision is made — purely performative
Informed	One-way notification after the decision	Informed people are asked to approve — wastes their time and yours

If you cannot name a single Accountable person in one minute, the recommendation is not ready to ship.

Writing for senior readers

Senior readers scan first, read second, and only re-read the parts they disagree with. Optimise for that pattern:

1. Lead with the recommendation, not the analysis. The reader should know what you want them to do before they finish the first paragraph.
2. One screen, one page, one decision. If the artefact needs scrolling on a laptop, it is too long for the audience it is written for.
3. Tables beat paragraphs for comparing options. Prose hides the trade-off; a table forces it into the open.
4. Numbers beat adjectives. Replace "significant" with the actual number. Replace "soon" with a date. Replace "improved" with a baseline and a target.
5. Name the disconfirming evidence. A recommendation that lists what would change the author's mind is read as honest; one that does not is read as advocacy.

Common failure modes

Failure mode	Symptom	Counter-move
Analysis paralysis	Weeks of investigation, no decision	Time-box the analysis. State the decision quality you can defend in the time available.
HiPPO override	Highest-paid person's opinion wins regardless of evidence	Force the trade-off table into the room before opinions are voiced
Sunk-cost gravity	Team defends the current path because of prior investment	Re-frame: what would we choose today with no prior investment?
Scope creep at the checkpoint	Review becomes a re-planning session	Separate "did this work?" from "what next?" Run them as two meetings.
Stealth de-scoping	Success metrics quietly soften between day 0 and day 90	Lock the day-0 metrics into the artefact; require an explicit amendment to change them.
Owner drift	Accountable person leaves, no one re-assigns	Owner reassignment is a mandatory step in onboarding/offboarding the role

A worked example

A product line is debating whether to invest in a major rewrite of a legacy service that has been failing under peak load.

A weak response: "We should rewrite it because the code is old."

A response that uses this skill:

Recommendation. Do not rewrite. Invest one quarter in targeted performance work on the existing service and a parallel strangler-fig migration of the top two failing endpoints. Confidence: medium. Would change my mind if peak-load incidents continue at the current rate for two consecutive months after the performance work ships.

Options considered. (1) Full rewrite — 9–12 months, ~$1.4M, high risk of partial delivery. (2) Performance fix in place — 6 weeks, ~$120K, addresses 80% of incident volume per last-quarter analysis. (3) Strangler-fig migration — 6 months for the two hottest endpoints, ~$400K, preserves optionality.

Plan. Owner: Platform tech lead. Day 30: performance fix in staging with load test results. Day 60: production rollout and a 30-day incident-rate comparison. Day 90: decision on whether to expand the strangler-fig scope.

Risks. (1) Performance fix masks a deeper architectural issue — mitigated by capturing flame graphs before and after. (2) Strangler-fig endpoints are not in fact the hottest ones — mitigated by re-running the traffic analysis at day 0. (3) Team capacity collides with a separate compliance deadline — escalated to the portfolio review on the next planning cycle.

That is the shape of output this skill should produce: a defensible, time-bound, owner-attached recommendation that respects the reader's time and survives turnover.

Quick reference card

• One paragraph of context, three options with trade-offs, one recommendation with confidence, one plan with an owner and a date.
• If you cannot name the owner, the metric, and the checkpoint date in one breath, the artefact is not done.
• A decision without a written rationale is a rumour. A rationale without a checkpoint is a wish. A checkpoint without a metric is theatre.
• Reversibility matters more than people admit: one-way doors deserve the slow lane, two-way doors deserve the fast lane.
• The best artefacts in this category are short, dated, signed, and easy to find six months later.

Deep dive: applying this in practice

The sections above describe what to produce. This section describes how practitioners actually run this in the field, including the conversations, artefacts, and review loops that turn a one-page recommendation into a sustained outcome.

The 30/60/90 cadence

A recommendation that is never revisited is a recommendation that quietly fails. Bake review checkpoints in from day one:

• Day 0 — Decision committed. Owner, scope, success metrics, and the first-checkpoint date are recorded in the decision log. The artefact is linked from the team's working space so it is discoverable without asking.
• Day 30 — Early-signal review. Look at the leading indicators, not the lagging ones. Has the team actually started? Are the assumed dependencies real? Have any of the named risks materialised? Adjust scope, not the goal.
• Day 60 — Course-correction window. This is the last cheap moment to change direction. If the leading indicators are flat or negative, escalate. Silence at day 60 is the most expensive form of optimism.
• Day 90 — Outcome review. Measure against the success criteria captured on day 0, not against the story the team is telling now. Write the post-mortem (or pre-mortem-confirmed) in the same artefact so the rationale, the outcome, and the lessons live together.

Stakeholder choreography

Decisions stall not because the analysis is wrong but because the choreography is wrong. Use a lightweight RACI on every recommendation:

Role	Meaning	Anti-pattern
Responsible	Does the work	More than two people listed
Accountable	Owns the outcome, signs off	Shared accountability (always becomes no accountability)
Consulted	Two-way input before the decision	Consulted after the decision is made — purely performative
Informed	One-way notification after the decision	Informed people are asked to approve — wastes their time and yours

If you cannot name a single Accountable person in one minute, the recommendation is not ready to ship.

Writing for senior readers

Senior readers scan first, read second, and only re-read the parts they disagree with. Optimise for that pattern:

1. Lead with the recommendation, not the analysis. The reader should know what you want them to do before they finish the first paragraph.
2. One screen, one page, one decision. If the artefact needs scrolling on a laptop, it is too long for the audience it is written for.
3. Tables beat paragraphs for comparing options. Prose hides the trade-off; a table forces it into the open.
4. Numbers beat adjectives. Replace "significant" with the actual number. Replace "soon" with a date. Replace "improved" with a baseline and a target.
5. Name the disconfirming evidence. A recommendation that lists what would change the author's mind is read as honest; one that does not is read as advocacy.

Common failure modes

Failure mode	Symptom	Counter-move
Analysis paralysis	Weeks of investigation, no decision	Time-box the analysis. State the decision quality you can defend in the time available.
HiPPO override	Highest-paid person's opinion wins regardless of evidence	Force the trade-off table into the room before opinions are voiced
Sunk-cost gravity	Team defends the current path because of prior investment	Re-frame: what would we choose today with no prior investment?
Scope creep at the checkpoint	Review becomes a re-planning session	Separate "did this work?" from "what next?" Run them as two meetings.
Stealth de-scoping	Success metrics quietly soften between day 0 and day 90	Lock the day-0 metrics into the artefact; require an explicit amendment to change them.
Owner drift	Accountable person leaves, no one re-assigns	Owner reassignment is a mandatory step in onboarding/offboarding the role

A worked example

A product line is debating whether to invest in a major rewrite of a legacy service that has been failing under peak load.

A weak response: "We should rewrite it because the code is old."

A response that uses this skill:

Recommendation. Do not rewrite. Invest one quarter in targeted performance work on the existing service and a parallel strangler-fig migration of the top two failing endpoints. Confidence: medium. Would change my mind if peak-load incidents continue at the current rate for two consecutive months after the performance work ships.

Options considered. (1) Full rewrite — 9–12 months, ~$1.4M, high risk of partial delivery. (2) Performance fix in place — 6 weeks, ~$120K, addresses 80% of incident volume per last-quarter analysis. (3) Strangler-fig migration — 6 months for the two hottest endpoints, ~$400K, preserves optionality.

Plan. Owner: Platform tech lead. Day 30: performance fix in staging with load test results. Day 60: production rollout and a 30-day incident-rate comparison. Day 90: decision on whether to expand the strangler-fig scope.

Risks. (1) Performance fix masks a deeper architectural issue — mitigated by capturing flame graphs before and after. (2) Strangler-fig endpoints are not in fact the hottest ones — mitigated by re-running the traffic analysis at day 0. (3) Team capacity collides with a separate compliance deadline — escalated to the portfolio review on the next planning cycle.

That is the shape of output this skill should produce: a defensible, time-bound, owner-attached recommendation that respects the reader's time and survives turnover.

Quick reference card

• One paragraph of context, three options with trade-offs, one recommendation with confidence, one plan with an owner and a date.
• If you cannot name the owner, the metric, and the checkpoint date in one breath, the artefact is not done.
• A decision without a written rationale is a rumour. A rationale without a checkpoint is a wish. A checkpoint without a metric is theatre.
• Reversibility matters more than people admit: one-way doors deserve the slow lane, two-way doors deserve the fast lane.
• The best artefacts in this category are short, dated, signed, and easy to find six months later.

---

Deep Reference Playbook

The sections below extend this skill into a complete operating playbook so it can run end-to-end inside Claude Code, CoWork, or any agentic tool without further prompting. Pull only the sections you need for a given engagement.

Inputs the skill must collect

Before producing any output, the skill confirms:

1. Objective — the single decision or artifact the user wants out of this session.
2. Context — system, team, customer, product, or domain the work sits inside.
3. Constraints — time, budget, headcount, regulatory, technical, political.
4. Definition of done — what "good" looks like and who signs it off.
5. Audience — who reads or consumes the output (engineer, exec, customer, regulator).
6. Existing artifacts — prior versions, related docs, dashboards, tickets.
7. Risk appetite — how reversible the decision is and how much ambiguity is acceptable.

If any of these are missing, the skill asks targeted clarifying questions before generating output. It never invents constraints the user did not state.

Operating workflow

The canonical workflow for Ab Test Power Calc runs in five stages. Each stage has an explicit exit criterion so the skill knows when to advance.

Stage 1 — Frame. Restate the problem in one paragraph. Name the decision, the deadline, the stakeholders, and the success metric. Surface assumptions explicitly so they can be challenged.

Stage 2 — Diagnose. Inventory the current state with concrete evidence: metrics, quotes, screenshots, configs, tickets. Separate facts from interpretations. Identify the two or three root causes that explain most of the gap, not the long tail of symptoms.

Stage 3 — Design. Generate at least two viable options. For each option, capture: what changes, who owns it, what it costs, what it unblocks, what it risks, and how it could fail. Recommend one with a written rationale.

Stage 4 — Execute. Convert the chosen option into a sequenced plan: milestones, owners, dependencies, gating checks, communication cadence, and rollback triggers. Anything that cannot be assigned an owner and a date is not yet a plan.

Stage 5 — Validate. Define how success will be measured, when the measurement happens, and what action follows each possible result. Schedule the retrospective before the work starts, not after.

Outputs the skill produces

Depending on the request, the skill returns one or more of:

• A one-page brief suitable for an executive reader.
• A detailed working document for the delivery team.
• A decision record capturing the choice, the alternatives, and the rationale.
• A risk register with probability, impact, owner, and mitigation.
• A sequenced action plan with named owners and explicit due dates.
• A measurement plan tied to the success metric.
• A communication plan for stakeholders inside and outside the team.

Every artifact uses clear headings, short paragraphs, and tables where comparison helps. No filler. No restating the prompt. No hedging language when a recommendation is warranted.

Decision logic and trade-offs

The skill applies the following heuristics when choices are not obvious:

• Prefer reversible decisions taken quickly over irreversible decisions taken slowly.
• Optimise for the constraint that bites first — usually time, attention, or trust, not money.
• Default to the simplest design that meets the stated definition of done; add complexity only when a specific requirement forces it.
• Make the cost of being wrong visible so the reader can judge whether the recommendation is proportionate.
• Name the people, not the roles, when assigning ownership; ambiguous ownership produces ambiguous outcomes.

Anti-patterns the skill refuses to emit

Anti-pattern	Why it fails	What the skill does instead
Generic best-practice list with no context	Reader cannot act on it	Tailors recommendations to the stated constraints
Recommendation without trade-offs	Hides the cost of being wrong	Names the price paid for the recommendation
Plan with no owners or dates	Cannot be executed or tracked	Assigns a named owner and a date to every action
Metrics theatre	Measures activity, not outcome	Ties every metric back to the user or business outcome
Boil-the-ocean scope	Nothing ships	Cuts scope to the smallest valuable slice
Buried recommendation	Reader misses the point	Leads with the recommendation in the first paragraph

Quality bar

The skill self-checks each output against these gates before returning it:

1. Can a busy executive understand the recommendation from the first 150 words?
2. Is every claim either evidenced, labelled as an assumption, or removed?
3. Does every action have an owner and a date?
4. Are the trade-offs of the recommendation stated honestly?
5. Is there a measurable success criterion?
6. Would the author be comfortable defending this artifact in a review meeting?

If any gate fails, the skill rewrites the section before returning it.

Worked micro-example

Context: Calculate sample size and runtime for an A/B test given variance and MDE.

Frame: the team needs a defensible recommendation within five working days; the audience is a cross-functional steering group; the cost of delay is higher than the cost of being slightly wrong.

Diagnose: the dominant constraint is decision latency, not analytical depth. Existing data is sufficient for a directional call.

Design: two viable options surfaced. Option A optimises for speed and reversibility. Option B optimises for completeness but slips the deadline by two weeks.

Execute: Option A recommended. Plan sequenced into a two-week sprint with named owners, a mid-point checkpoint, and a clear rollback trigger.

Validate: success measured against a single leading indicator at day 30 and a single lagging indicator at day 90. Retrospective scheduled for day 35.

Cadence and follow-through

A one-shot artifact rarely changes outcomes. The skill recommends a lightweight cadence to keep the work alive:

• Weekly: owner posts a five-line status (done, doing, blocked, risk, ask).
• Fortnightly: steering group reviews leading indicators and unblocks dependencies.
• Monthly: retrospective on what the data is teaching the team; adjust plan accordingly.
• Quarterly: revisit the original objective and decide whether to continue, pivot, or stop.

Closing rules of thumb

1. Lead with the recommendation; supporting analysis follows.
2. Treat every output as a draft that will be challenged; pre-empt the obvious objections.
3. Prefer one strong recommendation over three weak options.
4. When the evidence is thin, say so; do not launder uncertainty as confidence.
5. Optimise for the next decision, not for the perfect document.
6. Make it easy for the reader to disagree with you in a structured way.
7. Ship the artifact; iterate against feedback rather than in private.