experimental-design-ds

A/B testing, randomization, sample size calculation, confounding control, and causal inference for data science. Covers the full experimental lifecycle from hypothesis formulation through power analysis, randomization strategies, blocking, factorial designs, sequential testing, and the potential outcomes framework for causal claims. Use when designing experiments, planning A/B tests, calculating sample sizes, or reasoning about causation from data.

Jump to install

Repository: Tibsfox/gsd-skill-creator
Last source activity: April 14, 2026 at 07:57
Detected SKILL.md language: English
Stars: 69
Forks: 9

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

A direct command skips the review prompt. Inspect the source before running it.

npx skills add https://github.com/Tibsfox/gsd-skill-creator --skill experimental-design-ds

The command stays on one line. Scroll horizontally to inspect it before copying.

Prefer a local copy? Download the files currently available to SkillsMP.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

SKILL.md

Source instructions · Read-only preview

Back to search Back to installation

goal-ambiguity-gate

Use the moment a request is underspecified and you are about to act on an ASSUMED goal. Enumerate the candidate goals the request could mean, project the action or artifact each one implies, and check whether they diverge: if two or more plausible goals lead to materially different actions or artifacts, ask exactly ONE targeted clarifying question; otherwise proceed on the most-likely goal and state the assumption in one line. Clarification is an evidence-producing action, not a delay — but one question, never an interrogation. Distinct from gsd-spec-phase (GSD-phase-bound, emits a heavy SPEC.md) and intent-router (fetch strategy, not goal disambiguation). Backed by agent goal-state inference (arxiv 2606.16813v1). Triggers on acting under an assumed goal when the request admits more than one materially different reading.

2026-07-19

memory-hubness-gate

Use at write time to vector/embedding memory — Grove content-addressed insertion, chroma/pgvector upserts, memory-consolidation promoting session traces to MEMORY.md, or embedding externally-ingested content. Scores each candidate record against a fixed panel of sentinel queries and quarantines any record that would become the nearest neighbor of too many unrelated queries — a hub — whether from adversarial poisoning or accidental over-generality. This is the memory-record-side sibling of skill-injection-guardian (file-side) and the write-side complement of memory-use-warrant (read-side). Quarantine, never silently drop; a human reviews. Backed by the admission-time hubness gate (arxiv 2606.19692v1). Triggers on inserting into vector memory, consolidating memory, and embedding stranger content.

2026-07-19

memory-use-warrant

Run this appropriateness check the moment you are about to integrate a retrieved long-term memory — a Grove content-addressed hit, a chroma/pgvector neighbour, a memory-consolidation digest, or a MEMORY.md line — into a response, especially anything touching private origins, Fox Companies IP, credentials, or Center Camp / consent-governed content. It answers a question intent-router never asks: not WHAT to fetch or HOW, but WHETHER a correctly-retrieved item should reach output. Relevance is not appropriateness — a perfect similarity match can still be a boundary violation. Default is FAIL-CLOSED: if in-context authorization is uncertain, the memory may inform behaviour but must not be surfaced. Backed by RBI-Eval (arxiv 2606.06055v1). Triggers on surfacing recalled sensitive memory into a response.

2026-07-19

View all from this repository

name	experimental-design-ds
description	A/B testing, randomization, sample size calculation, confounding control, and causal inference for data science. Covers the full experimental lifecycle from hypothesis formulation through power analysis, randomization strategies, blocking, factorial designs, sequential testing, and the potential outcomes framework for causal claims. Use when designing experiments, planning A/B tests, calculating sample sizes, or reasoning about causation from data.
type	skill
category	data-science
status	stable
origin	tibsfox
modified	false
first_seen	"2026-04-12T00:00:00.000Z"
first_path	examples/skills/data-science/experimental-design-ds/SKILL.md
superseded_by	null

Experimental Design for Data Science

Experimental design is the discipline of collecting data so that the analysis can answer the intended question. Ronald Fisher, working at the Rothamsted agricultural station in the 1920s, formalized the three pillars -- randomization, replication, and blocking -- that remain the foundation of every modern experiment, from clinical trials to A/B tests on websites. This skill covers experimental design from the data scientist's perspective: planning experiments, ensuring valid causal inference, and avoiding the pitfalls that invalidate conclusions.

Agent affinity: fisher (experimental design, ANOVA), tukey (exploratory analysis of experimental results), benjamin (ethical review of experiments)

Concept IDs: data-hypothesis-testing, data-confidence-intervals, data-probability-basics, data-sampling-methods

Fisher's Three Pillars

1. Randomization

Random assignment of experimental units to treatment groups ensures that any observed difference is either due to the treatment or due to chance -- not due to confounders. Without randomization, the groups may differ systematically in ways that bias the result.

Mechanism: Each unit is assigned to treatment or control by a random process (coin flip, random number generator). This does not guarantee balance -- it guarantees that imbalances are random and quantifiable by probability theory.

Why it works: Randomization breaks the association between treatment assignment and all potential confounders, both observed and unobserved. No observational adjustment method can do this for unobserved confounders.

2. Replication

Multiple experimental units per group. Replication serves two purposes:

Statistical power: More units means smaller standard errors and greater ability to detect real effects.
Generalizability: Results from one unit might be idiosyncratic. Results replicated across many units are more convincing.

Replication is not repetition. Measuring the same unit 10 times is repetition (estimates measurement error). Measuring 10 different units is replication (estimates treatment effect variability).

3. Blocking

Grouping experimental units by a known source of variability and randomizing within blocks. If you know that males and females respond differently to a treatment, block by sex and randomize within each block. This reduces unexplained variance and increases power.

Blocking vs. stratification: Same concept, different fields. Blocking is the experimental design term (Fisher); stratification is the sampling term.

Stage	Goal	Key decisions
1. Hypothesis	Define what you want to learn	Primary hypothesis, secondary hypotheses, directional vs. two-sided
2. Metric selection	Define how you will measure the effect	Primary metric, guardrail metrics, sensitivity metrics
3. Power analysis	Determine required sample size	Minimum detectable effect, significance level, power
4. Design	Choose experimental structure	Completely randomized, blocked, factorial, sequential
5. Randomization	Assign units to groups	Simple, stratified, cluster randomization
6. Execution	Run the experiment	Monitor for implementation errors, compliance
7. Analysis	Test the hypothesis	Pre-specified analysis plan, intention-to-treat
8. Interpretation	Draw conclusions	Effect size, confidence interval, practical significance

Pitfall	Problem	Fix
Peeking	Checking results before full sample size; inflates false positive rate	Use sequential testing methods or commit to fixed horizon
Multiple comparisons	Testing many metrics inflates family-wise error	Bonferroni correction, FDR control, or pre-specify one primary metric
Network effects	One user's treatment affects another user's outcome (social platforms)	Cluster randomization (randomize by geographic region or social cluster)
Novelty/primacy effects	Users react differently to new features initially	Run experiments long enough for the effect to stabilize (2-4 weeks)
Sample ratio mismatch	Unequal assignment ratios indicate implementation bugs	Check actual vs. expected ratio before analyzing results
Simpson's paradox	Aggregate result reverses when stratified by a confounder	Pre-stratify or check for compositional changes across groups

Strategy	Description	Use when
Simple	Coin flip per unit	Default; works for large samples
Stratified	Randomize within strata of important covariates	Known prognostic factors (age, gender, etc.)
Cluster	Randomize groups (classrooms, hospitals, regions)	Treatment applies at group level or interference between individuals
Adaptive	Adjust allocation probabilities based on interim results	Clinical trials with ethical constraints (avoid allocating to inferior arm)
Switchback	Same unit alternates between treatment and control over time	Marketplace experiments where user-level randomization causes interference

Method	Assumption	Use when
Regression adjustment	All confounders are measured and included	Known, measurable confounders
Propensity score matching	Treatment assignment is ignorable given observed covariates	Observational data with rich covariate information
Instrumental variables	An instrument affects treatment but not outcome directly	Natural experiments (lottery assignment, geographic boundaries)
Difference-in-differences	Parallel trends in absence of treatment	Policy changes affecting one group but not another
Regression discontinuity	Treatment assigned by a threshold on a continuous variable	Program eligibility cutoffs (test scores, age)

Method	How it works	Trade-off
Group sequential (O'Brien-Fleming, Pocock)	Pre-specify interim analysis times with adjusted alpha spending	Slightly larger sample size; requires planning
Always-valid p-values	Construct p-values that are valid at any stopping time	More conservative; wider confidence intervals
Bayesian sequential	Update posterior continuously; stop when posterior probability exceeds threshold	Requires prior specification; more complex analysis

experimental-design-ds

Install options

Review the source files

Experimental Design for Data Science

Fisher's Three Pillars

1. Randomization

2. Replication

3. Blocking

The Experimental Lifecycle

Experimental Design for Data Science

Fisher's Three Pillars

1. Randomization

2. Replication

3. Blocking

The Experimental Lifecycle

A/B Testing

The Basic Framework

Sample Size Calculation

Common A/B Testing Pitfalls

Randomization Strategies

Factorial Designs

Full Factorial

Fractional Factorial

Causal Inference

The Fundamental Problem

Potential Outcomes Framework

Observational Causal Methods

Confounding

Directed Acyclic Graphs (DAGs)

Sequential Testing

The Problem with Peeking

Solutions

Common Mistakes

Cross-References

References

Mistake	Why it fails	Fix
No pre-registration	Post-hoc hypothesis selection inflates false positives (p-hacking)	Write the analysis plan before seeing the data
Underpowered experiment	Cannot detect real effects; waste of resources	Power analysis before running
Confusing statistical and practical significance	p < 0.05 with a tiny effect size is not actionable	Report effect size and confidence interval alongside p-value
Treating observational data as experimental	Cannot make causal claims without randomization	Use causal inference methods and state assumptions
Stopping early because "it looks significant"	Inflated false positive rate	Use sequential testing methods
Ignoring multiple comparisons	Testing 20 metrics at alpha = 0.05 expects 1 false positive	Bonferroni, Holm, or Benjamini-Hochberg correction

experimental-design-ds

Source facts

Install options

Review the source files

More from this repository

More from this repository

Experimental Design for Data Science

Fisher's Three Pillars

1. Randomization

2. Replication

3. Blocking

The Experimental Lifecycle

Experimental Design for Data Science

Fisher's Three Pillars

1. Randomization

2. Replication

3. Blocking

The Experimental Lifecycle

A/B Testing

The Basic Framework

Sample Size Calculation

Common A/B Testing Pitfalls

Randomization Strategies

Factorial Designs

Full Factorial

Fractional Factorial

Causal Inference

The Fundamental Problem

Potential Outcomes Framework

Observational Causal Methods

Confounding

Directed Acyclic Graphs (DAGs)

Sequential Testing

The Problem with Peeking

Solutions

Common Mistakes

Cross-References

References