Experimental Design

Experimental Design

Core principle: Define what counts as evidence before you collect it. A well-designed experiment eliminates alternative explanations so only one conclusion survives. Pre-registration is the firewall between discovery and self-deception.

When to Use This Skill

• A decision hinges on a factual claim no one has tested
• Someone proposes an A/B test but hasn't defined success criteria
• epistemic-mapping flagged a dangerous assumption needing validation
• A team is about to invest resources based on an unvalidated hypothesis
• Prior experiment results were inconclusive and need a better design

Core Methodology

Step 1: Convert the Question into a Testable Hypothesis

Take the vague question ("Does feature X improve retention?") and produce a precise, falsifiable hypothesis pair.

• H₀ (null): The default — no effect. "Feature X has no effect on 30-day retention."
• H₁ (alternative): The claim. "Feature X increases 30-day retention by ≥ 2 percentage points."

State the direction. One-tailed when you only care about improvement; two-tailed when you also need to detect harm. Default two-tailed unless strong prior reason to expect one direction.

If a hypothesis isn't falsifiable — if no observable outcome could disprove it — rewrite it.

Step 2: Identify Variables

• Independent variable (IV): What you manipulate. One per experiment unless factorial design with sufficient power.
• Dependent variable (DV): What you measure. The metric closest to the outcome you care about. Proxies introduce noise.
• Controlled variables: Held constant across conditions.
• Potential confounders: Could vary between groups and affect DV independently. List them all, then design controls (randomization, stratification, matching, statistical control).

Step 3: Select the Experimental Design

Randomized Controlled Trial (RCT / A/B test) — Gold standard. Random assignment eliminates confounding. Use when you can randomize and have sample size.

Quasi-experimental — When randomization is impossible:

• Difference-in-differences: Treated vs. untreated, before vs. after. Requires parallel trends.
• Regression discontinuity: Exploit a threshold (users above/below a score cutoff). Strong validity near cutoff.
• Interrupted time series: Track outcome over time, look for level/slope change at intervention.

Within- vs. between-subjects — Within-subjects eliminates individual differences but introduces order effects (counterbalance or use washout periods).

Factorial — Multiple IVs to detect interactions. Multiplicatively more sample. Only when interactions are the research question.

State what design you picked and why alternatives were rejected.

Step 4: Determine Sample Size

Run a power analysis before data collection:

• Minimum detectable effect (MDE): Smallest practically meaningful effect — not statistically significant, practically significant.
• Significance level (α): False positive rate. Standard 0.05; lower for high-stakes.
• Power (1-β): Probability of detecting a real effect. Standard 0.80; use 0.90 for critical experiments.
• Baseline rate: Current DV value (e.g., retention = 40%).
• Variance estimate: From historical data or pilot.

If required n exceeds feasible: accept larger MDE, run longer, or redesign for a more sensitive metric. Do not run underpowered and hope.

Step 5: Pre-Register Success Criteria

Before any data collection, commit in writing to:

• Primary metric: The one metric that determines success.
• Secondary metrics: Should move if hypothesis is correct (and shouldn't if effect is spurious).
• Success threshold: Effect size and significance required.
• Guardrail metrics: Must not degrade (error rate, latency, revenue).
• Decision rules: "If primary improves by ≥ MDE at p < α, ship. If guardrail degrades by > X%, halt."
• Stopping rules: Early stop conditions (sequential testing with adjusted thresholds) and extension criteria.

Pre-registration prevents the garden of forking paths — redefining success after seeing results.

Step 6: Mitigate Threats to Validity

Internal validity (did X actually cause Y?):

• Selection bias → randomization or matching
• Attrition → intent-to-treat analysis
• Maturation → control group captures natural change
• History → concurrent control experiences same external events
• Instrumentation → consistent measurement across conditions

External validity (does it generalize?):

• Sample representativeness → stratified sampling
• Temporal validity → run through full business cycles
• Setting validity → test in production, not staging

Construct validity (are we measuring what we think?):

• Metric validity → does the DV capture the construct?
• Treatment fidelity → did participants actually experience the treatment?

For each threat, state mitigated / partially mitigated / unmitigated and what that means for interpretation.

Step 7: Specify the Analysis Plan

Define the exact analytical approach before data arrives:

• Statistical test: t-test, chi-squared, regression, Bayesian — matched to data type and design.
• Multiple comparison correction: Bonferroni, Holm, or FDR if testing multiple hypotheses.
• Segmentation plan: Pre-specified subgroups (new vs. returning users). Exploratory segments labeled as such.
• Sensitivity analysis: Re-run under different assumptions (excluding outliers, different attribution windows).
• Effect size reporting: Practical effect size (Cohen's d, relative lift, absolute difference) alongside p-values. Statistical without practical significance is noise.

Step 8: Define Timeline and Resources

• Ramp-up plan: Traffic allocation schedule (5% → 20% → 50%).
• Minimum run duration: From sample size and business cycles.
• Maximum run duration: When to call it.
• Resource requirements: Engineering, data pipeline, stakeholder reviews.
• Go/no-go checkpoints: Dates to review interim data for safety (not for peeking at results).

Output Format

🎯 Research Question

• Plain language: [What are we trying to find out?]
• H₀: [Null — no effect]
• H₁: [Alternative — specific, falsifiable, directional]

📋 Variables

🔬 Design

• Type: [RCT / quasi-experimental / observational — with variant]
• Justification: [Why, and what alternatives were rejected]
• Assignment: [Random / stratified / threshold — mechanism]

📊 Sample Size

• MDE: [Minimum detectable effect]
• Power: [Target]
• α: [Significance level]
• Required n: [Per group and total]
• Feasibility: [Can we reach this? If not, what changes?]

✅ Pre-Registered Criteria

• Primary metric: [name] — success if [condition]
• Secondary metrics: [List with expected direction]
• Guardrail metrics: [List with degradation thresholds]
• Decision rule: [If...then ship / halt / extend]
• Stopping rule: [Early stop conditions]

⚠️ Threats to Validity

📊 Analysis Plan

• Test: [Statistical test with justification]
• Correction: [Multiple comparison method, if applicable]
• Segments: [Pre-specified subgroups]
• Sensitivity: [Alternative analyses]

📋 Timeline and Resources

• Duration: [Min — max]
• Ramp plan: [Traffic allocation schedule]
• Checkpoints: [Go/no-go review dates]
• Resources: [What's needed]

Thinking Triggers

• "What would we need to observe to abandon this hypothesis?"
• "If we saw this result, could we explain it without the hypothesis being true?"
• "Are we testing what we think we're testing, or a proxy?"
• "What's the smallest effect that would actually change our decision?"
• "If we can't randomize, what's the best natural experiment available?"
• "What would a skeptic say is wrong with this design?"

Common Traps

• Post-hoc hypothesis: Formulating after seeing data. If you discover something unexpected, label it exploratory and design a new experiment to confirm.
• Underpowered experiments: Running too small to detect realistic effects, then concluding "no effect." Absence of evidence is not evidence of absence — especially with n=47.
• Peeking: Checking daily and stopping when results look good. Inflates false positives dramatically. Use sequential testing with adjusted thresholds for interim looks.
• Surrogate metrics: Measuring clicks when you care about retention. The surrogate must have a validated causal link to what matters.
• Multiple testing without correction: Testing 20 metrics and celebrating the one significant at p=0.05 is finding noise.
• Ignoring practical significance: A statistically significant 0.01% improvement at p=0.03 isn't worth shipping if engineering cost exceeds value.

Example Applications