Experiment Design

Design product experiments (A/B tests, holdouts, staged rollouts) that produce trustworthy learning about what works. Use when testing hypotheses about features, pricing, UX, messaging, or growth levers.

itsual Updated 7 repo stars

File contents

Experiment Design

Overview

Experiments turn opinions into evidence. Good experiments have a clear hypothesis, a sensible design, and pre-committed success criteria.

When to Use

  • Testing feature or UX changes
  • Evaluating growth, pricing, or messaging ideas
  • Reducing uncertainty before full rollout
  • Measuring incremental impact of product changes

Core Elements

  • Hypothesis and rationale
  • Primary metric and guardrail metrics
  • Target population and sample size / duration reasoning
  • Variants and what differs between them
  • Randomization and assignment method
  • Success criteria and decision rules
  • Risks and mitigations

Principles

  • Decide how you will interpret results before looking at them
  • Don’t peek and stop early without a plan
  • Watch for novelty effects, seasonality, and interference
  • Prefer fewer, well-powered experiments over many inconclusive ones
  • Document outcomes and share learning even when the test “loses”

Verification

  • Hypothesis and metrics are explicit
  • Experiment can actually produce a decision
  • Results (including null results) are recorded

itsual/agent-skills-collection commit b3e1e7012c

Frequently asked questions

npx get-skill add itsual/experiment-design