# Experiment Design

> Design product experiments (A/B tests, holdouts, staged rollouts) that produce trustworthy learning about what works. Use when testing hypotheses about features, pricing, UX, messaging, or growth levers.

- Skill: `itsual/experiment-design` (Agent Skill)
- Install (CLI): `npx skillmds@latest add itsual/experiment-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/itsual/experiment-design/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- Author: itsual (https://skillmd.com/u/itsual)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/itsual/experiment-design

---


# Experiment Design

## Overview

Experiments turn opinions into evidence. Good experiments have a clear hypothesis, a sensible design, and pre-committed success criteria.

## When to Use

- Testing feature or UX changes
- Evaluating growth, pricing, or messaging ideas
- Reducing uncertainty before full rollout
- Measuring incremental impact of product changes

## Core Elements

- Hypothesis and rationale
- Primary metric and guardrail metrics
- Target population and sample size / duration reasoning
- Variants and what differs between them
- Randomization and assignment method
- Success criteria and decision rules
- Risks and mitigations

## Principles

- Decide how you will interpret results *before* looking at them
- Don’t peek and stop early without a plan
- Watch for novelty effects, seasonality, and interference
- Prefer fewer, well-powered experiments over many inconclusive ones
- Document outcomes and share learning even when the test “loses”

## Verification

- [ ] Hypothesis and metrics are explicit
- [ ] Experiment can actually produce a decision
- [ ] Results (including null results) are recorded