Experiment Design Validation
Core Workflow
- Define the decision the experiment will inform.
- Write a falsifiable hypothesis with target audience, change, expected behavior, and reason.
- Choose method: A/B test, holdout, fake-door test, concierge test, prototype test, usability test, smoke test, survey, landing-page test, or qualitative validation.
- Define primary metric, guardrail metrics, segmentation, exposure rules, sample/traffic constraints, duration, and stop criteria.
- Plan instrumentation and QA before launch.
- Decide in advance how results will be interpreted and what action follows.
- Record result, learning, caveats, and next experiment.
Freshness Rule
Verify current analytics, experimentation platform, privacy, consent, and ad-platform docs before giving tactical setup guidance for A/B tools, GA4/Firebase events, conversion tracking, targeting, or experiment allocation.
Deliverable Shape
For experiment work, provide:
- Decision and hypothesis
- Target audience and eligibility
- Variant/control design
- Primary and guardrail metrics
- Instrumentation and QA plan
- Duration/sample considerations
- Decision rules and follow-up actions
References
- Read
references/experiment-design-checklist.md when designing or reviewing an experiment.
1---2name: experiment-design-validation3description: Design, evaluate, and document experiments, A/B tests, product tests, growth tests, validation tests, hypotheses, primary metrics, guardrails, sample considerations, decision rules, and learning plans. Use when Codex is asked to test an idea, validate a change, run an experiment, compare variants, or create an experimentation plan.4license: MIT5---6
7# Experiment Design Validation
8
9## Core Workflow
10
111. Define the decision the experiment will inform.
122. Write a falsifiable hypothesis with target audience, change, expected behavior, and reason.
133. Choose method: A/B test, holdout, fake-door test, concierge test, prototype test, usability test, smoke test, survey, landing-page test, or qualitative validation.
144. Define primary metric, guardrail metrics, segmentation, exposure rules, sample/traffic constraints, duration, and stop criteria.
155. Plan instrumentation and QA before launch.
166. Decide in advance how results will be interpreted and what action follows.
177. Record result, learning, caveats, and next experiment.
18
19## Freshness Rule
20
21Verify current analytics, experimentation platform, privacy, consent, and ad-platform docs before giving tactical setup guidance for A/B tools, GA4/Firebase events, conversion tracking, targeting, or experiment allocation.
22
23## Deliverable Shape
24
25For experiment work, provide:
26
27- Decision and hypothesis
28- Target audience and eligibility
29- Variant/control design
30- Primary and guardrail metrics
31- Instrumentation and QA plan
32- Duration/sample considerations
33- Decision rules and follow-up actions
34
35## References
36
37- Read `references/experiment-design-checklist.md` when designing or reviewing an experiment.
gabrielmoreira