Experiment Design Readout

Experiment Design Readout

Use this skill when the user wants to design, prioritize, or review product/growth experiments.

What this skill must produce

Always produce:

1. Experiment brief
2. Hypothesis
3. Primary and secondary metrics
4. Sample / ramp plan
5. Implementation notes
6. Decision rules
7. Risks
8. Post-test readout template

Working rules

• Prefer minimum viable tests.
• State the causal claim explicitly.
• Define what success and failure mean before the test launches.
• Use one primary metric.
• Guard against underpowered or low-quality tests.
• Distinguish operational guardrails from outcome metrics.
• Call out when a proposal is not suitable for an A/B test.

Step-by-step method

Step 1: define the problem

State the specific product or growth problem.

Step 2: define the hypothesis

Use this format:

If [change]
then [primary metric] will move by [expected direction / magnitude]
because [behavioral or product assumption].

Step 3: define the minimum viable test

Specify the smallest intervention that could validate or invalidate the hypothesis.

Step 4: define metrics

Choose:

• one primary metric
• 2-4 secondary metrics
• 2-4 guardrail metrics

Step 5: define the audience and segmentation

Specify:

• eligible population
• exclusions
• key segments to inspect

Step 6: define ramp and duration

Provide:

• initial ramp
• escalation path
• minimum duration
• sample size considerations
• stop conditions

Step 7: define interpretation rules

State what outcome would mean:

• launch
• iterate
• kill
• rerun with higher power / cleaner implementation

Step 8: define readout

Create the post-test structure before the test starts.

Output structure

Experiment brief

• Problem:
• User / segment:
• Proposed change:
• Why now:

Hypothesis

...

Test design

• Test type:
• Control:
• Treatment:
• Minimum viable version:
• Why this is the minimum viable test:

Metrics

Primary metric

• Name:
• Why it matters:
• Expected movement:

Secondary metrics

Guardrail metrics

Audience and segmentation

• Eligible audience:
• Exclusions:
• Segments to inspect:

Ramp / duration / sample plan

• Ramp plan:
• Target split:
• Duration:
• Sample / power caveats:
• Abort conditions:

Implementation notes

• Product / eng tasks:
• Analytics tasks:
• QA tasks:
• Dependencies:

Decision rules

• Launch if:
• Iterate if:
• Kill if:
• Rerun if:

Risks

• Underpowered test:
• Confounded implementation:
• Novelty effects:
• Long-term effects not captured:
• Maintenance cost vs gain:

Post-test readout template

Use this exact structure after the test finishes:

1. Experiment summary
2. What happened on the primary metric
3. What happened on secondary and guardrail metrics
4. Segment-level observations
5. Did the result support the hypothesis?
6. Most likely explanation
7. Decision
8. Follow-up actions
9. New questions created

Special instructions

If the requested test is pricing-related, state plainly when classic A/B testing is a bad fit due to low sample size, customer fairness concerns, or poor statistical power. Offer an alternative such as:

• pricing research
• segmented rollout
• minimum viable pricing test
• sales-assisted validation
• willingness-to-pay study

If the requested test has major irreversible downside, recommend a smaller-risk pretest or staged rollout.