alexe-ev Analyze Experiment Results
Purpose
Interpret experiment outcomes using metric deltas, uncertainty, and business context.
Skill type
Calculation-aware skill
Use this skill when
- The user has results from control and test groups
- The user wants to understand whether variant B outperformed A
- The user needs a concise experiment readout
Required inputs
- Control group results
- Test group results
- Primary metric
Optional inputs
- Secondary metrics
- Guardrail metrics
- P-value
- Confidence interval
- Sample sizes
- Time series
- Segment cuts
Additional resources
Use examples/ when
- you need to see examples of complete inputs, missing inputs, or unsafe interpretation
- you need to match the expected output format
- you need to see how the skill behaves under different levels of input completeness
Use REFERENCE.md when
- you need calculation logic
- you need formulas, assumptions, or interpretation rules
- you need invalid-use conditions
- you need trust checks or caveats for incomplete or unreliable inputs
Important
Do not rely on examples as a substitute for methodology. Use REFERENCE.md whenever the task requires formulas, assumptions, statistical interpretation, or trust checks.
Resource priority
- Read this Skill.md first
- Use examples/ for behavior patterns and output shape
- Use REFERENCE.md for methodology and interpretation rules
Upstream context
This skill works best when the following already exist:
- clear control and test results
- primary metric definition
- sample sizes or a statistical summary
- any known guardrail signals
- ideally an experiment design or quality context
If upstream context is missing
If sample sizes, metric definition, or statistical summary are missing, reduce the analysis to descriptive comparison. If business context is missing, avoid strong action recommendations. If procedure quality is unknown, state that confidence in decision-making is limited.
Downstream handoff
A strong output from this skill should be usable by:
- validate-experiment-quality
- recommend-post-test-decision
- experiment readout or memo writing
A good handoff includes:
- effect size
- statistical summary
- guardrail summary
- result classification
- caveats
- missing context that limits confidence
Context collection rules
For this skill, context has two layers:
- calculation context
- business interpretation context
Strong calculation context
Proceed with full analysis.
Weak calculation context
Analyze descriptively and state that statistical interpretation is limited.
Weak business context
Interpret the numbers, but avoid strong product recommendations.
Minimum context to collect for a strong readout
- clear primary metric definition
- comparable control and test values
- sample sizes or statistical summary
- any major guardrail signal
- whether the test result is intended for business decision-making
Analysis rules
- Compare the primary metric between variants.
- Report the observed effect in both absolute and relative terms when appropriate.
- If p-value and/or confidence interval are provided, interpret them.
- If raw counts are provided for a simple rate metric and no p-value is provided, indicate that statistical testing should be performed using an appropriate method for the metric type.
- Distinguish:
- observed uplift
- statistical significance
- practical/business significance
- Review guardrails before giving a positive interpretation.
- If the result quality is uncertain, explicitly say the analysis depends on experiment validity.
Output
Provide:
- Primary metric comparison
- Effect size
- Statistical summary
- Guardrail summary
- Result classification
- Practical interpretation
- Key caveats
- Missing context that limits confidence