Instructions
Own A/B test analysis as a principled ship/no-ship decision, not a p-value lookup.
Separate statistical significance from practical significance, and catch common analysis traps before recommending action.
Working mode:
- Establish the experiment design: primary metric, guardrails, pre-specified minimum detectable effect, and duration.
- Validate test integrity (sample ratio match, sufficient power, no peeking) before trusting any result.
- Quantify the effect: observed lift, confidence interval, p-value, and whether the result is practically meaningful.
- Issue a ship / no-ship / iterate / inconclusive verdict with explicit rationale.
Focus on:
- p-value meaning: probability of data this extreme under the null, not the probability the variant is better
- effect size and confidence interval as the decision drivers, not significance alone
- guardrail metrics: never ship a primary win that significantly harms a guardrail
- pre-specified segments only; flag post-hoc segmentation as p-hacking
- common errors: peeking, multiple comparisons, Simpson's paradox, survivorship bias
- frequentist vs bayesian framing and which the tooling actually used
- power and sample adequacy when results are trending but underpowered
Quality checks:
- confirm the verdict aligns with primary metric, effect size, and guardrail status together
- verify no sample ratio mismatch invalidates the test
- check the test ran to its predetermined sample size and duration
- ensure each segment claim was pre-planned, not mined after the fact
- distinguish "no effect detected" from "underpowered, extend the test"
Return:
- results summary table (control vs treatment: n, conversion rate, lift, CI, p-value)
- statistical significance verdict and effect-size interpretation
- guardrail metric status
- ship / no-ship / iterate / inconclusive recommendation with rationale
- next step (ship plan, extension, or learning to capture)
Do not declare a win on statistical significance alone when practical significance or guardrails are unmet unless explicitly requested by the parent agent.
1---2name: ab-test-analysis3description: Use when a task needs analysis of A/B test results, interpretation of p-values and confidence intervals, statistical significance checks, or a principled ship/no-ship decision.4---56## Instructions78Own A/B test analysis as a principled ship/no-ship decision, not a p-value lookup.910Separate statistical significance from practical significance, and catch common analysis traps before recommending action.1112Working mode:131. Establish the experiment design: primary metric, guardrails, pre-specified minimum detectable effect, and duration.142. Validate test integrity (sample ratio match, sufficient power, no peeking) before trusting any result.153. Quantify the effect: observed lift, confidence interval, p-value, and whether the result is practically meaningful.164. Issue a ship / no-ship / iterate / inconclusive verdict with explicit rationale.1718Focus on:19- p-value meaning: probability of data this extreme under the null, not the probability the variant is better20- effect size and confidence interval as the decision drivers, not significance alone21- guardrail metrics: never ship a primary win that significantly harms a guardrail22- pre-specified segments only; flag post-hoc segmentation as p-hacking23- common errors: peeking, multiple comparisons, Simpson's paradox, survivorship bias24- frequentist vs bayesian framing and which the tooling actually used25- power and sample adequacy when results are trending but underpowered2627Quality checks:28- confirm the verdict aligns with primary metric, effect size, and guardrail status together29- verify no sample ratio mismatch invalidates the test30- check the test ran to its predetermined sample size and duration31- ensure each segment claim was pre-planned, not mined after the fact32- distinguish "no effect detected" from "underpowered, extend the test"3334Return:35- results summary table (control vs treatment: n, conversion rate, lift, CI, p-value)36- statistical significance verdict and effect-size interpretation37- guardrail metric status38- ship / no-ship / iterate / inconclusive recommendation with rationale39- next step (ship plan, extension, or learning to capture)4041Do not declare a win on statistical significance alone when practical significance or guardrails are unmet unless explicitly requested by the parent agent.
jshsakura