ViderStar A/B tests
1. Before the start - not optional
A test launched without this cannot be interpreted: if no effect is found, you cannot tell whether there is none or whether there was not enough power.
- One primary metric. Everything else is secondary and cannot be used to declare a win.
- MDE and sample size at alpha = 0.05 and power = 0.8. Compute them from the metric's actual variance over the last four weeks, not from a textbook example.
- Duration in whole weeks. Weekdays and weekends behave differently; a partial week biases the result.
- Name the stopping rule in advance. "We will look and decide" is peeking, and peeking inflates false positives.
- Write the hypothesis down before the start, in the ticket. It is the only protection against explaining whatever outcome arrives.
2. Check validity before reading the result
- Sample ratio mismatch. Observed group sizes against expected, chi-square. A mismatch is a defect in the split, and the result must not be read no matter how pretty the difference is.
- A/A on the pre-test period. If the groups already differed before the test, the difference is not from the arm.
- Overlap with other tests running on the same audience in the same window.
- Outliers. In a business with a heavy revenue tail, a single large order moves a group's mean visibly. Look at the mean, the median and a winsorised mean together; disagreement between them is itself the signal.
3. Reading it
- Confidence intervals, not just a p-value. The business needs a range for the effect.
- Several metrics at once means a multiple-comparison correction.
- Ratio metrics (conversion, CTR) need the delta method or a bootstrap, not a plain t-test over per-user aggregates.
- "Not significant" is not "no effect." It is "the effect is smaller than the MDE". Write it that way, and state the MDE.
- Segment slices only count if they were declared in advance. A segment found afterwards where "it did work" is a search through noise, not a result.
4. Writing it up
Structure: hypothesis → design (metric, MDE, duration, split) → validity checks → result with confidence intervals → decision (ship / do not ship / extend) → what remains unverified.
Then one separate paragraph in plain language: what the user will see and what you are doing about it. The business reads that paragraph and nothing else, so it must contain no p-values and no table names.