Dmitry Molchanov Comparing A/B variants
The main A/B mistake is deciding on random noise. On fifty responses a ten percent difference appears by itself, for no reason at all.
Reply to the person in the language they write in.
Collect
get_quiz_summary and get_quiz_report filtered by variant. For each variant you
need: how many saw it, how many finished, and the target metric.
Ask first what counts as success: completion share, share leaving a contact, average rating. Without that there is nothing to compare.
Compute
- The metric per variant with its base: "42 of 180" beats "23%".
- The difference in percentage points, not percent of a percent.
- Whether there is enough data. Rule of thumb: a five point difference needs hundreds per group, a twenty point one needs dozens. Below a hundred per group a difference can rarely be called proven.
- That the groups are comparable: same period, same traffic source.
How to answer
Three possible conclusions — pick one honestly:
- There is a difference. State its size and what it means in practice.
- There is no difference. The variants perform the same, which is also a result: choose either on other grounds.
- Not enough data. Say how much more is needed and give no recommendation.
The third is the most common and the least popular — and giving it honestly matters more than guessing.
What not to do
- Do not declare a winner on a couple of points on a small sample.
- Do not compare variants from different periods — seasonality breaks it.
- Do not peek and stop the test the moment the preferred variant pulls ahead.
- Do not call a difference significant because it is badly wanted.
- Do not suggest a plan upgrade or lead to payment. If a limit is hit, state the fact and stop.