Quantitative Methods

Designing an A/B Test That Can Actually Detect the Effect

Work out, before you spend a single session of traffic, whether the test you are about to run could detect the effect you are hoping for, and what to do when the honest answer is no.

  • Advanced
  • 13 min total
  • 14 chapters

What decision this helps you make: Whether to run this test at all, at what unit of randomization, and for how long, or whether to admit the traffic cannot resolve the question and decide another way.

What this topic is

An A/B test randomly assigns each unit (usually a user) to see one version or another, then compares an outcome between the groups. Randomization is the whole mechanism: it makes the two groups alike on everything you measured and everything you did not. So a difference at the end is attributable to the version rather than to who ended up in which group. Designing one means fixing five things in advance: the decision it feeds, the unit you randomize, the single metric that settles it, the smallest effect worth detecting, and the sample size that follows from all four.

Why it matters

Most business A/B tests are underpowered, which means they are structurally incapable of finding the effect they were built to find. They run for two weeks, come back "not significant," and get read as "the change did nothing." In fact the test could only ever have detected a change three times larger than anything the team was proposing. The traffic is spent, the calendar is spent, and the organisation now believes something false with the authority of a number attached.

Who should learn it

Operators who run or commission experiments (growth, product, marketing, pricing, and merchandising leaders) and anyone who has to decide whether a test result is worth acting on.

What you will understand

  • Why the sample size falls out of the effect you want to detect, and why halving that effect quadruples the traffic you need
  • What a p-value actually says, and the two sentences people write about it that are wrong
  • How to choose the unit of randomization, and what breaks when the unit and the analysis disagree
  • What to do when the arithmetic says the test cannot be run, which is the most common outcome

Prerequisites

Common misconception

"We ran the test and it was not significant, so the change does not work." A non-significant result from an underpowered test carries almost no information. If your test could only detect a 20% lift and the true effect is 6%, "not significant" is the expected outcome whether the change works or not. The design question is whether this test could have found the effect, and it has to be answered before the test runs. Afterwards there is no way to tell a real null from an unpowered one.