Quantitative Methods

Multiple Comparisons and the False-discovery Rate

Run twenty comparisons at a 5% threshold and you will find a winner about two-thirds of the time even when nothing works. This is the arithmetic of that, and the procedure that lets you ship from a batch anyway.

  • Advanced
  • 12 min total
  • 15 chapters

What decision this helps you make: Which results from a batch — of metrics, of arms, of segments, of tests — you are entitled to act on, and what threshold you commit to before the sweep begins.

What this topic is

A multiple-comparisons problem exists whenever one decision is informed by more than one statistical test. The 5% threshold controls the error rate of a single comparison; run many, and the chance that at least one of them is a false positive rises quickly. The two standard remedies control different things: family-wise error rate holds the probability of any false positive at 5%, while false discovery rate holds the expected share of your declared winners that are false at a level you choose.

Why it matters

Real experiments almost never produce one number. A test has a primary metric, six secondary metrics, four guardrails, three variants, and a dashboard that slices by device, country, tenure, and channel. That is not one comparison; it is dozens, and someone will find something. Every organisation with a metrics tool has shipped changes on the strength of a subgroup result that was pure arithmetic.

Who should learn it

Anyone who reviews a batch of experiment results, runs metric sweeps or segment analyses, or has to decide which of several apparent winners is real.

What you will understand

  • Why twenty independent tests at 5% give a 64% chance of at least one false winner
  • The difference between family-wise error rate and false discovery rate, and which one your decision needs
  • How to run the Benjamini-Hochberg procedure by hand on a list of p-values
  • Why correcting is sometimes the wrong move, and what makes a batch a family at all

Prerequisites

Common misconception

"We corrected for multiple comparisons, so the results are clean." Correction handles the tests you counted. It cannot handle the ones you never ran but would have — the segment you would have sliced if the first slice had been flat, the metric you would have promoted if the primary had failed. Those unrun analyses are part of the procedure that produced your finding, and no correction applied afterwards can account for them. The only defence is deciding what you will test before you look.