Quantitative Methods

Statistical Significance and What a P-value Does not Mean

Get the definition right, learn what a p-value can and cannot license, and stop treating a number just under 0.05 as permission to spend money.

  • Advanced
  • 11 min total
  • 13 chapters

What decision this helps you make: Whether a "significant" test result is enough to ship, roll out, or reallocate budget — and what else you must know before it is.

What this topic is

A p-value is the probability of getting data at least as extreme as what you observed, assuming the null hypothesis is true and every other assumption in your model holds. It is a statement about data under an assumption, not a statement about the assumption. "Statistically significant" simply means that number fell below a threshold someone chose in advance, conventionally 0.05.

Why it matters

The p-value is the most widely quoted and most widely misdescribed number in commercial analysis. Two specific misreadings — that it is the probability the hypothesis is false, and that a large p-value proves no effect — are behind an enormous share of the decisions companies make and then quietly reverse. Getting the definition right changes what you can claim, what you must still find out, and how much money a single test result is allowed to move.

Who should learn it

Anyone who reads or presents test results: growth and product teams running experiments, marketers reading platform lift reports, operators evaluating pilots, and executives being told a result is significant.

What you will understand

  • The exact definition of a p-value, and the four things it is routinely confused with
  • How to compute one for a two-proportion test, and what moves it
  • Why the share of your "wins" that are false depends on your hit rate and your power, not on the threshold alone
  • Why an effect that barely clears significance is also probably overstated in size

Prerequisites

Common misconception

"p = 0.03 means there is a 97% chance the variant really is better." This is the transposed conditional, and it is wrong in the same way as concluding that because most people who own a private jet are wealthy, most wealthy people own a private jet. The p-value is computed by ASSUMING the variant is identical to control and asking how often data this extreme would show up. It cannot tell you the probability of the assumption it started by assuming. To get that, you would also need to know how often ideas like this one turn out to work — and that number never appears anywhere in the test report.