Quantitative Methods

Confidence Intervals and the Range You Should Have Quoted

Replace every point estimate in your reporting with the range the data actually supports, and learn the one sentence about intervals that almost everyone, including most analysts, gets wrong.

  • Intermediate
  • 10 min total
  • 12 chapters

What decision this helps you make: What number goes in the plan, the board deck, or the price change: the point estimate, or the bottom of a range you can defend.

What this topic is

A confidence interval is a range constructed by a procedure that, applied repeatedly to samples from the same process, would contain the true value a stated share of the time. A 95% interval comes from a method with 95% long-run coverage. It shows which values of the underlying quantity are reasonably compatible with the data you collected, given your model.

Why it matters

Businesses report point estimates and then plan against them as though they were facts. A measured 16.7% conversion lift whose interval runs from roughly zero to 33% is not a 16.7% lift. It is a finding worth somewhere between six hundred dollars and a hundred thousand dollars a year, and those two conclusions call for different decisions. The interval is where the honest version of the number lives, and quoting it is the cheapest analytical upgrade available to most companies.

Who should learn it

Anyone who puts a number in a plan, a forecast, a pricing decision, or a board pack: operators, finance leads, product managers, and analysts who present results to people who will act on them.

What you will understand

  • The precise meaning of the 95%, and why it describes the method rather than this interval
  • How to compute an interval on a rate, on a difference, and on a count of zero
  • Why two overlapping intervals do not mean two groups are indistinguishable
  • How to convert an interval into a range of dollars and plan against the correct end of it

Prerequisites

Common misconception

"There is a 95% probability the true value is inside this interval." This is the sentence nearly everyone says and it does not follow from how the interval was built. In the standard framework the true value is a fixed unknown number: it is either inside your interval or it is not, and no probability attaches to that. The 95% is a property of the PROCEDURE across repeated samples. A Bayesian credible interval does support the probability statement people want, but only relative to a prior you had to state in advance, which the standard interval never asked for.