Quantitative Methods

Bayesian Updating and the Posterior You Should Have Carried

Turn a surprising test result into a number you would actually bet on, by combining it with everything you already knew instead of throwing that away.

  • Advanced
  • 14 min total
  • 13 chapters

What decision this helps you make: How much to move your belief, and your budget, after a single result, and whether the effect you just measured is big enough to be worth the change you are about to make.

What this topic is

Bayesian updating is arithmetic for changing your mind by the right amount. You start with a prior: what your accumulated experience says an effect is likely to be. You observe data. You combine them into a posterior, which is your revised belief, and which sits somewhere between the two, closer to whichever one is better supported. The whole method is one line of algebra applied honestly, and the honesty is the hard part.

Why it matters

Most operators already do a crude version of this, badly. A test comes back at plus eighteen percent and the room splits between the people who want to ship it and re-forecast the year on it, and the people who say small samples lie. Both are reasoning from a prior; neither has written it down. Writing it down produces a specific, defensible number, usually far below the headline, and that number is what you should staff, budget, and promise against. It also tells you when a result is genuinely strong enough to overturn what you believed, which is the case worth acting on hard.

Who should learn it

Anyone who has to decide what a single result means: operators reading experiment output, finance teams re-forecasting on a pilot, and anyone who has ever been handed a win and asked whether to build on it.

What you will understand

  • Why a statistically significant result in a low-yield testing program is often close to a coin flip, and how to compute how close
  • How to build a prior out of your own history instead of inventing one, and how to state its strength as a sample size
  • How much an observed effect shrinks toward the prior, and why the shrunk number is the one to budget on
  • The precise difference between a credible interval and a confidence interval, and which one you are allowed to talk about as a probability

Prerequisites

Common misconception

"The test was significant at p under 0.05, so there is a 95% chance the effect is real." That sentence is wrong twice over. A p-value is the probability of seeing data at least this extreme if there were no effect at all. It is a statement about data given a hypothesis, never about a hypothesis given data. And the probability that the effect is real depends entirely on how often effects are real in your program, which the p-value does not know. Run a testing program where one idea in seven genuinely works, and a batch of significant results at conventional power will contain roughly as many false alarms as true findings.