Quantitative Methods

Multi-armed Bandits and the Explore-exploit Tradeoff

Learn when to stop splitting traffic evenly and start shifting it toward whatever is winning — and what you permanently give up when you do, which is the ability to say how much better the winner was.

  • Advanced
  • 13 min total
  • 14 chapters

What decision this helps you make: Whether this test should be an A/B test or a bandit — that is, whether you need an effect size you can defend or simply the most conversions during the test.

What this topic is

A multi-armed bandit is an algorithm for allocating traffic among competing options while it is still learning which is best. Instead of fixing the split in advance, it continuously reallocates toward whichever option currently looks best, keeping just enough traffic on the others to notice if it was wrong. The quantity it minimises is regret: the conversions you lost relative to having known the best option from the start.

Why it matters

An A/B test spends half its traffic on a loser on purpose, and pays for that with a clean, defensible estimate of the difference. A bandit spends much less on losers and pays for that with an estimate you cannot cleanly defend, because the amount of data each option received depended on how well it was doing. Choosing between them is a real business decision with real money on both sides, and it hinges on one question that almost nobody asks explicitly: do you need the number, or do you need the conversions?

Who should learn it

Anyone running creative, headline, offer, or layout tests at volume; growth and marketing teams choosing between testing tools; and operators who have been sold "AI-powered optimization" and want to know what they are actually buying.

What you will understand

  • What regret is, and how to compute what an even split costs you against an adaptive one
  • How Thompson sampling works mechanically, with a worked posterior update on real-shaped numbers
  • Why the sample averages a bandit reports are biased, and what to do if you need an unbiased number
  • The four conditions under which a bandit is the wrong tool and a fixed-horizon test is right

Prerequisites

Common misconception

"A bandit is a smarter A/B test — it finds the winner faster." It is not a faster test; it is a different objective. An A/B test is built to estimate a difference precisely, and the even split is optimal for that. A bandit is built to earn while learning, and it deliberately sacrifices estimation precision to do so. It will often take longer to be confident which option is best, because it starves the losing arms of exactly the data that would prove they are losing. If someone hands you a bandit result and a confidence interval on the lift, be careful: the ordinary interval is not valid on adaptively collected data.