Quantitative Methods

Bandit-based Price Testing Without Burning the Market

Adaptive price testing promises to earn while it learns. Find out how much it actually earns, why profit is enormously harder to measure than conversion, and which guardrails stop a price test from doing permanent damage to the market you sell into.

  • Advanced
  • 12 min total
  • 13 chapters

What decision this helps you make: Whether to run an adaptive bandit or a fixed-horizon test on price. And, before either, whether your traffic can resolve the profit differences you are trying to find at all.

What this topic is

A multi-armed bandit is an allocation rule that shifts traffic toward whichever option is currently winning while continuing to sample the others. Applied to price, each candidate price is an arm and the reward is what a visitor is worth at that price. The appeal is that you stop paying full price for the losing options while you learn. The complication is that prices are not independent options: they sit in an order, they change what the same customer expects next time, and they are visible to people who are not in your test.

Why it matters

Price is the highest-leverage number in most businesses and the one companies test least, because testing it feels dangerous. Adaptive methods are marketed as the safe answer: less traffic wasted on bad prices, faster convergence, revenue protected during the test. Some of that is true. What is almost never said is that the quantity you care about, contribution per session, is many times noisier than the conversion rate everyone reports, so the amount of traffic required to tell two nearby prices apart on profit is often more than a mid-sized business generates in a year. A bandit that converges in three weeks on that data has not found the best price. It has found a lucky one, and it has stopped exploring the others.

Who should learn it

Operators, growth and pricing leads at businesses with enough transaction volume to consider testing price, and anyone being sold an automated pricing optimiser.

What you will understand

  • What a bandit optimises, how it differs from a fixed-horizon test, and when each is the right instrument
  • Why measuring a profit difference takes an order of magnitude more traffic than measuring a conversion difference
  • Why price arms are not independent, and what that does to the guarantees a bandit comes with
  • The specific guardrails that stop a price test from contaminating your market permanently

Prerequisites

Common misconception

"A bandit is strictly better than an A/B test, because it makes money during the experiment instead of wasting traffic." It optimises a different objective, and you have to know which one you want. A bandit minimises cumulative regret (how much you gave up along the way), and it achieves that by sampling losing arms less. Sampling an arm less is exactly what makes its estimated value unreliable, and estimates from adaptively collected data are biased, typically downward for arms that got unlucky early and were then starved of traffic.[4] So if you need a number (an elasticity for a model, a figure for a board, an input to a capacity decision), a bandit is the wrong instrument. If you only need to keep the winner and will never quote the losers, it is the right one.