Quantitative Methods

The Value of Information and What a Test Is Worth Before You Run It

Put a dollar ceiling on any piece of research before you commission it, and find out that a large share of what your company tests could not change a single decision.

  • Advanced
  • 13 min total
  • 13 chapters

What decision this helps you make: Whether to run the test, survey, pilot, or study at all — and how accurate it would have to be, and how much you may spend on it, before it stops being worth the delay.

What this topic is

The value of information is the increase in expected value that comes from knowing something before you decide. It has an exact definition and an exact ceiling. The expected value of perfect information is what you would gain if an oracle told you the truth for free; nothing you can buy is worth more than that. The expected value of sample information is what an imperfect signal is worth, and it is usually a small fraction of the perfect-information ceiling — sometimes exactly zero.

Why it matters

Companies commission research constantly and almost never price it first. The result is a testing culture that feels rigorous and is largely decorative: studies that confirm what was going to happen anyway, surveys whose every possible result leads to the same decision, and pilots whose six-week duration costs more than the information could ever be worth. The value of information gives you a number, computed before you spend anything, and the number is frequently zero. Knowing which of your tests are the zero ones frees the budget for the few that are not.

Who should learn it

Anyone who approves research spend, decides how long an experiment runs, or has to justify a pilot — and anyone who has sat through a study readout and realised nobody would have acted differently either way.

What you will understand

  • How to compute the expected value of perfect information, which caps what any study can ever be worth
  • Why information is worth nothing whenever it cannot change your action, and how to test that in one question
  • Why the value of an imperfect test is brutally nonlinear in its accuracy — zero below a threshold, then rising fast
  • How to set a research budget and an accuracy requirement before commissioning anything

Prerequisites

Common misconception

"More information is always better, so a study can only help." Information has a price and a delay, and its benefit is bounded by something that has nothing to do with how interesting it is: whether any possible result would change what you do. A study that would leave your decision unchanged under every outcome has an expected value of exactly zero, no matter how well designed it is, how large the sample is, or how much you learn from it. In the worked example in this lesson, a study that is right 70% of the time is worth precisely nothing — not a little, not "hard to quantify," but zero — because its most pessimistic possible finding still leaves the launch as the better choice.