Quantitative Methods

Propensity-score Matching and the Selection-on-observables Bet

Compare customers who adopted something against customers who look identical on paper and did not — then measure exactly how strong an unmeasured difference would have to be to erase the whole result.

  • Advanced
  • 13 min total
  • 14 chapters

What decision this helps you make: Whether an adoption-versus-retention comparison is strong enough to fund a spending programme, or whether the honest output is a randomized nudge on a small slice of customers and a much smaller cheque.

What this topic is

Propensity score matching estimates the probability of receiving treatment from observed characteristics, then pairs treated units with untreated units who had similar probabilities. Because the score summarizes everything you measured about who gets treated, matching on it balances all of those characteristics at once. The comparison of outcomes between matched groups is a causal estimate if — and only if — everything that drives both treatment and outcome was measured.

Why it matters

The most common analysis in any company with a product analytics stack is a comparison of people who did something against people who did not, and it is almost always reported as if the doing caused the difference. Matching is the standard way to make that comparison defensible, and it does genuinely remove the part of the gap explained by observed differences — often two thirds of it. What it cannot do is remove the part driven by whatever made those customers different in ways nobody recorded, which in a self-selected behaviour is usually the larger part.

Who should learn it

Product and growth analysts evaluating feature adoption, customer success teams valuing an engagement programme, finance teams asked to underwrite an initiative built on an adoption correlation, and anyone reviewing a deck that says "users who do X retain better."

What you will understand

  • What the propensity score does, and why balancing on one number balances everything you measured
  • How to check balance and overlap properly, and what a failed check means for the estimate
  • How to quantify the bet with a sensitivity analysis, so the unobserved confounder becomes a number rather than a caveat
  • When matching is the right tool and when the honest answer is a small randomized test instead

Prerequisites

Common misconception

"We controlled for everything, so the remaining difference is causal." Matching balances what you measured, and only that. In the worked example, matching on eleven observed characteristics cuts an apparent 32-point retention gap to 11 points — real progress. But an unmeasured factor associated with both adoption and retention by a modest amount, around a risk ratio of 1.6 on each, would account for the whole remaining 11 points. "Teams that were already committed to the product are more likely to set up shared workspaces" clears that bar easily and was never in the data. Matching converts an indefensible number into a smaller number that rests on an assumption you should state in a sentence and price.