Quantitative Methods

Difference-in-differences and the Parallel-trends Assumption

Recover the effect of a rollout you could not randomize by subtracting what happened to everyone else, and find out how much of your headline result was just the calendar.

  • Advanced
  • 14 min total
  • 15 chapters

What decision this helps you make: Whether a change you rolled out to some locations and not others actually caused the lift, and whether the number that survives is large enough to fund the next wave.

What this topic is

Difference-in-differences compares the change in an outcome for a group that got a treatment against the change over the same period for a group that did not. The first difference removes everything permanent about each group. The second removes everything that happened to both of them. What is left is attributed to the treatment. In its simplest form it is four averages and two subtractions; in its usual form it is a regression with unit and time fixed effects and standard errors clustered on the unit.

Why it matters

Most business changes are not randomized. A price rise goes to one region, a new store layout to one format, a loyalty program to whichever twelve stores the operations director thought were ready. The before-and-after number those rollouts produce is contaminated by season, by macro conditions, by whatever else the company did that quarter, and it is almost always far too large. Difference-in-differences is the cheapest honest correction available, and it usually cuts the headline effect by most of its size.

Who should learn it

Operators evaluating a staged rollout, finance teams asked to underwrite a program's return, marketing and growth teams running geo tests, and anyone who has been handed a before-and-after chart and asked to approve spending against it.

What you will understand

  • How the two subtractions work and what each one removes
  • What parallel trends actually asserts, why it cannot be tested, and why a clean pre-trend plot is weaker evidence than it looks
  • How to compute the estimate, cluster the errors correctly, and turn the interval into a funding decision
  • Why staggered rollouts break the standard regression, and which estimators exist because of it

Prerequisites

Common misconception

"The pre-trends look parallel, so the assumption holds." Parallel trends is a claim about a counterfactual: how the treated group would have moved if it had not been treated. That path does not exist and no test can observe it. A pre-period plot tests something adjacent and weaker: whether the two groups moved together before, over however many periods you happen to have. Those tests have low power against exactly the slow divergences that would matter most, and conditioning your analysis on having passed one distorts the estimate you then report. A clean pre-trend is a reason not to abandon the design. It is not evidence that the design is right.