How to Run an Experiment That Actually Decides Something

Key takeaways

• Halving the effect you want to detect quadruples the traffic you need, because sample size scales with 1/Δ², which is why the 5% test almost never finishes.
• Peeking is not free: Optimizely measured 26% false-winner rates when experimenters checked every 500 visitors, and 57% when they checked continuously, against a 5% target.
• About 6% of Microsoft A/B tests fail a sample ratio mismatch check, which is broken randomization that voids the result before any statistics are run.
• A flat result never means no effect; it means no effect large enough for this sample to see, and only the confidence interval says how large that was.

An experiment exists to end an argument

The point of a controlled experiment is not to produce a number. It is to make a decision that was previously contested, and to make it in a way that survives the next meeting. If the result would not change what you do, the experiment was expensive theater.

The canonical example comes from Microsoft. In 2012 an employee working on Bing proposed changing the way the search engine displayed ad headlines. It was one of hundreds of ideas, program managers rated it low priority, and it sat for more than six months until an engineer, noticing the change was only a few days of work, simply ran it as an A/B test. Within hours the new variation was producing revenue so far above normal that the platform fired a too-good-to-be-true alert. The change had increased revenue by 12%, more than $100 million a year in the United States alone, without damaging key user-experience metrics. Kohavi and Thomke call it the best revenue-generating idea in Bing's history, and note that until the test ran, nobody in the building knew.

The reason that story matters is the base rate underneath it. At Google and Bing, only about 10% to 20% of experiments generate positive results. Across Microsoft as a whole the split is roughly a third effective, a third neutral, a third actively negative. Your considered, well-argued, senior-approved idea is more likely than not to do nothing. That is not a comment on your judgment; it is the measured base rate at organizations with the best product people in the world, and it is why they run so many tests. Microsoft, Amazon, Booking.com, Facebook and Google each run more than 10,000 online controlled experiments a year; Google has over a thousand running concurrently on any given day.

The same paper supplies the counterweight. Integrating Bing with social media, pulling Facebook and Twitter content into a third pane on the results page, cost Microsoft more than $25 million to build and produced negligible increases in engagement and revenue. Big investments frequently return nothing; small ones frequently return a great deal. The only way to tell which is which in advance is that you cannot, which is the entire argument for experimenting.

One consequence follows immediately and is the most commonly skipped step in the whole practice: you must name the metric that decides, before you look. Microsoft calls it the Overall Evaluation Criterion. Bing has built more than 6,000 metrics over the years. If you choose which of them counts after seeing the data, you have not run an experiment. You have run a search, and a search over 6,000 metrics will always find something. The American Statistical Association put the same rule plainly in its 2016 statement on p-values: proper inference requires full reporting and transparency, and p-values should not be reported selectively. Kohavi's version is blunter, borrowed from Twyman's law: any figure that looks interesting or different is usually wrong.

Sample size is the design, not a formality

How many users you need is not something you discover as the test runs. It is fixed before you start by three numbers you choose and one you inherit: how big an effect is worth detecting, how often you accept a false alarm, how often you accept missing a real effect, and how noisy your metric is.

The standard rule of thumb for two equal arms, 5% significance and 80% power, is that each arm needs roughly 16σ²/Δ² observations, where σ is the standard deviation of the metric and Δ the absolute difference you want to catch. The 16 is not magic: it is 2 × (1.96 + 0.84)², the squared sum of the z-scores for a two-sided 5% test and 80% power, rounded. Everything interesting about experiment design lives in the fact that Δ is squared.

Illustrative only, with round numbers. Your checkout converts at 3%. For a proportion, σ² = p(1−p) = 0.03 × 0.97 = 0.0291. You want to detect a 10% relative lift, so Δ = 0.003 absolute. Then n = 16 × 0.0291 ÷ 0.000009 ≈ 51,700 per arm, about 103,000 visitors in total. Now be more ambitious and ask for a 5% relative lift instead: Δ halves to 0.0015, Δ² quarters, and the requirement jumps to about 207,000 per arm, or 414,000 total. Relax to a 20% lift and it collapses to roughly 12,900 per arm, about 26,000 total. Same page, same traffic, same team: three completely different projects.

The published version of this arithmetic is worth memorizing because it shows the second lever. Kohavi, Henne and Sommerfield work an e-commerce example where 5% of visitors purchase and purchasers spend about $75, so average revenue per user is $3.75 with a standard deviation around $30. Detecting a 5% change in revenue takes over 1.6 million users. Detecting a 5% change in the conversion rate, a Bernoulli variable with far less spread, takes under 500,000. And if the change is in the checkout flow, analyzing only the 10% of users who actually start checkout brings it to 25,600 checkout starters, equivalent to about 256,000 site visitors: half the previous figure, so half the running time for identical power. Nothing about the change got easier. The variance got smaller.

That is the real craft: you buy power with sample size, and you buy sample size back with variance reduction. Pick the least noisy metric that still answers the question. Exclude users who could not possibly have seen the change, because they contribute noise and no signal. And where you have history on the same users, use it. CUPED, Deng, Xu, Kohavi and Walker's technique from WSDM 2013, regresses out each user's pre-experiment behavior, leaving var(Δ_cv) = var(Δ)(1 − ρ²) where ρ is the correlation between the pre-period and in-period metric. On Bing's system this cut variance by about 50%, which the authors describe exactly as it should be described: the same statistical power with half the users, or half the duration.

Now the part most write-ups omit. Run those numbers against your actual traffic before you plan anything. A site with 20,000 monthly visitors converting at 3% needs a bit over five months to power the 10% test and close to two years to power the 5% test, and that is one test, occupying the whole site, assuming nothing else changes. Kohavi and Thomke put the practical floor at a few thousand daily active users. Below that, A/B testing small changes is not a discipline you are failing at; it is a tool that does not fit. The honest alternatives are to test only changes large enough to show up (a different offer, a different price, a different page, not a different button color), to run before-and-after comparisons while being explicit that seasonality and trend are uncontrolled, or to decide on judgment and say out loud that it was judgment.

The peeking problem, and why medicine solved it first

A p-value is only valid at the sample size you committed to in advance. If you watch the dashboard and stop the moment it dips under 0.05, the 5% you think you are risking is not the 5% you are actually running.

The mechanism is easy to picture. With no true effect, the test statistic wanders like a random walk. Over a long enough run it will drift across the significance boundary at some point purely by chance, and if you are allowed to stop whenever you like, you are effectively sampling the most extreme moment of that walk rather than a single fixed draw. Each additional look is another door at which the 5% can be paid.

Optimizely measured this on its own platform and published the figures. Simulating A/A tests (comparing a page against itself, where the true effect is exactly zero), checking results after every visitor produced a false declaration of a winner or loser in 57% of runs. Checking every 500 visitors produced 26%. Checking every 1,000 produced 20%. Their summary is that even conservative monitoring policies can push error rates from a target of 5% to over 25%. That is why they replaced the underlying statistics with an always-valid sequential method in January 2015 and re-ran 48,000 historical experiments through it.

The damage is easier to feel in decisions than in p-values, so work it through. Illustrative only, using the Google/Bing base rate above: you run 100 experiments and 20 of them contain a real effect. Run properly at 80% power and 5% alpha, you detect 16 of the 20 real ones and pick up 4 false positives from the 80 nulls, for 20 declared winners, of which 80% are genuine. Now peek every 500 visitors, at Optimizely's measured 26%: the 80 nulls throw off about 21 false winners while the real detections stay around 16. You now have 37 declared winners of which 16 are real. Fewer than half. You will ship all 37, your quarterly metric will not move, and the conclusion drawn in the room will be that experimentation does not work here.

There is a second, quieter cost. Stopping at the moment a result crosses the line selects for the high random draw, so the effect you report is biased upward, sometimes badly. Teams then forecast next year on a number that was inflated by the stopping rule itself, and the shortfall gets blamed on execution.

Clinical research hit this problem decades earlier and built the fix into regulation. The FDA's November 2019 guidance on adaptive designs states it directly: in a group sequential design with an interim efficacy test at 50% enrolment, performing both the interim and final tests at the conventional .025 one-sided level would push the overall Type I error probability above .025. The guidance then names the established remedies: O'Brien-Fleming boundaries, which demand very persuasive early evidence to stop; Pocock boundaries, which demand less and stop more often; and the Lan-DeMets alpha-spending approach, which specifies a function for how the error budget is spent across the trial and so allows flexibility in when you look. It also notes the payoff nobody expects: a single O'Brien-Fleming interim analysis at the halfway point reduces the expected sample size by roughly 15% relative to a fixed-sample trial at 90% power. Done properly, monitoring makes experiments finish sooner, not later.

The operator's rule is therefore simple and admits no middle: either fix the horizon and do not read the result until you reach it, or use a sequential test that has priced the looking in. What you may never do is run a fixed-horizon test and read it continuously, which is what almost every dashboard invites you to do by default.

Everything that breaks before the statistics do

Most wrong experiment results are not statistical errors. They are plumbing. The test was fine; the data going into it was not.

The single highest-yield check is sample ratio mismatch. You configured a 50/50 split; you got something else. Kohavi and Thomke give the scale at which this matters: 821,588 users against 815,482 is a ratio of 50.2/49.8, and the probability of a divergence that large by chance is less than one in 500,000. Microsoft's Experimentation Platform refuses to show results until the SRM check passes, at a deliberately conservative threshold of p < 0.0005, and a recent internal analysis found about 6% of its A/B tests have an SRM. The reason an SRM voids the result rather than merely denting it is survivorship: the users who went missing are rarely a random slice. They are disproportionately the ones the treatment affected: the slow devices that timed out on the heavier variant, the bounce traffic a redirect dropped, the bot filter that fired asymmetrically. Whatever you measure afterwards is measured on a population the treatment already selected.

The second check is an A/A test: run your system against itself and confirm it declares no significant difference about 95% of the time. Kohavi describes this simple safeguard as having identified hundreds of invalid experiments and improper applications of formulas at Microsoft, including cases where a formula assumed independent measurements that were not independent.

Then the data itself. More than 50% of requests to Bing come from bots, and if they land in your arms unevenly, or at all, they add noise and can add bias. Amazon found individual accounts placing massive book orders that could skew an entire A/B test; they turned out to be libraries. Reusing the same users in the same control and treatment buckets across consecutive experiments produces carryover effects, where the previous experiment's exposure is still shaping this one's behavior, which is why Microsoft shuffles users between experiments.

Averages also hide segments. One Microsoft experiment came out negative overall because a JavaScript bug stopped Internet Explorer 7 users clicking on results at all; every other segment was positive. Without segment-level detection, an experimenter looking at the average would have thrown out a good idea as a bad one. The reverse happens too: a single unusually good or bad segment can carry an average that is unrepresentative of everyone.

Two pieces of arithmetic close this out, and both argue for restraint. First, bundle features at your peril: if any new feature has a 10% chance of an egregious bug that aborts its test, then a change containing seven new features fails with probability 1 − 0.9⁷, which is 52%, more likely than not. Second, watch how many metrics you consult. Testing 20 independent metrics at the 5% level on a treatment that does nothing gives a 1 − 0.95²⁰ = 64% chance that at least one comes back significant; at 40 metrics it is 87%. Bing has over 6,000 metrics available. Declare one OEC in advance, treat everything else as guardrails and diagnostics rather than evidence, and remember that a guardrail is there to veto a launch, not to authorize one.

What a flat result actually means

A result that fails to reach significance is a statement about your measuring instrument, not about the world. It says the effect, if any, was too small for this design to distinguish from noise. It does not say the effect is zero, and treating the two as the same is the most expensive reading error in the practice.

Read the confidence interval instead. Illustrative only: baseline conversion 3%, observed relative lift +1.2%, 95% interval running from −2.6% to +5.1%. That result is compatible with a small win, a small loss, and nothing at all. What you have actually learned is a boundary: it is probably not a 6% winner and probably not a 4% disaster. If your decision rule was ship anything above +2%, you have not decided; you have paid for the test and still have the argument. The ASA's fifth principle is the general form: a p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

The fix is to design the null in advance. Before the test, write down the smallest effect that would change your action, the minimum detectable effect you are powering for, and state what you will do if the interval lands entirely inside ±MDE. Done that way, a flat result is a genuine decision: the change is not worth the complexity, ship the simpler option, close the workstream, and move the team. That is a real return on the experiment. Ten flat results a year that each stop a bad roadmap item are worth more than one inflated winner.

It also helps to be precise about which claim you are making. "We could not detect a difference" and "we established the difference is too small to care about" are different statements, and only the second supports a decision. The second requires that you powered for a threshold and that the interval sits inside it. Say which one you have.

Kohavi and Thomke frame the learning value in a way worth adopting verbatim in readouts: look at the difference between the expected outcome and the actual result. If you thought something would happen and it did, you have learned little. If you thought it would happen and it did not, you have learned something important. And if you thought a change was minor and the result was a major surprise, that is the most valuable outcome available, because it means your model of the customer was wrong in a direction you can now exploit.

The organizational failure mode to watch for is what happens when teams are rewarded for wins rather than for decisions. A flat headline result becomes a hunt: split by device, split by geography, split by new versus returning, split by traffic source, until something crosses 0.05. That is the 64% arithmetic from the previous section, applied deliberately. If you genuinely want to explore segments, say so in advance, treat every finding as a hypothesis rather than a result, and go re-test the interesting one on fresh users.

The short-term win that is a long-term loss

The metric you can move in two weeks is often not the metric you are paid on, and the gap between them is where experimentation programs quietly destroy value while reporting success.

Google's advertising team documented the sharpest version of this. Hohnhold, O'Brien and Tang measured what they call ads blindness and sightedness: users' inherent propensity to click on ads shifts over time depending on the quality of the ads they were shown before. That shift is a learning effect, it accumulates over months of elapsed time, and it is therefore invisible inside a two-week experiment. Optimizing which ads to show on short-term revenue is, in their words, the obvious and easy thing to do, and may be detrimental in the long term.

The payoff of measuring it properly was a decision no short-horizon readout would ever have permitted. Google cut the ad load on its mobile search interface by 50%. In the short term that change was substantially revenue-negative. Once the long-term user learning was modelled, the long-term revenue impact was neutral, and the user experience was strongly improved. The company gave up measured near-term revenue for a change its own experiments said would pay for itself through behavior, and it needed a purpose-built methodology to see that at all.

Bing hit the mirror image while arguing about its own OEC. Degrading search results makes users issue more queries and click on more ads, which raises query share and revenue on a two-week horizon. It also, obviously, makes people switch search engines eventually. Bing's answer was to define success as minimizing queries per task while maximizing tasks per session, a metric deliberately constructed so that the cheap short-term win scores badly.

Not every effect is slow, and it is worth knowing which are fast so you do not over-correct. Latency is immediate and brutally linear: Bing's tests with artificially injected delays found every 100 milliseconds of page performance moved revenue by 0.6%, so against annual revenue over $3 billion a 100-millisecond speedup was worth about $18 million a year in incremental revenue. That is a number you can measure in a fortnight and bank.

Three practices keep the horizons honest. Hold back a small long-term control group that never receives the winning treatment, so you can compare months later rather than weeks. Re-run winners after the novelty and primacy effects have decayed, since a change that looks strong because it is new will regress, and a change that looks weak because users had a habit to unlearn will improve. And build the OEC so it contains at least one metric that a genuinely bad change would damage. An evaluation criterion that only goes up is not measuring anything.

Put it to work

Before the next test, write down the one metric that decides and the smallest effect worth shipping. Compute 16σ²/Δ² per arm, divide by weekly traffic, and if the answer exceeds six weeks, change the test rather than the deadline. Fix the horizon in advance and do not read early. On readout day check the sample ratio first, then report the interval, not the p-value.

Sources & references

Linked entries open the named source directly. Entries without a link say exactly what kind of reference they are — and how to check them yourself.

• Kohavi & Thomke, "The Surprising Power of Online Experiments," Harvard Business Review, Sept–Oct 2017 (reprint R1705E)
• Kohavi, Henne & Sommerfield, "Practical Guide to Controlled Experiments on the Web," KDD 2007 (sample-size arithmetic)
• Optimizely — "The story behind our Stats Engine" (measured false-positive rates under continuous monitoring)
• U.S. Food and Drug Administration — Adaptive Designs for Clinical Trials of Drugs and Biologics, Guidance for Industry (CDER/CBER, November 2019)
• Microsoft Research — Diagnosing Sample Ratio Mismatch in A/B Testing
• Deng, Xu, Kohavi & Walker, "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data" (CUPED), WSDM 2013
• Hohnhold, O'Brien & Tang, "Focusing on the Long-term: It's Good for Users and Business," KDD 2015 (Google)
• Wasserstein & Lazar, "The ASA's Statement on p-Values: Context, Process, and Purpose," The American Statistician 70(2), 2016