The Peeking Problem and Why You Cannot Watch a Test Run

What this topic is

The peeking problem is what happens to a significance threshold when the data are examined repeatedly and the test can be stopped at any point that looks convincing. The 5% threshold is a promise about a single, pre-specified reading of a fixed sample. Repeated looks with an option to stop convert it into a much weaker promise, and the more often you look, the weaker it gets.

Why it matters

Every modern experimentation dashboard updates in real time and shows a p-value or a "probability to beat baseline" that moves each morning. Someone is looking. If the team has any discretion about when to call a result — and every team does — the nominal error rate is fiction. This is the single most common reason an experimentation programme produces a steady flow of wins that do not reproduce.

Who should learn it

Anyone who runs experiments on a platform with a live dashboard, and anyone who has ever been asked whether a test can be called early because the number looks good.

What you will understand

• How much the false-positive rate actually inflates with each additional look, with the published numbers
• Why continuous monitoring under a true null eventually declares a winner with probability approaching one
• The three legitimate fixes — fixed horizon, group sequential boundaries, and always-valid inference — and what each costs
• Why switching to a Bayesian readout does not by itself solve optional stopping, and what part of that claim is contested

Prerequisites

• The Value of Information and What a Test Is Worth Before You Run It
• Holdouts and Geo Experiments When You Cannot Randomize Users

Common misconception

"Peeking is fine as long as we do not stop the test — we just want to see how it is going." The problem is not looking; it is the option to act. If nobody would ever stop early, the looks are harmless. But the option is almost always live: a promising result gets escalated, a flat one gets extended, and the definition of "the planned end" quietly becomes "when the number settled somewhere we liked." A stopping rule that responds to the data is the thing that breaks the threshold, whether or not anyone admits to having one.