Experiments & Data
Experiment Readout Calculator
The other half of the test, and the half that goes wrong. Two conversion rates always differ; the question is whether they differ by more than chance would produce anyway. This gives you the lift, the range the true lift plausibly sits in, and a p-value — and then tells you plainly when that range includes zero, which means you learned nothing, or when the sample is too small for any of it to mean much.
Inputs
- Control visitors
- Control conversions
- Variant visitors
- Variant conversions
How to use this calculator
- Enter the visitors and conversions for each arm exactly as your analytics reports them: visitors who were exposed to the variant, and the conversions among those visitors. Counting all site traffic against test-only conversions is the most common way this comes out wrong.
- Read the confidence interval before the lift. The headline lift is one sample's answer; the interval is the range of true effects consistent with what you saw, and it is usually far wider than people expect.
- If the interval contains zero, the test did not find anything. That is not 'trending positive' and not 'directionally good'. It means a loss is as consistent with your data as a gain. Do not ship on it.
- If the interval clears zero, look at how wide it still is. An interval running from +1% to +30% tells you the change works and tells you almost nothing about how much, so plan on the low end.
- Check the smallest cell in the table. Under about ten conversions the maths behind every number here stops being trustworthy, and the tool will refuse to print a p-value rather than dress up four events as a finding.
What each term means
- Conversion rate
- Conversions divided by visitors in that arm. Two rates always differ; the question is whether they differ by more than chance produces.
- Absolute difference (pp)
- Variant rate minus control rate, in percentage points. Going from 4% to 4.65% is 0.65 percentage points, and also a 16% relative lift. Both are true, and the relative one always sounds bigger.
- Confidence interval
- The range of true differences consistent with this data at 95%. Run the same test twenty times and roughly nineteen of these intervals would contain the real effect.
- p-value
- The chance of seeing a difference at least this large if the two versions were actually identical. It is not the probability that the variant is better, and a great many decisions have been made on that misreading.
- Statistically significant
- Only means the result is unlikely to be pure chance. It says nothing about whether the effect is large enough to be worth the engineering, and a big enough sample makes trivial differences significant.
- Normal approximation
- The shortcut that lets binomial counts be treated as a bell curve. It needs roughly ten conversions and ten non-conversions per arm before it can be relied on.
Educational disclaimer: Outputs are simplified educational estimates built from the numbers you enter — they are not financial, legal, tax, or investment advice, and real decisions deserve verified figures and qualified professionals.