Lift test reader
Type in test and control results and read them the way an analyst would.
Lift is the test group's conversion rate minus the control group's, divided by the control rate. In the pre-filled sample, a 1.2% test rate against a 1.0% control rate is a 20% lift, with a 95% interval of roughly 5% to 35%. Report the interval, not just the point estimate, and plan with its cautious end.
Test group (could see ads)
Control group (held out)
Pre-filled with a sample test. Change any box.
Sample data: a made-up app to show the method, not a benchmark.
- Test install rate
- 1.20%
- Control install rate
- 1.00%
- Lift
- +20.0%
- 95% interval for lift
- +5.3% to +34.7%
- z score
- 2.67
- p-value (two-sided)
- 0.008
- Incremental installs in test group
- 360
- Share of test installs that were incremental
- 17%
- Holdout share
- 10%
Positive and clear of zero at 95% confidence. The ads most likely caused extra conversions: about 20% more than the control group's rate, but anywhere from 5% to 35% fits the data. Plan with the cautious end.
Five common misreadings
- Treating the point estimate as the answer. A 20% lift with an interval of 5% to 35% means the true lift could be a quarter of what you report. Budget plans should survive the low end.
- Reading "not significant" as "no effect". An interval of −10% to +40% says the test was too small to tell, not that the ads do nothing.
- Comparing incremental installs with all attributed installs. Incremental installs are measured in the test group only; scale them to the whole campaign before comparing.
- Ignoring what the control group saw. If control users were reached by another campaign aimed at the same people, the measured lift shrinks.
- Reading a percentage of a small rate as a big number. A 20% lift on a 1% install rate is 0.2 extra installs per 100 people reached.
The maths
Rates are installs divided by people in each group. The interval uses the two-proportion normal approximation without pooling:
The lift interval divides both ends by the control rate, treating it as fixed. That is a fair approximation when the control group has a few hundred conversions or more; with fewer, treat the interval as rough. The approximation also assumes people were split at random and that one person's choice does not affect another's.
Questions
What is a good p-value for a lift test?
Below 0.05 is the usual cut-off, matching a 95% confidence interval that excludes zero. Some platform tools let you choose lower confidence levels such as 80% or 90%; lower confidence gives quicker answers and more false wins.
What does incremental share mean in the reader?
It is incremental installs divided by all installs in the test group: the part of the test group's installs that the ads caused. The rest would have happened at the control group's rate anyway.
Why does the control group need so many conversions?
The uncertainty in the result comes mostly from the smaller group. A control group with only a few dozen conversions gives an interval too wide to act on, whatever the test group shows.
What is incrementality in app marketing?
Incrementality is the share of conversions that would not have happened without the ad. It is measured by comparing people who could see the ads with a similar group who could not, so it captures what the ads changed rather than what they were credited with.
Is attributed the same as incremental?
No. Attribution credits a campaign when a user touched an ad before installing, even if that user would have installed anyway. Incremental installs are only the ones the ads caused, so they are always equal to or fewer than attributed installs.
How big should my holdout group be?
Big enough that the control group collects enough conversions to detect the lift you care about within the time you can wait. Lower conversion rates and smaller expected lifts need bigger holdouts or longer tests; the holdout simulator on this site shows the trade-off.
Other labs
- 01Holdout test simulatorPick a holdout size, an audience and a true lift, then watch a simulated test day by day until the confidence interval clears zero.
- 02Cannibalisation labSee how a paid push can be credited with installs that were going to arrive organically, and what that does to the organic line.
- 03Attributed vs incremental CPI and ROASTwo sample channels side by side: how ranking them on attributed CPI and on incremental CPI can point budget in opposite directions.