Holdout, geo lift, ghost ads, PSA and on/off tests

What is an incrementality test?

An incrementality test compares a group that could see your ads with a comparable group that could not, so the difference in conversions can be put down to the ads.

Control group: the people, regions or time periods that did not get the ads. Its results stand in for what would have happened without advertising.

Every method below is a different way of building that control group. The cleaner the control, the more you can trust the answer; the cleaner methods usually need more control over delivery than an advertiser has.

The five methods at a glance

User-level holdouts and ghost ads

In a user-level holdout the platform randomly assigns people to test or control before delivery and never shows the control group your ads. Randomisation is what makes it clean: on average, the two groups differ only in ad exposure.

Ghost ads: a holdout refinement where the platform records which control users would have been shown your ad in the auction. Comparing those "ghost" users with the users who actually saw the ad avoids diluting the result with people who would never have been reached anyway.

These are the methods behind most platform lift studies. They are precise but measure one platform in isolation, and you depend on the platform's own randomisation and reporting. The platform guide covers what Google and Meta offer and require.

PSA tests

In a PSA test the control group is shown a public service announcement instead of your ad. It was a common way to approximate a holdout before platforms offered native studies.

• Pro: you can see that control users were reachable, since they were served something.
• Con: you pay for the PSA impressions, and modern delivery systems optimise each ad separately, so the PSA may reach a different kind of person than your ad. That breaks the comparison.

On/off tests

The simplest test: pause a channel for two to four weeks and compare total installs with the weeks before and after.

• Cheap, needs no platform support and works for any channel, including brand search and Apple Search Ads.
• Weak against seasonality, store featuring, competitor moves and anything else that changes over the same weeks. Run it outside festivals and launches, and repeat it if the answer matters.

How to run a geo lift test for app installs

A geo lift test switches ads off in some regions and keeps them on in matched regions. It is the most practical test for teams without access to platform lift studies, and it works across channels.

1. Pick the unit. States, cities or metro areas within one country; countries or regions for apps sold in several markets. Each unit needs enough daily installs to be measurable on its own.
2. Match the groups. Split regions into test and control so their install trends over the last 8 to 12 weeks track each other closely. Random assignment within matched pairs is better than choosing by hand.
3. Fix the length and the read-out date. At least two weeks, ideally four, covering whole weeks. Work out in advance how big a lift you could detect (the holdout simulator shows how volume and length trade off).
4. Turn ads off in the control regions using each platform's location targeting. Keep everything else the same, including other channels.
5. Measure total installs per region, not attributed installs. The question is what changed overall.
6. Compare against the pre-period. Estimate what each test region would have done from its own history and the control regions, then read the difference with its interval.

Keep campaign names and tracking parameters consistent across regions so the read-out is not muddied by tagging differences; Deep Link Doctor's note on inconsistent UTM naming covers the usual slips.

The data you need is daily installs for each test unit, split by organic and attributed. MMPs such as Linkrunner, AppsFlyer and Adjust report daily installs by country; for regions within a country, check what your mobile measurement partner's exports include.

Sample data: a made-up app to show the method, not a benchmark.

A sample read-out: control regions grew 2% against their pre-period while test regions, with ads on, grew 14%. The ads' estimated effect is about 12 points of growth, which you would then convert into installs and compare with what those regions were credited with.

Which test should you run first?

• You spend mostly on one platform and qualify for its lift study: start there. It is the cleanest answer for that platform.
• You want to test several channels or brand search: run a geo lift or a careful on/off test.
• You have low volume: test your biggest channel only, for longer, and accept that only a large effect will show.

This practical incrementality testing guide has a step-by-step plan for a first holdout, and the lift test reader helps you read whatever you run.