---
title: Holdout Group Size for App Incrementality Tests
description: How to size an incrementality holdout for an app: control conversions needed, and how holdout share, install rate and expected lift set the test length.
url: https://incrementalitycalculator.com/learn/holdout-group-size/
---

[Start](https://incrementalitycalculator.com/) / [Learn](https://incrementalitycalculator.com/learn/) / Sizing a holdout

# How big should my holdout group be?

How to size an incrementality holdout for an app: control conversions needed, and how holdout share, install rate and expected lift set the test length.

**Short answer**

Size the holdout so the control group collects enough conversions to detect the smallest lift worth acting on. At 95% confidence and 80% power, detecting a 15% lift needs roughly 400 control conversions and a 10% lift nearly 900, whatever the install rate. Most app tests hold out 5% to 20% of the audience.

## What a holdout group is for

A holdout group is a random slice of the audience that is deliberately not shown the ads, so its conversion rate can stand in for what everyone would have done without them.

**Holdout share:** the fraction of the audience placed in the control group. A 10% holdout means one person in ten is kept from the ads.

The holdout is a cost: those people do not see ads that might have converted them. It is also where most of the uncertainty in the result comes from, because it is the smaller group. Sizing is the balance between the two.

## How big should my holdout group be?

Big enough that the control group gathers the conversions needed to detect the smallest lift you would act on, within a test length you can live with. In practice that is usually 5% to 20% of the audience.

Four inputs decide it:

- **Minimum lift worth detecting.** If a 10% lift would not change any decision, plan for 20%. Smaller lifts need far bigger tests: halving the lift needs roughly four times the sample.
- **Conversion rate.** Rarer events need more people. An install is more common than a first order, so install tests finish sooner.
- **Daily reach.** More people per day means more conversions per day in both groups.
- **Holdout share.** A bigger holdout fills the control group faster but withholds ads from more people.

## A rule of thumb: count control conversions, not people

The standard sample size formula (95% confidence, 80% power) gives a useful shortcut. For a 10% holdout, the number of *conversions* the control group needs is nearly the same whatever the conversion rate:

| Lift you want to detect | Control conversions needed (about) |
|---|---|
| 5% | 3,500 |
| 10% | 870 |
| 15% | 390 |
| 20% | 220 |
| 30% | 100 |

These are outputs of the formula, not benchmarks. They hold for install rates from about 0.5% to 2%, and the [about page](https://incrementalitycalculator.com/about/#method) lists the assumptions.

So the planning question becomes: how many days until my control group has collected that many conversions?

## Worked example

Sample data: a made-up app to show the method, not a benchmark.

Forkly, a made-up app, reaches 40k people a day with a campaign. About 1% of them would install anyway, and the team wants to detect a 15% lift.

| Holdout share | Control people per day | Control people needed | Days needed |
|---|---|---|---|
| 5% | 2k | about 37k | about 18 |
| 10% | 4k | about 39k | about 10 |
| 20% | 8k | about 44k | about 6 |
| 30% | 12k | about 52k | about 4 to 5 |
| 50% | 20k | about 74k | about 4 |

Going from 5% to 10% nearly halves the test. Going from 30% to 50% saves very little time and withholds ads from far more people. That flattening is why most tests settle between 10% and 20%.

Try these settings in the [holdout simulator](https://incrementalitycalculator.com/labs/holdout-simulator/): the planned length line moves as you change the holdout share, and the simulated estimate shows how noisy the early days are.

## Set the length before you start

**Peeking:** checking a running test every day and stopping the first time it looks significant.

Peeking is the most common way tests go wrong. Early on, the confidence interval is wide and the estimate swings; by chance it will clear zero on some day even when the effect is small. In the simulator's default draw, the interval clears zero on day 6, slips back on day 7 and only stays clear from day 9. Stopping at day 6 would have reported a win too early.

- Decide the length from the sizing maths.
- Round up to whole weeks so weekdays and weekends are balanced.
- Read the result once, on the planned day.

## When volume is too low

Small apps often find the maths asks for months. Options, in order:

1. **Test a bigger effect.** Pause the whole channel in a geo test rather than looking for a small lift from a creative change.
2. **Use a higher-volume event.** Installs rather than purchases, if installs are what the channel is bought on.
3. **Raise the holdout to 20% to 30%** for a shorter, more expensive test.
4. **Accept a directional answer** and say so when you report it.

A [low-budget attribution guide](https://linkrunner.io/blog/low-budget-attribution-guide-measuring-roas-when-you-re-spending-under-rs2l-month?utm_source=incrementalitycalculator&utm_medium=referral&utm_campaign=learn-holdout-group-size&utm_content=inline) covers measurement when spend is small. For a sense of how much of your volume is organic to begin with, AttributionBenchmarks publishes [organic vs paid install shares](https://attributionbenchmarks.com/organic-vs-paid-installs/) by category.

## Where the counts come from

For a platform lift study, the platform counts conversions in each group. For geo or on/off tests you count them yourself, which needs daily installs split into organic and attributed. An MMP is the usual source: [Linkrunner](https://linkrunner.io/pricing?utm_source=incrementalitycalculator&utm_medium=referral&utm_campaign=learn-holdout-group-size&utm_content=inline), for example, includes organic attribution on every plan and has unrestricted CSV, API and webhook exports, so daily counts can go straight into a spreadsheet or notebook.

Run this on your own data

Sizing a holdout starts from your own baseline: how many installs arrive with no ad touch on a normal day. Your MMP's organic install counts give you that rate. Linkrunner's free plan covers 25k attributed installs, with no credit card and every feature included.

- [A practical guide to a first incrementality test](https://linkrunner.io/blog/incrementality-testing-guide-mobile-app-marketers?utm_source=incrementalitycalculator&utm_medium=referral&utm_campaign=learn-holdout-group-size&utm_content=panel)
- [Incrementality testing, defined](https://linkrunner.io/glossary/what-is-incrementality-testing?utm_source=incrementalitycalculator&utm_medium=referral&utm_campaign=learn-holdout-group-size&utm_content=panel)
- [Create a free account](https://dashboard.linkrunner.io/sign-in?utm_source=incrementalitycalculator&utm_medium=referral&utm_campaign=learn-holdout-group-size&utm_content=panel)

## Questions

**Is a 10% holdout enough?**

Often, if the control group can collect a few hundred conversions in two to four weeks. If it cannot, either test for a larger lift, raise the holdout, or run longer.

**Why not use a 50/50 split?**

A 50/50 split gives the most precise answer per person, but it withholds ads from half the audience. For an incrementality test on a running campaign, that cost is usually too high, and beyond about 30% the test gets only slightly shorter.

**Does the holdout size change the lift?**

No. The holdout share changes how precisely you measure the lift, not the lift itself, as long as both groups are chosen at random from the same audience.

**What is incrementality in app marketing?**

Incrementality is the share of conversions that would not have happened without the ad. It is measured by comparing people who could see the ads with a similar group who could not, so it captures what the ads changed rather than what they were credited with.

**Is attributed the same as incremental?**

No. Attribution credits a campaign when a user touched an ad before installing, even if that user would have installed anyway. Incremental installs are only the ones the ads caused, so they are always equal to or fewer than attributed installs.

**How big should my holdout group be?**

Big enough that the control group collects enough conversions to detect the lift you care about within the time you can wait. Lower conversion rates and smaller expected lifts need bigger holdouts or longer tests; the holdout simulator on this site shows the trade-off.

## More guides

- [01 **What is incrementality in app marketing?** Incrementality in app marketing explained: the share of installs and revenue your ads really caused, how it differs from attribution, and how to measure it.](https://incrementalitycalculator.com/learn/what-is-incrementality/)
- [02 **Holdout, geo lift, ghost ads, PSA and on/off tests** Compare incrementality test types for app marketers: user holdouts, geo lift, ghost ads, PSA and on/off tests, plus how to run a geo lift test for installs.](https://incrementalitycalculator.com/learn/incrementality-test-types/)
- [03 **How to read lift test results without fooling yourself** How to read an incrementality lift test: lift, confidence interval, p-value and incremental installs, the mistakes that turn noise into wins, what to report.](https://incrementalitycalculator.com/learn/read-lift-test-results/)
- [04 **Incrementality on Meta, Google and Apple Search Ads** How to test incrementality on Meta, Google App Campaigns and Apple Search Ads: what each platform's lift tools offer, their published rules and the fallbacks.](https://incrementalitycalculator.com/learn/incrementality-meta-google-apple/)
- [05 **Which UA campaigns are cannibalising organic installs?** How to identify which UA campaigns are cannibalising organic installs: brand search and retargeting patterns, a four-step data check, and how to test it.](https://incrementalitycalculator.com/learn/cannibalisation-brand-retargeting/)
- [06 **MMM vs MTA vs incrementality testing** Media mix modelling, multi-touch attribution and incrementality tests compared for app marketers: what each measures, the data needed, how they fit together.](https://incrementalitycalculator.com/learn/mmm-mta-incrementality/)
- [07 **How to act on incrementality results** Turn incrementality test results into budget decisions: incremental CPI and iROAS by channel, moving spend in steps, when to retest, and reading CTV and brand.](https://incrementalitycalculator.com/learn/act-on-incrementality-results/)