---
title: Holdout Test Simulator: Size an Incrementality Test
description: Simulate an app incrementality holdout test day by day on sample data: pick the holdout share, audience and true lift, and see when the result becomes clear.
url: https://incrementalitycalculator.com/labs/holdout-simulator/
---

[Start](https://incrementalitycalculator.com/) / [Labs](https://incrementalitycalculator.com/labs/) / Holdout test simulator

# Holdout test simulator

How big should a holdout group be, and how long should the test run? Change the settings and watch a simulated test unfold.

**Short answer**

A holdout needs to be big enough that the control group collects enough conversions to separate the lift you expect from day-to-day noise. In the sample settings, a 10% holdout with a 1% install rate needs about 39k control people to detect a 15% lift, which is about 10 days at this audience size. Halve the expected lift and you need roughly four times as many.

> Interactive tool (holdout simulator): runs in the browser at https://incrementalitycalculator.com/labs/holdout-simulator/#holdout-sim. Its text content follows.

**Holdout test simulator** Runs in your browser. Sample data.

Everyone eligible for the campaign, test and control together.

How many of them would install anyway (the control group's rate).

The answer the test is trying to find. In real life you never see it.

Share of people deliberately not shown the ads.

Sample data: a made-up app to show the method, not a benchmark.

Estimated lift (filled = interval clear of zero) 95% confidence interval True lift

**Control people per day**

4,000

**Planned length (80% power)**

10 days

**Interval first clears zero**

day 6

**Clear of zero from**

day 9

**Estimated lift on last day**

+14.8%

**95% interval on last day**

+8.7% to +20.9%

The interval cleared zero on day 6, then crossed back. Stopping the first time it looks good is how false wins get reported. Decide the length up front (about 10 days here) and read it once.

Random draw number 2. Each draw is seeded, so the same settings always give the same path.

Default draw, day by day (cumulative). This table is what the lab shows before you change anything.

| Day | Test installs | Control installs | Lift | 95% interval | Clear of zero |
|---|---|---|---|---|---|
| 1 | 383 | 38 | +12.0% | -21.6% to +45.5% | No |
| 2 | 782 | 76 | +14.3% | -9.4% to +38.1% | No |
| 3 | 1,200 | 119 | +12.0% | -6.9% to +31.0% | No |
| 4 | 1,627 | 163 | +10.9% | -5.3% to +27.1% | No |
| 5 | 2,046 | 199 | +14.2% | -0.4% to +28.9% | No |
| 6 | 2,475 | 241 | +14.1% | +0.8% to +27.4% | Yes |
| 7 | 2,872 | 299 | +6.7% | -5.2% to +18.6% | No |
| 8 | 3,272 | 339 | +7.2% | -4.0% to +18.4% | No |
| 9 | 3,727 | 364 | +13.8% | +2.9% to +24.6% | Yes |
| 10 | 4,218 | 405 | +15.7% | +5.4% to +26.0% | Yes |
| 14 | 5,892 | 585 | +11.9% | +3.4% to +20.5% | Yes |
| 21 | 8,870 | 846 | +16.5% | +9.4% to +23.6% | Yes |
| 28 | 11,839 | 1,146 | +14.8% | +8.7% to +20.9% | Yes |

## How to read the chart

- **Violet line:** the lift the test would report if you stopped on that day. Filled dots mean the 95% interval no longer includes zero.
- **Shaded band:** the 95% confidence interval. It narrows as people and installs pile up, roughly with the square root of the sample.
- **Dashed line:** the true lift. A real test never shows you this; here it lets you see how far early estimates can wander.
- **Dotted line:** the planned length, worked out before the test from the expected lift with 80% power.

## What changes the length of a test?

Four inputs decide how long a holdout test needs, and the simulator lets you move each one:

- **Install rate without ads:** rarer conversions mean fewer events per day, so a low-rate event (a first order rather than an install) needs a much bigger test.
- **Expected lift:** the required sample grows with one over the lift squared. Detecting a 5% lift takes about nine times as long as a 15% lift.
- **Holdout share:** the control group is usually the bottleneck. Going from 5% to 10% roughly halves the test length; above about 30% the gains flatten while the cost of not advertising rises.
- **Audience per day:** more people reached means more conversions per day in both groups.

## The maths

Required control-group people, for 95% confidence (two-sided) and 80% power, with *k* test people for every control person:

n control = (1.96 + 0.84)² × [p 1(1 − p 1) ÷ k + p 0(1 − p 0)] ÷ (p 1 − p 0)²

With the sample settings (p 0 = 1%, p 1 = 1.15%, k = 9) that is about 38,941 control people. At 4,000 control people a day, that is about 10 days. Each simulated day draws test and control installs from a normal approximation to the binomial with a seeded random generator, so a given draw number always reproduces the same path.

## Read next

- [How big should my holdout group be?](https://incrementalitycalculator.com/learn/holdout-group-size/)
- [How to read lift test results without fooling yourself](https://incrementalitycalculator.com/learn/read-lift-test-results/)

Run this on your own data

To size a real holdout you need your own install rate without ads: daily organic installs by country over a few ordinary weeks, exported from your MMP. Put that rate into the simulator instead of the sample one. Linkrunner's free plan covers 25k attributed installs, with no credit card and every feature included.

- [Export daily installs (data export docs)](https://docs.linkrunner.io/features/data-export?utm_source=incrementalitycalculator&utm_medium=referral&utm_campaign=lab-holdout-simulator&utm_content=panel)
- [Incrementality testing, defined](https://linkrunner.io/glossary/what-is-incrementality-testing?utm_source=incrementalitycalculator&utm_medium=referral&utm_campaign=lab-holdout-simulator&utm_content=panel)
- [Create a free account](https://dashboard.linkrunner.io/sign-in?utm_source=incrementalitycalculator&utm_medium=referral&utm_campaign=lab-holdout-simulator&utm_content=panel)

## Questions

**How big should my holdout group be?**

Usually between 5% and 20% of the audience. Use the smallest share that still reaches the planned sample within a test length you can live with: a lower install rate or a smaller expected lift both push the share up.

**Why did the result look significant early and then stop?**

Early in a test the interval is wide and the estimate swings, so it can briefly clear zero by chance. Checking every day and stopping at the first good-looking result raises the false-win rate well above 5%. Fix the length in advance.

**What is statistical power?**

Power is the chance the test detects a lift of the planned size if it is really there. At 80% power, a real lift of exactly that size is missed one time in five.

**What is incrementality in app marketing?**

Incrementality is the share of conversions that would not have happened without the ad. It is measured by comparing people who could see the ads with a similar group who could not, so it captures what the ads changed rather than what they were credited with.

**Is attributed the same as incremental?**

No. Attribution credits a campaign when a user touched an ad before installing, even if that user would have installed anyway. Incremental installs are only the ones the ads caused, so they are always equal to or fewer than attributed installs.

**Are the numbers on this site real benchmarks?**

No. Every number is sample data for a made-up app, chosen to show how the method works. Your own incrementality depends on your app, channels and audience, so measure it with your own test.

## Other labs

- [01 **Lift test reader** Enter test and control results and get the lift, a 95% confidence interval, a plain-English verdict and the usual misreadings.](https://incrementalitycalculator.com/labs/lift-reader/)
- [02 **Cannibalisation lab** See how a paid push can be credited with installs that were going to arrive organically, and what that does to the organic line.](https://incrementalitycalculator.com/labs/cannibalisation/)
- [03 **Attributed vs incremental CPI and ROAS** Two sample channels side by side: how ranking them on attributed CPI and on incremental CPI can point budget in opposite directions.](https://incrementalitycalculator.com/labs/incremental-roas/)