Incrementality testing is how companies can truly measure the impact of their marketing activities, allowing for a better understanding of how media or ads generate new conversions. So, what does measuring incrementality mean for marketing, and what benefits does it provide? Let’s dive deeper.

What is Incrementality Testing?

An incrementality test measures the rate of new conversions in relation to a marketing activity, content, or ad. As a result, it can act as a measure of effectiveness for a company’s output. While attribution measures tend to be fine for most businesses, incrementality is better at finding causal relationships between actions and growth.

One of the other reasons to opt for incrementality testing in marketing analytics is that a lot of the numbers many platforms provide can be unreliable. Many platforms have a vested interest in overstating their contributions to MROI, conversions, or sales. Many of the stats they provide can often be rose-tinted as a result. Incrementality marketing measures allow businesses to cut through the noise and crunch numbers that better reflect reality.

Consequently, marketers and companies can better understand how to allocate the funds for their marketing budget. While incrementality testing can take longer to implement and is less convenient than many tools that are readily available within standard analytics dashboards, it provides better insights to build future operations on.

The steps of incrementality tests include:

• Design: Set your research goals, KPIs, minimum detectable lift, and pick a test/control unit.
• Randomise or Match: Use matched‑market (similar groups in different markets with similar external factors) or synthetic control to build a valid study.
• Launch & Guardrail: Enforce spending limits and rates, monitor with benchmarks of pre-trend audience fit, and track power.
• Read‑out: Report lift percentages, confidence intervals (p-value), iROAS (rate of ad spend), and heterogeneity by region/audience.
• Calibrate models: Feed learnings into your marketing tools and day‑to‑day attribution to keep them honest.

Let’s look at how incrementality works and what tools you can use to measure it.

Incrementality Measurement

Incrementality requires two fundamental elements: a control conversion rate and a test conversion rate. The former acts as a control group, much like a standard effectiveness study in any other field. This controlled experiment allows the company to get definitive numbers on how their activities are faring online without having to rely on potentially flawed data collection. Moreover, incrementality tests get as close to a causative test (as opposed to correlative) as possible.

The incrementality formula is “(Test Conversion Rate – Control Conversion Rate) / (Test Conversion Rate) = Incrementality”.

Measuring incrementality in marketing thus requires measuring the difference between a control group unexposed to the marketing activity we’re measuring and a group that was exposed to a specific marketing action, and comparing it as a fraction of the test conversion rate.

Measuring marketing incrementality should be done on the basis of each individual activity. Needless to say, you may generate false positives if you bundle the effect of all activities together and may not be able to separate effective strategies from ineffective ones.

However, it’s worth noting that there are multiple types of incrementality tests. Let’s go over some.

Holdout Experiments

A holdout experiment isolates a media channel and segments the audience away from it. This allows for the observation of their conversion behaviour in comparison to a control group who receive that specific media. In this case, the group targeted with the media is the “control” group, as they represent the typical audience. In this case, a decrease in buying behaviour or interest would be seen as a sign of the effectiveness of the media (i.e., exposure to media correlates to desirable outcomes).

Scale Experiment

Scale experiments are the opposite of holdout studies. Here, the media channel is “scaled-up” in investment, subjected to an audience, and then the study measures conversion behaviour with an audience that received the normal level of spend. As the name suggests, this one tests the effectiveness of increasing the scale of spending on an audience. This can be great for making advertising spend more lean.

Multi-treatment Incrementality Testing

This is an experimental design where an audience is split into more than two test groups (“cells”) for the purpose of evaluating different combinations of media and comparing their relative lifts against a holdout group. It can be used to find overlapping effects in channels.

You can do an experiment with three test cells, such as a search ads holdout, a social media holdout, and a combined holdout for both. Each of these has to run up against a control group, which is the typical audience. The experiment will highlight the individual contribution and the combined effect of each channel. This helps them determine whether the relationship is synergistic (meaning they enhance each other’s performance) or cannibalistic (one channel diminishes the impact of another).

Audience Splitting

There are two main types of splits for audiences when doing incrementality tests: Known-Audience splits and Geographical splits.

A Known-Audience Split classifies individual users from an existing user list into a different media test. Therefore, this is best when you have data on user-based targeting and associated tactics. It would apply best to email, catalogue, SMS, etc., where user information is readily available.

The test needs to account for and control for the recency, frequency, and monetary value of recent purchases and user eligibility (e.g., opt-in vs. opt-out) to receive the media in question or any other related media.

In contrast, a geographical split is for audiences that are unaddressable. This splitting method foregoes pre-existing lists and is best when Known-Audience Splits are infeasible. This can be best for broad targeting like social prospecting, CTV prospecting, and paid search.

The geo-split method finds specific markets within a broader region, such as a country, that are statistically representative of that broader region. The method then requires grouping these markets into a test cell for different types of testing. A treatment, such as Google Holdout, is then applied to this test, checking to see if adding or removing marketing activities affects behaviour. The results are then compared to a control group.

Best Channels for Incrementality Testing

When it comes to channels, incrementality is best divided alongside the purpose of the test or audience splitting method. Certain channels like SMS, email, catalogues, and direct social media contact allow for known-audience splitting. Conversely, a wider test with a mass media approach will allow for scale experiments and geo-splitting. Multi-treatment tests can be best administered when you have enough information about the audience and what channels they are likely to be on.

Best Incrementality Testing Tools

• Lifesight: Provides a great platform for marketing incrementality tests with causal modelling and causal graphs. Its design is user-friendly with easy optimisation and AI recommendations.
• Haus: Great for performing an incrementality test on Search, YouTube, and social campaigns. It’s quick and easy to set up, and offers pricing based on the specific tests, geographic region, and number of experiments.
• Measured: While more expensive than others, this may be best for extensive eCommerce connectors, with retail‑friendly reporting, and is ideal for consumer brands.
• Billy Grace: AI-powered, accurate tracking with great ease-of-use. It provides smart insights into budget allocation and promises GDPR-compliant first-party data.
• Workmagic: This one is first and foremost an incrementality tool. It has great geographic features that can divide groups accurately and quickly. It also promises a fully automated experience.
• LiftLab: LiftLab is great for every stage of the funnel, providing up-to-date information. The company promises comprehensive customer support for all its users.

The latest incrementality testing news indicates that companies are focusing on it rather than classic attribution modelling. If you’re looking for an agency to help build a robust marketing strategy with robust analytics, look no further than Promoguy! Check out our marketing services page to learn more.