2014/05/08 12:28

Real attribution of retargeting ads (ultimate A/B test)

Retargeting is one of the most successful online marketing features. By using cookies, advertisers can target their ads to users who previously visited the advertiser’s website and, for example, have viewed the product but ultimately did not decide to buy it. Advanced retargeting networks use display banners featuring the exact product(s) that the users previously viewed.

The target audience is narrowed to small group of potential customers, and can be run on low-cost placements. For this reason, retargeting ads usually have very good post-click conversion ROI, much better than in any other type of display advertising.

Retargeting providers (including Google, who name it remarketing) use this argument to persuade advertisers to spend more money on retargeting ads, especially that for many advertisers it is the only form of display ads that brings positive ROI measured by post-click, last-click conversions.

Retargeting visit decision conversion

Self-fulfilling theory

Retargering providers very often claim that retargeting has positive impact on conversions even without clicks, and persuade advertisers to look at post-view conversions. However, this is a manipulation.

Usually, certain time elapses from the click to conversion (see the example below). Czas do konwersji

The retargeting ads are served to visitors who started the purchase process. Some users will click the retargeting ad and buy, but a significant part of them would buy this product without seeing the ads too. Many users who convert may not even view the ads served, but the system will still show the post-view, or precisely: post-impression conversion.

At the moment, the most popular web analytics tools are not ready to measure the actual attribution of the last-interaction and assisting clicks and impressions/views. The fact, that interaction (a click or impression) appeared in conversion funnel, does not prove that it had positive impact.

Numerous studies show that during the conversion process, customers interact with advertisements served by competitors. However, it is hard to believe that these interactions could increase sale of our product, and most likely, competitor ads have negative impact on our conversions.

Of course, we should rather not expect that the impact of our own ads can be negative, but what is the actual attribution to the conversion, is it 1%, 10%, 30% or more? Maybe the ads have negative ROI, because majority of ad spending goes to target visitors who would buy this product anyway?

For sure, retargeting does not have 100% attribution in conversion. This is because we have previously paid for each user who converted from retargeting: in SEM, display, social media, e-mailing. Even if the original visit was a direct visit, it was result of brand awareness investments. For this reason, the post-click conversion ROI of retargeting ads can’t be compared with, for example, search engine sponsored link for non-brand keywords.

Concept of the experiment

In case of remarketing, we can, however, run an experiment that will show the exact ROI of retargeting campaigns.

We assign to each user who visited the website, on a random basis, a Google Analytics custom variable with a value of “A” or “B”. It is a user-level variable (not session-level or page-level), and re-visit does not overwrite the value.

Then we use Google Analytics remarketing list (custom variable “A” and “B”) and display the remarketing campaign only to the users from the “A” list.

retargeting A:B test

In other words, the potentially retargeted users are divided in two statistically identical populations, “A” and “B”, of which only the population of “A” is subject to remarketing. Comparing conversions in both populations, we will be able to determine to what extent the remarketing has increased the number of conversions.

The stages of the experiment

We measured visits and e-Commerce transactions (purchases). The experiment, run on an e-commerce website, was divided into stages:

1. A = B – Both “A” and “B” users see remarketing ads.

2. Remarketing only in the “A” group  – Remarketing in the “B” group has been turned off.

3. A = B – Remarketing for the “B” group has been re-enabled. This stage is split into two parts:

(3a) interim stage (P) – the first three days after “B” remarketing was re-enabled;
(3b) A = B – fourth day and subsequent days.

Retargeting test results

The results

Step 1 (A=B). The purpose of the first step was to ascertain that the two populations of “A” and “B” are statistically identical. Number of visits and transactions in groups “A” and “B” turned out to be almost identical (the difference was much smaller than ± 1%), and other metrics, such as visit duration and pages per visit were almost the same.

    Remartekting test 1

Step 2 (only remarketing “A”). After turning off remarketing in the “B” group, we have obviously recorded decrease of visits in the “B” group, and the number of visits in the “A” group was higher by 5.2%. At the same time the number of transactions in the “A” group was higher by 18.5%.

  remarketing test 2

Step 3a (P). Having restored remarketing in the “B” group, the number of visits in “A” and “B” became identical again, but during the first three days, the “A” group had still about 15% more transactions.

retargeting test 3

Step 3b. On the subsequent days, the number of transactions in “A” and “B” became similar again.

remarketing test 4

How is this possible? This experiment confirmed that the impact of online advertising is not only click-based. Advertising banners work, even without being clicked. Nobody clicks on the classic TV ad, nobody clicks outdoor or radio ads either, but these ads have impact on sales.

Users who saw remarketing ads made 18.5% more transactions. In the same time, in Google Analytics we could see that the remarketing ads caused only 1.7% of all conversions in the “A” group. So, the actual impact was 11x higher than we could see by observing last non-direct click conversion in Google Analytics.

  remarketing test 5

Interesting observation: during the experiment (Remarketing “A” only), Google Analytics has still reported significant traffic and conversions from remarketing source in the “B” group:

retargeting remains remarketing

The reason is that Google Analytics assigns a traffic source to the last non-direct visit. Therefore if the user came to website from remarketing, and then, in the course of the experiment, he re-entered the website directly (direct visits), the source of this visit will be still shown as “remarketing” in standard reports. This, actually, shows that the difference between post-click and “real” impact is even higher than 11x.


In this case, the positive impact of remarketing ads was much higher than the cannibalization. The cost of remarketing ads was much lower than the income from additional 18% transactions. This, however, is not a general conclusion. The result of this experiment can be completely different, depending on:

  • product type and the conversion time lag;
  • competitive environment – with low competition retargeting may have smaller impact;
  • seasonality – the remarketing impact can be different in case of last-minute deals;
  • retargeting strategy – Non-optimized retargeting can more likely cannibalise other campaigns.

For this reason, we encourage our clients to conduct this type of experiment individually. Please contact us if you need assistance.

You can read this article in Polish language (extended version).


Witold Wrodarczyk

Dyrektor Operacyjny Adequate Interactive Boutique Google Analytics & AdWords Qualified Professional

Adequate - Interactive boutique

ul. Okopowa 47/23
01-059 Warszawa
(+48) 22 299 50 28

(Polski) Nasza strona korzysta z plików cookie w celu realizacji usługi zgodnie z Polityką Prywatności.