Optimize campaigns through A/B testing

  • Created : Aug, 21, 2020
  • Last Updated : Dec, 08, 2021

What is campaign management and optimization?

Campaign management is a tedious task and optimizing them takes double the efforts. Why? You must be in sync with the interests of the audiences as to what appeals to them most. 

It is inevitable for marketers to engage users with the best possible version of the campaign that can easily convert.

What are the ways to optimize the campaigns :

You can use the marketing automation tools (AB testing and multivariate testing) to optimize the campaigns and further send the campaigns to the audience. 

How can you manage campaigns in NotifyVisitors :

NotifyVisitors enables you to create campaigns and optimize them through AB testing & multivariate testing. At the same time, you can optimize and send campaigns to the targeted audience.

Let’s have a look at how you can optimize the campaigns :

  • The very first tool to optimize your campaigns is AB testing and Multivariate testing through which you can create variations of the original version. i.e. the control version of the campaigns.
  • Split delivery: Choose the ratio of the audience whom you are intending to send the campaigns. i.e. Sending the variations to a specific ratio of audience.
  • Target to a specific audience: Send the campaigns only to a  set of audience. Read further to know more.
How to optimize your campaigns in NotifyVisitors dashboard :

1.AB testing: It is one of the most efficient tools to optimize your campaigns. It allows you to create two or more versions of your campaigns. You can create different versions with the combination of different elements such as a different call to action buttons, different titles, and images for your campaigns.


How to optimize the campaign in NotifyVisitors dashboard through AB testing :

  • Once you have created the variants for the control version of the campaign, pick up a ratio of the audience whom you want to send the variations of the campaigns.
  • The variant winner will be determined on the basis of the highest amount of clicks on a specific campaign variation
  • Next, you can deliver the winning variation to the rest of your targeted audience.

Follow the steps to know how to create ab test push campaigns in NotifyVisitors :

  • Navigate to the NotifyVisitors dashboard> Notifications> Create new notification.
  • Click on the campaign type and give a name to your notification.
  • Once you assign the name, you will see a plus button as soon as you save the name of the notification. 
  • Click on the "Plus button" from where you can create a variation of the control version or copy the existing one and make changes to it.

          add push variation           

  • In order to create a variation, you have to first create a control version of the push. See below for more clarity, give the title and image to the notification.

      Control version -

  push campaign

  •  Now create a variant by clicking on the "plus button" where you will view the option to  "create a new variation". Click on it and get started. You can add a different message and add an emoji to make your notification more engaging.


         push variant

  •  After creating the variation, its time to set targeting rules. Here you can click on the targeting rules and view a/b test settings          
  1.  View the two options of ab testing and split delivery. You can split test your push notifications and ab test them as well.
  2.  Click on the split delivery option and send the control version and variant version of push in an equal ratio of traffic. Click on the schedule notification and determine the results of your split test. See which one of the two wins.

         split delivery

c. Click on the A/B Test Settings to A/B test your push campaign, divide the traffic into a 60-40 percent ratio.

Now here is a difference lies in the A/B or multi-variate Test and Split Delivery Test, here In A/B Test you have to send the control version and the variant in a determined ratio(as shown below). After determining the ratio, specify the waiting time. Once you get the results as to which one of the two wins, our system will send that winning variation to the remaining users.


ab test settings

  • Similarly, you can create a duplicate copy of the existing push and make changes to it and follow the similar process of determining the results. 

  duplicate copy

How to determine the audience for your campaigns :

  • Send the campaigns to the segmented audience on the basis of their past behavior.
  • Sending campaigns to the live user segments.

Let us see both the above insights in brief :

  • Send campaigns to the already segmented audience :audience-segment

Run an AB test on the campaigns and send them to a segmented audience on the basis of their past behavior on your apps or website.

   How to execute the process :

  1.  First of all, create the variants of the campaigns.
  2. Determine the campaign reach. i.e. Users you want to engage your campaigns with.
  3. Choose a testing percentage of users out of the total no. of users and send the campaigns to that percentage of users in a specific ratio.
  4. Determine the winning variant.
  5. Send the rest of the audience that winning variant.

For example: Suppose you have created a push campaign and you want to optimize the campaign. Create three variants of it, variant A, variant B, and C with different combinations of elements. Say if your campaign reach is 3,00,000 users. Choose a test population for your variants to see how the audience reacts to them. Say it is 20 percent, we will be sending variant A to the 20,000 users and variant B to a different set of the 20,000 users and variant C to another set of 20,000 users. If the variant A  wins then we will automatically send the winning variation to the remaining 2,40,000 users.

  • Send the campaigns to the live segments : 

When you are sending the campaigns to the live user segments, the user will receive the message immediately if the user qualifies the criteria you have determined. In other words, If you want to send the campaign to the users who make a purchase and the customer buys a product, the campaign will be sent immediately to him as they match your determined criteria.

For example - You want to send campaigns to the user while the user subscribes. As you cannot determine the amount of the users for your campaigns, you can make an assumption for the qualified audience and send the variants with a specific ratio. Say you have selected 1000 users as your test audience, now send variant A to the 500 users and variant B to another 500 users. You can determine the winning variation through clicks and send the winning variation to the rest of the users

You can determine the test audience on the basis of the previous and similarly triggered campaigns. In case the test audience exceeds the total amount of users then there will be no winner.

2. Split delivery: Split delivery determines how you can optimize the campaign by first determining the percentage of your target audience. Then the campaigns are delivered accordingly.


The major difference between AB testing and Split testing is as follows:

In AB testing, you can add multiple variants of the campaigns and in split delivery, you have to determine the percentage of users for the variants such as delivering variant A to 40 percent of the users and variant B to 60 percent of users.

How to deliver the campaign :

Let’s understand the use case with an example :

Split delivery comes into handy when you want to send the campaigns to a subset of the target audience. On the contrary, if the no. of qualified users exceeds the specified number of campaigns, then unfortunately you cannot send.

For example - You want to reward your users by distributing the specific number of coupon codes. In case the total no. of users for your campaign exceeds the number of coupons, you then have to limit the users to the number of coupons you want to distribute.

How split delivery works in NotifyVisitors dashboard :

  • You can control the campaign creation flow by deciding whom you want to send the messages to. You can limit the delivery of the campaigns to the number of users by choosing the amount.
  • You can select the segmented users based on past behavior and send them the campaigns. In the case of live user segments, send the campaigns to the users as they will qualify. The campaigns will be sent to the users until a total amount of quality is specified.

Note: You can limit the number of users for a campaign per day and select the time-zone. i.e. user time-zone to schedule the campaign.

3. Email AB testing: The email AB testing feature allows you to compare different versions of your email campaign so that you can find the best one out of many. 

Note: Pick up the size of the audience for the testing and know the winning version through the total no. of clicks. Consequently, send the winning variation to the rest of the audience.

How can you set up the email AB testing in the panel :

  • Create variations for the email campaign. Modify the elements of the campaign.
  • Run AB tests on the campaigns.
  • You can choose the segment that you want to target.
  • Choose the frequency of communication.
  • Pick up the sample size ( percentage of users to whom variants will be sent) and set the trigger time after the variations are sent to the specific percentage of users.
  • Determine the winning variation through the amount of the number of clicks on each variation.

For example  :

You have created an email campaign to nudge users to buy your products with catchy offers. You can send the campaigns to segmented users on the basis of their performed actions. Pick up a sample size of the targeted audience to whom you want to send the variants. Suppose you have decided to send the variants to 10 percent of the users. You can now send the variant A to 4 percent and variant B to 6 percent. Set the time trigger and send the winning variation to the rest of the users.

4. Multivariate testing: It may take various attempts to optimize the campaign by testing the multiple variations of a campaign. Also, varying the elements such as images, CTAs, banners, and much more somehow zeroes the effectiveness of the campaigns after so many attempts. In such a scenario, it's difficult to increase and enhance conversions. But NotifyVisitors simplifies your dilemma. 

Now, You can easily determine a winner and target your audience in a much better way.

Let’s see how can you test multiple variations of the campaign in NotifyVisitors :

  • Create multiple variations of the campaigns.
  • Send the winning variations automatically to the target audience.

Just two steps and you are good to get started.

The variations are tested and the most effective/the winning variant is sent to the larger audience.

5. Variations of push, SMS, and other campaigns: When we talk about the campaigns for user engagement. We are usually referring to the push, SMS, Email, or WhatsApp journey.

You can create variations of the campaigns and test all the aspects of the variation such as the image, CTA buttons, links, and so on.

How to automate the process of testing and winning variation in NotifyVisitors Dashboard :
  • NotifyVisitors dashboard enables you to automate the process of testing variations and at the same time sending the winning variation to the audience. 
  • First of all, you can test the variations by sending them to the smaller test audience. Afterward, you determine the winning variant and send it to a larger audience.

Let’s get acquainted with some of the key terms in the process :

  • How to determine the size of the test audience: Generally the variations are equally distributed amongst the test audience in order to test. From where does the test audience come? It comes from the target audience of the campaign.
  • Determine the duration of the test: It is the duration for which the variations are tested against the control version of the campaign.
  • Winning criteria: It is your determined goal - the foundation on which you are testing the variations. In other words, winning criteria is your end goal on the basis of which you want to measure the success of your campaigns. It could be conversions, clicks and impressions, and open rates.
How to send winning variation automatically :
  • You can determine the winning variation on the basis of the winning criteria you have set for the campaigns.

Points to remember :

  • The test results will only be determined for the campaigns which are sent successfully and not for the campaigns whose delivery is failed.
  • In case the winning rates are the same for all the variations, then variant A will be sent to the target audience.
  • You can watch out the performance of the variations to analyze minutely.
How to send winning variations for different types of campaigns :

1. How to test one-time campaigns: The test audience for the one-time campaign variations are distributed equally. You can determine the winning variant on the basis of the winning criteria.

 Configure the settings for one-time campaigns :

  • Size of the test audience: Size of the test audience can vary between 5 to 25 percent of the total target audience.
  • Test duration: You can specify the duration of the test in hours and minutes.

2. How to test recurring campaigns: Recurring campaigns are sent periodically to the audience after a specific amount of time.

 Configure the settings for recurring campaigns :

  • Size of the test audience: The test variations must be distributed equally amongst the target audience for the first time you launch the campaign.
  • Test duration: The test time varies depending upon the frequency of the recurring campaign. For example- It could be 24 hours for a daily recurring campaign, 7 days, and 24 hrs for a weekly recurring campaign. Also, for the monthly campaigns, it is 30 or 31 days and 24 hours.

3. How to test the journey campaigns: Journey campaigns are usually the campaigns that are sent to the users in the context of their attributes and events performed by them. These campaigns are highly personalized. The campaigns define the journey and based on different conditions.

Configure the settings of journey campaigns :

  • Size of the test audience: The size of the test audience depends upon the entries or the number of times users enter the campaign block of the journey. The testing of the variation will continue unless and until a good count of entries occurs in the block. Divide the variations amongst the first 500 entrants who enter the campaign block.
  • Test time duration: Once the specific number of entries occurs in the campaign’s block, you can wait for the closest time slot to calculate variation’s performance. Also, you can send the winning variation to all the users who enter the campaign’s block. 

Managing campaigns and optimizing them manually takes a lot of time. NotifyVisitors makes it easy and stable for you to manage different types of campaigns for your business growth.