Skip to main content
Here we have curated a few FAQs regarding Static and Dynamic Multivariate Testing Q. What is multi-variate testing? Ans. Multivariate testing refers to the presence of more than two variations of a communication that are sent to different users chosen at random to compare the performance of message variations. Q. How are the users distributed across variations?
Ans. The audience is divided into two parts, and each audience receives any one variation of email.
  • In Manual A/B testing, you define the distribution, which remains fixed throughout the campaign. This distribution will not change unless you edit it manually. In this method, the analysis is presented at the end of the campaign, which helps to decide the better-performing variation to utilize in upcoming campaigns.
  • In Dynamic A/B testing, Users are equally distributed across variations at the beginning of the campaign. In this method, the distribution and analysis are continuously made based on the performance of each variation during the campaign’s running time.
Q. Is the message variation sticky in nature, e.g., if a user received Variation 1 of any periodic campaign on Day 1 and is re-eligible to receive the same campaign on any day later, will he/she receive only Variation 1? Our A/B Testing feature is not sticky in nature. Every time, a periodic instance is scheduled to run, we fetch all the eligible users and divide them randomly across message variations. Hence the user may or may not receive the Variation 1. Similarly, if a user becomes re-eligible to receive an already received active campaign, same scenario will be repeated. Q. What is a Control Group?
Ans. A control group is a subset of the customers we are targeting with a particular campaign, who will not receive the campaign and will hence serve as a baseline to compare campaign performance. Q. How do we measure campaign Uplift?
Ans. Campaign Uplift measures the impact of your message variations to drive conversion goal. Control Group performance serves as base-line to calculate the Uplift. You can check this in detail here. Q. How can I identify the users who received a certain variation of my general push campaign? Ans. You can find these users by making a segmentation query by navigating to MoEngage Dashboard > Segment > Create Segment. The query will be: has executed Notification Received Android where campaign id contains <cid>_F_T_GP_AB_< VariationSerialNumber> e.g for cid:57f112bc2b8dbc480b457ffe Variation 1 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_ GP_AB_1 Variation 2 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_ GP_AB_2 Variation 3 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_ GP_AB_3 The resultant query for Variation 1 will be: has executed; Notification Received Android where campaign id contains 57f112bc2b8dbc480b457ffe_F_T_ GP_AB_1 Note: Currently we do not allow to identify members of Control Group. Q. How can I identify the users who clicked a certain variation of my general push campaign? Ans. You can find these users by making a segmentation query by navigating to MoEngage Dashboard > Segment > Create Segment. The query will be: has executed Notification Clicked Android where campaign id contains <cid>_F_T_GP_AB_< VariationSerialNumber> OR has executed Notification Clicked iOS where campaign id contains <cid>_F_T_GP_AB_< VariationSerialNumber> e.g for cid:57f112bc2b8dbc480b457ffe Variation 1 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_ GP_AB_1 Variation 2 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_ GP_AB_2 Variation 3 can be identified with key: 57f112bc2b8dbc480b457ffe_F_T_ GP_AB_3 The resultant query for Variation 1 will be: has executed; Notification Clicked Android where campaign id contains 57f112bc2b8dbc480b457ffe_F_T_ GP_AB_1 Q. What is our statistical approach - Frequentist or Bayesian? Ans. We use Bayesian Testing because the results of Bayesian Testing are easily understandable for marketers plus absorb the priorlearning which refines the result over time. You can read a primer on Bayesian Testing here.