Control Group In Flows

Overview

In MoEngage Flows, a Control Group is a percentage of users who enter the flow and traverse through the stages but are intentionally held back from receiving any of the campaigns you have configured within that flow. While it might seem counter-intuitive to withhold a marketing message, the Control Group is essential for accurately measuring the true impact of your engagement strategy.

By not sending any messages to this group, you establish a baseline for organic user behavior. This baseline shows how many users would convert on their own, without any influence from your flow’s campaigns.

The real value of this approach becomes clear when you compare the conversion rate (CVR) of the users who experienced the flow against the CVR of the Control Group. This comparison allows you to validate two key assumptions:

• Positive Impact: The flow is genuinely encouraging users to convert at a higher rate.
• Necessity: There is a clear need to engage users with this specific flow to drive desired actions.

How to Interpret the Results:

• If the Conversion Rate (CVR) of the main flow is higher than the Control Group’s CVR, it demonstrates the effectiveness of your campaign. You can quantify this success by calculating the “conversion uplift” – the percentage increase in conversions directly attributable to your flow.
• Conversely, if the Control Group’s CVR is higher than or equal to the flow’s CVR, it signals a critical insight: your flow is not performing as expected. This outcome suggests that users are more likely to convert organically, or worse, that the messages in your flow might be discouraging them. In such a scenario, the data indicates that the flow should be paused and re-evaluated, as it is not adding value.

Conversion Uplift

The uplift analysis is the most common way to analyze the extent to which a MoEngage Flow has nudged your users toward a conversion. Uplift is measured against the organic conversions from the control group users and conversions of the users who received at least one of the action campaigns of the flow. Uplift = [(Flow CVR - Control group CVR )/Control Group CVR] x 100.

A 50% conversion uplift signifies that the configured flow has fetched 50% more conversions than what would be possible had the Flow never been configured.

Understanding Uplift Percentage

The uplift calculation is dependent on the stage and the timeframe in consideration. Thus, the upliftment for the flow shown at the top of the Detailed Stats section might vary from the upliftment shown in the Engagement Trends in the Detailed stats as the latter is calculated only for the chosen Channel while the former is calculated for the entire flow.

Example

Let’s take a Flow that has a 36-hour attribution window and has been configured for a Push and an Email campaign.
Default scenario of 36 hr attribution window:

Timeline

• Hour 0 - User Enters flow (90 TG, 10 CG)
• All of them are sent the push campaign Push-1, assuming all of them have received (100% impression/view-through)
• 0 to 24 hours - 5 users**(4 TG, 1 CG)** convert. These users are attributed only to the push campaign Push-1.
• Hour 24 - Email campaign Email-1 is sent out, assuming all of them have opened the email (100% impression/view-through)
• 24 to 36 hours - 8 users (8 TG, 0 CG) convert. These users are attributed to both the push campaign Push-1 and the Email campaign Email-1.
• Hours 36 to 60 - 5 users (4 TG, 1 CG) convert. These users are attributed only to the email campaign Email-1.

The uplift percentage is calculated as shown below for this example.

Uplift Percentages and CVR at the Child Level

Channel	Metrics
Push	TG CVR = 12/90 = 13.3% CG CVR = 1/10 = 10% Uplift = 33%
Email	TG CVR = 12/90 = 13.3% CG CVR = 1/10 = 10% Uplift = 33%

The Uplift Percentage at the Flow level

• Total TG converted users = 16 / 90 = 17.7%
• Total CG converted users = 2 / 10 = 20%
• Uplift = -11.5%

How to configure Control Group?

The option to add a Control Group is available in the Entry Condition node as well as on the flow setting page. The control group is enabled by default and can be changed anytime during the flow lifecycle. The users will become part of the control group only after the flow is published with the control group setting enabled.

Control group users are randomly selected out of the total users entering the trip out of all the users in the Flow’s target audience.

In order to maintain the sanctity of the experiment, we ensure that once a user enters the flow as part of the control group, they would continue to be part of the control in all their subsequent entries into the flow until the control group is removed. You can choose to disable this if required.

How Control Group users are allocated?

Due to the scale at which MoEngage operates, we split users randomly into subgroups and apply the control group split percentage to each of the groups on a daily basis. The above process is made with the assumption that an equal number of users can be allotted to the different subgroups on any given day. So there is a need to have a minimum number of users to enter the flow for the split to be maintained. Also, with time the system scales down/up the number of users allotted as CG depending on the condition so that the % of users in CG is always close to the defined value.

When marketers check the “Users part of Control Group should continue to be part of it for their subsequent entries”(as shown in the above image), the system makes sure that a user marked as CG will always be a CG user. So, if CG users repeatedly enter the Flow more than the other type of users, then the % between CG trips and overall trips is going to be greater than the defined value. However, the % split on the unique user level will be as defined.

For example, let’s take the case where a marketer defines a Flow with a 20% Control Group split and wants “Users part of the Control Group should continue to be part of it for their subsequent entries”. Let users U1, U2, U3, U4, and U5 enter 1,2,3,4,5 times, respectively. Let U5 be marked as the Control Group. In this situation:

Total Trips = 15
CG trips = 5
However, the CG split has been maintained at 20% as only 1 in 5 users was marked as the Control Group.

How big should the control group be?

You can make the control group as big as 95% or as small as 1%, depending on your target audience. For example, if you expect 100k users to enter the flow then 1% CG would be enough. The smaller the target group the larger the control group should be configured to effectively measure the efficacy of the flow.

How do Control Groups work?

Users are randomly allocated to the control group at the time of entering into the flow. The control group users after entering the flow traverse through the flow as per their behavior during the trip and eventually exit the flow. These users are not sent any campaign during their flow trip.

Conversion tracking for the Control group

Similar to the flow conversion tracking, control group users’ conversion tracking starts as soon as they move past any one of the action campaigns configured in the flow. We assume that these users have received the campaign and we start tracking their conversion for the configured attribution window. As with the non-control group users also, conversions performed by CG users are attributed to the action campaigns as well as the flow.

Since the control group users will skip all the action campaigns, we get an interesting problem “What should the conversion attribution window be for the flow in such a case?” We can try one of the following solutions:

• Approach 1: Calculate the longest path of the flow and keep waiting for that much period for control group users to convert after entering the flow. But this assumes that all the control group users who had not been part of the control group would have gone traversed the flow through the longest path, which seems very unlikely.
• Approach 2: Wait for the attribution window period for control group users to convert after entering the flow. This approach gives an unfair advantage to the control group users if the configured flow is long. For example, if the largest path of the flow takes 3 days to traverse and exit, and if the attribution window of the flow is 24 hours. Then, a control group can enter and convert for the flow approximately thrice before the flow user completes one trip through the longest path.
• Approach 3: We let control group users traverse through any of the possible paths based on their behavior after they enter the flow and continue to track the conversions. After they exit the flow from one of the nodes we track the conversion for an additional period that is equal to the attribution window for the flow. This way we let the control group user behavior govern the period till which the conversion should be tracked.

We identified the third approach to be the least assumptive and most intuitive. If you have any concerns or suggestions, please contact your MoEngage Customer Success Manager (CSM) or the Support team.

Editing the control group

While Marketers can choose to change the Control group variation at any time during the lifetime of the flow, we recommend against it as changing the traffic distribution may invalidate the experiment results.

For Example, let’s say you publish your flow with a 50% control group on Day 1 and 2000 user entries happen on that day. 1000 Flow user entries result in 250 converted trips and 1000 control group(CG) user entries fetch 200 conversions. Hence, the CG CVR is 20% whereas flow CVR is 25% which is a 25% uplift from the control group.

Now the next day if you change the CG variation to 10% and 1000 user entries happen on the next day. Now out of 900 regular flow entries, 45 lead to at least one conversion( i.e. become converted trips), and out of 100 CG user entries only 4 converts. Now for Day 2, CG CVR is 4% and Flow CVR is 5% which is again a 25% uplift from the CG.

Now if combine the Day 1 and Day 2 results you will get CG CVR as 18.55% [(204/1100)x100) whereas the flow CVR during the same period drops down to 15.52% [(295/1900)x100] which gives an overall negative uplift of -16.33%.

Even though Flow CVR is better on both days individually, the Control group wins the experiment. Changing the CG distribution has eventually altered the results of the experiment, this in statistical theory is known as Simpson’s Paradox. Because of this, we strongly recommend fixing the control group variation before publishing the flow and not changing it unless you absolutely have to or you have validated the efficacy of that flow over time.

Removing the control group

If you are observing conversion uplift for your flows over a significant period of time, we recommend you disable the Control group setting for your flow. Once this is done, all the users entering the flow can potentially receive the action campaigns configured in the flow.

If the Control group users re-enter the flow, they will also be eligible to receive the action campaigns. The list of previous control group users is also purged, so if the control group is added again, the CG user distribution will start from scratch without any influence from the previous control group.

In such a scenario, there is no way to restore the earlier configuration, but you can choose to add the CG again and restart, analyzing the flow performance.

Finding the users marked as the control group

Once you have removed the control group, you might want to find them to target the users that were part of the control group and did not receive any of the flow action campaign events after going through the flow.

Users Marked as the Control Group for the Flow

You can create a segment with all the users who have performed the “User entered flow” event with the Flow Id Attribute value being the flow-id of your flow, with the “with control group” attribute as “true” and “Control Group Type” as Flow.

Users Marked as the Global Control Group

You can create a segment with all the users who have performed the “User entered flow” event with the Flow Id Attribute value being the flow-id of your flow, with the “with control group” attribute as “true” and “Control Group Type” as Global.