Overview - Imports

Introduction

MoEngage allows customers to import users and events through files stored in various sources like S3 Buckets and SFTP servers, as well as directly from tables in your data warehouses. MoEngage Dashboard enables you to configure multiple imports directly by defining file locations, configuring one or more file import sources, and import methods.

Supported Sources

MoEngage currently supports importing files from:

1. Amazon S3 Buckets
2. SFTP Servers

MoEngage also supports importing from data warehouses:

1. Snowflake Imports
2. Databricks Imports

Types of Imports

MoEngage can import the following from your sources:

1. Registered Users: These are users who are already registered on MoEngage. You can also use this type of import to update existing users in bulk.
2. Anonymous Users: These are users who must register on MoEngage.
3. Events (Standard and User Defined): MoEngage can import standard events like Campaign Interaction Events as well as your own user-defined events.
4. Auxiliary Data: There are user data temporarily ingested into MoEngage for specific business purposes.

If your system requires IP whitelisting, refer to this document here before setting up the data import.

Prepare the Files

Before configuring the imports on the MoEngage dashboard, make sure that all your files are in the expected format and within the sanctioned limits.

Supported File Types

Source	S3 Imports		SFTP Imports		Upload		All Sources
File Type	Users	Events	Users	Events	Users	Events	Aux
CSV	✅	✅	✅	✅	✅	✅	✅
JSON	✅	✅	✅	✅	❎	❎	❎

All files should be in UTF-8 file encoding.

Naming Conventions

Ensure your every file has a unique name. MoEngage skips any file name it has already encountered, regardless of whether the previous import was successful, partially successful, or failed.

The file name is made up of two parts: <import type>_<date time format>.<file_extension>.. The final name depends on the type of import and the date-time format you pick.

Import Type

• Registered users
  • File names should start with the prefix registered_user_data_
  • Example: registered_user_data_01311997.csv or registered_user_data_01311997.json
• Anonymous users
  • File names should start with the prefix anonymous_user_data_
  • Example: anonymous_user_data_jan_2022.csv or anonymous_user_data_jan_2022.json
• Events
  • File names should start with the prefix <event name>_
  • In certain cases, the Event Name and Display Name can be different. To manage your events, go to the Data Management page in MoEngage.
  • For example, if the event name is “Purchase Summary”: Purchase_Summary_31011997.csv or Purchase_Summary_31011997.json
  • For example, if the event name is “Purchase_Transaction”: Purchase_Transaction_19970131_1259.csv or Purchase_Transaction_19970131_1259.json

Make sure not to use “moe_” as a prefix while naming events, event attributes, or user attributes. It is a system prefix and using it might result in periodic blacklisting without prior communication.

File Extensions Supported

• Imports without having File Encryption enabled support only the following extensions: .csv, .json
• Imports having File Encryption enabled support only the following extensions: .csv.asc, .json.asc, .csv.pgp, .json.pgp

Date Time Format

Your file names should end with the suffix <date time format>.<file extension>This “date-time” represents the day on which the import executes. For example, if the import is executed on “7th January 2022”, and the selected date-time format is ddmmyyyy, MoEngage looks for all the files that end with the suffix 07012022.csv or 07012022.json

MoEngage supports multiple date-time formats:

Type	Description	Example
{mm}_{yyyy}	mm: Month from 01 to 12 yyyy: 4-digit year
{ddmmyyyy}	dd: Days from 01 to 31 mm: Month from 01 to 12 yyyy: 4-digit year	31012022
{yyyymmdd}	yyyy: 4-digit year mm: Month from 01 to 12 dd: Days from 01 to 31	20220112
{mon\_yyyy}	mon: 3-letter abbreviation of the month from jan to dec (in lowercase) yyyy: 4-digit year	jan_2022
{yyyymmdd}_{hhmm}	yyyy: 4-digit year mm: Month from 01 to 12 dd: Days from 01 to 31 hh: 2-digit hours from 00 to 23 mm: 2-digit minutes from 00 to 59	20220131_1259

While setting up the import, you can configure the date-time formats for the file names on the MoEngage dashboard.

MoEngage also supports custom filenames, as long as the following naming convention is followed:

Naming Convention: <custom name>_<date time format>.<file extension>.

Filenames should contain the custom prefix name and date-time format separated by ”.” The custom name should consist of letters, numbers, a space, and ”.”

For example: abc_31012024.csv or abc_31012024.json

Valid File Name Examples

The following are examples for valid file names:

• Registered users import with date time format {yyyymmdd}_{hhmm} have file names like registered_user_data_20220131_1259.csv or registered_user_data_20220131_1259.json
• Anonymous users Import with date time format {mon_yyyy} have file names like anonymous_user_data_feb_2022.csv or anonymous_user_data_feb_2022.json
• Item purchased event import with date time format {ddmmyyyy} have file names like Item Purchased_31012022.csv or Purchased_31012022.json

File Structure

MoEngage supports importing data from CSV and JSON files. Your files must adhere to the following rules:

CSV Files

MoEngage supports importing user and event data using Comma-Delimited CSV files. All files should be in UTF-8 file encoding

• The first row of your file must be the column names. MoEngage always treat the first row of your file as the header row.
• Ensure there are no duplicate column names in your file. Do note that column names are case-sensitive.
• Ensure there are no missing column names.
• MoEngage only supports the comma (,) as a delimiter.
• Avoid empty rows (either the row is blank, or all the columns in that row are blank).

MoEngage recognizes the first row of your files as the header file and parses the column names. You can then map them to MoEngage attributes on the MoEngage dashboard or choose to upload a manifest file.

Sample Files

Here are some sample files:

• Import Registered Users Sample File
• Import Anonymous Users Sample File
• Import Events Sample File
• Sample Manifest/Mapping File

JSON Files

MoEngage supports importing user and event data using Newline-Delimited JSON (NDJSON) files. All files should be in UTF-8 file encoding.

NDJSON Format

Your JSON files must adhere to the NDJSON format, having a .json file extension, meaning each line in the file is a complete, self-contained JSON object, and each object is separated by a newline character:

JSON

{"user\_id": "user123", "name": "John Doe", "email": "[e-mail]", "address": {"city": "New York", "zip": "10001"}}
{"user\_id": "user124", "name": "Jane Smith", "email": "[e-mail]", "address": {"city": "Los Angeles", "zip": "90001"}, "preferences": ["email\_updates", "sms\_alerts"]}

Here are some sample files:

• Import Registered Users Sample File
• Import Anonymous Users Sample File
• Import Events Sample File
• Sample Manifest/Mapping File

File Structure and Mapping JSON Levels

When you upload or select a JSON file, MoEngage generates a flattened, tabular preview of the data. This preview will represent keys up to two levels deep (root level + one nested level) as distinct columns:

JSON

{
Level 1 "item-id": 123,
Level 1 "item-type": "books",
Level 1 "item-cost": {
Level 2     "amount": {
Level 3        "cost-price": 100,
Level 3        "discount": 0
},
Level 2     "currency": "USD"
}
}

Mapping Capabilities

• Mapping Primitive Keys: You can map any primitive key from Level 1 or Level 2 directly to a standard MoEngage attribute.
• Mapping Entire Objects or Arrays of Objects: You can map a complete JSON object or an array of objects to a single MoEngage attribute of type “Object” or “Array of Objects” respectively. When an entire object is mapped, its internal structure and all its nested keys are preserved and imported. It is not possible to rename these keys within MoEngage.
• Cherry-Picking Fields: If you only want to map specific primitive fields from an object (up to Level 2) and not the entire parent object, you must skip the parent object attribute in the mapping table, and then map its individual nested primitive keys.
• Manifest Files: You can use manifest files to automate your JSON key mappings. The manifest file will reference the flattened JSON paths (for example.,"column": "user\_profile.id")

Map Limitations

Level-2 Keys within an Array of Objects: You cannot directly map individual Level 2 keys that are nested within an array of objects. You can map the entire array of objects to an “Array of Objects” attribute, which imports its full nested structure.

Behavior during the data ingestion

• Null or Blank Strings: If a mapped key’s value in your JSON file is null or an empty string, MoEngage will skip it and not update the corresponding data.
• Unmapped/Skipped Keys: Any additional keys in your JSON file that are neither explicitly mapped nor marked as skipped will be ignored and will not be ingested into MoEngage during the import process.

For each type of import, MoEngage requires you to have a column that can be mapped to a mandatory attribute:

1. Registered users import: Your files should have a column that can be mapped to the user ID (user identifier to identify User Profile in MoEngage).
2. Anonymous users import: Your files should have a column (email, mobile, etc) that can be mapped to an anonymous ID. An anonymous ID such as email, mobile, etc can help you identify users who have used your app/website but have not yet signed up.
3. Event Import: Your files must have separate columns for event time and user ID.

MoEngage does not support the | (pipe) character in non-array type columns. Ensure your String, Numeric, and Boolean columns do not contain this character.

Manifest Files

A manifest file contains mappings between the source column and a MoEngage Attribute. It also contains the data type of the column. This file must be in JSON format. An example of a manifest file is shown below. Using the MoEngage Dashboard, you can map the columns in your CSV or JSON files to attributes inside MoEngage. Or optionally, you can also choose to upload a manifest file.

CSV Files

{
"mapping": [
{
 "column": "ID",
 "moe_attr": "uid",
 "type": "string",
 "is_skipped": false
},
{
 "column": "First Name",
 "moe_attr": "u_fn",
 "type": "string",
 "is_skipped": false
},
{
 "column": "First Seen",
 "moe_attr": "cr_t",
 "type": "datetime",
 "datetime_format": "YYYY-MM-DD hh:mm:ss",
 "is_skipped": false
},
{
 "column": "LTV",
 "moe_attr": "t_rev",
 "type": "double",
 "is_skipped": false
},
{
 "column": "Install Status",
 "moe_attr": "installed",
 "type": "bool",
 "is_skipped": false
}
]
}

JSON Files

{
"mapping": [
   {
   "column": "customer_id",
   "moe_attr": "uid",
   "type": "string",
   "is_skipped": false
 },
{
   "column": "user.first_name",
   "moe_attr": "u_fn",
   "type": "string",
   "is_skipped": false
 },
{
   "column": "user.first_seen",
   "moe_attr": "cr_t",
   "type": "datetime",
   "datetime_format": "YYYY-MM-DD hh:mm:ss",
   "is_skipped": false
},
{
   "column": "attribution.lifetime_value",
   "moe_attr": "t_rev",
   "type": "double",
   "is_skipped": false
 },
{
   "column": "attribution.device_installed",
   "moe_attr": "installed",
   "type": "bool",
   "is_skipped": false
 }
]
}

For each column, you must mention the following fields:

1. “column”: (required) Represents the column name from the source file. When mapping Level 2 columns of a JSON file, you can use dot-notations: key1.key2.
2. “moe_attr”: (required) The MoEngage Attribute (attribute name) to which you want to map the source file column. You can view the number of Events, User Attributes, Volume of User and Events on the Data Management page. Ensure each column is mapped to a unique “moe_attr”.
3. “type”: (optional) This field represents the data type of the column. Refer to the table below for the acceptable “type” formats for attributes. Refer to the Data Management page for more information.
4. “datetime_format”: (optional) This field indicates the date and time format. It is mandatory for DateTime fields only. Refer to the table below for the complete list of supported date-time formats.
5. “is_skipped”: (optional) This field represents whether MoEngage should import the column or not. It is a boolean field, and any attribute marked as true will be skipped during import.

The above sample manifest file consists of 5 standard attributes. You can refer to the list below for more such standard attributes, or refer to your Data Management dashboard to get the exhaustive standard attribute list.

Some other standard user attributes in MoEngage for reference:

Key	Attribute Name on Dashboard	Datatype	Description
name	Name	String	Full name of the user.
first_name	First Name	String	First name of the user.
last_name	Last Name	String	Last name of the user.
email	Email (Standard)	String	The Email Address of the user. For example, [e-mail]
age	Age	Numeric	Age of the user
gender	Gender	String	Gender of the user
mobile	Mobile Number (Standard)	String	Mobile Number of the user. For example, 918888444411
moe_geo_location	Location	Array of [lat,lng] in double in the format {"lat": 12.11, "lon": 123.122}	A sample value would be the location of the user. For example: {"lat": 12.11, "lon": 123.122}
source	Publisher Name	String	This is the Publisher Name of Install. For example, Google Ads
revenue	LTV	Numeric	Life Time Value of the user.
moe_unsubscribe	Unsubscribe	Boolean	Email Unsubscribe Attribute. Emails are not sent to the user when the set value is true.
moe_hard_bounce	Hard Bounce	Boolean	Email Hard Bounce Attribute. The emails are not sent to the user when the set value is true.
moe_spam	Spam	Boolean	Email Spam Attribute. The emails are not sent to the user when the set value is true.

Supported User Attribute Types

Type	Description	Value in Manifest
String	Any string value. Examples of acceptable values: ABC, ABC XYZ, ABC123, etc.	"type": "string"
Double	Any decimal value. Examples of acceptable values: 3.14159, 241.23, -123.1	"type": "double"
Boolean	Examples of acceptable values: true, false	"type": "bool"
Date Time	Any date time value. The supported formats are covered in the next table. Example of acceptable values: 2019/02/22 17:54:14.933	"type": "datetime"

MoEngage does not support the | (pipe) character in non-array type columns. Ensure your String, Numeric, and Boolean columns do not contain this character.

Supported Datetime Formats

Datetime Format	Examples
“datetime_format”: "YYYY-MM-DD"	- 2022-01-22
“datetime_format”: "YYYY/MM/DD"	- 2022/01/22
“datetime_format”: "DD/MM/YYYY"	- 22/01/2022
“datetime_format”: "DD-MM-YYYY"	- 22-01-2022
“datetime_format”: "DD-MM-YYYY hh:mm:ss"	- 31-12-2022 12:10:33
“datetime_format”: "DD/MM/YYYY hh:mm:ss"	- 31/12/2022 12:10:33
“datetime_format”: "YYYY-MM-DD hh:mm:ss"	- 2019-02-22 17:54:14
“datetime_format”: "YYYY/MM/DD hh:mm:ss"	- 2019/02/22 17:54:14
“datetime_format”: "DD-MM-YYYYThh:mm:ss.s"	- 31-12-2022T12:10:33.882
“datetime_format”: "DD/MM/YYYYThh:mm:ss.s"	- 31/12/2022T12:10:33.882
“datetime_format”: "DD-MM-YYYYThh:mm:ssTZD"	- 31-12-2022T12:10:33Z - 31-12-2022T12:10:33+08:00 - 31-12-2022T12:10:33-08:00
“datetime_format”: "DD/MM/YYYYThh:mm:ssTZD"	- 31/12/2022T12:10:33Z - 31/12/2022T12:10:33+08:00 - 31/12/2022T12:10:33-08:00
“datetime_format”: "YYYY-MM-DD hh:mm:ss.s"	- 2019-02-22 17:54:14.933
“datetime_format”: "YYYY/MM/DD hh:mm:ss.s"	- 2019/02/22 17:54:14.933
“datetime_format”: "YYYY-MM-DDThh:mm:ssTZD"	- 2019-11-14T00:01:02Z - 2019-11-14T00:01:02+08:00 - 2019-11-14T00:01:02Z-08:00
“datetime_format”: "YYYY/MM/DDThh:mm:ssTZD"	2019/11/14T00:01:02Z 2019/11/14T00:01:02+08:00 2019/11/14T00:01:02-08:00
“datetime_format”: "YYYY-MM-DDThh:mm:ss.sTZD"	2019-02-22T17:54:14.957Z 2019-02-22T17:54:14.957299-08:00 2019-02-22T17:54:14.957299+08:00
“datetime_format”: "YYYY/MM/DDThh:mm:ss.sTZD"	2019/02/22T17:54:14.957Z 2019/02/22T17:54:14.957299-08:00 2019/02/22T17:54:14.957299+08:00
“datetime_format”: "YYYY-MM-DD hh:mm:ss.sTZD" (only for Snowflake Imports)	2019-01-31 17:54:14.957299-08:00 2019-01-31 17:54:14.957299+08:00

After you are ready with the files, you mustplace them inside a folder.

Folder Structure

Place all files into a folder. You can configure the folder path while setting up the import.
Note:MoEngage do not look for files in the root folder or sub-folders.

Setting Up Imports

You can set up the S3/SFTP Imports in the following navigation on the MoEngage Dashboard: Segment -> S3/SFTP Imports. You can also find user-specific imports in the following navigation on the Dashboard: Segment -> Import users.

Amazon S3

In order to properly set up S3 Imports, you must give MoEngage relevant permissions and ensure all your files are properly accessible to us. Learn how to set up S3 Imports.

SFTP Servers

To learn about the types of authentication methods MoEngage support and the steps to configure, see how to set up SFTP Imports.

Imports Dashboard

The S3/SFTP Imports Dashboard contains all the information you must keep track of your imports. You can view the most important information at a glance:

Name	Description
Name	The name of the import you gave while setting it up.
Type	The source of the imports, MoEngage support S3, SFTP, CSV, Snowflake The type of imports, One-Time or Periodic, is also mentioned in the brackets.
Custom Segment	If the user import happens in a custom segment, the name of the segment is displayed in this column.
Created at	The date and time of the import when it was created.
Last Sync Status	The last status of the import. One-Time Imports: The current status of the import. Scheduled: The import is scheduled to run in the future. Processing: The import is currently ongoing. Successful: The import was successful. Partial Success: At least one file was imported with Partial Success. Failed: The import has failed. Recurring Imports: The current status of the latest scheduled import. Scheduled: The import is scheduled to run in the future. The scheduled time is given below. Processing: The import is currently ongoing. An estimated time of completion is also mentioned. Successful: The import was successful. The completion time is also provided. Partial Success: When at least one file was imported with Partial Success. Failed: The import has failed. The failure time is given for reference.
Files Processed	The number of files processed to date.
Rows Processed	The total number of rows processed to date.
Actions	You can choose to View Details, Edit, Duplicate, or Delete an import.

You can also view the details of each import by clicking on the three-dot actions menu against it and selecting View Details.

Name	Description
Scheduled At	The name of the import you gave while setting it up.
File Name	The name of the file imported.
Status	The status of the import. Successful Failed - Hover to see the failure reason. Partial Success - At least 1 row failed to import in the file.
Rows in file	The number of data rows present in the file.
Events Added	The number of events that were imported from the file.
Events Failed	The number of events that failed to import.
Users Updated	The number of records identified in your import for existing users that were submitted for potential update in MoEngage.
Users Created	The number of records identified in your import for new users that were submitted for potential creation in MoEngage.
Users Failed	The number of user records from the import that could not be initially processed due to errors (e.g., invalid format).
Aux Data Added	The number of auxiliary data rows added during the import.
Aux Data failed	The number of auxiliary data rows failed during the import.
Rows Skipped	The number of rows that were skipped during the import.
Actions	You can choose “Export File” to download a copy of the file that was imported. You can export files for up to 60 days.

To verify the actual number of creates and updates to your users in MoEngage, you must ensure about the segmentation queries.

Triggering Imports

You can trigger imports using one of the following options:

1. Manually : Periodic imports can be triggered manually from the imports page by clicking Actions Note: Triggering an import within five minutes of the scheduled time may lead to an error.
2. API : Imports can be triggered using the File Imports Trigger API. For more information, refer to File Imports Trigger API.

File Limits

The maximum file size that MoEngage supports is 200MB per file. In case of a bigger file, the file must be split into 200MB files each.

1. Rows: MoEngage allows a maximum of 1M rows in a single CSV/JSON file.
2. Columns/Attributes: MoEngage allows 100 columns (attributes) for users and 100 columns (attributes) per event.
3. File size: The preferred file size should be less than 200MB (per CSV/JSON) for S3/SFTP and less than 150MB (per CSV) for Manual Uploads.
4. File name prefix: The maximum number of characters in the file name prefix should be 100. These limits apply to S3 and SFTP Imports.
5. Number of files: In each import run, MoEngage can import a maximum of 250 files.
6. Import limits: Refer to the table below.

Import Type	Users	Events
Default Hourly	600K/hr	5M/hr
Default Daily	Total 14M/day	Total 40M/day

• The values mentioned above are the default rate limits. If your requirement exceeds these rate limits, you can get in touch with MoEngage to increase them.
• The default hourly rate indicates the total data that can be ingested in an hour and does not indicate the total data limit for the day. The Default Daily limit is the total cap on the data that can be ingested in a day, as indicated in the table above.
• For one-time historical imports, get in touch with the support team to guide you.
• When the rate limits for the imports are breached, you receive a Rate limit breached alert in the Segment-> Imports section on the MoEngage Dashboard.

Best Practices

To ensure your data imports run smoothly and efficiently, consider the following recommendations:

• Optimize File Size for Speed: For faster processing, it’s recommended to keep individual import files under 1 million rows.
• Split Large Datasets: If you have a large amount of data to import, splitting it into multiple smaller files can generally result in faster overall processing than uploading a single, huge file.

Frequently Asked Questions

My imports have failed. How do I check what went wrong?

Click the ellipsis on the right and click View to look up the Import details. Hover over the Failed Status to learn the reason.

What happens when there is an error in fetching the files from the S3/SFTP folder?

By default, imports are not retried when there is a failure. You can, however, configure to receive an email alert upon failure.

What if a scheduled import adds the data into a recently archived segment?

In such cases, the new data will still be added to the archived segment. You can unarchive the segment as required.

How does MoEngage pick up new files in a periodic run?

In a scheduled run for a given workspace, MoEngage picks up all the new files identified in the S3/SFTP folder based on file name prefix and processes them. Data import in a given MoEngage workspace uniquely identifies these files with the combination of Folder location and File name.

Will a periodic Data import pick up an earlier processed file?

No. A file once processed will not be processed again if the file name remains the same and in the same folder location. However if the same file is also placed in a different folder location, a new data import setup in MoEngage with the new folder location configured can pick up the file.

Can a CSV file contain both User and Event data?

No. Data imports in your MoEngage workspace dashboard have to be setup separately for User data and Event data, hence the files should either contain User attribute data or Event data.

Can an import be stopped while it's running?

Once an import process starts, it can’t be stopped midway. This is because the data goes through several steps, and interrupting it could lead to incomplete or inconsistent results. It’s best to let the current import finish.

How can I stop future runs of a scheduled import?

Yes, you can stop future scheduled imports from running automatically. To do this, find the import schedule and select the “Archive” option from the Actions menu on the Data Imports dashboard. This will prevent it from running on its next scheduled time.

What should I do if an import seems stuck or is taking a long time?

If an import appears to be stuck or is taking longer than usual, it’s best to wait. The system has checks in place to handle these situations automatically and retry if necessary. Manually starting the same import again while it’s still processing can cause conflicts and may prevent the original import from completing successfully.

What is NDJSON, and why is it required?

NDJSON (Newline-Delimited JSON) means each line in your .json file is a complete JSON object, separated by a new line. This format is required to efficiently process large JSON datasets line by line, allowing for scalable and robust imports.

Are compressed files (e.g.,.gz,.zip) supported?

No, MoEngage don’t support importing compressed files.

If I change the file type of an existing import, what happens to my mappings?

If you change the file type, the MoEngage removes the existing preview and re-fetches the list of available files based on the new file type. You must select File to Preview again and configure your mappings based on the new file’s structure.