Configuration: Building Your Pipeline

Owned by EazyDI Main Account
Last updated: Oct 18, 2024 by Neha Awalwar
4 min read

Empower your data journey with EazyDI's intuitive pipeline creation. Choose connectors, map fields, schedule seamlessly. Elevate your data integration experience—simple, powerful, and effective

1. Enter Mapping Integration Name

Start Building Your Pipeline: Begin by providing a distinctive name for identification. Click the pencil icon, and an input box will appear for entering the integration pipeline name.

2. Configure Source Connection

Step 1: Select Source Connector

Browse the list and choose a source connector that aligns with your business requirements.

Step 2: Connect to Source

  • If a connection exists, click "Connect" to access the Source Configuration page.

  • If no connection, create one:

    • Redirected to the Source Configuration page. A pop-up screen with connection attributed is displayed

    • Enter connection attributes.

    • Test and validate the connection.

    • Create the connection.

Step 3: Choose Connection Name

Select an existing connection name or create a new one on-the-fly by clicking "Add Connection."

Step 4: Define Connection Objects

  • Depending on the connector type, specify connection objects:

    • Database (SQL): Tables

    • NoSQL Database: Documents

    • Application Services: Modules (e.g., accounts, users, campaigns)

    • Excel: Workbook and sheet names

    • File system: File names

    • Social Media: Ad metrics (campaigns, Ads, Adsets)

Step 5: Configure Connection Parameters (Custom Connectors)

Custom application connection might have some additional attributes . Specify connection parameters as required.

Step 6: Proceed

  • Click "Next" button to advance to the next stage of pipeline configuration.

  • The user will be directed to the Target Connection page

3. Configure Target Connection

Step 1: Select Target Connector

Browse the list and choose a target connector that aligns with your business requirements.

Step 2: Connect to Target

  • If a connection exists, click "Connect" to access the Target Configuration page.

  • If no connection, create one:

    • Redirected to the Target Configuration page. A pop-up screen with connection attributed is displayed

    • Enter connection attributes.

    • Test and validate the connection.

    • Create the connection.

Step 3: Choose Connection Name

Select an existing connection name or create a new one on-the-fly by clicking "Add Connection."

Step 4: Define Connection Objects

  • Depending on the connector type, specify connection objects:

    • Database (SQL): Tables

    • NoSQL Database: Documents

    • Application Services: Modules (e.g., accounts, users, campaigns)

    • Excel: Workbook and sheet names

    • File system: File names

Step 5: Configure Connection Parameters (Custom Connectors)

Custom application connection might have some additional attributes . Specify connection parameters as required.

Step 6: Proceed

  • Click "Next" to advance to the next stage of pipeline configuration.

  • The  user will be directed to the Field Mapping Connection page

4. Mapping Source Attributes to Target Attributes

This screen provides the flexibility to map unstructured source attributes to structured target attributes. The attributes are referred to as fields here in this application 

Upon selecting the Connection objects, the application fetches the metadata (attributes) of the connector object, presenting it on the field mapping page.

On the left side of the screen, a list of source metadata fields is displayed, while the right side of the table showcases the target metadata fields. This interface allows you to efficiently map raw source data to structured target data.

The features available on this screen are 

  • Auto Match: This is the default view as well – it acts as a name pair match (source fields will be mapped if it matches the same target field name)

  • Drag and Drop: You can drag a source field to the target field that you want to map to

  • Clear Mapping: It clears the existing mapping (i.e.: removes everything from the Mapped Field/Expression area

  • Search box: Using this you can find the field name

  • Delete a mapping: You can remove a mapping by clicking on the trash icon

  • FX: On click of a pop-up screen will open. Here you can add simple operations on your target data like :

    • Removing white spaces using Trim() , LTRIM(), RTRIM()

    • combine data of 2 source  field  to one target  field using Concat(),  OR 

    • Set the desired date-format using To_Date()

    • Convert the other data types using TO_Float(), To_BigInt(), To_Char, To_Decimal(), To_Integer()

5. Configure Pipeline Scheduling

Access Scheduling Configuration: Navigate to the designated screen to configure scheduling for your pipeline based on business requirements

Features:

  • Scheduling Options: The screen offers users the flexibility to set up scheduling in two ways:

    • Multiple Runs: Choose this option if you want your pipeline to run at specified intervals.

    • Manual Run: Opt for this option if you prefer a one-time manual execution of the pipeline.

  • Radio Buttons: Utilize the provided radio buttons to easily toggle between the two scheduling options. Select the appropriate button based on your scheduling preference.

Details for Setting up Manual Run or Multiple Runs for Pipeline:

i) Manually run pipeline without Schedule

  • If the user wants to just save the pipeline and manually trigger the pipeline, you can choose this option. 

  • Click and select the radio button, Without Schedule.

  •  It will  minimize the scheduling configuration section of the screen.

  • You can click on Next button to browse to Summary screen

ii) Run pipeline to run on given schedule 

The user can choose this option to schedule the pipeline to run multiple time at a given schedule as per business requirements 

You will have to first enter the following details :

  • Schedule Name: Provide the schedule name 

  • Description: Short description for the schedule for future reference

You can choose from the following Scheduling Options 

  • Start time:   Select the start time

  • Time Zone:  Select the Timezone from the available list

  • Repeats:  This dropdown box allows user to select the frequency at which you want the pipeline to run:

    • Hourly

    • Daily

    • Monthly

    • Every N hours

    • Every N minutes

    • DoesNotRepeat

After selecting the scheduling options, you can click on Next button to browse to Summary screen

5. Summary and Save Mapping pipeline

This screen provides an overview of the pipeline created or edited by the user, showcasing key details. 

  • Summary: The page displays a concise overview of connection details and field mapping settings configured for pipeline construction.

  • Back Button:  Clicking this button redirects the user to the scheduling configuration page.

  • Save Button: Once the user has verified all pipeline details, clicking the save button will store the information and present a success message pop-up.

Upon clicking the "Done" button within the success message box, the application will navigate to the Manage Mapping page.

 

Note : - To explore more details on the read and save strategies for your data, kindly refer to the below link.

Read/Write Strategies for Pipelines in EazyDI