Skip to main content

CreatePipeline Embeddable Guide

The CreatePipeline embeddable allows you to set up Ingestro pipelines directly within your application, whether for internal teams to configure or to enable your customers to create pipelines within your product. Like all Ingestro pipeline components, the CreatePipeline embeddable is easy to integrate into your existing application.

This guide will help you implement the CreatePipeline component and configure it based on your specific requirements:

  1. Install the Ingestro package
  2. Get your Ingestro license key
  3. Get the access token
  4. Implement the embeddable component in your application
  5. Configure the component based on your specific use case

Let's get started​

Step 1: Install the Ingestro package​

Installing the Ingestro package is super easy.

Install with NPM:

React
Angular
Vue
VanillaJS
npm install @ingestro/pipelines-react

Install with Yarn:

React
Angular
Vue
VanillaJS
yarn add @ingestro/pipelines-react

Install via CDN:

VanillaJS
https://unpkg.com/@ingestro/pipelines-vanilla-js

Step 2: Get your Ingestro license key​

You can get your Ingestro license key from the Ingestro User Platform.

If you don’t have an account yet, you can sign up here.

info

Please note that pipelines in development mode only send the first 100 rows to your defined output destination. If you would like to increase your quota, please contact us at [email protected].

Step 3: Get the access token​

In order to authorise yourself or rather the component you have to first get the access token via our authentication API.

You can get access tokens for two different cases:

  1. You want your internal users to create pipelines
  2. You want your customers to create pipelines

You want your internal users to create pipelines

  1. To get the access token for an internal user, you need to first create an user in the Ingestro User Platform or use the user API
  2. Use the user’s email address and your license key to request the access token. Here is the cURL or JavaScript request you can use to get the access token:
cURL
Javascript
curl -X 'POST' 'https://api-gateway.ingestro.com/dp/api/v1/access/token' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "license_key": LICENSE_KEY,
  "auth": {
    "identifier": E-MAIL,
    "type": "USER"
  }
}' # Replace LICENSE_KEY with your license key & E-MAIL with a user's email

You want your customers to create pipelines

  1. To get the access token for your customer, you need to first create a sub-organization (sub-org) via the sub-organization API
  2. Use the sub-orgs’s identifier and your license key to retrieve the access token. Here is the cURL or JavaScript request you can use to retrieve the access token:
cURL
Javascript
curl -X 'POST' 'https://api-gateway.ingestro.com/dp/api/v1/access/token' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d {
  "license_key": LICENSE_KEY,
  "auth": {
    "identifier": SUB_ORG_IDENTIFIER,
    "type": "SUB_ORG"
  }
}  # Replace LICENSE_KEY with your license key & SUB_ORG_IDENTIFIER with a sub-org identifier

Step 4: Implement the embeddable component in your application​

With the package installed, you can add the CreatePipeline component to your application.

First, import the component in the file where you want to use the CreatePipeline component:

React
Angular
Vue
VanillaJS
import { CreatePipeline } from "@ingestro/pipelines-react";
import "@ingestro/pipelines-react/index.css";

Step 5: Configure the component based on your specific use case​

Add the code snippet below and insert the component on the page where you want it to appear:

React
Angular
Vue
VanillaJS
<CreatePipeline
  accessToken="ACCESS_TOKEN"
  templateId="6695093afg66536c28b97222"
  configuration={{
    developerMode: false,
    name: "My pipeline",
    tdmId: "6695092afg66536c28b97222",
    outputConnectorId: "6695092a8266536c28b97000",
    inputConnectorId: "66bb5afde31b39b57a580000",
    errorConfig: {
      error_threshold: 10,
    },
    scheduleConfig: {
      frequency: "DAILY",
      interval: 1,
      startsOn: "2024-09-21T06:57:06.672Z",
      endsOn: "2024-09-30T06:57:06.672Z",
    },
  }}
  settings={{
    i18nOverrides: {},
    language: "en",
    modal: false,
    allowTdmCreation: false,
    allowInputConnectorCreation: false,
    allowOutputConnectorCreation: false,
    runPipelineOnCreation: false,
  }}
  onPipelineCreate={({ data }) => {
    // runs after the user has confirmed the final step of the flow to create a pipeline
    // data: pipeline object after creation
  }}
  onClose={() => {
    // runs when the creation workflow is closed via the "Cancel" button or the "X" button
  }}
  onConnectorCreate={({ reload, connectorType }) => {
    // runs when the user clicks on "Create connector" when selecting an input or output connector
    // reload: on function call, refetch the connectors
    // connectorType: "input" or "output"
  }}
  onTdmCreate={({ reload }) => {
    // runs when the user clicks on "Create target data model" when selecting a target data model
    // reload: on function call, refetch the TDMs
  }}
  onExecutionView={({ data }) => {
    // runs when the user selects an execution from the list of triggered pipelines
    // data: the id of the selected execution
  }}
  onSuccessContinue={({ data }) => {
    // runs when the user clicks the "Continue" button on the success screen
    // data: object of the created pipeline
  }}
/>

accessToken

Add here the access token you’ve got in Step 3

templateId

Add here the ID of the template you want to use when a pipeline is created

configuration

The configuration determines if certain settings or components, such as connectors, target data model, schedule config etc., are already set for the pipeline, meaning that users going through the flow won’t have to set them themselves

developerMode

Set developer mode to true to test pipelines in your testing environment. Pipeline executions in developer mode are free of charge. Set it to false for production use. Please note that pipelines executed in developer mode will only output 100 rows

name

The name of the pipeline

tdmId

The ID of the target data model that should be used for the created pipeline. If this is set, the user won’t be able to select another target data model

inputConnectorId

The ID of the input connector that should be used for the created pipeline. If this is set, the user won’t be able to select another input connector

outputConnectorId

The ID of the output connector that should be used for the created pipeline. If this is set, the user won’t be able to select another output connector

errorConfig

Defines how the pipeline should handle errors that might occur during pipeline execution

errorThreshold

A float between 0 and 100, representing the allowed percentage of erroneous cells during a pipeline execution. For example, if it is set to 10, it means that pipeline executions with less than 10% erroneous cells will be considered successful and will not fail

scheduleConfig

Defines when the pipeline is executed for the first and last time, as well as the interval at which it is executed

frequency

Sets how often the pipeline is executed. It is intertwined with interval. For example, if frequency is set to HOURLY and interval is set to 2, the pipeline is executed every 2 hours:

  • HOURLY
  • DAILY
  • WEEKLY
  • MONTHLY

interval

Sets the interval based on the frequency at which the pipeline is executed. For example, if interval is set to 2 and frequency is set to HOURLY, the pipeline is executed every 2 hours. The next execution cannot be scheduled further into the future than 1 year from the set start date and time

startsOn

The date and time when the pipeline is first executed, provided as a timestamp in UTC (e.g. 2024-09-02T13:26:13.642Z). The date and time cannot be in the past

endsOn

The date and time when the pipeline is last executed, provided as a timestamp in UTC (e.g. 2024-09-02T13:26:13.642Z). This date and time cannot be earlier than the start date and time

settings

i18nOverrides

Allows you to override each text element in the interface

language

Defines the language of the embeddable (so far we only support English ("en"))

modal

Defines whether the component is shown inline (false) or within a modal view (true)

allowTdmCreation

Defines whether the "Create target data model" button is shown in the TDM selection dropdown

allowInputConnectorCreation

Defines whether the "Create connector" button is shown in the input connector selection dropdown

allowOutputConnectorCreation

Defines whether the "Create connector" button is shown in the output connector selection dropdown

runPipelineOnCreation

Defines whether the pipeline is executed after it was created.

onPipelineCreate

Runs after the user has confirmed the final step of the flow to create a pipeline

onClose

Runs when the user attempts to exit the "Create Pipeline" flow by clicking "Cancel" or closing the modal using the "X" button

onConnectorCreate

Runs when the user clicks on "Create connector" when selecting an input or output connector

onTdmCreate

Runs when the user clicks on "Create target data model" when selecting a target data model

onExecutionView

Runs when the user clicks on the "View" or "Fix" button of the execution that was created after the new pipeline was created. When defined, each execution element shows a "View" or "Fix" button that triggers this hook when clicked.

onSuccessContinue

Runs when the user clicks the "Continue" button on the success screen after successfully creating a pipeline.