Skip to main content

Core Concepts

Our embeddable Importer SDK provides the building blocks for scaling the import of diverse and messy data.

Often, data from your customers, suppliers, or partners is not formatted or structured as required by your system. Handling this mismatch manually can significantly drain time and resources during data onboarding.

With our AI-powered column and option mapping, along with fully customizable transformation functions, even your non-technical users can effortlessly import clean data.

Our Importer SDK is fully customizable to meet your specific requirements. Depending on your use case, the import workflow can include more or fewer steps. Typically, our importer guides users through three main steps: uploading files, matching columns, and reviewing the transformed and cleaned data.

  1. Upload your file(s): Users can upload one or multiple files by either dragging them into the drop zone or pressing the "Select file" button. We currently support XLS(X), CSV, TSV, XML, and JSON files. The UI is fully customizable (white-label) to match your corporate identity and can be implemented with only a few lines of code.
  2. Match columns: Our Ingestro AI restructures the data and matches the imported columns to the columns in your defined output schema. It learns from initial imports, so if a similar file structure is uploaded in the future, this step can be skipped. During this process, users can confirm recommended matches and create new mappings if necessary.
  3. Review entries: In this step, users review the restructured data, resolve potential errors either manually or by utilizing our Ingestro AI for data cleaning, and submit their data. You can flexibly define what should happen with the submitted and cleaned data, whether it should be downloaded directly as an Excel file or sent to your preferred target destination via an HTTP call.

Ingestro’s power lies in its intuitive UI, AI-powered mappings, and flexible data validation and cleaning functions. After the upload, Ingestro reorganizes the data. Once the mappings are complete, the cleaning functions you configured take effect, validating and transforming the imported data into the desired format. With Ingestro, you can release an advanced importer to your users in days, not months.

Core Concepts​

Before diving into the details of how to set up your own Ingestro importer, let’s review the most important concepts and elements of Ingestro.

Target Data Model (TDM)​

The most essential element of our importer is the target data model (TDM), which reflects the data structure required by your system. The TDM includes column labels, validation rules such as required, uniqueness, regex, and cross-column validations, and column types, our interpretation and extension of data types, such as string, int, float, country code, currency, email address, and many others.

You can create your preferred TDM in our User Platform using one of the following four methods:

  1. Generate from an existing file
  2. Use the no-code generator
  3. Feed prompts to our Ingestro AI
  4. Write the TDM code manually

Please note, the TDM can be generated dynamically within your application in real time, allowing you to present a customized and personalized schema to each user. It can be generated before initializing the importer and then inserted during runtime.

Cleaning Functions​

Our cleaning functions are callbacks that are triggered after specific events. These functions allow you to transform imported data into the desired format and provide feedback to the user. Typical use cases include correcting date formats, enriching entries, comparing imported data against your database or third-party services, validating cross-column dependencies, and flagging duplicates with customizable error messages. Cleaning functions provide full flexibility for adding your own cleaning logic in JavaScript/TypeScript.

Step Handler Functions​

Our step handler functions are powerful callback functions that execute after each step of the import workflow, providing complete control over data validation, transformation, and user experience. These functions allow you to intercept the import process at four key moments: after file upload (uploadStep), after header selection (headerStep), after column mapping (mappingStep), and before final submission (reviewStep). Step handler functions enable sophisticated import customization including:

  • Data validation and transformation: Access raw files, parsed data, and mapped results to validate against external systems or apply custom business logic
  • Schema modification: Dynamically add or remove columns from your target data model during the import process
  • User flow control: Show custom alert or block modals to guide users or prevent progression based on data conditions

These functions receive detailed context about the import state. This includes file metadata, column and option mappings, customization logs, and the complete dataset. With this information, you can implement complex validation workflows, apply advanced data cleaning, and deliver personalized user experiences throughout the import journey.

Dynamic Import​

The Dynamic Import feature allows you to trigger an import based on various events, such as user interactions or receiving data from an external source. Additionally, you can initiate the import process at different stages, such as starting directly at the mapping or review steps, with your preferred data. This feature offers the flexibility to customize the import workflow to meet your specific requirements.

Supported File Types & Advanced Parsing​

The Input Types setting defines which file formats your importer supports. In addition to standard formats like csv, tsv, xls, and xlsx, it also unlocks advanced parsing for json, xml, and pdf.

  • XML: Supports complex XML structures by breaking them into multiple sheets—similar to Excel workbooks—based on individual datasets within the file.
  • JSON: Handles both flat and nested JSON files (up to four levels deep). Nested data is transformed into a 2D table by converting:
    • Arrays → keys joined with underscores _
    • Objects → keys joined with dots .

This makes it easy to visualize and process structured data directly in the importer interface.

Smart Table​

This feature adds Excel-like functionality to the Ingestro importer in the review step, making it easier and quicker for users to resolve potential errors during the import process. For example, users can sort, filter, freeze, and hide columns, as well as find and replace values.

Knowledge Base​

Ingestro’s knowledge base includes 70+ sandboxes, providing real-life examples that you can incorporate into your Ingestro importer. To access them, log into the Ingestro User Platform, click on the SDK section in the sidebar, and select the "Knowledge Base" tab.

info

Try a compact version of the importer in our User Platform. No code or setup is required. Choose from our example target data models or create your own with the target data model generator, customize the importer, and start importing cleaned data instantly.