Data Processing
This page lists every feature of the importer that sends data to our server side. Features that aren't listed here run entirely in the user's browser.
Always sent
| Request | When | What is sent | What we store |
| License check | When the importer loads | The session ID, the SDK version, the framework and its version, and the domain | The SDK version, the framework and its version, the domain, and whether a dev or live license key is used |
| Import log | When the user completes the import | The session ID, the importer ID, the SDK version, the framework and its version, the domain, the target data model, the column names of the uploaded file and the column each one was mapped to, the settings, the number of rows, columns and errors, the time spent in each step, and the size and type of each uploaded file | The session ID, the importer ID, the SDK version, the framework and its version, the domain, the target data model, the column names of the uploaded file and the column each one was mapped to, the settings, the number of rows, columns and errors, the time spent in each step, the size and type of each uploaded file, and whether a dev or live license key is used |
Features
| Feature | What is sent | Processed by AI | What we store |
Column mapping (columnMappingConfiguration). On by default. Set processingMode to "browser" to run it in the browser instead. | The session ID, the column names of the uploaded file, and the label and alternative matches of each column in the target data model | Yes, with the "smart" layer (on by default) | Nothing during column mapping. However, when the user completes the import, the column names of the uploaded file and the column each one was mapped to are saved with the import log. |
Option mapping (optionMappingConfiguration). Off by default. Set processingMode to "node" to run it on our server side. | The session ID, the column names and every unique value of every uploaded column, and the options of the matching dropdown column in the target data model | Yes, with the "smart" layer (on by default) | The column names and every unique value of every uploaded column, which are deleted when the user clicks "Back" on the "Match Columns" step, closes the importer or completes the import. They can remain if the user closes the browser tab, or clicks "Back" while persistReviewSession is on. When the user completes the import, and the "historic" layer is used (on by default), each value and the option it was mapped to are saved, including options that the user added. |
Saved mappings (skipConfiguration.mapping) | The column names of the uploaded file and the column each one was mapped to, and for dropdown columns, each value and the option it was mapped to | No | The column names of the uploaded file and the column each one was mapped to, and for dropdown columns, each value and the option it was mapped to |
Join and append suggestions (sheetMappingConfiguration) | The session ID and the column names of both sheets. With "headers_with_samples", also the first 25 rows of each sheet. | Yes | Nothing |
Pre-Processing prompts (preProcessing) | The session ID, the prompt, and the column names of the selected sheets and 3 rows in total | Yes | The session ID, the prompt, which of the resulting steps the user applied or rejected, and whether a dev or live license key is used |
Pre-Processing suggestions (preProcessingSuggestions) | The session ID, the column names of the selected sheets and 3 rows in total, and the target data model | Yes | The session ID, each suggestion and whether the user added, dismissed or ignored it, and whether a dev or live license key is used |
PDF parsing (pdfConfiguration). On by default if your plan includes PDF upload and baseUrl isn't set. Set pdfConfiguration.mode to "browser" to parse PDF files in the user's browser instead. | The PDF file | No | The PDF file |
| PDF extraction instructions | The session ID, the instruction, the user's previous instructions for the same PDF, the selected tables and text, and the current result | Yes | The session ID, the instruction, the number of tables it produced, the error if it failed, and whether a dev or live license key is used |
Cleaning Assistant (cleaningAssistant) | The session ID, the SDK version, up to 1,500 rows with errors and their error messages, 5 rows without errors, and the target data model | Yes | The session ID, whether a dev or live license key is used, and for each cell it suggests a fix for: the column, the original value, the error message, the suggested value, whether the user applied, ignored, deleted or changed it, and the final value |
Contextual Engine (contextualEngine) | The session ID, the whole uploaded sheet and the target data model. The AI receives the first 50 rows. If that attempt fails, the retry sends it the whole sheet. | Yes | The session ID, the whole uploaded sheet, whether the result was accepted, rejected, failed, skipped or cancelled, the error message if it failed, and whether a dev or live license key is used |
AI Prompts (prompts) | The session ID, the prompt, the selected column and the target data model | Yes | The session ID, each prompt and the function it generated, the final function, whether the user applied it, and whether a dev or live license key is used |
| Export as Excel | Which export the user chose (all rows or error rows), the SDK version, the framework and its version, and the domain | No | Which export the user chose (all rows or error rows), the SDK version, the framework and its version, the domain, and whether a dev or live license key is used |
Step events (stepTracking) | The session ID, the importer's identifier and importer ID, the name of each step the user completes, the time, the SDK version, the framework and its version, and the domain | No | The session ID, the importer's identifier and importer ID, the name of each step the user completes, the time, the SDK version, the framework and its version, the domain, and whether a dev or live license key is used |
Storage and AI
Data stored in our database is kept indefinitely and can't be set to expire. The one exception is option mappings saved by the "historic" layer. Each organization can keep up to 400,000 of them, and once that limit is reached, the ones updated least recently are deleted first. Files in our file storage and entries in our AI tracing logs have no set retention period.
AI requests are recorded in our tracing logs. If your organization uses its own LLM, these requests still pass through our server side before they reach your LLM. Their content is left out of the tracing logs unless payload logging is turned on.
Self-hosted installations
When baseUrl is set, the requests listed on this page go to your own deployment instead of our server side. There are two exceptions. If pdfConfiguration.mode is set to "node", PDF files are still sent to our server side for parsing. If import logging is turned on, the import log and step events are also sent to our server side.