Skip to main content

​
Overview

Data Ingestion is where data from various sources is imported into the platform for processing, analysis, and storage. This process is managed through two key concepts: Sources and Extracts. A Source holds the connection to an external system — credentials, host, project. You create it once and reuse it. An Extract selects what to pull from that source — a table, a collection, a file pattern — and how to pull it. One source can back many extracts. That split is also how configuration is split: Each source page documents both. Parameter names match the API and CLI payloads.

​
Getting Started

To configure a new source:
  1. Select your source type from the available connectors
  2. Provide the required connection details
  3. Start creating extracts from your source
For detailed configuration steps, visit the specific documentation page for your chosen source type.

​
Shared Extract Parameters

These parameters apply to every extract regardless of source type. Source-specific parameters are documented on each connector’s page.
​
'overwrite' | 'append'
default:"overwrite"
Extract Mode — overwrite replaces the dataset on every run (full table). append reads only new rows since the last run and appends them, using replication_key to track position.
​
string
Replication Key — the column Datazone compares against the last extracted value to find new rows. Required when mode is append. Use a monotonically increasing column such as an auto-increment id or an updated_at timestamp.
​
FieldMapping[]
Schema Mapping — per-field selection, renaming, and type casting applied before the data is written. Each entry accepts source_field, source_type, target_field (rename, null keeps the original), target_type (cast, null keeps the inferred type), and include (set false to drop the field).
​
object
Custom Reader Config — an escape hatch. Keys in this object are merged over the resolved reader configuration for every execution, overriding anything computed from the source and extract. Use it only for parameters not exposed elsewhere.
The following go in the extract’s source_parameters and are supported by all connectors that use the reader framework:
​
integer
Rows read and written per batch. Higher values reduce round trips but raise memory use. The default varies per connector — see each source page.
​
string
default:"*"
Columns to read, as a comma-separated list (for example id, name, created_at). Defaults to every column.
​
string
A row filter applied at the source, written without the WHERE keyword — for example status = 'active' AND region = 'EU'. Pushing filters down here is far cheaper than filtering later in a pipeline.
reset_state is set per execution, not on the extract. It re-reads all data while still capturing the latest replication state, and is only valid for an append extract that already has state.

​
Connectors

MySQL

Direct connection to MySQL and MySQL-compatible databases

PostgreSQL

Direct connection to PostgreSQL databases

Microsoft SQL Server

Direct connection to SQL Server instances

Oracle

Direct connection to Oracle Database

MongoDB

Document collections from MongoDB

Google BigQuery

Tables and views from BigQuery datasets

AWS S3

Delimited files from S3 buckets

Azure Blob Storage

Files from Azure Blob containers

SAP HANA

Direct SQL access to SAP HANA

SAP ERP

SAP ERP tables via the CloudFeed connector

SAP S/4HANA

SAP S/4HANA tables and CDS views via CloudFeed

SAP BW

SAP BW InfoProviders via CloudFeed

HubSpot

CRM objects from HubSpot