Skip to main content

​
Overview

Salesforce stores CRM and custom business data as sObjects such as Accounts, Opportunities, or custom objects ending in __c. Datazone connects to an org with a connected app (or external client app) through the OAuth 2.0 client credentials flow, and extracts one sObject per extract using SOQL.

​
Prerequisites

In Salesforce Setup:
  1. Create a connected app or external client app.
  2. Enable the client credentials flow for the app and set a Run As user.
  3. Copy the consumer key and consumer secret.
  4. Note your org’s My Domain URL (Setup → My Domain). A sandbox has its own, for example https://acme--uat.sandbox.my.salesforce.com.
The Run As user’s profile and permission sets decide which objects and fields Datazone can read. Give that user read access to each object you plan to extract.

​
Connection Parameters

Set on the source when you create it. Names match the API and CLI payload.
​
string
required
Name: a unique identifier for your Salesforce source.
​
string
required
My Domain URL: the org’s My Domain URL, for example https://acme.my.salesforce.com. Use the sandbox’s own My Domain URL for sandbox orgs.
​
string
required
Consumer Key: the consumer key of the connected app or external client app.
​
string
required
Consumer Secret: the consumer secret of the app. Stored encrypted.
Datazone signs in when an extract runs and signs in again if the session expires during a long extraction. The flow issues no refresh token, so no token needs to be stored.

​
Extract Parameters

Set per extract, in source_parameters. See Shared Extract Parameters for mode, replication_key, schema_mapping, and reset_state.
​
string
required
Object — the API name of the sObject to extract, for example Account, Opportunity, or Invoice__c. Use the source explorer to browse queryable objects.
​
string
Filter — an optional SOQL condition, written without the WHERE keyword. For example Type = 'Customer' AND CreatedDate = LAST_N_DAYS:365.In append mode the filter is combined with the incremental condition using AND.
Datazone builds the field list from the object’s describe metadata. If you set a schema_mapping selection, only those fields are queried.

​
Output Shape

Each row is one sObject record. Column names match the Salesforce field API names (for example Id, Name, SystemModstamp). Fields of type address, location, and base64 are not extracted. Address and location values are available through their component fields, such as BillingStreet and BillingCity.

​
Incremental Extraction

Use mode: append to extract only new and changed records. Set replication_key to a field that Salesforce updates on every create and edit. SystemModstamp is the usual choice. Any field of the object can be used, but it should be a datetime that changes on every write. On the first run, Datazone reads every record that matches the filter. On later runs, it reads records where the replication key is greater than the last value it saw.
An edited record is appended again, so the dataset can hold more than one row per Id. Keep the latest row per Id (highest replication key value) in a downstream pipeline transform.In append mode, rows already extracted stay in the dataset after the record is deleted in Salesforce. Use overwrite if the dataset must mirror deletions.
To re-read everything, run the extract with reset_state set for that execution.

​
Limitations

  • One sObject per extract
  • Deleted records are not extracted
  • Compound fields (address, location) and base64 fields are skipped
  • Fields of type anyType, such as OldValue and NewValue on history objects, are stored as JSON strings
  • Rate limits (HTTP 429) and transient server errors are retried automatically with backoff, which can extend the runtime of large extracts
  • Each query page holds up to 2,000 records

​
Next Steps

After configuring your Salesforce source:
  1. Browse the objects with the source explorer and create an extract for each one
  2. Choose overwrite for full reloads, or append with a replication_key for incremental loads
  3. Configure scheduling for recurring extracts
  4. If you use append, keep the latest row per Id in a pipeline transform