Project Repository

Overview

A Datazone project is a Git repository containing your data pipelines, actions, apps, knowledge objects, flows, and endpoints. All project components are defined in a central config.yml file.

Project Structure

my-project

config.yml

pipelines

hello_world.py

data_processing.py

actions

apps

objects

flows

studio

utils.py

Each project must have a config.yml file
Each pipeline should have a unique alias and be defined in separate files
Aliases must be unique within pipelines and within studio_apps
Unknown keys are rejected on deploy rather than ignored, so a typo fails loudly
You can organize your code with utility files for shared logic

Configuration File

The config.yml file defines all project resources:

config.yml

project_name: my-pretty-project
project_id: 67280ba2f4a0960d02159675

pipelines:
  - alias: hello_world
    path: pipelines/hello_world.py
    compute: LARGE
    spark_config:
      deploy_mode: client
      executor_instances: 3
    python_dependencies:
      - name: pandas
        version: 1.3.3

actions:
  - path: actions/send_notification.py

apps:
  - path: apps/my_app.py

objects:
  - path: objects/order.yml

flows:
  - path: flows/daily_report.yml

studio_apps:
  - alias: orders
    name: Orders
    path: studio/orders

endpoints:
  - path: endpoints/api_config.yml

Configuration Reference

Project Fields

string

required

Name of your project. Used for display and identification.

project_name: data-processing-platform

string

required

Unique identifier for your project. Generated when you create a project.

project_id: 67280ba2f4a0960d02159675

array

List of data pipeline definitions. Each pipeline processes and transforms data.

pipelines:
  - alias: etl_pipeline
    path: pipelines/etl.py

array

List of serverless action functions. Actions can be triggered by endpoints or used by AI agents.

actions:
  - path: actions/send_email.py

Learn more in the Actions documentation.

array

List of intelligent AI applications.

apps:
  - path: apps/customer_support.py

array

List of Knowledge Object definitions — versioned business entities with a CRUD API.

objects:
  - path: objects/order.yml

Removing an entry deletes the object and schedules cleanup of its data. Learn more in the Knowledge Objects documentation.

array

List of declarative YAML orchestration flows.

flows:
  - path: flows/daily_report.yml

Learn more in the Flows documentation.

array

List of Studio Apps — React applications built and served by Datazone.

studio_apps:
  - alias: orders
    name: Orders
    path: studio/orders

Learn more in the Studio Apps documentation.

array

List of API endpoint configurations.

endpoints:
  - path: endpoints/webhooks.yml

Learn more in the Endpoints documentation.

Pipeline Configuration

string

required

Short, unique identifier for the pipeline. Used in CLI commands and UI.

alias: daily_etl

string

required

Relative path to the pipeline Python file from project root.

path: pipelines/etl_pipeline.py

string

default:"SMALL"

Compute instance size for pipeline execution.Available sizes:

XSMALL - 2 vCPU, 8 GB RAM
SMALL - 4 vCPU, 16 GB RAM
MEDIUM - 8 vCPU, 32 GB RAM
LARGE - 16 vCPU, 64 GB RAM
XLARGE - 32 vCPU, 128 GB RAM (Enterprise only)

compute: LARGE

string

default:"local"

Spark deployment mode for distributed processing.Options:

local - Runs on a single machine (default)
client - Driver runs in the same process, executors run separately
cluster - Both driver and executors run in separate processes (Enterprise only)

spark_config:
  deploy_mode: client

integer

Number of Spark executors for parallel processing. Only applies when deploy_mode is client or cluster.

spark_config:
  deploy_mode: client
  executor_instances: 5

object

Additional Spark configuration properties. Pass any custom Spark configuration key-value pairs.

spark_config:
  deploy_mode: client
  executor_instances: 3
  extra_spark_config:
    spark.sql.shuffle.partitions: "200"
    spark.default.parallelism: "100"
    spark.sql.adaptive.enabled: "true"

array

Python packages required by the pipeline. Installed before execution.

python_dependencies:
  - name: pandas
    version: 2.0.0
  - name: requests
    version: 2.31.0
    index_url: https://pypi.org/simple
  - name: my-package
    extra_arguments:
      - "--no-dependencies"

Python Dependency Fields

string

required

Python package name from PyPI or custom index.

- name: numpy

string

Specific package version. If omitted, installs the latest version.

- name: pandas
  version: 2.0.0

string

Custom Python package index URL. Useful for private packages or mirrors.

- name: my-private-package
  version: 1.2.3
  index_url: https://pypi.mycompany.com/simple

array

Additional arguments passed directly to the package installer for this dependency.

- name: my-package
  version: 1.2.3
  extra_arguments:
    - "--no-dependencies"

Action Configuration

string

required

Relative path to the action Python file containing an @action decorated function.

actions:
  - path: actions/send_notification.py
  - path: workflows/data_validator.py

Each file should contain one action function. Learn more in the Actions documentation.

App Configuration

string

required

Relative path to the intelligent app Python file.

apps:
  - path: apps/customer_assistant.py

Studio App Configuration

string

required

Short, unique identifier for the app. Used in its served URL.

alias: orders

string

Display name shown in Datazone. Defaults to the alias.

name: Orders

string

required

Relative path to the app directory — the folder holding package.json.

studio_apps:
  - alias: orders
    name: Orders
    path: studio/orders

The entry is created for you when you add a Studio App, and removing it deletes the app and its directory. Each app is built per branch; see the Studio Apps documentation.

Knowledge Object Configuration

string

required

Relative path to the object definition YAML file. One file may hold several objects, separated by ---.

objects:
  - path: objects/order.yml
  - path: objects/customer.yml

Learn more in the Knowledge Objects documentation.

Flow Configuration

string

required

Relative path to the flow definition YAML file.

flows:
  - path: flows/daily_report.yml

Learn more in the Flows documentation.

Endpoint Configuration

string

required

Relative path to the endpoint configuration YAML file.

endpoints:
  - path: endpoints/webhooks.yml
  - path: endpoints/api_routes.yml

Learn more in the Endpoints documentation.

Example Projects

Data Processing Pipeline

project_name: sales-analytics
project_id: 67280ba2f4a0960d02159675

pipelines:
  - alias: daily_sales_etl
    path: pipelines/sales_pipeline.py
    compute: MEDIUM
    spark_config:
      deploy_mode: client
      executor_instances: 3
    python_dependencies:
      - name: pandas
        version: 2.0.0
      - name: sqlalchemy
        version: 2.0.0

Multi-Component Project

project_name: customer-platform
project_id: 67280ba2f4a0960d02159675

pipelines:
  - alias: customer_data_sync
    path: pipelines/sync.py
    compute: SMALL
    python_dependencies:
      - name: requests
      - name: pydantic
        version: 2.0.0

actions:
  - path: actions/send_welcome_email.py
  - path: actions/validate_customer.py

apps:
  - path: apps/support_chatbot.py

objects:
  - path: objects/customer.yml

flows:
  - path: flows/onboarding.yml

studio_apps:
  - alias: customer_portal
    name: Customer Portal
    path: studio/customer_portal

endpoints:
  - path: endpoints/customer_api.yml

Get Started

Reference

Deployment

Tutorial

Project Repository

Overview

Project Structure

Configuration File

Configuration Reference

Project Fields

Pipeline Configuration

Python Dependency Fields

Action Configuration

App Configuration

Studio App Configuration

Knowledge Object Configuration

Flow Configuration

Endpoint Configuration

Example Projects

Data Processing Pipeline

Multi-Component Project

Next Steps

​Overview

​Project Structure

​Configuration File

​Configuration Reference

​Project Fields

​Pipeline Configuration

​Python Dependency Fields

​Action Configuration

​App Configuration

​Studio App Configuration

​Knowledge Object Configuration

​Flow Configuration

​Endpoint Configuration

​Example Projects

​Data Processing Pipeline

​Multi-Component Project

​Next Steps

Overview

Project Structure

Configuration File

Configuration Reference

Project Fields

Pipeline Configuration

Python Dependency Fields

Action Configuration

App Configuration

Studio App Configuration

Knowledge Object Configuration

Flow Configuration

Endpoint Configuration

Example Projects

Data Processing Pipeline

Multi-Component Project

Next Steps