# AI Skills
Source: https://docs.datazone.co/ai-skills
Install the official Datazone agent skills so your AI coding assistant can build pipelines, endpoints, flows, and apps correctly
# AI Skills
Datazone publishes a set of **public agent skills** that teach AI coding assistants how to work with Datazone. Install them once, and your agent knows the YAML schemas, the `config.yml` registration rules, the deploy loop, and the mistakes that produce a project which validates and then fails at runtime.
Without them, an assistant guesses at Datazone's conventions. With them, it writes a `@transform` that deploys, an endpoint YAML that resolves, and a Studio App that does not 404 on its own assets.
Eight skills covering pipelines, endpoints, flows, objects, apps, and the API
## Install
Run this in the root of your Datazone project repository:
```bash theme={null}
npx skills add datazoneco/datazone-skills
```
The CLI detects which assistant you use and installs the skills where it expects to find them. Supported clients include **Claude Code**, **Cursor**, **Codex**, **GitHub Copilot**, **Windsurf**, **Gemini**, **Cline**, **AMP**, and **Antigravity**.
To install a single skill instead of the whole collection, pass `--skill`:
```bash theme={null}
npx skills add datazoneco/datazone-skills --skill datazone-pipeline
```
Skills are instructions, not credentials. They tell your agent how Datazone works — they do not grant it access. Your agent still authenticates with your own [API key](/reference/development/api-key) or `datazone` CLI [profile](/reference/development/command-line), under your existing [policies](/reference/development/policy).
## What's included
| Skill | Use it for | Reference |
| --------------------------- | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------- |
| `datazone-project-setup` | Installing the CLI, creating profiles, cloning a project, and the edit-deploy-verify loop | [Command line](/reference/development/command-line) |
| `datazone-pipeline` | Python `@transform` functions, `input_mapping`, `output_mapping`, materialization | [Pipelines](/reference/development/pipeline) |
| `datazone-knowledge-object` | Versioned business entities in YAML — fields, primary keys, relationships, action handlers | [Knowledge Objects](/reference/knowledge-objects/overview) |
| `datazone-endpoint` | Exposing a query, action, or vector search as an authenticated REST API | [Endpoints](/reference/integration/endpoints) |
| `datazone-flow` | Declarative orchestration — `llm_call`, `rest_call`, `for_each`, Python nodes, connections | [Flows](/reference/flows/overview) |
| `datazone-intelligent-app` | YAML dashboards — charts, filters, tabs, widgets | [Intelligent Apps](/reference/intelligent-apps/overview) |
| `datazone-studio-app` | Vite + React SPAs served behind your organisation's session | [Studio Apps](/reference/studio-apps/overview) |
| `datazone-api` | Calling the REST API — `x-api-key` auth, filtering, sorting, pagination, linked documents | [API reference](/api-reference/introduction) |
Each skill loads only when it is relevant. Ask your agent to "add a chart to the sales app" and it reads `datazone-intelligent-app`; ask it to "expose this query as an API" and it reads `datazone-endpoint`. You do not choose between them by hand.
## Using them
Once installed, work in plain language. The skill supplies the schema and the conventions.
```text theme={null}
Add a pipeline that reads the orders dataset, aggregates revenue by
region, and materializes the result.
```
The agent writes `pipelines/revenue_by_region.py`, registers it in `config.yml` under `pipelines:` with a unique alias, commits, and pushes — because pushing is what deploys. It then checks the deploy result, since validation is server-side and asynchronous.
Some further examples that map cleanly onto a single skill:
* *"Create a Knowledge Object for Contract with a relationship to Customer."*
* *"This flow fails validation — the `for_each` node isn't receiving its input."*
* *"Expose the unpaid-invoices query as an endpoint and show me how to call it."*
* *"My Studio App builds but shows a blank page."*
Skills are most useful inside your project repository, where the agent can read your actual `config.yml`, datasets, and existing definitions. Several skills begin by confirming they are in a Datazone project — one with a `config.yml` at the root — and will stop and ask if they are not.
## Keeping them current
Datazone's YAML schemas and CLI evolve. Re-run the install command to pull the latest published versions:
```bash theme={null}
npx skills add datazoneco/datazone-skills
```
The collection is versioned alongside the platform, so refreshing after a [release](/changelog) keeps your agent's knowledge aligned with what your deployment actually accepts.
## Related resources
Install the `datazone` CLI and manage profiles
How a project repository is structured and deployed
Datazone's built-in assistant for queries and apps
The REST API your agents and scripts call
# Change Logs
Source: https://docs.datazone.co/changelog
Release notes and feature updates for Datazone
* **[Studio Apps](reference/studio-apps/overview)**: When a dashboard is not enough, write the app. A Studio App is a **Vite + React single-page application that lives in your project repository** — Datazone installs its dependencies, builds it in a sandboxed job, and serves the bundle at a real URL **behind your organisation's session**. The browser sends the signed-in user's session with every API call, so the app ships **no API key** and runs with that user's permissions. Each branch builds and serves its own version, so you can review a change before it reaches `main`. Routing, Tailwind, the shadcn setup, and a small Datazone client are **scaffolded for you** — there is no Dockerfile, no deployment pipeline, and no CDN to configure. Use it for the things a declarative dashboard cannot express: multi-step forms, write-back to [Knowledge Objects](reference/knowledge-objects/overview), and bespoke internal tools. See [Getting Started](reference/studio-apps/getting-started).
* **[Intelligent App Builder](reference/intelligent-apps/overview)**: Intelligent Apps no longer have to start in YAML. Build them **visually** — drag a chart onto the grid, then edit its query, metrics, and dimensions **directly in the UI**. [Orion](reference/intelligent-apps/orion-ai) is still there and does considerably more than help you write config: ask it to *"add a new customer trend"* and it **adds the chart to your app**, query and all. The YAML has not gone anywhere — the builder and the file are two views of the same app, so you can lay an app out by hand and still review it as a diff.
* **Generate policies with Orion**: [Policies](reference/development/policy) give you very granular control — per resource, per action, per project, down to row- and column-level restrictions — but writing that JSON by hand is tedious and easy to get subtly wrong. Now you describe the access you want in plain language and **Orion drafts the policy statements** for you. You review the suggestion before it is applied, so you still decide what gets granted.
* **[Flows](reference/flows/overview)**: Orchestrate work as a **directed graph of typed nodes** instead of glue code. Call an **LLM**, hit a **REST API**, branch with `if`, loop with `for_each`, run **Python**, or invoke a project [Action](reference/development/actions) — wired together on the Flow Builder canvas or written directly as **declarative YAML**. The document you edit is exactly what executes, so every run is **reproducible and versionable**, triggered **manually or on a schedule**, with per-node step status for each run.
* **[Custom Widgets in Intelligent Apps](reference/intelligent-apps/components#widgets)**: When none of the built-in chart types fit, render **your own React component** right in the app grid. Widgets are **TSX components compiled in the browser at runtime** — define them inline or keep them as `file` components in your repository for editor support and reviewable diffs. Attach a SQL query and its rows arrive as the component's `data` prop, with **Jinja templating** over the app's filter variables and the same caching as charts. Use `setFilter` from `appContext` to make a widget an **interactive participant**: a click inside your component drives every other chart and filter in the app.
* **[Knowledge Objects](reference/knowledge-objects/overview)**: Until now, Datazone could store and transform your data — now it can **create** it. Define your own tables in a couple of clicks and start inserting rows right away. Model the entities of your domain — `Employee`, `Invoice`, `Company` — as declarative YAML, get a governed CRUD API over every instance, with versioning, relationships, and branch awareness built in.
* **SQL Explorer — Generate Query**: Write SQL without writing SQL. Click the sparkles icon, type what you want in plain language — "show me unpaid orders" — and Orion generates the query and inserts it straight into your tab. "Show me all orders above \$10,000 in the last 30 days, grouped by region" becomes a working `SELECT` statement before you've opened a second tab. No syntax lookup, no schema guessing.
* **[Embedded Agents](reference/agents/embedding)**: Take the agents you've built on top of your lakehouse and drop them straight into **your own apps, websites, or any third-party tool** — with **just one line of code**. The embedded agent connects to the **same data sources, tools, and model configuration** you set up in Datazone, so your users get the full conversational experience without ever leaving your product. Choose between **Full**, **Bot**, or **Drawer** widget styles, and generate a secure, signed snippet right from the **Embedding** tab of any agent.
* **SQL Explorer**: A complete SQL workspace inside Datazone — **write, run, save, and revisit** your queries without ever losing your place. Get **AI-powered autocomplete** on every column, **sub-second results**, and **one-click access** to every table in your lakehouse. Work across **multiple tabs** that persist between sessions, save your favourite queries to a reusable library, and **replay any past run** from query history. Built for **SQL-first analyst workflows**, with transpose and export to CSV, Parquet, or JSON.
* **[Vectors](reference/development/vectors)**: Transform your data into searchable embeddings for AI-powered applications. Create vector indexes from datasets or file containers, configure embedding models and chunking strategies, and enable semantic search that understands meaning and context. Attach vectors to [Agents](reference/agents/overview) for RAG-powered conversations or create [Vector Endpoints](reference/integration/endpoints#vector-based-endpoints) to build semantic search backends for your applications.
* **[Dynamic Chart Inputs](reference/intelligent-apps/components#dynamic-charts-with-chart-inputs)**: Add interactive dropdown controls directly to charts for dynamic query modification. Users can switch dimensions, change grouping levels, or adjust filters without leaving the chart view. Works seamlessly with Jinja templating for powerful, flexible visualizations.
* **Data Profiling**: Understand your data at a glance without manual exploration. Data profiling automatically analyzes your datasets and views, revealing key insights through visual charts. For categorical columns, see value distribution with histograms and pie charts. For numeric columns, view statistical metrics like average and standard deviation. Get instant visibility into your data's characteristics.
* **[Channels](reference/platform/channels)**: Integrate Datazone with external communication platforms. Configure Slack or email channels to deliver scheduled reports and notifications to your team.
* **[Reports](reference/platform/reports)**: Automate Intelligent App delivery on a schedule. Generate reports as PDF or PNG and send them via Slack or email using cron expressions for flexible scheduling.
* **[Quotas](reference/platform/resources/quotas)**: Set resource limits to control costs and prevent unexpected usage. Create quotas for DCU, tokens, storage, queries, and actions. Get email notifications at 80% usage and automatic service pause at 100% to stay in control of consumption.
* **[Actions](reference/development/actions)**: Deploy **serverless Python functions** that execute on-demand via **API endpoints** or as **tools for AI agents**. Write custom logic with the `@action` decorator, use the **context API** for logging, and automate workflows like **sending notifications**, **processing data**, or **calling external APIs**.
* **[Datazone Agents](reference/agents/overview)**: Create **custom AI assistants** that interact with your data through **natural language**. Agents can execute **SQL queries**, run **Python code**, generate **charts**, and **search the web** automatically. Supports **multiple languages** and remembers **conversation context** for intelligent, multi-step analysis.
* **[Model Accounts](reference/development/model-accounts)**: Securely manage **AI provider credentials** for **OpenAI**, **Anthropic (Claude)**, and **AWS Bedrock**. Configure once and reuse across projects and agents with **encrypted storage** and **organization-level access control**.
* **[New Clear Layout](reference/ui-overview)**: Redesigned interface with **card-based home page**, **streamlined project workspace**, and **⌘K quick search** for instant navigation to any resource from anywhere in the app.
* **[Scatter Plot Chart Type](reference/intelligent-apps/components#scatter-plot-charts)**: Introduced a new scatter plot chart type for visualizing relationships between two numerical variables. Each point represents a data record positioned according to two metrics on the x and y axes, ideal for identifying correlations, patterns, or outliers in your data.
* **[Custom Theme Styling](reference/intelligent-apps/yaml-reference#custom-style-attributes)**: Added `custom_style_attributes` for advanced theme customization. Override specific theme colors using Tailwind CSS variables in OKLCH format, giving you fine-grained control over backgrounds, foregrounds, and chart color palettes.
* **[Conditional Chart Visibility](reference/intelligent-apps/yaml-reference#example-conditional-chart-visibility)**: Added `hide_expression` to `chart_config`, allowing you to conditionally hide charts based on query results. Use JavaScript expressions to control chart visibility dynamically (e.g., hide charts with insufficient data).
* **[Policy System](reference/development/policy)**: Introduced a comprehensive role-based access control system with hierarchical permissions! Define fine-grained access policies with support for **flat and hierarchical resource patterns** (e.g., `project::*`), explicit allow/deny rules, and resource-specific constraints. Policies support wildcards for flexible matching, branch-aware permissions, and extra constraints like **row-level security** for views and **path restrictions** for projects. Bind policies to roles for scalable permission management across your organization.
* **[Views](reference/integration/views)**: Introduced a powerful new feature - Views! Transform your datasets into optimized relational database structures with advanced configurations like **partitioning, primary keys, and ordering**. Choose between **materialized views** for lightning-fast query performance or **non-materialized views** for real-time data access. Create views by replicating datasets or writing custom SQL queries that combine multiple datasets and views. Access your views through SQL interface, endpoints, and intelligent apps for enhanced performance and flexibility.
* **[Heatmap Chart Type](reference/intelligent-apps/components#heatmap-charts)**: Introduced a new heatmap chart type for visualizing data where values are represented as colors. Heatmaps require exactly two dimensions and one metric, making them perfect for showing relationships between categorical dimensions.
* **[Markdown Component](reference/intelligent-apps/components#text-markdown)**: Added support for Markdown components in Intelligent Apps. Users can now add rich text sections with formatting, headers, lists, and more to their dashboards.
* **[Multiple Selection Filters](reference/intelligent-apps/filters#handling-multiple-selection-filters)**: Enhanced dropdown filters with multiple selection capability. Users can now select multiple values from a dropdown filter, with proper handling in SQL queries using Jinja templates.
* **[Embedded Intelligent Apps](reference/intelligent-apps/embedding)**: Introduced the ability to embed Intelligent Apps into external applications via iframes. Generate a secure JWT token containing app and user details, and easily integrate dashboards into your own products.
* **[Theme selection for Intelligent Apps](reference/intelligent-apps/yaml-reference#style-configuration)**: Added theme customization options for Intelligent Apps with multiple color schemes: default, teal, blue, green, purple, orange, amber, and mono. Set themes through the `style.theme` configuration property.
* **[Chart export features](reference/intelligent-apps/yaml-reference#config-section)**: Added export capabilities for Intelligent App charts. Users can now download chart data as CSV files or save visualizations as PNG images. Enable with the `chart_export_enabled` configuration property.
* **[File Container](reference/development/file-container)**: Introduced File Container, a **storage solution** for managing files in Datazone. It allows you to create, update, and delete file containers, and use them to store data for your pipelines and notebooks. You can interact with file containers like AWS S3 buckets or Google Cloud Storage buckets using the `FileContainerClient` toolkit.
* **Orion Notebook Assistant:** Added support for **code suggestion** and **error suggestions** in Orion Notebooks. Just click to Sparkles ✨ and tell what you want to do, and Orion will suggest code snippets.
* **[Endpoints](reference/integration/endpoints)**: Introduced a new core concept - Endpoints! Users can now create secure, controlled API interfaces for their datasets. Define endpoints using YAML configuration with filters, queries, and authentication. Perfect for exposing analytical results, feeding data to downstream systems, or creating data products that other teams can consume.
* **[chart\_config](reference/intelligent-apps/components#chart-configuration) attribute**: The `chart_config` attribute is now available for all chart types, allowing you to customize chart appearance with options like `fill_donut`, `show_labels`, and `show_legend`.
* **[Stacked bar charts](reference/intelligent-apps/components#stacked-bar-charts)**: Added support for stacked bar charts using the `is_stacked` attribute, allowing you to visualize multiple metrics in a single bar
* **[fill\_donut](reference/intelligent-apps/components#chart-configuration) for pie charts**: The `fill_donut` attribute is now available for pie charts, allowing you to fill the donut area for a more visually appealing design.
* **[Radial chart type](reference/intelligent-apps/components#radial-charts)**: Added support for the new `radial` chart type, perfect for displaying progress or completion metrics in a circular format. Requires exactly two metrics where the first represents the current value and the second represents the total.
* **[Filter dependencies and cascading filters](reference/intelligent-apps/filters#filter-dependencies)**: Enhanced filter functionality with dependent filters that create cascading effects. Filters can now reference other filter variables in their SQL queries, enabling hierarchical filtering like country → city relationships.
* **AI-driven deployment error assistance**: New deployment activity feature allows users to directly ask Orion Code Assistant for help when encountering deployment errors, providing intelligent troubleshooting and fix suggestions.
* **[Icon feature for number charts](reference/intelligent-apps/components#number-charts)**: You can now add Lucide icons next to number values in number charts using the `icon` and `icon_variant` attributes.
* **[metric\_format to format](reference/intelligent-apps/yaml-reference#charts)**: The `metric_format` attribute is deprecated. Use the new `format` attribute under each metric instead.
* **[Composed (combined) chart type](reference/intelligent-apps/components#composed-charts)**: Added support for the new `composed` chart type, allowing you to combine line and bar series in a single chart.
* **[Axis support for line, bar, and composed charts](reference/intelligent-apps/components#axis)**: You can now define multiple axes using the `axis` property and assign metrics to axes with `axis_name`.
* **[affected\_filter for bar and pie charts](reference/intelligent-apps/components#bar-charts)**: Add `affected_filter` under a dimension to allow users to update filter values by clicking chart items (bars or pie slices).
# Private Cloud
Source: https://docs.datazone.co/deployment/private-cloud
Enterprise deployment of Datazone in your cloud environment
# Private Cloud Deployment
Private cloud deployment allows enterprises to run Datazone in their own cloud infrastructure with enhanced security, control, and customization options.
## Supported Cloud Providers
### AWS
* EKS for container orchestration
* S3 for data lake storage
### Azure
* AKS for container orchestration
* Blob Storage for data lake
## Enterprise Features
### Enhanced Security
* VPC/VNET integration
* Customer-managed encryption keys
* LDAP/Active Directory integration
* IP whitelisting
* Audit logging
* Custom security policies
### Advanced Operations
* Multi-region deployment
* Custom backup policies
* Disaster recovery options
* Load balancing configuration
* Auto-scaling policies
* Resource quotas
### Data Governance
* Data lineage tracking
* Access control policies
* Compliance monitoring
* Data retention rules
* Audit trails
* Custom metadata
## System Requirements
### Kubernetes Cluster
* Minimum 3 nodes
* 4 CPU cores per node
* 16GB RAM per node
* 100GB storage per node
### Object Storage
* S3-compatible storage
* Minimum 1TB capacity
* High availability setup
## Support & SLA
### Enterprise Support
* 24/7 technical support
* 1-hour response for critical issues
* Dedicated support engineer
* Quarterly review meetings
* Custom feature development
### SLA Terms
* 99.99% uptime guarantee
* 15-minute RPO
* 1-hour RTO
# Public Cloud
Source: https://docs.datazone.co/deployment/public-cloud
Datazone public cloud deployment for community and pro users
# Public Cloud Deployment
Datazone's public cloud offering is hosted at [https://app.datazone.co](https://app.datazone.co), providing a fully managed solution for community and professional users.
## Available Tiers
### Community Edition
* Free tier for individual users and small teams
* 1 Data Project
* 2 Users
* 5 DCU Compute
* 250 MB of Hot Storage
* 1GB of Cold Storage
* Community support
### Professional Edition
* Suitable for growing teams
* Including all Community features
* Unlimited Data Projects
* Orchestration
* Data Branching & Time Travel
* API access
* Email support
## Features
### Infrastructure
* Hosted on AWS infrastructure
* Automatic scaling
* Daily backups
* 99.9% uptime SLA (Professional)
* Global CDN
### Security
* SOC2 compliant
* Data encryption at rest and in transit
* Regular security audits
* Two-factor authentication
* Single Sign-On (Professional)
### Maintenance
* Automatic updates
* 24/7 monitoring
* Zero-downtime deployments
* Automated backups
## Getting Started
1. Sign up at [https://app.datazone.co/app/sign-up](https://app.datazone.co/app/sign-up)
2. Create your organization and choose your plan
3. Create your first project
4. Configure data sources
5. Start building pipelines
## Support Options
### Community Edition
* Documentation access
* Bug reporting/Feedback system
### Professional Edition
* Email support (24-hour response)
* Screen sharing sessions
* Priority bug fixes
# From Zero to Production!
Source: https://docs.datazone.co/from-zero-to-production
In this guide, you will learn how to build a Data Lakehouse from scratch using Datazone, including creating intelligent apps, deploying AI agents, and exposing data via secure endpoints.
## Prerequisites
Before you start building your Data Lakehouse, make sure you have the following prerequisites:
* A Datazone account. If you don't have one, you can sign up [here](https://app.datazone.co/app/sign-up).
* Datazone CLI installed on your local machine. You can install it by following the instructions [here](/installation).
## Task List
To understand how to build a Data Lakehouse from scratch using Datazone, let's follow these steps:
1. 🔌 **Connecting your data source**: Start by connecting AWS S3 as a data source.
2. 📁 **Initialize first project**: Set up your first project and add an Extract component.
3. 🚀 **Run first execution**: Launch your first execution to fetch data from the source.
4. 📄 **Create first pipeline**: Design a simple pipeline to process the data.
5. 🚂 **Run first pipeline**: Execute the pipeline to transform your data.
6. ⏰ **Create first schedule**: Configure periodic runs for automated processing.
7. 🧠 **Create an Intelligent App**: Turn your data pipelines into context-aware applications that observe, reason, and act — not just visualize.
8. 🤖 **Deploy an Agent**: Set up AI-powered automation for monitoring and insights.
9. 🔗 **Expose data via Endpoints**: Create secure APIs for your processed data.
10. 🏆 **Access the data**: Learn how to query and use the processed data.
## 🔌 Connect Source
1.Go to **Settings** from the top-right user menu, then select **Sources** under the **Integrations** section.
1. Click on the **Create Source** button.
1. Fill in the required fields and click on the **Create** button. And you are done! You have successfully connected your source. Check your source in the **Settings** > **Sources** page.
## 📁 Create Project
1. Go to the **Projects** page by clicking on the **Projects** tab in the sidebar.
1. Click on the **Create Project** button.
1. Fill in the required fields and click on the **Create** button. Boom! 🚀 You have successfully created your first project.
### Define your Extract
1. On your project page, click on the **Add** button in the top right corner to add a new entity.
2. Select **Extract** as the entity type.
1. Fill in the required fields and click on the **Create** button. You have successfully created your first Extract entity.
**Base Attributes**
* `name`: The name of the extract.
* `source`: The source you want to extract data from. (It is already selected)
* `mode`: The mode of the extract. Options are;
* `Overwrite`: Fetch all the data from the source every time.
* `Append`: Fetch only the new data from the source.
**Source Dependent Attributes** (In this case, AWS S3). Check the [AWS S3](/sources/awss3) page for more details.
* `search_prefix`: The prefix you want to search for in the bucket.
* `search_pattern`: The pattern you want to search for in the bucket.
## 🚀 Run First Execution and Check the Data
1. Click to the created Extract entity and move to the **Executions** tab. Via clicking the **Run** button, you can start your first execution.
1. Simultaneously, you can check the execution logs and the other details in the **Logs** tab. You can cancel the execution if you want.
After a while, execution will be completed and you notice the new dataset in left explorer. You can check the data by clicking on the dataset.
1. On the dataset drawer, you can see the data fetched from the source. You can also check the schema and make queries on the data to explore it.
1. With above way, we can fetch the other csv files from the source and create the datasets for each of them.
## ⌨️ Click Less, Code More: Create First Pipeline
If you have already created your project on the UI, open the project page and move to the "Code" section in the left tabs. For local development, clone it using the Datazone CLI.
```shell theme={null}
datazone project clone
```
You will see:
```text theme={null}
Repository has initialized
👉 Go to repository directory: cd ecommerce-project/
```
Check your project folder
```shell theme={null}
> cd ecommerce-project/
> ls -ll
ecommerce-project/
├── README.md
├── hello-world.py
├── config.yml
```
1. We can create our pipeline file in the project folder. Let's create a new file named `order_reports.py` in the project folder.
```python order_reports.py theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql import functions as F
@transform(
input_mapping={
"orders": Input(Dataset(alias="orders_c90dc0")),
"order_lines": Input(Dataset(alias="order_lines_8c5238")),
"customers": Input(Dataset(alias="customers_9cd9ab")),
}
)
def join_tables(orders, order_lines, customers):
return orders.join(order_lines, on="OrderID", how="inner").join(
customers, on="CustomerID", how="inner"
)
@transform(input_mapping={"joined": Input(join_tables)}, materialized=True)
def sales_by_country(joined):
country_report = (
joined.groupBy("Country")
.agg(
F.sum("TotalAmount").alias("TotalSales"),
F.count("OrderID").alias("OrderCount"),
)
.orderBy("TotalSales", ascending=False)
)
return country_report
@transform(input_mapping={"joined": Input(join_tables)}, materialized=True)
def most_popular_products(joined):
product_report = (
joined.groupBy("ProductID")
.agg(
F.sum("TotalAmount").alias("TotalSales"),
F.sum("Quantity").alias("TotalQuantity"),
F.avg("UnitPrice").alias("AveragePrice"),
F.count("OrderID").alias("OrderCount"),
)
.orderBy("TotalSales", ascending=False)
)
return product_report
```
Above code is a simple pipeline that joins three tables and creates two reports.
* You can see that we have two functions that are decorated with `@transform`. These functions are the steps of the pipeline.
You can specify the input datasets and the output dataset of the function by using the `input_mapping` and `output_mapping` classes.
* The first function `join_tables` joins the `orders`, `order_lines`, and `customers` tables.
* The second function `sales_by_country` calculates the total sales and order count by country.
* The third function `most_popular_products` calculates the total sales, total quantity, average price, and order count by product.
You can create your own pipeline according to your needs. Also you can check the [Transform Functions](/transform-functions)
page to see the available functions.
1. Then you need to reference this pipeline in the `config.yml` file.
```yaml config.yml theme={null}
project_name: ecommerce-project
project_id: 673f8ef62a466524757a7de1
pipelines:
- alias: order_reports
path: order_reports.py
```
1. Write your code in the "Code" section of the project page and click the Deploy button.
And you will see the deployed pipeline if you click the newly created pipeline in the project page.
1. Click the Execute button in the pipeline page.
There are many ways to do something in Datazone. You can run your pipeline via
UI, CLI or API.
1. While execution is running, you can check the logs both in the terminal and in the UI.
After the execution is completed, you can check the logs and the output dataset in the **Executions** tab.
1. Our new **Dataset** is ready to use. You can check and explore the data in the dataset drawer.
## ⏰ Orchestrate Your Pipeline
1. Select the pipeline you want to schedule in the Explorer.
2. Open the **Schedules** tab and click on the **+ Set Schedule** button.
3. Attributes are:
* `pipeline`: The pipeline you want to schedule. (It is already selected)
* `name`: The name of the schedule.
* `expression`: The cron expression for the schedule. You can use the presets or write your own.
## 🧠 Create an Intelligent App
At this point, your pipelines are producing reliable and structured datasets.
In Datazone, an Intelligent App is built on top of these pipelines to turn data into **context-aware applications**.
An Intelligent App can:
* Observe changes in datasets over time
* Understand relationships between entities
* Surface meaningful insights without manual queries
This layer moves your system beyond data processing and enables it to support decisions and workflows.
***
## 🤖 Deploy an Agent
Agents bring automation and intelligence into your Datazone applications.
An Agent continuously monitors selected datasets, pipelines, or executions and reacts based on context.
Typical use cases include:
* Detecting anomalies in data or execution behavior
* Monitoring pipeline health and failures
* Identifying unusual trends or patterns
* Providing proactive insights without user interaction
Once deployed, agents run continuously and become part of your production system.
***
## 🔗 Expose Data via Endpoints
After processing and enriching your data, you can expose it using Endpoints.
Endpoints allow external applications and services to securely access your datasets via APIs.
Key benefits:
* Secure access using API keys
* Controlled exposure at dataset or view level
* No direct database access required
This enables frontend applications, integrations, and services to consume trusted data safely.
## 🏆 Access the Data
### SQL Interfaces
```sql Clickhouse theme={null}
clickhouse-connect --host=app.datazone.co --port=8443 --user=your-user --password=your-password
```
```sql MySQL theme={null}
mysql -h app.datazone.co -P 3306 -u your-user -p
```
```sql PostgreSQL theme={null}
psql -h app.datazone.co -U your-user -d your-database
```
## Related Resources
Create secure, controlled API interfaces for your datasets
Generate and manage API keys for programmatic access
Connect to Datazone using Clickhouse ODBC or JDBC drivers
Transform datasets into optimized relational database structures
# Installation
Source: https://docs.datazone.co/installation
Before starting to use Datazone, you need to install the Datazone Python CLI. The client is a Python package that provides a simple interface to interact with the Datazone API.
## Requirements
Use the following command to check that you have a supported Python version installed:
```bash theme={null}
python --version
```
Currently, Python 3.8, 3.9, 3.10, 3.11 and 3.12 are supported
## Installation
* You can install the Datazone Python SDK using pip:
```bash theme={null}
pip install datazone
```
* Check that the installation was successful by running the following command:
```bash theme={null}
datazone --version
```
Check Datazone CLI section for more information on how to use the CLI.
## Get API Key
To use the Datazone CLI, you need to create an API key. You can do this by following these steps:
1. Go to the **Settings** page by clicking on the **Settings** tab in the sidebar.
2. Click on the **API Keys** tab in the left sidebar.
3. Click on the **+Create** button.
## Login via CLI
If you have already created an account, you can log in to the Datazone CLI using the following command:
```text theme={null}
datazone profile create
> Profile [default]:
> Host [app.datazone.co]: (If you are using a self-hosted version)
> API Key:
> Password:
> Repeat for confirmation:
You logged in successfully 🎉
```
After you have successfully logged in, you can start using the Datazone CLI to interact with the Datazone API.
You can check you session status by running the following command:
```bash theme={null}
datazone project list
```
For more information on how to use the Datazone CLI, check the [Command Line](/reference/development/command-line) section.
# Introduction
Source: https://docs.datazone.co/introduction
Build, manage, and serve your data pipelines and intelligent applications with ease using Datazone in minutes.
Datazone is a modern data platform that empowers you to create, orchestrate, and analyze data workflows with a few clicks, while building interactive apps and deploying AI-powered agents.
* Create your **[Sources](/key-concepts#source)** and integrate them with your data lakehouse.
* Define your source entities with **[Extracts](/key-concepts#extract)**.
* Build **[Pipelines](/key-concepts#pipeline)** to clean, filter, and manipulate your data.
* Orchestrate your pipelines with the **[Schedule](/key-concepts#schedule)** feature.
* Analyze your data with the **[Notebook](/key-concepts#notebook)**.
* Create **[Intelligent Apps](/key-concepts#intelligent-app)** for interactive dashboards, charts, and real-time insights.
* Deploy **[Agents](/key-concepts#agent)** for automated data processing, AI-driven analytics, and custom workflows.
## Getting Started
Start your journey by setting up your environment and building your first project.
You need an account to create and manage your data pipelines and apps.
Install the Datazone Python CLI to interact with the Datazone API via command line.
## Next Steps
Follow our step-by-step guide to create your first project, pipelines, and schedules.
Build interactive dashboards from your pipelines with drag-and-drop components
and AI chat.
Integrate Datazone with your applications using REST APIs, webhooks, and
embedding options.
Learn more about how Datazone handles Sources, Extracts, Pipelines, and Agents.
# Key Concepts
Source: https://docs.datazone.co/key-concepts
Datazone is a modern data platform that simplifies your data engineering journey by providing a unified environment for data ingestion, processing, analysis, and AI-driven automation. It seamlessly connects your data sources to a robust data lakehouse while offering powerful tools for transformation, orchestration, exploration, interactive apps, and intelligent agents.
## Core Entities
### Source
Think of Sources as secure gateways to your data. They act as bridges between your external data systems (databases, cloud storage, streaming platforms) and Datazone. Sources handle the crucial task of credential management and access control, typically managed by organization administrators. They ensure your data connections are both secure and efficient.
### Project
Projects are your data initiatives home base. They provide a structured workspace where you can organize related data work - from ingestion to analysis. Each project is a self-contained environment housing **Extracts**, **Pipelines**, **Notebooks**, **Datasets**, **Schedules**, **Intelligent Apps**, and **Agents**. With flexible permission settings, you can control who accesses what, making it perfect for both team collaboration and data governance.
### Extract
Extracts are your data ingestion powerhouses. Connected to Sources and living within Projects, they define how data should be pulled from external systems. When executed, Extracts create standardized Datasets in your data Lakehouse. Think of them as smart data movers that handle the heavy lifting of data ingestion while ensuring data quality and consistency.
### Pipeline
Pipelines are where data engineering magic happens. Built as Directed Acyclic Graphs (DAGs), they represent your data transformation workflows. Pipelines are defined in code, making them version-controlled, reusable, and maintainable. They can clean, filter, join, and reshape your data, turning raw information into valuable insights.
### Schedule
Schedules bring automation to your data workflows. Using cron expressions, they orchestrate when your Extracts and Pipelines should run. They're the timekeepers of your data platform, ensuring your data processes run like clockwork, whether it's hourly updates or monthly aggregations.
### Notebook
Notebooks are your interactive playground for data exploration and analysis. Similar to Jupyter notebooks but integrated into Datazone, they provide a user-friendly interface where you can write code, visualize data, and debug your transformations. They're perfect for both quick data investigations and detailed analysis.
### Intelligent App
Intelligent Apps transform your data into interactive dashboards and applications without requiring frontend development. Using declarative YAML configuration, you can create multi-tab dashboards with charts, filters, and dynamic visualizations that query your datasets directly. They're perfect for building executive dashboards, operational monitoring tools, or self-service analytics interfaces that stakeholders can interact with through filters and drill-down capabilities.
### Agent
Agents are AI-powered assistants that automate data processing, analysis, and insights generation. Configured with custom instructions, they can monitor data pipelines, generate reports, trigger actions based on data conditions, and provide intelligent recommendations. Agents integrate with your data workflows to enable proactive data management, anomaly detection, and automated decision-making processes.
### Endpoints
Endpoints enable you to create secure, controlled API interfaces for your datasets. They transform your processed data into RESTful APIs that can be consumed by external applications, services, or third-party integrations. With built-in authentication and authorization controls, Endpoints ensure your data is shared securely while maintaining governance standards. They're perfect for exposing analytical results, feeding data to downstream systems, or creating data products that other teams can reliably consume.
## How It All Fits Together
Your data journey in Datazone typically flows like this:
1. Connect to external systems through **Sources**
2. Organize your work in **Projects**
3. Ingest data using **Extracts**
4. Transform data with **Pipelines**
5. Automate workflows using **Schedules**
6. Analyze results in **Notebooks**
7. Share insights through **Intelligent Apps**
8. Deploy intelligent automation with **Agents**
9. Expose data via secure **Endpoints**
This integrated approach ensures a smooth data lifecycle, from ingestion to insights and automation, while maintaining security, scalability, and ease of use.
Each entity in Datazone is designed to solve specific data engineering
challenges while working harmoniously with others. This modular yet integrated
approach makes Datazone powerful enough for complex data operations and
AI-driven automation yet simple enough for quick data tasks.
# Best Practices
Source: https://docs.datazone.co/reference/agents/best-practices
Optimize your agents for better performance, accuracy, and cost efficiency
# Best Practices
Follow these guidelines to build **effective, efficient agents** that deliver **accurate insights** while **managing costs**.
## Model Selection
### Choosing the Right Model
Different models excel at different tasks:
| Model Type | Best For | Cost | Speed |
| --------------------- | ---------------------------------------- | ------ | ------ |
| **GPT-4** | **Complex reasoning**, detailed analysis | High | Slower |
| **GPT-4o** | **Balanced performance**, general use | Medium | Medium |
| **GPT-4o-mini** | **Simple queries**, high volume | Low | Fast |
| **Claude 3.5 Sonnet** | **Code generation**, structured data | Medium | Medium |
| **Claude 3 Haiku** | **Quick answers**, basic queries | Low | Fast |
### Use Case Guidelines
**Complex Analysis** → GPT-4 or Claude 3.5 Sonnet
* **Multi-step reasoning** required
* **Deep data analysis**
* **Code generation** needs
**General Use** → GPT-4o or Claude 3.5 Sonnet
* **Balanced performance**
* Most common questions
* **Good for production** agents
**High Volume** → GPT-4o-mini or Claude 3 Haiku
* **Simple queries**
* **Cost-sensitive** applications
* **Fast response times** needed
Start with a **mid-tier model** (GPT-4o or Claude 3.5 Sonnet) and adjust based on actual usage patterns.
## Model Parameters
### Temperature
**Temperature** controls how **creative** or **focused** the agent's responses are:
* **0.0 - 0.3** (Focused)
* **Consistent, deterministic** answers
* Best for: **Data queries**, factual analysis
* Use when **accuracy is critical**
* **0.4 - 0.7** (Balanced)
* Mix of **consistency and creativity**
* Best for: **General purpose** agents
* **Recommended default**
* **0.8 - 1.0** (Creative)
* **Varied, exploratory** responses
* Best for: Brainstorming, recommendations
* Use **sparingly for data agents**
For most data analysis agents, keep temperature between **0.2 and 0.5** for reliable results.
### Max Tokens
**Max Tokens** controls the **maximum response length**:
* **1,000 - 2,000** - **Short, focused** answers
* **2,000 - 4,000** - **Standard responses** (recommended)
* **4,000+** - **Detailed analysis** and long explanations
**Higher limits = more cost**. Set based on expected answer complexity.
## Data Source Optimization
### Use Views Instead of Datasets
**Why Views Are Better:**
* ✅ **Faster query execution**
* ✅ **Pre-filtered**, relevant data only
* ✅ **Better security** (control data access)
* ✅ **Lower token usage**
* ✅ **More accurate** agent responses
**Example:**
```
Instead of: Full "orders" dataset (millions of rows)
Create: "recent_orders" view (last 90 days, key columns)
```
The agent queries **faster** and gets **relevant data immediately**.
**Always use Views instead of raw datasets** for better performance and cost efficiency.
### Limit Data Sources
Only grant access to **necessary datasets/views**:
**Too Broad:**
* Sales data
* HR data
* Marketing data
* Finance data
**Focused:**
* Sales summary view
* Revenue trends view
**Fewer sources = faster decisions + lower costs**.
### Optimize View Definitions
Create views with:
* **Only necessary columns**
* **Pre-aggregated data** where possible
* **Relevant date ranges**
* **Indexed fields**
## Agent Instructions
### Write Clear System Prompts
**Good Instructions:**
```
You are a sales analytics expert. Help users analyze:
- Revenue trends and forecasting
- Customer behavior and segmentation
- Product performance metrics
Always provide specific numbers and dates. When showing
trends, create visualizations. Be concise and actionable.
```
**Poor Instructions:**
```
You are helpful. Answer questions about data.
```
### Define Scope
Tell the agent:
* **What data** it has access to
* **What questions** it should handle
* **What format** responses should take
* Any **business rules** to follow
### Include Examples
Provide **specific examples** in instructions:
```
When asked about "top customers," show:
1. Customer name
2. Total revenue
3. Number of orders
4. Comparison to average
Example format:
"Top 5 customers by revenue:
1. Acme Corp - $150K (45 orders) - 3x average
..."
```
## Cost Optimization
### Control Token Usage
**Input Tokens:**
* Keep **system instructions concise**
* Limit **conversation history** length
* Use **focused data sources**
**Output Tokens:**
* Set **appropriate max\_tokens**
* Request **concise answers** when possible
* Avoid asking for **repeated information**
### Monitor Spending
* Review **token usage per conversation**
* **Track costs by agent**
* Set up **alerts for high usage**
* Regularly **audit agent performance**
### Batch Similar Questions
If running automated analysis:
* **Group related queries**
* **Reuse context** where possible
* **Cache frequently** accessed data
## Agent Configuration
### Tool Selection
Enable **only what you need**:
**Optional Tools:**
* **Chart Generator** - For data visualizations (recommended for most agents)
* **Python Executor** - For complex calculations and advanced analytics
* **Web Search** - For external context and current information
**Query Executor** is enabled by default and cannot be disabled.
### Data Access Control
* Grant **minimum necessary access**
* Use **views to restrict data**
* **Separate agents** by use case
* **Review permissions** regularly
### Regular Maintenance
* **Update instructions** based on user feedback
* **Refine data sources** as needs change
* **Adjust model selection** for cost/performance
* **Archive unused agents**
## Query Performance
### Faster Queries
1. **Use Views** - **Pre-filtered, optimized data**
2. **Limit Result Sets** - Ask for **"top 10"** not "all"
3. **Specific Time Ranges** - **"last month"** not "all time"
4. **Indexed Columns** - Ensure views use **indexed fields**
### Example Optimizations
**Slow:**
"Show all customer transactions"
**Fast:**
"Show top 20 customers by revenue this quarter"
## Testing & Validation
### Test Common Questions
Before deploying:
* Ask **typical user questions**
* **Verify data accuracy**
* Check **tool usage patterns**
* Measure **response times**
### Validate Answers
* Compare agent responses to **known results**
* **Verify calculations** manually
* Check **chart accuracy**
* Test **edge cases**
### Iterate
* Gather **user feedback**
* **Refine instructions**
* **Adjust parameters**
* **Optimize data sources**
## Security
### Data Access
* Only grant **necessary permissions**
* Use **views to limit sensitive data**
* **Review agent access** regularly
* **Audit query logs**
### Credentials
* **Secure model account** API keys
* **Rotate credentials** periodically
* **Monitor for unusual usage**
* Implement **access controls**
## Next Steps
* [Create Your First Agent](/reference/agents/overview)
* [Configure Model Accounts](/reference/development/model-accounts)
* [Set Up Data Views](/reference/integration/views)
* [Chat with Your Agent](/reference/agents/chat)
# Chat Interface
Source: https://docs.datazone.co/reference/agents/chat
Interact with your agents through conversational AI
# Chatting with Agents
The chat interface is where you **interact with your agents**. Ask questions, get insights, and explore your data through **natural language conversations**.
## Starting a Conversation
1. In home page, find your agent in **Agents** section
2. Select an agent from your list
3. Click **New Chat** to start a fresh conversation
4. Type your question and press **Enter**
## Chat Features
### Real-Time Streaming
Responses **stream in real-time** as the agent thinks and processes:
* See the agent's **thought process**
* Watch **tool executions** happen
* Get answers **progressively**
### Tool Indicators
**Visual indicators** show when the agent uses tools:
* **🔧 Using Tool** - Agent is **executing a query** or running code
* **💭 Analyzing** - Agent is **processing information**
* **✓ Completed** - Tool execution **finished**
These indicators help you understand **what the agent is doing** behind the scenes.
### Follow-Up Questions
After each response, the agent **automatically suggests** relevant follow-up questions:
```
💡 Suggested follow-ups:
• What is the revenue trend for the top customer?
• How do these customers compare to last quarter?
• Which products do these customers buy most?
```
Click any suggestion to **automatically ask** that question.
### Conversation History
Each chat maintains **full conversation history**:
* Return to **previous conversations** anytime
* Agent **remembers context** from earlier messages
* **Build on previous analysis**
### Token Usage & Cost
Each response shows:
* **Token count** (input + output)
* **Estimated cost**
* **Model used**
Monitor usage to **manage costs effectively**.
## Asking Effective Questions
### Be Specific
**Good:** "What were sales in California last month?"
**Vague:** "Tell me about sales"
### Context Helps
**Good:** "Show revenue by product category for Q4 2025"
**Unclear:** "Show revenue"
### Build on Context
```
You: "What were our top 5 products last month?"
Agent: [Provides list]
You: "Now show their profit margins"
Agent: [Analyzes same products from previous context]
```
### Ask for Visuals
```
"Show me..." → Agent generates charts
"Create a chart of..." → Visual response
"Compare..." → Comparison visualization
```
## Response Types
### Text Answers
**Simple, direct responses** to straightforward questions.
### Data Tables
**Structured data** presented in table format with:
* Column headers
* **Formatted numbers**
* Sortable results
### Visualizations
**Charts and graphs** embedded directly in responses:
* **Line charts** for trends
* **Bar charts** for comparisons
* **Pie charts** for distributions
### Multi-Part Answers
**Complex questions** get comprehensive responses:
1. **Initial analysis**
2. **Data query results**
3. **Visualization**
4. **Summary and insights**
## Managing Chats
### Chat List
View **all your conversations**:
* **Recent chats** appear first
* See chat **titles and timestamps**
* Preview **last message**
### Automatic Chat Titles
Chat titles **auto-generate** from your first question. The agent creates a **concise, descriptive title** automatically.
### Deleting Chats
Remove conversations you no longer need:
1. Open the chat
2. Click **delete option**
3. Confirm deletion
Deleted chats **cannot be recovered**. All messages and history will be **permanently removed**.
## Tips
* **Start broad, then narrow** - Begin with overview questions, then dive into specifics
* **Use follow-up suggestions** - Click suggested questions to explore deeper
* **Reference previous results** - Agent remembers context, build on earlier answers
* **Monitor token usage** - Check costs per response to manage budget
## Next Steps
* [Optimize Agent Performance](/reference/agents/best-practices)
* [Set Up Model Accounts](/reference/development/model-accounts)
# Embedding
Source: https://docs.datazone.co/reference/agents/embedding
Embed your agents into any website, app, or third-party tool with a single line of code
# Embedding Agents
You already have the ability to **create agents on top of your lakehouse**. With **embedding**, you can take those same agents and drop them into **your own apps, websites, or any third-party tool** — with **just one line of code**.
The embedded agent connects to the **same data sources, tools, and model configuration** you set up in Datazone, so your users get the full conversational experience without ever leaving your product.
## Overview
Embedding works through a secure, token-based mechanism:
1. Your server generates a **signed JWT token** containing the agent ID and embedding configuration.
2. The token is used to build a **signed script URL**.
3. You drop a single `
```
Replace `{scriptUrl}` with the signed URL generated on your server.
The container `
` is only needed for the **full** widget. The **bot** and **drawer** widgets render their own floating UI and ignore the container.
## Embedding Parameters
The JWT token payload can include the following parameters:
| Parameter | Type | Description |
| ------------------ | ------ | -------------------------------------------------------------------------------- |
| `agent_id` | string | **Required.** The ID of your agent. |
| `email` | string | Email of the user interacting with the agent (for access control and analytics). |
| `user_id` | string | Optional, instead of email — a unique identifier for the user. |
| `embedding_config` | object | Configuration for how the widget renders. See below. |
### `embedding_config` Options
| Option | Type | Description |
| -------- | ------ | -------------------------------------------------- |
| `widget` | string | The widget type: `"full"`, `"bot"`, or `"drawer"`. |
## Example: Full Server-Side Implementation
Here's a more complete example using Node.js / Express:
```javascript theme={null}
const express = require("express");
const jwt = require("jsonwebtoken");
const app = express();
// Configuration
const DATAZONE_SITE_URL = "https://dev.datazone.co";
const DATAZONE_SECRET_KEY = process.env.DATAZONE_SECRET_KEY; // Store in environment variables
app.get("/agent", (req, res) => {
// Get user information from your auth system
const currentUser = req.user;
// Create the token
const payload = {
agent_id: "6a0e502103bdcc40271925f1",
email: currentUser.email,
embedding_config: {
widget: "drawer",
},
};
const token = jwt.sign(payload, DATAZONE_SECRET_KEY);
const scriptUrl = DATAZONE_SITE_URL + "/app/api/widget/agent?token=" + token;
// Render your page with the script URL
res.render("agent", { scriptUrl });
});
app.listen(3000, () => console.log("Server running on port 3000"));
```
## Security Considerations
* Keep your Datazone secret key secure and **never expose it in client-side code**.
* Generate tokens **server-side** and pass the complete script URL to your frontend.
* Use the `email` or `user_id` field to scope access and track usage per user.
* Set appropriate content security policies for embedding.
## Troubleshooting
If your embedded agent isn't loading correctly:
1. Check that your token is signed correctly with the right secret key.
2. Verify that the agent ID is correct.
3. For the **full** widget, ensure the `datazone-agent-embed-container` div exists on the page.
4. Look for CORS or CSP issues in your browser's developer tools.
## Next Steps
* [Agents Overview](/reference/agents/overview) - Learn how to create and configure agents
* [Chat Interface](/reference/agents/chat) - Explore the conversational experience
* [Contact Support](mailto:support@datazone.co) - Get help with embedding setup
# Overview
Source: https://docs.datazone.co/reference/agents/overview
Create custom AI agents to interact with your data through natural language
# Datazone Agents
Datazone Agents are **custom AI assistants** that can interact with your data through **natural language conversations**. Each agent is configured with specific **data sources**, **tools**, and **models** to help users explore, analyze, and extract insights from their data.
## What is an Agent?
An agent is an AI-powered assistant that:
* **Understands natural language** - Ask questions in any language
* **Accesses your data** - Queries datasets and views you've configured
* **Uses tools** - Executes SQL queries, runs Python code, creates charts, and more
* **Provides insights** - Analyzes data and delivers actionable answers
Unlike **static dashboards**, agents **adapt to your questions in real-time** and can handle **complex, multi-step analysis** automatically.
## Creating an Agent
1. Navigate to your **Project** page
2. Click the **Add** button (+ icon)
3. Select **Agent** from the dropdown
The agent creation flow will guide you through **4 steps**:
### Step 1: Basic Information
Configure the basic settings:
* **Name** - A descriptive name for your agent (e.g., "Sales Analyzer", "Customer Support Assistant")
* **Instructions** - System prompt that defines the agent's **behavior**, **personality**, and **expertise**
* **Response Tone** - How the agent communicates:
* **Casual** - Informal, conversational style
* **Friendly** - Warm and approachable
* **Technical** - Precise and technical (default)
* **Educational** - Explanatory and teaching-focused
* **Professional** - Formal and business-like
* **Response Length** - How detailed responses should be:
* **Brief** - Short, concise answers
* **Moderate** - Balanced detail (default)
* **Detailed** - Comprehensive, in-depth responses
* **Language** - The language your agent will use (supports multiple languages)
Write **clear instructions**. For example: "You are a sales analytics expert. Help users analyze revenue trends, customer behavior, and product performance. Always provide specific numbers and actionable insights."
### Step 2: Model Configuration
Select your **AI model provider** and configure parameters:
* **Model Account** - Choose from configured [Model Accounts](/reference/development/model-accounts) (OpenAI, Anthropic, AWS Bedrock)
* **Model** - Select the specific model (e.g., GPT-4, Claude 3.5 Sonnet, GPT-4o-mini)
* **Temperature** - Controls creativity and randomness (0 = **focused and consistent**, 1 = **creative and varied**)
* **Max Tokens** - Maximum response length (higher = longer responses, higher cost)
### Step 3: Data Sources
Choose which data sources the agent can access:
* **Datasets** - Direct access to raw datasets
* **Views** - Optimized, filtered data views
* **Vectors** - Semantic search on vectorized data for RAG (Retrieval Augmented Generation)
The agent will **only be able to access data** from these selected sources.
Use **[Views](/reference/integration/views)** instead of raw datasets for **better performance** and security. Views are **faster to query** and let you **control what data** the agent can access. Add **[Vectors](/reference/development/vectors)** to enable **semantic search** and provide contextual information to your agent.
### Step 4: Tools
Enable **optional tools** that extend what your agent can do:
* **Python Code Executor** - Execute Python code for calculations and analysis
* **Chart Generator** - Create visualizations (line, bar, pie charts)
* **Web Search** - Search the internet for external information
* **Actions** - Call custom functions you've deployed (e.g., send emails, trigger workflows)
**Query Executor** is enabled by default and allows the agent to run SQL queries on your data. Select **Actions** to give your agent access to specific custom functions from your project. After completing all steps, click **Create** to save your agent. You can modify any of these settings later from the agent's detail page.
## Using Your Agent
Once created, you can start chatting with your agent:
1. Click on your agent from the **agents list** in the **project page**.
2. Click to **Go to Agent** to open the chat interface.
3. Then click to **New Chat** to start a fresh conversation
4. Ask questions in **natural language** (in any language you configured)
5. The agent will **automatically use its tools** to analyze data and provide answers
## Agent Capabilities
### Natural Language Queries
Ask questions in **any language**:
* "What were our top 5 customers last month?"
* "Show me the revenue trend for Q4"
* "Which products have declining sales?"
* "Compare this quarter to last year"
### Semantic Search with Vectors (RAG)
When you add **[Vectors](/reference/development/vectors)** as data sources, agents can perform **semantic similarity search** to retrieve relevant context from your vectorized data. This enables:
* **Context-aware responses** - Agent finds relevant information based on meaning, not just keywords
* **Document retrieval** - Search through documents, knowledge bases, and unstructured data
* **Enhanced accuracy** - Provide agents with precise contextual information for better answers
The agent **automatically decides** when to use vector search based on the question and available data sources.
### Action Tools
Agents can **call custom actions** you've deployed:
* **Send notifications** when certain conditions are met
* **Trigger workflows** based on data analysis
* **Integrate with external services** (Slack, email, APIs)
* **Process or transform data** using custom logic
The agent **automatically decides** when to use actions based on the conversation context.
Learn more about [creating actions](/reference/development/actions).
### Multi-Step Analysis
Agents can perform **complex analysis automatically**:
1. **Execute SQL queries** to fetch data
2. **Run Python code** for calculations
3. **Generate visualizations** automatically
4. **Provide insights** and recommendations
### Context Awareness
Agents **remember conversation history** and can:
* Reference previous questions
* Build on earlier analysis
* Maintain context throughout the chat session
## Best Practices
1. **Clear Instructions** - Write **specific system instructions** that define the agent's expertise and behavior
2. **Right Data Sources** - Only grant access to **relevant datasets/views**
3. **Appropriate Model** - Choose models that **balance cost and capability** for your use case
4. **Use Views** - Configure views for **faster queries** and better data access control
5. **Test Thoroughly** - Ask various questions to ensure the agent **understands your data correctly**
## Next Steps
* [Chat with Your Agent](/reference/agents/chat)
* [Optimize Performance](/reference/agents/best-practices)
* [Configure Model Accounts](/reference/development/model-accounts)
# Kernels
Source: https://docs.datazone.co/reference/analysis/kernel
Kernels are the execution environments for your Datazone notebooks.
## Overview
A kernel is the computational engine that executes the code contained in a notebook. It acts as an isolated environment that:
* Processes the code you write
* Manages the memory and computations
* Returns the results back to your notebook
## Available Kernel Images
Datazone provides two main kernel images:
* **Python**: Standard Python environment with common data science libraries
* **Python Pyspark**: Python environment with Apache Spark support for distributed computing
## Compute Resources
You can select from different compute sizes based on your workload:
| Size | Resources | Best for |
| ----------- | ------------------- | --------------------------------------- |
| **X-Small** | 2 vCPU, 8 GB RAM | Light data processing, simple scripts |
| **Small** | 4 vCPU, 16 GB RAM | Standard data analysis, medium datasets |
| **Medium** | 8 vCPU, 32 GB RAM | Heavy computations, large datasets |
| **Large** | 16 vCPU, 64 GB RAM | Big data processing, complex analytics |
| **X-Large** | 32 vCPU, 128 GB RAM | Enterprise-level distributed computing |
## Environment Variables
Before initializing a kernel, you can set custom environment variables to:
* Configure access credentials
* Set runtime parameters
* Define application-specific settings
* Control library behaviors
# Notebooks
Source: https://docs.datazone.co/reference/analysis/notebook
Notebooks are the interactive documents that you can write and run your code in Datazone.
## Overview
A Jupyter notebook is an interactive computing environment that enables you to create and share documents containing:
* Live code that you can run
* Rich text explanations
* Visualizations and charts
* Mathematical equations
The notebook is divided into cells, which can contain either:
* **Code cells**: Where you write and execute code
* **Markdown cells**: Where you write formatted text, images and equations
When you run a code cell, the results appear directly below the cell, making it easy to:
* Experiment with data analysis
* Document your workflow
* Share your findings with others
* Create interactive reports
Datazone's integrated Jupyter notebook environment provides several key benefits:
* **Direct Data Access**: Instant connection to your data lakehouse
* **Security & Governance**: Built-in security controls and policies
* **Collaboration**: Easy notebook sharing and version control
* **Resource Management**: Fully managed compute resources
* **Pre-configured Environment**: Ready-to-use data science tools
# Toolkit
Source: https://docs.datazone.co/reference/analysis/toolkit
The Toolkit is a collection of tools and utilities that help you manage your notebooks in Datazone.
## Overview
The Toolkit provides utilities to help you work with data in Datazone notebooks. Currently, it includes the Dataset class for accessing datasets.
## Dataset
The Dataset class allows you to easily load datasets into your notebooks as Pandas or PySpark DataFrames.
```python theme={null}
from datazone import Dataset
dataset = Dataset(id="")
# or
dataset = Dataset(alias="")
```
You can also load a dataset by providing a specific branch name:
```python theme={null}
from datazone import Dataset
dataset = Dataset(id="", branch="")
```
Thanks to the Dataset class, you can now easily load datasets into your notebooks as Pandas or PySpark DataFrames.
```python theme={null}
from datazone import Dataset
pandas_df = dataset.get_pandas_df()
# or
pyspark_df = dataset.get_pyspark_df()
```
`get_pyspark_df()` is only available in the PySpark kernel.
## Variable
You can access the Variables from the kernel environment using the `Variable` class.
```python theme={null}
from datazone import Variable
variable = Variable(key="")
```
### Example
```python theme={null}
from datazone import Variable
api_key = Variable(key="OPENAI_API_KEY")
client = OpenAI(api_key=str(api_key))
...
```
Also you can check the [Variables](/reference/development/variables)
# Access Keys
Source: https://docs.datazone.co/reference/development/access-keys
Create and manage access keys for programmatic access to your project resources
## Overview
**Access Keys** provide **programmatic access** to your project's file containers and resources. Each access key consists of an **Access Key ID** and a **Secret Access Key** that work like AWS credentials for S3-compatible operations.
## Creating an Access Key
1. Open your **Project**
2. Click **Settings** in the project navigation
3. Navigate to **Access Keys** section
4. Click **Create Access Key**
5. **Copy both credentials immediately** - the secret key won't be shown again
**Save your Secret Access Key immediately!** For security reasons, it cannot be retrieved after creation. If lost, you'll need to create a new access key.
## Access Key Components
Each access key has two parts:
* **Access Key ID** - Public identifier (can be viewed anytime)
* **Secret Access Key** - Private credential (shown only once at creation)
Both are required for **S3-compatible authentication**.
## Usage
Use access keys to connect to **File Containers** via:
* **AWS CLI** - Standard AWS S3 commands
* **Python (boto3)** - S3 client operations
* **JavaScript (AWS SDK)** - Node.js applications
* **Java (AWS SDK)** - Java applications
See the [File Container Connection Guide](/reference/development/file-container#local-access) for detailed examples.
Access keys are **project-specific**. Each project has its own set of access keys that only work with that project's resources.
## Environment Variables
Store credentials securely using environment variables:
```bash theme={null}
export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
```
Most AWS SDKs automatically read these variables.
## Next Steps
* [Connect to File Container](/tutorial/file-container-connection)
* [Learn About File Containers](/reference/development/file-container)
* [Explore AWS CLI Operations](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html)
# Actions
Source: https://docs.datazone.co/reference/development/actions
Deploy serverless Python functions to automate workflows and extend agent capabilities
## Overview
**Actions** allow you to deploy **serverless Python functions** that can be **triggered on-demand** by endpoints or used as **tools by AI agents**. Think of actions as **lambda-like functions** that run in isolated environments.
```python send_mail_action.py theme={null}
from datazone.actions import action
@action
def send_email(to: str, subject: str, body: str):
"""
Send an email to a recipient.
"""
# Your email sending logic here
print(f"Sending email to {to}")
# Return structured result
return {
"status": "sent",
"recipient": to,
"timestamp": "2026-02-11T10:00:00Z"
}
```
```yaml config.yaml theme={null}
project_name: my-project
project_id: proj_abc123
actions:
- path: actions/send_mail_action.py
```
* **Project Structure**
## What Are Actions?
Actions are **custom Python functions** that run in **isolated environments**, accept **parameters**, return **structured results**, and can be **triggered via endpoints** or **used as agent tools**.
## Use Cases
Inside an action you can:
* **[Invoke an LLM](/reference/development/actions#agent)** — call a model through your model account with `Agent`, optionally with structured output
* **[Run SQL queries](/reference/development/actions#execute_query)** — read from your datasets with `execute_query`
* **[Read and write knowledge objects](/reference/development/actions#knowledgeobject)** — get, list, update and batch upsert instances with `KnowledgeObject`
* **[Use secure variables](/reference/development/actions#variable-api)** — access API keys and credentials with `Variable`, without hardcoding them
* **[Log execution details](/reference/development/actions#context-api)** — emit info, warning and error logs with `context`
* **[Connect to SAP](/reference/development/actions#cloudfeedclient)** — read tables and execute function modules with `CloudFeedClient`
* **Call any external API** — the runtime ships with `requests`, so any HTTP service is one call away
Which makes actions a good fit for sending notifications, processing data, generating reports, automating workflows, and extending AI agent capabilities with custom logic.
### Action with Logging
Use **`context`** to log information during execution:
```python theme={null}
from datazone.actions import action, context
@action
def process_data(dataset_id: str, operation: str = "transform"):
"""
Process dataset with specified operation.
Args:
dataset_id: ID of dataset to process
operation: Type of operation (default: transform)
"""
context.log_info(f"Starting {operation} on dataset {dataset_id}")
try:
# Your processing logic
result_count = 42
context.log_info(f"Processed {result_count} records")
return {
"status": "success",
"dataset_id": dataset_id,
"records_processed": result_count
}
except Exception as e:
context.log_error(f"Processing failed: {str(e)}")
raise
```
### Action with Optional Parameters
```python theme={null}
from datazone.actions import action, context
@action
def generate_report(report_type: str, format: str = "pdf", email: str = None):
"""
Generate and optionally email a report.
Args:
report_type: Type of report to generate
format: Output format (default: pdf)
email: Optional email address to send report
"""
context.log_info(f"Generating {report_type} report in {format} format")
report_url = f"https://example.com/reports/{report_type}.{format}"
if email:
context.log_info(f"Sending report to {email}")
# Send email logic
return {
"report_url": report_url,
"format": format,
"emailed": email is not None
}
```
## Actions SDK Reference
### Decorator
```python theme={null}
from datazone.actions import action
@action
def my_function():
"""Your function logic"""
pass
```
The **`@action`** decorator marks a function as executable by Datazone.
### Context API
The **context** object provides logging capabilities during action execution:
```python theme={null}
from datazone.actions import context
```
Log informational messages during action execution.
```python theme={null}
context.log_info("Processing started")
context.log_info(f"Processed {count} records")
```
Log warning messages for potential issues.
```python theme={null}
context.log_warning("Potential issue detected")
context.log_warning(f"Unusual data pattern in {field}")
```
Log error messages when operations fail.
```python theme={null}
context.log_error("Operation failed")
context.log_error(f"Failed to connect: {str(e)}")
```
All logs are **collected and returned** with the action response.
### Variable API
The **Variable** class allows actions to access **secure variables** stored in Datazone:
```python theme={null}
from datazone.actions import Variable
```
Access workspace variables securely within actions.
```python theme={null}
# Access a variable
api_key = Variable("API_KEY")
# Use as a string (Variable stringifies to its value)
headers = {"Authorization": f"Bearer {Variable('AUTH_TOKEN')}"}
```
Variables are **workspace-scoped** and can be marked as **secret** to encrypt sensitive data like API keys, passwords, and tokens.
Learn more in the [Variables documentation](/reference/development/variables).
#### Example: Using Variables
```python theme={null}
from datazone.actions import action, Variable
@action
def send_notification(message: str):
"""Send notification using API key from Variables."""
api_key = str(Variable("API_KEY"))
print(f"Sending notification with key: {api_key[:4]}...")
return {"status": "sent", "message": message}
```
**Security Best Practice**: Always store sensitive credentials as Variables rather than hardcoding them in your action code.
### execute\_query
The **`execute_query`** function runs SQL queries against your Datazone datasets and returns structured results:
```python theme={null}
from datazone.actions import execute_query
```
Execute a SQL query against Datazone datasets.
```python theme={null}
result = execute_query("SELECT * FROM my_dataset LIMIT 10")
print(result.data) # list of row dicts
print(result.status) # query status string
```
Returns a `QueryResult` object with:
* **`data`** — list of row dictionaries
* **`status`** — execution status string
#### Example: Query and Transform
```python theme={null}
from datazone.actions import action, context, execute_query
@action
def summarize_orders(min_amount: float = 100.0):
"""Return order totals above a threshold."""
context.log_info(f"Querying orders with amount >= {min_amount}")
result = execute_query(f"""
SELECT customer_id, SUM(amount) AS total
FROM orders
WHERE amount >= {min_amount}
GROUP BY customer_id
ORDER BY total DESC
LIMIT 20
""")
context.log_info(f"Found {len(result.data)} customers")
return {
"status": result.status,
"row_count": len(result.data),
"rows": result.data
}
```
### Agent
The **`Agent`** class invokes an LLM through one of your [Model Accounts](/reference/development/model-accounts), without any provider SDK or API key in your code — the call is proxied by Datazone and billed against your workspace.
```python theme={null}
from datazone.actions import Agent
```
Create an agent bound to a model account and model.
Parameters:
* **`model`** *(optional)* — model to use, e.g. `CLAUDE_45_SONNET`. Defaults to your organisation's default model.
* **`model_account`** *(optional)* — model account id. Defaults to your organisation's AI settings account.
* **`response_format`** *(optional)* — a Pydantic model or JSON schema dict for structured output.
```python theme={null}
agent = Agent(model="CLAUDE_45_SONNET")
result = agent.invoke("Summarize this in one sentence: ...")
result["content"]
```
Invoke the model and return a result dictionary.
Accepts either a plain string or a message list:
```python theme={null}
agent.invoke("Say hi")
agent.invoke({
"messages": [
{"role": "system", "content": "You are concise."},
{"role": "user", "content": "Say hi"},
]
})
```
Message roles are **`system`**, **`user`** and **`assistant`**. Returns a dict with:
* **`content`** — the raw text answer (empty when `response_format` is used)
* **`structured_response`** — the parsed object when `response_format` is used, otherwise `None`
* **`model`** — the model that produced the answer
Raises `RuntimeError` if the invocation fails.
#### Example: Summarize Text
```python theme={null}
from datazone.actions import action, Agent
@action
def summarize(text: str):
"""Summarize a piece of text in one sentence."""
agent = Agent(model="CLAUDE_45_SONNET")
result = agent.invoke(f"Summarize in one sentence:\n\n{text}")
return {"summary": result["content"]}
```
#### Example: Structured Output
Pass a **Pydantic model** as `response_format` to get a validated object back instead of raw text:
```python theme={null}
from typing import Literal
from pydantic import BaseModel, Field
from datazone.actions import action, context, Agent
class TicketTriage(BaseModel):
"""Triage decision for a support ticket."""
priority: Literal["low", "medium", "high"] = Field(description="Urgency of the ticket")
team: str = Field(description="Team that should own it, e.g. billing, platform, data")
summary: str = Field(description="One sentence summary of the problem")
@action
def triage_ticket(subject: str, body: str):
"""Classify an incoming support ticket."""
agent = Agent(model="CLAUDE_45_SONNET", response_format=TicketTriage)
result = agent.invoke({
"messages": [
{"role": "system", "content": "You triage incoming support tickets. Be concise."},
{"role": "user", "content": f"Subject: {subject}\n\n{body}"},
]
})
triage = result["structured_response"]
context.log_info(f"Triaged as {triage.priority} for {triage.team}")
return {"priority": triage.priority, "team": triage.team, "summary": triage.summary}
```
`response_format` also accepts a plain JSON schema dict — in that case `structured_response` is returned as a dict rather than a model instance.
Token usage is recorded per invocation and attributed to the action, so LLM costs show up alongside your other workspace usage.
### KnowledgeObject
The **`KnowledgeObject`** client reads and writes [Knowledge Object](/reference/knowledge-objects/overview) instances from within an action. Instances are addressed by their opaque `_key`, and operations run on the action's project (main branch by default).
```python theme={null}
from datazone.actions import KnowledgeObject
```
Access instances of a knowledge object by name.
```python theme={null}
employees = KnowledgeObject("Employee") # main branch
employees = KnowledgeObject("Employee", branch="feature-x")
```
Methods:
* **`get(key, add_relationships=False)`** — fetch one instance (a `dict`) by its `_key`.
* **`list(filters=None, page=1, page_size=50, fields=None)`** — iterate pages of instances (see below).
* **`update(key, payload)`** — partially update an instance; returns the updated instance.
* **`delete(key)`** — delete an instance by its `_key`.
* **`batch_upsert(payloads)`** — insert/update up to 1000 instances in one call; returns `{"created", "updated", "total"}`.
**Filtering and pagination.** `list(...)` returns an iterator of pages; each page exposes `.items` and `.total_count`. `filters` is a list of `{"column", "operator", "value"}` objects combined with **AND** (operators: `equal`, `not_equal`, `contains`, `not_contains`, `greater_than`, `less_than`).
```python theme={null}
from datazone.actions import action, context, KnowledgeObject
@action
def deactivate_inactive_employees():
"""Mark employees with no recent login as inactive."""
employees = KnowledgeObject("Employee")
updated = 0
for page in employees.list(
filters=[{"column": "is_active", "operator": "equal", "value": True}]
):
for employee in page.items:
context.log_info(f"Checking {employee['name']}")
employees.update(key=employee["_key"], payload={"is_active": False})
updated += 1
return {"updated": updated}
```
#### Example: Read and upsert
```python theme={null}
from datazone.actions import action, KnowledgeObject
@action
def sync_employee(employee_id: int, email: str):
"""Read one employee and upsert a batch."""
employees = KnowledgeObject("Employee")
# Read a single instance by its key
existing = employees.get(key="8A3F9C2E1B4D6F7A8A3F9C2E1B4D6F7A")
# Insert or update many at once
result = employees.batch_upsert([
{"id": employee_id, "name": "John", "email": email},
{"id": employee_id + 1, "name": "Jane", "email": "jane@acme.com"},
])
return {"existing": existing, "upsert": result}
```
See the [Actions SDK](/reference/knowledge-objects/actions-sdk) page under Knowledge Objects for the full method reference.
### FileContainerClient
**`FileContainerClient`** provides access to files stored in the Datazone file container (LakeFS/S3-compatible storage). It is **automatically initialized** as `file_client` in agent code execution environments — no import is required.
`file_client` is pre-initialized in agent code execution contexts. It is **not** available inside `@action` decorated functions.
Fetch raw bytes for a file from the file container.
```python theme={null}
# path is relative to the file container root
raw = file_client.read("reports/data.csv")
```
* **`path`** — path relative to the file container root
* Returns raw `bytes`
* Raises `ValueError` if the file exceeds 10 MB or cannot be fetched
#### Examples: Reading Different File Types
```python theme={null}
# CSV / plain text
text = file_client.read("data.csv").decode("utf-8")
print(text[:500])
# JSON
import json
config = json.loads(file_client.read("config.json"))
print(config["setting"])
# Excel
import io
import openpyxl
wb = openpyxl.load_workbook(io.BytesIO(file_client.read("report.xlsx")))
ws = wb.active
for row in ws.iter_rows(values_only=True):
print(row)
# Agent-uploaded file
raw = file_client.read(f"agent-uploads/{agent_id}/{upload_id}/{filename}")
```
### CloudFeedClient
**`CloudFeedClient`** is a simplified SAP connector for reading tables and executing function modules from within actions:
```python theme={null}
from datazone.actions.clients import CloudFeedClient
```
SAP CloudFeed client for table access and function execution.
Constructor parameters:
* **`base_url`** — base URL of the SAP system
* **`username`** — SAP username
* **`password`** — SAP password
* **`timeout`** *(optional)* — request timeout in seconds (default: 30)
* **`max_retries`** *(optional)* — max retries on failure (default: 10)
* **`backoff_factor`** *(optional)* — exponential backoff factor (default: 0.3)
| Method | Description |
| --------------------------------------------------------------- | ---------------------------------------------------- |
| `get_table_list(filter_name, filter_desc, rows)` | List available SAP tables, supports wildcard filters |
| `get_table_schema(table_name)` | Get field definitions for a table |
| `preview_table(table_name, rows)` | Preview table contents |
| `execute_function(obj_name, input_data, output_fields, commit)` | Execute a SAP function module |
#### Example: Query SAP Sales Orders
```python theme={null}
from datazone.actions import action, context, Variable
from datazone.actions.clients import CloudFeedClient
@action
def fetch_sap_orders(table_name: str = "VBAP", preview_rows: int = 10):
"""Fetch sales order data from SAP."""
client = CloudFeedClient(
base_url=str(Variable("SAP_BASE_URL")),
username=str(Variable("SAP_USERNAME")),
password=str(Variable("SAP_PASSWORD")),
)
context.log_info(f"Fetching schema for {table_name}")
schema = client.get_table_schema(table_name)
fields = [f["FIELDNAME"] for f in schema]
context.log_info(f"Previewing {preview_rows} rows")
preview = client.preview_table(table_name, rows=preview_rows)
return {
"table": table_name,
"fields": fields,
"preview": preview
}
```
#### Example: Execute SAP Function Module
```python theme={null}
from datazone.actions import action, context, Variable
from datazone.actions.clients import CloudFeedClient
import json
@action
def create_sap_order(material: str, quantity: int, plant: str):
"""Create a purchase order in SAP via function module."""
with CloudFeedClient(
base_url=str(Variable("SAP_BASE_URL")),
username=str(Variable("SAP_USERNAME")),
password=str(Variable("SAP_PASSWORD")),
) as client:
input_data = json.dumps({
"MATERIAL": material,
"QUANTITY": quantity,
"PLANT": plant
})
context.log_info(f"Creating order for {material} x{quantity} at {plant}")
result = client.execute_function(
obj_name="Z_CREATE_PO",
input_data=input_data,
commit=True
)
context.log_info("Order created successfully")
return {"result": result, "material": material, "quantity": quantity}
```
#### Example: Search and Inspect SAP Tables
```python theme={null}
from datazone.actions import action, context, Variable
from datazone.actions.clients import CloudFeedClient
@action
def search_sap_tables(name_filter: str = "VBAP*"):
"""List SAP tables matching a filter and return their schemas."""
client = CloudFeedClient(
base_url=str(Variable("SAP_BASE_URL")),
username=str(Variable("SAP_USERNAME")),
password=str(Variable("SAP_PASSWORD")),
)
tables = client.get_table_list(filter_name=name_filter, rows=20)
context.log_info(f"Found {len(tables)} tables matching '{name_filter}'")
result = []
for table in tables[:5]: # inspect first 5
name = table["TABNAME"]
schema = client.get_table_schema(name)
result.append({"table": name, "field_count": len(schema)})
return result
```
## Configuration
### Add to config.yaml
Register your actions in **`config.yaml`**:
```yaml theme={null}
project_name: my-project
project_id: proj_abc123
actions:
- path: actions/send_email.py
- path: actions/process_data.py
- path: actions/generate_report.py
- path: workflows/cleanup_data.py
```
### Repository Structure
Each file should contain **one `@action` decorated function**. Action function names must be **unique within your project**.
## Return Values
Actions should return **structured data** (dict, list, or primitives):
```python theme={null}
@action
def my_action():
# Good - structured dictionary
return {
"status": "success",
"data": [1, 2, 3],
"metadata": {"count": 3}
}
# Good - simple values
return 42
# Good - lists
return [{"id": 1}, {"id": 2}]
```
## Error Handling
Actions can **raise exceptions** - they'll be captured and returned:
```python theme={null}
@action
def validate_input(value: int):
"""Validate input value."""
if value < 0:
raise ValueError("Value must be positive")
return {"validated": True, "value": value}
```
## Using Actions
### 1. In Endpoints
Connect actions to **API endpoints** for webhook-style triggers:
```yaml theme={null}
endpoint:
name: send-notification
type: action
config:
action_id: "507f1f77bcf86cd799439011"
```
When the endpoint is called, the action **executes automatically**.
Learn more in the [Endpoints documentation](/reference/integration/endpoints#action-based-endpoints).
### 2. In Agents
Enable actions as **tools for AI agents**:
When creating an agent:
1. Select **"Action"** in the tools section
2. Choose which actions the agent can use
3. The agent will **automatically call actions** when needed
The AI agent decides **when and how** to use actions based on user questions.
Learn more in the [Agents documentation](/reference/agents/overview#action-tools).
## Examples
### Slack Notification
```python theme={null}
from datazone.actions import action, context
import requests
@action
def send_slack_message(channel: str, message: str, webhook_url: str):
"""
Send message to Slack channel.
Args:
channel: Slack channel name
message: Message to send
webhook_url: Slack webhook URL
"""
context.log_info(f"Sending message to {channel}")
payload = {
"channel": channel,
"text": message
}
response = requests.post(webhook_url, json=payload)
if response.status_code == 200:
context.log_info("Message sent successfully")
return {"status": "sent", "channel": channel}
else:
context.log_error(f"Failed to send: {response.text}")
raise Exception(f"Slack API error: {response.status_code}")
```
### Data Validation
```python theme={null}
from datazone.actions import action, context
@action
def validate_customer_data(customer_id: str, email: str, age: int):
"""
Validate customer data before processing.
Args:
customer_id: Customer identifier
email: Customer email address
age: Customer age
"""
errors = []
# Validate email
if "@" not in email:
errors.append("Invalid email format")
# Validate age
if age < 18 or age > 120:
errors.append("Age must be between 18 and 120")
if errors:
context.log_error(f"Validation failed: {', '.join(errors)}")
return {
"valid": False,
"errors": errors
}
context.log_info("Validation passed")
return {
"valid": True,
"customer_id": customer_id
}
```
### API Integration
```python theme={null}
from datazone.actions import action, context, Variable
import requests
@action
def fetch_weather(city: str):
"""
Fetch current weather for a city.
Args:
city: City name
"""
context.log_info(f"Fetching weather for {city}")
api_key = str(Variable("WEATHER_API_KEY"))
url = f"https://api.weather.com/data?city={city}&key={api_key}"
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
data = response.json()
context.log_info("Weather data retrieved")
return {
"city": city,
"temperature": data["temp"],
"conditions": data["conditions"],
"humidity": data["humidity"]
}
except requests.exceptions.RequestException as e:
context.log_error(f"API request failed: {str(e)}")
raise
```
## Next Steps
* [Create Action Endpoints](/reference/integration/endpoints#action-based-endpoints)
* [Use Actions in Agents](/reference/agents/overview#action-tools)
* [Manage Variables](/reference/development/variables) - Store secure credentials and configuration
* [Learn About Projects](/reference/development/project)
# API Keys
Source: https://docs.datazone.co/reference/development/api-key
API keys are used to authenticate requests to the Datazone API. You can create and manage your API keys from the Datazone dashboard.
API Keys is only available for the Professional and Enterprise plans. Check the [plans](https://www.datazone.co/pricing) for more information.
## Overview
1. Click **Settings** in the top right corner session context menu.
2. Click **API Keys** in the left sidebar.
3. Click **Create** to create a new API key.
4. Enter a name for the API key and click **Create**.
5. Copy the API key with `dz-xxxxxx` format and store it in a secure place.
## Usage
You can use the API key to authenticate requests to the Datazone API. You can pass the API key in the `x-api-key` header.
```bash theme={null}
curl -X GET "https://app.datazone.co/api/v1/dataset/list" -H "x-api-key: dz-xxxxxx"
```
# Command Line
Source: https://docs.datazone.co/reference/development/command-line
One of the interaction way with Datazone is using Command Line Interface. You can manage your projects, datasets, and models with Datazone CLI commands.
## Installation
You can install the **Datazone Python SDK** using pip:
```bash theme={null}
pip install datazone
```
Check that the installation was successful by running the following command:
```bash theme={null}
datazone --version
```
## Common Command Pattern
On the Datazone CLI, generally, the command pattern is as follows:
```bash theme={null}
datazone [options]
```
## Help
You can get help about the Datazone CLI commands by add `--help` flag to the command.
```bash theme={null}
datazone --help
datazone project --help
datazone dataset list --help
```
## Commands
### `profile`
```bash theme={null}
## Create a new profile. It will ask you some questions to create a new profile.
datazone profile create
## List all profiles
datazone profile list
## Delete a profile
datazone profile delete
## Set default profile
datazone profile set-default
```
### `source`
```bash theme={null}
## List all sources
datazone source list
## Create a new source. It will ask you some questions to create a new source.
datazone source create
## Delete a source
datazone source delete
```
### `project`
```bash theme={null}
## List all projects
datazone project list
## Create a new project
datazone project create
## Clone a project
datazone project clone
## Deploy a project
datazone project deploy [--commit-message ] [--file ]
## Pull the project
datazone project pull
## Show the summary of the project
datazone project summary
```
### `dataset`
```bash theme={null}
## List all datasets
datazone dataset list
## Show sample data of a dataset
datazone dataset show
## List all transactions of a dataset
datazone dataset transactions
```
### `pipeline`
```bash theme={null}
## Create a new pipeline
datazone pipeline create
```
### `extract`
```bash theme={null}
## Create a new extract
datazone extract create
## List all extract
datazone extract list
## Delete an extract
datazone extract delete
## Run an execution for an extract
datazone extract execute
```
### `execution`
```bash theme={null}
## List all executions
datazone execution list
## Show logs of an execution
datazone execution logs
## Run an execution
datazone execution run [--extract-id ] [--pipeline-id ]
```
### `auth`
```bash theme={null}
## Test the authentication
datazone auth test
```
### `version`
```bash theme={null}
## Show the version of the Datazone CLI
datazone version
```
### `info`
```bash theme={null}
## Show the information of the current session
datazone info
```
# Context
Source: https://docs.datazone.co/reference/development/context
Each pipeline in Datazone has a context object that provides access to resources and configuration settings.
```python theme={null}
@transform
def my_transform(context):
context.state.write(key="my_key", value="my_value")
```
## State
The context object provides a `state` attribute that allows you to read and write key-value pairs.
The state is stored in database and can be accessed in any transform function in the pipeline.
Also you can access the state in the next executions of the pipeline.
```python theme={null}
from datazone import transform
@transform
def my_transform(context):
context.state.write(key="my_key", value="my_value")
@transform(depends=[my_transform])
def my_transform(context):
value = context.state.read(key="my_key")
print(value)
```
### State Callbacks
You can write your states by various conditions like `success`, `failure`, `now`.
```python theme={null}
from datazone import transform
@transform
def my_transform(context):
context.state.write(key="my_key", value="my_value", on="success")
raise Exception("Error")
```
In this example, the state will not be written because the transform function raises an exception.
## Resources
### PySpark Session
You can access the PySpark session using the `pyspark` attribute of the `resources` object.
```python theme={null}
from datazone import transform
@transform
def my_transform(context):
spark = context.resources["pyspark"].spark
# Use the PySpark session
df = spark.createDataFrame({
"name": ["Alice", "Bob"],
"age": [25, 30]
})
return df
```
# File Container
Source: https://docs.datazone.co/reference/development/file-container
File Container is a storage solution in the Datazone platform that allows you to manage and store files. You can create, update, and delete file containers, and use them to store data for your pipelines and notebooks.
## Overview
File Containers allows you to manage and store files. Each project has its own isolated file container. You can interact with file containers like **AWS S3 buckets** or **Google Cloud Storage buckets**.
Also, Datazone provides **toolkits** to interact with file containers. You can use the `FileContainerClient` in notebooks and pipelines to interact with file containers.
1. Click **Projects** in the left sidebar.
2. Choose a project from the card list.
3. Click **File Containers** tab in the left sidebar.
## Client Usage
The `FileContainerClient` provides a convenient interface to interact with file containers using S3-compatible storage. It handles authentication and bucket management automatically.
The `FileContainerClient` is **only available in the execution environment** (pipelines and notebooks running on Datazone). For **local development and external applications**, see the [Local Access](#local-access) section below.
You can access the `FileContainerClient` in your **pipelines** and **notebooks** like this:
```python theme={null}
from datazone import FileContainerClient
```
#### `list_objects`
Lists the files and directories directly under a path in the file container.
```python theme={null}
from datazone import FileContainerClient
# List everything at the root of the file container
objects = FileContainerClient.list_objects()
# List the contents of a directory
documents = FileContainerClient.list_objects("documents/")
```
**Parameters:**
* `prefix` (str): Optional directory to list, such as `documents/`. Should end with `/`. Defaults to the file container root.
**Returns:**
* `list`: List of entries, each tagged with a `Type` field of either `file` or `directory`. Directory entries carry only a `Key`, file entries also carry their S3 metadata. Example:
```
[
{
'Key': 'documents/',
'Type': 'directory'
},
{
'Key': 'customer_list.csv',
'Type': 'file',
'LastModified': datetime.datetime(2025, 7, 24, 19, 44, 12, 475000, tzinfo=tzlocal()),
'ETag': '"bf36dc829c4229254b7df3c428d0a349"',
'Size': 18311622,
'StorageClass': 'STANDARD'
}
]
```
The listing is **one level deep**, like `ls`. It does not descend into subdirectories — pass a directory's `Key` back as the `prefix` to list what is inside it.
To walk the whole tree, recurse into every directory you get back:
```python theme={null}
from datazone import FileContainerClient
def walk_file_container(prefix: str = ""):
for entry in FileContainerClient.list_objects(prefix):
if entry["Type"] == "directory":
yield from walk_file_container(entry["Key"])
else:
yield entry
for file in walk_file_container():
print(file["Key"], file["Size"])
```
#### `get_object`
Retrieves an object from the file container by its key.
```python theme={null}
# Get a file's content
file_data = FileContainerClient.get_object("data/sample.csv")
# Convert bytes to string for text files
content = file_data.decode('utf-8')
```
**Parameters:**
* `key` (str): The key/path of the object to retrieve
**Returns:**
* `bytes`: The object's raw data
Objects are stored as bytes, so you may need to encode/decode text data appropriately
#### `put_object`
Stores data in the file container at the specified key.
```python theme={null}
# Store text data
text_data = "Hello, World!".encode('utf-8')
FileContainerClient.put_object("messages/hello.txt", text_data)
# Store binary data
with open("local_file.pdf", "rb") as f:
file_data = f.read()
FileContainerClient.put_object("documents/file.pdf", file_data)
```
**Parameters:**
* `key` (str): The key/path where the object will be stored
* `data` (bytes): The data to store
#### `delete_object`
Removes an object from the file container.
```python theme={null}
# Delete a specific file
FileContainerClient.delete_object("temp/old_file.txt")
```
**Parameters:**
* `key` (str): The key/path of the object to delete
## Examples
### Periodically Uploading Files in a Pipeline
You can use the FileContainerClient to periodically upload files to your file container. This can be useful for tasks like logging, data collection, or backups.
```python theme={null}
from datazone import transform, FileContainerClient
import requests
from datetime import datetime
import io
@transform
def fetch_and_store_llm_data():
# URL to fetch the data from
url = "https://docs.datazone.co/llms-full.txt"
# Get the current timestamp in ISO format
timestamp_as_iso = datetime.now().isoformat()
# Fetch the data from the URL
response = requests.get(url)
# Check if the request was successful
if response.status_code == 200:
# Get the content
content = response.text
# Store the content in the file system
FileContainerClient.put_object(f"daily_llm/{timestamp_as_iso}/llm.txt", io.BytesIO(content.encode('utf-8')))
return f"Successfully stored LLM data with timestamp {timestamp_as_iso}"
else:
raise Exception(f"Failed to fetch data: HTTP {response.status_code}")
```
### Read a Parquet File in a Notebook
```python theme={null}
import io
import pandas as pd
from datazone import FileContainerClient
# Read a Parquet file from the file container
data = FileContainerClient.get_object("datasets/sample.parquet")
# Convert bytes to a Pandas DataFrame
df = pd.read_parquet(io.BytesIO(data))
```
## Local Access
For **local development** and **external applications**, you can access File Containers using **S3-compatible tools** and SDKs. This requires **Access Keys** for authentication.
### Prerequisites
Before connecting locally, you need:
1. **Access Keys** - Create from your project settings ([Learn how](/reference/development/access-keys))
2. **Endpoint URL** - Your Datazone instance URL (e.g., `your-instance.datazone.co:3333`)
3. **Project Path** - Format: `{project-name}/main/file-container/`
### AWS CLI
#### Install AWS CLI
```bash theme={null}
# macOS
brew install awscli
# Linux
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
# Windows
msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi
```
#### Configure Credentials
Set your **access keys** as environment variables:
```bash theme={null}
export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
```
#### List Files
```bash theme={null}
aws s3 ls --endpoint-url https://your-instance.datazone.co:3333 \
your-project-bucket/main/file-container/
```
#### Upload a File
```bash theme={null}
aws s3 cp local-file.txt \
--endpoint-url https://your-instance.datazone.co:3333 \
s3://your-project-bucket/main/file-container/local-file.txt
```
#### Download a File
```bash theme={null}
aws s3 cp --endpoint-url https://your-instance.datazone.co:3333 \
s3://your-project-bucket/main/file-container/remote-file.txt \
local-file.txt
```
#### Sync Directory
```bash theme={null}
# Upload entire directory
aws s3 sync ./local-folder \
--endpoint-url https://your-instance.datazone.co:3333 \
s3://your-project-bucket/main/file-container/remote-folder/
# Download entire directory
aws s3 sync --endpoint-url https://your-instance.datazone.co:3333 \
s3://your-project-bucket/main/file-container/remote-folder/ \
./local-folder
```
### Python (boto3)
#### Install boto3
```bash theme={null}
pip install boto3
```
#### Configure S3 Client
```python theme={null}
import boto3
s3_client = boto3.client(
's3',
endpoint_url='https://your-instance.datazone.co:3333',
aws_access_key_id='your-access-key-id',
aws_secret_access_key='your-secret-access-key'
)
bucket_name = 'your-project-bucket'
prefix = 'main/file-container/'
```
#### List Files
```python theme={null}
response = s3_client.list_objects_v2(
Bucket=bucket_name,
Prefix=prefix
)
for obj in response.get('Contents', []):
print(obj['Key'])
```
#### Upload a File
```python theme={null}
s3_client.upload_file(
'local-file.txt',
bucket_name,
f'{prefix}local-file.txt'
)
print('File uploaded successfully')
```
#### Download a File
```python theme={null}
s3_client.download_file(
bucket_name,
f'{prefix}remote-file.txt',
'local-file.txt'
)
print('File downloaded successfully')
```
#### Upload with Metadata
```python theme={null}
s3_client.upload_file(
'data.csv',
bucket_name,
f'{prefix}data.csv',
ExtraArgs={
'Metadata': {
'source': 'analytics',
'date': '2026-01-20'
}
}
)
```
### JavaScript (AWS SDK)
#### Install AWS SDK
```bash theme={null}
npm install aws-sdk
```
#### Configure S3 Client
```javascript theme={null}
const AWS = require('aws-sdk');
const s3 = new AWS.S3({
endpoint: 'https://your-instance.datazone.co:3333',
accessKeyId: 'your-access-key-id',
secretAccessKey: 'your-secret-access-key',
s3ForcePathStyle: true,
signatureVersion: 'v4'
});
const bucketName = 'your-project-bucket';
const prefix = 'main/file-container/';
```
#### List Files
```javascript theme={null}
s3.listObjectsV2({
Bucket: bucketName,
Prefix: prefix
}, (err, data) => {
if (err) console.error(err);
else {
data.Contents.forEach(obj => {
console.log(obj.Key);
});
}
});
```
#### Upload a File
```javascript theme={null}
const fs = require('fs');
const fileContent = fs.readFileSync('local-file.txt');
s3.putObject({
Bucket: bucketName,
Key: `${prefix}local-file.txt`,
Body: fileContent
}, (err, data) => {
if (err) console.error(err);
else console.log('File uploaded successfully');
});
```
#### Download a File
```javascript theme={null}
s3.getObject({
Bucket: bucketName,
Key: `${prefix}remote-file.txt`
}, (err, data) => {
if (err) console.error(err);
else {
fs.writeFileSync('local-file.txt', data.Body);
console.log('File downloaded successfully');
}
});
```
### Java (AWS SDK)
#### Add Dependency
```xml theme={null}
com.amazonawsaws-java-sdk-s31.12.400
```
#### Configure S3 Client
```java theme={null}
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.*;
import java.io.File;
BasicAWSCredentials credentials = new BasicAWSCredentials(
"your-access-key-id",
"your-secret-access-key"
);
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(
new AwsClientBuilder.EndpointConfiguration(
"https://your-instance.datazone.co:3333",
"us-east-1"
)
)
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withPathStyleAccessEnabled(true)
.build();
String bucketName = "your-project-bucket";
String prefix = "main/file-container/";
```
#### List Files
```java theme={null}
ListObjectsV2Request listRequest = new ListObjectsV2Request()
.withBucketName(bucketName)
.withPrefix(prefix);
ListObjectsV2Result result = s3Client.listObjectsV2(listRequest);
for (S3ObjectSummary objectSummary : result.getObjectSummaries()) {
System.out.println(objectSummary.getKey());
}
```
#### Upload a File
```java theme={null}
File file = new File("local-file.txt");
s3Client.putObject(
bucketName,
prefix + "local-file.txt",
file
);
System.out.println("File uploaded successfully");
```
#### Download a File
```java theme={null}
S3Object s3Object = s3Client.getObject(
bucketName,
prefix + "remote-file.txt"
);
S3ObjectInputStream inputStream = s3Object.getObjectContent();
// Save to file or process the input stream
System.out.println("File downloaded successfully");
```
## Common Operations
### Check if File Exists
**Python:**
```python theme={null}
try:
s3_client.head_object(Bucket=bucket_name, Key=f'{prefix}file.txt')
print('File exists')
except:
print('File does not exist')
```
### Delete a File
**AWS CLI:**
```bash theme={null}
aws s3 rm --endpoint-url https://your-instance.datazone.co:3333 \
s3://your-project-bucket/main/file-container/file.txt
```
**Python:**
```python theme={null}
s3_client.delete_object(
Bucket=bucket_name,
Key=f'{prefix}file.txt'
)
```
### Get File Metadata
**Python:**
```python theme={null}
response = s3_client.head_object(
Bucket=bucket_name,
Key=f'{prefix}file.txt'
)
print(f"Size: {response['ContentLength']} bytes")
print(f"Last Modified: {response['LastModified']}")
```
## Best Practices
1. **Use Environment Variables** - Never hardcode credentials in code
2. **Handle Errors** - Always wrap operations in try-catch blocks
3. **Stream Large Files** - Use streaming uploads/downloads for large files
4. **Set Timeouts** - Configure appropriate timeouts for your use case
5. **Clean Up** - Delete temporary files after processing
6. **Monitor Usage** - Track file operations for cost management
## Next Steps
* [Create Access Keys](/reference/development/access-keys)
* [AWS S3 CLI Reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html)
* [boto3 Documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html)
# Model Accounts
Source: https://docs.datazone.co/reference/development/model-accounts
Configure and manage AI model provider credentials for building intelligent applications with Datazone
## Overview
Model Accounts allow you to securely store and manage API credentials for AI model providers like OpenAI, Anthropic (Claude), and AWS Bedrock. Once configured, these credentials can be used across your Datazone projects to build intelligent applications, data pipelines with AI capabilities, and automated workflows.
## Supported Providers
Datazone supports the following AI model providers:
| Provider | Description | Required Credentials |
| --------------- | ------------------------------------------------- | -------------------------------------------- |
| **OpenAI** | Access to GPT models (GPT-4, GPT-3.5, etc.) | API Key |
| **Anthropic** | Access to Claude models (Claude 3.5 Sonnet, etc.) | API Key |
| **AWS Bedrock** | Access to multiple models via AWS Bedrock service | AWS Access Key ID, Secret Access Key, Region |
## Creating a Model Account
1. Click **Settings** in the top right corner session context menu.
2. Click **Model Accounts** in the left sidebar.
3. Click **Create Model Account** to add a new provider.
4. Enter a descriptive name for your model account (e.g., "Production OpenAI", "Development Claude").
5. Select your AI provider from the dropdown.
6. Enter the required credentials based on the provider:
### OpenAI Configuration
* **API Key** (required): Your OpenAI API key starting with `sk-`
* **API Base** (optional): Custom API endpoint if using a proxy or Azure OpenAI
### Anthropic Configuration
* **API Key** (required): Your Anthropic API key starting with `sk-ant-`
* **API Base** (optional): Custom API endpoint if needed
### AWS Bedrock Configuration
* **AWS Access Key ID** (required): Your AWS access key
* **AWS Secret Access Key** (required): Your AWS secret access key
* **AWS Region** (required): The AWS region for Bedrock (e.g., `us-east-1`, `us-west-2`)
* **AWS Session Token** (optional): Temporary session token if using STS
7. Click **Create** to save your model account.
## Security
Datazone takes security seriously when handling your AI provider credentials:
* **Encrypted Storage**: All API keys and sensitive credentials are encrypted at rest using industry-standard encryption.
* **Access Control**: Only organization members with MANAGE permissions can view or modify model accounts.
* **Organization Isolation**: Model accounts are scoped to your organization and cannot be accessed by other organizations.
Never share your API keys or credentials publicly. Each model account is tied to your organization, so treat credentials with the same care as database passwords.
## Managing Model Accounts
### Viewing Model Accounts
All your configured model accounts are listed in the Model Accounts page. Each entry shows:
* Account name
* Provider type
* Creation date
* Last updated date
### Updating Credentials
To update a model account:
1. Click on the account you want to update.
2. Modify the account name or credentials as needed.
3. Click **Update** to save changes.
Updating credentials will affect all projects and pipelines currently using this model account. Ensure the new credentials are valid before saving.
### Deleting Model Accounts
To delete a model account:
1. Click on the account you want to remove.
2. Click the **Delete** button.
3. Confirm the deletion.
Deleting a model account is permanent. Any projects or pipelines using this account will fail until reconfigured with a different model account.
## Next Steps
* [Build Intelligent Apps](/reference/intelligent-apps/overview) powered by AI
* [Create Data Pipelines](/reference/development/pipeline) with AI transformations
* [Manage Variables](/reference/development/variables) to reference model accounts in projects
# Pipeline
Source: https://docs.datazone.co/reference/development/pipeline
Define data processing steps using transforms and dependencies to build workflows
```python theme={null}
from datazone import transform
@transform
def say_hello():
print("Hello, World!")
@transform(depends=[say_hello])
def say_goodbye():
print("Goodbye, World!")
```
* Each pipeline should have a **unique alias** and should be defined in the different files.
* A pipeline should have at least one transform function.
* You can define dependencies between pipelines using the `depends` or `input_mapping` parameter to create a directed acyclic graph (DAG).
### Complex Pipeline Example
```python theme={null}
from datazone import transform
@transform
def prepare():
print("Preparing data...")
@transform(depends=[prepare])
def build_project():
print("Building project...")
@transform(depends=[prepare])
def build_report():
print("Building report...")
@transform(depends=[build_project, build_report])
def notify_email():
print("Sending email notification...")
```
```mermaid theme={null}
graph LR;
prepare-->build_project;
prepare-->build_report;
build_project-->notify_email;
build_report-->notify_email;
```
### Data Flow Management
The `@transform` decorator enables you to define data transformation functions efficiently.
Each function should:
* Accept input data as arguments
* Process the data
* Return the transformed data
Data is handled as PySpark DataFrames both for input and output operations.
```python theme={null}
from datazone import transform
@transform(input_mapping={'data': Input(Dataset(alias='input_data')})
def clean_data(data):
return data.filter(data['column'] > 0)
@transform(input_mapping={'clean_data': Input(clean_data)}, materialized=True)
def aggregate_data(clean_data):
return clean_data.groupBy('column').agg({'column': 'sum'})
```
In above example,
1. `clean_data` function takes `input_data` as input. You can check the dataset alias in the Datazone UI or use the `datazone dataset list` command to list all datasets.
2. After cleaning the data, the `clean_data` function returns the cleaned data PySpark DataFrame as lazy evaluation.
3. The `aggregate_data` function takes the cleaned data as input and aggregates it and returns the aggregated data.
4. Since the `materialized` parameter is set to `True`, the `aggregate_data` function will be materialized and create a new dataset in Datazone.
Check the [Transform](/reference/development/transform) section for more information on how to define a transform decorator.
### Invoking LLM Models
Transforms can call an LLM through one of your [Model Accounts](/reference/development/model-accounts) with the **`Agent`** class — no provider SDK or API key in your pipeline code. The call is proxied by Datazone and its token usage is attributed to the pipeline.
```python theme={null}
import pandas as pd
from datazone import transform, Agent, logger
TEXT = """
Datazone connects data sources to a lakehouse and provides tools for transformation,
orchestration and exploration.
"""
@transform(materialized=True, engine="pandas")
def summarize_text():
agent = Agent(model="CLAUDE_45_SONNET")
summary = agent.invoke(f"Summarize this in one sentence:\n\n{TEXT}")["content"]
logger.info(f"Summary: {summary}")
return pd.DataFrame([{"text": TEXT.strip(), "summary": summary}])
```
Create an agent bound to a model account and model.
Parameters:
* **`model`** *(optional)* — model to use, e.g. `CLAUDE_45_SONNET`. Defaults to your organisation's default model.
* **`model_account`** *(optional)* — model account id. Defaults to your organisation's AI settings account.
* **`response_format`** *(optional)* — a Pydantic model or JSON schema dict for structured output.
Invoke the model and return a result dictionary.
Accepts either a plain string or a message list:
```python theme={null}
agent.invoke("Say hi")
agent.invoke({
"messages": [
{"role": "system", "content": "You are concise."},
{"role": "user", "content": "Say hi"},
]
})
```
Message roles are **`system`**, **`user`** and **`assistant`**. Returns a dict with:
* **`content`** — the raw text answer (empty when `response_format` is used)
* **`structured_response`** — the parsed object when `response_format` is used, otherwise `None`
* **`model`** — the model that produced the answer
Raises `RuntimeError` if the invocation fails.
#### Structured Output
Pass a **Pydantic model** as `response_format` to get a validated object back instead of raw text:
```python theme={null}
from typing import Literal
import pandas as pd
from pydantic import BaseModel, Field
from datazone import Dataset, Input, Output, transform, Agent
reviews = Dataset(alias="product_reviews")
class ProductReview(BaseModel):
"""Analysis of a product review."""
rating: int | None = Field(description="The rating of the product", ge=1, le=5)
sentiment: Literal["positive", "negative"] = Field(description="The sentiment of the review")
key_points: list[str] = Field(description="Key points of the review, 1-3 words each")
@transform(
input_mapping={"data": Input(reviews)},
output_mapping={"result": Output(materialized=True)},
engine="pandas",
)
def analyze_reviews(data):
agent = Agent(model="CLAUDE_45_SONNET", response_format=ProductReview)
rows = []
for comment in data["comment"]:
analysis = agent.invoke(f"Analyze this review: '{comment}'")["structured_response"]
rows.append({
"comment": comment,
"rating": analysis.rating,
"sentiment": analysis.sentiment,
"key_points": ", ".join(analysis.key_points),
})
return pd.DataFrame(rows)
```
Each `invoke` is one request, so a row-by-row loop over a large dataset is slow and costly. Filter the input first, or batch several records into a single prompt and ask for a list back.
### Transform Selection
When executing a pipeline, you can selectively run specific transforms using the `transform_selection` parameter in the "Run with Config" modal. This allows you to execute only the transforms you need, along with their dependencies if required.
#### Selection Patterns
| Pattern | Description |
| ------------------- | -------------------------------------------------------------- |
| `some_transform` | Select the transform only |
| `*some_transform` | Select transform and all ancestors (upstream dependencies) |
| `some_transform*` | Select transform and all descendants (downstream dependencies) |
| `*some_transform*` | Select transform with all ancestors and descendants |
| `+some_transform` | Select transform and its direct parents |
| `some_transform+` | Select transform and direct children |
| `some_transform++` | Select transform and 2 levels of children |
| `some_transform+++` | Select transform and 3 levels of children |
Use transform selection to optimize execution time by running only the necessary parts of your pipeline during development and testing.
#### Usage Examples
For the pipeline example above with `prepare`, `build_project`, `build_report`, and `notify_email`:
* `build_project` - Runs only the `build_project` transform
* `*build_project` - Runs `prepare` and `build_project` (transform with all ancestors)
* `prepare*` - Runs `prepare`, `build_project`, and `build_report` (transform with all descendants)
* `*notify_email*` - Runs the entire pipeline (transform with all ancestors and descendants)
# Policy
Source: https://docs.datazone.co/reference/development/policy
Role-based access control with hierarchical permissions for fine-grained authorization
Datazone's policy system provides flexible, fine-grained access control through role-based permissions. Policies define what actions users can perform on resources, supporting both flat and hierarchical resource patterns with explicit allow/deny rules.
### Key Features
* **Role-based**: Policies are bound to roles, not individual users
* **Hierarchical**: Support for project-scoped resources (e.g., `project::dataset:*`)
* **Explicit deny**: Deny statements override allow statements
* **Extra constraints**: Resource-specific restrictions (row-level security, column filtering, path prefixes)
* **Wildcard support**: Use `*` for flexible matching across resources and actions
* **Branch-aware**: Optional branch specification for version control
## Policy Structure
A policy is a **list of statements** that define access rules:
```json theme={null}
[
{
"resource": "",
"branch": "",
"actions": [""],
"effect": "allow" | "deny",
"extra_constraints": {
"": ""
}
}
]
```
### Fields
| Field | Type | Description |
| ------------------- | -------------- | --------------------------------------------------------------- |
| `resource` | string | Resource pattern (supports wildcards and hierarchical patterns) |
| `branch` | string \| null | Optional branch name (defaults to `main`) |
| `actions` | array | List of actions in `:` format |
| `effect` | enum | Either `"allow"` or `"deny"` |
| `extra_constraints` | object | Resource-specific constraints (optional) |
## Resource Patterns
Resources follow a hierarchical pattern that supports various levels of specificity:
### Flat Resources
| Pattern | Scope | Description |
| -------------- | ------------- | -------------------------------------- |
| `*` | All | All resources in the organization |
| `dataset` | Type | Dataset type (for creation permission) |
| `dataset:*` | All instances | All dataset instances |
| `dataset:` | Specific | A specific dataset by ID |
### Hierarchical Resources
Hierarchical patterns enable project-scoped permissions:
| Pattern | Scope | Description |
| --------------------------- | -------------- | ----------------------------------- |
| `project::*` | All children | All entities within the project |
| `project::dataset:*` | Typed children | All datasets within the project |
| `project::dataset:` | Specific child | Specific dataset within the project |
### Supported Resource Types
* `dataset` - Data tables in the lakehouse
* `project` - Project containers
* `view` - Virtual views over datasets
* `schedule` - Automated execution schedules
* `extract` - Data ingestion jobs
* `compute` - Compute resources
* `api_key` - API authentication keys
* `user` - User accounts
* `role` - User roles
* `notebook` - Interactive analysis notebooks
* `pipeline` - Data transformation pipelines
* `endpoint` - REST API endpoints
* `intelligent_app` - Dashboard applications
* `variable` - Environment variables
* `agent` - AI agents
* `action` - Executable actions
* `organisation` - Organisation-level administration
* `orion` - Orion logs
* `source` - Data source connections
* `notification` - Notification channels and settings
* `tag` - Resource tags
## Actions
Actions follow the `:` format and define what operations can be performed:
### Action Patterns
* `dataset:read` - Read access to datasets
* `dataset:write` - Modify datasets
* `dataset:delete` - Delete datasets
* `dataset:execute` - Execute operations on datasets
* `dataset:manage` - Full management access
* `dataset:create` - Create new datasets
* `dataset:*` - All dataset actions
* `*:read` - Read access to all resources
* `*:*` - All actions on all resources
### Common Actions
| Action | Description |
| -------------------- | --------------------------------- |
| `:read` | View the resource |
| `:write` | Modify the resource |
| `:delete` | Delete the resource |
| `:create` | Create new instances |
| `:execute` | Execute operations |
| `:manage` | Full control (includes all above) |
| `:*` | All actions for that resource |
### Exceptional Actions
Some resources support custom actions beyond the standard CRUD operations:
* `project:read_repository` - Read access to project code repository
* `project:write_repository` - Write access to project code repository (push commits, merge pull requests)
* `project:pr_create` - Permission to open a pull request targeting a branch
* `project:deploy` - Permission to deploy a project
* `endpoint:invoke` - Permission to call an API endpoint
* `agent:ask` - Permission to ask (query) an AI agent
## Manage Permissions and Settings Access
Some parts of the application — mainly the **Settings** area — are gated by the `manage` action rather than by `read`. A user who holds `:manage` sees that resource's menu in Settings and can view its entries.
`manage` grants **visibility only**. It does not imply creating, editing, or deleting — grant `:create`, `:write`, or `:delete` alongside it for those operations.
Previously these areas were visible only to users with `organisation:manage`. Now each resource can be delegated independently, so you can grant access to a single Settings menu without handing over organisation-wide administration.
### Resources that support `manage`
| Resource | Action | Grants access to |
| -------------- | --------------------- | ---------------------------------------------------------------------------------- |
| `organisation` | `organisation:manage` | Organisation-wide administration (users, roles, all Settings menus) |
| `orion` | `orion:manage` | Orion logs |
| `source` | `source:manage` | Source menu in Settings |
| `api_key` | `api_key:manage` | API Keys menu in Settings (keys are user-owned, so this also covers creating them) |
| `notification` | `notification:manage` | Notification settings and channels |
| `compute` | `compute:manage` | Compute resource configuration |
| `variable` | `variable:manage` | Variable management |
| `tag` | `tag:manage` | Tag management |
### Example: access to API keys and sources
This policy reveals the **API Keys** and **Source** menus in Settings without granting any other organisation administration:
```json theme={null}
[
{
"resource": "api_key",
"actions": ["api_key:manage"],
"effect": "allow"
},
{
"resource": "source",
"actions": ["source:manage"],
"effect": "allow"
}
]
```
The user can open both menus and see their contents. All other Settings menus stay hidden. Because API keys belong to the user who owns them, `api_key:manage` is enough to create and revoke their own keys. Sources are shared organisation resources, so this policy does **not** let them create or delete a source.
### Example: adding source create permission
To let the same role also create sources, add `source:create` on the resource type:
```json theme={null}
[
{
"resource": "api_key",
"actions": ["api_key:manage"],
"effect": "allow"
},
{
"resource": "source",
"actions": ["source:manage", "source:create"],
"effect": "allow"
}
]
```
Deleting or editing existing sources requires `source:delete` and `source:write` on the instances (for example `source:*`).
`organisation:manage` still implies access to every Settings menu. Use the per-resource `manage` actions when you want to delegate one area only.
## Row-Level and Column-Level Security
Datazone supports fine-grained data access control through row-level and column-level restrictions using the `extra_constraints` field. This enables you to restrict what data users can see within a dataset or view, beyond just granting or denying access to the entire resource.
### Extra Constraints Structure
For datasets and views, you can specify columnar constraints:
```json theme={null}
{
"resource": "dataset:",
"actions": ["dataset:read"],
"effect": "allow",
"extra_constraints": {
"row_level_restrictions": [""],
"column_level_restrictions": [""]
}
}
```
### Fields
| Field | Type | Description |
| --------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `row_level_restrictions` | array of strings | SQL WHERE conditions to filter rows - only matching rows are accessible (e.g., `["region = 'US'", "status = 'active'"]`) |
| `column_level_restrictions` | array of strings | Column names to allow - only these columns are accessible (e.g., `["id", "name", "email"]`) |
### Row-Level Restrictions
Row-level restrictions apply SQL conditions to filter which rows a user can access:
```json theme={null}
{
"resource": "dataset:507f1f77bcf86cd799439011",
"actions": ["dataset:read"],
"effect": "allow",
"extra_constraints": {
"row_level_restrictions": ["country = 'USA'", "department = 'Sales'"]
}
}
```
When a user queries this dataset, these conditions are automatically appended to the WHERE clause:
```sql theme={null}
SELECT * FROM dataset WHERE (country = 'USA') AND (department = 'Sales')
```
### Column-Level Restrictions
Column-level restrictions specify which columns users can access (allowlist):
```json theme={null}
{
"resource": "view:507f1f77bcf86cd799439012",
"actions": ["view:read"],
"effect": "allow",
"extra_constraints": {
"column_level_restrictions": ["id", "name", "email", "department"]
}
}
```
Only the specified columns will be accessible in query results. All other columns will be filtered out automatically.
### Combined Restrictions
You can use both row and column restrictions together:
```json theme={null}
{
"resource": "dataset:507f1f77bcf86cd799439011",
"actions": ["dataset:read"],
"effect": "allow",
"extra_constraints": {
"row_level_restrictions": ["region = 'EMEA'"],
"column_level_restrictions": ["id", "name", "region", "department"]
}
}
```
**Important Restrictions for Row/Column-Level Security:**
When using `extra_constraints` with `row_level_restrictions` or `column_level_restrictions`:
1. **Must target a specific resource**: Use `dataset:` or `view:`, not wildcards like `dataset:*`
2. **Read-only actions**: Only `dataset:read` or `view:read` actions are allowed
3. **Single action**: Statement must contain exactly one action
4. **Single resource**: Statement must target exactly one dataset or view
Invalid examples:
* ❌ `"resource": "dataset:*"` (wildcard not allowed)
* ❌ `"actions": ["dataset:read", "dataset:write"]` (multiple actions)
* ❌ `"actions": ["dataset:*"]` (wildcard action not allowed)
* ❌ `"actions": ["dataset:write"]` (write action not allowed)
Valid example:
* ✅ `"resource": "dataset:507f1f77bcf86cd799439011"` with `"actions": ["dataset:read"]`
## Policy Examples
### Read-Only Access
Grant read access to all resources:
```json theme={null}
[
{
"resource": "*",
"actions": ["*:read"],
"effect": "allow"
}
]
```
### Dataset Admin
Full control over all datasets:
```json theme={null}
[
{
"resource": "dataset",
"actions": ["dataset:create"],
"effect": "allow"
},
{
"resource": "dataset:*",
"actions": ["dataset:*"],
"effect": "allow"
}
]
```
### Project Admin
Full control over a specific project and all its resources:
```json theme={null}
[
{
"resource": "project:66be5fc75158d037e9970c6d",
"actions": ["project:*"],
"effect": "allow"
},
{
"resource": "project:66be5fc75158d037e9970c6d:*",
"actions": ["*:*"],
"effect": "allow"
}
]
```
This allows the user to:
* Manage the project itself (`project:*`)
* Create and manage all child resources (datasets, notebooks, pipelines, etc.)
### Restricted Access with Deny
Allow read access to all datasets except one specific dataset:
```json theme={null}
[
{
"resource": "dataset:*",
"actions": ["dataset:read"],
"effect": "allow"
},
{
"resource": "dataset:507f1f77bcf86cd799439011",
"actions": ["dataset:read"],
"effect": "deny"
}
]
```
Deny statements always override allow statements, regardless of order.
### Project-Scoped Dataset Access
Grant access to datasets within a specific project only:
```json theme={null}
[
{
"resource": "project:66be5fc75158d037e9970c6d:dataset:*",
"actions": ["dataset:read", "dataset:write"],
"effect": "allow"
}
]
```
### Branch Protection
Repository permissions can be scoped to a branch with the `branch` field. Combined with the separation between `project:write_repository` (pushing commits and merging pull requests) and `project:pr_create` (opening a pull request), this lets you protect a branch such as `main` while keeping the rest of the project fully editable.
```json theme={null}
[
{
"effect": "allow",
"actions": ["project:*"],
"resource": "project:6a2ac2b4b45557181b0ecf04"
},
{
"effect": "allow",
"actions": ["*:*"],
"resource": "project:6a2ac2b4b45557181b0ecf04:*"
},
{
"effect": "deny",
"actions": ["project:write_repository"],
"resource": "project:6a2ac2b4b45557181b0ecf04",
"branch": "main"
}
]
```
How this evaluates:
| Operation | Result | Why |
| ------------------------------------ | ------- | ---------------------------------------------------------- |
| Push a commit to a feature branch | Allowed | The deny only applies to `main` |
| Push a commit directly to `main` | Denied | `project:write_repository` is denied on `main` |
| Create a branch from `main` | Allowed | Branch creation writes a new ref, not `main` itself |
| Open a pull request targeting `main` | Allowed | `project:pr_create` on `main` still comes from `project:*` |
| Merge a pull request into `main` | Denied | Merging requires `project:write_repository` on `main` |
The result is a familiar protected-branch workflow: contributors work on feature branches and open pull requests into `main`, but nobody with this policy can push or merge into `main` directly. Grant merge rights to a separate reviewer role that is not denied `project:write_repository` on `main`.
`project:pr_create` is checked against the **base** branch of the pull request — the branch being merged into. To block pull requests into `main` entirely, add `project:pr_create` to the deny statement's actions.
### Data Analyst Role
Typical permissions for a data analyst:
```json theme={null}
[
{
"resource": "dataset:*",
"actions": ["dataset:read"],
"effect": "allow"
},
{
"resource": "notebook",
"actions": ["notebook:create"],
"effect": "allow"
},
{
"resource": "notebook:*",
"actions": ["notebook:*"],
"effect": "allow"
},
{
"resource": "view:*",
"actions": ["view:read"],
"effect": "allow"
}
]
```
## Built-in Policies
Datazone provides several built-in policies for common use cases:
### Admin Policy
Full access to all resources:
```json theme={null}
[
{
"resource": "*",
"actions": ["*:*"],
"effect": "allow"
}
]
```
## Best Practices
### Policy Design
1. **Start restrictive**: Begin with minimal permissions and add as needed
2. **Use hierarchical patterns**: Organize permissions by project for better management
3. **Leverage deny sparingly**: Use deny for exceptions to broad allow rules
4. **Document policies**: Add clear descriptions to explain policy intent
### Role Assignment
1. **Bind to roles only**: Policies are assigned to roles, not individual users
2. **Create role hierarchies**: Use multiple roles for different permission levels (Viewer, Editor, Admin)
3. **Audit regularly**: Review policy assignments periodically
### Performance
1. **Cache aware**: Policies are cached; changes may take a few seconds to propagate
2. **Granular resources**: Use specific resource IDs when possible to reduce evaluation complexity
3. **Minimize deny statements**: They require checking all policies
### Security
1. **Principle of least privilege**: Grant only necessary permissions
2. **Explicit denies**: Use deny statements to override broad allows for sensitive resources
3. **Extra constraints**: Apply row-level and column-level security for sensitive data
4. **Path restrictions**: Use `path_prefix` constraints to sandbox project access
## Validation Rules
Policies are validated automatically to ensure correctness:
### Action Format
* Must follow `:` pattern
* Both parts must be lowercase with underscores
* Wildcards allowed: `*:*`, `dataset:*`, `*:read`
**Valid:**
* `dataset:read`
* `project:*`
* `*:*`
**Invalid:**
* `dataset` (missing action)
* `Dataset:Read` (uppercase)
* `read` (missing resource)
### Resource Pattern
* Must be valid resource type or wildcard
* ObjectIds must be valid MongoDB ObjectIds
* Hierarchical patterns must follow `parent::child` format
**Valid:**
* `dataset:*`
* `project:507f1f77bcf86cd799439011`
* `project:507f1f77bcf86cd799439011:dataset:*`
**Invalid:**
* `dataset:invalid-id`
* `project::dataset:*`
### Action-Resource Matching
Actions must match the resource type they're applied to:
```json theme={null}
{
"resource": "dataset:*",
"actions": ["dataset:read"], // ✓ Valid
"effect": "allow"
}
```
```json theme={null}
{
"resource": "dataset:*",
"actions": ["project:read"], // ✗ Invalid - mismatched types
"effect": "allow"
}
```
Exception: Wildcard actions (`*:*`) can be used on any resource.
## Troubleshooting
### Permission Denied Errors
If you encounter permission denied errors:
1. **Check user roles**: Verify the user has the appropriate role assigned
2. **Review policy statements**: Ensure the policy includes the required action and resource
3. **Look for deny statements**: Check if an explicit deny is overriding an allow
4. **Verify resource IDs**: Ensure you're using the correct resource identifier
5. **Check cache**: Wait a few seconds for policy changes to propagate
### Hierarchical Permissions Not Working
If project-scoped permissions aren't working:
1. **Verify pattern format**: Use `project::*` not `project:*:`
2. **Check parent context**: Ensure the resource creation includes project reference
3. **Review cache**: Hierarchical relationships are cached; wait 5 minutes or invalidate cache
4. **Validate ObjectIds**: All IDs must be valid MongoDB ObjectIds
### Performance Issues
If policy evaluation is slow:
1. **Reduce policy complexity**: Simplify nested hierarchies
2. **Use specific resources**: Prefer `dataset:` over broad wildcards when possible
3. **Monitor cache health**: Ensure Redis is functioning properly
4. **Check database queries**: Hierarchical policies should use cache, not database
## Related Resources
Learn about role management and user assignment
Understand authentication and token management
Generate and manage API keys for programmatic access
Organize resources within projects
# Project Repository
Source: https://docs.datazone.co/reference/development/project
Organize and deploy your pipelines, actions, apps, and endpoints in a single project structure.
## Overview
A Datazone project is a **Git repository** containing your data pipelines, actions, apps, knowledge objects, flows, and endpoints. All project components are defined in a central **`config.yml`** file.
## Project Structure
* Each project **must have** a `config.yml` file
* Each pipeline should have a **unique alias** and be defined in separate files
* Aliases must be unique within `pipelines` and within `studio_apps`
* Unknown keys are **rejected** on deploy rather than ignored, so a typo fails loudly
* You can organize your code with utility files for shared logic
## Configuration File
The `config.yml` file defines all project resources:
```yaml config.yml theme={null}
project_name: my-pretty-project
project_id: 67280ba2f4a0960d02159675
pipelines:
- alias: hello_world
path: pipelines/hello_world.py
compute: LARGE
spark_config:
deploy_mode: client
executor_instances: 3
python_dependencies:
- name: pandas
version: 1.3.3
actions:
- path: actions/send_notification.py
apps:
- path: apps/my_app.py
objects:
- path: objects/order.yml
flows:
- path: flows/daily_report.yml
studio_apps:
- alias: orders
name: Orders
path: studio/orders
endpoints:
- path: endpoints/api_config.yml
```
## Configuration Reference
### Project Fields
Name of your project. Used for display and identification.
```yaml theme={null}
project_name: data-processing-platform
```
Unique identifier for your project. Generated when you create a project.
```yaml theme={null}
project_id: 67280ba2f4a0960d02159675
```
List of data pipeline definitions. Each pipeline processes and transforms data.
```yaml theme={null}
pipelines:
- alias: etl_pipeline
path: pipelines/etl.py
```
List of serverless action functions. Actions can be triggered by endpoints or used by AI agents.
```yaml theme={null}
actions:
- path: actions/send_email.py
```
Learn more in the [Actions documentation](/reference/development/actions).
List of intelligent AI applications.
```yaml theme={null}
apps:
- path: apps/customer_support.py
```
List of Knowledge Object definitions — versioned business entities with a CRUD API.
```yaml theme={null}
objects:
- path: objects/order.yml
```
Removing an entry deletes the object and schedules cleanup of its data. Learn more in the [Knowledge Objects documentation](/reference/knowledge-objects/overview).
List of declarative YAML orchestration flows.
```yaml theme={null}
flows:
- path: flows/daily_report.yml
```
Learn more in the [Flows documentation](/reference/flows/overview).
List of Studio Apps — React applications built and served by Datazone.
```yaml theme={null}
studio_apps:
- alias: orders
name: Orders
path: studio/orders
```
Learn more in the [Studio Apps documentation](/reference/studio-apps/overview).
List of API endpoint configurations.
```yaml theme={null}
endpoints:
- path: endpoints/webhooks.yml
```
Learn more in the [Endpoints documentation](/reference/integration/endpoints).
### Pipeline Configuration
Short, unique identifier for the pipeline. Used in CLI commands and UI.
```yaml theme={null}
alias: daily_etl
```
Relative path to the pipeline Python file from project root.
```yaml theme={null}
path: pipelines/etl_pipeline.py
```
Compute instance size for pipeline execution.
**Available sizes:**
* `XSMALL` - 2 vCPU, 8 GB RAM
* `SMALL` - 4 vCPU, 16 GB RAM
* `MEDIUM` - 8 vCPU, 32 GB RAM
* `LARGE` - 16 vCPU, 64 GB RAM
* `XLARGE` - 32 vCPU, 128 GB RAM (Enterprise only)
```yaml theme={null}
compute: LARGE
```
Spark deployment mode for distributed processing.
**Options:**
* `local` - Runs on a single machine (default)
* `client` - Driver runs in the same process, executors run separately
* `cluster` - Both driver and executors run in separate processes (Enterprise only)
```yaml theme={null}
spark_config:
deploy_mode: client
```
Number of Spark executors for parallel processing. Only applies when `deploy_mode` is `client` or `cluster`.
```yaml theme={null}
spark_config:
deploy_mode: client
executor_instances: 5
```
Additional Spark configuration properties. Pass any custom Spark configuration key-value pairs.
```yaml theme={null}
spark_config:
deploy_mode: client
executor_instances: 3
extra_spark_config:
spark.sql.shuffle.partitions: "200"
spark.default.parallelism: "100"
spark.sql.adaptive.enabled: "true"
```
Python packages required by the pipeline. Installed before execution.
```yaml theme={null}
python_dependencies:
- name: pandas
version: 2.0.0
- name: requests
version: 2.31.0
index_url: https://pypi.org/simple
- name: my-package
extra_arguments:
- "--no-dependencies"
```
### Python Dependency Fields
Python package name from PyPI or custom index.
```yaml theme={null}
- name: numpy
```
Specific package version. If omitted, installs the latest version.
```yaml theme={null}
- name: pandas
version: 2.0.0
```
Custom Python package index URL. Useful for private packages or mirrors.
```yaml theme={null}
- name: my-private-package
version: 1.2.3
index_url: https://pypi.mycompany.com/simple
```
Additional arguments passed directly to the package installer for this dependency.
```yaml theme={null}
- name: my-package
version: 1.2.3
extra_arguments:
- "--no-dependencies"
```
### Action Configuration
Relative path to the action Python file containing an `@action` decorated function.
```yaml theme={null}
actions:
- path: actions/send_notification.py
- path: workflows/data_validator.py
```
Each file should contain **one action function**. Learn more in the [Actions documentation](/reference/development/actions).
### App Configuration
Relative path to the intelligent app Python file.
```yaml theme={null}
apps:
- path: apps/customer_assistant.py
```
### Studio App Configuration
Short, unique identifier for the app. Used in its served URL.
```yaml theme={null}
alias: orders
```
Display name shown in Datazone. Defaults to the alias.
```yaml theme={null}
name: Orders
```
Relative path to the app directory — the folder holding `package.json`.
```yaml theme={null}
studio_apps:
- alias: orders
name: Orders
path: studio/orders
```
The entry is created for you when you add a Studio App, and removing it deletes the app and its directory. Each app is built per branch; see the [Studio Apps documentation](/reference/studio-apps/overview).
### Knowledge Object Configuration
Relative path to the object definition YAML file. One file may hold several objects, separated by `---`.
```yaml theme={null}
objects:
- path: objects/order.yml
- path: objects/customer.yml
```
Learn more in the [Knowledge Objects documentation](/reference/knowledge-objects/overview).
### Flow Configuration
Relative path to the flow definition YAML file.
```yaml theme={null}
flows:
- path: flows/daily_report.yml
```
Learn more in the [Flows documentation](/reference/flows/overview).
### Endpoint Configuration
Relative path to the endpoint configuration YAML file.
```yaml theme={null}
endpoints:
- path: endpoints/webhooks.yml
- path: endpoints/api_routes.yml
```
Learn more in the [Endpoints documentation](/reference/integration/endpoints).
## Example Projects
### Data Processing Pipeline
```yaml theme={null}
project_name: sales-analytics
project_id: 67280ba2f4a0960d02159675
pipelines:
- alias: daily_sales_etl
path: pipelines/sales_pipeline.py
compute: MEDIUM
spark_config:
deploy_mode: client
executor_instances: 3
python_dependencies:
- name: pandas
version: 2.0.0
- name: sqlalchemy
version: 2.0.0
```
### Multi-Component Project
```yaml theme={null}
project_name: customer-platform
project_id: 67280ba2f4a0960d02159675
pipelines:
- alias: customer_data_sync
path: pipelines/sync.py
compute: SMALL
python_dependencies:
- name: requests
- name: pydantic
version: 2.0.0
actions:
- path: actions/send_welcome_email.py
- path: actions/validate_customer.py
apps:
- path: apps/support_chatbot.py
objects:
- path: objects/customer.yml
flows:
- path: flows/onboarding.yml
studio_apps:
- alias: customer_portal
name: Customer Portal
path: studio/customer_portal
endpoints:
- path: endpoints/customer_api.yml
```
## Next Steps
* [Learn about Pipelines](/reference/development/pipeline)
* [Create Actions](/reference/development/actions)
* [Build Intelligent Apps](/reference/intelligent-apps/overview)
* [Build Studio Apps](/reference/studio-apps/overview)
* [Define Knowledge Objects](/reference/knowledge-objects/overview)
* [Orchestrate with Flows](/reference/flows/overview)
* [Set up Endpoints](/reference/integration/endpoints)
# Transform
Source: https://docs.datazone.co/reference/development/transform
Transform functions are the bricks of your pipeline. You can atomize your data processing steps into small functions and chain them together to build a pipeline.
| Parameter Name | Default | Description |
| ------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| **compute\_fn** | - | The main computation function to be transformed. |
| **name** | - | Name of the transform function. If not provided, uses the function name. |
| **description** | - | Description of the transform function for documentation purposes. |
| **materialized** | False | If True, the output will be persisted/cached for reuse. |
| **input\_mapping** | - | Dictionary defining input dependencies and their sources. Maps input parameter names to their corresponding datasets or transforms. |
| **depends** | - | List of transform functions that must complete before this transform can run. |
| **partition\_by** | - | List of column names to partition the output data by. |
| **output\_mapping** | - | Dictionary defining how the output should be mapped or stored. |
| **tags** | - | List of tags for categorizing and organizing transforms. |
| **engine** | pyspark | The computation engine to use. Options are 'pyspark' or 'pandas'. |
### Basic Transform Example
```python theme={null}
from datazone import transform
@transform(name="say_hello", description="Prints 'Hello, World!'")
def say_hello():
print("Hello, World!")
```
## Transform Engines
The transform decorator supports two computation engines: PySpark and Pandas. You can specify the engine using the `engine` parameter.
```python theme={null}
from datazone import transform
@transform(
input_mapping={"user_data": Input(Dataset(id="66280b3f49ae018b4c0c904a"))},
engine="pandas"
)
def filter_by_age(user_data):
adult_users = user_data[user_data.age >= 30]
return adult_users
```
When using transforms with dependencies (via `input_mapping` or `depends`), all connected transforms must use the same engine.
For example, if a transform uses the Pandas engine, all transforms it depends on or that depend on it must also use the Pandas engine.
### Engine Characteristics
* **PySpark (default)**:
* Distributed processing capabilities
* Better for large-scale data processing
* Supports all materialization features
* **Pandas**:
* Better for smaller datasets
* More intuitive Python-native syntax
* Ideal for local development and testing
## Materialization
Materialization is the process of storing the output of a transform function for reuse.
This can be useful when a transform function is computationally expensive and its output is used multiple times in the pipeline.
Materialization is allowed for PySpark DataFrames only. So the output of the transform function should be a PySpark DataFrame.
```python theme={null}
from datazone import transform
@transform(name="fetch_data", materialized=True)
def fetch_data(context):
data = [("Alice", 9), ("Bob", 6), ("Charlie", 3), ("Maria", 7)]
columns = ["Name", "Score"]
spark = context.resources["pyspark"].spark
return spark.createDataFrame(data, columns)
```
In above code, we used `context.resources["pyspark"].spark` to access the PySpark session. For more information,
check the [Context](/reference/development/context) section.
After running the pipeline, the output of the `fetch_data` function will be stored in Datazone as a dataset.
You can check the dataset alias in the Datazone UI or use the `datazone dataset list` command to list all datasets.
## Input Mapping
You can define input mappings to specify the data sources and dependencies for your transform functions. Input mappings enable you to:
* Chain multiple transform functions
* Create directed acyclic graphs (DAGs)
* Connect to different data sources
* Apply data transformations sequentially
| Parameter Name | Default | Description |
| ---------------- | ------- | ---------------------------------------------------------------------------------- |
| **entity** | - | The entity to be used as input. It can be a dataset or another transform function. |
| **output\_name** | - | If you use a transform that has multiple outputs, you can specify the output name. |
Here are the common input mapping patterns:
```python theme={null}
from datazone import transform
@transform(input_mapping={"data": Input(Dataset(alias="raw_data"))})
def clean_data(data):
return data.filter(data["column"] > 0)
@transform(
input_mapping={
"clean_data": Input(clean_data),
"another_data": Input(Dataset(alias="another_data")),
}
)
def aggregate_data(clean_data, another_data):
return (
clean_data.join(another_data, "column")
.groupBy("column")
.agg({"column": "sum"})
.select("column", "sum(column)")
)
```
* Input mappings should be defined as a dictionary where the key is the input parameter name and the value is an instance of the `Input` class.
* The `Input` class accepts a `Dataset` or another transform function as an argument.
```mermaid theme={null}
graph LR
subgraph Pipeline
clean[clean_data filter column > 0] --> agg[aggregate_data join and groupBy]
end
data[(Dataset raw_data)]-->clean
another[(Dataset another_data)]-->agg
style Pipeline fill:#f0f0f0,stroke:#333,stroke-width:2px
style data fill:#b8e994,stroke:#333,stroke-width:1px
style another fill:#b8e994,stroke:#333,stroke-width:1px
```
## Output Mapping
Output mapping defines how the output of a transform function should be stored or mapped.
You can specify the output mapping using the `output_mapping` parameter.
| Parameter Name | Default | Description |
| ----------------- | --------- | ---------------------------------------------------------------------- |
| **dataset** | - | The dataset where the output should be stored. |
| **materialized** | False | If True, the output will be stored as a materialized dataset. |
| **partition\_by** | - | List of column names to partition the output data by. |
| **mode** | overwrite | The write mode for the output data. Options are `overwrite`, `append`. |
```python theme={null}
from datazone import transform, Output, Dataset
@transform(output_mapping={dataset: Output(dataset=Dataset(alias="clean_data"))})
def clean_data(data):
return data.filter(data["column"] > 0)
```
* Output mapping should be defined as a dictionary where the key is the output parameter name and the value is an instance of the `Output` class.
### Multiple Outputs
```python theme={null}
@transform(
input_mapping={"orders": Input(Dataset(alias="orders"))},
output_mapping={
"daily_sales": Output(materialized=True),
"monthly_sales": Output(materialized=True),
"yearly_sales": Output(
partition_by=["year"],
materialized=True
)
}
)
def process_sales(orders):
daily = orders.groupBy("date").agg(...)
monthly = orders.groupBy("year", "month").agg(...)
yearly = orders.groupBy("year").agg(...)
return daily, monthly, yearly
```
```mermaid theme={null}
graph LR
subgraph Transform
process_sales
end
orders[(orders)]-->process_sales
process_sales-->daily[(daily_sales)]
process_sales-->monthly[(monthly_sales)]
process_sales-->yearly[(yearly_sales)]
```
## Transform Hooks
Transform hooks allow you to execute custom logic on success or failure of a transform function.
You can define hooks using the `on_success` and `on_failure` parameters in the `transform` decorator.
```python theme={null}
from datazone import transform
@transform(name="example_transform")
def generate_sales_data():
return [("Alice", 100), ("Bob", 150), ("Charlie", 200)]
@generate_sales_data.on_success
def log_success(context):
print(f"Transform {context.state.name} completed successfully.")
@generate_sales_data.on_failure
def log_failure(context):
print(f"Transform {context.state.name} failed with error: {context.error}")
```
## Partitioning
Partitioning helps organize and optimize your datasets in our data platform.
When you create a transform function, you can specify partition columns using the `partition_by` parameter.
```python theme={null}
@transform(
partition_by=["date", "country"],
materialized=True
)
def sales_by_country(data):
return data.filter(...)
```
### Why Partition?
* Improve query performance when filtering by partition columns
* Efficiently manage large datasets
* Enable data retention policies by date partitions
### Common Partition Strategies
```python theme={null}
# Daily partitioning for time-series data
@transform(partition_by=["date"])
# Geographic partitioning
@transform(partition_by=["country", "region"])
# Multiple partition columns
@transform(partition_by=["date", "customer_type"])
```
All transformed datasets are stored in Delta Lake format. Choose partition columns based on your most common filtering needs, typically date-based or categorical columns with reasonable cardinality.
If you partition by a high-cardinality column, it may lead to a large number of small files, which can impact query performance.
### Best Practices
* Use date partitioning for time-series data
* Avoid partitioning by columns with high cardinality
* Consider your query patterns when choosing partition columns
## Generator Transforms
Generator transforms allow you to yield data multiple times from a single transform function. This is useful for processing data in chunks, creating multiple batches from a single input, or streaming incremental results.
### Usage
Use Python's `yield` statement to return data in chunks. Each yielded value creates a separate batch that gets written to the output dataset.
```python theme={null}
@transform(
input_mapping={"user_data": Input(Dataset(alias="users"))},
output_mapping={"processed_data": Output(Dataset(alias="processed-users"), materialized=True, mode="append")},
)
def process_in_chunks(user_data):
# Split data into chunks of 1000 rows
chunk_size = 1000
total_rows = user_data.count()
for offset in range(0, total_rows, chunk_size):
chunk = user_data.limit(chunk_size).offset(offset)
processed_chunk = chunk.filter(chunk.age >= 18)
yield processed_chunk
```
```mermaid theme={null}
graph TD
input[(users 10,000 rows)] --> start[Start Transform]
start --> loop{For each chunk}
loop -->|offset 0-999| process1[Process & Filter chunk 1]
loop -->|offset 1000-1999| process2[Process & Filter chunk 2]
loop -->|offset 2000-2999| process3[Process & Filter chunk 3]
loop -->|offset N| processN[Process & Filter chunk N]
process1 -->|yield| output1[(Append to processed-users)]
process2 -->|yield| output2[(Append to processed-users)]
process3 -->|yield| output3[(Append to processed-users)]
processN -->|yield| outputN[(Append to processed-users)]
output1 --> check1{More chunks?}
output2 --> check2{More chunks?}
output3 --> check3{More chunks?}
outputN --> end1[End]
check1 -->|Yes| loop
check2 -->|Yes| loop
check3 -->|Yes| loop
```
**Multiple Transactions**: Each `yield` creates a separate transaction. If your transform fails after some yields have succeeded, the already-written batches will remain in the output dataset. Consider idempotency in your pipeline design.
**Mode Must Be Append**: Generator transforms require `mode="append"` in the output mapping. Using `mode="overwrite"` will cause each yielded batch to overwrite the previous one, leaving only the last batch in the output dataset.
# Variables
Source: https://docs.datazone.co/reference/development/variables
Variables are used to store and manage data in the Datazone platform. You can define variables in the Datazone dashboard and use them in your pipelines.
## Overview
1. Click **Settings** in the top right corner session context menu.
2. Click **Variables** in the left sidebar.
3. Click **Create** to create a new Variable.
4. Enter a name for the Variable and value. Also, you can define whether the variable is secret or not. If you define the variable as secret, the value will be encrypted and will not be shown in the UI.
5. Click **Create**.
## Usage
You can use the Variable in your pipelines and notebooks. You can access the Variable value using the `Variable` class.
```python theme={null}
from datazone import Variable, transform
@transform
def my_transform():
variable = Variable(key="my-secret-variable")
print(str(variable))
```
Also in Notebooks, you can use the `Variable` class to access the Variable value.
# Vectors
Source: https://docs.datazone.co/reference/development/vectors
Transform your data into searchable embeddings for AI-powered applications
Datazone Vectors enable you to **transform your data into embeddings** and store them in a vector database. This allows you to build **AI-powered search**, **RAG (Retrieval Augmented Generation)** applications, and **semantic similarity** features on top of your data.
## What are Vectors?
Vectors convert your text data into **numerical representations** (embeddings) that capture **semantic meaning**. This enables:
* **Semantic search** - Find relevant content based on meaning, not just keywords
* **RAG applications** - Enhance AI agents with contextual information from your data
* **Similarity matching** - Identify related documents or records
Unlike traditional keyword search, vector-based search understands **context and intent**, delivering more accurate and relevant results.
## Creating a Vector Index
1. Navigate to your **Project** page
2. Click the **Add** button (+ icon)
3. Select **Vector** from the dropdown
The vector creation flow guides you through **configuring your vector index**:
### Step 1: Choose Data Source
Select what data to vectorize:
* **Dataset** - Index data from a dataset (CSV format)
* Select the dataset you want to vectorize
* Choose a **primary key column** to uniquely identify records
* Select one or more **text columns** to embed
* **File Container** - Index files from a file container path
* Supported formats: **PDF**, **CSV**, **DOCX**, **XLSX**, **Markdown**, **TXT**
* Files will be automatically processed and chunked
For large datasets, consider using **Views** or filtered datasets to index only the most relevant data.
### Step 2: Embedding Configuration
Configure how your data will be embedded:
* **Model Account** - Select a configured [Model Account](/reference/development/model-accounts) with embedding support
* **Embedding Model** - Choose the embedding model (e.g., `text-embedding-3-small`, `text-embedding-ada-002`)
* **Vector Dimension** - Embedding dimension (typically 1536 for OpenAI models)
### Step 3: Chunking Strategy
Configure how your text is split into chunks before embedding:
* **Text Splitting** - Split by character count
* **Chunk Size** - Number of characters per chunk
* **Chunk Overlap** - Characters shared between chunks (helps maintain context)
* **Length Splitting** - Split by token count using a specific encoding
* **Encoding Name** - Tokenizer to use (e.g., `cl100k_base` for GPT models)
* **Chunk Size** - Number of tokens per chunk
* **Chunk Overlap** - Tokens shared between chunks
* **Document Splitting** - Split based on document structure
* **Document Type** - Choose format: Markdown, JSON, Code, or HTML
* Preserves logical document boundaries
Smaller chunks provide more precise search results but may lose broader context. Larger chunks retain more context but may be less specific. A typical chunk size is **500-1000 characters** with **10-20% overlap**.
After completing configuration, click **Create** to start the indexing process.
## Indexing Process
Once created, your vector index goes through several states:
1. **Not Indexed** - Initial state after creation
2. **Scheduled** - Queued for indexing
3. **Indexing** - Currently processing and embedding your data
4. **Indexed** - Successfully completed, ready to use
5. **Failed** - Error occurred during indexing (check error details)
You can monitor the **indexing status** and view **statistics** including:
* Total chunks created
* Total chunks indexed
* Total tokens processed
## Using Vectors
Once indexed, your vectors can be used for:
### Semantic Search
Search your data using **natural language queries** that understand meaning and context, not just exact keyword matches. The **Explore** tab provides an interface to:
* Enter search queries in natural language
* View semantically similar results ranked by relevance
* See matching chunks with their context and metadata
* Test and refine your vector search results
### Agent RAG
Attach vectors as data sources for your [Agents](/reference/agents/overview), enabling them to retrieve relevant context from your data to provide more accurate and informed responses. Agents can automatically perform semantic search when answering questions.
### Similarity Endpoints
Create [API endpoints](/reference/integration/endpoints#vector-based-endpoints) that return similar records based on vector similarity, enabling semantic search in your applications. Perfect for building search features or recommendation systems.
## Best Practices
1. **Choose the Right Source** - Use datasets for structured data and file containers for documents
2. **Optimize Chunk Size** - Balance between context (larger chunks) and precision (smaller chunks)
3. **Add Overlap** - Include 10-20% overlap to maintain context across chunk boundaries
4. **Select Appropriate Models** - Smaller embedding models are faster and cheaper, larger models may provide better quality
5. **Use Cosine Similarity** - Works well for most text similarity use cases
6. **Monitor Indexing** - Check indexing status and statistics to ensure successful processing
## Next Steps
* [Configure Model Accounts](/reference/development/model-accounts) for embedding models
* [Create Agents](/reference/agents/overview) to leverage your vectors
* [File Containers](/reference/development/file-container) for document management
# Node Reference
Source: https://docs.datazone.co/reference/flows/nodes
Detailed reference for every node type available in Datazone Flows
## Common Properties
Every node in a flow document shares the same envelope:
| Property | Type | Description |
| -------- | ------ | ---------------------------------------------------------------------------------------------------------------------- |
| `id` | string | Unique identifier for the node within the flow |
| `type` | string | Node type: `start`, `if`, `for_each`, `llm_call`, `rest_call`, `action_call`, `python`, `sql`, `response`, or `output` |
| `label` | string | (Optional) Display name shown on the canvas |
| `config` | object | Type-specific configuration, validated against that node type's schema |
Each node type also declares a fixed set of **input** and **output** ports (e.g. `in` / `out`). [Connections](/reference/flows/yaml-reference#connections) wire an upstream node's output port to a downstream node's input port.
## Start
The entry point of a flow. It takes no configuration and no inputs — it simply emits the run's parameters and a trigger timestamp so downstream nodes have a well-defined starting payload.
| Property | Category | Inputs | Outputs |
| -------- | -------- | ------ | ------- |
| `start` | trigger | — | `out` |
```yaml theme={null}
- id: trigger
type: start
label: Start
config: {}
```
**Output:**
```json theme={null}
{
"kind": "start",
"parameters": { "...": "the run's parameters" },
"triggered_at": "2026-07-24T12:00:00+00:00"
}
```
Every flow must have exactly one `start` node as its entry point.
## If
Branches the flow based on a condition evaluated against upstream data and flow parameters.
| Property | Category | Inputs | Outputs |
| -------- | ----------------------- | ------ | --------------- |
| `if` | control (**branching**) | `in` | `true`, `false` |
| Attribute | Type | Default | Description |
| ---------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------- |
| `left` | string | — | Python expression evaluated with `inputs`/`parameters` in scope (required) |
| `operator` | string | `eq` | One of `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `contains`, `is_true`, `is_false`, `is_empty`, `is_not_empty` |
| `right` | any | `null` | Right-hand operand. Ignored for `is_true`/`is_false`/`is_empty`/`is_not_empty` |
```yaml theme={null}
- id: check_status
type: if
label: Is approved?
config:
left: "inputs['in']['status']"
operator: eq
right: "approved"
```
Because `if` is a **branching** node, every outgoing connection from it must set `from_port` to `true` or `false`:
```yaml theme={null}
connections:
- from: check_status
to: send_email
from_port: "true"
- from: check_status
to: log_rejection
from_port: "false"
```
A connection leaving a branching node without a valid `from_port` fails validation. Nodes only reachable through the branch that isn't chosen at run time are marked `SKIPPED` instead of executed.
When the upstream input is a dict, its fields are merged into the output alongside `branch` and `value`, so downstream nodes on either branch still see the original payload.
## For Each
Iterates over a list, re-running a nested **body** sub-flow once per item.
| Property | Category | Inputs | Outputs |
| ---------- | ----------------------- | ------ | ------- |
| `for_each` | control (**container**) | `in` | `out` |
| Attribute | Type | Default | Description |
| ---------------- | ------ | --------- | ------------------------------------------------------------------------------------------------------------ |
| `items` | string | — | Python expression evaluated with `inputs`/`parameters` in scope; must evaluate to a list/iterable (required) |
| `item_as` | string | `item` | Parameter name the current item is exposed under inside the body, as `parameters[item_as]` |
| `index_as` | string | `index` | Parameter name the current index is exposed under inside the body (omit to disable) |
| `max_iterations` | int | `1000` | Safety cap — exceeding it raises a run error |
| `save_as` | string | `results` | Output key the collected per-iteration results are stored under |
| `body` | object | — | A full nested flow document (its own `nodes`/`connections`), required |
```yaml theme={null}
- id: process_orders
type: for_each
label: Process each order
config:
items: "inputs['in']['orders']"
item_as: order
index_as: index
max_iterations: 500
save_as: results
body:
flow:
name: process_orders_body
nodes:
- id: body_start
type: start
config: {}
- id: format_order
type: python
config:
code: |
def handler(inputs, parameters):
order = parameters["order"]
return {"id": order["id"], "total": order["amount"] * 1.1}
- id: body_response
type: response
config: {}
connections:
- from: body_start
to: format_order
- from: format_order
to: body_response
```
The `body` document is validated the exact same way a top-level flow is, including supporting another `for_each` nested inside it (up to a max depth of 5). It **must contain exactly one `response` node** — that node's output is what gets collected into `save_as` for each iteration.
## LLM Call
Calls a Large Language Model through a configured [Model Account](/reference/development/model-accounts), or the organization's Orion AI default account when omitted.
| Property | Category | Inputs | Outputs |
| ---------- | ---------------------- | ------ | ------- |
| `llm_call` | ai (needs credentials) | `in` | `out` |
| Attribute | Type | Default | Description |
| ------------------ | ------ | ---------- | -------------------------------------------------------------------------------------- |
| `model_account_id` | string | `null` | Model account to use. Omit to use the organization's Orion AI settings |
| `model` | string | `null` | Model enum value (e.g. `GPT_4O`). Omit to use the organization default |
| `prompt` | string | — | Prompt template, required. Supports `{inputs[...]}` / `{parameters[...]}` substitution |
| `system_prompt` | string | `null` | (Optional) System prompt, same templating support |
| `save_as` | string | `response` | Output key the model's response text is stored under |
| `timeout_sec` | int | `60` | Request timeout in seconds |
```yaml theme={null}
- id: summarize
type: llm_call
label: Summarize ticket
config:
model_account_id: "664f1c2e5a2ac9f0d39326"
model: GPT_4O
system_prompt: "You are a support triage assistant."
prompt: "Summarize this ticket in one sentence: {inputs[in][body]}"
save_as: summary
timeout_sec: 45
```
`prompt`/`system_prompt` use Python's `str.format()` templating (not Jinja) — see [Templating & Expressions](/reference/flows/templating#llm_call-and-action_call-templating) for the exact rules and error behavior when a referenced field is missing.
## REST Call
Calls an external HTTP/REST endpoint.
| Property | Category | Inputs | Outputs |
| ----------- | ----------- | ------ | ---------- |
| `rest_call` | integration | `in` | `response` |
| Attribute | Type | Default | Description |
| -------------------------------- | ------ | ---------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `url` | string | — | Target URL, required. Must start with `http://` or `https://` |
| `method` | string | `GET` | One of `GET`, `POST`, `PUT`, `PATCH`, `DELETE` |
| `headers` | object | `{}` | Request headers |
| `query` | object | `{}` | Query parameters. Values that reference `inputs`/`parameters` are evaluated as Python expressions; plain literals are sent as-is |
| `json_body` | object | `null` | JSON request body |
| `timeout_sec` | int | `30` | Request timeout in seconds |
| `save_as` | string | `response` | Output key the parsed response body is stored under |
| `auth_type` | string | `none` | One of `none`, `basic`, `bearer`, `api_key` |
| `username` / `password` | string | `null` | Used when `auth_type: basic` |
| `token` | string | `null` | Used when `auth_type: bearer` — sent as `Authorization: Bearer ` |
| `api_key_name` / `api_key_value` | string | `null` | Used when `auth_type: api_key` — sent as a `: ` header |
```yaml theme={null}
- id: fetch_weather
type: rest_call
label: Fetch London weather
config:
url: "https://api.open-meteo.com/v1/forecast"
method: GET
query:
latitude: "51.5074"
longitude: "-0.1278"
current_weather: "true"
timeout_sec: 30
save_as: weather
```
Authenticated example:
```yaml theme={null}
- id: call_billing_api
type: rest_call
label: Get invoice
config:
url: "https://billing.example.com/api/invoices/{parameters[invoice_id]}"
method: GET
auth_type: bearer
token: "sk_live_..."
save_as: invoice
```
The response is always returned alongside its `status_code`:
```json theme={null}
{ "response": { "...": "parsed JSON or raw text" }, "status_code": 200 }
```
## Action Call
Calls a project [Action](/reference/development/actions) — a user-defined Python function — with parameters.
| Property | Category | Inputs | Outputs |
| ------------- | ------------------------------- | ------ | ------- |
| `action_call` | integration (needs credentials) | `in` | `out` |
| Attribute | Type | Default | Description |
| ------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------- |
| `action_id` | string | — | Id of the Action to call, required |
| `parameters` | object | `{}` | Parameters passed to the action. String values support `{inputs[...]}`/`{parameters[...]}` templating |
| `save_as` | string | `result` | Output key the action's return value is stored under |
| `timeout_sec` | int | `30` | Timeout in seconds |
```yaml theme={null}
- id: send_welcome_email
type: action_call
label: Send welcome email
config:
action_id: "664f1c2e5a2ac9f0d39326"
parameters:
to: "{inputs[in][email]}"
name: "{inputs[in][full_name]}"
save_as: email_result
timeout_sec: 30
```
Actions are audited the same way as when an agent calls them — every `action_call` node execution appears in the Action's call history.
## Python
Runs custom Python against upstream data and flow parameters, in an isolated subprocess.
| Property | Category | Inputs | Outputs |
| -------- | --------- | ------ | ------- |
| `python` | transform | `in` | `out` |
| Attribute | Type | Description |
| --------- | ------ | ------------------------------------------------------------------------------------- |
| `code` | string | A complete Python module defining a `handler(inputs, parameters)` function (required) |
```yaml theme={null}
- id: convert_temperature
type: python
label: Convert temperature
config:
code: |
def handler(inputs, parameters):
# inputs – dict of upstream node outputs, keyed by port name
# parameters – dict of flow run parameters
weather = inputs.get("in", {}).get("weather", {})
celsius = weather.get("current_weather", {}).get("temperature")
fahrenheit = round(celsius * 9 / 5 + 32, 1)
return {"celsius": celsius, "fahrenheit": fahrenheit}
```
`code` must define a `handler(inputs, parameters)` function and be valid Python — both are checked before the flow can run.
## SQL
Runs a read-only SQL `SELECT` against the project's own datasets, the same way SQL Explorer does, and returns the rows to the flow.
| Property | Category | Inputs | Outputs |
| -------- | ------------------------ | ------ | ------- |
| `sql` | data (needs credentials) | `in` | `out` |
| Attribute | Type | Default | Description |
| ------------- | ------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `query` | string | — | The `SELECT` query, required. Supports `str.format()` templating — see [Templating & Expressions](/reference/flows/templating#llm_call-action_call-and-sql-templating) |
| `save_as` | string | `result` | Output key the query result is stored under |
| `timeout_sec` | int | `60` | Request timeout in seconds |
```yaml theme={null}
- id: get_orders
type: sql
label: Fetch recent orders
config:
query: "SELECT * FROM orders WHERE id = {parameters[order_id]} LIMIT 100"
save_as: result
timeout_sec: 60
```
**Output:**
```json theme={null}
{
"result": {
"data": [{ "id": 1, "amount": 10 }, { "id": 2, "amount": 20 }],
"columns": ["id", "amount"],
"row_count": 2
}
}
```
`query` uses the same `str.format()` templating as `llm_call`/`action_call` — a field referenced in `query` but missing from `inputs`/`parameters` fails the node with a clear error naming the missing key, rather than sending a malformed query.
Only `SELECT` (and `UNION`/`INTERSECT`/`EXCEPT`) queries are accepted — multi-statement queries and `INTO OUTFILE`/`INTO DUMPFILE` are rejected. The query runs with the permissions of whoever triggered the flow run, against the project's own datasets — there's no separate connection or data source to configure. Unlike SQL Explorer, **no default row limit is applied** to a flow's `sql` node, so add your own `LIMIT` or a large result set can be slow to return.
Every `sql` node execution is recorded as an auditable query result against the flow run, and counts against the organization's query quota — a run made while that quota is exceeded fails with `Query quota exceeded for this account`.
## Response
A terminal node that returns the upstream result, unchanged, to the flow's caller.
| Property | Category | Inputs | Outputs |
| ---------- | -------- | --------------- | ------- |
| `response` | terminal | `in` (required) | — |
```yaml theme={null}
- id: finish
type: response
label: Response
config: {}
```
A flow may contain **at most one** top-level `response` node. Inside a `for_each` body, exactly **one** `response` node is required — it defines that iteration's result.
## Output
A sink node — writes the upstream result to a dataset, webhook, or file.
| Property | Category | Inputs | Outputs |
| -------- | -------- | --------------- | ------- |
| `output` | sink | `in` (required) | — |
| Attribute | Type | Default | Description |
| --------------- | ------ | ------- | ------------------------------------------------------------------------- |
| `kind` | string | — | One of `dataset`, `webhook`, `file` (required) |
| `dataset_alias` | string | `null` | Required when `kind: dataset` |
| `materialized` | bool | `false` | Reserved for a future materialized dataset write path — not yet supported |
| `url` | string | `null` | Required when `kind: webhook`. Must start with `http://` or `https://` |
| `path` | string | `null` | Required when `kind: file` |
```yaml theme={null}
- id: post_result
type: output
label: Send to webhook
config:
kind: webhook
url: "https://hooks.example.com/flow-result"
```
```yaml theme={null}
- id: write_dataset
type: output
label: Write to dataset
config:
kind: dataset
dataset_alias: "processed_orders"
```
Setting `materialized: true` on a `kind: dataset` output currently fails validation — materialized dataset writes are not supported yet.
## Next Steps
* [Overview](/reference/flows/overview) - flow structure and core concepts
* [Templating & Expressions](/reference/flows/templating) - exactly how each node type consumes `inputs`/`parameters`
* [YAML Reference](/reference/flows/yaml-reference) - the complete flow document schema
# Overview
Source: https://docs.datazone.co/reference/flows/overview
Build orchestration graphs that call LLMs, REST APIs, and Actions with a declarative YAML flow
# Flows
A Flow is a **directed graph of typed nodes** you connect together to orchestrate a task: call an LLM, hit an external API, branch on a condition, loop over a list, or invoke a project [Action](/reference/development/actions). Flows are defined declaratively — the same document your Flow Builder canvas edits is what gets executed — so every run is reproducible and versionable.
## Overview
A Flow consists of:
1. **Nodes** - Typed steps (`start`, `if`, `llm_call`, `rest_call`, `action_call`, `python`, `sql`, `for_each`, `output`, `response`) that read inputs and produce outputs
2. **Connections** - Directed edges wiring one node's output port to another node's input port
3. **Parameters** - Values passed in at run time (manual trigger or schedule), available to every node
## Flow Structure
Flows use a YAML-based document format — the same envelope the Flow Builder posts when you edit on the canvas or switch to the YAML view:
```yaml theme={null}
flow:
name: "My Flow"
nodes:
- id: trigger
type: start
label: Start
config: {}
- id: call_api
type: rest_call
label: Fetch data
config:
url: "https://api.example.com/data"
method: GET
- id: finish
type: response
label: Response
config: {}
connections:
- from: trigger
to: call_api
- from: call_api
to: finish
```
## Key Concepts
### Nodes
Every node has an **id** (unique within the flow), a **type** (which node class runs it), an optional **label**, and a **config** bag whose shape is validated against that node type's schema. See the [Node Reference](/reference/flows/nodes) for every type.
### Connections and ports
A connection wires one node's **output port** to another node's **input port**:
```yaml theme={null}
connections:
- from: fetch_weather
to: process
from_port: response # optional — defaults to the node's first output
to_port: in # optional — defaults to the node's first input
```
Most nodes have a single input (`in`) and a single output, so `from_port`/`to_port` can usually be omitted. Two node types are the exception:
* **`if`** is a **branching** node — its outputs are the mutually-exclusive ports `true`/`false`, and every outgoing connection from it **must** set `from_port`.
* **`for_each`** is a **container** node — it owns a nested `body` sub-flow (its own `nodes`/`connections`) instead of branching, executed once per item.
### Data flow between nodes
A node receives its upstream nodes' outputs as `inputs`, keyed by input port name, plus the run's `parameters`. Most nodes read `inputs.get("in")` and `save_as` a named key in their own output — see [Templating & Expressions](/reference/flows/templating) for exactly how each node type consumes them.
### Runs
A flow is executed as a **run**, triggered either **manually** or on a **schedule**. Each node execution is tracked as a step with its own status (`PENDING` -> `RUNNING` -> `SUCCESS`/`ERROR`/`SKIPPED`). See [Triggers & Runs](/reference/flows/triggers-and-runs).
## Best Practices
1. **One `response` node per flow** - a flow may contain at most one top-level `response` node; it's what gets returned to the caller
2. **Name your `save_as` keys clearly** - downstream nodes reference upstream output by the key you chose (e.g. `weather`, `result`)
3. **Set `from_port` on every branch** - connections leaving an `if` node must declare `true` or `false` explicitly
4. **Keep loops bounded** - `for_each` enforces a `max_iterations` cap (default `1000`) to guard against runaway iteration
5. **Avoid cycles** - a flow must be a DAG; the validator rejects any cycle before a run starts
## Next Steps
* [Node Reference](/reference/flows/nodes) - every node type, its config, and its ports
* [Triggers & Runs](/reference/flows/triggers-and-runs) - starting a flow manually or on a schedule, and reading run/step status
* [Templating & Expressions](/reference/flows/templating) - how nodes reference upstream data and flow parameters
* [YAML Reference](/reference/flows/yaml-reference) - the complete flow document schema
# Templating & Expressions
Source: https://docs.datazone.co/reference/flows/templating
How Flow nodes reference upstream data and run parameters.
Flow nodes read two things at run time: **`inputs`** (a dict of upstream nodes' outputs, keyed by input port name) and **`parameters`** (the run's parameters, plus — inside a `for_each` body — the current item/index). Different node types expose two different ways to reference them in `config`.
## `llm_call`, `action_call`, and `sql` templating
`llm_call`'s `prompt`/`system_prompt`, `action_call`'s string `parameters` values, and `sql`'s `query` are plain Python [`str.format()`](https://docs.python.org/3/library/string.html#format-string-syntax) templates, rendered against `inputs`/`parameters`:
```yaml theme={null}
- id: summarize
type: llm_call
config:
prompt: "Summarize this ticket: {inputs[in][body]}"
```
```yaml theme={null}
- id: notify
type: action_call
config:
action_id: "664f1c2e5a2ac9f0d39326"
parameters:
to: "{inputs[in][email]}"
greeting: "Hello, {parameters[first_name]}!"
```
```yaml theme={null}
- id: get_orders
type: sql
config:
query: "SELECT * FROM orders WHERE id = {parameters[order_id]} LIMIT 100"
```
* Nested dict/list access uses `[key]` chains — no dots (e.g. `{inputs[in][user][email]}`, not `{inputs.in.user.email}`).
* `action_call` renders every **string** value in `parameters` this way, recursively through nested dicts/lists; non-string values (numbers, booleans) pass through unchanged.
* If a referenced key is missing, the node fails with a clear error naming the field and the missing key — e.g. `llm_call 'prompt' references 'foo', which is missing from inputs/parameters on this run` (or, for `sql`, `sql 'query' references 'foo', which is missing from inputs/parameters on this run`) — rather than silently rendering an empty string or sending a malformed query.
## `if`, `for_each`, and `rest_call` query values: Python expressions
`if.config.left`, `for_each.config.items`, and any `rest_call.config.query` value that mentions `inputs` or `parameters` are evaluated as a **Python expression** (via `eval`), with `inputs` and `parameters` as the only names in scope:
```yaml theme={null}
- id: check_amount
type: if
config:
left: "inputs['in']['amount'] > parameters['threshold']"
operator: is_true
```
```yaml theme={null}
- id: process_items
type: for_each
config:
items: "inputs['in']['orders']"
```
```yaml theme={null}
- id: search
type: rest_call
config:
url: "https://api.example.com/search"
query:
q: "inputs['in']['query']" # evaluated (mentions `inputs`)
limit: "20" # sent as a literal string — no reference
```
This is a genuine `eval()` against untrusted-shaped input — the subprocess sandbox each node runs in is the isolation boundary, not the expression syntax itself. Only reference `inputs`/`parameters`; there is no other trust model layered on top.
## `python` node: full function body
The `python` node doesn't template strings at all — it runs a complete Python module. Your `handler(inputs, parameters)` function receives both dicts directly and returns the node's output:
```yaml theme={null}
- id: transform
type: python
config:
code: |
def handler(inputs, parameters):
order = inputs.get("in", {})
return {"total": order["amount"] * 1.1}
```
## Passing data across a branch
When an `if` node's upstream input is a dict, its fields are merged through to the chosen branch alongside `branch` and `value`, so a node downstream of `true` (or `false`) still sees the original payload, not just the condition's result:
```json theme={null}
// if node output on the "true" branch, given upstream input {"ticket": {...}}
{ "ticket": { "...": "..." }, "branch": "true", "value": true }
```
## Reference
| Node type | Field(s) | Syntax |
| ------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `llm_call` | `prompt`, `system_prompt` | `str.format()` — `{inputs[...]}`, `{parameters[...]}` |
| `action_call` | `parameters` (string values, recursively) | `str.format()` — `{inputs[...]}`, `{parameters[...]}` |
| `sql` | `query` | `str.format()` — `{inputs[...]}`, `{parameters[...]}` |
| `if` | `left` | Python expression (`eval`) with `inputs`/`parameters` in scope |
| `for_each` | `items` | Python expression (`eval`) with `inputs`/`parameters` in scope |
| `rest_call` | `query` values | Python expression (`eval`) when the string mentions `inputs`/`parameters`; otherwise sent as a literal |
| `python` | `code` | Full Python module; `handler(inputs, parameters)` receives both dicts as-is |
See also: [Node Reference](/reference/flows/nodes), [Overview](/reference/flows/overview).
# Triggers & Runs
Source: https://docs.datazone.co/reference/flows/triggers-and-runs
How Datazone Flows start, and how to read the status of a run.
Every execution of a flow is a **run**. A run is created either manually or by a schedule, and progresses through the graph one node at a time, recording a per-node **step** as it goes.
## Trigger Types
Datazone supports two ways to start a flow:
| Type | Description |
| ---------- | ---------------------------------------------------------------------- |
| `MANUAL` | Started on demand, from the UI or by calling the run endpoint directly |
| `SCHEDULE` | Started automatically on a cron expression |
### Manual Runs
A manual run takes an optional set of **parameters** — a flat object made available to every node as `parameters` (and, on the `start` node's output, echoed back under `parameters`):
```json theme={null}
{
"parameters": {
"city": "London",
"invoice_id": "INV-2024-0001"
}
}
```
### Scheduled Runs
A **Flow Schedule** runs a flow's deployed definition automatically on a cron expression:
| Attribute | Type | Description |
| ------------ | ------ | ---------------------------------------------------------------- |
| `name` | string | Schedule name |
| `flow` | string | Id of the flow to run |
| `branch` | string | Branch whose definition should be scheduled (defaults to `main`) |
| `expression` | string | Cron expression (e.g. `0 * * * *` for hourly) |
| `parameters` | object | (Optional) Parameters passed to every scheduled run |
| `is_active` | bool | Whether the schedule is currently enabled |
```yaml theme={null}
name: "Hourly weather refresh"
flow: "664f1c2e5a2ac9f0d39326"
branch: main
expression: "0 * * * *"
parameters:
city: "London"
```
A schedule always runs the flow's definition on the **branch it was created against**. Deploying a new version of the flow to that branch changes what the next scheduled run executes.
## Run Lifecycle
A run moves through the following statuses:
| Status | Description |
| ---------- | ---------------------------------------------- |
| `CREATED` | The run has been accepted and queued |
| `RUNNING` | The graph is currently executing |
| `SUCCESS` | Every required node finished without error |
| `FAILURE` | A node raised an error that wasn't recoverable |
| `CANCELED` | The run was canceled before it finished |
## Step Lifecycle
Each node execution inside a run is tracked as a **step**, with its own status:
| Status | Description |
| ---------- | ----------------------------------------------------------------------- |
| `PENDING` | Not started yet |
| `RUNNING` | Currently executing |
| `SUCCESS` | Finished without error |
| `ERROR` | Raised an error |
| `TIMEOUT` | Exceeded its `timeout_sec` |
| `CANCELED` | The run was canceled while this step was pending/running |
| `SKIPPED` | Unreachable because a branching node (`if`) chose the other output port |
Each step also records `started_at`, `finished_at`, `execution_time`, `stdout`/`stderr`, and, on failure, `error_type`/`error_message`/`traceback` — everything you need to debug a failing node without re-running the whole flow.
Nodes inside a `for_each` body re-execute once per item. Rather than one step per iteration, the step for a body node is overwritten on each pass (last-iteration-wins), with `iteration`/`iterations_total` reporting progress.
## Canceling a Run
A running flow can be canceled while it's in progress. In-flight and not-yet-started steps are marked `CANCELED`, and the run transitions to `CANCELED` once cancellation completes.
## Validation Before a Run
Before a run starts, the flow document is validated as a whole. The most common issues you'll see surfaced per-node:
* A **required input port** must be connected (e.g. `response` and `output` always require their `in` port to be wired).
* Connections leaving a **branching** node (`if`) must set `from_port` to a valid output port (`true`/`false`).
* A flow may contain **at most one** top-level `response` node.
* A `for_each` node's `body` must contain **exactly one** `response` node.
* The flow (and every `for_each` body) must be a **DAG** — cycles are rejected.
* `for_each` nesting is capped at a maximum depth of 5.
See the [Node Reference](/reference/flows/nodes) for the full set of per-node-type validation rules.
## Next Steps
* [Overview](/reference/flows/overview) - flow structure and core concepts
* [Node Reference](/reference/flows/nodes) - every node type and its config
* [YAML Reference](/reference/flows/yaml-reference) - the complete flow document schema
# YAML Reference
Source: https://docs.datazone.co/reference/flows/yaml-reference
Comprehensive reference for every attribute in a Flow document.
# Flow YAML Reference
This page documents every attribute available in a Flow document — the envelope the Flow Builder posts, whether you edit on the canvas or in the YAML view.
## Top-Level Structure
```yaml theme={null}
flow: # Flow metadata (required)
name: string
runtime: # (Optional) flow-level runtime defaults
nodes: # List of node definitions (required)
- id: string
type: string
label: string
config: {}
runtime: {} # (Optional) per-node runtime override
connections: # List of directed edges between nodes (optional)
- from: string
to: string
from_port: string
to_port: string
```
A minimal flow needs a `flow.name`, at least a `start` node, and at least one `response` or `output` node:
```yaml theme={null}
flow:
name: "Minimal Flow"
nodes:
- id: trigger
type: start
config: {}
- id: finish
type: response
config: {}
connections:
- from: trigger
to: finish
```
***
## `flow`
| Attribute | Type | Description |
| --------- | ------ | ---------------------------------------------------------------------------------------------- |
| `name` | string | Flow display name (required) |
| `runtime` | object | (Optional) Default [runtime override](#runtime) applied to every node that doesn't set its own |
***
## `nodes`
The list of steps that make up the flow. Every node's `config` is validated against its `type`'s schema on deploy/run — an unknown type or an invalid config fails validation with a structured, per-node issue.
| Attribute | Type | Default | Description |
| --------- | ------ | ------- | ---------------------------------------------------------------------------------------- |
| `id` | string | — | Unique identifier for the node within the flow (required) |
| `type` | string | — | Node type — see the [Node Reference](/reference/flows/nodes) for all 10 types (required) |
| `label` | string | `null` | (Optional) Display name shown on the canvas |
| `config` | object | `{}` | Type-specific configuration |
| `runtime` | object | `null` | (Optional) [Runtime override](#runtime) for this node only |
### `runtime`
An optional block — settable at the flow level (as a default) or per-node (as an override) — controlling how a node's compiled code executes:
| Attribute | Type | Default | Description |
| ----------------- | ------ | ------- | --------------------------------------------------------------------- |
| `mode` | string | `auto` | One of `auto`, `subprocess`, `pod` |
| `engine` | string | `null` | (Optional) Execution engine override |
| `timeout_sec` | int | `null` | (Optional) Timeout in seconds, overriding the node type's own default |
| `memory_mb` | int | `null` | (Optional) Memory limit in MB |
| `max_input_rows` | int | `null` | (Optional) Cap on input row count |
| `max_input_bytes` | int | `null` | (Optional) Cap on input payload size in bytes |
```yaml theme={null}
- id: heavy_call
type: rest_call
config:
url: "https://api.example.com/report"
runtime:
timeout_sec: 120
memory_mb: 512
```
***
## `connections`
Directed edges between nodes. Each entry wires one node's output port to another node's input port.
| Attribute | Type | Default | Description |
| ----------- | ------ | ------------------- | ------------------------------------------------------------------ |
| `from` | string | — | Id of the upstream node (required) |
| `to` | string | — | Id of the downstream node (required) |
| `from_port` | string | node's first output | (Optional) Which output port of `from` this connection leaves from |
| `to_port` | string | node's first input | (Optional) Which input port of `to` this connection arrives at |
```yaml theme={null}
connections:
- from: fetch_weather
to: process
from_port: response
to_port: in
```
### Branching connections
Outgoing connections from a **branching** node (currently only `if`) must explicitly set `from_port` to one of that node's declared output ports (`true`/`false`) — an omitted or invalid `from_port` fails validation:
```yaml theme={null}
connections:
- from: check_status
to: send_email
from_port: "true"
- from: check_status
to: log_rejection
from_port: "false"
```
### Required inputs
Some node types declare `required_inputs` — input ports that must have an incoming connection for the flow to be valid. `response` and `output` both require their `in` port to be connected.
***
## Node types at a glance
Full attribute tables for every type live in the [Node Reference](/reference/flows/nodes). Quick summary:
| Type | Category | Inputs | Outputs | Notes |
| ------------- | ----------- | --------------- | --------------- | --------------------------------------------------------------------------- |
| `start` | trigger | — | `out` | Entry point; no config |
| `if` | control | `in` | `true`, `false` | **Branching** — `from_port` required on outgoing connections |
| `for_each` | control | `in` | `out` | **Container** — owns a nested `body` [`FlowDocument`](#top-level-structure) |
| `llm_call` | ai | `in` | `out` | Needs credentials; calls a Model Account |
| `rest_call` | integration | `in` | `response` | Calls an external HTTP endpoint |
| `action_call` | integration | `in` | `out` | Needs credentials; calls a project Action |
| `python` | transform | `in` | `out` | Runs a `handler(inputs, parameters)` function |
| `sql` | data | `in` | `out` | Needs credentials; runs a `SELECT` against project datasets |
| `response` | terminal | `in` (required) | — | At most one per flow (exactly one inside a `for_each` body) |
| `output` | sink | `in` (required) | — | Writes to a dataset, webhook, or file |
***
## `for_each.config.body`
The one node type whose `config` embeds a full nested document, following the exact same [top-level structure](#top-level-structure) as a flow:
```yaml theme={null}
- id: process_orders
type: for_each
config:
items: "inputs['in']['orders']"
item_as: order
save_as: results
body:
flow:
name: process_orders_body
nodes:
- id: body_start
type: start
config: {}
- id: format_order
type: python
config:
code: |
def handler(inputs, parameters):
return {"total": parameters["order"]["amount"] * 1.1}
- id: body_response
type: response
config: {}
connections:
- from: body_start
to: format_order
- from: format_order
to: body_response
```
* The body may itself contain a nested `for_each` (up to a maximum depth of 5).
* The body must contain **exactly one** `response` node — its output becomes that iteration's result.
***
## Full example
The "London Weather Demo" starter template — fetches current weather, transforms it, and returns a structured summary:
```yaml theme={null}
flow:
name: "London Weather Demo"
nodes:
- id: trigger
type: start
label: Start
config: {}
- id: fetch_weather
type: rest_call
label: Fetch London weather
config:
url: "https://api.open-meteo.com/v1/forecast"
method: GET
query:
latitude: "51.5074"
longitude: "-0.1278"
current_weather: "true"
timeout_sec: 30
save_as: weather
- id: process
type: python
label: Convert temperature
config:
code: |
def handler(inputs, parameters):
# inputs – dict of upstream node outputs, keyed by port name
# parameters – dict of flow run parameters
payload = inputs.get("in") or {}
weather = payload.get("weather", {})
current = weather.get("current_weather", {})
celsius = current.get("temperature")
if celsius is None:
raise ValueError("No temperature in weather response")
fahrenheit = round(celsius * 9 / 5 + 32, 1)
wind = current.get("windspeed", 0)
return {
"city": parameters.get("city", "London"),
"celsius": celsius,
"fahrenheit": fahrenheit,
"windspeed_kmh": wind,
"wind_category": "calm" if wind < 10 else "breezy" if wind < 25 else "windy",
"weather_code": current.get("weathercode"),
}
- id: answer
type: response
label: Return to caller
config: {}
connections:
- from: trigger
to: fetch_weather
to_port: in
- from: fetch_weather
to: process
to_port: in
- from: process
to: answer
to_port: in
```
## Next steps
* [Overview](/reference/flows/overview) - flow structure and core concepts
* [Node Reference](/reference/flows/nodes) - every node type, its config, and its ports
* [Triggers & Runs](/reference/flows/triggers-and-runs) - starting a flow and reading run status
* [Templating & Expressions](/reference/flows/templating) - how nodes reference upstream data and parameters
# API Key Authentication
Source: https://docs.datazone.co/reference/integration/authentication/api-key
Learn how to authenticate with Datazone using API keys
# API Key Authentication
Datazone provides secure API key-based authentication for accessing its APIs and services.
## Obtaining an API Key
1. Log into your Datazone account
2. Navigate to Settings > API Keys
3. Click "Generate New Key"
4. Save the key securely - it won't be shown again
## Using API Keys
Include the API key in your requests:
```bash theme={null}
curl -H "Authorization: Bearer YOUR_API_KEY" https://api.datazone.co/v1/datasets
```
## API Key Best Practices
* Keep keys secure and never expose them in client-side code
* Rotate keys regularly
* Use different keys for different environments
* Limit key permissions to only what's needed
## Role-Based Access Control
### Available Roles
* **admin**: Full access to all resources
* **read\_only**: Read-only access to resources
## Error Handling
### Common Authentication Errors
| Status Code | Description | Solution |
| ----------- | ------------------------ | -------------------------------- |
| 401 | Invalid API key | Verify key is correct and active |
| 403 | Insufficient permissions | Check required roles |
| 429 | Rate limit exceeded | Implement backoff strategy |
# Azure AD (Entra ID) SAML Setup
Source: https://docs.datazone.co/reference/integration/authentication/saml/azure-ad
Configure SAML authentication with Microsoft Azure Active Directory
## Prerequisites
* Administrator access to the [Azure Portal](https://portal.azure.com)
* Your Datazone instance domain (e.g., `app.datazone.co`)
## Configuration Steps
### 1. Access Azure Portal
Log into the [Azure Portal](https://portal.azure.com) with administrator privileges.
### 2. Create a New Enterprise Application
Navigate to **Enterprise applications** and click **New application**.
### 3. Create Custom Application
Select **Create your own application**, provide a name (e.g., "Datazone SSO"), and choose **Integrate any other application you don't find in the gallery (Non-gallery)**.
### 4. Configure SAML Sign-On
In the application overview, go to **Single sign-on** and select **SAML** as the sign-on method.
### 5. Configure Basic SAML Settings
In the **Basic SAML Configuration** section, click **Edit** and set the following parameters:
* **Identifier (Entity ID)**: `https://YOUR_DATAZONE_DOMAIN/api/v1/auth/saml/acs`
* **Reply URL (Assertion Consumer Service URL)**: `https://YOUR_DATAZONE_DOMAIN/api/v1/auth/saml/acs`
Replace `YOUR_DATAZONE_DOMAIN` with your actual Datazone instance domain.
Save the configuration.
### 6. Collect Azure AD Configuration Details
From the **SAML Signing Certificate** section, gather the following information:
* **Login URL**: Found in the SAML configuration
* **Azure AD Identifier**: Found in the SAML configuration
* **Certificate (Base64)**: Download the certificate
### 7. Configure User Attributes (Optional)
You can assign users or groups to the application in the **Users and groups** section. Ensure that the `email` attribute is mapped correctly for user identification.
### 8. Organization Routing by Attribute (Optional)
If you have a single Azure tenant serving users that belong to different organizations in Datazone, you can let Azure decide which organization a user is signed into by sending an extra claim in the SAML assertion.
When this claim is present, Datazone matches its value against your organizations:
* If a matching organization is found, the user is signed into that organization.
* If the user does not exist yet, the account is created directly inside that organization.
* If the claim is missing, login falls back to the default organization behavior.
To enable it, go to **Single sign-on** and click **Edit** on the **Attributes & Claims** card.
Then click **Add new claim** and define a claim whose value comes from a user property that already holds the organization name — for example the built-in **companyName** field:
| Claim name | Source attribute |
| -------------------------------------------------------------------- | ------------------ |
| `http://schemas.xmlsoap.org/ws/2005/05/identity/claims/company_name` | `user.companyname` |
You can use any other user attribute or a custom claim name instead — Datazone only needs to know which claim to read.
Then, in the Datazone SAML configuration, enter that same claim name so Datazone knows which attribute carries the organization:
The claim value must match the organization name in Datazone exactly. Make sure
the `companyName` field is populated for every user who should be routed this way,
otherwise their login falls back to the default organization.
### 9. Enter Configuration in Datazone
Finally, enter the collected information into the Datazone SAML configuration settings as described in the [SAML Overview](overview) guide:
* **Entity ID (Issuer)**: Use the **Azure AD Identifier** from Azure
* **SSO URL**: Use the **Login URL** from Azure
* **Certificate**: Upload the downloaded **Certificate (Base64)**
## Testing Your Configuration
After completing the setup:
1. Navigate to your Datazone login page
2. Click on the SSO/SAML login option
3. You should be redirected to Microsoft for authentication
4. Upon successful authentication, you'll be redirected back to Datazone
If you encounter any issues during setup, refer to the [SAML troubleshooting
guide](overview#troubleshooting) or contact support at [support@datazone.co](mailto:support@datazone.co)
# Google Workspace SAML Setup
Source: https://docs.datazone.co/reference/integration/authentication/saml/google-workspace
Configure SAML authentication with Google Workspace
## Prerequisites
* Administrator access to your Google Workspace account
* Your Datazone instance domain (e.g., `app.datazone.co`)
## Configuration Steps
### 1. Access Web and Mobile Apps
In your [Google Admin Console](https://admin.google.com/ac/apps/unified), navigate to **Apps** > **Web and mobile apps** from the left sidebar menu.
### 2. Add Custom SAML App
Click **Add app** and select **Add custom SAML app** from the dropdown menu.
### 3. Configure App Details
Provide a name for the application (e.g., "Datazone SSO"). You can optionally add an icon for the application.
### 4. Collect Google Identity Provider Details
Google will display the following configuration details that you'll need for Datazone:
* **SSO URL**: Identity Provider Single Sign-On URL
* **Entity ID**: Identity Provider Issuer
* **Certificate**: Download the certificate (PEM format)
Make sure to save these details securely. You'll need them to configure SAML
in Datazone.
### 5. Configure Service Provider Details
In the **Service provider details** section, set the following parameters:
* **ACS URL**: `https://YOUR_DATAZONE_DOMAIN/api/v1/auth/saml/acs`
* **Entity ID**: `https://YOUR_DATAZONE_DOMAIN/api/v1/auth/saml/acs`
Leave the rest of the form with default values.
Replace `YOUR_DATAZONE_DOMAIN` with your actual Datazone instance domain.
### 6. Add Attribute Mapping
In the **Attribute mapping** section, add the following mapping:
| Google Directory attributes | App attributes |
| --------------------------- | -------------- |
| Primary email | `email` |
### 7. Configure User Access
After the app is created, you'll see the **User access** section on the main app page. Configure which users or organizational units should have access to Datazone through Google SSO.
### 8. Assign Users to the App
Assign the appropriate users or groups to the application to control who can access Datazone through Google Workspace SSO.
### 9. Enter Configuration in Datazone
Finally, enter the collected information from Step 4 into the Datazone SAML configuration settings as described in the [SAML Overview](overview) guide:
* **Entity ID (Issuer)**: Use the Entity ID from Google
* **SSO URL**: Use the SSO URL from Google
* **Certificate**: Upload the downloaded certificate (PEM format)
## Testing Your Configuration
After completing the setup:
1. Navigate to your Datazone login page
2. Click on the SSO/SAML login option
3. You should be redirected to Google for authentication
4. Upon successful authentication, you'll be redirected back to Datazone
## Additional Resources
For more information about configuring custom SAML applications in Google Workspace, refer to [Google's official documentation](https://knowledge.workspace.google.com/admin/apps/set-up-your-own-custom-saml-app).
If you encounter any issues during setup, refer to the [SAML troubleshooting
guide](overview#troubleshooting) or contact support at [support@datazone.co](mailto:support@datazone.co)
# Okta SAML Setup
Source: https://docs.datazone.co/reference/integration/authentication/saml/okta
Configure SAML authentication with Okta
## Prerequisites
* Administrator access to your Okta account
* Your Datazone instance domain (e.g., `app.datazone.co`)
## Configuration Steps
### 1. Create a New SAML Application in Okta
In your Okta Admin Console, create a new SAML 2.0 application integration for Datazone.
### 2. Choose SAML 2.0 as Sign-On Method
When prompted to select a sign-on method, choose **SAML 2.0**.
### 3. Configure General Settings
In the **General Settings** section, provide a name for the application (e.g., "Datazone SSO").
### 4. Configure SAML Settings
In the **Configure SAML** section, set the following parameters:
* **Single sign-on URL**: `https://YOUR_DATAZONE_DOMAIN/api/v1/auth/saml/acs`
* **Audience URI (SP Entity ID)**: `https://YOUR_DATAZONE_DOMAIN/api/v1/auth/saml/acs`
* **Application username**: `Email`
Replace `YOUR_DATAZONE_DOMAIN` with your actual Datazone instance domain.
### 5. Add Attribute Statements
Once the application is created, navigate to the **Sign On** tab. Under **Attribute Statements**, add the following attribute mapping:
| Name | Value |
| ------- | ------------ |
| `email` | `user.email` |
### 6. Assign Users or Groups (Optional)
Assign the appropriate users or groups to the application to control who can access Datazone through Okta.
### 7. Collect Okta Configuration Details
After configuring the application, gather the following information from the **Sign On** tab:
* **Identity Provider Single Sign-On URL** (SSO URL)
* **Identity Provider Issuer** (Entity ID)
* **X.509 Certificate** (Download the certificate)
### 8. Enter Configuration in Datazone
Finally, enter the collected information into the Datazone SAML configuration settings as described in the [SAML Overview](overview) guide:
* **Entity ID (Issuer)**: Use the Identity Provider Issuer from Okta
* **SSO URL**: Use the Identity Provider Single Sign-On URL from Okta
* **Certificate**: Upload the downloaded X.509 certificate
## Testing Your Configuration
After completing the setup:
1. Navigate to your Datazone login page
2. Click on the SSO/SAML login option
3. You should be redirected to Okta for authentication
4. Upon successful authentication, you'll be redirected back to Datazone
If you encounter any issues during setup, refer to the [SAML troubleshooting
guide](overview#troubleshooting) or contact support at [support@datazone.co](mailto:support@datazone.co)
# SAML Authentication
Source: https://docs.datazone.co/reference/integration/authentication/saml/overview
Configure SAML-based single sign-on for your Datazone instance
# SAML Authentication
Datazone supports SAML 2.0-based single sign-on (SSO), allowing you to integrate with your organization's identity provider for centralized authentication.
## Overview
SAML (Security Assertion Markup Language) enables secure authentication between your SAML identity provider (IdP) and Datazone. This allows your users to access Datazone using their existing corporate credentials.
## Configuration
To configure SAML authentication in Datazone, you need to provide the following information from your SAML identity provider:
### Required Fields
You can find these values in your identity provider's SAML configuration
settings. Refer to our provider-specific guides below for detailed
instructions.
#### Entity ID (Issuer)
The unique identifier for your SAML identity provider (IdP). This is typically provided by your IdP configuration and ensures that SAML assertions are validated against the correct identity provider.
**Example:** `https://accounts.google.com/o/saml2?idpid=ABC123`
#### SSO URL (Sign-On Endpoint)
The URL where authentication requests should be sent to your IdP. When users attempt to log in to Datazone, they will be redirected to this endpoint for authentication.
**Example:** `https://sso.yourcompany.com/saml/login`
#### Certificate
Upload the X.509 certificate from your SAML identity provider. This certificate is used to verify the authenticity of SAML assertions and ensure secure communication between Datazone and your IdP.
Ensure your certificate is in PEM format and includes the full certificate
chain if required by your identity provider.
## Setup Steps
1. **Access SAML Settings**: Navigate to Settings > Authentication > SAML in your Datazone dashboard
2. **Enter Configuration Details**: Fill in the Entity ID, SSO URL, and upload your certificate
3. **Save Configuration**: Click "Update" to apply your SAML settings
## Provider-Specific Guides
For detailed setup instructions with specific identity providers, refer to our provider guides:
Configure SAML with Microsoft Azure Active Directory
Set up SAML authentication with Okta
Configure SAML with Google Workspace
## Troubleshooting
### Common Issues
| Issue | Possible Cause | Solution |
| --------------------- | ------------------- | ------------------------------------------------- |
| Authentication fails | Invalid certificate | Verify certificate is current and in PEM format |
| Redirect loops | Incorrect SSO URL | Double-check the SSO URL in your IdP settings |
| Users not provisioned | Missing attributes | Ensure email attribute is mapped in SAML response |
Need help with SAML configuration? Contact our support team at
[support@datazone.co](mailto:support@datazone.co)
# Endpoints
Source: https://docs.datazone.co/reference/integration/endpoints
Create custom API endpoints for secure data access
## Overview
Endpoints allow you to create **custom API interfaces** with **secure, controlled access**. You can define **three types of endpoints**:
1. **Query Endpoints** - Execute **SQL queries** on your datasets with **dynamic filters**
2. **Action Endpoints** - Execute **serverless Python functions** for custom logic
3. **Vector Endpoints** - Perform **semantic similarity search** on vectorized data
## Creating Endpoints
1. Navigate to your **Project**
2. Create a new **YAML file** for your endpoint (e.g., `api-orders.yaml`)
3. Define your endpoint configuration
4. Reference it in your **`config.yaml`** file
## Endpoint Types
Each endpoint file defines **exactly one endpoint** under a top-level `endpoint:` key. Use a separate YAML file for each endpoint and register each file individually in `config.yaml`.
### Query-Based Endpoints
Query endpoints execute **SQL queries** on your data with **dynamic filtering** using Jinja templating.
**Example YAML Configuration:**
```yaml theme={null}
endpoint:
name: consolidated-sales-endpoint
type: query
config:
filters:
- name: country
type: string
optional: true
- name: city
type: string
optional: true
query: |
SELECT *
FROM consolidated_sales_df_299ceb
WHERE 1 = 1
{% if country is defined %}
AND CustomerCountry = '{{ country }}'
{% endif %}
{% if city is defined %}
AND CustomerCity = '{{ city }}'
{% endif %}
```
### Action-Based Endpoints
Action endpoints **execute serverless Python functions** when called. Perfect for sending notifications, processing data, calling external APIs, or automating workflows.
**Example YAML Configuration:**
```yaml theme={null}
endpoint:
name: process-data-endpoint
type: action
config:
action_id: "507f1f77bcf86cd799439011"
```
The `action_id` references an action function in your project. Get the ID from your action details page.
The action function must **return a list**. Returning any other type raises `ActionEndpointResultMustBeListError`.
### Vector-Based Endpoints
Vector endpoints enable **semantic similarity search** on your vectorized data via HTTP API. Perfect for building search features, recommendation systems, or RAG applications.
**Example YAML Configuration:**
```yaml theme={null}
endpoint:
name: document-search-endpoint
type: vector
config:
vector_id: "69d597a6e9713d78370a50d9"
```
The `vector_id` references a [Vector](/reference/development/vectors) in your project. Get the ID from your vector details page.
### Register in config.yaml
Reference your endpoint file in **`config.yaml`**:
```yaml theme={null}
project_name: TestProject
project_id: 688b3d0b7c5f93f0763a028c
endpoints:
- path: api-orders.yaml
```
Learn more about the configuration file in the [Project Configuration](/reference/development/project#configuration-file) section.
## Configuration Reference
### Common Attributes
| Attribute | Type | Required | Description |
| --------- | ------ | -------- | ------------------------------------------------- |
| `name` | string | Yes | **Unique endpoint identifier** |
| `type` | string | Yes | **Endpoint type**: `query`, `action`, or `vector` |
| `config` | object | Yes | **Type-specific configuration** |
### Query Config Attributes
| Attribute | Type | Required | Description |
| --------- | ------ | -------- | ----------------------------------- |
| `query` | string | Yes | **SQL query** with Jinja templating |
| `filters` | array | No | **List of filter parameters** |
### Action Config Attributes
| Attribute | Type | Required | Description |
| ----------- | ------ | -------- | --------------------------------- |
| `action_id` | string | Yes | **Action function ID** to execute |
### Vector Config Attributes
| Attribute | Type | Required | Description |
| ----------- | ------ | -------- | ----------------------------------- |
| `vector_id` | string | Yes | **Vector ID** for similarity search |
### Filter Configuration
Filters are **only for query endpoints** and define dynamic parameters:
| Attribute | Type | Required | Description |
| ---------- | ------- | -------- | ------------------------------------------------ |
| `name` | string | Yes | **Filter parameter name** (alphanumeric, -, \_) |
| `type` | string | Yes | **Data type** (see types below) |
| `optional` | boolean | No | Whether filter is **optional** (default: `true`) |
### Filter Types
| Type | Description | Example Usage |
| ---------- | ------------------------- | -------------------------- |
| `string` | **Text parameter** | `country`, `category` |
| `integer` | **Numeric parameter** | `year`, `limit` |
| `float` | **Decimal parameter** | `price`, `rating` |
| `date` | **Date parameter** | `start_date`, `end_date` |
| `datetime` | **DateTime parameter** | `created_at`, `updated_at` |
| `boolean` | **Boolean parameter** | `active`, `is_featured` |
| `enum` | **Enumeration parameter** | `status`, `priority` |
## Using Endpoints
Endpoints are accessible via **HTTP requests**. Query and vector endpoints use **GET requests**, while action endpoints may vary based on implementation:
```bash cURL theme={null}
curl -X GET \
"https://app.datazone.co/api/v1/endpoint/your-endpoint-name?param1=value1¶m2=value2" \
-H "x-api-key: " \
-H "Content-Type: application/json"
```
```python Python theme={null}
import requests
url = "https://app.datazone.co/api/v1/endpoint/your-endpoint-name"
headers = {
"x-api-key": "",
"Content-Type": "application/json"
}
params = {
"param1": "value1",
"param2": "value2"
}
response = requests.get(url, headers=headers, params=params)
data = response.json()
print(data)
```
```javascript JavaScript theme={null}
const url = 'https://app.datazone.co/api/v1/endpoint/your-endpoint-name';
const params = new URLSearchParams({
param1: 'value1',
param2: 'value2'
});
const response = await fetch(`${url}?${params}`, {
method: 'GET',
headers: {
'x-api-key': '',
'Content-Type': 'application/json'
}
});
const data = await response.json();
console.log(data);
```
```java Java theme={null}
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
HttpClient client = HttpClient.newHttpClient();
String url = "https://app.datazone.co/api/v1/endpoint/your-endpoint-name?param1=value1¶m2=value2";
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.header("x-api-key", "")
.header("Content-Type", "application/json")
.GET()
.build();
HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.body());
```
## Response Format
### Query Endpoint Response
Query endpoints return **JSON data** with query results:
```json theme={null}
{
"records": [
{
"column1": "value1",
"column2": "value2"
}
],
"metadata": {
"total_records": 100,
"query_time_ms": 250,
"endpoint_name": "consolidated-sales-endpoint"
}
}
```
### Action Endpoint Response
Action endpoints return the **list** your Python function returns:
```json theme={null}
[
{
"status": "success",
"message": "Action executed successfully"
}
]
```
Your action function **must return a list**. Returning any other type (dict, string, `None`, etc.) raises `ActionEndpointResultMustBeListError`. Always wrap your result in a list, even when there is only one item.
```python theme={null}
# ✅ Correct — return a list
def run(context):
return [{"status": "success", "message": "Done"}]
# ❌ Wrong — raises ActionEndpointResultMustBeListError
def run(context):
return {"status": "success", "message": "Done"}
```
### Vector Endpoint Response
Vector endpoints return **semantically similar results** based on your search query:
```json theme={null}
{
"results": [
{
"content": "Relevant text chunk from your data...",
"metadata": {
"source": "document.pdf",
"page": 5,
"chunk_id": "abc123"
},
"score": 0.92
},
{
"content": "Another relevant text chunk...",
"metadata": {
"source": "report.docx",
"section": "Introduction"
},
"score": 0.87
}
],
"metadata": {
"vector_id": "69d597a6e9713d78370a50d9",
"endpoint_name": "document-search-endpoint",
"query": "what is the company revenue?",
"total_results": 10
}
}
```
**Using Vector Endpoints:**
```bash cURL theme={null}
curl -X GET \
"https://app.datazone.co/api/v1/endpoint/document-search-endpoint?query=what+is+the+company+revenue" \
-H "x-api-key: " \
-H "Content-Type: application/json"
```
```python Python theme={null}
import requests
url = "https://app.datazone.co/api/v1/endpoint/document-search-endpoint"
headers = {
"x-api-key": "",
"Content-Type": "application/json"
}
params = {
"query": "what is the company revenue?"
}
response = requests.get(url, headers=headers, params=params)
results = response.json()
for result in results["results"]:
print(f"Score: {result['score']}")
print(f"Content: {result['content']}")
print(f"Metadata: {result['metadata']}\n")
```
**Query Parameters for Vector Endpoints:**
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | ------------------------------------ |
| `query` | string | Yes | **Search query** in natural language |
### Authentication
Endpoints use API key authentication via the `x-api-key` header:
```bash theme={null}
-H "x-api-key: YOUR_API_KEY"
```
### Error Responses
| Status Code | Description |
| ----------- | ---------------------------------- |
| 200 | Success |
| 400 | Bad Request - Invalid parameters |
| 401 | Unauthorized - Invalid API key |
| 403 | Forbidden - Access denied |
| 404 | Not Found - Endpoint doesn't exist |
| 500 | Internal Server Error |
Example error response:
```json theme={null}
{
"error": "Invalid parameter 'country': must be a string",
"code": "INVALID_PARAMETER",
"status": 400
}
```
### Advanced Query Features
#### Jinja Templating
Endpoints support Jinja templating for dynamic queries:
```yaml theme={null}
query: |
SELECT *
FROM sales_data
WHERE 1 = 1
{% if start_date is defined %}
AND order_date >= '{{ start_date }}'
{% endif %}
{% if end_date is defined %}
AND order_date <= '{{ end_date }}'
{% endif %}
{% if categories is defined %}
AND category IN ({{ categories | join("','") | surround("'") }})
{% endif %}
```
### Caching
Endpoint responses can be cached:
* Default cache TTL: 60 minutes
* Cache headers indicate freshness
* Use cache-busting parameters when needed
# ODBC/JDBC Connections
Source: https://docs.datazone.co/reference/integration/odbc-jdbc-connection
Connect to Datazone using Clickhouse ODBC or JDBC drivers
# ODBC/JDBC Connections
Datazone uses Clickhouse as its SQL interface, allowing you to connect to your data using standard Clickhouse ODBC and JDBC drivers.
## ODBC Connection
### System Requirements
* Windows, Linux, or macOS
* Clickhouse ODBC Driver installed
* Network access to Datazone instance
### Installing Clickhouse ODBC Driver
#### Windows
1. Download the Clickhouse ODBC driver from the official website
2. Run the installer
3. Configure the ODBC data source in Windows ODBC Data Source Administrator
#### Linux
```bash theme={null}
# Ubuntu/Debian
apt-get install clickhouse-odbc
# CentOS/RHEL
yum install clickhouse-odbc
```
#### macOS
```bash theme={null}
brew install clickhouse-odbc
```
### Connection String
```
Driver={Clickhouse ODBC Driver};Server=app.datazone.co;Port=8123;Database=default;UID=your_username;PWD=your_password;Protocol=http;
```
### Configuration Parameters
| Parameter | Required | Description |
| --------- | -------- | --------------------------------------- |
| Server | Yes | Your Datazone instance hostname |
| Port | Yes | Connection port (default: 8123) |
| Database | Yes | Target database name (default: default) |
| Protocol | Yes | Connection protocol (http/https) |
| UID | Yes | Username |
| PWD | Yes | Password |
## JDBC Connection
### Requirements
* Java 8 or later
* Clickhouse JDBC driver
* Network access to Datazone instance
### Maven Dependency
```xml theme={null}
com.clickhouseclickhouse-jdbc0.3.2
```
### Connection URL
```
jdbc:clickhouse://app.datazone.co:8123/default
```
### Java Example
```java theme={null}
String url = "jdbc:clickhouse://app.datazone.co:8123/default";
Properties props = new Properties();
props.setProperty("user", "your_username");
props.setProperty("password", "your_password");
Connection conn = DriverManager.getConnection(url, props);
```
## Common Use Cases
### BI Tool Integration
1. **Tableau**
```
Connection Type: Other Databases (ODBC)
Driver: Clickhouse ODBC Driver
```
2. **Power BI**
```
Data Source: ODBC
Driver: Clickhouse ODBC Driver
```
3. **Looker**
```
Dialect: Clickhouse
Connection: JDBC
```
### DBeaver Configuration
1. Create new connection
2. Select Clickhouse database
3. Enter connection details:
```
JDBC URL: jdbc:clickhouse://app.datazone.co:8123/default
Username: your_username
Password: your_password
```
## Troubleshooting
### Common Issues and Solutions
1. **Connection Timeout**
```
Error: Connection timeout
Solutions:
- Check network connectivity
- Verify port is open (8123)
- Check firewall rules
```
2. **Authentication Failed**
```
Error: Authentication failed
Solutions:
- Verify credentials
- Check user permissions
- Ensure proper protocol (http/https)
```
3. **Query Performance**
```
Issue: Slow queries
Solutions:
- Check query execution plan
- Review data types
- Use appropriate indexes
```
### Logging
Enable detailed logging in your connection string:
```
jdbc:clickhouse://app.datazone.co:8123/default?log_level=TRACE
```
## Next Steps
1. Install appropriate drivers for your system
2. Configure your connection using the provided parameters
3. Test the connection with a simple query
4. Integrate with your BI tools or applications
# Overview
Source: https://docs.datazone.co/reference/integration/overview
Overview of Datazone Integration and API capabilities
# Integration & APIs Overview
Datazone provides multiple ways to integrate with your existing data infrastructure and applications. This section covers the various integration methods and APIs available to interact with Datazone.
## Available Integration Methods
### 1. REST API
The Datazone REST API provides programmatic access to all platform functionality, allowing you to:
* Manage data sources and extracts
* Control pipeline executions
* Access metadata and lineage information
* Retrieve data from datasets
* Manage users and permissions
### 2. ODBC/JDBC Connections
Direct database-style connections to Datazone enable:
* Query data using standard SQL
* Integration with BI tools
* Real-time data access
### 3. Authentication
Secure access to Datazone APIs through:
* API key authentication
* Token-based authentication
* Role-based access control
## Getting Started
1. Review the authentication documentation to set up secure access
2. Choose the appropriate integration method for your use case
3. Follow the specific guides for your chosen integration method
4. Test your integration using the provided examples
## Integration Use Cases
### Data Analysis Tools
* Connect Tableau, Power BI, or other BI tools
* Use ODBC/JDBC for direct SQL access
* Leverage REST API for custom analytics applications
### Automation
* Schedule data extractions via API
* Automate pipeline operations
* Integrate with workflow tools
### Custom Applications
* Build custom data applications
* Embed Datazone functionality
* Create specialized data interfaces
## Next Steps
* Review the authentication documentation to get started
* Explore the REST API reference
* Set up your first ODBC/JDBC connection
# Views
Source: https://docs.datazone.co/reference/integration/views
Create optimized relational database views from your datasets with advanced partitioning and indexing
## Overview
Views enable you to transform your **datasets into optimized relational database structures** with advanced configurations like **partitioning, primary keys, and ordering**. This feature bridges the gap between Datazone's lakehouse architecture and traditional relational database capabilities, providing enhanced performance and flexibility for your data access patterns.
With Views, you can replicate datasets or create custom aggregations using SQL queries across multiple datasets and even other views. The resulting views are accessible through Datazone's SQL interface, endpoints, and intelligent apps.
## View Types
* **Materialized View**: Stores the query results in the database, so **queries are very fast** and consistent until refreshed. Best for dashboards and reports you check often.
* **Non-Materialized View**: Runs the query every time you access it, always showing the latest data. **Good for real-time needs or when you want to save storage.**
If you create non-materialized views, keep in mind that query performance may vary based on the complexity of the underlying SQL and the size of the datasets involved. For frequently accessed views, consider using materialized views to enhance performance.
## Let's Create a View
With the intuitive interface, you can easily create and manage views to suit your data needs.
1. Go to the **Views** page by clicking on the **Views** tab in the sidebar.
2. Click on the **Create** button.
3. Choose between creating a **Materialized** or **Non-Materialized** view.
4. Select whether it's a **replicate** of a dataset or a **custom SQL** query.
5. Fill the **optimization customization options** as needed.
6. Check the preview and create the view. 🚀
## Why I Should Use Views
Let's say you have a large dataset that is frequently queried for reporting purposes. By creating a materialized view with appropriate partitioning and primary keys, you can significantly speed up query performance, reduce load times, and improve the overall user experience for your data consumers.
In the following scenario, you have a sales dataset that is regularly queried to generate monthly sales reports.
By creating a materialized view that aggregates sales data by category or day of week, you can optimize the performance of these queries, making report generation faster and more efficient.
## Next Steps
Learn how to query views using SQL
Expose views through secure APIs
Build dashboards using views
Understand source datasets
# Components
Source: https://docs.datazone.co/reference/intelligent-apps/components
Detailed reference for components in Datazone Intelligent Apps
## Common Properties
All chart types share these properties:
| Property | Type | Description |
| -------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------- |
| `type` | string | Chart type: `number`, `line`, `bar`, `pie`, `radial`, `table`, `data_table`, `item_list`, `composed`, `heatmap`, or `scatter` |
| `name` | string | Unique identifier for the chart |
| `title` | string | Display title |
| `description` | string | Optional description |
| `query` | string | SQL query that provides chart data |
| `chart_config` | object | (Optional) Chart configuration options |
### Chart Configuration
The `chart_config` object allows you to customize chart appearance and behavior:
| Property | Type | Description |
| ------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `show_legend` | boolean | (Optional) Whether to show the chart legend (default: false) |
| `show_labels` | boolean | (Optional) Whether to show labels on the chart (default: false) |
| `fill_donut` | boolean | (Optional, pie charts only) Whether to fill the donut chart (default: false) |
| `is_stacked` | boolean | (Optional, bar/pie charts) Whether to stack elements (default: false) |
| `line_type` | string | (Optional, line/composed charts) Line style: `linear`, `monotone`, `step`, `natural` |
| `fill_area` | boolean | (Optional, line charts only) Whether to fill the area under the line |
| `custom_chart_type` | string | (Optional, custom charts only) Custom chart type identifier |
| `sub_expression` | string | (Optional, number/radial charts) Sub-expression for metric calculation |
| `page_size` | integer | (Optional, data\_table charts) Number of rows per page |
| `layout` | string | (Optional, bar charts) Layout orientation: `horizontal` or `vertical` |
| `hide_expression` | string | (Optional, all chart types) JavaScript expression to conditionally hide the chart based on data (e.g., `"(data) => {return data.length < 5}"`) |
### Metric Properties
Each entry in a chart's `metrics` list supports the following properties:
| Property | Type | Description |
| ---------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `name` | string | Column name from the query result that holds the metric value |
| `label` | string | Display label for the metric |
| `format` | string | (Optional) Number format string for the value |
| `icon` | string | (Optional, number/radial charts only) Lucide icon name |
| `icon_variant` | string | (Optional, number/radial charts only) One of: `default`, `neutral`, `success`, `warning`, `error` |
| `axis_name` | string | (Optional, line/bar/composed charts) Axis to assign this metric to |
| `composed_type` | string | (Optional, composed charts) Series type: `bar` or `line` |
| `submetric_name` | string | (Optional, number/radial charts only) Name of the submetric to display |
| `submetric_type` | string | (Optional, number/radial charts only) Submetric display: `plain`, `change`, `delta` |
| `color` | string | (Optional) Custom color for this metric's series, overriding the theme palette (e.g., a hex code like `#2563eb` or a chart color variable) |
| `show_label` | boolean | (Optional) Whether to show the data label for this specific metric, overriding the chart-level `show_labels` setting |
`show_label` is a per-metric override that controls labels for a single series, whereas `chart_config.show_labels` toggles labels for the whole chart. Use `color` to give an individual series a distinct color instead of the default theme palette.
## Number Charts
Number charts display a single metric value, often used for KPIs.
You can add an icon next to the number using any [Lucide icon](https://lucide.dev/icons/), and control its color with `icon_variant`.
Required properties for number metrics:
| Property | Type | Description |
| -------------- | ------ | ---------------------------------------------------------------------- |
| `icon` | string | (Optional) Lucide icon name (e.g., `check`, `star`) |
| `icon_variant` | string | (Optional) One of: `default`, `neutral`, `success`, `warning`, `error` |
For detailed information, check the [YAML reference](./yaml-reference#charts).
Datazone currently uses Lucide version 0.515.0. If you try to use an icon introduced in a newer version, it may cause an error or not display correctly. See the [Lucide icon gallery](https://lucide.dev/icons/) for available icons in 0.515.0.
```yaml theme={null}
- type: number
name: completed_tasks
title: "Completed Tasks"
query: "SELECT count(*) as completed FROM tasks WHERE status = 'done'"
metrics:
- name: completed
label: Completed Tasks
icon: check
icon_variant: success
format: "0"
```
## Composed Charts
Composed charts allow you to combine line and bar series in a single chart, each mapped to a specific axis. Use the `axis` property to define axes, and set `axis_name` and `composed_type` for each metric.
```yaml theme={null}
- type: composed
name: top_products_composed
title: Top Performing Products
description: Top 10 products by revenue
query: |
SELECT
ProductName as product,
SUM(OrderLineTotalAmount) as revenue,
AVG(OrderLineTotalAmount) as avg_revenue
FROM consolidated_sales_df_299ceb
WHERE 1=1
{% if region_filter is defined %}
AND CustomerRegion = '{{ region_filter }}'
{% endif %}
{% if product_group_filter is defined %}
AND ProductGroup = '{{ product_group_filter }}'
{% endif %}
GROUP BY ProductName
ORDER BY revenue DESC
LIMIT 10;
axis:
- name: left
- name: right
position: right
dimensions:
- name: product
label: Product
metrics:
- name: revenue
label: Revenue
axis_name: "left"
composed_type: "bar"
format: "$0,0.00"
- name: avg_revenue
label: "Average Revenue"
axis_name: "right"
composed_type: "line"
format: "0,0[.]00 $"
```
## Line Charts
Line charts visualize trends over time or continuous data. You can use the `axis` property to define multiple axes and assign metrics to them using `axis_name`. Use `fill_area` to control whether the area under the line is filled.
```yaml theme={null}
- type: line
name: sales_trend
title: "Sales Trend"
query: "SELECT date, sum(amount) as total, avg(amount) as avg FROM sales GROUP BY date ORDER BY date"
chart_config:
fill_area: true
axis:
- name: left
- name: right
position: right
dimensions:
- name: date
label: Date
metrics:
- name: total
label: Daily Sales
axis_name: left
- name: avg
label: Average Sales
axis_name: right
```
### Line Chart Configuration
| Property | Type | Description |
| ----------- | ------- | ------------------------------------------------------------------ |
| `fill_area` | boolean | (Optional) Whether to fill the area under the line (default: true) |
## Bar Charts
Bar charts compare values across categories. You can use the `affected_filter` attribute under a dimension to allow users to update a filter by clicking a bar.
### Standard Bar Charts
```yaml theme={null}
- type: bar
name: sales_by_country
title: Sales by Country
query: SELECT country, sum(amount) as total FROM sales GROUP BY country
dimensions:
- name: country
label: Country
affected_filter: country_filter
metrics:
- name: total
label: Total Sales
```
### Dynamic Charts with Chart Inputs
You can add **interactive controls** directly to charts using `chart_inputs`. This allows users to modify queries dynamically using **dropdown menus** without leaving the chart view.
Chart inputs work seamlessly with **Jinja templating** in your queries, enabling dynamic GROUP BY clauses, WHERE conditions, and more:
```yaml theme={null}
- type: bar
name: top_by_revenue
title: Top 10 by Revenue
chart_inputs:
- type: dropdown
name: group_by_column
label: Group By
default: Product
options:
- Product
- Customer
query: |
SELECT
{% if group_by_column == 'Product' %}
ProductName as group,
{% else %}
CustomerName as group,
{% endif %}
SUM(OrderLineTotalAmount) as revenue
FROM consolidated_sales_df
WHERE 1=1
{% if date_from is defined %}
AND OrderDate >= '{{ date_from }}'
{% endif %}
{% if group_by_column == 'Product' %}
GROUP BY ProductName
{% else %}
GROUP BY CustomerName
{% endif %}
ORDER BY revenue DESC
LIMIT 10
dimensions:
- name: group
label: Group
metrics:
- name: revenue
label: Revenue
format: "$0,0.00"
```
#### Chart Input Properties
| Property | Type | Description |
| --------------- | ------ | ----------------------------------------------------------------------------- |
| `type` | string | Input type (currently supports `dropdown`) |
| `name` | string | Variable name accessible in the query via Jinja |
| `label` | string | Display label shown in the UI |
| `default` | string | Default selected value |
| `options` | array | (Optional) List of static options for the dropdown |
| `options_query` | string | (Optional) SQL query to fetch dynamic options (cannot be used with `options`) |
Chart inputs are perfect for allowing users to **switch dimensions** (e.g., by Product vs. by Customer) or **change grouping levels** (e.g., daily vs. monthly) without creating multiple separate charts.
#### Dynamic Chart Input Options
Chart inputs can fetch their dropdown options dynamically from database queries, enabling data-driven dropdowns and cascading input scenarios. This is particularly useful when:
* **Filter-Dependent Options**: You want the available options to change based on active tab-level filters
* **Cascading Inputs**: One input's value should affect another input's available options
* **Data-Driven Dropdowns**: Options need to come from your database rather than being hard-coded
**Example: Cascading Inputs**
```yaml theme={null}
chart_inputs:
- type: dropdown
name: country
label: Select Country
default: USA
options:
- USA
- Canada
- UK
- type: dropdown
name: region
label: Select Region
options_query: "SELECT DISTINCT region FROM sales_data WHERE country = '{{ country }}' ORDER BY region"
```
In this example:
1. User selects a country from the first dropdown (static options)
2. The second dropdown automatically fetches regions for the selected country
3. The `options_query` has access to the `country` variable from the first input
**Example: Filter-Dependent Options**
```yaml theme={null}
chart_inputs:
- type: dropdown
name: product_category
label: Select Category
options_query: "SELECT DISTINCT category FROM products WHERE date >= '{{ start_date }}' ORDER BY category"
```
Here, the available categories change based on the tab-level `start_date` filter value.
You cannot define both `options` and `options_query` for the same chart input. The `options_query` can reference both tab-level filter variables and other chart input values using Jinja template syntax.
### Stacked Bar Charts
Stacked bar charts allow you to show multiple metrics stacked on top of each other for each category. Use the `chart_config.stacked` property to enable stacking:
```yaml theme={null}
- type: bar
name: monthly_order_performance
title: Monthly Order Performance
query: |
SELECT
month,
sum(orders) as orders,
sum(customers) as customers
FROM sales_data
GROUP BY month
ORDER BY month
chart_config:
is_stacked: true
show_legend: true
dimensions:
- name: month
label: Month
metrics:
- name: orders
label: Orders
- name: customers
label: Customers
```
### Chart Configuration for Bar Charts
| Property | Type | Description |
| ------------- | ------- | ------------------------------------------------------------- |
| `is_stacked` | boolean | (Optional) Whether to stack bars on top of each other |
| `show_legend` | boolean | (Optional) Whether to show the legend for multi-series charts |
| `show_labels` | boolean | (Optional) Whether to show labels on the bars |
### Vertical Layout Bar Charts
You can create vertical layout bar charts by setting `layout: vertical` in the chart configuration:
```yaml theme={null}
- type: bar
name: monthly_order_performance
title: Monthly Order Performance
query: |
SELECT
month,
sum(orders) as orders,
sum(customers) as customers
FROM sales_data
GROUP BY month
ORDER BY month
axis:
- name: top_axis
position: top
- name: bottom_axis
position: bottom
chart_config:
layout: vertical
show_legend: true
dimensions:
- name: month
label: Month
metrics:
- name: orders
label: Orders
axis_name: top_axis
- name: customers
label: Customers
axis_name: bottom_axis
```
## Pie Charts
Pie charts show part-to-whole relationships. You can use the `affected_filter` attribute under a dimension to allow users to update a filter by clicking a pie slice.
```yaml theme={null}
- type: pie
name: sales_distribution
title: Sales Distribution
query: SELECT category, sum(amount) as total FROM sales GROUP BY category
dimensions:
- name: category
label: Category
affected_filter: category_filter
metrics:
- name: total
label: Total Sales
```
### Donut Charts
You can create donut charts by setting `fill_donut: false` in the chart configuration:
```yaml theme={null}
- type: pie
name: revenue_by_region
title: Revenue by Region
query: SELECT region, sum(revenue) as total_revenue FROM sales GROUP BY region
chart_config:
fill_donut: false
show_labels: true
show_legend: false
dimensions:
- name: region
label: Region
metrics:
- name: total_revenue
label: Total Revenue
format: "$0,0.00"
```
## Radial Charts
Radial charts display progress or completion metrics in a circular format. They require exactly two metrics: the first metric represents the current value (displayed with a label), and the second metric represents the total or maximum value.
```yaml theme={null}
- type: radial
name: task_completion
title: "Task Completion"
description: "Current progress on project tasks"
query: "SELECT sum(case when status = 'completed' then 1 else 0 end) as completed_tasks, count(*) as total_tasks FROM tasks"
metrics:
- name: completed_tasks
label: "Completed Tasks"
format: "0,0"
- name: total_tasks
label: "Total Tasks"
format: "0,0"
```
## Table Charts
Table charts display raw data in tabular format.
```yaml theme={null}
- type: table
name: recent_orders
title: "Recent Orders"
description: "Last 100 orders placed"
query: "SELECT * FROM orders ORDER BY date DESC LIMIT 100"
```
## Data Table Charts
Data Table charts provide advanced tabular display with custom formatting and alignment for each column.
* **Sorting**: Users can click on column headers to sort the table by any dimension or metric, ascending or descending.
* **Pagination**: Large result sets are automatically split into pages, allowing users to navigate through data efficiently.
This makes `data_table` ideal for exploring large datasets interactively within your Intelligent App.
```yaml theme={null}
- type: data_table
name: detailed_orders
title: "Detailed Orders"
description: "All order details with formatting"
query: "SELECT order_id, amount, status FROM orders ORDER BY date DESC LIMIT 100"
dimensions:
- name: order_id
label: Order ID
table_align: left
- name: amount
label: Amount
number_format: "$0,0.00"
table_align: right
- name: status
label: Status
table_align: left
```
### Chart Config
Chart configuration options allow you to customize the appearance and behavior of your charts. Common properties include:
* `page_size`: (For `data_table`) Number of rows per page
```yaml theme={null}
chart_config:
page_size: 20
```
### Formatting
Use the `format` attribute under each metric to control how numbers are displayed:
| Format | Example | Result |
| ----------- | ------- | ------------ |
| `"0,0"` | 1234 | "1,234" |
| `"$0,0.00"` | 1234.5 | "\$1,234.50" |
| `"0.0%"` | 0.123 | "12.3%" |
| `"0.00a"` | 1234 | "1.23k" |
### Dimensions
| Property | Type | Description |
| --------------- | ------ | --------------------------------------------- |
| `name` | string | Unique identifier for the dimension |
| `label` | string | Display label for the dimension |
| `number_format` | string | (Optional) Number format for this dimension |
| `table_align` | string | (Optional) Table alignment: `left` or `right` |
## Heatmap Charts
Heatmap charts visualize data where individual values are represented as colors. They require exactly two dimensions and one metric. Heatmaps are ideal for showing relationships between two categorical dimensions with a numeric value determining the color intensity.
```yaml theme={null}
- type: heatmap
name: sales_by_region_product
title: "Sales by Region and Product"
description: "Sales distribution across regions and products"
query: |
SELECT
region,
product_category,
SUM(sales_amount) as total_sales
FROM sales_data
GROUP BY region, product_category
ORDER BY region, product_category
chart_config:
show_legend: true
dimensions:
- name: region
label: Region
- name: product_category
label: Product Category
metrics:
- name: total_sales
label: Total Sales
format: "$0,0.00"
```
## Scatter Plot Charts
Scatter plot charts visualize the relationship between two numerical variables. Each point represents a data record, positioned according to the values of two metrics on the x and y axes. Scatter plots are ideal for identifying correlations, patterns, or outliers in your data.
```yaml theme={null}
- type: scatter
name: example_scatter_chart
title: Sales Profiles by Countries
query: |
SELECT
CustomerCountry,
ROUND(AVG(UnitPrice), 2) as avg_unit_price,
ROUND(SUM(InvoiceTotalAmount), 2) as total_invoice_count
FROM consolidated_sales_df_9cb4a3
GROUP BY CustomerCountry
ORDER BY total_invoice_count DESC
LIMIT 20;
dimensions:
- name: CustomerCountry
label: "Country"
metrics:
- name: avg_unit_price
label: Average Unit Price
- name: total_invoice_count
label: Total Invoice Count
chart_config:
show_labels: true
```
Scatter plots require exactly one dimension and two to three metrics. The first metric is plotted on the x-axis, the second metric on the y-axis, and an optional third metric can be used for additional visualization properties.
## Advanced Features
### Multi-Series Charts
For line and bar charts, you can include multiple metrics:
```yaml theme={null}
- type: line
name: revenue_vs_cost
title: "Revenue vs Cost"
query: "SELECT date, sum(revenue) as rev, sum(cost) as cost FROM financials GROUP BY date"
dimensions:
- name: date
label: Date
metrics:
- name: rev
label: Revenue
- name: cost
label: Cost
```
## Text (Markdown)
Render rich text using Markdown in your Intelligent App.
````yaml theme={null}
components:
texts:
- name: features_md
title: Features
content: |
## Key Features
### Data Analysis
- Real-time data processing
- Advanced analytics
- Custom visualizations
### User Experience
- Intuitive interface
- Responsive design
- Mobile-friendly
```sql
SELECT
id,
name,
created_at
FROM users
WHERE active = true;
```
layout:
tabs:
- name: features
title: Features
items:
- type: text
name: features_md
span: 12
````
For full attributes, see the [YAML Reference: Texts](/reference/intelligent-apps/yaml-reference#texts).
## Item List
List-style chart for items (orders, activity): icon, title, description, optional timestamp and badge; opens detail modal on click.
```yaml theme={null}
components:
charts:
- type: item_list
name: recent_orders_list
title: "Recent Orders"
query: |
SELECT
'ShoppingCart' AS icon,
'text-indigo-600' AS icon_color,
concat('Order #', order_id) AS title,
concat(customer_name, ' | ', product_name, ' (Qty: ', quantity, ')') AS description,
formatDate(order_date, 'MMM dd, yyyy') AS timestamp,
payment_status AS badge_text,
CASE
WHEN payment_status = 'Paid' THEN 'success'
WHEN payment_status = 'Unpaid' THEN 'warning'
ELSE 'neutral'
END AS badge_variant
FROM orders
ORDER BY order_date DESC
LIMIT 20;
layout:
tabs:
- name: overview
title: Overview
items:
- type: chart
name: recent_orders_list
span: 12
```
For full attributes and layout usage, see the [YAML Reference: Item Lists](/reference/intelligent-apps/yaml-reference#item-lists).
## Widgets
When none of the chart types above fit, render **your own React component** in the app grid with a widget. Widgets are TSX components that Datazone compiles in the browser at runtime, placed in the layout with `type: widget`. Attach a SQL query and its rows arrive as the component's `data` prop.
```yaml theme={null}
components:
widgets:
- name: top_customers_widget
title: Top Customers
source_type: file
file: widgets/TopCustomers.tsx
query: |
SELECT CustomerName as customer, SUM(OrderLineTotalAmount) as revenue
FROM consolidated_sales_df
GROUP BY CustomerName
ORDER BY revenue DESC
LIMIT 5
layout:
tabs:
- name: overview
title: Overview
items:
- type: widget
name: top_customers_widget
span: 6
height: 320
```
The container gives your widget a bordered, padded surface and scrolls it when `height` is set. Widgets can also be nested inside a `chart-group`, next to charts.
`title` and `description` are metadata on the definition — no header is rendered for you. Your component owns its entire visual output, so render the heading yourself.
### Source Types
With `source_type: inline` (the default) the component code lives in the app YAML under `source`. With `source_type: file` it lives in a `.tsx` file in your project repository, referenced by a path relative to the repository root and read when the app is loaded.
```yaml theme={null}
widgets:
# Inline: must define `source`, must not define `file`
- name: status_banner
title: Status Banner
source_type: inline
source: |
import React from 'react'
export default function StatusBanner({ appContext }) {
return
Active theme: {appContext.theme}
}
# File: must define `file`, must not define inline `source`
- name: revenue_breakdown
title: Revenue Breakdown
source_type: file
file: widgets/RevenueBreakdown.tsx
```
Prefer `file` widgets for anything non-trivial: you get editor support, syntax highlighting, and reviewable diffs, and the same component can be reused across apps.
### Component Contract
Your source must **`export default`** a React component. Datazone passes it two props:
| Prop | Type | Description |
| ------------ | ------ | ---------------------------------------------------------------------------------------------------- |
| `data` | array | Rows returned by the widget's `query`, keyed by column alias. Empty array when no `query` is defined |
| `appContext` | object | `filters` (current filter values by filter name), `setFilter(name, value)`, and `theme` |
`setFilter` makes widgets **interactive participants** in the app rather than display surfaces: a click inside your component can drive every chart and filter, the same way `affected_filter` does for bar and pie charts.
```tsx theme={null}
import React from "react"
import { Card, CardContent } from "@datazone/widget-sdk"
export default function RegionPicker({ data, appContext }) {
return (
{data.map((row) => (
))}
)
}
```
### Available Imports
The widget runtime resolves a fixed set of imports — anything else fails to compile.
| Module | Contents |
| ---------------------- | ------------------------------------------------------------------- |
| `react` | The full React API — hooks, `Fragment`, and the default export |
| `@datazone/widget-sdk` | `Card`, `CardHeader`, `CardTitle`, `CardDescription`, `CardContent` |
Third-party packages cannot be installed or imported inside a widget — no chart libraries, data-fetching clients, or utility libraries. Build with React and the widget SDK, and let the widget's `query` do the data work.
### Styling
SDK components inherit the active theme automatically, so `Card` and friends match the rest of the dashboard, including [custom themes](/reference/intelligent-apps/yaml-reference#style-configuration). For your own styling, prefer inline `style` objects: utility classes are resolved from the platform's precompiled stylesheet, so a class the platform itself never uses may not exist at runtime.
Theme CSS variables come in two shapes, and mixing them up produces an invalid color:
| Variables | Value | Use as |
| ------------------------------------------------------------------------------------------------------------ | -------------- | ------------------- |
| `--chart-1` … `--chart-5` | Complete color | `var(--chart-1)` |
| `--foreground`, `--muted`, `--muted-foreground`, `--border`, `--accent`, `--success`, `--warning`, `--error` | HSL components | `hsl(var(--error))` |
```tsx theme={null}
```
`--primary` and `--ring` are redefined by the color themes, so their shape depends on the active theme. Use `--chart-1` … `--chart-5` for a theme-following accent, and the semantic variables above for text, borders, and status colors.
### Widget Queries
A widget's `query` behaves like a chart query: **Jinja templating** with the app's current filter variables (use the same `{% if ... is defined %}` guards), **cached** per the app's `cache` and `cache_ttl` config, re-run whenever a filter changes, and restricted to `SELECT` statements executed with the permissions of the user viewing the app.
```yaml theme={null}
widgets:
- name: filtered_orders
title: Filtered Orders
source_type: file
file: widgets/FilteredOrders.tsx
query: |
SELECT OrderDate as order_date, CustomerName as customer, OrderLineTotalAmount as amount
FROM consolidated_sales_df
WHERE 1=1
{% if order_date is defined %}
AND OrderDate >= '{{ order_date }}'
{% endif %}
ORDER BY OrderDate DESC
LIMIT 50
```
A widget without a `query` never hits the database — use that for purely presentational widgets, or ones that only react to `appContext`.
Widget queries read **tab-level filter variables** only. [Chart inputs](#dynamic-charts-with-chart-inputs) are a chart-level feature and are not available to widgets — render your own controls with React state instead.
### Validation and Errors
At **load time**, the app definition is rejected — with an error naming the widget — when a layout item references a widget that is not defined in `components.widgets`, when the `source` / `file` rules above are broken, or when a `file` widget points at a path that does not exist in the repository.
At **render time**, compile and runtime errors appear as an inline message in the widget's slot. A broken widget only breaks itself; the surrounding tab, charts, and filters keep working.
Widget code is not validated when the app is saved — it is compiled in the viewer's browser. Open the app after deploying a new widget to confirm it renders. Widgets work in [embedded apps](/reference/intelligent-apps/embedding) too.
For full attributes and layout usage, see the [YAML Reference: Widgets](/reference/intelligent-apps/yaml-reference#widgets).
# Embedding
Source: https://docs.datazone.co/reference/intelligent-apps/embedding
You can build a Data Intensive Application in a couple of minutes with Datazone.
# Embedding Intelligent Apps
Datazone allows you to **embed** your Intelligent Apps directly into **your own applications** using a simple iframe approach. This enables you to integrate powerful data visualizations and dashboards into your product while maintaining your own application's user experience.
## Overview
Embedding works through a secure token-based authentication mechanism:
1. Your application generates a signed JWT token containing the necessary parameters
2. This token is used to construct an iframe URL
3. The iframe renders your Intelligent App inside your application
## Integration Steps
### 1. Prerequisites
* An Intelligent App in your Datazone account. You can look at [Intelligent Apps](/reference/intelligent-apps/overview) to create your first app.
* Go to Intelligent App settings page under the Project section and click the Access Control tab to enable embedding and get your code snippet.
* A web application where you want to embed the Intelligent App
### 2. Example Code Snippet
```javascript theme={null}
const jwt = require("jsonwebtoken");
const DATAZONE_SITE_URL = "https://comet.datazone.co";
const DATAZONE_SECRET_KEY = "your_secret_key"; // Replace with your actual secret key
const payload = {
intelligent_app_id: "your_app_id", // Replace with your app's ID
email: "user@example.com" // User's email.
};
const token = jwt.sign(payload, DATAZONE_SECRET_KEY);
const iframeUrl = DATAZONE_SITE_URL + "/app/embedding/" + token;
```
You can use `user_id` instead of `email` in the payload for user identification based on your application's needs.
### 3. Embed the App
Insert an iframe in your application pointing to the generated URL:
```html theme={null}
```
## Embedding Parameters
The JWT token payload can include the following parameters:
| Parameter | Type | Description |
| -------------------- | ------ | -------------------------------------------------------------------- |
| `intelligent_app_id` | string | **Required.** The ID of your Intelligent App |
| `email` | string | Email of the user viewing the app (for access control and analytics) |
| `user_id` | string | Optional, instead of email, a unique identifier for the user |
| `variables` | object | Initial values for app variables |
| `filter_values` | object | Initial values for filter components |
| `config_overwrite` | object | Override app configuration settings |
### Example: Setting Initial Variables and Filters
```javascript theme={null}
const payload = {
intelligent_app_id: "685ff04dab58322fd751e040",
email: "user@example.com",
variables: {
selected_region: "Europe",
time_period: "last_30_days"
},
filter_values: {
region_filter: "Europe",
date_range: "2025-08-01,2025-09-01"
}
};
```
### Example: Overriding App Configuration
You can override app configuration settings:
```javascript theme={null}
const payload = {
intelligent_app_id: "685ff04dab58322fd751e040",
email: "user@example.com",
config_overwrite: {
hide_header: true,
hide_filters: false,
style: {
theme: "orange"
}
}
};
```
## Security Considerations
* Keep your Datazone secret key secure and never expose it in client-side code
* Generate tokens server-side and pass the complete iframe URL to your frontend
* Set appropriate content security policies for iframe embedding
* Consider implementing additional authentication mechanisms if needed
## Example: Full Server-Side Implementation
Here's a more complete example using Node.js/Express:
```javascript theme={null}
const express = require('express');
const jwt = require('jsonwebtoken');
const app = express();
// Configuration
const DATAZONE_SITE_URL = "https://comet.datazone.co";
const DATAZONE_SECRET_KEY = process.env.DATAZONE_SECRET_KEY; // Store in environment variables
app.get('/embedded-dashboard', (req, res) => {
// Get user information from your auth system
const currentUser = req.user;
// Create the token
const payload = {
intelligent_app_id: "685ff04dab58322fd751e040",
email: currentUser.email,
variables: {
department: currentUser.department
}
};
const token = jwt.sign(payload, DATAZONE_SECRET_KEY);
const iframeUrl = DATAZONE_SITE_URL + "/app/embedding/" + token;
// Render your page with the iframe URL
res.render('dashboard', { iframeUrl });
});
app.listen(3000, () => console.log('Server running on port 3000'));
```
## Customizing the Embedded Experience
You can customize how your embedded app appears by overriding configuration settings in the token:
```javascript theme={null}
const payload = {
intelligent_app_id: "685ff04dab58322fd751e040",
config_overwrite: {
hide_header: true, // Hide the app header for a cleaner embed
hide_llm: true, // Hide the Orion AI assistant
hide_insights: true, // Hide the insights panel
style: {
theme: "teal" // Match your application's theme
}
}
};
```
## Troubleshooting
If your embedded app isn't loading correctly:
1. Check that your token is signed correctly with the right secret key
2. Verify that the Intelligent App ID is correct
3. Ensure your domain is allowed for embedding (contact Datazone support if needed)
4. Look for CORS or CSP issues in your browser's developer tools
## Next Steps
* [Customize App Appearance](/reference/intelligent-apps/yaml-reference#config-section) - Learn about configuration options
* [Working with Variables](/reference/intelligent-apps/filters) - Learn how to use variables in your app
* [Contact Support](mailto:support@datazone.co) - Get help with embedding setup
# Filters
Source: https://docs.datazone.co/reference/intelligent-apps/filters
How to use filters for interactivity in Datazone Intelligent Apps.
Filters allow users to interactively control the data displayed in your Intelligent App dashboards. They are linked to variables and can be referenced in SQL queries to dynamically filter results.
## Filter Types
Datazone supports several filter types:
| Type | Description |
| ---------- | ----------------------------- |
| `text` | Free text input |
| `number` | Numeric input |
| `dropdown` | Select from a list of options |
| `date` | Date picker |
## Filter Configuration
Each filter is defined in the `components.filters` section of your YAML configuration. Key properties:
* `type`: The filter type (`text`, `number`, `dropdown`, `date`)
* `name`: Unique filter name
* `title`: Label shown to users
* `affected_variable`: The variable this filter updates
* `default_value`: (Optional) Default value
* `options`: (For dropdown) Static or SQL-driven options
* `multiple`: (Optional, **dropdown only**) Enable multi-select functionality (default: `false`)
* `placeholder`: (Optional) Placeholder text
The `options` object also accepts `use_input` (SQL options only) to enable [server-side search](#searchable-dropdown-filters).
The `multiple` attribute is only allowed for `dropdown` filters. Using it with other filter types (`text`, `number`, `date`) will result in a validation error.
If you define a filter in the Intelligent App layout but do not use it any layout tab, validations will fail. Make sure all defined filters are included in at least one tab's `filters` list.
### Example: Static Dropdown Filter
```yaml theme={null}
- type: dropdown
name: membership_type_filter_limited
title: Membership Type
affected_variable: membership_type_second
default_value: 'all'
options:
type: static
values:
- value: all
label: All
- value: basic
label: Basic
- value: premium
label: Premium
- value: vip
label: VIP
```
### Example: SQL-Driven Dropdown Filter
```yaml theme={null}
- type: dropdown
name: membership_type_filter
title: Membership Type
affected_variable: membership_type
options:
type: sql
query: |
SELECT distinct ProductGroup_Name FROM sales_order LIMIT 100;
```
The query should return only one column with the values you want to display in the dropdown.
### Example: Date and Number Filters
```yaml theme={null}
- type: date
name: order_date_filter
title: Order Date Filter
affected_variable: order_date
default_value: '2024-04-01'
- type: number
name: penalty_count_filter
title: Penalty Count Filter
affected_variable: penalty_count
default_value: '10'
placeholder: 'Enter penalty count'
```
All filter values are treated as strings. Ensure your queries handle type conversion as needed. Also if you pass
a non-numeric string from SQL based dropdown to a number filter, it will be ignored.
Example: `SELECT distinct toString(year) as year FROM sales_order;`
## Using Filters in Queries
Filters update variables, which you can reference in your chart queries using double curly braces:
```yaml theme={null}
query: |
select * from sales_order WHERE OrderDate > '{{ order_date }}' limit 100;
```
## Multi-Select Dropdown Filters
Dropdown filters support multi-select functionality by setting `multiple: true`. This allows users to select multiple values from the dropdown list simultaneously.
### Configuration Example
```yaml theme={null}
- type: dropdown
name: region_filter
title: Select Regions
affected_variable: selected_regions
multiple: true
default_value: North # Optional: single default value
options:
type: static
values:
- value: North
label: North Region
- value: South
label: South Region
- value: East
label: East Region
- value: West
label: West Region
```
### Using Multi-Select Values in Queries
When `multiple: true` is enabled, the variable will contain an array of selected values. Use Jinja's `join` filter to format these values for SQL queries:
```yaml theme={null}
query: |
SELECT * FROM customer_data
WHERE 1=1
{% if selected_regions %}
AND CustomerRegion IN ('{{ selected_regions | join("','") }}')
{% endif %}
LIMIT 100;
```
**How it works:**
* When users select `['North', 'South']`, the Jinja template transforms it to: `'North','South'`
* The final SQL becomes: `AND CustomerRegion IN ('North','South')`
* If no values are selected, the condition is skipped entirely
### Multi-Select with SQL-Driven Options
You can also use SQL queries to populate multi-select dropdowns:
```yaml theme={null}
- type: dropdown
name: product_filter
title: Select Products
affected_variable: selected_products
multiple: true
options:
type: sql
query: |
SELECT DISTINCT ProductName
FROM products
ORDER BY ProductName
```
## Searchable Dropdown Filters
By default, an SQL-driven dropdown fetches its full list of options once and then filters them **client-side** as the user types. This is fine for short lists, but it does not scale to high-cardinality columns (thousands of customers, product references, IDs, etc.) where you cannot — or do not want to — load every value into the browser.
Setting `use_input: true` on an `sql` options block switches the dropdown to **server-side search**. The text the user types into the dropdown's search box is sent back to your options `query` as the `search_term` variable, so the database does the filtering and returns only matching rows.
### Configuration Example
```yaml theme={null}
- type: dropdown
name: order_reference_filter
title: Order Reference
affected_variable: order_reference
multiple: true
options:
type: sql
use_input: true
query: |
SELECT DISTINCT order_reference
FROM orders
WHERE 1=1
{% if search_term %}
AND order_reference LIKE '%{{ search_term }}%'
{% endif %}
LIMIT 100
```
**How it works:**
* As the user types in the dropdown, the input is debounced (\~300ms) and sent to the options `query` as `search_term`.
* Wrap the search condition in `{% if search_term %}` so the query still returns a sensible initial list (e.g. the first 100 rows) before the user types anything.
* The database returns only the matching values, which populate the dropdown. Because filtering happens server-side, client-side filtering is disabled for that dropdown.
* Always add a `LIMIT` to keep the result set small and responsive.
`search_term` is only injected when `use_input: true`.
`use_input` is only valid for `type: sql` options. Setting it on a `type: static` options block results in a validation error.
`use_input` works for both single-select and multi-select (`multiple: true`) dropdowns.
## Filter Dependencies
Filters can depend on other filters to create cascading filter effects. When one filter changes, dependent filters can update their options based on the selected value. This is particularly useful for hierarchical data like country/city relationships.
### Example: Country and City Filters
In this example, the city filter's options are limited based on the selected country:
```yaml theme={null}
filters:
- type: dropdown
name: country_filter
title: Country
affected_variable: selected_country
default_value: 'USA'
options:
type: sql
query: "SELECT DISTINCT country FROM locations ORDER BY country"
- type: dropdown
name: city_filter
title: City
affected_variable: selected_city
options:
type: sql
query: |
SELECT DISTINCT city FROM locations
WHERE 1=1
{% if selected_country is defined %}
AND country = '{{ selected_country }}'
{% endif %}
ORDER BY city
```
### Key Points for Filter Dependencies
* Use conditional logic with `{% if variable_name is defined %}` to check if a filter value exists
* Always include a fallback condition (like `WHERE 1=1`) to ensure valid SQL when no filters are applied
* Dependent filters will automatically refresh when their parent filter changes
* You can chain multiple levels of dependencies (e.g., country → state → city)
## Best Practices
* Use descriptive titles and placeholders for better UX
* Use SQL-driven dropdowns for dynamic option lists
* Set sensible defaults to improve initial dashboard state
* Group filters logically on tabs for clarity
See also: [Intelligent Apps Overview](./overview), [Chart Reference](./charts)
# Orion AI (LLM Assistant)
Source: https://docs.datazone.co/reference/intelligent-apps/orion-ai
Overview of Orion AI, the LLM-powered assistant for Intelligent Apps.
Orion AI is an advanced Large Language Model (LLM) assistant integrated into Datazone's Intelligent Apps. It enables users to ask natural language questions about their dashboards and data, making analytics more accessible and interactive for everyone—no SQL or technical expertise required.
## What is Orion AI?
Orion AI is a chatbot assistant that leverages state-of-the-art LLMs (such as GPT-4O and Claude) to help you:
* **Understand your dashboards**: Ask questions about any chart, metric, or filter in your Intelligent App.
* **Perform data analysis**: Get instant insights, summaries, and explanations about your data.
* **Discover trends and anomalies**: Let Orion AI highlight important changes or outliers in your data.
* **Learn how to use the platform**: Ask for help with features, navigation, or best practices.
Orion AI is always context-aware. When you use it on an Intelligent App page, it automatically receives the app's metadata and configuration, so its answers are tailored to your current dashboard and data context.
## Why Use Orion AI?
* **No technical skills required**: Anyone can ask questions in plain English (or other supported languages).
* **Faster insights**: Skip manual exploration—just ask Orion AI for what you need.
* **Guided analysis**: Orion can suggest next steps, deeper dives, or related metrics to explore.
* **Customizable behavior**: App creators can set custom instructions for Orion AI using the `llm_instruction` field in the app's YAML config, tailoring the assistant's tone and focus for each dashboard.
## How It Works
* On any Intelligent App, look for the Orion AI chat interface.
* Type your question (e.g., "What is the sales trend for the last quarter?" or "Why did revenue drop in March?").
* Orion AI will analyze your app's metadata, charts, and filters, and respond with a relevant, data-driven answer.
* If the app creator has provided a custom instruction (via `llm_instruction`), Orion will follow those guidelines in its responses.
## Customizing Orion AI
To customize Orion's behavior for your app, set the `llm_instruction` field in your YAML config:
```yaml theme={null}
config:
llm_instruction: "You are a helpful assistant for sales analytics. Always explain trends simply."
```
## Configuration Options
In addition to `llm_instruction`, you can also use these optional config options:
* `hide_llm`: (Optional) Hide the LLM assistant (Orion AI) in the app UI
* `hide_insights`: (Optional) Hide the Insights panel in the app UI
***
Orion AI makes data analysis conversational, intuitive, and accessible—empowering every user to get more from their data.
# Overview
Source: https://docs.datazone.co/reference/intelligent-apps/overview
Build interactive data applications with Datazone's Intelligent Apps feature
# Intelligent Apps
Datazone Intelligent Apps provide a powerful way to build interactive data dashboards and applications without writing frontend code. Using a declarative YAML configuration, you can create multi-tab dashboards with charts, filters, and interactive elements that query your data directly.
## Overview
An Intelligent App consists of:
1. **Layout** - How your application is organized (tabs, charts, groups)
2. **Components** - The building blocks of your app (charts, filters, variables, texts, widgets)
3. **Configuration** - Settings that control app behavior
## App Structure
Intelligent Apps use a YAML-based configuration format:
```yaml theme={null}
app:
name: "My Dashboard"
description: "Dashboard description"
config:
cache: true
cache_ttl: 3600
hide_llm: false # (Optional) Hide the LLM assistant (Orion AI) in the app UI
hide_insights: false # (Optional) Hide the Insights panel in the app UI
llm_instruction: "You are a helpful assistant." # (Optional) Custom LLM instruction
layout:
tabs:
- name: main_tab
title: "Main Dashboard"
filters: [...]
items: [...]
components:
charts: [...]
variables: [...]
filters: [...]
texts: [...]
widgets: [...]
```
## Key Components
### Charts
Charts are the primary visualization elements. Datazone supports several chart types:
| Chart Type | Description |
| ------------ | --------------------------- |
| `number` | Single metric display |
| `line` | Time-series/trend chart |
| `bar` | Categorical comparisons |
| `pie` | Part-to-whole relationships |
| `radial` | Radial gauge for metrics |
| `table` | Tabular data display |
| `data_table` | Advanced tabular display |
Example chart definition:
```yaml theme={null}
- type: line
name: sales_over_time
title: "Sales Over Time"
description: "Monthly sales trend"
query: |
SELECT
toStartOfMonth(order_date) AS month,
sum(amount) AS total_amount
FROM orders
GROUP BY month
ORDER BY month
dimensions:
- name: month
label: Month
metrics:
- name: total_amount
label: Total Sales
format: "$0,0.00"
```
Example number chart definition:
```yaml theme={null}
- type: number
name: completed_tasks
title: "Completed Tasks"
query: "SELECT count(*) as completed FROM tasks WHERE status = 'done'"
metrics:
- name: completed
label: Completed Tasks
icon: check
icon_variant: success
format: "0"
```
#### Dimension Properties
| Property | Type | Description |
| --------------- | ------ | --------------------------------------------- |
| `name` | string | Unique identifier for the dimension |
| `label` | string | Display label for the dimension |
| `number_format` | string | (Optional) Number format for this dimension |
| `table_align` | string | (Optional) Table alignment: `left` or `right` |
### Widgets
When the built-in chart types aren't enough, you can render **your own React component** inside a tab with a widget. Widgets are written in TSX — inline in the YAML or in a `.tsx` file in your project repository — and can attach a SQL query whose rows arrive as a `data` prop.
```yaml theme={null}
widgets:
- name: top_customers_widget
title: Top Customers
source_type: file
file: widgets/TopCustomers.tsx
query: |
SELECT CustomerName as customer, SUM(OrderLineTotalAmount) as revenue
FROM consolidated_sales_df
GROUP BY CustomerName
ORDER BY revenue DESC
LIMIT 5
```
Place it in the layout with `type: widget`:
```yaml theme={null}
items:
- type: widget
name: top_customers_widget
span: 6
height: 320
```
See [Components: Widgets](/reference/intelligent-apps/components#widgets) for the component contract, available imports, and interactivity through `appContext`.
### Layout
The layout defines how components are arranged in your app using a responsive grid system:
* **Tabs**: Organize content into different sections
* **Items**: Individual elements like charts, text, or widgets
* **Chart Groups**: Collections of related charts
* **Span/Height**: Control sizing and layout
Example layout:
```yaml theme={null}
layout:
tabs:
- name: overview
title: "Overview"
filters: ["start_date", "product_category"]
items:
- type: chart-group
name: kpi_group
span: 12
entities:
- type: chart
name: total_sales
span: 4
- type: chart
name: total_orders
span: 4
- type: chart
name: sales_by_region
height: 400
span: 6
- type: chart
name: top_products
height: 400
span: 4
- type: chart
name: top_customers
height: 400
span: 4
- type: chart
name: monthly_performance_order
height: 400
span: 8
```
### Interactivity
Make your apps interactive with variables and filters:
* **Variables**: Store values that can be used in queries
* **Filters**: UI elements that update variables
Example filter:
```yaml theme={null}
filters:
- type: dropdown
name: category_filter
title: "Product Category"
affected_variable: product_category
options:
type: sql
query: "SELECT DISTINCT category FROM products"
```
## Query Variables
You can reference variables in your queries using double curly braces:
```yaml theme={null}
query: |
SELECT * FROM orders
WHERE order_date > '{{ start_date }}'
AND order_date < '{{ end_date }}'
{% if product_category != 'all' %}
AND category = '{{ product_category }}'
{% endif %}
```
## Best Practices
1. **Organize with Tabs**: Group related content into logical tabs
2. **Use Chart Groups**: Group related metrics together
3. **Filter Placement**: Place filters on tabs where they're most relevant
4. **Query Optimization**: Keep queries efficient for better performance
5. **Consistent Formatting**: Use the `format` attribute under each metric to ensure consistent number formatting
## App Configuration
The `config` section supports the following attributes:
| Attribute | Type | Description |
| ----------------- | ------ | ---------------------------------------------------------------- |
| `cache` | bool | Enable/disable caching of query results |
| `cache_ttl` | int | Cache time-to-live in seconds (default: 3600) |
| `hide_llm` | bool | (Optional) Hide the LLM assistant (Orion AI) in the app UI |
| `hide_insights` | bool | (Optional) Hide the Insights panel in the app UI |
| `llm_instruction` | string | (Optional) Custom instruction for the LLM (prompt customization) |
| `style` | object | (Optional) Custom styling configuration (theme and colors) |
Example usage:
```yaml theme={null}
config:
cache: true
cache_ttl: 3600
llm_instruction: "You are a helpful assistant."
```
### Theme Customization
You can customize your app's appearance by selecting from predefined themes or using custom style attributes for advanced color customization. See the [YAML Reference](/reference/intelligent-apps/yaml-reference#style-configuration) for detailed styling options.
## Next Steps
* [Detailed Chart Reference](/reference/intelligent-apps/components)
* [Custom Widgets](/reference/intelligent-apps/components#widgets)
* [Working with Filters](/reference/intelligent-apps/filters)
* [Advanced Queries](/reference/intelligent-apps/queries)
# Variable Usage
Source: https://docs.datazone.co/reference/intelligent-apps/query-manipulation
Examples of using variables and Jinja-style templating in Intelligent App queries.
Intelligent Apps support dynamic query generation using variables and Jinja-style templating. This allows you to build flexible dashboards that respond to user input from filters and variables.
## Basic Variable Substitution
You can reference variables in your SQL queries using double curly braces:
```yaml theme={null}
query: |
SELECT * FROM sales_orders WHERE order_date > '{{ order_date }}'
```
## Conditional Query Blocks
You can use Jinja-style control structures to include or exclude parts of your query based on variable values:
```yaml theme={null}
query: |
SELECT * FROM sales_orders
WHERE 1=1
{% if product_group != '' %}
AND product_group = '{{ product_group }}'
{% endif %}
{% if membership_type != 'all' %}
AND membership_type = '{{ membership_type }}'
{% endif %}
```
## Using Multiple Variables
Combine several variables for more complex filtering:
```yaml theme={null}
query: |
SELECT * FROM sales_orders
WHERE order_date BETWEEN '{{ start_date }}' AND '{{ end_date }}'
AND amount > {{ min_amount }}
```
## Example: Dynamic LIMIT
You can use variables to control query limits:
```yaml theme={null}
query: |
SELECT * FROM sales_orders
LIMIT {{ row_limit }}
```
## Example: IN Clauses with Lists
If your variable is a list, you can use Jinja to join values for an IN clause:
```yaml theme={null}
query: |
SELECT * FROM sales_orders
WHERE region IN ({{ regions | join(", ") }})
```
## Best Practices
* Always provide sensible defaults for variables to avoid empty queries.
* Use `{% if ... %}` blocks to prevent SQL errors when variables are empty or unset.
* Sanitize and validate user input where possible.
## Reference: Jinja Syntax
* `{{ variable }}`: Insert variable value
* `{% if ... %} ... {% endif %}`: Conditional blocks
* `{% for ... %} ... {% endfor %}`: Loops (for advanced use)
* Filters like `| join(", ")` for lists
See also: [Filters](./filters), [Overview](./overview), [Charts](./charts)
# YAML Reference
Source: https://docs.datazone.co/reference/intelligent-apps/yaml-reference
Comprehensive reference for all YAML attributes in Intelligent App definitions.
# Intelligent App YAML Reference
This page provides a detailed reference for all attributes available in Intelligent App YAML definitions. Use this as a guide when authoring or reviewing your app configuration files.
## Top-Level Structure
```yaml theme={null}
app:
name: string # App display name
description: string # App description
icon: string # (Optional) App icon (Lucide icon name)
config: # App configuration (see below)
layout: # Layout definition (see below)
components: # App components (see below)
```
***
## `config` Section
| Attribute | Type | Description |
| --------------------------- | ------ | --------------------------------------------------------------------------- |
| `cache` | bool | Enable/disable caching of query results (default: true) |
| `cache_ttl` | int | Cache time-to-live in seconds (default: 3600) |
| `hide_llm` | bool | (Optional) Hide the LLM assistant (Orion AI) in the app UI (default: false) |
| `hide_insights` | bool | (Optional) Hide the Insights panel in the app UI (default: false) |
| `hide_filters` | bool | (Optional) Hide the filters panel in the app UI (default: false) |
| `hide_header` | bool | (Optional) Hide the app header in the UI (default: false) |
| `chart_export_enabled` | bool | (Optional) Enable export functionality for charts (default: false) |
| `llm_instruction` | string | (Optional) Custom instruction for the LLM (prompt customization) |
| `insight_agent_instruction` | string | (Optional) Custom instruction for the insights agent |
| `style` | object | (Optional) Custom styling configuration (see below) |
### Style Configuration
| Attribute | Type | Description |
| ------------------------- | ------ | ---------------------------------------------------------------------------- |
| `theme` | string | UI theme color scheme (default: "default") |
| `custom_style_attributes` | object | (Optional) Custom CSS variables for advanced theme customization (see below) |
Available themes: "default", "teal", "blue", "green", "purple", "orange", "amber", "mono"
#### Custom Style Attributes
You can override theme colors using Tailwind CSS variables in OKLCH format. This allows fine-grained control over your app's appearance:
```yaml theme={null}
style:
theme: blue
custom_style_attributes:
"--background": "oklch(0.9540 0.0134 286.2151)"
"--foreground": "oklch(0.4396 0.0330 279.3275)"
"--chart-1": "oklch(0.5547 0.2503 297.0166)"
"--chart-2": "oklch(0.6820 0.1448 226.1885)"
"--chart-3": "oklch(0.5916 0.1596 135.3018)"
"--chart-4": "oklch(0.6979 0.1926 44.0932)"
"--chart-5": "oklch(0.7078 0.1259 21.4179)"
```
Common customizable CSS variables:
| Variable | Description |
| ------------------------------- | ---------------------- |
| `--background` | Main background color |
| `--foreground` | Main text color |
| `--chart-1` through `--chart-5` | Chart color palette |
| `--primary` | Primary accent color |
| `--secondary` | Secondary accent color |
| `--muted` | Muted background color |
| `--accent` | Accent color |
Colors must be specified in OKLCH format, including the `oklch()` wrapper (e.g., "oklch(0.9540 0.0134 286.2151)"). Other color formats such as HSL, RGB, or hex are not accepted.
***
## `layout` Section
Defines the structure of the app (tabs, items, chart groups).
```yaml theme={null}
layout:
tabs:
- name: string # Tab identifier
title: string # Tab display title
filters: [string] # (Optional) List of filter names
items: # List of layout items (see below)
```
### Layout Items
| Attribute | Type | Description |
| ---------- | ------ | ------------------------------------------------- |
| `type` | string | `chart`, `chart-group`, `text`, `table`, `widget` |
| `name` | string | Unique item name |
| `span` | int | (Optional) Grid span (width) |
| `height` | int | (Optional) Height in px |
| `entities` | list | (For `chart-group`) Nested layout items |
***
## `components` Section
Holds the main building blocks: charts, variables, filters, texts, and widgets.
### Charts
```yaml theme={null}
charts:
- type: string # `number`, `radial`, `line`, `bar`, `vertical_bar`, `pie`, `table`, `data_table`, `item_list`, `composed`, `heatmap`, `scatter`, `custom`
name: string # Unique chart name
title: string # Chart title
description: string # (Optional) Description
query: string # SQL query (Jinja supported)
chart_inputs: # (Optional) Interactive controls for dynamic queries
- type: string # Input type: "dropdown"
name: string # Variable name accessible in query via Jinja
label: string # Display label
default: string # Default selected value
options: [string] # (Optional) List of static options for dropdown
options_query: string # (Optional) SQL query to fetch dynamic options (cannot be used with 'options')
axis: # (Optional, for line, bar, vertical_bar, composed, heatmap)
- name: string # Axis name (e.g., left, right)
position: string # (Optional) Axis position: left, right, top, bottom (default: left)
format: string # (Optional) Number format for this axis
hidden: bool # (Optional) Whether to hide this axis
dimensions: # (Optional) List of dimensions
- name: string
label: string
number_format: string # (Optional) Number format for numeric dimensions
table_align: left|right # (Optional) Table alignment for this dimension
affected_filter: string # (Optional, bar/pie only) Filter to update when this dimension is clicked
metrics: # (Optional) List of metrics
- name: string
label: string
format: string # (Optional) Number format string
icon: string # (Optional, number/radial charts only) Lucide icon name (see https://lucide.dev/icons/)
icon_variant: string # (Optional, number/radial charts only) One of: "default", "neutral", "success", "warning", "error"
axis_name: string # (Optional, for line, bar, vertical_bar, composed) Axis to use for this metric
composed_type: string # (Optional, for composed) "line" or "bar"
submetric_name: string # (Optional, number/radial charts only) Name of the submetric to display
submetric_type: string # (Optional, number/radial charts only) Type of submetric display: "plain", "change", "delta"
color: string # (Optional) Custom color for this metric's series (e.g., hex code or Tailwind color)
show_label: bool # (Optional) Whether to show the data label for this metric
pivot_by: [string] # (Optional) List of columns to pivot by
show_labels: bool # (Optional, default: true)
chart_config: # (Optional) Chart configuration
show_legend: bool # (Optional) Whether to show the chart legend (default: false)
show_labels: bool # (Optional) Whether to show labels on the chart (default: false)
fill_donut: bool # (Optional, pie charts only) Whether to fill the donut chart (default: false)
is_stacked: bool # (Optional, pie/bar/vertical_bar charts) Whether to stack elements (default: false)
line_type: string # (Optional, line/composed charts) "linear", "monotone", "step", "natural"
fill_area: bool # (Optional, line charts only) Whether to fill the area under the line
custom_chart_type: str # (Optional, custom charts only) Custom chart type identifier
sub_expression: string # (Optional, number/radial charts) Sub-expression for metric calculation
page_size: int # (Optional, data_table charts) Number of rows per page
layout: string # (Optional, bar/vertical_bar charts) "horizontal" or "vertical"
hide_expression: string # (Optional, all chart types) JavaScript expression to conditionally hide the chart based on data
```
#### Composed Chart Example
```yaml theme={null}
- type: composed
name: top_products_composed
title: Top Performing Products
description: Top 10 products by revenue
query: |
SELECT
ProductName as product,
SUM(OrderLineTotalAmount) as revenue,
AVG(OrderLineTotalAmount) as avg_revenue
FROM consolidated_sales_df_299ceb
WHERE 1=1
{% if region_filter is defined %}
AND CustomerRegion = '{{ region_filter }}'
{% endif %}
{% if product_group_filter is defined %}
AND ProductGroup = '{{ product_group_filter }}'
{% endif %}
GROUP BY ProductName
ORDER BY revenue DESC
LIMIT 10;
axis:
- name: left
- name: right
position: right
dimensions:
- name: product
label: Product
metrics:
- name: revenue
label: Revenue
axis_name: "left"
composed_type: "bar"
format: "$0,0.00"
- name: avg_revenue
label: "Average Revenue"
axis_name: "right"
composed_type: "line"
format: "0,0[.]00 $"
```
#### Example: Bar Chart with Click-to-Filter
```yaml theme={null}
type: bar
name: sales_by_country
title: Sales by Country
query: SELECT country, sum(amount) as total FROM sales GROUP BY country
axis:
- name: left
dimensions:
- name: country
label: Country
affected_filter: country_filter
metrics:
- name: total
label: Total Sales
```
#### Example: Dynamic Chart with Chart Inputs
Chart inputs allow users to modify the query dynamically using interactive controls like dropdowns.
##### Static Options
```yaml theme={null}
type: bar
name: top_by_revenue
title: Top 10 by Revenue
chart_inputs:
- type: dropdown
name: group_by_column
label: Group By
default: Product
options:
- Product
- Customer
query: |
SELECT
{% if group_by_column == 'Product' %}
ProductName as group,
{% else %}
CustomerName as group,
{% endif %}
SUM(OrderLineTotalAmount) as revenue
FROM consolidated_sales_df
WHERE 1=1
{% if date_from is defined %}
AND OrderDate >= '{{ date_from }}'
{% endif %}
{% if date_to is defined %}
AND OrderDate <= '{{ date_to }}'
{% endif %}
{% if region_filter is defined and region_filter != 'all' %}
AND CustomerRegion = '{{ region_filter }}'
{% endif %}
{% if group_by_column == 'Product' %}
GROUP BY ProductName
{% else %}
GROUP BY CustomerName
{% endif %}
ORDER BY revenue DESC
LIMIT 10
axis:
- name: left
format: "$0,0"
dimensions:
- name: group
label: Group
metrics:
- name: revenue
label: Revenue
format: "$0,0.00"
axis_name: left
```
##### Dynamic Options
Chart inputs can also fetch their dropdown options dynamically from database queries. This enables data-driven dropdowns and cascading input scenarios where one input's value affects another input's options.
```yaml theme={null}
type: bar
name: sales_by_region
title: Sales by Region
chart_inputs:
- type: dropdown
name: country
label: Select Country
default: USA
options:
- USA
- Canada
- UK
- type: dropdown
name: region
label: Select Region
options_query: "SELECT DISTINCT region FROM sales_data WHERE country = '{{ country }}' ORDER BY region"
query: |
SELECT
region,
SUM(amount) as total_sales
FROM sales_data
WHERE country = '{{ country }}'
AND region = '{{ region }}'
GROUP BY region
dimensions:
- name: region
label: Region
metrics:
- name: total_sales
label: Total Sales
format: "$0,0.00"
```
Chart inputs work with all chart types and seamlessly integrate with Jinja templating, allowing you to create highly dynamic and interactive visualizations.
**Key Features:**
* **Filter-Dependent Options**: Chart input options can change based on active tab-level filters
* **Cascading Inputs**: One input's value can affect another input's available options
* **Template Variables**: Use both tab-level filters and other chart input values in your `options_query`
You cannot define both `options` and `options_query` for the same chart input. Use `options` for static lists and `options_query` for dynamic, data-driven dropdowns.
#### Example: Pie Chart with Click-to-Filter
```yaml theme={null}
type: pie
name: sales_distribution
title: Sales Distribution
query: SELECT category, sum(amount) as total FROM sales GROUP BY category
dimensions:
- name: category
label: Category
affected_filter: category_filter
metrics:
- name: total
label: Total Sales
```
#### Example: Heatmap Chart
```yaml theme={null}
type: heatmap
name: sales_heatmap
title: Sales Heatmap by Region and Product
query: |
SELECT region, product, sum(amount) as total
FROM sales
GROUP BY region, product
dimensions:
- name: region
label: Region
- name: product
label: Product
metrics:
- name: total
label: Total Sales
format: "$0,0.00"
```
Heatmap charts require exactly 2 dimensions and at least 1 metric.
#### Example: Conditional Chart Visibility
You can conditionally hide charts based on the query result data using `hide_expression`. The expression receives the data array and should return `true` to hide the chart:
```yaml theme={null}
type: bar
name: sales_by_category
title: Sales by Category
query: SELECT category, sum(amount) as total FROM sales GROUP BY category
dimensions:
- name: category
label: Category
metrics:
- name: total
label: Total Sales
chart_config:
hide_expression: "(data) => {return data.length < 5}"
```
In this example, the chart will only be displayed if the query returns 5 or more rows. This is useful for hiding charts when there's insufficient data to display meaningfully.
The `hide_expression` is a JavaScript function that receives the `data` parameter (an array of query results) and must return a boolean value. Return `true` to hide the chart, or `false` to show it.
#### Example: Scatter Plot Chart
```yaml theme={null}
type: scatter
name: example_scatter_chart
title: Sales Profiles by Countries
query: |
SELECT
CustomerCountry,
ROUND(AVG(UnitPrice), 2) as avg_unit_price,
ROUND(SUM(InvoiceTotalAmount), 2) as total_invoice_count
FROM consolidated_sales_df_9cb4a3
GROUP BY CustomerCountry
ORDER BY total_invoice_count DESC
LIMIT 20;
dimensions:
- name: CustomerCountry
label: "Country"
metrics:
- name: avg_unit_price
label: Average Unit Price
- name: total_invoice_count
label: Total Invoice Count
chart_config:
show_labels: true
```
Scatter plot charts require exactly 1 dimension and 2-3 metrics. The first metric is plotted on the x-axis, the second metric on the y-axis, and an optional third metric can be used for additional visualization properties.
#### Example: Custom Chart
```yaml theme={null}
type: custom
name: custom_visualization
title: Custom Visualization
query: SELECT * FROM custom_data
chart_config:
custom_chart_type: my_custom_chart
```
### Variables
```yaml theme={null}
variables:
- name: string # Variable name
type: string # `string`, `integer`, `float`, `boolean`, `date`
```
### Filters
```yaml theme={null}
filters:
- type: string # `text`, `number`, `dropdown`, `date`
name: string # Filter name
title: string # Display label
affected_variable: str # Variable updated
default_value: any # (Optional) Default value
multiple: bool # (Optional, dropdown only) Enable multi-select (default: false)
options: # (For dropdown)
type: string # `static` or `sql`
values: # (For static)
- value: string
label: string
query: string # (For sql)
use_input: bool # (Optional, sql only) Pass the dropdown's search text to the query as `search_term` (default: false)
placeholder: string # (Optional)
```
### Texts
Markdown text components allow adding rich textual content to your apps.
```yaml theme={null}
texts:
- name: string # Unique text component name
title: string # (Optional) Display title
content: string # Markdown content (supports GFM and fenced code blocks)
```
Layout usage with a text item:
```yaml theme={null}
layout:
tabs:
- name: overview
title: Overview
items:
- type: text # Layout item type must be 'text'
name: overview_md # References a component from components.texts
span: 12 # (Optional) Grid span 1–12
height: 240 # (Optional) Height in px; enables scroll when set
```
Text layout item properties:
| Attribute | Type | Description |
| --------- | ------ | ------------------------------------------------------------- |
| `type` | string | Must be `text` |
| `name` | string | References a component from `components.texts` |
| `span` | int | (Optional) Grid span (width) |
| `height` | int | (Optional) Height in px; when set, content becomes scrollable |
### Widgets
Custom React (TSX) components rendered inside the app grid. See [Components: Widgets](/reference/intelligent-apps/components#widgets) for the component contract.
```yaml theme={null}
widgets:
- name: string # Unique widget name
title: string # Widget title (metadata; the widget renders its own header)
description: string # (Optional) Description
source_type: string # `inline` (default) or `file`
source: string # (Required for `inline`) Component TSX source
file: string # (Required for `file`) Path to a .tsx file in the project repository
query: string # (Optional) SQL query (Jinja supported); result is passed as the `data` prop
```
Widget attributes:
| Attribute | Type | Description |
| ------------- | ------ | ------------------------------------------------------------------------------------- |
| `name` | string | Unique widget name, referenced from the layout |
| `title` | string | Widget title (metadata only — no header is rendered for you) |
| `description` | string | (Optional) Widget description |
| `source_type` | string | `inline` or `file` (default: `inline`) |
| `source` | string | (Required for `inline`, forbidden for `file`) Component TSX source |
| `file` | string | (Required for `file`, forbidden for `inline`) Path to a `.tsx` file in the repository |
| `query` | string | (Optional) SQL query whose rows are passed to the component as the `data` prop |
Layout usage with a widget item:
```yaml theme={null}
layout:
tabs:
- name: overview
title: Overview
items:
- type: widget # Layout item type must be 'widget'
name: my_widget # References a component from components.widgets
span: 6 # (Optional) Grid span 1–12
height: 320 # (Optional) Height in px; enables scroll when set
```
Widget layout item properties:
| Attribute | Type | Description |
| --------- | ------ | ------------------------------------------------------------- |
| `type` | string | Must be `widget` |
| `name` | string | References a component from `components.widgets` |
| `span` | int | (Optional) Grid span (width) |
| `height` | int | (Optional) Height in px; when set, content becomes scrollable |
Example:
```yaml theme={null}
widgets:
- name: top_customers_widget
title: Top Customers
source_type: file
file: widgets/TopCustomers.tsx
query: |
SELECT CustomerName as customer, SUM(OrderLineTotalAmount) as revenue
FROM consolidated_sales_df
WHERE 1=1
{% if region_filter is defined %}
AND CustomerRegion = '{{ region_filter }}'
{% endif %}
GROUP BY CustomerName
ORDER BY revenue DESC
LIMIT 5
```
The component must `export default` a React component. It receives `data` (query rows, or an empty array when no `query` is set) and `appContext` (`filters`, `setFilter`, `theme`). Only `react` and `@datazone/widget-sdk` can be imported.
### Item Lists
List-style chart type. Each row supports icon, title, description, optional timestamp, and badge.
**Query result fields:**
| Field | Type | Description |
| --------------- | ------ | ------------------------------------------------------------------ |
| `icon` | string | (Optional) Lucide icon name (default: `Mail`) |
| `icon_color` | string | (Optional) Tailwind color class (default: `text-blue-600`) |
| `title` | string | Primary title |
| `description` | string | Secondary text |
| `timestamp` | string | (Optional) Right-aligned timestamp |
| `badge_text` | string | (Optional) Badge label |
| `badge_variant` | string | (Optional) Badge variant: `neutral`, `success`, `warning`, `error` |
```yaml theme={null}
charts:
- type: item_list
name: my_item_list
title: "Items"
query: |
SELECT
icon, -- e.g., 'Mail'
icon_color, -- e.g., 'text-blue-600'
title,
description,
updated_at AS timestamp,
status AS badge_text,
CASE
WHEN status = 'Success' THEN 'success'
WHEN status = 'Pending' THEN 'warning'
WHEN status = 'Error' THEN 'error'
ELSE 'neutral'
END AS badge_variant
FROM my_items
ORDER BY updated_at DESC
LIMIT 20;
```
***
Example:
```yaml theme={null}
config:
cache: true
cache_ttl: 3600
llm_instruction: "You are a helpful assistant."
hide_filters: false
hide_header: false
chart_export_enabled: true
style:
theme: "teal"
```
***
## Example
```yaml theme={null}
app:
name: Billing Collection Dashboard
description: Dashboard showing billing and collection metrics
config:
cache: true
cache_ttl: 3600
chart_export_enabled: true
style:
theme: "blue"
layout:
tabs:
- name: billing_data
title: Billing Data
filters: [product_group, membership_type_filter]
items:
- type: chart-group
name: kpis
span: 12
entities:
- type: chart
name: total_count
span: 4
- type: chart
name: detailed_table
span: 12
components:
charts:
- type: number
name: total_count
title: Total Row Count
query: select count(*) as row_count from sales_data;
metrics:
- name: row_count
label: Row Count
variables:
- name: order_date
type: date
filters:
- type: date
name: order_date_filter
title: Order Date Filter
affected_variable: order_date
default_value: '2024-04-01'
```
### Example for Number Chart Metrics
```yaml theme={null}
charts:
- type: number
name: completed_tasks
title: Completed Tasks
query: SELECT count(*) as completed FROM tasks WHERE status = 'done';
metrics:
- name: completed
label: Completed Tasks
icon: check
icon_variant: success
format: "0"
```
* `number` charts support 1-2 metrics: one main metric and optionally one secondary metric (submetric)
* `radial` charts support the same metric configuration as `number` charts
* Use `submetric_name` and `submetric_type` to display a secondary metric below the main value
* `submetric_type` options: `plain` (just show the value), `change` (show percentage change), `delta` (show difference)
* For a full list of Lucide icons, see [lucide.dev/icons](https://lucide.dev/icons/)
* `icon_variant` must be one of: `default`, `neutral`, `success`, `warning`, `error`
## Value Format Strings
All Intelligent App value formattings use [Numeral.js](http://numeraljs.com/) formatting strings. Examples:
| Number | Format | Result |
| ----------- | ------------ | ----------- |
| 10000 | `0,0.0000` | 10,000.0000 |
| 10000.23 | `0,0` | 10,000 |
| 10000.23 | `+0,0` | +10,000 |
| -10000 | `0,0.0` | -10,000.0 |
| 1000.234 | `$0,0.00` | \$1,000.23 |
| 1000.2 | `0,0[.]00 $` | 1,000.20 \$ |
| 1001 | `$ 0,0[.]00` | \$ 1,001 |
| 1230974 | `($ 0.00 a)` | \$ 1.23 m |
| 100 | `0b` | 100B |
| 1024 | `0b` | 1KB |
| 2048 | `0 ib` | 2 KiB |
| 3072 | `0.0 b` | 3.1 KB |
| 7884486213 | `0.00b` | 7.88GB |
| 1 | `0%` | 100% |
| 0.974878234 | `0.000%` | 97.488% |
| 0.43 | `(0.000 %)` | 43.000 % |
| 25 | `00:00:00` | 0:00:25 |
| 63846 | `00:00:00` | 17:44:06 |
***
## Notes
* All attributes are case-sensitive.
* Use Jinja templating for dynamic queries.
* See [Query Manipulation](./query-manipulation) for advanced usage.
* For more examples, see [example\_intelligent\_app.yaml](../../example_intelligent_app.yaml).
# Actions SDK
Source: https://docs.datazone.co/reference/knowledge-objects/actions-sdk
Read and write Knowledge Object instances from within an action.
# Knowledge Objects in actions
[Actions](/reference/development/actions) can read and write Knowledge Object instances through the `KnowledgeObject` client, exposed by the action SDK. It calls the same governed instance operations as the [REST API](/reference/knowledge-objects/api) — validation, options, filters, and versioning all apply — and runs as the action's user, on the action's project.
```python theme={null}
from datazone.actions import KnowledgeObject
employee = KnowledgeObject("Employee").get(key="8A3F9C2E1B4D6F7A8A3F9C2E1B4D6F7A")
print(employee["email"])
```
Instances are addressed by their opaque **`_key`** (returned by `get`, `list`, and create). Operations run on the **main branch** by default; pass `branch=` to target another branch.
## Constructing a client
```python theme={null}
KnowledgeObject(name, branch=None)
```
* **`name`** — the object's name (e.g. `"Employee"`).
* **`branch`** — (optional) branch to read/write on. Defaults to the project's main branch.
```python theme={null}
employees = KnowledgeObject("Employee") # main
employees = KnowledgeObject("Employee", branch="dev") # a feature branch
```
## Methods
### get
```python theme={null}
get(key, add_relationships=False) -> dict
```
Fetch a single instance by its `_key`. Pass `add_relationships=True` to also resolve the object's relationship fields (one level deep) under an `_relationships` key.
```python theme={null}
employee = KnowledgeObject("Employee").get(key="8A3F...")
company = KnowledgeObject("Company").get(key="AB12...", add_relationships=True)
print(company["_relationships"]["owner"]) # resolved related instance
```
### list
```python theme={null}
list(filters=None, page=1, page_size=50, fields=None) -> iterator of pages
```
Returns an iterator over **pages**; each page exposes `.items` (a list of instance dicts) and `.total_count`. Pages are fetched lazily as you iterate.
* **`filters`** — a list of `{"column", "operator", "value"}` objects, combined with **AND**. Operators: `equal`, `not_equal`, `contains`, `not_contains`, `greater_than`, `less_than`.
* **`fields`** — restrict the columns returned (the meta fields `_key` and `_version` are always included).
```python theme={null}
employees = KnowledgeObject("Employee")
for page in employees.list(
filters=[
{"column": "gender", "operator": "equal", "value": "female"},
{"column": "age", "operator": "greater_than", "value": 30},
],
fields=["name", "email"],
):
for employee in page.items:
print(employee["name"], employee["email"])
```
### update
```python theme={null}
update(key, payload) -> dict
```
Partially update an instance by its `_key`; returns the full updated instance. Primary key fields cannot be changed.
```python theme={null}
KnowledgeObject("Employee").update(key="8A3F...", payload={"email": "new@acme.com"})
```
### delete
```python theme={null}
delete(key) -> None
```
Delete an instance by its `_key`.
```python theme={null}
KnowledgeObject("Employee").delete(key="8A3F...")
```
### batch\_upsert
```python theme={null}
batch_upsert(payloads) -> dict
```
Insert or update up to **1000** instances in one call. A payload whose primary key does not exist is inserted; one that already exists is updated (a new version). Returns a summary.
```python theme={null}
result = KnowledgeObject("Employee").batch_upsert([
{"id": 1, "name": "John", "email": "john@acme.com"},
{"id": 2, "name": "Jane", "email": "jane@acme.com"},
])
# result -> {"created": 1, "updated": 1, "total": 2}
```
## Full example
```python theme={null}
from datazone.actions import action, context, KnowledgeObject
@action
def deactivate_inactive_employees():
"""Mark all active employees as inactive."""
employees = KnowledgeObject("Employee")
updated = 0
for page in employees.list(
filters=[{"column": "is_active", "operator": "equal", "value": True}]
):
for employee in page.items:
context.log_info(f"Deactivating {employee['name']}")
employees.update(key=employee["_key"], payload={"is_active": False})
updated += 1
return {"updated": updated}
```
## Next steps
* [Actions](/reference/development/actions) — writing and deploying actions.
* [Instance API](/reference/knowledge-objects/api) — the REST equivalent of these operations.
* [YAML Reference](/reference/knowledge-objects/yaml-reference) — defining objects and their fields.
# Instance API
Source: https://docs.datazone.co/reference/knowledge-objects/api
REST API for managing Knowledge Objects and their instances.
# Knowledge Object API
Every Knowledge Object is backed by a REST API for managing its **instances** — the rows of its materialized table. This page covers the object endpoints (create, list, read) and the full instance CRUD API.
* **Base path:** `/knowledge-object`
* **Auth:** standard logged-in session, same as every other app endpoint.
* **Permissions:** resource type `knowledge_object` under the project hierarchy — read endpoints require `read`, instance writes require `write`.
## Branching
Every endpoint takes an optional **`branch` query parameter, defaulting to `main`**. Objects are stored in your repository, so each branch has its own definition and its own ClickHouse table:
* The object id in the URL is **stable across branches** — only the `branch` param changes when you switch.
* Object reads return `definition: null` when the object is not deployed on the requested branch.
* Instance calls return `404` when the object is not deployed on the requested branch.
* **Instance operations require the branch's definition to be `READY`** — otherwise they return `400`. Gate instance UIs on `definition.status === "READY"`.
See [Overview → Branching](/reference/knowledge-objects/overview#branching) for the full model.
***
## Object endpoints
### Create object
Creates an object by writing its YAML file to the repository and deploying it — the object and its definition are then materialized by the loader, exactly as if you had committed the file yourself.
```http theme={null}
POST /knowledge-object/create
Content-Type: application/json
{
"metadata": {
"name": "Company",
"description": "A company.",
"fields": [
{ "name": "id", "type": "int", "primary_key": true },
{ "name": "name", "type": "str" },
{ "name": "owner", "type": "str", "optional": true, "relationship": [{ "object": "Person" }] }
],
"settings": { "label_column": "name", "icon": "building" }
},
"project": "6748d8b66a5a2ac9f0d39326",
"branch": "main"
}
```
| Field | Type | Description |
| ---------- | ------ | ---------------------------------------------------------------------------------------------------------------- |
| `metadata` | object | The object definition ([YAML reference](/reference/knowledge-objects/yaml-reference), as JSON). Fully validated. |
| `project` | string | Id of the project to create the object in. |
| `branch` | string | (Optional) Branch to deploy on. Defaults to `main`. |
* `metadata` is validated exactly like a deployed YAML file (primary key required, relationship rules, `label_column` must name a field, …) — an invalid definition returns `400`.
* The server writes `objects/_.yaml`, registers it in the project's `config.yaml`, and deploys on `branch`. The definition starts at `PENDING_MIGRATION` and is materialized asynchronously.
* **Response:** `200` with no body. Poll [Get object](#get-object) or [List objects](#list-objects) for the definition and its `status`.
* `400` if an object with the same name already exists in the project.
### Update object
```http theme={null}
PUT /knowledge-object/update/{id}
Content-Type: application/json
{ "metadata": { /* name, description, fields, settings */ }, "branch": "main" }
```
Updates the object's metadata **on one branch** by rewriting its YAML file and redeploying.
* Send the full `metadata` (same shape as create). `branch` is optional (defaults to `main`).
* **Renaming is not supported** — `metadata.name` must equal the current name (`400` otherwise).
* **Changing the primary key** (columns or types) is rejected (`400`) — the instance key derives from it.
* A schema change re-runs migration (definition returns to `PENDING_MIGRATION` → `READY`); metadata/settings-only changes apply in place.
* **Response:** `200`, no body. Poll [Get object](#get-object) for the updated definition and `status`.
### Delete object
```http theme={null}
DELETE /knowledge-object/delete/{id}?branch=main
```
Removes the object's **definition on the given branch only** (other branches keep theirs; `branch` defaults to `main`).
* The object is removed from its YAML file (the file and its `config.yaml` entry are dropped when it was the only object in the file) and redeployed. The branch definition is deleted, and the object identity is deleted once no branch has a definition.
* A background task then drops that branch's ClickHouse table and view.
* **Response:** `204`, no body.
### List objects
```http theme={null}
GET /knowledge-object/list?branch=main
```
Standard list parameters — `filters`, `page`, `page_size`, `sort_by` — plus `branch` (default `main`). Filters apply to the object identity (e.g. `name`, `project.$id`); the definition for the selected branch is attached to each item.
```http theme={null}
# Objects of one project, with their feature-x definitions
GET /knowledge-object/list?filters=[project.$id][$eq]:6748d8b66a5a2ac9f0d39326&branch=feature-x
```
```json theme={null}
{ "total_count": 3, "items": [ { "id": "…", "name": "Employee", "branches": ["main"], "definition": { } } ] }
```
Only objects deployed on the selected branch are returned.
### Get object
```http theme={null}
GET /knowledge-object/get-by-id/{id}?branch=main
```
Returns the object identity plus the definition for the requested branch. `definition` is `null` when the object is not deployed on that branch.
```json theme={null}
{
"id": "6863f1a2c9d4b2a7e1f00001",
"name": "Employee",
"project": { "id": "6748d8b66a5a2ac9f0d39326", "collection": "project" },
"branches": ["feature-x", "main"],
"definition": {
"id": "6863f1a2c9d4b2a7e1f00002",
"branch": "main",
"file_path": "objects/employee.yaml",
"status": "READY",
"error_message": null,
"table_name": "object_main_employee__raw",
"view_name": "object_main_employee",
"metadata": {
"name": "Employee",
"description": "Employee object.",
"settings": { "object_type": "STANDALONE", "store_versions": false, "icon": "users", "label_column": "name" },
"fields": [
{ "name": "id", "type": "int", "primary_key": true, "optional": false, "mutable": true, "default": null, "relationship": null },
{ "name": "name", "type": "str", "primary_key": false, "optional": false, "mutable": true },
{ "name": "owner", "type": "str", "optional": true, "relationship": [{ "object": "Person" }] }
],
"actions": []
}
}
}
```
Use `definition.metadata.fields` to build instance forms and table columns, and `definition.view_name` to query the object in the SQL editor. The view exposes the object fields plus `__version`, `__timestamp`, and `__primary_key` (the instance key as hex).
***
## Instance endpoints
All instance endpoints are nested under an object id: `/knowledge-object/{id}/instances`. `{id}` is the object's id (not its name). Each accepts `branch` (default `main`) and operates on that branch's table.
### List instances
```http theme={null}
GET /knowledge-object/{id}/instances?page=1&page_size=50&branch=main
```
| Param | Type | Default | Description |
| ----------- | --------------- | -------- | -------------------------------------- |
| `page` | int | `1` | Page number. |
| `page_size` | int | `50` | Rows per page. |
| `branch` | string | `main` | Branch to read from. |
| `fields` | repeated string | *(all)* | [Column selection](#column-selection). |
| `filters` | repeated JSON | *(none)* | [Row filtering](#row-filtering). |
```json theme={null}
{
"total_count": 128,
"items": [
{ "_key": "8A3F9C2E1B4D6F7A8A3F9C2E1B4D6F7A", "id": 1, "name": "john", "last_login": "2026-07-01T09:30:00Z", "_version": 1 }
]
}
```
Each item is the selected object fields plus the meta fields `_key` and `_version`.
#### Column selection
Use the repeated `fields` query parameter to fetch only the columns you need — for example a dropdown that shows the label column and stores the primary key:
* **omitted** → all object fields (default).
* `?fields=id&fields=name` → only those fields, plus `_key` and `_version`.
* `?fields=` (present but empty) → only the meta fields `_key` and `_version`.
* An unknown field name → `400` (`detail.unknown_fields`).
The meta fields `_key` and `_version` are always included.
#### Row filtering
Use the repeated `filters` query parameter to filter rows. Each value is a JSON object `{ "column", "operator", "value" }`, and multiple filters are combined with **AND**. Filtering affects both the returned rows and `total_count`.
```http theme={null}
GET /knowledge-object/{id}/instances
?filters={"column":"is_active","operator":"equal","value":true}
&filters={"column":"salary","operator":"greater_than","value":50000}
```
| `operator` | Meaning |
| -------------- | ------------------------------ |
| `equal` | column = value |
| `not_equal` | column ≠ value |
| `greater_than` | column > value |
| `less_than` | column \< value |
| `contains` | case-sensitive substring match |
| `not_contains` | negated substring match |
* `column` must be an object field name → an unknown column returns `400` (`detail.unknown_filter_column`).
* `value` is **not** type-checked against the field; it is sent to the database as a safely-escaped literal.
* A filter that isn't valid JSON returns `400` (`detail.invalid_filter`).
* Combine freely with `fields`, `page`, and `page_size`.
### Create instance
```http theme={null}
POST /knowledge-object/{id}/instances
Content-Type: application/json
{ "id": 1, "name": "john", "last_login": null }
```
* The payload is validated against the object's field definitions (types, required, nullable).
* Optional fields, and fields with a `default`, may be omitted — the database fills in defaults (including functional ones like `now()`).
**Response:** `201 Created` with the created instance (its `_key` is in the body).
```json theme={null}
{ "_key": "8A3F9C2E1B4D6F7A8A3F9C2E1B4D6F7A", "id": 1, "name": "john", "last_login": null }
```
In the UI, instances are created and edited through a form generated from the object's fields:
* `409` if an instance with the same primary key values already exists.
* Creating with the primary key of a previously **deleted** instance succeeds — the instance is revived under the same `_key`.
### Batch upsert instances
Insert or update many instances in a single request — useful for bulk imports and syncing.
```http theme={null}
POST /knowledge-object/{id}/instances/batch?branch=main
Content-Type: application/json
[
{ "id": 1, "name": "john" },
{ "id": 2, "name": "jane", "last_login": "2026-07-01T09:30:00Z" },
{ "id": 3, "name": "amir" }
]
```
The request body is a **JSON array of instance payloads** (each shaped exactly like a single [create](#create-instance) payload). Semantics are **upsert**:
* If an instance with the same primary key does **not** exist, it is inserted as a new instance (version `1`).
* If it **already exists**, a new version is appended with an incremented version number (the previous value is kept in [history](#instance-history)). A previously **deleted** instance is revived the same way.
* Optional fields and fields with a `default` may be omitted per item — the database fills them in.
Constraints:
* **Maximum 1000 items per request** — more returns `400` (`detail.max_batch_size`, `detail.received`).
* The **whole batch is validated before any write**. If any item is invalid, nothing is written and the response is `400` with the offending item's index (`detail.index`, `detail.errors`).
* A primary key that appears **more than once within the same batch** is rejected with `400` (`detail.index`, `detail.error`).
**Response:** `200` with a summary of what was applied.
```json theme={null}
{ "created": 2, "updated": 1, "total": 3 }
```
To retrieve the resulting instances (with their `_key`s), read them back via [List](#list-instances).
### Get instance
```http theme={null}
GET /knowledge-object/{id}/instances/{key}?branch=main&add_relationships=false
```
Returns the instance, or `404` if it does not exist or was deleted.
**Relationships** — pass `add_relationships=true` to resolve the object's relationship fields. Each relationship field keeps its raw `_key` value, and a parallel `_relationships` object is added, keyed by field name, holding the resolved related instance (or `null` when the key is empty, the target object is not deployed/ready on the branch, or the related instance was deleted). Resolution is **one level deep**. Default is `false` (no `_relationships` key, no extra queries).
```json theme={null}
{
"_key": "AB12...",
"id": 5,
"name": "Acme",
"owner": "9F3C...",
"_relationships": {
"owner": { "_key": "9F3C...", "id": 1, "name": "John", "email": "j@acme.com" }
}
}
```
### Instance history
```http theme={null}
GET /knowledge-object/{id}/instances/{key}/history?branch=main
```
Returns **all versions** of an instance, newest first — including deletion rows, and even for instances that are currently deleted (so it works where a plain `GET` returns `404`).
```json theme={null}
[
{ "_key": "8A3F…", "id": 1, "name": "John Smith", "_version": 2, "_timestamp": "2026-07-06T10:00:00", "_deleted": false, "_user_id": "6748d8…" },
{ "_key": "8A3F…", "id": 1, "name": "john", "_version": 1, "_timestamp": "2026-07-05T09:30:00", "_deleted": false, "_user_id": "6748d8…" }
]
```
Each item is the object fields plus `_version`, `_timestamp`, `_deleted`, and `_user_id` (empty string when not written by a user). `404` if no instance ever existed for the key.
For objects with `store_versions: false`, superseded versions may be compacted away over time — history is best-effort in that mode.
### Update instance
```http theme={null}
PATCH /knowledge-object/{id}/instances/{key}
Content-Type: application/json
{ "name": "John Smith" }
```
* Send only the fields to change (partial update).
* Primary key fields in the payload → `400` (they are immutable; a different primary key is a different instance).
* Immutable fields (`mutable: false`) cannot be changed after creation.
* Returns the full updated instance.
### Delete instance
```http theme={null}
DELETE /knowledge-object/{id}/instances/{key}
```
`204 No Content`. A subsequent `GET` on the key returns `404`. Deletion is logical — history is kept internally and the key can be revived by creating an instance with the same primary key.
***
## Errors
| Status | When | Detail |
| ------ | ---------------------------------------- | ------------------------------------------------------------------------- |
| `400` | Definition not `READY` on the branch | Instance operations require a completed migration. |
| `400` | Payload fails validation | `detail.errors` carries the validation message (`detail.index` on batch). |
| `400` | Batch larger than 1000 items | `detail.max_batch_size`, `detail.received`. |
| `400` | Duplicate primary key within a batch | `detail.index`, `detail.error`. |
| `400` | Primary key field in a `PATCH` payload | `detail.fields` lists the offending fields. |
| `400` | Value outside a field's `options` | `detail.invalid_options` maps field → rejected value. |
| `400` | Unknown `fields` value | `detail.unknown_fields` lists the unknown fields. |
| `400` | Unknown filter column | `detail.unknown_filter_column`. |
| `400` | Malformed `filters` JSON | `detail.invalid_filter`. |
| `400` | Object with the same name already exists | On create object. |
| `404` | Object id not found / other project | Standard not-found. |
| `404` | Object not deployed on the branch | `detail.branch`. |
| `404` | Instance missing or deleted | — |
| `409` | Create with existing primary key | `detail.key` is the existing `_key`. |
***
## Not available yet
* Executing object actions via the API (`POST /{id}/instances/{key}/actions/{action}`).
* Backed objects (`object_type: BACKED`).
# Overview
Source: https://docs.datazone.co/reference/knowledge-objects/overview
Define business entities as versioned objects and manage their instances through a governed API
# Knowledge Objects
Knowledge Objects let you model the business entities of your domain — `Employee`, `Invoice`, `Company`, `Ticket` — as declarative YAML definitions in your project repository. On deploy, Datazone stores each definition as metadata and materializes a ClickHouse table behind it. Every row in that table is an **instance**, created, read, updated, and deleted through a governed REST API.
Unlike an Extract (which ingests data from an external source) or a Pipeline (which transforms datasets), a Knowledge Object is a **first-class, hand-curated entity**: its schema is versioned in Git, its instances are edited by users or agents, and every change is tracked.
## What you get
* **Declarative schema** — fields, types, primary keys, defaults, and relationships defined in YAML and validated on deploy.
* **A materialized table + view** — each object is backed by a database table you can query in the SQL editor and join with the rest of your data.
* **A CRUD API** — list, create, read, update, and delete instances, with pagination, column selection, and filtering.
* **Versioning** — every write appends a new version; the API always serves the latest, and full history is available.
* **Relationships** — a field can point at another object's instance, resolved on demand.
* **Branch awareness** — objects live in your repository, so they follow the same branch model as the rest of your project.
## How it works
An object is a YAML file in your repository, registered in `config.yaml`, exactly like a pipeline or an endpoint:
1. Create a YAML file under `objects/` (e.g. `objects/employee.yaml`) defining the object.
2. Register the file in your project's `config.yaml`.
3. Deploy. Datazone parses the definition, stores it, and enqueues a migration that builds the ClickHouse table and view.
4. Once the migration completes, the object becomes `READY` and its instance API is live.
```yaml objects/employee.yaml theme={null}
name: Employee
description: A person employed by the company.
fields:
- name: id
type: int
primary_key: true
- name: name
type: str
- name: email
type: str
optional: true
- name: hired_at
type: datetime
default: now()
settings:
icon: users
label_column: name
```
You can also create an object directly from the UI (or the `POST /knowledge-object/create` endpoint). Datazone writes the YAML file, registers it in `config.yaml`, and deploys it for you — the object is still materialized by the same loader, so the outcome is identical to committing the file yourself.
## Core concepts
### Object vs. Definition
A Knowledge Object is **one identity per (project, object name)** — its id is stable no matter which branch you are on. Deploying the object to a branch produces a **definition** for that branch, and each definition carries its own schema, migration status, and ClickHouse table/view.
```
KnowledgeObject "Employee" (stable id, shared across branches)
├── definition @ main → READY, table object_main_employee
└── definition @ feature-x → PENDING, table object_feature_x_employee
```
This means you can evolve an object's schema on a feature branch — add a field, change a default — and try it in isolation before merging to `main`. See [Branching](#branching) below.
Relationships between objects form a graph you can explore visually:
### Instances and the instance key
Each row is an **instance**, addressed by an opaque **instance key** (`_key`): a hex string derived server-side from the instance's primary key values. The key is stable across updates — the same primary key values always map to the same `_key`, so an instance keeps its URL for life.
### Versioning
Updates and deletes never modify a row in place; they append a new version. The API always serves the latest non-deleted version, so you can treat instances as plain mutable records — while still being able to read the full [history](/reference/knowledge-objects/api#instance-history) of any instance.
## Branching
Because objects are stored in your repository, they follow your project's branch model end to end:
* The **object identity** (and its id) is shared across branches.
* Each **branch has its own definition** — schema, status, and a dedicated ClickHouse table (`object__`), so instance data is isolated per branch.
* Every API call takes an optional **`branch` query parameter, defaulting to `main`**. The object id in the URL never changes when you switch branches — only the `branch` param does.
* Object responses include a **`branches`** array listing every branch the object is deployed on, which powers the branch switcher in the UI.
* If an object is not deployed on the requested branch, object reads return `definition: null` and instance calls return `404`.
Working on a feature branch is the safe way to change an object's schema. Deploy the change to your branch, verify the migration and instances there, then merge to `main`.
## Next steps
* [YAML Reference](/reference/knowledge-objects/yaml-reference) — every attribute you can declare on an object.
* [Instance API](/reference/knowledge-objects/api) — the full REST API for managing objects and their instances.
* [Actions SDK](/reference/knowledge-objects/actions-sdk) — read and write instances from within an [action](/reference/development/actions).
# YAML Reference
Source: https://docs.datazone.co/reference/knowledge-objects/yaml-reference
Comprehensive reference for every attribute in a Knowledge Object definition.
# Knowledge Object YAML Reference
This page documents every attribute available in a Knowledge Object YAML definition. Each file declares **one object per YAML document**; you can define multiple objects in one file by separating them with `---`.
## Top-level structure
```yaml theme={null}
name: string # Object name, unique per project (required)
description: string # (Optional) Human-readable description
fields: # Field definitions (required, at least one)
settings: # (Optional) Object settings
actions: # (Optional) Action definitions
```
A minimal object needs a `name` and at least one field, and at least one of those fields must be a primary key:
```yaml theme={null}
name: Tag
fields:
- name: slug
type: str
primary_key: true
- name: label
type: str
```
***
## `fields`
The list of columns that make up the object. Every field is validated on deploy; an unknown attribute or an unsupported type fails the deploy.
| Attribute | Type | Default | Description |
| -------------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------- |
| `name` | string | — | Field name (required). |
| `type` | string | — | Field type (required). One of the [supported types](#field-types). |
| `primary_key` | bool | `false` | Whether the field is part of the object's primary key. |
| `optional` | bool | `false` | Whether the field is nullable / may be omitted on create. |
| `mutable` | bool | `true` | Whether the value can be changed after creation. |
| `default` | any | `null` | Default value applied at the database level when the field is omitted. |
| `db_column` | string | `null` | (Optional) Override the underlying database column name. |
| `db_type` | string | `null` | (Optional) Override the underlying database column type. |
| `relationship` | list | `null` | (Optional) Declares the field as a [relationship](#relationships) to another object. |
| `options` | list | `null` | (Optional, `str` only) Allowed values; renders a dropdown and is validated on write. See [Options](#options). |
### Field types
| Type | Description |
| ---------- | --------------------- |
| `int` | Integer |
| `str` | String |
| `float` | Floating-point number |
| `bool` | Boolean |
| `datetime` | Timestamp |
| `date` | Calendar date |
By default each type maps to a sensible database column type. Use [`db_type`](#custom-database-types) to override the exact type.
### Primary keys
At least one field must have `primary_key: true`. The combination of primary key fields uniquely identifies an instance, and it is used to compute the stable instance key (`_key`).
* Primary key fields **cannot be optional**.
* Primary key fields are **immutable** — attempting to change one through the API is an error. A different primary key is a different instance.
```yaml theme={null}
fields:
- name: order_id
type: int
primary_key: true
- name: line_no
type: int
primary_key: true # composite primary key
- name: quantity
type: int
```
### Defaults
`default` sets a **database-level default** that the database applies whenever the field is omitted from an insert. This means the default is enforced regardless of how the instance is created — through the API, or directly against the table.
Because it is a database expression, you can use functional defaults:
```yaml theme={null}
fields:
- name: created_at
type: datetime
default: now() # evaluated by the database on insert
- name: status
type: str
default: active
- name: is_active
type: bool
default: true
```
A field with a `default` may be omitted when creating an instance — the database fills it in. Functional defaults such as `now()` can only be produced at insert time, so the client should leave the field out rather than sending a value.
### Mutability
Set `mutable: false` to freeze a field after the instance is created. Non-primary-key fields are mutable by default.
```yaml theme={null}
fields:
- name: external_ref
type: str
mutable: false # set once at creation, never updated
```
### Custom database columns and types
Use `db_column` and `db_type` to control the underlying database storage independently of the field name and logical type:
```yaml theme={null}
fields:
- name: amount
type: float
db_type: Decimal(18, 2) # store as an exact decimal
- name: label
type: str
db_column: display_label # store under a different column name
```
### Relationships
A relationship field points at **another object's instance**. It is metadata only — there is no database-level foreign key. The field itself is a plain string that stores the target instance's key (`_key`).
* The field's `type` must be `str`.
* `relationship` must reference **exactly one** target object, by name.
```yaml theme={null}
name: Company
fields:
- name: id
type: int
primary_key: true
- name: name
type: str
- name: owner
type: str
optional: true
relationship:
- object: Person # `owner` stores the _key of a Person instance
```
At read time you can ask the API to resolve relationships (one level deep) via the `add_relationships` flag — see the [Instance API](/reference/knowledge-objects/api#get-instance). In the UI, a relationship field is rendered as a picker over the target object's instances, using the target's [`label_column`](#settings) for display.
### Options
A `str` field can declare a fixed set of allowed values with `options`. The UI renders it as a dropdown instead of a free-text input, and the API rejects any create/update whose value is outside the list.
```yaml theme={null}
fields:
- name: gender
type: str
options:
- male
- female
- other
```
* `options` is only valid on `str` fields, and must not be empty.
* Writing an instance with a value not in `options` returns `400`.
***
## `settings`
Object-level settings. All are optional.
| Attribute | Type | Default | Description |
| ---------------- | ------ | ------------ | ----------------------------------------------------------------------- |
| `object_type` | string | `STANDALONE` | `STANDALONE` (managed instances) or `BACKED` (backed by a dataset). |
| `backed_dataset` | string | `null` | Alias of the backing dataset — required when `object_type` is `BACKED`. |
| `store_versions` | bool | `false` | Whether previous instance versions are retained. |
| `icon` | string | `null` | Icon name used to represent the object in the UI. |
| `label_column` | string | `null` | Which field to display as an instance's human label in the UI. |
```yaml theme={null}
settings:
icon: users
label_column: name
store_versions: true
```
* **`label_column`** must name one of the object's fields. It is used wherever an instance is shown by name — most importantly in relationship pickers.
* **`icon`** is purely for UI presentation.
* **`store_versions`** keeps superseded versions of an instance available in history. With it off, old versions may be compacted away over time (history becomes best-effort).
`BACKED` objects (`object_type: BACKED`, backed by a dataset) are not yet generally available. Define `STANDALONE` objects — the default — for managed instances.
***
## `actions`
Actions bind a named operation on the object to a Python handler function in your repository.
| Attribute | Type | Description |
| ------------- | ------ | ----------------------------------------------------------------- |
| `name` | string | Action name (required). |
| `description` | string | (Optional) Human-readable description. |
| `handler` | string | Reference to the handler, in the form `path/to/file.py:function`. |
```yaml theme={null}
actions:
- name: send_welcome_email
description: Send a welcome email to the employee.
handler: objects/employee_actions.py:send_welcome_email
```
The `handler` must point at a Python file and function in your repository. Changing an object's actions does **not** trigger a schema migration — only field and setting changes do.
Executing actions via the API is on the roadmap; today, actions are declared and validated as part of the object definition.
***
## Full example
```yaml objects/employee.yaml theme={null}
name: Employee
description: A person employed by the company.
fields:
- name: id
type: int
primary_key: true
- name: name
type: str
- name: email
type: str
optional: true
- name: department
type: str
optional: true
relationship:
- object: Department
- name: salary
type: float
db_type: Decimal(18, 2)
- name: hired_at
type: datetime
default: now()
- name: is_active
type: bool
default: true
settings:
icon: users
label_column: name
store_versions: true
actions:
- name: send_welcome_email
description: Send a welcome email to the employee.
handler: objects/employee_actions.py:send_welcome_email
```
***
## Deploying multiple objects
To define more than one object in a single file, separate documents with `---`:
```yaml objects/hr.yaml theme={null}
name: Department
fields:
- name: id
type: int
primary_key: true
- name: name
type: str
settings:
label_column: name
---
name: Employee
fields:
- name: id
type: int
primary_key: true
- name: department
type: str
relationship:
- object: Department
```
Each object still becomes its own identity, definition, and database table. Object names must be unique within the file and within the project.
## Next steps
* [Overview](/reference/knowledge-objects/overview) — concepts, branching, and the lifecycle.
* [Instance API](/reference/knowledge-objects/api) — managing instances of your objects.
# Channels
Source: https://docs.datazone.co/reference/platform/channels
Configure notification channels for sending reports and alerts
## Overview
Channels provide a way to integrate Datazone with external communication platforms. You can configure channels to send scheduled reports, alerts, and notifications through Slack or email.
## Supported Channel Types
### Slack Integration
To set up a Slack channel, you need to create a Slack App and obtain a Bot Token.
#### Step 1: Create a Slack App
1. Go to [https://api.slack.com/apps](https://api.slack.com/apps)
2. Click **"Create New App"**
3. Choose **"From scratch"**
4. Enter an app name (e.g., "Datazone")
5. Select your workspace
6. Click **"Create App"**
#### Step 2: Add Bot Token Scopes
1. In your app settings, go to **"OAuth & Permissions"**
2. Scroll to **"Bot Token Scopes"**
3. Add the following required scopes:
* `channels:read` - View basic information about public channels
* `chat:write` - Send messages as your app
* `files:write` - Upload files as your app
#### Step 3: Install App to Workspace
1. Scroll to **"OAuth Tokens for Your Workspace"**
2. Click **"Install to Workspace"**
3. Review permissions and click **"Allow"**
4. Copy the **Bot User OAuth Token** (starts with `xoxb-`)
#### Step 4: Add App to Channels
For each Slack channel where you want to receive notifications:
1. Open the channel in Slack
2. Click the channel name at the top
3. Go to **"Integrations"** tab
4. Click **"Add an App"**
5. Select your Datazone app
#### Required Credential
* **Bot User OAuth Token** - The `xoxb-` token from Step 3
### Email Integration
To set up an Email channel, you need SMTP server credentials.
#### Required Credentials
* **SMTP Host** - Your SMTP server address (e.g., `smtp.gmail.com`)
* **SMTP Port** - Server port (typically `587` for TLS or `465` for SSL)
* **Email Address** - The email address to send from
* **Username** - SMTP authentication username (often same as email address)
* **Password** - SMTP authentication password or app-specific password
* **Use TLS** - Enable TLS/SSL encryption (recommended)
#### Common SMTP Settings
**Gmail:**
* Host: `smtp.gmail.com`
* Port: `587`
* Use TLS: `Yes`
* Note: Use an [App Password](https://support.google.com/accounts/answer/185833) instead of your regular password
**Outlook/Office 365:**
* Host: `smtp.office365.com`
* Port: `587`
* Use TLS: `Yes`
**Other Providers:**
Check your email provider's documentation for SMTP settings.
## Using Channels
Once you've configured a channel with the credentials above, you can use it to send [Reports](/reference/platform/reports) to your team via Slack or email.
## Security
All channel credentials (OAuth tokens, SMTP passwords) are encrypted at rest. Datazone uses industry-standard encryption to protect your sensitive integration data.
# Reports
Source: https://docs.datazone.co/reference/platform/reports
Schedule and automate intelligent app report delivery
## Overview
Reports enable you to automatically generate and send snapshots of your Intelligent Apps on a schedule. Reports are rendered as PDF, PNG, or Excel files and delivered through configured [Channels](/reference/platform/channels) via Slack or email.
## Key Features
* **Scheduled Delivery**: Use cron expressions to define when reports run (daily, weekly, monthly, etc.)
* **Multiple Formats**: Export as PDF (multi-page), PNG (single image), or Excel (tabular data)
* **Channel Integration**: Send via Slack or email through configured channels
* **Flexible Targeting**: Send the same report to different destinations
* **Manual Triggers**: Test reports or send on-demand outside the schedule
## How Reports Work
1. **Create a Channel**: Set up Slack or email credentials (done once, reused for multiple reports)
2. **Configure a Report**: Select an Intelligent App, set a schedule, choose a format, and specify recipients
3. **Automatic Execution**: Datazone renders the app and sends it via your channel at the scheduled time
4. **Track History**: Review execution logs to monitor delivery status and timing
## Report Configuration
### Schedule
Reports use cron expressions to define when they run. Common examples:
* `0 9 * * *` - Every day at 9:00 AM
* `0 9 * * MON` - Every Monday at 9:00 AM
* `0 9 * * MON-FRI` - Every weekday at 9:00 AM
* `0 */6 * * *` - Every 6 hours
* `0 0 1 * *` - First day of every month at midnight
### Output Format
**PDF Format:**
* Combines multiple tabs into a single document
* Ideal for comprehensive reports
* Can select specific tabs to include
**PNG Format:**
* Single image snapshot
* Best for dashboards and visual summaries
* Faster generation time
**Excel Format:**
* Exports the underlying chart data instead of a rendered image
* Iterates over each tab in the Intelligent App and, for every chart, writes its data as a table on a separate sheet
* Best for recipients who need to analyze the raw numbers in a spreadsheet
* Each sheet is named after its chart, preserving the order in which tabs and charts appear in the app
# Resources Overview
Source: https://docs.datazone.co/reference/platform/resources/overview
Understanding Datazone resource types and their usage
## Overview
Datazone tracks several resource types to help you monitor and manage your platform usage. Understanding these resources is essential for capacity planning, cost management, and setting appropriate [Quotas](/reference/platform/resources/quotas).
## Resource Summary
| Resource | Description | Features Using It |
| ---------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| **DCU** | Datazone Compute Units - measures compute time (1 DCU = 1 vCPU for 1 hour) | Executions, Notebook kernels |
| **Tokens** | Total LLM tokens consumed for AI operations | Orion AI Code Assistant, Intelligent App Chat, Agents, Embeddings |
| **Hot Storage** | Fast relational database storage in megabytes | Materialized tables, Transformed datasets, Query results |
| **Cold Storage** | Long-term object storage in megabytes | Raw ingested data, Historical datasets, Archives |
| **Query** | Total data read by queries in bytes | All analytical queries and data scans |
| **Actions** | Number of action function executions | [Action](/reference/development/actions) endpoints, Agent tools |
## Resource Types
### DCU (Datazone Compute Units)
DCU measures compute resources consumed by your workloads.
**Definition:** 1 DCU = 1 vCPU for 1 hour
**Example:**
* An XSMALL instance (2 vCPU, 8GB RAM) running for 1 hour = **2 DCU**
* A SMALL instance (4 vCPU, 16GB RAM) running for 30 minutes = **2 DCU**
* A MEDIUM instance (8 vCPU, 32GB RAM) running for 15 minutes = **2 DCU**
**What uses DCU:**
* Executions
* Notebook kernels
### Tokens
Tokens measure the usage of Large Language Models (LLMs) in your Datazone workspace.
**Definition:** Total number of input and output tokens consumed by LLM operations
**What uses tokens:**
* Orion AI Code Assistant
* Intelligent App Chat
* Agent
* Embeddings
**Note:** Token consumption varies based on the complexity of prompts and length of responses. Longer conversations and detailed outputs consume more tokens.
### Hot Storage
Hot storage refers to data stored in the relational database for fast querying and analysis.
**Definition:** Storage measured in megabytes (MB)
**What uses hot storage:**
* Materialized tables from pipelines
* Transformed datasets
* Cached query results
* Metadata and system tables
**Characteristics:**
* High-performance SSD storage
* Optimized for fast analytical queries
* More expensive than cold storage
* Best for frequently accessed data
### Cold Storage
Cold storage refers to data stored in the lakehouse (object storage) for long-term retention and archival.
**Definition:** Storage measured in megabytes (MB)
**What uses cold storage:**
* Raw ingested data
* Historical datasets
* Archived pipeline outputs
* Backup and disaster recovery data
**Characteristics:**
* Cost-effective object storage
* Suitable for infrequent access
* Longer retrieval times compared to hot storage
* Ideal for compliance and long-term retention
### Query
Query measures the amount of data processed when executing analytical queries.
**Definition:** Total bytes of data read by queries
**What affects query usage:**
* Number of rows scanned
* Number of columns selected
* Use of filters and aggregations
* Query optimization
**Optimization Tips:**
* Use column pruning (select only needed columns)
* Apply filters early to reduce data scanned
* Leverage partitioning when available
* Use materialized views for frequently accessed aggregations
### Actions
Actions measure the number of serverless function executions in your workspace.
**Definition:** Total number of action function calls
**What uses actions:**
* [Action](/reference/development/actions) endpoints triggered via API
* Agent tools calling actions
* Manual action triggers
**How it's counted:**
Each time an action function is executed, it counts as 1 action usage, regardless of execution duration or complexity.
## Resource Planning
**For Development Environments:**
* Start with smaller compute instances (XSMALL/SMALL)
* Monitor DCU consumption during testing
* Use hot storage sparingly for prototypes
**For Production Environments:**
* Size compute resources based on workload patterns
* Balance hot vs. cold storage based on access frequency
* Set quotas to prevent unexpected resource consumption
* Review query patterns to optimize data scanning
## Cost Considerations
Resource usage directly impacts your Datazone costs:
* **DCU** scales with compute power and duration
* **Hot Storage** is more expensive per MB than cold storage
* **Query** costs increase with data scanned (optimize your queries)
* **Tokens** vary based on LLM usage frequency and complexity
* **Actions** are counted per execution (each function call)
Understanding your resource patterns helps optimize both performance and cost.
# Quotas
Source: https://docs.datazone.co/reference/platform/resources/quotas
Set limits and manage resource consumption with quotas
## Overview
Quotas allow you to set limits on resource consumption in your Datazone workspace. They help prevent unexpected usage, control costs, and ensure fair resource allocation across teams and projects.
## How Quotas Work
When you create a quota for a resource type (DCU, Tokens, Storage, Query, Action), Datazone monitors your usage and takes action based on consumption levels:
### Usage Thresholds
**80% - Warning Notification**
When usage reaches 80% of the quota limit, you receive an email notification alerting you that you're approaching the limit.
**100% - Service Pause**
When usage exceeds the quota limit, your services are automatically paused to prevent further consumption. You'll need to either increase the quota or reduce usage to resume operations.
## Setting Up Quotas
When you create a quota, you choose a **resource type**, a **scope**, and a **limit**. The scope determines how broadly the limit is enforced — across the whole organisation, within a single project, per user, or for a specific parent entity such as an intelligent app or agent.
### Scopes
A scope defines the boundary that a quota is measured and enforced against. Datazone supports the following scopes:
* **Organisation** - Applies the quota to the whole organisation. Usage is aggregated across every project, user, and app.
* **Project** - Limits usage only within a single project. You select the target project when creating the quota.
* **User** - Each user gets their own usage limit for the resource.
* **Parent Entity** - Limits usage for a specific app or agent (for example, an intelligent app or a single agent).
Not every resource supports every scope. The table below shows which scopes are available for each resource type:
| Resource | Organisation | Project | User | Parent Entity (App / Agent) |
| ---------------- | :----------: | :-----: | :--: | :-------------------------: |
| **Hot Storage** | ✅ | ✅ | — | — |
| **Cold Storage** | ✅ | ✅ | — | — |
| **DCU** | ✅ | ✅ | — | — |
| **Query** | ✅ | ✅ | ✅ | — |
| **Action** | ✅ | ✅ | ✅ | — |
| **Tokens** | ✅ | ✅ | ✅ | ✅ |
**Tokens** are the most flexible resource — they can be scoped all the way down to an intelligent app or an individual agent, in addition to organisation, project, and user levels.
#### Example: Project-scoped token limit
To cap LLM token consumption for a single project, choose the **Tokens** resource type, select the **Project** scope, pick the target project, and set a quota limit:
In this example, the `SalesProject` project is limited to 100,000 tokens. Usage in other projects is unaffected by this quota.
### Supported Resources
You can set quotas for any of the following [resource types](/reference/platform/resources/overview):
* **DCU** - Datazone Compute Units (compute time)
* **TOKENS** - LLM token consumption
* **HOT\_STORAGE** - Relational database storage (MB)
* **COLD\_STORAGE** - Lakehouse/object storage (MB)
* **QUERY** - Total data scanned by queries (MB)
* **ACTION** - Number of action function executions
# AWS S3 CSV
Source: https://docs.datazone.co/reference/sources/aws-s3
AWS S3 is a scalable object storage service that can be used to store and retrieve files.
# Overview
Amazon Simple Storage Service (S3) is an object storage service offering industry-leading scalability, availability, and durability. Datazone provides native integration with AWS S3 to read data files directly from your S3 buckets.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your AWS S3 source.
**Bucket Name** — the S3 bucket containing your data files.
**AWS Access Key ID** — the access key ID from your AWS credentials.
**AWS Secret Access Key** — the secret access key from your AWS credentials. Stored encrypted.
**AWS Region** — the region the bucket lives in, for example `eu-west-2`.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Search Pattern** — a regular expression matched against object keys in the bucket. Every matching object is read as one dataset, so all matches must share a schema. For example `^exports/orders/.*\.csv$`.
**Search Prefix** — an S3 key prefix that limits which objects are listed before the pattern is applied, for example `exports/orders/`. Always set this on a large bucket: without it, every key is listed on every run.
Field separator in the files. Use `\t` for tab-separated exports.
Character encoding of the files.
Rows read and written per batch.
Columns to read, as a comma-separated list.
Row filter applied to the parsed rows.
In `append` mode the extract tracks the last-modified timestamp of the objects it has read, so a later run picks up newly written files rather than re-reading the whole prefix.
## Required Permissions
The AWS IAM user account needs the following permissions on the specified S3 bucket:
* `s3:GetObject` - For reading files from the bucket
* `s3:ListBucket` - For listing contents of the bucket
* `s3:GetBucketLocation` - For determining the bucket's region
## Limitations
Be aware of the following limitations when working with AWS S3 CSV sources:
* CSV, TXT, Parquet, JSON files are supported
* UTF-8 encoding is recommended
* Individual file size limits apply based on your AWS S3 configuration
* The S3 bucket and Datazone instance should ideally be in the same region for optimal performance
* Cross-region access may incur additional AWS charges
## Next Steps
After configuring your AWS S3 source:
1. Create extracts to specify which CSV files to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# Azure Blob Storage
Source: https://docs.datazone.co/reference/sources/azure-blob
Azure Blob Storage is Microsoft's object storage solution for the cloud, designed to store massive amounts of unstructured data.
# Overview
Azure Blob Storage is a massively scalable object storage solution for unstructured data. Datazone provides native integration with Azure Blob Storage to read data files directly from your containers.
## Connection Parameters
| Parameter | Required | Description |
| -------------- | -------- | ------------------------------------------------- |
| Name | Yes | A unique identifier for your Azure Blob source |
| Account URL | Yes | The URL endpoint for your Azure Storage account |
| Token | Yes | The access token or SAS token for authentication |
| Container Name | Yes | The name of the container holding your data files |
## Required Permissions
The Azure Storage account needs the following permissions:
* `Storage Blob Data Reader` - For reading blob data
* `Storage Blob Data List` - For listing blobs in containers
* `Storage Account List` - For accessing storage account properties
## Limitations
Be aware of the following limitations when working with Azure Blob sources:
* CSV, TXT, Parquet, JSON files are supported
* UTF-8 encoding is recommended
* Individual file size limits apply based on your Azure Storage configuration
* The Storage account and Datazone instance should ideally be in the same region for optimal performance
* Cross-region access may incur additional Azure charges
## Next Steps
After configuring your Azure Blob source:
1. Create extracts to specify which files to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# Google BigQuery
Source: https://docs.datazone.co/reference/sources/bigquery
Connect to Google BigQuery for large-scale analytics data extraction.
# Overview
Google BigQuery is a fully managed, serverless data warehouse that enables scalable analysis over large datasets. Datazone connects to BigQuery using a **Google Cloud service account**, allowing you to extract tables and views from any dataset within your GCP project.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your BigQuery source.
**Project ID** — your Google Cloud project ID, for example `my-gcp-project`. Query jobs are billed to this project.
**Credentials JSON** — the full contents of your service account key file, as JSON. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Table Name** — the table or view to extract, qualified as `dataset_id.table_id` — for example `analytics.events`. The project comes from the source's `project_id`, so do not include it here; a value that is not exactly two dot-separated parts is rejected.
Rows fetched and written per batch.
Hard cap on the total number of rows extracted, applied as a `LIMIT` on the generated query. Unset means no limit. Useful for sampling a large table before committing to a full extract.
Run the generated query with BigQuery legacy SQL instead of standard SQL. Leave off unless the table requires it.
Columns to read, as a comma-separated list. Selecting only what you need is the single biggest lever on BigQuery cost, since a columnar scan is billed by bytes read.
Row filter pushed down into the query, written without the `WHERE` keyword — for example `status = 'active'`. On a partitioned table, filtering on the partition column prunes partitions and lowers cost.
## Setting Up a Service Account
1. Go to **IAM & Admin → Service Accounts** in the Google Cloud Console
2. Create a new service account (e.g. `datazone-reader`)
3. Grant the following roles:
* `BigQuery Data Viewer` — read access to datasets and tables
* `BigQuery Job User` — permission to run query jobs
4. Create a JSON key for the service account
5. Copy the full contents of the downloaded JSON key file into the **Credentials JSON** field
## Required Permissions
The service account needs the following IAM roles:
* `roles/bigquery.dataViewer` — for reading table data
* `roles/bigquery.jobUser` — for executing queries
For cross-project access, grant `BigQuery Data Viewer` on the specific datasets in the source project.
## Limitations
* BigQuery extracts use the BigQuery Storage API for efficient large-scale reads
* Partitioned and clustered tables are supported
* Supported BigQuery regions: all standard GCP regions
## Next Steps
After configuring your BigQuery source:
1. Create extracts to specify which tables or views to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# HubSpot
Source: https://docs.datazone.co/reference/sources/hubspot
Extract CRM objects from HubSpot using a private app access token.
# Overview
HubSpot is a CRM platform whose records — contacts, companies, deals, tickets, and custom objects — are exposed through its CRM API. Datazone connects with a **HubSpot private app access token** and extracts one CRM object type per extract via the CRM Search API.
## Prerequisites
Create a **private app** in your HubSpot account (**Settings → Integrations → Private Apps**), grant it read scopes for the object types you intend to extract, and copy its access token.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your HubSpot source.
**Access Token** — the private app access token, for example `pat-eu1-…`. Stored encrypted.
**Base URL** — HubSpot API base URL. Leave the default unless you have been given a region-specific host.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Object Type** — the singular CRM object type to extract: `contact`, `company`, `deal`, `ticket`, or the name of a custom object.
Comma-separated list of properties to fetch, for example `email,firstname,lastname,hubspot_owner_id,lifecyclestage`.
This **replaces** HubSpot's default property set rather than adding to it. Only `hs_object_id`, `createdate`, and `lastmodifieddate` are always returned — every other field, including custom properties and owner fields, must be listed explicitly. Omit the parameter to get HubSpot's default set. Discover available names with `GET /crm/v3/properties/{object}`.
Records requested per API call. The Search API's maximum is 200.
Records written per batch on the Datazone side.
Per-request timeout in seconds.
Whether to fetch archived records. The Search API returns active records only, so this has no effect on the current extraction path.
CRM API version segment. Retained for compatibility; it does not affect the Search endpoint used for extraction.
## Output Shape
Each record is flattened one level, so the schema stays stable across records regardless of which properties are populated:
| Column | Type | Notes |
| ------------ | ------- | ------------------------------------ |
| `id` | string | Object id |
| `properties` | string | JSON blob of the object's properties |
| `createdAt` | string | ISO 8601 |
| `updatedAt` | string | ISO 8601 |
| `archived` | boolean | |
| `url` | string | Record URL, when returned |
Because `properties` arrives as a JSON string, unpack the fields you need in a downstream pipeline transform.
## Incremental Extraction
Records are read sorted by `hs_object_id` ascending, and each page is fetched with an `hs_object_id > last_id` filter. This sidesteps the Search API's 10,000-record offset cap, so collections of any size can be extracted. In `append` mode the last id seen becomes the resume cursor.
Incremental extraction advances by object id, so it captures **newly created** records only — edits to records already extracted are not picked up. Schedule a periodic `overwrite` run to refresh updated records.
## Required Permissions
Grant the private app read scopes for each object type you extract, for example:
| Scope | Purpose |
| ---------------------------- | --------------------------------- |
| `crm.objects.contacts.read` | Reading contacts |
| `crm.objects.companies.read` | Reading companies |
| `crm.objects.deals.read` | Reading deals |
| `crm.schemas.custom.read` | Reading custom object definitions |
## Limitations
* One CRM object type per extract
* `page_size` is capped at 200 by the Search API
* Rate limits (HTTP 429) and transient errors are retried automatically with backoff, which can extend the runtime of large extracts
* Nested property values are JSON-stringified rather than expanded into columns
## Next Steps
After configuring your HubSpot source:
1. Create extracts to specify which CRM object types to ingest
2. Configure scheduling for recurring extracts
3. Unpack the `properties` JSON in a pipeline transform
# MongoDB
Source: https://docs.datazone.co/reference/sources/mongodb
MongoDB is a popular NoSQL database that stores data in flexible, JSON-like documents.
# Overview
MongoDB is a document-oriented NoSQL database that provides high performance, high availability, and easy scalability. Datazone provides native integration with MongoDB to read data directly from your collections.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your MongoDB source.
**Host** — hostname or IP address of your MongoDB server. You may also supply a full connection string beginning with `mongodb://` or `mongodb+srv://`, in which case it is used verbatim and `port`, `user`, and `password` are ignored. Use this form for replica sets, Atlas clusters, and any connection needing extra options.
**Port** — port the MongoDB server listens on. Typically `27017`.
**Database Name** — the database holding the collections you want to extract.
**User** — username with read permission on the database.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Collection Name** — the collection to extract, within the source's `database_name`.
Documents fetched and written per batch. Lower than the relational default because documents are typically wider than rows.
Number of documents sampled to infer the schema. Raise it when a collection is heterogeneous and fields present in only a few documents are being missed.
Cast values to string rather than inferring narrower types. Keeps the schema stable across documents whose field types disagree.
Fields to read, as a comma-separated list.
Row filter applied to the documents read.
## Required Permissions
The MongoDB user account needs the following permissions:
* `find` - For reading documents from collections
* `listCollections` - For listing available collections
* `listIndexes` - For accessing collection indexes
* `read` - For reading data from the database
## Limitations
Be aware of the following limitations when working with MongoDB sources:
* Complex MongoDB data types may be converted to standard formats
* Individual document size limits apply based on your MongoDB configuration
* The schema is inferred from a sample of documents, not from the whole collection
## Next Steps
After configuring your MongoDB source:
1. Create extracts to specify which collections to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# Microsoft SQL Server
Source: https://docs.datazone.co/reference/sources/mssqlserver
Microsoft SQL Server is a relational database management system developed by Microsoft.
# Overview
Microsoft SQL Server is a robust relational database management system that enables secure and efficient data management. Datazone provides native integration with SQL Server to read data directly from your databases.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your SQL Server source.
**Host** — hostname or IP address of your SQL Server instance.
**Port** — port the instance listens on. Typically `1433`.
**Database Name** — the database to connect to.
**Schema Name** — schema that extracts resolve table names within.
**User** — username with read permission on the tables you intend to extract.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Table Name** — the table or view to extract, resolved within the source's `schema_name`.
Columns to read, as a comma-separated list.
Row filter pushed down to SQL Server, written without the `WHERE` keyword — for example `modified_at >= '2024-01-01'`.
`batch_size` is fixed at 100,000 rows for SQL Server extracts and cannot be overridden in `source_parameters`.
## Required Permissions
The SQL Server user account needs the following permissions:
* `SELECT` - For reading data from tables and views
* `VIEW DEFINITION` - For accessing object metadata
* `VIEW DATABASE STATE` - For viewing database state information
* `CONNECT` - For connecting to the database
## Limitations
Be aware of the following limitations when working with SQL Server sources:
* Some SQL Server-specific data types may be converted to standard formats
* CLR data types are not supported
* Individual query size and timeout limits apply
* Connection pooling settings may affect performance
## Next Steps
After configuring your SQL Server source:
1. Create extracts to specify which tables to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# MySQL
Source: https://docs.datazone.co/reference/sources/mysql
MySQL is an open-source relational database management system (RDBMS).
# Overview
MySQL is one of the world's most popular open-source relational database management systems. Datazone provides seamless integration with MySQL databases, allowing you to easily ingest and manage your MySQL data.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your MySQL source.
**Host** — hostname or IP address of your MySQL server.
**Port** — port the MySQL server listens on. Typically `3306`.
**Database Name** — the database to connect to.
**Schema Name** — optional schema, if different from the database name. Left empty, extracts resolve tables within `database_name`.
**User** — username with read permission on the tables you intend to extract.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Table Name** — the table or view to extract. Resolved within the source's `schema_name` when set, otherwise `database_name`.
Rows fetched and written per batch.
Columns to read, as a comma-separated list.
Row filter pushed down to MySQL, written without the `WHERE` keyword — for example `created_at >= '2024-01-01'`.
## Required Permissions
The MySQL user account needs the following permissions:
* `SELECT` - For reading data from tables
* `SHOW DATABASES` - For listing available databases
* `SHOW VIEW` - For accessing views
* `REFERENCES` - For foreign key constraints
## Limitations
Be aware of the following limitations when working with MySQL sources:
* Supported MySQL versions: 5.6 and above
* Some MySQL-specific data types might be converted to standard types
* Large table scans may impact source database performance
* Consider using read replicas for heavy data extraction
## Next Steps
After configuring your MySQL source:
1. Create extracts to specify which tables to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# Oracle
Source: https://docs.datazone.co/reference/sources/oracle
Oracle is a powerful enterprise-grade relational database management system.
# Overview
Oracle Database is an enterprise-class database management system that provides comprehensive and advanced data management features. Datazone provides native integration with Oracle to read data directly from your databases.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your Oracle source.
**Host** — hostname or IP address of your Oracle database server.
**Port** — port the listener runs on. Defaults to `1521` when omitted.
**Service Name** — the Oracle service name for your database.
**Schema Name** — schema that extracts resolve table names within.
**User** — username with read permission on the tables you intend to extract.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Table Name** — the table or view to extract, resolved within the source's `schema_name`.
Rows fetched and written per batch.
Columns to read, as a comma-separated list.
Row filter pushed down to Oracle, written without the `WHERE` keyword — for example `status = 'ACTIVE'`.
## Required Permissions
The Oracle user account needs the following permissions:
* `SELECT` - For reading data from tables and views
* `SELECT ANY TABLE` - For reading from tables in other schemas
* `SELECT_CATALOG_ROLE` - For accessing data dictionary views
* `CREATE SESSION` - For connecting to the database
## Limitations
Be aware of the following limitations when working with Oracle sources:
* Some Oracle-specific data types may be converted to standard formats
* LONG and LONG RAW data types have size limitations
* Individual query size and timeout limits apply
* The Oracle instance and Datazone should ideally be in the same network for optimal performance
* Binary data types require special handling
## Next Steps
After configuring your Oracle source:
1. Create extracts to specify which tables to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# Data Ingestion
Source: https://docs.datazone.co/reference/sources/overview
Data ingestion is the first step in Datazone's data journey.
# Overview
Data Ingestion is where data from various sources is imported into the platform for processing, analysis, and storage. This process is managed through two key concepts: [Sources](/key-concepts#source) and [Extracts](/key-concepts#extract).
A **Source** holds the connection to an external system — credentials, host, project. You create it once and reuse it. An **Extract** selects what to pull from that source — a table, a collection, a file pattern — and how to pull it. One source can back many extracts.
That split is also how configuration is split:
| Layer | Where it is set | What it holds |
| ------------------------- | -------------------------------------- | ----------------------------------------------------------- |
| **Connection parameters** | On the Source, at creation | Host, port, credentials, database, bucket, project ID |
| **Extract parameters** | On the Extract, in `source_parameters` | Table or collection name, file pattern, batching, filtering |
Each source page documents both. Parameter names match the API and CLI payloads.
## Getting Started
To configure a new source:
1. Select your source type from the available connectors
2. Provide the required connection details
3. Start creating extracts from your source
For detailed configuration steps, visit the specific documentation page for your chosen source type.
## Shared Extract Parameters
These parameters apply to every extract regardless of source type. Source-specific parameters are documented on each connector's page.
**Extract Mode** — `overwrite` replaces the dataset on every run (full table). `append` reads only new rows since the last run and appends them, using `replication_key` to track position.
**Replication Key** — the column Datazone compares against the last extracted value to find new rows. Required when `mode` is `append`. Use a monotonically increasing column such as an auto-increment id or an `updated_at` timestamp.
**Schema Mapping** — per-field selection, renaming, and type casting applied before the data is written. Each entry accepts `source_field`, `source_type`, `target_field` (rename, `null` keeps the original), `target_type` (cast, `null` keeps the inferred type), and `include` (set `false` to drop the field).
**Custom Reader Config** — an escape hatch. Keys in this object are merged over the resolved reader configuration for every execution, overriding anything computed from the source and extract. Use it only for parameters not exposed elsewhere.
The following go in the extract's `source_parameters` and are supported by all connectors that use the reader framework:
Rows read and written per batch. Higher values reduce round trips but raise memory use. The default varies per connector — see each source page.
Columns to read, as a comma-separated list (for example `id, name, created_at`). Defaults to every column.
A row filter applied at the source, written without the `WHERE` keyword — for example `status = 'active' AND region = 'EU'`. Pushing filters down here is far cheaper than filtering later in a pipeline.
`reset_state` is set per execution, not on the extract. It re-reads all data while still capturing the latest replication state, and is only valid for an `append` extract that already has state.
## Connectors
Direct connection to MySQL and MySQL-compatible databases
Direct connection to PostgreSQL databases
Direct connection to SQL Server instances
Direct connection to Oracle Database
Document collections from MongoDB
Tables and views from BigQuery datasets
Delimited files from S3 buckets
Files from Azure Blob containers
Direct SQL access to SAP HANA
SAP ERP tables via the CloudFeed connector
SAP S/4HANA tables and CDS views via CloudFeed
SAP BW InfoProviders via CloudFeed
CRM objects from HubSpot
# PostgreSQL
Source: https://docs.datazone.co/reference/sources/postgresql
PostgreSQL is a powerful, open source object-relational database system.
# Overview
PostgreSQL is an advanced open-source relational database that supports both SQL (relational) and JSON (non-relational) querying. Datazone provides native integration with PostgreSQL databases, allowing you to easily connect and extract data from your PostgreSQL instances.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your PostgreSQL source.
**Host** — hostname or IP address of your PostgreSQL server.
**Port** — port the server listens on. Defaults to `5432` when omitted.
**Database Name** — the database to connect to.
**Schema Name** — schema that extracts resolve table names within.
**User** — username with read permission on the tables you intend to extract.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Table Name** — the table or view to extract, resolved within the source's `schema_name`.
Rows fetched and written per batch.
Columns to read, as a comma-separated list.
Row filter pushed down to PostgreSQL, written without the `WHERE` keyword — for example `status = 'active'`.
## Required Permissions
The PostgreSQL user account needs the following permissions:
* `SELECT` - For reading data from tables
* `USAGE` - For accessing schemas
* `CONNECT` - For connecting to the database
## Limitations
Be aware of the following limitations when working with PostgreSQL sources:
* Supported PostgreSQL versions: 9.6 and above
* Some PostgreSQL-specific types will be converted to standard types
## Next Steps
After configuring your PostgreSQL source:
1. Create extracts to specify which tables to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# SAP BW
Source: https://docs.datazone.co/reference/sources/sap-bw
Connect to SAP Business Warehouse InfoProviders via the Datazone CloudFeed SAP Connector.
# Overview
SAP BW (Business Warehouse) is SAP's data warehousing platform, where business data is modelled into **InfoProviders** — InfoCubes, DataStore Objects, and CompositeProviders. Datazone connects to SAP BW via the **Datazone CloudFeed SAP Connector**, an HTTP adapter installed on your SAP system, and extracts InfoProviders rather than transparent tables.
## Prerequisites
The **Datazone CloudFeed SAP Connector** must be installed on your SAP environment before creating this source. Contact your SAP administrator to confirm the connector is active.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your SAP BW source.
**Base URL** — base URL of the SAP system running the CloudFeed connector, for example `https://your-bw-host:8080`.
**Username** — SAP technical user username.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**InfoProvider Name** — the InfoProvider to extract, for example `0MATERIAL`.
Rows SAP puts in each extraction package. Packages are streamed rather than buffered whole, so this bounds peak memory. Lower it for very wide objects.
Rows written per batch on the Datazone side.
Fetch ready packages concurrently instead of one at a time. Turn it off to minimise load on the SAP system.
Number of concurrent package fetches when `parallel_fetch` is on. Peak memory scales with this multiplied by `package_size`, so raise the two together with care.
Per-request timeout in seconds when talking to the CloudFeed connector.
SAP range-table filters, evaluated inside SAP before any data leaves the system. Each entry takes `FIELDNAME`, `SIGN` (`I` to include, `E` to exclude), `OPTION` (`EQ`, `NE`, `GT`, `GE`, `LT`, `LE`, `BT`, `CP`), `LOW`, and `HIGH` (upper bound, for `BT`).
```json theme={null}
[{ "FIELDNAME": "MTART", "SIGN": "I", "OPTION": "EQ", "LOW": "FERT", "HIGH": "" }]
```
Delta configuration, **required** when `mode` is `append`. Delta state is held server-side by SAP against `ID`, not tracked by Datazone, and is committed only after the extraction succeeds.
```json theme={null}
{
"ID": "unique-extraction-id",
"TYPE": "MULTI",
"OPERATOR": "GE",
"COLUMNS": ["LAEDA"]
}
```
`ID` identifies the delta stream and must stay stable across runs; `TYPE` is `SINGLE` or `MULTI` column; `OPERATOR` is the comparison applied to `COLUMNS`.
Columns to read, as a comma-separated list.
Row filter applied to the extracted rows, written without the `WHERE` keyword. For filtering inside SAP, prefer `filters`.
`extraction_timeout` (default 7200 seconds) and `check_interval` (default 5 seconds) govern how long Datazone waits for SAP to finish an extraction job and how often it polls. Neither is read from `source_parameters` — set them through the extract's `custom_reader_config` if you need to change them.
## Required Permissions
Assign the following roles to the SAP technical user:
| Role | Description |
| -------------------- | --------------------------- |
| `/CLF/BASE` | CloudFeed Base Service Role |
| `/CLF/TABLE_ALL` | All Table Access |
| `/CLF/WRITEBACK_ALL` | All Writeback Access |
## How It Works
Datazone requests an extraction job from the CloudFeed connector, polls it until packages become ready, and streams each package as it arrives. On success the extraction is committed, which advances the server-side delta state for `append` extracts.
## SAP ERP vs SAP BW
Both connectors speak to the same CloudFeed connector and share their filter syntax, delta settings, package streaming, and commit-on-success behaviour. They differ in what they read:
| | SAP ERP | SAP BW |
| ---------------------- | -------------------------------- | ------------------------------------ |
| **Extraction target** | Transparent tables (e.g. `MARA`) | InfoProviders (e.g. `0MATERIAL`) |
| **CloudFeed endpoint** | `/sap/cloudfeed/module/table` | `/sap/cloudfeed/module/infoprovider` |
| **Use case** | Raw operational tables | Modelled, aggregated warehouse data |
## Limitations
* Maximum payload size per request: 10 MB
* `delta_settings` is required for `append` extracts; without it, incremental extraction has nothing to track
* Delta state lives in SAP, so resetting an extract's position is done on the SAP side against the `delta_settings.ID`
## Next Steps
1. Create extracts to specify which InfoProviders to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
# SAP ERP
Source: https://docs.datazone.co/reference/sources/sap-erp
Connect to SAP ERP systems via the Datazone CloudFeed SAP Connector.
# Overview
Datazone connects to SAP ERP via the **Datazone CloudFeed SAP Connector**, an HTTP adapter installed on your SAP system. It provides access to SAP tables, schemas, and function modules without requiring a direct database connection.
## Prerequisites
The **Datazone CloudFeed SAP Connector** must be installed on your SAP environment before creating this source. Contact your SAP administrator to confirm the connector is active.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your SAP ERP source.
**Base URL** — base URL of the SAP system running the CloudFeed connector, for example `https://your-sap-host:8080`.
**Username** — SAP technical user username.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Table Name** — the table to extract, for example `MARA`.
Rows SAP puts in each extraction package. Packages are streamed rather than buffered whole, so this bounds peak memory. Lower it for very wide objects.
Rows written per batch on the Datazone side.
Fetch ready packages concurrently instead of one at a time. Turn it off to minimise load on the SAP system.
Number of concurrent package fetches when `parallel_fetch` is on. Peak memory scales with this multiplied by `package_size`, so raise the two together with care.
Per-request timeout in seconds when talking to the CloudFeed connector.
SAP range-table filters, evaluated inside SAP before any data leaves the system. Each entry takes `FIELDNAME`, `SIGN` (`I` to include, `E` to exclude), `OPTION` (`EQ`, `NE`, `GT`, `GE`, `LT`, `LE`, `BT`, `CP`), `LOW`, and `HIGH` (upper bound, for `BT`).
```json theme={null}
[{ "FIELDNAME": "MTART", "SIGN": "I", "OPTION": "EQ", "LOW": "FERT", "HIGH": "" }]
```
Delta configuration, **required** when `mode` is `append`. Delta state is held server-side by SAP against `ID`, not tracked by Datazone, and is committed only after the extraction succeeds.
```json theme={null}
{
"ID": "unique-extraction-id",
"TYPE": "MULTI",
"OPERATOR": "GE",
"COLUMNS": ["LAEDA"]
}
```
`ID` identifies the delta stream and must stay stable across runs; `TYPE` is `SINGLE` or `MULTI` column; `OPERATOR` is the comparison applied to `COLUMNS`.
Columns to read, as a comma-separated list.
Row filter applied to the extracted rows, written without the `WHERE` keyword. For filtering inside SAP, prefer `filters`.
`extraction_timeout` (default 7200 seconds) and `check_interval` (default 5 seconds) govern how long Datazone waits for SAP to finish an extraction job and how often it polls. Neither is read from `source_parameters` — set them through the extract's `custom_reader_config` if you need to change them.
## Required Permissions
Assign the following roles to the SAP technical user:
| Role | Description |
| -------------------- | --------------------------- |
| `/CLF/BASE` | CloudFeed Base Service Role |
| `/CLF/TABLE_ALL` | All Table Access |
| `/CLF/WRITEBACK_ALL` | All Writeback Access |
## How It Works
Datazone communicates with SAP ERP via the Datazone CloudFeed SAP Connector over HTTP installed on the SAP system. No direct database port needs to be open — all data flows over HTTPS and SAP's own authorization model is enforced.
## Limitations
* Maximum payload size per request: 10 MB
* Supported versions: SAP ERP ECC 6.0 and above
## Next Steps
1. Create extracts to specify which tables to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
# SAP HANA
Source: https://docs.datazone.co/reference/sources/sap-hana
SAP HANA is an in-memory, column-oriented, relational database management system.
# Overview
SAP HANA is a high-performance in-memory database platform that provides real-time analytics and transaction processing capabilities. Datazone offers native integration with SAP HANA, enabling you to efficiently extract and process data from your SAP HANA instances.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your SAP HANA source.
**Host** — hostname or IP address of your SAP HANA server.
**Port** — port the SAP HANA SQL interface listens on. Typically `30015`.
**User** — database username with read permission on the tables and views you intend to extract.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Table Name** — the table or view to extract. Since the source carries no schema, qualify it here as `SCHEMA.TABLE` — for example `SAPABAP1.MARA`.
Rows fetched and written per batch.
Columns to read, as a comma-separated list.
Row filter pushed down to SAP HANA, written without the `WHERE` keyword — for example `MANDT = '100'`.
## Required Permissions
The SAP HANA user account needs the following permissions:
* `SELECT` - For reading data from tables and views
* `CATALOG READ` - For accessing system views and metadata
## Limitations
Be aware of the following limitations when working with SAP HANA sources:
* Supported SAP HANA versions: 2.0 and above
## Next Steps
After configuring your SAP HANA source:
1. Create extracts to specify which tables or views to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.
# SAP S/4HANA
Source: https://docs.datazone.co/reference/sources/sap-s4hana
Connect to SAP S/4HANA systems via the Datazone CloudFeed SAP Connector.
# Overview
Datazone connects to SAP S/4HANA via the **Datazone CloudFeed SAP Connector**, an HTTP adapter installed on your SAP system. It provides access to SAP tables, CDS views, and function modules without requiring a direct database connection.
## Prerequisites
The **Datazone CloudFeed SAP Connector** must be installed on your SAP environment before creating this source. Contact your SAP administrator to confirm the connector is active.
## Connection Parameters
Set on the source when you create it. Names match the API and CLI payload.
**Name** — a unique identifier for your SAP S/4HANA source.
**Base URL** — base URL of the SAP system running the CloudFeed connector, for example `https://your-s4hana-host:8080`.
**Username** — SAP technical user username.
**Password** — password for the specified user. Stored encrypted.
## Extract Parameters
Set per extract, in `source_parameters`. See [Shared Extract Parameters](/reference/sources/overview#shared-extract-parameters) for `mode`, `replication_key`, and `schema_mapping`.
**Table Name** — the table or CDS view to extract, for example `I_BusinessPartner`.
Rows SAP puts in each extraction package. Packages are streamed rather than buffered whole, so this bounds peak memory. Lower it for very wide objects.
Rows written per batch on the Datazone side.
Fetch ready packages concurrently instead of one at a time. Turn it off to minimise load on the SAP system.
Number of concurrent package fetches when `parallel_fetch` is on. Peak memory scales with this multiplied by `package_size`, so raise the two together with care.
Per-request timeout in seconds when talking to the CloudFeed connector.
SAP range-table filters, evaluated inside SAP before any data leaves the system. Each entry takes `FIELDNAME`, `SIGN` (`I` to include, `E` to exclude), `OPTION` (`EQ`, `NE`, `GT`, `GE`, `LT`, `LE`, `BT`, `CP`), `LOW`, and `HIGH` (upper bound, for `BT`).
```json theme={null}
[{ "FIELDNAME": "MTART", "SIGN": "I", "OPTION": "EQ", "LOW": "FERT", "HIGH": "" }]
```
Delta configuration, **required** when `mode` is `append`. Delta state is held server-side by SAP against `ID`, not tracked by Datazone, and is committed only after the extraction succeeds.
```json theme={null}
{
"ID": "unique-extraction-id",
"TYPE": "MULTI",
"OPERATOR": "GE",
"COLUMNS": ["LAEDA"]
}
```
`ID` identifies the delta stream and must stay stable across runs; `TYPE` is `SINGLE` or `MULTI` column; `OPERATOR` is the comparison applied to `COLUMNS`.
Columns to read, as a comma-separated list.
Row filter applied to the extracted rows, written without the `WHERE` keyword. For filtering inside SAP, prefer `filters`.
`extraction_timeout` (default 7200 seconds) and `check_interval` (default 5 seconds) govern how long Datazone waits for SAP to finish an extraction job and how often it polls. Neither is read from `source_parameters` — set them through the extract's `custom_reader_config` if you need to change them.
## Required Permissions
Assign the following roles to the SAP technical user:
| Role | Description |
| -------------------- | --------------------------- |
| `/CLF/BASE` | CloudFeed Base Service Role |
| `/CLF/TABLE_ALL` | All Table Access |
| `/CLF/WRITEBACK_ALL` | All Writeback Access |
## How It Works
Datazone communicates with SAP S/4HANA via the Datazone CloudFeed SAP Connector over HTTP installed on the SAP system. No direct database port needs to be open — all data flows over HTTPS and SAP's own authorization model is enforced.
## SAP HANA vs SAP S/4HANA
| | SAP HANA | SAP S/4HANA |
| ------------------- | ---------------------------------- | ------------------------------------------- |
| **Connection type** | Direct JDBC (host + port) | HTTP via CloudFeed connector |
| **Use case** | Direct SQL access to HANA database | ERP business data via SAP application layer |
| **Auth model** | Database user | SAP application user |
## Limitations
* Maximum payload size per request: 10 MB
* Supported versions: SAP S/4HANA 1610 and above
## Next Steps
1. Create extracts to specify which tables or CDS views to ingest
2. Configure scheduling for recurring extracts
3. Integrate the source into your data pipelines
# Getting Started
Source: https://docs.datazone.co/reference/studio-apps/getting-started
Create, build, and extend your first Studio App
# Getting Started with Studio Apps
This walks through creating a Studio App, building it, and making it show your own data. You need a Datazone project and permission to write to its repository.
## 1. Create the app
Open your project, go to **Studio Apps**, and choose **New studio app**. Give it a name — the alias is derived from it — and pick the branch to create it on.
Datazone commits the whole scaffold to `studio//` and adds the entry to `config.yml` in one commit:
```yaml theme={null}
studio_apps:
- alias: sales_dashboard
name: Sales Dashboard
path: studio/sales_dashboard
```
Creating the app does not build it. That is the next step, and it is always explicit.
## 2. Build it
Press **Build**. Datazone queues a sandboxed job that runs `npm install` and `vite build`, then publishes the bundle. The first build takes a couple of minutes; later ones are faster.
When the status reaches `READY`, open the app. The scaffold greets you with the signed-in user's email — proof that the session and the API are wired up:
```tsx theme={null}
const user = await getMe()
```
If the build fails, the **Builds** tab has the full log.
## 3. Edit it
You have three ways to change the app, and they all go through the repository:
The project's **Code** tab edits files and commits them in place.
Describe the change; Orion writes the files and can design the objects behind them.
`git clone` the project, edit with your own tools, and push.
After any change: **push, then build again.** Datazone flags the served build as stale once the branch has moved on, but it never rebuilds on its own.
## 4. Show your own data
`src/lib/datazone.ts` is the client. Replace the body of `Home` in `src/App.tsx` with a query of your own:
```tsx theme={null}
import { useEffect, useState } from "react"
import { AppLayout } from "@/components/app-layout"
import { executeQuery } from "@/lib/datazone"
type Row = { region: string; total: number }
function Home() {
const [rows, setRows] = useState([])
const [error, setError] = useState(null)
useEffect(() => {
executeQuery("select region, sum(amount) as total from sales group by region")
.then(setRows)
.catch((problem) => setError(problem.message))
}, [])
return (
{error &&
{error}
}
{rows.map((row) => (
{row.region}{row.total}
))}
)
}
```
The query runs with the signed-in user's permissions. A dataset they cannot read fails with an error you can show them — which is exactly what the `error` state above is for.
For a query you will use more than once, or one that would otherwise interpolate user input into SQL, publish an [endpoint](/reference/integration/endpoints) and call it with `callEndpoint`.
## 5. Add a page
Create the component, then **add its route** — an imported component with no route is removed from the bundle at build time and its page will 404:
```tsx theme={null}
// src/App.tsx
import { Route, Routes } from "react-router-dom"
import { OrdersPage } from "@/pages/orders"
export default function App() {
return (
} />
} />
} />
)
}
```
## 6. Add UI components
The scaffold ships a layout shell and leaves `src/components/ui/` empty. Add components with the shadcn CLI, which reads the `components.json` already in your app:
```bash theme={null}
cd studio/sales_dashboard
npx shadcn@latest add button card table
```
The files land in `src/components/ui/` and are yours to edit. Check `package.json` afterwards and pin any version the CLI added as a range — a `^` lets two builds of the same commit install different code.
Style from the theme tokens (`bg-background`, `text-muted-foreground`, `bg-card`, `text-primary`) rather than literal colours, so the app follows Datazone's light and dark themes. The theme lives in `src/index.css`; Tailwind 4 is configured there, not in a `tailwind.config.js`.
## 7. Store data
To let users create or edit records, add a [Knowledge Object](/reference/knowledge-objects/overview) and have the app read and write its instances. Objects are YAML in the same repository, so both ship in the same push:
```yaml theme={null}
# config.yml
objects:
- path: objects/order.yml
studio_apps:
- alias: orders
name: Orders
path: studio/orders
```
Objects migrate before they can be written to — an app whose objects are not `READY` will load and fail on its first write. Deploy the objects, wait for `READY`, then build the app.
Do not keep records in `localStorage` or in a file in the repository. `localStorage` is per-browser and lost on the next device, and the bundle is read-only at runtime. Use Knowledge Objects.
## Things that build fine and break in the browser
A Studio App can compile cleanly and still fail once served. These are the causes, in order of how often they happen:
| Symptom | Cause |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Every asset 404s, blank page | `base` was set in `vite.config.js` — remove it; Datazone passes the correct one |
| Every route resolves to nothing | The router's `basename` must stay `import.meta.env.BASE_URL` |
| A page 404s | Its component has no `` in `App.tsx` |
| An image is missing | Reference `public/` files through `` `${import.meta.env.BASE_URL}logo.svg` ``, not `/logo.svg` |
| Every API call is unauthorised | Use `@/lib/datazone` with relative paths; an absolute URL or an added `Authorization` header drops the session |
| The app reads the wrong data | A branch-scoped call omitted `branch` and read the default branch |
| A colour renders unstyled | A custom token needs both a `:root` value and an `@theme inline` entry |
## Working locally
`npm install && npm run dev` renders the app, but API calls will not work: in dev the client resolves the API to `/api`, which the Vite dev server does not serve. Develop layout locally and verify data against a built app, or add your own `/api` proxy to `vite.config.js`.
## Next steps
* [Studio Apps Overview](/reference/studio-apps/overview) — how builds, branches, and serving work
* [Knowledge Objects](/reference/knowledge-objects/overview) — modelling the data behind your app
* [Endpoints](/reference/integration/endpoints) — publishing a query for your app to call
# Overview
Source: https://docs.datazone.co/reference/studio-apps/overview
Build and host custom React applications on your Datazone data, served behind your organisation's authentication
# Studio Apps
A Studio App is a **Vite + React single-page application that lives in your project repository**. Datazone installs its dependencies, builds it in an isolated sandbox, and serves the result at a URL inside your deployment — behind the same session that protects the rest of Datazone.
Unlike an [Intelligent App](/reference/intelligent-apps/overview), which you describe declaratively in YAML, a Studio App is code you write. You get the whole React ecosystem and, with it, everything a dashboard cannot do: multi-step forms, write-back to [Knowledge Objects](/reference/knowledge-objects/overview), custom interactions, bespoke layouts.
## What you get
* **A real URL, behind your authentication** — the app is served from your Datazone deployment and only to signed-in users of the organisation that owns it. There is no separate hosting, no separate login.
* **The API, already authenticated** — the browser sends the user's Datazone session with every request. The app ships no API key, and every call runs with that user's permissions.
* **One build per branch** — each branch of your repository builds and serves its own version of the app, so you can review a change on a feature branch before it reaches `main`.
* **A pre-wired scaffold** — routing, Tailwind 4, the shadcn component setup, and a small Datazone client are generated for you.
* **Nothing to operate** — no Dockerfile, no deployment pipeline, no CDN configuration. Push, then build.
## When to use one
| Use | Choose |
| -------------------------------------------- | --------------------------------------------------------------------------- |
| Charts, KPIs, filters over existing datasets | [Intelligent App](/reference/intelligent-apps/overview) |
| Users need to create or edit records | **Studio App** + [Knowledge Objects](/reference/knowledge-objects/overview) |
| A custom workflow, wizard, or internal tool | **Studio App** |
| Exposing data to another system | [Endpoint](/reference/integration/endpoints) |
An Intelligent App is considerably less work when it fits. Reach for a Studio App when you need behaviour a dashboard cannot express.
## How it works
1. **Create the app.** Datazone scaffolds a complete Vite + React app into `studio//` and registers it in your project's `config.yml`, in a single commit.
2. **Write your app.** Edit the files in the repository — in the built-in code editor, with Orion, or locally with `git`. Push your changes.
3. **Build.** Datazone runs `npm install` and `vite build` in a sandboxed job, then publishes the static bundle.
4. **Open it.** The app is served at its URL for signed-in members of your organisation.
Registering an app makes it appear in Datazone; it does **not** build it. Building is an explicit step, so pushing a broken commit never takes down a working app.
## The app in your repository
```
studio/sales_dashboard/
├── package.json pinned dependencies
├── index.html
├── vite.config.js
├── components.json shadcn configuration
├── tsconfig.json
└── src/
├── main.tsx router entry
├── App.tsx your routes
├── index.css Tailwind 4 theme
├── lib/datazone.ts the Datazone client
├── lib/utils.ts
├── components/app-layout.tsx
└── components/ui/ shadcn components you add
```
And in `config.yml`:
```yaml theme={null}
studio_apps:
- alias: sales_dashboard
name: Sales Dashboard
path: studio/sales_dashboard
```
## Talking to Datazone
The scaffold includes `src/lib/datazone.ts`, a small client for the Datazone API. It sends requests same-origin and relative, so the browser attaches the user's session cookie automatically.
```tsx theme={null}
import { callEndpoint, executeQuery, getMe } from "@/lib/datazone"
const user = await getMe()
const rows = await executeQuery<{ region: string; total: number }>(
"select region, sum(amount) as total from sales group by region",
)
const { records } = await callEndpoint("daily-revenue", { page_size: 50 })
```
| Export | Purpose |
| ---------------------------- | ------------------------------------------------------------- |
| `getMe()` | The signed-in user |
| `executeQuery(sql)` | SQL over the datasets this user can read |
| `callEndpoint(slug, params)` | Call a published [endpoint](/reference/integration/endpoints) |
| `apiFetch(path, init)` | Any other API path |
| `branch`, `projectId` | The branch this build came from, and the project |
| `branchQuery(filters)` | Build a `branch=…&filters=…` query string |
Permissions are enforced by the API on every call, so a user who cannot read a dataset cannot read it through your app either. There is no API key in the bundle, and none is needed.
Never put an API key, token, or secret in a Studio App. The bundle is JavaScript delivered to the browser — anything in it is visible to everyone who can open the app.
## Storing data
A Studio App is static files. It has no database and no server-side code, so anything users create or edit belongs in a [Knowledge Object](/reference/knowledge-objects/overview) — a versioned entity declared in your repository with a governed CRUD API.
That combination is the usual shape of an internal tool: objects for the records, a Studio App for the interface.
```tsx theme={null}
import { apiFetch, branch } from "@/lib/datazone"
// Instances are addressed by their `_key`, and every call takes the app's branch.
await apiFetch(`/knowledge-object/${objectId}/instances/${key}?branch=${branch}`, {
method: "PATCH",
body: JSON.stringify({ status: "SHIPPED" }),
})
```
See the [Knowledge Objects API](/reference/knowledge-objects/api) for the full surface.
## Branches and builds
Every app is built **per branch**. A branch has its own bundle, its own URL, and its own build history — the app on `feat/new-layout` is a different deployment from the one on `main`.
Two consequences worth remembering:
* After pushing, build again. Datazone marks the served build **stale** when the branch has moved on, but it does not rebuild on its own.
* Branch-scoped data (Knowledge Objects, for example) defaults to your default branch when a request does not name one. The client exports `branch` for exactly this reason — pass it, or an app on a feature branch will quietly read `main`'s data.
Build status is one of `NOT_BUILT`, `QUEUED`, `BUILDING`, `READY`, `ERROR`, or `TIMEOUT`. Logs for every build, successful or not, are on the app's **Builds** tab.
## Building with Orion
Orion knows how Studio Apps are structured and can write one for you — including designing the Knowledge Objects behind it. Describing what you want ("an app to manage orders, with a status filter and a create form") is usually faster than starting from the scaffold by hand.
## Limits
* **The app is client-side only.** No server-side rendering, no API routes, no server secrets.
* **Dependencies come from npm at build time**, from your `package.json`. Pin exact versions so a rebuild of the same commit installs the same code.
* **Builds are sandboxed and time-limited.** A build that hangs is marked `TIMEOUT`.
* **An app belongs to one organisation.** Users from other organisations are refused, not merely unable to find it.
## Next steps
* [Getting Started](/reference/studio-apps/getting-started) — create, build, and open your first app
* [Knowledge Objects](/reference/knowledge-objects/overview) — where a Studio App's data belongs
* [Endpoints](/reference/integration/endpoints) — publish a query for your app to call
* [Project Repository](/reference/development/project) — how `config.yml` ties it together
# UI Overview
Source: https://docs.datazone.co/reference/ui/overview
Navigate Datazone's main interface sections
# UI Overview
Datazone's interface is organized into **four main sections** to help you efficiently manage projects, build intelligent apps, create agents, and configure settings.
## Home Page
The **Home Page** is your starting point, displaying all your resources as cards:
* **Projects** - All your data projects
* **Intelligent Apps** - Interactive dashboards and applications
* **Agents** - AI assistants for data analysis
Click any card to open the resource.
## Project Page
The **Project Page** is where you spend **90% of your time**. This is your **core development workspace** where all data operations happen:
**Key Features:**
* **Code Editor** - Write transforms, pipelines, and notebooks
* **Workflow Builder** - Design and orchestrate data workflows
* **Executions** - Monitor running and completed jobs
* **Builds** - View build history and logs
* **Data Sources** - Manage datasets and views
* **Agents** - Create and configure AI agents
All **mutations and development** happen here - from writing code to executing pipelines.
## Intelligent App Page
When you **open an Intelligent App**, you see the interactive dashboard:
**Features:**
* **Interactive Charts** - Dynamic data visualizations
* **Filters** - Control what data is displayed
* **AI Assistant** - Ask questions about your data
* **Multi-Tab Layout** - Organize content logically
Learn more in the [Intelligent Apps documentation](/reference/intelligent-apps/overview).
## Settings
Access **Settings** from the **user dropdown** in the top-left corner:
**Available Settings:**
* **Profile** - Personal information and preferences
* **Organizations** - Manage teams and members
* **Model Accounts** - Configure AI providers
* **API Keys** - Generate authentication keys
* **Integrations** - SAML, OAuth, and external services
* **Platform Settings** - System configurations
Settings content **varies based on your permissions**. Some options are only visible to organization admins.
## Quick Search (⌘K / Ctrl+K)
Access the **Quick Search** menu from anywhere in the app by pressing **⌘K** (Mac) or **Ctrl+K** (Windows/Linux):
**Search Features:**
* **Global Search** - Find projects, agents, intelligent apps, datasets, and more
* **Quick Navigation** - Jump directly to any resource
* **Command Palette** - Execute actions without clicking
* **Recent Items** - Quick access to recently viewed resources
Simply press **⌘K** anywhere and start typing to find what you need.
## Navigation Tips
* **Home** - Click the logo to return to the home page
* **Quick Access** - Use the top navigation bar to switch between sections
* **⌘K Search** - Press ⌘K anytime for quick search and navigation
* **Context Menu** - Right-click resources for quick actions
## Next Steps
* [Create Your First Project](/reference/development/project)
* [Build an Intelligent App](/reference/intelligent-apps/overview)
* [Set Up an AI Agent](/reference/agents/overview)
# SQL Explorer
Source: https://docs.datazone.co/reference/ui/sql-explorer
A complete SQL workspace inside Datazone — write, run, save and revisit your queries without ever losing your place
# SQL Explorer
**New in v1.3.8 · May 2026** — SQL Explorer is now available in all Datazone workspaces.
SQL Explorer is a complete SQL workspace built directly into Datazone. Write, run, save, and revisit your queries without ever losing your place.
## One workspace for every query
Smart suggestions, instant results, autocomplete on every column. SQL Explorer turns ad-hoc analysis into a first-class workflow.
* **One-click lakehouse access** — every table in your lakehouse is instantly queryable
* **AI-powered completions** — smart suggestions on every column and keyword
* **Sub-second results** — results appear as fast as your warehouse allows
## Multi-Tab
Never lose your analysis. Tabs persist across sessions — come back days later and pick up exactly where you left off.
* Open multiple queries side by side
* Switch between **Draft** and named saved queries in the same tab bar
* Session state (cursor position, results, scroll) is fully preserved
## Saved Queries
Your favourite SQL, one keystroke away.
Press **⌘S** (Mac) or **Ctrl+S** (Windows/Linux) to save any query. Build a personal library you actually use — no more hunting through history to find that query from last week.
* Save and name queries for instant recall
* Accessible from the sidebar under **Saved Queries**
* Shared within your project so teammates can reuse them
## History
Every run, kept. Replay any query, anytime.
The **Query History** panel records every execution with its timestamp, duration, and result status. Click any entry to restore the exact SQL that produced those results.
## What's included
Everything an analyst needs, on day one:
Every table in your lakehouse, queryable. No exports, no connectors, no waiting.
Tabs, results, and cursor position are kept — exactly as you left them.
Smart completions on every keystroke, powered by your lakehouse schema.
Designed for SQL-first workflows — no setup, no boilerplate.
Pivot rows, then ship results to CSV, Parquet, or JSON in one click.
## Next Steps
* [Analysis Notebooks](/reference/analysis/notebook) — go deeper with Python-powered analysis
* [Toolkit](/reference/analysis/toolkit) — explore the full set of analysis utilities
* [Integration & APIs](/reference/integration/overview) — connect SQL Explorer output to external tools
# Datazone SDK
Source: https://docs.datazone.co/tutorial/datazone-sdk
Learn how to use Datazone SDK to access and manage your data in Datazone in your local environment.
### Get Dataset as Pandas Dataframe
Before use the Datazone SDK, you need to create a profile. It always use the default profile.
Check the [Installation](/installation) section for more information.
```python theme={null}
from datazone.sdk.client import DatazoneClient
client = DatazoneClient()
df = client.get_dataset_as_pandas(id="")
```
# Building an AI-Powered Customer Response Automation System with Datazone
Source: https://docs.datazone.co/tutorial/examples/ai-powered-message-automation
We built this cool email response system using Datazone, and its handling customer support emails like a champ! 🚀
## The Problem
Customer support inboxes get flooded with emails daily. Some need expert human attention, others are routine questions that keep coming up again and again.
We wanted to create a system that could:
* Process incoming customer emails automatically
* **Determine the sentiment** and urgency of each message
* Identify which ones **need human expertise**
* **Automatically respond** to straightforward inquiries
So we rolled up our sleeves and built a solution with Datazone
## The Setup: Three Core Components
Our solution consists of three main parts:
1. **Data Source**: A JSON file containing customer support messages
2. **AI Analysis Engine**: **Claude via Datazone's `Agent` SDK** for structured output
3. **Datazone Pipeline**: It's the simplest part of the flow thanks to **Datazone**. 😎
Here's how we built it:
### The Data: Raw Customer Messages
Here's what our sample data looks like:
```json theme={null}
[
{
"mail": "john.smith@gmail.com",
"subject": "Card Not Working",
"message_content": "Hello, my bank card has balance but it didn't work when I tried to make a purchase.",
"source": "email",
"timestamp": "2025-08-01T09:15:30Z"
}
]
```
We have used a sample JSON file to simulate incoming customer emails. You can find it in the example repository which I'll share in below.
### Structured Output, Straight From the Pipeline
We created a Pydantic model to structure the AI's analysis:
```python theme={null}
class SentimentAnalysisResultFormat(BaseModel):
issue_title: str = Field(description="Brief title for the issue")
summary: str = Field(description="Summary of the message")
response: str = Field(description="Response to the message")
solving_score: int = Field(description="Confidence score, 1-10")
request_type: Literal["information_request", "complaint", "suggestion"] = Field()
sentiment_category: Literal["positive", "negative", "neutral"] = Field()
urgency_level: int = Field(description="Urgency level, 1-10")
```
This gives us structured data we can actually use in our workflow, rather than raw text.
Creating the agent is a one-liner — no provider SDK, no API key in the code:
```python theme={null}
from datazone import Agent
agent = Agent(
model="CLAUDE_45_SONNET",
response_format=SentimentAnalysisResultFormat,
)
```
The model credentials live in a [Model Account](/reference/development/model-accounts), and Datazone injects them at run time. Omit `model` and the organisation's default model is used; token usage is recorded against the pipeline automatically.
### The Instructions: Guiding the AI
We wrote a clear set of instructions for the AI in a markdown file:
```markdown theme={null}
You are an automation agent for responding to user feedback.
...
Scale the solving_score as follows:
- 1-3: Not confident. Use when you're unsure.
- 4-6: Moderately confident. Use for complaints needing more info.
- 7-8: Confident. Use for basic information requests.
- 9-10: Very confident. Use for simple suggestions or appreciation.
```
This helps the AI make consistent assessments about each incoming message. You can access the full instructions in the [example repository](https://github.com/datazoneco/examples/blob/main/ai-automated-customer-message-response/prompt.md).
### The Pipeline: Orchestrating the Workflow
Here's the Datazone pipeline we built:
```python theme={null}
@transform(description="Reads emails from file", engine="pandas")
def read_emails():
with open("sample_data.json", "r") as file:
data = json.load(file)
return pd.DataFrame(data)
@transform(
input_mapping={"emails": Input(read_emails)},
engine="pandas"
)
def analyze_emails(emails):
for index, row in emails.iterrows():
user_message = f"Email: {row['mail']}\nSubject: {row['subject']}\nContent: {row['message_content']}"
# Process with AI
result = agent.invoke({
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
]
})
analysis = result["structured_response"]
# Store analysis results
emails.at[index, "issue_title"] = analysis.issue_title
emails.at[index, "sentiment_category"] = analysis.sentiment_category
emails.at[index, "solving_score"] = analysis.solving_score
emails.at[index, "response"] = analysis.response
emails.at[index, "status"] = "ANALYZED"
return emails
@transform(
input_mapping={"analyzed_emails": Input(analyze_emails)},
engine="pandas"
)
def handle_emails(analyzed_emails):
# Determine which emails need human review
for index, row in analyzed_emails.iterrows():
if row["solving_score"] < 8:
analyzed_emails.at[index, "status"] = "NEED_ACTION"
return analyzed_emails
```
Three transformations is all it takes to build a complete email processing system!
## The Decision Engine: Human or Machine?
The key part of the system is the logic that decides which emails need human attention:
```python theme={null}
# If the AI confidence is below our threshold, route to human team
if row["solving_score"] < 8:
# Mark for human review
emails.at[index, "status"] = "NEED_ACTION"
```
When the AI isn't confident in its response, it flags the message for human review. This ensures customers always get the right answer, whether it comes from AI or the support team.
## Final Data ✨
```json theme={null}
{
"mail": "john.smith@gmail.com",
"subject": "Card Not Working",
"message_content": "Hello, my bank card has balance but it didn't work when I tried to make a purchase.",
"source": "email",
"timestamp": "2025-08-01T09:15:30Z",
"issue_title": "Bank Card Payment Failure",
"sentiment_category": "negative",
"solving_score": 7,
"response": "I'm sorry to hear about the trouble with your bank card. This could be due to several reasons such as a temporary system issue, a hold on your account, or a security feature triggered by unusual purchase activity. Please try the following steps:\n\n1. Try the card again at a different terminal\n2. Contact your bank's customer service at the number on the back of your card\n3. Check your online banking to ensure there are no restrictions on your account\n\nIf you continue to experience issues, please reply with more details about where you attempted to make the purchase and we'll investigate further.",
"status": "NEED_ACTION"
}
```
## The Dashboard: Real-time Monitoring
We built a dashboard in Datazone to monitor the system:
The dashboard tracks:
* Total message volume
* Average urgency levels
* Messages requiring human intervention
* Sentiment distribution across all communications
Please don't assume I moved the data to another platform or used a traditional BI tool to build this app! 😊 I'm still in Datazone and accomplished this with a simple command to Orion AI. For more details, check out the [Intelligent App](/reference/intelligent-apps/overview) section.
## The Results: Real Business Impact
After deploying the system:
* Approximately **60% of customer emails** are now handled **automatically**
* Response time has decreased to **under 1 hour**
* Support team can focus on complex issues that truly need human expertise
* Customer satisfaction has improved due to faster response times
And we built the entire system in just a few hours using Datazone's platform.
## Next Steps for this Project
* Using tools for enhanced actions by LLM like accessing **user detail, previous interactions, and context** to improve response accuracy.
* Give **more context** about the business case to find better insights and responses.
* Use some **prompt engineering** methods like **few-shot prompting** to categorize and notice correct sentiment.
* Implement **feedback loops** to continuously improve the model's performance based on real-world interactions.
## Resources
* [Datazone Documentation](https://docs.datazone.com)
* [GitHub Project Repository](https://github.com/datazoneco/examples/blob/main/ai-automated-customer-message-response/)
# Pipeline Examples
Source: https://docs.datazone.co/tutorial/examples/pipeline-examples
Learn how to create and manage data pipelines in Datazone through practical examples
# Pipeline Examples
## Basic Data Pipeline
### Example 1: CSV to Processed Dataset
This example demonstrates uploading a CSV file to Datazone and then performing basic transformations, and saving the results.
```python theme={null}
from datazone import transform, Input, Dataset
# Upload data to project and then use it in the transform
@transform(
input_mapping={
"sales_raw": Input(Dataset(alias="sales_raw_csv"))
}
)
def clean_sales_data(sales_raw):
# Remove duplicates and null values
df = sales_raw.dropDuplicates()
df = df.na.drop()
# Convert date string to timestamp
df = df.withColumn("sale_date", to_timestamp("sale_date"))
return df
# Calculate daily metrics
@transform(
input_mapping={
"clean_sales": Input(clean_sales_data)
}
)
def calculate_daily_metrics(clean_sales):
return clean_sales.groupBy("sale_date").agg(
sum("amount").alias("daily_total"),
count("*").alias("transaction_count"),
avg("amount").alias("average_transaction")
)
```
## Data Quality Pipeline
### Example 2: Data Validation and Reporting
```python theme={null}
from datazone import transform, Input, Dataset
@transform(
input_mapping={
"customer_data": Input(Dataset(alias="customer_records"))
}
)
def validate_customer_data(customer_data):
# Check for required fields
validation_df = customer_data.select(
when(col("email").isNull(), "Missing Email")
.when(~col("email").rlike("^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}$"), "Invalid Email")
.otherwise("Valid").alias("email_status"),
when(col("phone").isNull(), "Missing Phone")
.when(~col("phone").rlike("^\\+?[1-9]\\d{1,14}$"), "Invalid Phone")
.otherwise("Valid").alias("phone_status")
)
return validation_df
@transform(
input_mapping={
"validation_results": Input(validate_customer_data)
}
)
def generate_validation_report(validation_results):
return validation_results.groupBy(
"email_status", "phone_status"
).count()
```
## Multi-Source Pipeline
### Example 4: Combining Data from Multiple Sources
```python theme={null}
from datazone import transform, Input, Dataset
@transform(
input_mapping={
"orders": Input(Dataset(alias="order_data")),
"customers": Input(Dataset(alias="customer_data")),
"products": Input(Dataset(alias="product_catalog"))
}
)
def create_order_summary(orders, customers, products):
# Join orders with customer data
orders_with_customers = orders.join(
customers,
orders.customer_id == customers.id,
"left"
)
# Join with product data
complete_orders = orders_with_customers.join(
products,
orders.product_id == products.id,
"left"
)
return complete_orders.select(
"order_id",
"order_date",
"customer_name",
"product_name",
"quantity",
"total_amount"
)
```
## Logging Pipeline
### Example 6: Pipeline with Comprehensive Logging
```python theme={null}
from datazone import transform, logger, Input, Dataset
from pyspark.sql.functions import col, count
@transform(
input_mapping={
"sales_data": Input(Dataset(id="daily_sales"))
}
)
def process_sales_with_logging(sales_data):
try:
logger.info(f"Starting sales data processing. Row count: {sales_data.count()}")
# Log data quality metrics
null_counts = sales_data.select([count(when(col(c).isNull(), c)).alias(c)
for c in sales_data.columns])
logger.info(f"Null value counts: {null_counts.toPandas().to_dict()}")
# Process data
logger.info("Applying transformations...")
processed_df = sales_data.filter(col("amount") > 0)
# Calculate aggregates
daily_totals = processed_df.groupBy("date").agg(
sum("amount").alias("total_sales")
)
logger.info(f"Generated daily totals. Output rows: {daily_totals.count()}")
return daily_totals
except Exception as e:
logger.error(f"Error processing sales data: {str(e)}")
logger.error(f"Error details:", exc_info=True) # Logs full stack trace
raise
@transform(
input_mapping={
"daily_totals": Input(process_sales_with_logging)
}
)
def validate_totals_with_logging(daily_totals):
logger.info("Starting totals validation")
try:
# Validate results
invalid_totals = daily_totals.filter(col("total_sales") < 0)
invalid_count = invalid_totals.count()
if invalid_count > 0:
logger.warning(f"Found {invalid_count} invalid total(s)")
logger.debug(f"Invalid records: {invalid_totals.collect()}")
else:
logger.info("All totals validated successfully")
return daily_totals
except Exception as e:
logger.critical(f"Critical error in validation: {str(e)}")
raise
```
Different logging levels are available:
* `logger.debug()`: Detailed information for debugging
* `logger.info()`: General information about pipeline progress
* `logger.warning()`: Warning messages for potential issues
* `logger.error()`: Error messages for caught exceptions
* `logger.critical()`: Critical failures that require immediate attention
## Usage Instructions
1. Save these transforms in your project's transform directory
2. Configure the dataset IDs to match your environment
3. Create a pipeline including the transforms in the desired order
4. Set up appropriate scheduling and monitoring
# Pyspark Examples in Datazone Transforms
Source: https://docs.datazone.co/tutorial/examples/pyspark-transform-examples
Welcome to this focused guide on utilizing PySpark within the Datazone platform. This document is tailored to illustrate how PySpark can be seamlessly integrated into Datazone Transforms, enabling efficient data processing and transformation.
Using three key datasets - `orders`, `SKUs`, and `customers` - as our foundation, we will explore various PySpark operations. Each example is designed to demonstrate practical applications in data transformation, showcasing the versatility and power of PySpark in a Datazone environment.
From basic operations like *projection* and *filtering* to more advanced techniques such as *joins*, *unions*, and *aggregations*, each section of this guide offers concise yet comprehensive insights into PySpark's capabilities. Whether you're a beginner or an experienced user, these examples provide a clear pathway to enhance your data processing workflows in Datazone using PySpark.
In the upcoming examples, the terms `dataset` and `dataframe` are used interchangeably. This means that whenever either term is mentioned, it refers to the same concept of a structured collection of data within our context.
Please note that the usage of `Dataset(id="")` in the provided code examples serves merely as an illustrative placeholder. It is important to replace the string `` with the actual identifier of your specific orders dataset when implementing these examples in your environment. This ensures that the code correctly references and interacts with your dataset.
## Projection
Projection in PySpark is used to select specific columns from a DataFrame. This operation is similar to the `SELECT` statement in SQL. It is useful when you want to work with only a subset of columns in your dataset.
*Example Use Case*: Selecting only the `order_id` and `order_date` columns from an orders DataFrame.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def select_orders(orders):
return orders.select("order_id", "order_date")
```
## Filter
The filter operation is used to retrieve rows from a DataFrame that meet a specific condition. This is akin to the `WHERE` clause in SQL. It allows for both simple and complex filtering criteria.
*Example Use Case*: Fetching orders made after a certain date.
Filtering records based on a condition.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def filter_orders(orders):
return orders.filter(orders.order_date > "2023-01-01")
```
Filtering using conditions on multiple columns.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def filter_orders_multi(orders):
return orders.filter((orders.order_date > "2023-01-01") & (orders.customer_id == 102))
```
## Column Rename
Column renaming is used to change the name of a column in a DataFrame. This is particularly useful for improving readability or when column names need to conform to certain naming conventions.
*Example Use Case*: Renaming `order_date` to `date_of_order` for clarity.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def rename_order_column(orders):
return orders.withColumnRenamed("order_date", "date_of_order")
```
## On-the-fly Columns
Creating on-the-fly columns involves adding new columns to a DataFrame, often with calculated or static values. This is useful for adding derived metrics or flags to your data.
*Example Use Case*: Adding a new column status to an orders DataFrame to indicate processing status.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql.functions import lit
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def add_new_column(orders):
return orders.withColumn("status", lit("processed"))
```
## Sorting
Sorting refers to arranging data in a specified order. In PySpark, the `orderBy` function is used to sort the DataFrame based on one or more columns, either in ascending or descending order.
Detailed Usage:
* `orderBy("column")`: Sorts the DataFrame in ascending order based on the specified column.
* `orderBy("column"`, ascending=False): Sorts in descending order.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql.functions import lit
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def sort_orders_by_date(orders):
return orders.orderBy("order_date")
```
## Joins
Join operations are used to combine two DataFrames based on a common key or condition. This is similar to joins in SQL and is essential for merging related datasets.
Types of Joins:
* *Inner Join*: Returns rows that have matching values in both DataFrames.
* *Left/Right Outer Join*: Returns all rows from the left/right DataFrame and matched rows from the other DataFrame.
* *Full Outer Join*: Returns all rows when there is a match in one of the DataFrames.
* *Anti Join*: Returns rows from the left DataFrame that do not have matching keys in the right DataFrame.
*Example Use Case*:
* Inner Join: Find customers who have placed orders (common in both datasets).
* Left Outer Join: Find all customers and their order details, if any.
* Right Outer Join: Find all orders and their customer details, if any.
* Anti Join: Find customers who have not placed any orders.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
@transform(input_mapping={"customers": Input(Dataset(id="dataset_customers_id")),
"orders": Input(Dataset(id="dataset_orders_id"))})
def perform_various_joins(customers, orders):
# Inner Join: Customers with orders
inner_join_df = customers.join(orders, customers.customer_id == orders.customer_id, "inner")
# Left Outer Join: All customers, with order details if available
left_outer_join_df = customers.join(orders, customers.customer_id == orders.customer_id, "left_outer")
# Right Outer Join: All orders, with customer details if available
right_outer_join_df = customers.join(orders, customers.customer_id == orders.customer_id, "right_outer")
# Anti Join: Customers without orders
anti_join_df = customers.join(orders, customers.customer_id == orders.customer_id, "left_anti")
return inner_join_df, left_outer_join_df, right_outer_join_df, anti_join_df
```
## Union
The union operation is used to combine two DataFrames with the same schema (i.e., number and type of columns) by appending the rows of one DataFrame to another.
*Example Use Case*: Merging two datasets of orders from different sources or time periods into a single DataFrame.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
@transform(input_mapping={"orders1": Input(Dataset(id="dataset_orders_id1")),
"orders2": Input(Dataset(id="dataset_orders_id2"))})
def union_orders(orders1, orders2):
return orders1.union(orders2)
```
## Aggregation
Aggregation operations are used to compute summary statistics or other complex aggregations on a DataFrame. These operations often go hand in hand with group by functionality.
*Example Use Case*: Calculating total sales per product or average order value.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql import functions as F
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def aggregate_total_orders_per_customer(orders):
return orders.groupBy("customer_id").agg(F.count("order_id").alias("total_orders"))
```
In this code snippet:
* `groupBy("customer_id")`: Groups the data by the customer\_id column.
* `agg(...)`: Performs the specified aggregation function, which is counting the number of order\_id for each group.
* `alias("total_orders")`: Renames the result of the aggregation to total\_orders for clarity.
## Pivot
Pivoting is used to rotate data from a long format to a wide format. It can summarize data and is useful in data reshaping and analysis.
Detailed Usage:
* `groupBy("column").pivot("pivot_column")`: Groups the data by a column and pivots on another column.
* Functions like `sum()`, `count()`, etc., can be applied to the pivoted data for aggregation.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql import functions as F
@transform(input_mapping={"sales": Input(Dataset(id="dataset_sales_id"))})
def pivot_sales_data(sales):
return sales.groupBy("year").pivot("category").sum("sales")
```
## Window Functions
Window functions are used for performing calculations across a set of rows that are somehow related to the current row. This is useful for running totals, moving averages, ranking, etc.
Detailed Usage:
* Define a Window specification using `Window.partitionBy("column").orderBy("other_column")`.
* Apply window functions like `rank()`, `row_number()`, `lead()`, `lag()`, etc.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql import functions as F
from pyspark.sql.window import Window
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def rank_orders(orders):
windowSpec = Window.partitionBy("customer_id").orderBy("order_date")
return orders.withColumn("rank", F.rank().over(windowSpec))
```
## UDFs (User Defined Functions)
UDFs allow you to extend the functionality of PySpark by defining custom functions in Python. These functions can then be used in DataFrame transformations.
Detailed Usage:
1. Define a Python function.
2. Register it as a `UDF`.
3. Apply the `UDF` to a DataFrame column.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql import functions as F
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def apply_discount(orders):
calculate_discount_udf = F.udf(lambda price: price * 0.9, FloatType())
return orders.withColumn("discounted_price", calculate_discount_udf(F.col("price")))
```
## Handling Missing Data
Dealing with null or missing data is a common task. PySpark provides functions to drop, fill, or replace these missing values.
Detailed Usage:
* `fillna()`: Fills null values with specified value(s).
* `dropna()`: Drops rows with null values.
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql import functions as F
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def fill_missing_prices(orders):
return orders.fillna({"price": 0})
```
## A comprehensive Transform Example
To demonstrate a scenario where multiple PySpark operations are used within a single transform block, let's consider a hypothetical analysis on an orders dataset. In this scenario, we will:
1. Filter the orders for a specific year.
2. Sort these orders by order date.
3. Add a new column indicating the order size category.
4. Calculate the total value of orders for each customer.
5. Finally, pivot this data to show the sum of order values per month for each customer.
Here is how you can implement this using the transform decorator and datazone module:
```python filename="transform.py" copy theme={null}
from datazone import transform, Input, Dataset
from pyspark.sql import functions as F
from pyspark.sql.types import StringType
@transform(input_mapping={"orders": Input(Dataset(id="dataset_orders_id"))})
def comprehensive_orders_analysis(orders):
# Filter orders from a specific year
filtered_orders = orders.filter(F.year("order_date") == 2023)
# Sort by order date
sorted_orders = filtered_orders.orderBy("order_date")
# Define a UDF to categorize order size
def order_size_category(order_value):
if order_value < 100:
return "Small"
elif order_value <= 500:
return "Medium"
else:
return "Large"
order_size_udf = F.udf(order_size_category, StringType())
categorized_orders = sorted_orders.withColumn("order_size", order_size_udf(F.col("order_value")))
# Aggregate total value of orders for each customer
total_value_per_customer = categorized_orders.groupBy("customer_id").agg(F.sum("order_value").alias("total_value"))
# Pivot data to show sum of order values per month for each customer
pivoted_data = total_value_per_customer.groupBy("customer_id").pivot("month").sum("total_value")
return pivoted_data
```
### Explanation of the Code:
* `Filtering`: Only includes orders from the year 2023.
* `Sorting`: Orders are sorted by their date.
* `UDF Application`: A user-defined function categorizes each order based on its value.
* `Aggregation`: The total value of orders is computed for each customer.
* `Pivoting`: The data is then pivoted to show the sum of order values per month for each customer.
### Note:
* Ensure that `dataset_orders_id` is replaced with the actual dataset ID in your implementation.
* The code assumes the presence of columns like `order_date`, `order_value`, etc., in your orders DataFrame.
* UDFs might have performance implications; consider using built-in functions wherever possible.