> ## Documentation Index
> Fetch the complete documentation index at: https://docs.datazone.co/llms.txt
> Use this file to discover all available pages before exploring further.
# Vectors
> Transform your data into searchable embeddings for AI-powered applications
Datazone Vectors enable you to **transform your data into embeddings** and store them in a vector database. This allows you to build **AI-powered search**, **RAG (Retrieval Augmented Generation)** applications, and **semantic similarity** features on top of your data.
## What are Vectors?
Vectors convert your text data into **numerical representations** (embeddings) that capture **semantic meaning**. This enables:
* **Semantic search** - Find relevant content based on meaning, not just keywords
* **RAG applications** - Enhance AI agents with contextual information from your data
* **Similarity matching** - Identify related documents or records
Unlike traditional keyword search, vector-based search understands **context and intent**, delivering more accurate and relevant results.
## Creating a Vector Index
1. Navigate to your **Project** page
2. Click the **Add** button (+ icon)
3. Select **Vector** from the dropdown
The vector creation flow guides you through **configuring your vector index**:
### Step 1: Choose Data Source
Select what data to vectorize:
* **Dataset** - Index data from a dataset (CSV format)
* Select the dataset you want to vectorize
* Choose a **primary key column** to uniquely identify records
* Select one or more **text columns** to embed
* **File Container** - Index files from a file container path
* Supported formats: **PDF**, **CSV**, **DOCX**, **XLSX**, **Markdown**, **TXT**
* Files will be automatically processed and chunked
For large datasets, consider using **Views** or filtered datasets to index only the most relevant data.
### Step 2: Embedding Configuration
Configure how your data will be embedded:
* **Model Account** - Select a configured [Model Account](/reference/development/model-accounts) with embedding support
* **Embedding Model** - Choose the embedding model (e.g., `text-embedding-3-small`, `text-embedding-ada-002`)
* **Vector Dimension** - Embedding dimension (typically 1536 for OpenAI models)
### Step 3: Chunking Strategy
Configure how your text is split into chunks before embedding:
* **Text Splitting** - Split by character count
* **Chunk Size** - Number of characters per chunk
* **Chunk Overlap** - Characters shared between chunks (helps maintain context)
* **Length Splitting** - Split by token count using a specific encoding
* **Encoding Name** - Tokenizer to use (e.g., `cl100k_base` for GPT models)
* **Chunk Size** - Number of tokens per chunk
* **Chunk Overlap** - Tokens shared between chunks
* **Document Splitting** - Split based on document structure
* **Document Type** - Choose format: Markdown, JSON, Code, or HTML
* Preserves logical document boundaries
Smaller chunks provide more precise search results but may lose broader context. Larger chunks retain more context but may be less specific. A typical chunk size is **500-1000 characters** with **10-20% overlap**.
The same step also sets the **Search Type** (and, for keyword search, the **Keyword Language**). See [Search Types](#search-types) below.
After completing configuration, click **Create** to start the indexing process.
## Search Types
Each vector uses one search type. Set it when you create the vector, or change it later from the vector's edit dialog. The vector **Overview** page shows the current search type and language.
| Search Type | How it matches | When to use |
| - | - | - |
| **Dense (Semantic)** — default | Embedding similarity: finds content with the same meaning, even when the words differ | Natural-language questions, RAG for agents |
| **Sparse (Keyword)** | BM25 keyword search: finds content that contains the query's words | Product codes, names, IDs, and other exact terms |
| **Hybrid** | Runs semantic and keyword search, then merges the two rankings with Reciprocal Rank Fusion (RRF) | Mixed queries where both meaning and exact terms matter |
### Keyword Language
Sparse and Hybrid search need a **Keyword Language** (default: English). Keyword search uses it to remove stopwords and to reduce words to their stems, so set it to the language of your content. Supported languages: Arabic, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, and Turkish.
### Hybrid Weights
With **Hybrid**, two extra fields set how much each ranking counts when the results are merged:
* **Semantic Weight** — weight of the semantic (dense) ranking
* **Keyword Weight** — weight of the keyword (sparse) ranking
Both default to `1` and accept values greater than `0` and up to `100`. Equal weights treat both rankings the same.
Weights apply to **ranks, not scores**. With a semantic weight of `3` and a keyword weight of `1`, a 3rd-place semantic match counts the same as a 1st-place keyword match. A higher weight lets more of that method's lower-ranked matches reach the top.
If keyword-heavy queries (codes, names) return too many loosely related semantic matches, raise the **Keyword Weight**. If natural-language questions miss relevant content, raise the **Semantic Weight**.
### Changing the Search Configuration
* Changing the **search type** or **keyword language** rebuilds the index. Until the reindex finishes, searches keep using the current index.
* Changing the **hybrid weights** does not rebuild the index. The new weights apply from the next search.
- Sparse vectors still need an embedding model in their configuration, but keyword search does not call it, so there is no embedding cost.
- Semantic scores are between 0 and 1. Keyword and hybrid scores are not, so the **Explore** tab shows them as raw scores instead of percentages, and a **score threshold** only filters semantic results (for Hybrid, it applies to the semantic part).
Agents, [endpoints](/reference/integration/endpoints#vector-based-endpoints), and actions that search a vector all use that vector's search type automatically.
## Indexing Process
Once created, your vector index goes through several states:
1. **Not Indexed** - Initial state after creation
2. **Scheduled** - Queued for indexing
3. **Indexing** - Currently processing and embedding your data
4. **Indexed** - Successfully completed, ready to use
5. **Failed** - Error occurred during indexing (check error details)
You can monitor the **indexing status** and view **statistics** including:
* Total chunks created
* Total chunks indexed
* Total tokens processed
## Using Vectors
Once indexed, your vectors can be used for:
### Search
Search your data using **natural language queries** that understand meaning and context, not just exact keyword matches. The **Explore** tab provides an interface to:
* Enter search queries in natural language
* View results ranked by relevance, using the vector's [search type](#search-types)
* See matching chunks with their context and metadata
* Test and refine your vector search results
### Agent RAG
Attach vectors as data sources for your [Agents](/reference/agents/overview), enabling them to retrieve relevant context from your data to provide more accurate and informed responses. Agents can automatically perform semantic search when answering questions.
### Similarity Endpoints
Create [API endpoints](/reference/integration/endpoints#vector-based-endpoints) that return similar records based on vector similarity, enabling semantic search in your applications. Perfect for building search features or recommendation systems.
## Best Practices
1. **Choose the Right Source** - Use datasets for structured data and file containers for documents
2. **Optimize Chunk Size** - Balance between context (larger chunks) and precision (smaller chunks)
3. **Add Overlap** - Include 10-20% overlap to maintain context across chunk boundaries
4. **Select Appropriate Models** - Smaller embedding models are faster and cheaper, larger models may provide better quality
5. **Use Cosine Similarity** - Works well for most text similarity use cases
6. **Monitor Indexing** - Check indexing status and statistics to ensure successful processing
## Next Steps
* [Configure Model Accounts](/reference/development/model-accounts) for embedding models
* [Create Agents](/reference/agents/overview) to leverage your vectors
* [File Containers](/reference/development/file-container) for document management
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.