> ## Documentation Index
> Fetch the complete documentation index at: https://docs.datazone.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Conversation Evaluations

> Let an AI reviewer score agent, intelligent app and Orion conversations so you can find poor answers and improve your assistants

# Evaluating Conversations

Your users ask your **agents**, **intelligent app chats** and **Orion** questions every day. Conversation evaluations tell you **how well those conversations went**, without anyone reading them one by one.

An **AI reviewer** reads each conversation once it is finished and gives it:

* a **score** from 1 to 5 stars
* a **verdict**: Good, Needs review or Bad
* a short **summary** of what the conversation was about
* **flags** for the problems it found, such as an answer that is not backed by the data
* **notes** explaining the score, and up to three **suggestions** for improving the assistant

You find the results on the **Conversations** page, where you can filter for the conversations that need a look, read them, and mark them as reviewed.

## Turning Evaluations On

Evaluations are **off by default**, since they use tokens of your model account. To turn them on, go to **Settings → Orion AI** and use the **Agent Evaluation** card.

![Agent Evaluation settings card](https://mintcdn.com/datazone/pg6wMEUvd8Ky6fkr/images/light/agent/agent-evaluation-settings.png?fit=max\&auto=format\&n=pg6wMEUvd8Ky6fkr\&q=85\&s=1a6d5bcec6f3519347edaaa00d0ee408)

**Evaluate conversations** `switch` (default: off)

Evaluates conversations automatically. When it is off, nothing is evaluated on its own, and you can still evaluate a single conversation by hand from the Conversations page.

**Conversations to evaluate** `checkboxes` (default: all)

Which assistants are evaluated: **Agents**, **Intelligent Apps** and **Orion**. Playground chats are never evaluated.

**Evaluation focus** `text` (default: empty)

Extra instructions for the reviewer, up to 2,000 characters. Use it for what matters in your organisation, for example *"Check that every number in an answer comes from a query result"* or *"Revenue always means net revenue"*. It is added to the reviewer's fixed rubric, so scores stay comparable over time.

### Choosing the Evaluation Model

The reviewer uses the model set in **Settings → Orion AI → Default Configurations → Agent Evaluation**. When nothing is set there, it uses the organisation's default model. The card shows which model is currently used.

> [!TIP]
>
> A **fast, low-cost model** is usually enough for evaluations. The reviewer reads the conversation; it does not have to answer it.

## When a Conversation Is Evaluated

* A conversation is evaluated **30 minutes after its last question**, so a conversation that is still going is not scored halfway through.
* When someone **asks something new** in an evaluated conversation, it is evaluated again and the score is updated.
* When you turn evaluations on, conversations from the **last day** are evaluated. Older conversations are not, so you are not billed for your whole history; you can still evaluate them by hand.

> [!NOTE]
>
> Evaluations run in the background. A conversation shows **Evaluating** while the reviewer is working on it, which usually takes less than a minute.

## The Conversations Page

Open **Settings → Conversations**. Every row is one conversation, the most recently asked first:

| Column | What it shows |
| - | - |
| **Conversation** | The conversation title, the assistant type (**Agent**, **Intelligent App** or **Orion**) and the agent, app or project it belongs to |
| **User** | Who started the conversation |
| **First question** | The first question that was asked |
| **Questions** | How many questions were asked, with markers for failed answers and thumbs down |
| **Score** | The score out of 5 stars, or *Not evaluated* / *Evaluating* |
| **Review** | Whether someone has already looked at the evaluation |
| **Summary** | What the conversation was about, with the flags the reviewer raised |
| **Last asked** | When the last question was asked |

### Filtering

Narrow the list by **date**, **user**, **assistant type** and **quality**:

* **To review** shows conversations judged *Needs review* or *Bad* that nobody has reviewed yet. This is the best place to start.
* **Good**, **Needs review** and **Bad** show conversations with that verdict.

The **Stars** filter works like a hotel search: tick any number of ratings to see only those conversations.

## Reading an Evaluation

Click a conversation to open it. The **evaluation** is at the top, followed by every **question** and **answer** in the conversation.

![Conversation detail with the evaluation card and the questions](https://mintcdn.com/datazone/pg6wMEUvd8Ky6fkr/images/light/agent/agent-evaluation-detail.png?fit=max\&auto=format\&n=pg6wMEUvd8Ky6fkr\&q=85\&s=5652578d97f32785d26a39ad5c7a308b)

### Score and Verdict

| Score | Meaning |
| - | - |
| ⭐ 5 | Excellent, nothing to improve |
| ⭐ 4 | Good, minor issues |
| ⭐ 3 | Acceptable, but clearly weak somewhere |
| ⭐ 2 | Poor, the user was not really helped |
| ⭐ 1 | Wrong, harmful or failed |

The **verdict** says whether a person should look at the conversation: **Good** (nothing to check), **Needs review** or **Bad** (the assistant failed the user).

### Flags

The reviewer judges each conversation against a fixed rubric: did the assistant **complete the task**, are its answers **backed by the data it queried**, is it **clear and helpful**, and did it **use its tools well**. Problems become flags:

| Flag | Raised when |
| - | - |
| **Wrong answer** | An answer is incorrect |
| **Ungrounded** | An answer states facts or numbers that the query results do not support |
| **Unanswered** | A question was left unanswered or answered off topic |
| **Tool misuse** | Wrong, needless or repeated tool calls, or tool errors left unhandled |
| **User frustrated** | The user repeats themselves, corrects the assistant or complains |
| **Error** | An answer failed with an error |
| **Thumbs down** | The user gave an answer a thumbs down |
| **Guardrail** | The agent's guardrails detected a violation |

The last three are read directly from the conversation, without the reviewer.

### Notes and Suggestions

**Notes** explain the score, often naming the turn where things went wrong. **Suggestions** are concrete changes to the assistant's instructions or setup that would have made the conversation better, for example *"Tell the agent to ask the user before substituting a proxy metric"*. Adding a good suggestion to the [agent's instructions](https://docs.datazone.co/reference/agents/overview.md) is often the quickest fix.

### Reviewing

* **Mark reviewed** records that you have looked at the evaluation, and takes the conversation out of the **To review** filter. **Reopen** puts it back.
* **Re-evaluate** evaluates the conversation again right away, for example after you change the evaluation focus. It also works when automatic evaluations are off.
* **Show trace** under a question opens everything the assistant did to answer it: the context it was given, its reasoning, every tool call and its result.

## What the Reviewer Sees

For every question, the reviewer reads the **question**, each **tool call** with its arguments and a preview of its result, the **final answer**, whether the answer **failed**, and the user's **thumbs up or down**. Long texts are shortened, and in a very long conversation the earliest turns are left out.

The reviewer does **not** see the agent's instructions or the full query results. It judges correctness from the tool results it is shown, and says so when it cannot verify something.

## Permissions

The Conversations page and the evaluation settings are available to users with the **Orion manage** permission.

> **[Agent Best Practices](https://docs.datazone.co/reference/agents/best-practices.md)**
>
> Use what your evaluations show to write better instructions and pick the right model


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.