source-empty

​
The Problem

Customer support inboxes get flooded with emails daily. Some need expert human attention, others are routine questions that keep coming up again and again.
source-empty
We wanted to create a system that could:
  • Process incoming customer emails automatically
  • Determine the sentiment and urgency of each message
  • Identify which ones need human expertise
  • Automatically respond to straightforward inquiries
So we rolled up our sleeves and built a solution with Datazone

​
The Setup: Three Core Components

Our solution consists of three main parts:
Customer Message Automation Architecture
  1. Data Source: A JSON file containing customer support messages
  2. AI Analysis Engine: Anthropic Claude 3.7 with Pydantic AI for structured output
  3. Datazone Pipeline: It’s the simplest part of the flow thanks to Datazone. 😎
Here’s how we built it:

​
The Data: Raw Customer Messages

Here’s what our sample data looks like: 
[
  {
    "mail": "john.smith@gmail.com",
    "subject": "Card Not Working",
    "message_content": "Hello, my bank card has balance but it didn't work when I tried to make a purchase.",
    "source": "email",
    "timestamp": "2025-08-01T09:15:30Z"
  }
]
We have used a sample JSON file to simulate incoming customer emails. You can find it in the example repository which I’ll share in below.

​
Datazone + Pydantic AI: Perfect Together

We created a Pydantic model to structure the AI’s analysis: 
class SentimentAnalysisResultFormat(BaseModel):
    issue_title: str = Field(description="Brief title for the issue")
    summary: str = Field(description="Summary of the message")
    response: str = Field(description="Response to the message")
    solving_score: int = Field(description="Confidence score, 1-10")
    request_type: Literal["information_request", "complaint", "suggestion"] = Field()
    sentiment_category: Literal["positive", "negative", "neutral"] = Field()
    urgency_level: int = Field(description="Urgency level, 1-10")
This gives us structured data we can actually use in our workflow, rather than raw text. 
agent = Agent(
    model=AnthropicModel(
        provider=AnthropicProvider(api_key=ANTHROPIC_API_KEY),
        model_name="claude-3-7-sonnet-latest",
    ),
    output_type=SentimentAnalysisResultFormat,
    system_prompt=SYSTEM_PROMPT,
)

​
The Instructions: Guiding the AI

We wrote a clear set of instructions for the AI in a markdown file: 
You are an automation agent for responding to user feedback.
...
Scale the solving_score as follows:
    - 1-3: Not confident. Use when you're unsure.
    - 4-6: Moderately confident. Use for complaints needing more info.
    - 7-8: Confident. Use for basic information requests.
    - 9-10: Very confident. Use for simple suggestions or appreciation.
This helps the AI make consistent assessments about each incoming message. You can access the full instructions in the example repository.

​
The Pipeline: Orchestrating the Workflow

Here’s the Datazone pipeline we built: 
@transform(description="Reads emails from file", engine="pandas")
def read_emails():
    with open("sample_data.json", "r") as file:
        data = json.load(file)
    return pd.DataFrame(data)

@transform(
    input_mapping={"emails": Input(read_emails)},
    engine="pandas"
)
def analyze_emails(emails):
    for index, row in emails.iterrows():
        user_message = f"Email: {row['mail']}\nSubject: {row['subject']}\nContent: {row['message_content']}"
        
        # Process with AI
        result = agent.run_sync(user_message)
        
        # Store analysis results
        emails.at[index, "issue_title"] = result.output.issue_title
        emails.at[index, "sentiment_category"] = result.output.sentiment_category
        emails.at[index, "solving_score"] = result.output.solving_score
        emails.at[index, "response"] = result.output.response
        emails.at[index, "status"] = "ANALYZED"
    
    return emails

@transform(
    input_mapping={"analyzed_emails": Input(analyze_emails)},
    engine="pandas"
)
def handle_emails(analyzed_emails):
    # Determine which emails need human review
    for index, row in analyzed_emails.iterrows():
        if row["solving_score"] < 8:
            analyzed_emails.at[index, "status"] = "NEED_ACTION"
    
    return analyzed_emails
Three transformations is all it takes to build a complete email processing system!

​
The Decision Engine: Human or Machine?

The key part of the system is the logic that decides which emails need human attention: 
# If the AI confidence is below our threshold, route to human team
if row["solving_score"] < 8:
    # Mark for human review
    emails.at[index, "status"] = "NEED_ACTION"
When the AI isn’t confident in its response, it flags the message for human review. This ensures customers always get the right answer, whether it comes from AI or the support team.

​
Final Data ✨

{
"mail": "john.smith@gmail.com",
"subject": "Card Not Working",
"message_content": "Hello, my bank card has balance but it didn't work when I tried to make a purchase.",
"source": "email",
"timestamp": "2025-08-01T09:15:30Z",
"issue_title": "Bank Card Payment Failure",
"sentiment_category": "negative",
"solving_score": 7,
"response": "I'm sorry to hear about the trouble with your bank card. This could be due to several reasons such as a temporary system issue, a hold on your account, or a security feature triggered by unusual purchase activity. Please try the following steps:\n\n1. Try the card again at a different terminal\n2. Contact your bank's customer service at the number on the back of your card\n3. Check your online banking to ensure there are no restrictions on your account\n\nIf you continue to experience issues, please reply with more details about where you attempted to make the purchase and we'll investigate further.",
"status": "NEED_ACTION"
}

​
The Dashboard: Real-time Monitoring

We built a dashboard in Datazone to monitor the system:
Customer Message Automation Architecture
The dashboard tracks:
  • Total message volume
  • Average urgency levels
  • Messages requiring human intervention
  • Sentiment distribution across all communications
Please don’t assume I moved the data to another platform or used a traditional BI tool to build this app! 😊 I’m still in Datazone and accomplished this with a simple command to Orion AI. For more details, check out the Intelligent App section.

​
The Results: Real Business Impact

After deploying the system:
Customer Message Automation Impact
  • Approximately 60% of customer emails are now handled automatically
  • Response time has decreased to under 1 hour
  • Support team can focus on complex issues that truly need human expertise
  • Customer satisfaction has improved due to faster response times
And we built the entire system in just a few hours using Datazone’s platform.

​
Next Steps for this Project

  • Using tools for enhanced actions by LLM like accessing user detail, previous interactions, and context to improve response accuracy.
  • Give more context about the business case to find better insights and responses.
  • Use some prompt engineering methods like few-shot prompting to categorize and notice correct sentiment.
  • Implement feedback loops to continuously improve the model’s performance based on real-world interactions.

​
Resources