Evaluations is a new tab in the Digital AI (LLM Chatbot) builder that reviews your chatbot's past conversations and gives each one a grade. Instead of reading through every transcript to check how your bot is doing, you get a graded list that shows you where things went well and where they didn't.
Think of it as a second pair of eyes on your chatbot. It goes through completed interactions, summarises each one, and flags the conversations worth a closer look, so you can spend your time fixing the responses that need it rather than hunting for them.
Evaluations is a testing and maintenance tool. It only looks at conversations that have already happened, so it has no effect on your live chatbot or on the customer experience.
🔍 What are Evaluations?
Every conversation your Digital AI has can be scored on how well it handled the customer. Each evaluation gives you three things, a short summary of what the conversation was about, when it started, and a grade.
The three grades
The grade is a quick signal of how the conversation went. There are three:
Grade | What it usually means | What to do |
Good | The chatbot handled the conversation well. It answered accurately and stayed on task. | Nothing. These are working as intended. |
Warning | Mostly fine, but something minor is worth a look, such as a slightly off or incomplete answer. | Skim it when you have time. Fix if it's part of a pattern. |
Needs Attention | The chatbot likely got something wrong or handled the conversation poorly. | Review it and act. This is where improvements matter most. |
📊 Reading the Evaluations tab
The Evaluations tab has two parts: a filter panel at the top and a results list below it.
1. Filter to what you need
Use the filters to narrow the list down before you start reviewing.
- Start date / End date: set the date range of conversations you want to review.
- Grades: choose which grades to show. Select Needs Attention and Warning on their own to jump straight to the conversations that need work.
- Apply filters: click this to update the list. Nothing changes until you do.
2. Review the results
Each row is one conversation, shown with its summary, start time and grade.
- Summary: a short description of the conversation (e.g. "Heating not working"). If one couldn't be generated, you'll see the interaction number instead.
- Started: the date and time the conversation began.
- Grade: Good, Warning or Needs Attention.
Click the arrow on the right of any row to open its full evaluation. You can also click on the Summary text to open the interaction in the interaction logs.
3. Check the Evaluation
Expanding a row shows two things:
- Summary: a plain-English recap of what happened in the conversation.
- Grades breakdown: the individual checks behind the grade, each with its own score (for example, "Tools used correctly: 2/3" or "Customer issue resolved: 0/1"). The overall grade is a summary of these checks, so the breakdown tells you exactly why a conversation was graded the way it was.
🔄 Evaluating more calls
Use the Evaluate More Interactions button in the bottom of the page to run evaluations on additional interactions.
Evaluating more interactions can take a moment to process. Once it's done, the new results appear in the list and you can filter them like any other.
⚙️ Adjusting the sampling rate
By default, every conversation is evaluated. If you'd rather evaluate only a portion of them, you can lower the sampling rate.
Changing the sampling rate
Go to Settings, then Company Configs, then Supervisor Controls, and find LLM Chatbot Evaluation Sampling Rate.
- Enter the percentage of interactions to evaluate, from 0 to 100.
- Click Submit to save.
A rate of 100 evaluates every conversation, and a lower rate evaluates fewer.
💡 Getting the most out of Evaluations
Evaluations is most useful when you act on what it shows you, not just read it.
Top tips:
- Start with the reds. Filter to Needs Attention first, then Warning. That's where your time is best spent.
- Look for patterns. If several conversations fail in the same way (a topic the bot can't answer, a link it keeps getting wrong), that's a prompt or Tools problem, not a one-off.
- Feed fixes back in. When you spot a recurring issue, update your Main Prompt or Tools to address it, then check the next batch of evaluations to see if the grades improve.
- Check it regularly. A quick review on a schedule catches issues before your customers do.
❓ Frequently Asked Questions
Where do I find Evaluations?
Open the Digital AI (LLM Chatbot) you want to review and click the Evaluations tab at the top, next to Prompt and Tools.
What do the three grades mean?
Good means the conversation was handled well. Warning means there's something minor worth a look. Needs Attention means the bot likely got something wrong and the conversation is worth reviewing.
Does running Evaluations affect my live chatbot?
No. Evaluations only reviews conversations that have already finished. It doesn't change your chatbot's behaviour, and customers never see it.
What should I do about a conversation graded "Needs Attention"?
Open the conversation to see what happened, work out what the bot got wrong, and update your Main Prompt or Tools to fix it. Then keep an eye on future evaluations to check the change worked.