How to tell if your AI replies are working
It is easy to feel that an AI agent is helping, and harder to know. You do not need a dashboard or a data person to find out. A handful of numbers, counted by hand once a week in a simple table, will tell you whether customers are getting faster and better answers, whether your team has more time, and whether anything is going wrong. This guide explains which numbers to track, how to count each one, what good and worrying look like, and how to compare a week before the agent with a week after.
Start with a week before the agent
Numbers mean more when you have something to compare them with. If you have not connected your AI agent yet, count one ordinary week of messages first. If you already have, scroll back through your WhatsApp, Instagram and inbox to a typical week before you started and count what you can.
Avoid comparing a quiet week with a busy one. A week in the run-up to a holiday, or a week with a sale on, will make any change look bigger or smaller than it is. If your business is seasonal, compare with the same few weeks last year where you can.
The six numbers worth counting
| Number | What it tells you | How to count it |
|---|---|---|
| Conversations started by customers | How busy the week was | Ask your agent how many new conversations came in, by channel |
| Time to first reply | Whether customers wait less | Check ten chats: time of the customer's first message and time of the first answer |
| Chats handed to a person | How much still needs you | Ask your agent to list conversations handed over this week |
| Replies you had to correct | Whether the answers are right | Count from your weekly review |
| Customers who asked for a person | Whether people trust the replies | Ask your agent to find chats where someone asked for a person |
| Bookings, orders or quotes from chats | Whether chats turn into business | Count from your diary, till or order book, marked "from a message" |
That is all. More numbers feel thorough but rarely change what you do. If one of these six moves the wrong way, you will know where to look.
How to count each one without software
Spaces can list your conversations with filters such as unread, unanswered, handed over or a tag, and it can export how many messages each account sent each day and how many new conversations started, as a spreadsheet, with no message text. Your AI agent can fetch both for you. The rest comes from your own records.
- Conversations: ask your agent "how many conversations did customers start this week, on each channel?" Write the totals down.
- Time to first reply: ask the agent to show you ten chats from different days and times, including evenings. Note the gap between the first customer message and the first answer in each, then take the middle value rather than the average, so one very slow chat does not skew it.
- Hand-overs: ask for the list of conversations handed to a person this week and count them.
- Corrections: during your weekly review, make a tally mark each time you had to put a reply right with a customer.
- Asked for a person: ask the agent to find chats where the customer asked to speak to someone. Count them.
- Bookings and orders: add a small "M" next to anything in your diary or order book that came from a message. Count the Ms at the end of the week.
If you also tag conversations, such as booking, order or quote, counting gets easier still. Your agent can list chats with a given tag, and our explainer on conversation tags shows how owners use them.
A weekly tracking table to copy
Before Week 1 Week 2 Week 3 Week 4
Customer conversations ____ ____ ____ ____ ____
Middle time to 1st reply ____ ____ ____ ____ ____
Handed to a person ____ ____ ____ ____ ____
Replies corrected ____ ____ ____ ____ ____
Asked for a person ____ ____ ____ ____ ____
Bookings/orders from chats ____ ____ ____ ____ ____
Notes (busy week? sale? closed a day?): ________________________Fill one column a week. A paper copy pinned by the till works as well as a spreadsheet. The notes line matters: it stops you blaming the agent for a quiet week that was really a bank holiday.
What good and worrying look like
| Number | A good sign | Worth a closer look |
|---|---|---|
| Time to first reply | Falls sharply, especially evenings and weekends | Stays the same, which often means a chat app is not being asked to check often enough |
| Handed to a person | Settles at a steady share of chats | Rises week on week, or nearly every chat is handed over |
| Replies corrected | Falls towards zero over a month | Stays the same, which means fixes are not reaching the brief |
| Asked for a person | Low and steady | Rising, which can mean replies sound robotic or miss the point |
| Bookings from chats | Holds or rises with the same number of conversations | Falls while conversations hold steady |
There is no single right number for hand-overs. A clinic will hand over more than a café, because more of its questions need a professional. What matters is the trend in your own business, not a figure from somewhere else.
A worked example
Here is how a small nail salon's first month might look. Names and details are examples.
| Number | Before | Week 1 | Week 2 | Week 4 |
|---|---|---|---|---|
| Customer conversations | 64 | 70 | 66 | 68 |
| Middle time to 1st reply | 3 hours | 5 minutes | 4 minutes | 4 minutes |
| Handed to a person | n/a | 19 | 12 | 9 |
| Replies corrected | n/a | 6 | 3 | 1 |
| Asked for a person | n/a | 5 | 3 | 2 |
| Bookings from chats | 21 | 24 | 25 | 27 |
The owner read it like this. Replies became much faster from the first week. Hand-overs and corrections fell as the brief improved, which is the pattern you want. Bookings rose a little while conversations stayed about the same, which suggests faster answers were catching people who used to book elsewhere. Week 1 had the most hand-overs because the brief was missing the gel removal price, which the weekly review found and fixed.
Numbers that mislead
- Messages sent. A higher number can mean the agent is chatty, not helpful. Count conversations and outcomes instead.
- Average reply time. One chat that waited overnight can make a good week look bad. The middle value of ten chats is fairer.
- Zero hand-overs. It can look like success, but it may mean the agent is answering things it should pass on. Read a few complaints to check.
- A single week. Look at four weeks before deciding anything, unless something has clearly gone wrong.
Asking customers directly
Numbers tell you what happened, not how it felt. Now and then, ask a few regulars in person: "Did you get a quick answer when you messaged us?" If they say the replies felt odd or slow, take it seriously, even if your table looks fine.
Avoid sending a survey to everyone who messaged you. It adds messages people did not ask for, and a short word at the counter usually tells you more.
What to do with the results
| If you see | Try this |
|---|---|
| Slow replies with a chat app | Ask it to check messages more often, or consider an always-on agent |
| Many hand-overs on simple questions | Add those answers to the FAQs in your business brief |
| Corrections that are not falling | Check you are updating the brief, not just the customer |
| More people asking for a person | Read five of those chats and adjust the tone with your own example replies |
| Fewer bookings with the same conversations | Check the agent asks for the booking clearly at the end of each answer |
Summing up
Count one ordinary week before, then six simple numbers every week after, in one table. Look for faster replies, falling corrections, steady hand-overs and bookings that hold or rise. If the numbers and your customers both say it is working, it is working. If not, the table shows you which part of the brief to fix next.
Pricing
FAQ
Do I need special software to measure this?
No. A paper table or a simple spreadsheet is enough. Your AI agent can list conversations and hand-overs through Spaces, and the rest comes from your diary, till or order book.
Can Spaces show how many messages went out each day?
Yes. It can export usage by day for each account, as a spreadsheet, with messages sent and new conversations and no message text. Ask your agent to fetch it for the week you are counting.
How long before I can tell if it is working?
Reply speed usually changes in the first week. For corrections, hand-overs and bookings, give it about four weeks, because the first week includes fixing gaps in your brief.
What is a good number of hand-overs?
There is no single right figure, because it depends on your trade. Watch the trend instead: hand-overs on simple questions should fall as you add answers to your brief.
How do I know a booking came from a message?
Mark it when you write it down, with a letter or a tag in your booking system. Tagging the conversation as a booking also lets your agent count them for you later.
Should I measure each channel separately?
Count conversations by channel, because it shows you where customers write most. The other numbers can usually be combined unless one channel clearly behaves differently.
What if the numbers look good but customers complain?
Trust the customers and read their chats. Numbers miss things like a tone that feels cold, and a few changes to your example replies often fix that.