Disclosure: This post was created as part of a paid partnership with Fonio.ai. For the practical test, TechPanther configured its own AI phone assistant and evaluated it using 15 different test scenarios. TechPanther independently conducted, evaluated, and presented the test results. The results mentioned in the article refer exclusively to the tested configuration and the tests performed.
How reliably can an AI phone assistant handle real customer conversations? I explored exactly this question as part of an in-depth practical test with Fonio.ai.
Fonio offers an AI-based phone assistant that can answer incoming calls and process or log the content of conversations. The assistant can be customized. Among other things, you can specify the assistant’s name, company, role, and various behavioral rules.
For my test, I set up my own AI customer service system for TechPanther. I provided the assistant with information about my blog, its topics, and the desired workflows.
I then called the AI phone assistant in a total of 15 different test scenarios. These included not only simple, standard questions but also deliberately difficult and, in some cases, unusual conversation scenarios.
The Practical Test: 15 Test Scenarios
To test the features as thoroughly as possible, I ran a total of 15 scenarios.
1. General Customer Inquiry
First, I simulated a classic customer inquiry. The AI was supposed to explain what TechPanther does and what topics are covered on the blog.
Result: 2/2 points
2. Question about information not stored in the knowledge base
Next, I asked for information that wasn’t clearly documented in the knowledge base. Specifically, I asked how much a review on TechPanther costs.
The purpose was to test how the assistant handles information for which it cannot provide a clear answer.
Result: 0/2 points
3. Request for a callback
In this scenario, a callback was requested. The AI was supposed to record the necessary information—such as name and phone number—and then summarize it.
Result: 2/2 points
4. Vague Request
For this test, a deliberately vague statement was used: “I’m calling about that thing from last week.”
The AI should recognize that the request isn’t specific enough and determine what it’s about by asking appropriate follow-up questions.
Result: 2/2 points
5. Interruption During the Conversation
During an ongoing conversation, the AI was deliberately interrupted, and a new question was then asked.
The purpose was to test how well the assistant handles interruptions and changes in the flow of the conversation.
Result: 2/2 points
6. Multiple inquiries at once
In another conversation, several questions were combined. These included, among other things, information about TechPanther, test reports, and contact options.
The AI was expected to recognize the various requests and address them accordingly.
Result: 2/2 points
7. Incorrect or Missing Information
Here, a false claim was deliberately made. For example, it was claimed that TechPanther had already published an article about the iPhone 18.
The assistant was not supposed to simply repeat the statement but rather verify whether such information actually existed.
Result: 2/2 points
8. Record and repeat a phone number
The AI was given a phone number and was supposed to record it correctly and then repeat it for verification.
This test is particularly relevant for callback or contact requests, since even a single error in a phone number can prevent a callback from being made.
Result: 2/2 points
9. Record and repeat an email address
An email address was also mentioned during the conversation. The AI was then supposed to repeat the address to verify it.
Result: 0/2 points
This revealed a limitation in the AI’s ability to correctly process the provided email address.
10. Fast or difficult speech input
For this test, a longer request was spoken as quickly as possible with few pauses.
The aim was to assess how reliably the speech recognition and subsequent comprehension function even with more challenging speech input.
Result: 1/2 points
11. Language switch
The assistant was initially addressed in English. Later in the conversation, the language was switched back to German.
The goal was to test how the AI phone assistant handles a language switch within the same conversation.
Result: 1/2 points
12. Canceling a Callback
After initially requesting a callback, this request was subsequently withdrawn during the course of the conversation.
The assistant was supposed to recognize the change in request and process it accordingly.
Result: 2/2 points
13. Complaint
A typical complaint scenario was also simulated. This involved portraying a situation of dissatisfaction or disappointment.
The test assessed whether the assistant responded appropriately to the complaint and used follow-up questions to determine the specific nature of the problem.
Result: 2/2 points
14. Complex Conversation Involving Multiple Pieces of Information
One particularly extensive scenario involved combining several different pieces of information.
Among other things, it involved a product or test report, contact options, contact with the team, an email address, and a request for a callback.
The AI was expected to process the various pieces of information during the conversation and then summarize them.
Result: 2/2 points
15. Unknown Information and Knowledge Base
Finally, a question was asked for which no relevant information was available in the knowledge base.
The goal here was to test whether the assistant handles gaps in knowledge correctly and refrains from providing a made-up answer when no suitable information can be found.
Result: 2/2 points
Scoring System
I used a simple point system for the evaluation. Each test situation could earn between 0 and 2 points:
- 2 points: The situation was solved completely and correctly.
- 1 point: The situation was partially resolved or minor limitations occurred.
- 0 points: The situation was not resolved satisfactorily.
In addition, I repeated individual scenarios in separate conversations. This allowed me to check how the assistant behaves when there is no prior conversational context.
Test result: 26 out of 30 points
Out of a possible 30 points, the AI phone assistant scored 26 points in my practical test.
This corresponds to a score of 86.7 percent.
| General Customer Inquiry The AI was supposed to explain what TechPanther does and what topics the blog covers. | 2 |
| Question about information not stored in the system I asked how much a review on TechPanther costs. This was intended to test how the AI handles information that isn’t explicitly stored in the system. | 0 |
| Request for a Callback A callback was requested. The AI was supposed to record the necessary information, such as name and phone number, and then summarize it. Unclear Request A deliberately vague statement such as “I’m calling about that thing from last week” was supposed to be interpreted appropriately by the AI and clarified through follow-up questions. | 2 2 |
| Interruption: Fast or Difficult Speech Input A longer request was spoken as quickly as possible and without long pauses to test speech recognition and comprehension. During the conversation, the AI was interrupted and presented with a new question. | 2 |
| Multiple Concerns at Once: During a single conversation, several questions were linked together—for example, regarding test reports, the content of TechPanther, and contact options. | 2 |
| Incorrect or Missing Information It was claimed that TechPanther had already published an article about the iPhone 18. The AI was supposed to verify this information rather than simply confirming it. | 2 |
| Record and repeat a phone number The AI was given a phone number and was supposed to record it correctly and then repeat it. | 2 |
| Recording and repeating an email address An email address was also provided and subsequently repeated by the AI for verification. | 0 |
| Fast or difficult speech input A longer request was spoken as quickly as possible and without long pauses to test speech recognition and comprehension. | 1 |
| Language switch The AI was initially addressed in English. The conversation then switched back to German to test how it handles a language change within a conversation. | 1 |
| Canceling a callback request After a callback was requested, the request was withdrawn later in the conversation. The AI was expected to correctly process the change in the request. | 2 |
| Complaint A situation involving dissatisfaction or disappointment was simulated. This tested how the AI reacts to a complaint and whether it asks appropriate questions about the problem. | 2 |
| Complex Conversation with Multiple Pieces of Information A conversation combined several topics, including a product or test report, contact options, how to reach the team, an email address, and a request for a callback. The AI should have captured and summarized the various pieces of information. | 2 |
| Unknown Information / Knowledge Base Finally, a question was asked for which no relevant information was available in the knowledge base. This tested whether the AI actually searches for the information and, if no results are found, refrains from providing a made-up answer. | 2 |
| Total | 26/30 |
Where the Assistant Performed Especially Well
In my test, the AI phone assistant performed particularly well in typical conversation scenarios.
These included, for example, general customer inquiries, requests for callbacks, unclear issues, and conversations covering multiple topics simultaneously.
Recording and then repeating a phone number also worked reliably in the test.
It was also interesting to see how the assistant handled deliberately incorrect or non-existent information. These tests showed that the assistant didn’t simply have to accept every statement made by the caller.
Where there were limitations
At the same time, the practical test also revealed some limitations.
The test scenario could not be fully resolved when processing the email address. Limitations also arose when speech input was particularly fast.
Switching between English and German within a conversation was also only partially successful.
When asked for information that wasn’t stored in the system, the situation was also not resolved satisfactorily.
In my view, these tests are particularly interesting because, in real-world use, an AI phone assistant isn’t exclusively confronted with perfectly formulated standard questions.
Conclusion
With 26 out of a possible 30 points—or 86.7 percent—the Fonio AI phone assistant I configured demonstrated a high overall success rate in the practical test.
Particularly with regard to typical customer inquiries, callback requests, unclear issues, and more complex conversation scenarios, the assistant was able to handle the tasks presented in the test almost entirely on its own.
At the same time, the results show that even an AI phone assistant does not handle every call situation with the same level of reliability. In my test, difficulties arose in particular when dealing with information not stored in the system, processing an email address, handling fast speech input, and when the language changed.
It’s important to put these results in context: The evaluation is based exclusively on my self-configured Fonio test environment and the 15 test scenarios I conducted. It therefore does not represent a universal assessment of all Fonio.ai features.
For me, it was particularly interesting to see how the assistant performs outside of simple, standard situations. It was precisely here that the 15 different test scenarios revealed where its strengths lie and where limitations might still arise in practical use.
Leave a Reply