The QIWI Wallet chatbot, Quick, is evolving into a fully-fledged customer support channel, handling inquiries across 70 user scenarios daily. Recently, the number and variety of chat requests have significantly increased, prompting the bot's team to assess how well the channel scales, processes new use cases, meets current communication standards, and how the user experience can be improved. To address these challenges, the QIWI product team reached out to Markswebb.
Contents
We conducted an audit of the chatbot, evaluated the capabilities of the automated service in terms of the number of tasks it handles and the quality of communication, formulated 18 growth directions, and proposed mechanisms to enhance the user experience and increase user engagement.
The audit of QIWI Quick led to significant improvements, enhancing the chatbot’s ability to deliver a more human-like experience. By implementing 26 specific tasks, the team focused on increasing response effectiveness and customer loyalty. Key improvements included better intent recognition and extending response timeouts, reducing repeated inquiries. These changes created a more natural and empathetic communication environment, adhering to conversational user interface principles and setting proper expectations for users. This project has laid a crucial foundation for QIWI Quick’s development, positioning it as a top-performing chatbot in the fintech market.
To assess the functionality and communication quality, we used a modified Chatbot Rank 2022 methodology. The benchmark of the study focuses on the principles of effective communication in a conversational user interface (CUI). It considers the chatbot's verbal behavior as the most comfortable speech actions for the user and provides an objective evaluation of the interaction, regardless of the digital service's domain.
It was also important to evaluate the chatbot's functionality - whether it could solve specific client tasks. For this, the initial list of requests (intents) from Chatbot Rank 2022 was almost doubled by adding inquiries typical for user scenarios in a digital wallet, totaling 96 intents. Researchers added four more requests related to product management, bringing the number of tested requests to 100.
The intents within the audit scope covered six blocks of user tasks: getting help with navigation, receiving general informational responses, getting responses related to personal client data, performing an action, submitting a complaint, and selecting a product.
As a result of the project, a metric was obtained that objectively evaluates two important qualities of QIWI Quick: how fully and empathetically it interacts with people (based on the principles and rules of CUI) and whether it can accomplish the assigned tasks (the number of resolved intents). Both metrics indicate the current level of the bot's development and allow the product team to set a target indicator and track its achievement.
The chatbot was studied in a desktop setting. Researchers first created wallets in QIWI, topped up the balance, and made various payments, based on target scenarios. This helped create a realistic working environment: transaction history, the necessary set of services, and data for the chatbot to operate.
Then they began testing the chatbot on intents, simulating the phrasings used by specific service clients. The researchers were aided by examples of QIWI user personas provided in advance by the product team. These personas allowed for a more accurate representation of client profiles, the products they have connected, and how they might behave in a text channel.
The study found that Quick adheres to most principles of constructing a conversational user interface (CUI). It exceeds the average fintech market level of CUI compliance, creating a comfortable communication environment in the text channel.
Quick adheres to the principle of politeness: it sends a message when the client's request is completed and ensures the client received exactly what they needed. It also creates transparent communication, for example, by informing the client in advance that the conversation is with a bot and setting the right expectations.
Growth points were identified in the principles of functionality and convenience: for example, the user could not return to previous steps of the conversation or resume a postponed dialogue. Sometimes, the bot gave the same response within different scenarios.
In processing intents, QIWI Quick showed even higher results. The bot handles 58 out of 100 requests perfectly: it fully manages tasks related to the most critical issues - direct wallet operations, providing information about conditions and products. It handles complaints and claims at a high level, helping clients resolve payment issues. In other intents, Quick clarifies data and prepares the user for a conversation with a support operator, which also creates a high-quality client experience.
Researchers described 18 growth areas for QIWI, each with 2 to 5 UX improvement recommendations. Solutions were selected from chatbot practices in fintech applications and e-commerce - more than 40 implementations in total, to match each CUI principle.
To illustrate the recommendations for the QIWI team, let's consider the principle "Adapt to client requests." This principle describes rules for creating a natural communication environment where the client does not need to adjust their requests to the channel's features.
Ideally, the bot should quickly recognize the type of request and respond accordingly: provide information, suggest an action, or perform it itself.
Adaptation to client requests is not only about the bot adequately understanding requests or texts with errors but also about the convenience of managing the conversation, having quick replies, or the ability to return to a specific step.
Therefore, we recommended several practices to the Quick team to facilitate dialogue navigation.
The same effect can be achieved by keeping quick replies always available and active in the chat log, so the dialogue can be rewound to the desired stage and another option selected. This eases the conversation and creates a sense of freedom in the text channel.
An important factor in forming a positive user experience is the ability to return to the dialogue even after a long break. To achieve this, the conversation timeout interval should be extended from 20 minutes to several hours, saving the client's efforts when returning to the chat.
The results of the study, evaluation, and recommendations for development were presented to the QIWI Quick team at the end of 2022. The main outcome of the project for the client was the ability to objectively assess the quality of the text channel compared to the entire digital market, and to understand which directions need investment to continuously raise the service level.
Audit from Markswebb
"The audit from Markswebb gave us a better understanding of the product and customer attitudes towards it. Some results were predictable (e.g., navigation in the bot), while others truly opened our eyes; for instance, we began considering how to handle negative customer feedback. Overall, we were pleasantly surprised that Quick is at the top of the ranking—we didn't have a realistic understanding of the bot's market position before this. The study proved very useful for evaluating the work already done and understanding where to go next."
Pavel Smirnov,
QIWI Quick Product Expert
The backlog was filled with specific tasks, with 26 of them moving into implementation as early as the beginning of February. The first recommendations implemented were those that directly impacted Quick's response effectiveness (FCR, First Call Resolution metric) and qualities that build loyalty (CSI). Recognition was improved in intent classes where growth points were identified, and the response timeout was extended, reducing the number of repeated bot inquiries.
All this has become an important foundation for a new development phase for the chatbot team, and for QIWI Quick itself - a step towards even greater "humanity" in communication.
This case study demonstrates a strategic approach to enhancing chatbot performance in a digital wallet. By partnering with Markswebb, QIWI was able to identify key growth areas, implement best practices, and significantly improve user experience, setting a new standard for customer support in the digital wallet industry.
Every year we conduct up to 15 studies of digital services. These are industry benchmarks that reflect the state of the market and trends.
