Is 1,000 evaluation criteria for user experience too many or too few? For a complex service like a modern mobile bank, we believe it’s just right. Mobile banks have long surpassed basic payment functions, offering hundreds of scenarios.
Each year since 2012, we’ve developed new ways to systematize and evaluate mobile banks in countries like Kazakhstan (Mobile Banking Rank Kazakhstan 2024), Uzbekistan (Mobile Banking Rank Uzbekistan 2024), and Azerbaijan (Mobile Banking Rank Azerbaijan 2024). This year, we adapted our approach to the UAE market (Mobile Banking Review UAE 2024).
Let’s take a closer look at this dynamic system: why our 2024 evaluation is structured the way it is, how we ensure objectivity, and how the Markswebb Mobile Banking Rank drives progress in the market.
Contents
Our research focuses on the customer experience, which is shaped by the breadth and convenience of task resolution as well as the variety of available solutions within a mobile bank.
Mobile banks handle hundreds of diverse user tasks, so in 2020, we divided our evaluation system into multiple components. The foundation - serving individuals and the banking business - includes daily money management (payments, transfers, balance tracking, transaction history, fee transparency, and support) and financial service management. Effectively, mobile banking has become a digital replacement for branches, enabling customers to open or close products and access services.
Evaluating customer experience is challenging to quantify. You can measure task completion speed or app ratings in stores, but those are subjective metrics that don’t provide actionable insights. We developed our own way to combat subjectivity: we break down each task available to mobile bank users into binary criteria, which comprehensively reflect both the functional depth and ease of use within the mobile bank interface.
Our evaluation encompasses nearly all client tasks related to managing debit and credit cards, accounts, deposits, as well as credit and investment products. It includes banking products and services available in both authenticated and unauthenticated areas of the app.
Here is an illustrative structure using the Mobile Banking Rank Kazakhstan as an example:
The Mobile Banking Rank is a comparative research. Researchers follow identical scenarios across all mobile banking apps, evaluating the same criteria and recording outcomes in detailed checklists. Structurally, these checklists are tables with an extensive list of criteria grouped by tasks and subtasks.
In addition to expert evaluation, the checklist incorporates feedback from real users, gathered during moderated usability tests. Respondents test banking apps by completing frequent user tasks, helping identify weaknesses. Participants follow predefined scenarios typical of real-life usage, while researchers observe their actions, documenting challenges, task success, ease of use, and time spent.
The result is a matrix with hundreds of rows and columns, reflecting the completeness and convenience of various digital services. While this dataset is the study’s core artifact, its volume can make it difficult for an untrained audience to derive insights.
To ensure fairness in the rating process, we establish a "freeze date" for each study, after which service updates are not considered. For Mobile Banking Rank 2024, the freeze date was November 5, meaning that on this day, all checklists were finalized and no further changes were made.
Once data collection is complete, researchers analyze the findings. They outline the strengths and weaknesses of individual apps, create comparative tables and implementation maps, and identify features that could become competitive advantages in the coming year, those that have become standard for users, and the quality benchmarks customers expect. At this stage, the team also identifies best practices and common service flaws—solutions that have proven effective or, conversely, risks that could lead users to abandon the service.
For Mobile Banking Rank 2024, the system includes over 1,000 binary criteria. This year, we added expert interviews with product teams from mobile banks to complement the checklist evaluation and UX testing.
The system enables an objective evaluation of user experience quality in mobile banking, providing a data-driven comparison of banks rather than relying on subjective opinions. The data we collect has direct practical value, empowering decision-making based on insights.
We can customize the system to fit your digital service. Discover how!
Each Markswebb study begins with a core participant list. These are banks selected based on our understanding of the market to include the most popular services shaping the public perception of customer experience quality.
For Mobile Banking Rank 2024, the core list includes major banks based on deposit and credit volumes for individuals, leaders in app downloads across popular stores, and the top five from the 2023 wave of rankings.
Any bank or fintech company developing a mobile app for private clients can request to participate in the study if they are not already on the core list. Participation offers a valuable tool for competitive analysis. Apps are evaluated equally alongside market leaders across all scenarios for daily money management and digital office tasks. This participation does not provide any advantages in the final ranking, except the option to decline publication of results.
Methodologically, the Mobile Banking Rank evaluation system belongs to the field of social sciences, specifically user research and behavioral economics. It also incorporates principles of information technology and human-computer interaction (HCI). Certain aspects of the evaluation system were developed in-house, tested, and refined through practical application.
The evaluation is based on weighting criteria that reflect the importance of user tasks and mobile banking functions. Each criterion has a specific weight, determined by the following factors:
All criteria are grouped into blocks of user tasks (e.g., transfers, payments, account management). Each block's score is calculated as the sum of its criteria multiplied by their respective weights. The final ranking is the result of a comprehensive analysis of all blocks, representing how closely the product aligns with the "ideal mobile bank" of the year.
In the diagram, you can see how the weights were distributed across blocks of user tasks in 2024 using the Daily Banking evaluation system as an example:
The evaluation checklist serves as the foundation for assessing customer experience. Each app is "broken down" into over 1,000 criteria, with the following ratings:
The "partially" rating is used for cases where functionality is inconsistently implemented. Examples include:
However, "partially" is not used if a criterion evaluates something essential, such as full account details, and the app provides only partial information. In such cases, we ask: "What task does the user aim to complete?" For example, a user might need full account details to share them for a transfer. If only partial details are available, the user cannot complete the task, and the rating is "no."
This nuanced approach ensures the evaluation captures the real-world utility of mobile banking functions and helps identify areas for improvement.
Every Markswebb evaluation system evolves to keep pace with market fluctuations, technological advancements, and regulatory changes. Banking clients respond instantly to shifts, reflected in changing demands and priorities among financial services. As researchers, we not only keep up with these changes but aim to anticipate them, identifying subtle UX differences—what we call the "ghosts" of customer experience. These nuances collectively provide robust, data-driven insights into advantages and gaps in mobile banking.
For example, in the 2023 study, we asked banks to rank user task blocks by frequency. We calculated the average ranking for each block and adjusted their order accordingly. This method allowed banks to indicate which functions or areas gained or lost importance in terms of reach, usage frequency, and audience impact. Based on this consensus, we adjusted the weights of corresponding criteria to better align with market trends. In 2023, we also replaced approximately 20% of the checklist criteria to optimize focus and efficiency.
New criteria arise from market observations, expert interviews, and user research. Here are a few examples:
We draw inspiration from several sources:
By regularly updating the checklist, we ensure it reflects current user expectations and the needs of banking businesses.
The Mobile Banking Rank research requires nearly 3,000 research hours. We thoroughly analyze customer experiences to collect comprehensive data on task completeness and usability.
These insights help product teams save resources on competitor analysis, accelerate feature rollouts, and make better-informed decisions. By adapting to industry trends and focusing on actionable data, our research drives innovation and competitiveness in mobile banking.
We respond to all messages as soon as possible.
We’ve evolved dozens of successful financial services and are eager to prove that our expertise can be implemented in other industries and around the world. Have a look at our success stories!