Discover the mobile banking evaluation system by Markswebb

Is 1,000 evaluation criteria for user experience too many or too few? For a complex service like a modern mobile bank, we believe it’s just right. Mobile banks have long surpassed basic payment functions, offering hundreds of scenarios.

Each year since 2012, we’ve developed new ways to systematize and evaluate mobile banks in countries like Kazakhstan (Mobile Banking Rank Kazakhstan 2024), Uzbekistan (Mobile Banking Rank Uzbekistan 2024), and Azerbaijan (Mobile Banking Rank Azerbaijan 2024). This year, we adapted our approach to the UAE market (Mobile Banking Review UAE 2024).

Let’s take a closer look at this dynamic system: why our 2024 evaluation is structured the way it is, how we ensure objectivity, and how the Markswebb Mobile Banking Rank drives progress in the market.

The essence of mobile banking evaluation system

Our research focuses on the customer experience, which is shaped by the breadth and convenience of task resolution as well as the variety of available solutions within a mobile bank.

Mobile banks handle hundreds of diverse user tasks, so in 2020, we divided our evaluation system into multiple components. The foundation - serving individuals and the banking business - includes daily money management (payments, transfers, balance tracking, transaction history, fee transparency, and support) and financial service management. Effectively, mobile banking has become a digital replacement for branches, enabling customers to open or close products and access services.

Evaluating customer experience is challenging to quantify. You can measure task completion speed or app ratings in stores, but those are subjective metrics that don’t provide actionable insights. We developed our own way to combat subjectivity: we break down each task available to mobile bank users into binary criteria, which comprehensively reflect both the functional depth and ease of use within the mobile bank interface.

Our evaluation encompasses nearly all client tasks related to managing debit and credit cards, accounts, deposits, as well as credit and investment products. It includes banking products and services available in both authenticated and unauthenticated areas of the app.

Here is an illustrative structure using the Mobile Banking Rank Kazakhstan as an example:

Mobile Banking Rank Kazakhstan

Mobile Banking Rank: a comparative research

The Mobile Banking Rank is a comparative research. Researchers follow identical scenarios across all mobile banking apps, evaluating the same criteria and recording outcomes in detailed checklists. Structurally, these checklists are tables with an extensive list of criteria grouped by tasks and subtasks.

In addition to expert evaluation, the checklist incorporates feedback from real users, gathered during moderated usability tests. Respondents test banking apps by completing frequent user tasks, helping identify weaknesses. Participants follow predefined scenarios typical of real-life usage, while researchers observe their actions, documenting challenges, task success, ease of use, and time spent.

The result is a matrix with hundreds of rows and columns, reflecting the completeness and convenience of various digital services. While this dataset is the study’s core artifact, its volume can make it difficult for an untrained audience to derive insights.

To ensure fairness in the rating process, we establish a "freeze date" for each study, after which service updates are not considered. For Mobile Banking Rank 2024, the freeze date was November 5, meaning that on this day, all checklists were finalized and no further changes were made.

Once data collection is complete, researchers analyze the findings. They outline the strengths and weaknesses of individual apps, create comparative tables and implementation maps, and identify features that could become competitive advantages in the coming year, those that have become standard for users, and the quality benchmarks customers expect. At this stage, the team also identifies best practices and common service flaws—solutions that have proven effective or, conversely, risks that could lead users to abandon the service.

For Mobile Banking Rank 2024, the system includes over 1,000 binary criteria. This year, we added expert interviews with product teams from mobile banks to complement the checklist evaluation and UX testing.

Mobile Banking Rank: a comparative research methods

The system enables an objective evaluation of user experience quality in mobile banking, providing a data-driven comparison of banks rather than relying on subjective opinions. The data we collect has direct practical value, empowering decision-making based on insights.

Key questions addressed in Mobile Banking Rank 2024

  • What changes have occurred in mobile banking from the end-user perspective over the past year?
  • How have the competitive positions of market participants evolved?
  • What priorities and focus areas are banks emphasizing in their mobile apps for private clients, and what drives these decisions?
  • What customer experience issues remain unresolved?
  • What new best practices have emerged in the market?
  • What opportunities exist for business growth through mobile banking?
Want to learn more about the capabilities of our evaluation system?

We can customize the system to fit your digital service. Discover how!

Get a quote

A representative selection of participants reflects the market

Each Markswebb study begins with a core participant list. These are banks selected based on our understanding of the market to include the most popular services shaping the public perception of customer experience quality.

For Mobile Banking Rank 2024, the core list includes major banks based on deposit and credit volumes for individuals, leaders in app downloads across popular stores, and the top five from the 2023 wave of rankings.

  • The best mobile banks from last year’s study serve as a source of advanced solutions.
  • The largest and most popular banks have sufficient resources to develop their apps and reach a broad audience, significantly influencing end-users.

Any bank or fintech company developing a mobile app for private clients can request to participate in the study if they are not already on the core list. Participation offers a valuable tool for competitive analysis. Apps are evaluated equally alongside market leaders across all scenarios for daily money management and digital office tasks. This participation does not provide any advantages in the final ranking, except the option to decline publication of results.

How customer experience is translated into numbers

Methodologically, the Mobile Banking Rank evaluation system belongs to the field of social sciences, specifically user research and behavioral economics. It also incorporates principles of information technology and human-computer interaction (HCI). Certain aspects of the evaluation system were developed in-house, tested, and refined through practical application.

The evaluation is based on weighting criteria that reflect the importance of user tasks and mobile banking functions. Each criterion has a specific weight, determined by the following factors:

  1. Task criticality. The more essential a function is for daily user tasks, the higher its weight.
  2. Audience reach. Tasks commonly encountered by the majority of users are given higher priority.
  3. Frequency of use. Functions used regularly are considered more significant.

All criteria are grouped into blocks of user tasks (e.g., transfers, payments, account management). Each block's score is calculated as the sum of its criteria multiplied by their respective weights. The final ranking is the result of a comprehensive analysis of all blocks, representing how closely the product aligns with the "ideal mobile bank" of the year.

In the diagram, you can see how the weights were distributed across blocks of user tasks in 2024 using the Daily Banking evaluation system as an example:

Markswebb's Daily Banking evaluation system

The evaluation checklist serves as the foundation for assessing customer experience. Each app is "broken down" into over 1,000 criteria, with the following ratings:

  • Yes — the functionality is fully implemented.
  • Partially — the functionality is partially implemented (e.g., not for all product types or only some data fields, such as partial card details). These ratings are always accompanied by researcher comments explaining the reason, which can provide valuable insights.
  • No — the functionality is absent.
  • Numerical — some criteria are evaluated with numeric values representing scores or sums.

The role of the "partially" rating

The "partially" rating is used for cases where functionality is inconsistently implemented. Examples include:

  • When evaluating the availability of tariff details for debit cards, one card might display the information while another does not. In such cases, we rate it "partially."
  • If service provider logos are displayed for some providers but not others, this also warrants a "partially" rating.

However, "partially" is not used if a criterion evaluates something essential, such as full account details, and the app provides only partial information. In such cases, we ask: "What task does the user aim to complete?" For example, a user might need full account details to share them for a transfer. If only partial details are available, the user cannot complete the task, and the rating is "no."

This nuanced approach ensures the evaluation captures the real-world utility of mobile banking functions and helps identify areas for improvement.

Updates to the evaluation system

Every Markswebb evaluation system evolves to keep pace with market fluctuations, technological advancements, and regulatory changes. Banking clients respond instantly to shifts, reflected in changing demands and priorities among financial services. As researchers, we not only keep up with these changes but aim to anticipate them, identifying subtle UX differences—what we call the "ghosts" of customer experience. These nuances collectively provide robust, data-driven insights into advantages and gaps in mobile banking.

For example, in the 2023 study, we asked banks to rank user task blocks by frequency. We calculated the average ranking for each block and adjusted their order accordingly. This method allowed banks to indicate which functions or areas gained or lost importance in terms of reach, usage frequency, and audience impact. Based on this consensus, we adjusted the weights of corresponding criteria to better align with market trends. In 2023, we also replaced approximately 20% of the checklist criteria to optimize focus and efficiency.

Updates to the evaluation system

What criteria are removed?

  1. Outdated criteria. Some criteria lose relevance due to changes in technology, economics, or user behavior.
    • Example: A criterion for displaying the cardholder's name during debit card ordering was removed. Personalized cards are becoming less common, and the associated workflows were primarily relevant for international transactions, which are now less accessible.
  2. Unverifiable criteria. Some functionalities exist but are impractical or impossible to evaluate.
    • Example: Scenarios involving account seizures are difficult to simulate or verify due to limited access to relevant respondents.
  3. Criteria with no market dynamics. If a feature is consistently implemented by only 1–2 banks over several years, it may not be relevant to most users.
    • Example: Scenarios related to tax deductions were found to be low-frequency and were removed.

What criteria are added?

New criteria arise from market observations, expert interviews, and user research. Here are a few examples:

  1. Automated transfers:We introduced a criterion evaluating whether transaction history data is auto-filled when transferring to a previously used account. This saves time and simplifies the process.
  2. Phone number scanning:A new criterion assesses the ability to scan a phone number using a smartphone camera. This feature was added to the "Transfers" block as it streamlines data entry.
  3. Auto-payment management:A criterion for managing automatic payments, such as setting up credit repayment schedules from a specific account, was introduced to prevent missed payments.

Where do ideas for changes come from?

We draw inspiration from several sources:

  • User research data: Interviews and testing sessions reveal what bank clients need most.
  • News and trends: We monitor updates in mobile apps, analyze new releases in app stores, and track features introduced by market leaders.
  • Bank feedback: Insights from consulting projects often lead to discussions and testing of new ideas.

By regularly updating the checklist, we ensure it reflects current user expectations and the needs of banking businesses.

Delivering actionable insights

The Mobile Banking Rank research requires nearly 3,000 research hours. We thoroughly analyze customer experiences to collect comprehensive data on task completeness and usability.

Mobile Banking Rank

These insights help product teams save resources on competitor analysis, accelerate feature rollouts, and make better-informed decisions. By adapting to industry trends and focusing on actionable data, our research drives innovation and competitiveness in mobile banking.

Looking for a partner?

Get in Touch

    Fields requiring an asterisk (*) are essential for submission. By submitting this form, you agree to our Terms and Conditions.

    Markswebb

    We respond to all messages as soon as possible.

    Become a client