How to measure the ROI of UX
The general public probably considers 'design' to be simply the surface features of an object, the 'skin' which covers the working parts. Perhaps nice to have, rather than essential. Yet we are all using designed objects, and nowhere more so than in the software and apps which underpin our lives. Anyone who logs into their bank or financial service is utilizing a highly sophisticated design that must function cleanly and with the greatest possible ease of use. In the last decade, we have all learned that negotiating with apps does not have to be complicated. Big Tech in the form of Google, Facebook, or Amazon has increasingly demonstrated to the financial markets that there is a huge public appetite for great, well-functioning software.
Value for Money
It's easier to spot when the User Experience – UX – is not good. If there are glitches and problems, we notice them. If a choice is available, we'll dump the bad UX experience and move on to something more seamless and pleasing. If the UX and User Interface – UI – is good, then in a way, we don't see it. This is an issue for UX designers and their clients because if the definition of “perfect” is something no one notices, how can you measure and evaluate that?
As is often the case, it's the bad news that attracts attention. There's no story involved here: “Plane takes off, has an uneventful flight, lands safely.” On the other hand, everyone pays attention if things go wrong. So from the client side and the design side, the UX aim is in some senses to always have an “uneventful journey”, but one which also delivers maximum effectiveness and sometimes even pleasure in using a service. These are significant demands that corporations and clients of UX sometimes struggle with: “You want us to pay a ton of money so that no-one notices?”
This is where Return On Investment comes into the picture – to assess the value for money delivered by the UX process. Few organizations today would argue that UX isn't necessary to at least some extent. However, evaluating the degree and depth of User Experience and User Interface might be a dark art. Or not: let's look at some factors in determining ROI for UX.
Looking to the end
Right at the start of a project, there needs to be a framework put in place that will enable assessment at the end. If we set out to design an airplane, it has to be defined as more than simply 'an object that flies'. The design process can only get underway when we've decided whether it will carry passengers, what distances it will cover, and the heights and speeds it should attain. After that, all the results will be referred back to those original criteria and the client gets what they wanted: a 200-seat passenger jet, rather than a single-engine, solo stunt plane.
So what kind of metrics should be put in place at the beginning of a UX design project to ensure at the end that all criteria are met? Forget the aircraft analogy now; we'll get straight down to business and start on a financial technology project.
Looking at 'Best in class'
First of all, benchmark: Find out comprehensively what is already 'out there' and offered by other banks and fintechs. How do 'best in class' financial products perform? What is their history? What is their level of uptake? What do users like (or hate) about other apps? Thorough benchmarking defines the project's scope by giving clear comparators for similar products to the new proposed offering. This research period is vital and should never be rushed through; it is often where a good UX agency can bring some 'wisdom' to the proceedings. Product managers in financial institutions might be keen to push ahead with their fantastic new idea. In contrast, an experienced agency can examine the proposal compared to other products. Eager startups are often keen to power forward with their “game-changing” app, so benchmarking is vital, both upfront and as an ongoing process.
UX metrics
Benchmarking produces UX metrics through surveys, analytics, usability testing and customer interviews – all of which are ways of getting to a Single Source of Truth that will create an outstanding product. With UX metrics established, these must be expressed as Key Performance Indicators, which are not always the same for every stakeholder. For example, a customer KPI might be spending as little time engaging with an app as possible, while the opposite could be valid for a sales department. And what does the bank or financial organization care about? These factors must be collated upfront and rigorously challenged where necessary. Organizations’ Mission Statements are notoriously “wooly” and all-encompassing, but by definition KPIs must be as precise and measurable as possible.
Surveys form the first basis for information gathering. These may include satisfaction ratings, ease-of-use ratings, and perceived usability, measured by interviewing as many potential users as possible (with the number of interviewees being agreed at the start of the project).
Tools such as NPS (Net Promoter Score), SUS (System Usability Scale), and SUPR-Q (Standardized User Experience Percentile Rank Questionnaire) are used to define 'personas' and accurately drill down to discover what users need and often find frustrating. For example, the Net Promoter Score method typically involves a single survey question, asking people to rate the likelihood of recommending a product or service to someone else. A low NPS may be an early predictor of customer loss, which signals a decrease in income. So using such techniques to assess UX-UI effectiveness are not only about increasing income but also about avoiding financial and reputational loss. ROI can therefore also be viewed in the light of how spending on UX-UI can help a company avoid losing income.
Analytics then measure developing solutions for factors that include frequency of return visits, conversion rates, possible customer churn, and error counts. They show how the final product will be used before its launch. Of course, not all products will be entirely new launches, and often an app is still being redesigned and updated. In such cases, there is already a wealth of user material to evaluate. However, proposed changes shouldn't simply come from the client – they must also engage with, and acknowledge, end users' feedback in a fully measurable way.
Quantitative Usability Testing is then used to assess the success rate of using the app, the time spent on the task, and the associated error rate. This is about precise measurement, rather than possibly more subjective feedback from users, and is designed to demonstrate the productivity and effectiveness of the program in achieving what it is being designed to do. For example, a bank onboarding program must work seamlessly and, for the user, quickly. The benchmarks already exist with finely-tuned fintech offerings and neobank onboarding, demonstrating that swift onboarding is possible. Potential customers know that the process does not have to take forever.
ROI tools
How can these metrics be calculated to determine the ROI of UX? There are three overlapping approaches: the Single Usability Metric, the Conversion Rate, and the Drop Off Rate (fewer drop-offs being a solid indication of fewer errors in the UX design).
SUM – Single Usability Metric – is a standardized usability metric measuring essential components of usability such as effectiveness, efficiency through task completion rates, error counts, task times, and task-time satisfaction. SUM calculations can be accomplished using an agreed algorithm that analyzes UX errors to optimize conversion.
The Conversion Rate is the ratio of total visitors who take the desired action, such as new customers onboarding. Onboarding would fall into the macro conversion category. At the same time, there is also a subset that might feature – for example – blog viewing, which then tends to feed forward to the macro conversion results.
It's unlikely that any app will ever achieve a 100% conversion rate, so measuring the Drop Off Rate is essential, too. How many potential site users are not following through from their initial contact? And why? Is it because the UX-UI is too tricky to navigate? Is it too complex, with too many options? 'Keep It Simple Stupid' (the KISS principle) is an old idea, but it has survived because it is true. Calculating the Drop Off rate is a vital part of the UX process because it tells the real story of the difference between how an app is being developed and how it may be used or not used.
Research and testing, then more testing
Classic design is pleasing to the senses, so great UX-UI design must likewise be just that. However, it goes much further than that. Some agencies have pioneered the importance of careful research and rigorous testing to create products that are as perfect as possible and truly ready for use. That means being aware of all the stakeholders in a project. With tremendous experience in UX-UI design, MNDWRK is an example of a breed of agencies that, while certainly delivering stylish and elegant solutions, also concentrate a vast amount of their firepower on measurement and analysis. Banks and fintechs are, after all, very interested in their Return On Investment, and the ROI of UX can be measured and assessed.
Napkin Math
So let's take a brief look at a typical calculation:
A bank has an internal interface used by 1,000 employees whose wages, taxes and insurance contributions, plus a share of the infrastructure costs and equipment, can be averaged out to €50 an hour. With each employee working approximately 2,000 hours a year, the workforce costs 1,000 x 2,000 x €50 = €100M a year.
However, if the bank replaces its software with something more efficient that costs – say – €10M (comprising €9M dev and business + €1m UX) and that €1M spent on UX results in software with a 10% speed increase, then ROI is achieved in 1 year.
Influencing and engaging
Having said all this about the measurement of the ROI of UX, of course, great design also serves to influence and engage. Just look at how Apple Computers won a loyal and dedicated audience for its products over many years. The design demonstrates a brand's message and expresses core values and beliefs, which go on to create differentiation from other brands within the same market. Both banks and fintechs operate in crowded and highly competitive fields, so they must ensure that they have great UX from every point of view – for effectiveness, functionality and outstanding ROI – so that the end user might say, “Wow! Will you take a look at that!”
How to measure the ROI of UX
The general public probably considers 'design' to be simply the surface features of an object, the 'skin' which covers the working parts. Perhaps nice to have, rather than essential. Yet we are all using designed objects, and nowhere more so than in the software and apps which underpin our lives. Anyone who logs into their bank or financial service is utilizing a highly sophisticated design that must function cleanly and with the greatest possible ease of use. In the last decade, we have all learned that negotiating with apps does not have to be complicated. Big Tech in the form of Google, Facebook, or Amazon has increasingly demonstrated to the financial markets that there is a huge public appetite for great, well-functioning software.
Value for Money
It's easier to spot when the User Experience – UX – is not good. If there are glitches and problems, we notice them. If a choice is available, we'll dump the bad UX experience and move on to something more seamless and pleasing. If the UX and User Interface – UI – is good, then in a way, we don't see it. This is an issue for UX designers and their clients because if the definition of “perfect” is something no one notices, how can you measure and evaluate that?
As is often the case, it's the bad news that attracts attention. There's no story involved here: “Plane takes off, has an uneventful flight, lands safely.” On the other hand, everyone pays attention if things go wrong. So from the client side and the design side, the UX aim is in some senses to always have an “uneventful journey”, but one which also delivers maximum effectiveness and sometimes even pleasure in using a service. These are significant demands that corporations and clients of UX sometimes struggle with: “You want us to pay a ton of money so that no-one notices?”
This is where Return On Investment comes into the picture – to assess the value for money delivered by the UX process. Few organizations today would argue that UX isn't necessary to at least some extent. However, evaluating the degree and depth of User Experience and User Interface might be a dark art. Or not: let's look at some factors in determining ROI for UX.
Looking to the end
Right at the start of a project, there needs to be a framework put in place that will enable assessment at the end. If we set out to design an airplane, it has to be defined as more than simply 'an object that flies'. The design process can only get underway when we've decided whether it will carry passengers, what distances it will cover, and the heights and speeds it should attain. After that, all the results will be referred back to those original criteria and the client gets what they wanted: a 200-seat passenger jet, rather than a single-engine, solo stunt plane.
So what kind of metrics should be put in place at the beginning of a UX design project to ensure at the end that all criteria are met? Forget the aircraft analogy now; we'll get straight down to business and start on a financial technology project.
Looking at 'Best in class'
First of all, benchmark: Find out comprehensively what is already 'out there' and offered by other banks and fintechs. How do 'best in class' financial products perform? What is their history? What is their level of uptake? What do users like (or hate) about other apps? Thorough benchmarking defines the project's scope by giving clear comparators for similar products to the new proposed offering. This research period is vital and should never be rushed through; it is often where a good UX agency can bring some 'wisdom' to the proceedings. Product managers in financial institutions might be keen to push ahead with their fantastic new idea. In contrast, an experienced agency can examine the proposal compared to other products. Eager startups are often keen to power forward with their “game-changing” app, so benchmarking is vital, both upfront and as an ongoing process.
UX metrics
Benchmarking produces UX metrics through surveys, analytics, usability testing and customer interviews – all of which are ways of getting to a Single Source of Truth that will create an outstanding product. With UX metrics established, these must be expressed as Key Performance Indicators, which are not always the same for every stakeholder. For example, a customer KPI might be spending as little time engaging with an app as possible, while the opposite could be valid for a sales department. And what does the bank or financial organization care about? These factors must be collated upfront and rigorously challenged where necessary. Organizations’ Mission Statements are notoriously “wooly” and all-encompassing, but by definition KPIs must be as precise and measurable as possible.
Surveys form the first basis for information gathering. These may include satisfaction ratings, ease-of-use ratings, and perceived usability, measured by interviewing as many potential users as possible (with the number of interviewees being agreed at the start of the project).
Tools such as NPS (Net Promoter Score), SUS (System Usability Scale), and SUPR-Q (Standardized User Experience Percentile Rank Questionnaire) are used to define 'personas' and accurately drill down to discover what users need and often find frustrating. For example, the Net Promoter Score method typically involves a single survey question, asking people to rate the likelihood of recommending a product or service to someone else. A low NPS may be an early predictor of customer loss, which signals a decrease in income. So using such techniques to assess UX-UI effectiveness are not only about increasing income but also about avoiding financial and reputational loss. ROI can therefore also be viewed in the light of how spending on UX-UI can help a company avoid losing income.
Analytics then measure developing solutions for factors that include frequency of return visits, conversion rates, possible customer churn, and error counts. They show how the final product will be used before its launch. Of course, not all products will be entirely new launches, and often an app is still being redesigned and updated. In such cases, there is already a wealth of user material to evaluate. However, proposed changes shouldn't simply come from the client – they must also engage with, and acknowledge, end users' feedback in a fully measurable way.
Quantitative Usability Testing is then used to assess the success rate of using the app, the time spent on the task, and the associated error rate. This is about precise measurement, rather than possibly more subjective feedback from users, and is designed to demonstrate the productivity and effectiveness of the program in achieving what it is being designed to do. For example, a bank onboarding program must work seamlessly and, for the user, quickly. The benchmarks already exist with finely-tuned fintech offerings and neobank onboarding, demonstrating that swift onboarding is possible. Potential customers know that the process does not have to take forever.
ROI tools
How can these metrics be calculated to determine the ROI of UX? There are three overlapping approaches: the Single Usability Metric, the Conversion Rate, and the Drop Off Rate (fewer drop-offs being a solid indication of fewer errors in the UX design).
SUM – Single Usability Metric – is a standardized usability metric measuring essential components of usability such as effectiveness, efficiency through task completion rates, error counts, task times, and task-time satisfaction. SUM calculations can be accomplished using an agreed algorithm that analyzes UX errors to optimize conversion.
The Conversion Rate is the ratio of total visitors who take the desired action, such as new customers onboarding. Onboarding would fall into the macro conversion category. At the same time, there is also a subset that might feature – for example – blog viewing, which then tends to feed forward to the macro conversion results.
It's unlikely that any app will ever achieve a 100% conversion rate, so measuring the Drop Off Rate is essential, too. How many potential site users are not following through from their initial contact? And why? Is it because the UX-UI is too tricky to navigate? Is it too complex, with too many options? 'Keep It Simple Stupid' (the KISS principle) is an old idea, but it has survived because it is true. Calculating the Drop Off rate is a vital part of the UX process because it tells the real story of the difference between how an app is being developed and how it may be used or not used.
Research and testing, then more testing
Classic design is pleasing to the senses, so great UX-UI design must likewise be just that. However, it goes much further than that. Some agencies have pioneered the importance of careful research and rigorous testing to create products that are as perfect as possible and truly ready for use. That means being aware of all the stakeholders in a project. With tremendous experience in UX-UI design, MNDWRK is an example of a breed of agencies that, while certainly delivering stylish and elegant solutions, also concentrate a vast amount of their firepower on measurement and analysis. Banks and fintechs are, after all, very interested in their Return On Investment, and the ROI of UX can be measured and assessed.
Napkin Math
So let's take a brief look at a typical calculation:
A bank has an internal interface used by 1,000 employees whose wages, taxes and insurance contributions, plus a share of the infrastructure costs and equipment, can be averaged out to €50 an hour. With each employee working approximately 2,000 hours a year, the workforce costs 1,000 x 2,000 x €50 = €100M a year.
However, if the bank replaces its software with something more efficient that costs – say – €10M (comprising €9M dev and business + €1m UX) and that €1M spent on UX results in software with a 10% speed increase, then ROI is achieved in 1 year.
Influencing and engaging
Having said all this about the measurement of the ROI of UX, of course, great design also serves to influence and engage. Just look at how Apple Computers won a loyal and dedicated audience for its products over many years. The design demonstrates a brand's message and expresses core values and beliefs, which go on to create differentiation from other brands within the same market. Both banks and fintechs operate in crowded and highly competitive fields, so they must ensure that they have great UX from every point of view – for effectiveness, functionality and outstanding ROI – so that the end user might say, “Wow! Will you take a look at that!”
