We’re heading into the holiday season of 2022, and still, the words that Philip Kotler said decades ago ring truer than ever: “customer is king.”
With the approval of your kings (and queens), your product - app, website, software, or any other offering to them - is bound to do well in the market. Which brings us to the question:
How can you tell what goes on in your customer’s minds?
Put another way, is your product going to help your target segment overcome their challenges?
Before you go all-in and launch, ensure that you’ve made a sure bet - with usability testing.
This guide answers ‘what is usability testing’ and delves into some methods, questions, and ways to analyze the data.
TL;DR tip: Scroll to the end for a convenient list of FAQs about usability testing.
Usability testing answers your questions about any and all parts of your product’s design. From the moment users land on your product page to the time they show exit intent, you can get feedback about user experience.
Using this feedback, you can create products from perfect prototypes that delight your customers and improve conversion rates. This also works out well if you want to improve your product that is already in the market.
Usability testing is essential because it makes your website or product easier to use. With so much competition out there, a user-friendly experience can be the difference between converting the visitors to customers or making them bounce off to your competitors.
For example, the bounce rate on a web page that loads slower than 3 seconds can jump up to 38%, meaning more people leaving the site without converting.
In the same way, a product with a complicated information architecture will add to the learning curve for the users. It may confuse them enough to abandon it and move to a more valuable product.
When you’re told by unbiased testers or users that your product “feels like it’s missing something,” but they can’t tell you what “something” is, isn’t it frustrating? It’s a pity you can’t turn into Sherlock Holmes and ‘deduce’ what they mean by that vague statement.
Philip Kotler even quoted the author of Sherlock Holmes, Sir Arthur Conan Doyle: “It is a capital mistake to theorize before one has data.” in his book Kotler on Marketing: How to Create, Win, and Dominate Markets.
How cool would it be to turn the subjective, vague survey responses into objective, undeniable data? This data can be the source of quite ‘elementary’ as well as pretty deep insights that you’ve been seeking.
That’s where usability testing comes in.
It puts the users in the driver's seat to guide you to all the points which can deter them from having a good experience.
Be it a prototype or working product, you can use usability tests at any stage to streamline the flows and other elements.
It’s time to see what benefits usability testing can bring to your product development cycle. Irrespective of the stage at which your product is, you can run usability testing to pinpoint critical issues and optimize it to improve user experience.
There are so many processes & flows on your website or product, from creating an account to checking out. How would you know if they are easy to use and understand or not?
For example, a two-step checkout process may seem like a better optimization strategy than a three-step process. You can also use A/B testing to see which one brings more conversion.
But the question is, which is more straightforward to understand? Even though a two-step checkout involves fewer steps than 3 step checkout, it may complicate the checkout process more, leaving first-time shoppers confused.
That is where usability testing helps to find the answer. It helps to simplify the processes for your users.
Case Study- How McDonald's improved its mobile ordering process
In 2016, McDonald's rolled out their mobile app and wanted to optimize it for usability. They ran the first usability session with 15 participants and at the same time collected survey feedback from 150 users after they had used the app.
It allowed them to add more flexibility to the ordering process by introducing new collection methods. They also added the feature which allowed users to place the order from wherever they want. The customers could then collect the food upon arriving at the restaurant. It improved customer convenience and avoided congestion at the restaurants.
Following the success of their first usability test iteration, they ran successive usability tests before releasing the app nationwide.
They wanted to test the end-to-end process for all collection methods to understand the app usability, measure customer satisfaction, identify areas of improvement, and check how intuitive the app was.
They ran 20 usability tests for all end-to-end processes and surveyed 225 first-time app users.
With the help of the findings, the app received a major revamp to optimize the User Interface or UI elements, such as more noticeable CTAs. The testing also uncovered new collection methods which were added to the app.
Just like in the case of McDonald's in the previous point, usability testing can help you evaluate whether your prototype or proof of concept meets the user's expectations or not.
It is an important step during the early development stages to improve the flows, information architecture, and other elements before you finalize the product design and start working on it.
It saves you from the painstaking effort of tearing down what you've built because it does not agree with your users.
Case Study - SoundCloud used testing to build an optimized mobile app for users
When SoundCloud moved its focus from desktop to mobile app, it wanted to build a user-friendly experience for the app users and at the same time explore monetization options. With increased demand on the development schedule because of the switch from website to mobile, the SoundCloud development team decided to use usability testing to discover issues and maintain a continuous development cycle.
The first remote usability test iteration involving 150 testers from 22 countries found over 150 bugs that affected real users. It also allowed the team to scale up the testing to include participants from around the world.
Soundcloud now follows an extensive testing culture to release smoother updates with fewer bugs. It helps them to add new features with better mobile compatibility to streamline their revenue models.
Bonus Read: Step by Step: Testing Your Prototype
With usability, it is possible to map whether the product performs as intended in the actual users' environment. You can test whether the product is easy to use or not. The question that testing answers is - Does it have all the features to help the user complete their goal in their preferred conditions?
Case Study - Udemy maps expectations and behavior of mobile vs. desktop app users
Udemy, an online education platform, always follows a user-centric approach and learns from its users' feedback. It has helped them to put customer insights into the product roadmap.
The team wanted to understand the behavioral differences between users from different platforms such as desktop and mobile to map their expectations.
So, they decided to create a diary study using remote unmoderated testing as it would allow participants from different user segments and provide rich insights into the product.
They asked the participants to use a mobile camera to show what they were doing while using the app to study their behavior and the environment.
The data from the studies were centralized into one place for the teams to take notes of the issues and problems that people faced while using the Udemy platform.
With remote testing, the Udemy team got insights into how students used the app in their environment, whether on mobile or desktop. For example, the initial assumption that people used the mobile app on the go was proven wrong. Instead, the users were stationary even when they were on the Udemy mobile app.
It also shed new light on the behavior of mobile users helping the team to use the insights into future product and feature planning.
Bonus Read: Product Feedback Survey Questions & Examples
With every usability test, you can find major and minor issues that hinder the user experience and fix them to optimize your website or product.
Case Study - Satchel uses usability testing to optimize their website for conversions
Satchel, an online learning platform, wanted to test the usability of their website to use the inputs into their conversion rate optimization process. The test recruited 5 participants to perform specific tasks and answer the questions to help the team review the functionality and usability of user journeys.
The finding revealed one major usability issue with the website flow of one process in which users were asked to fetch the pricing information.
It indicated a high lostness score (0.6), which is the ratio of optimal to the actual number of steps taken to complete a task. It means that users were getting lost while completing the assignment or were taking longer than expected to complete it. Some participants even got frustrated, which meant churned customers in the case of real users.
Using the insights, the team decided to test a new design by adding the pricing and 'booking the demo' link into the navigation menu.
The results showed a 34% increase in the 'book a demo' requests validating their hypothesis.
The example clearly shows how different tests can be used to develop a working hypothesis and test it out with statistical confidence.
In all the studies discussed above, the ultimate aim of usability testing was to improve user experience. Because if the users find your website or product interactive and easy to use, they are more likely to convert into customers.
Case study - Autotrader.com improves user experience with live interviews
Autotrader.com, an online marketplace for car buyers and sellers, was looking to improve users' car buying experience on the website. To understand user behavior and map their journey, the team used live conversations to connect with recent customers.
Remote interviews made it possible to test users from across the country to understand how the behavior changed with location, geography, demographics, and other socioeconomic factors. It helped the testing team to connect with users across different segments to compare their journeys.
They discovered one shared experience - both new and experienced customers found the process of finding a new car very exhausting.
“Live Conversation allows me to do journey mapping type interviews and persona type work that I couldn’t do before because of staff and budget constraints— bringing these insights into the company faster and at a much lower cost.”
Bradley Miller, Senior User Experience Researcher at Autotrader
Live interviews also helped uncover new insights about the consumer shopping process. The team found out that most consumers started their car buying journey from search engine queries.
The users were not looking for a third-party car website as thought earlier. It meant they could land on any page on the website. It became clear that the landing pages needed to be revamped to provide a more customer-centric experience to the visitors.
The team redesigned each page to act as a starting point for the visitor journey, dispelling the assumption that people already knew about the website.
Before you conduct a usability test, it is crucial to understand different usability testing types to pick the one that suits your needs and resource availability.
There are mainly six types of usability testing:
|Moderated Usability Testing||Unmoderated Usability Testing|
|Carried out in the presence of a moderator who oversees the entire sessions.||Does not require a moderator. But you need a tool that can guide the participants through the test and display test instructions clearly.|
|Usually carried out in lab conditions.||Can be carried out in lab settings, or users’ environments like home, office, etc.|
|The moderator observes the user behavior and actions. So this method may or may not require a recording tool.||Requires a recording tool to collect data for analysis after the test.|
|Provides deeper insights than unmoderated testing as both the moderator and participant can exchange questions.||Insights can only be collected with predefined questions that are shown to the users with the help of the testing tool.|
The difference between remote and unmoderated testing is that a moderator may be present during a remote usability test.
|In-person Usability Testing||Remote Usability Testing|
|It is carried out in the physical presence of a moderator in lab testing conditions.||Can be carried out over the internet or phone in the absence or presence of a moderator.|
|Expensive and takes more time than remote testing.||It is a cost-effective method as it does not require any physical venue and other preparations.|
|Provides in-depth feedback as the moderator can study the behavior of the user.||The data collected is limited as compared to in-person testing.|
|Can test only a few participants at a time.||Can test a large number of users simultaneously.|
|Qualitative Usability Testing||Quantitative Usability Testing|
|Involves testing usability in terms of qualitative assessments such as easy or hard to use.||Measure quantifiable metrics such as task completion time, success rate, number of failures, etc.|
|A direct measure of product usability as it gives clear insights about issues and problems directly from the participants’ feedback.||An indirect method of testing the product usability as it provides the measurement of usability based on user performance.|
|Points out the exact problems that participants faced during the test. For example, 70% of the participants who failed a task provided feedback about the missing step in the checkout process.||Only provides direction to the problem area. For example, if 70% of users could not complete a task, it means there is a problem with the flow or design but what?|
|Findings are purely based on the moderator's experience, interpretations, and knowledge. They can change from moderator to moderator.||Provides statistically significant numbers that can be compared with successive testings.|
Benchmark or Comparison testing is done to compare two or more design flows to find out which works best for the users.
For example, you can test two different designs for your shopping cart menu -
It is a great way to test different solutions to the same problem/issue to find the optimal solution preferred by your users.
You can run benchmark testing at any stage of your product development cycle.
Now that you know about usability tests types, let's discuss different usability testing methods.
Each method has different applicability and approach towards testing the participants. You can use multiple methods in conjunction to get deeper insights into your users.
Lab usability tests are conducted in the presence of a moderator under controlled conditions. The users perform the tasks on the website, product, or software, and the moderator observes them making notes of their actions and behavior. The moderator may ask the users to explain their actions to collect more information.
The designers, developers, or other personnel related to the project may be present during the test as observers. They do not interfere with the testing conditions.
Limitations of lab usability testing
Tips for conducting effective lab usability tests:
Paper prototyping is an early-stage lab usability testing performed before the product, website, or software is put into production. It uses wire diagrams and paper sketches of the design interface to perform the usability test.
Different paper screens are created for multiple scenarios in the test tasks. The participants are given the tasks, and they point to the elements they would click on the paper model interface. The human-computer (developer or moderator) then changes the paper sketches to bring a new layout, snippets, or dropdowns as it would occur in the product UI.
The moderator observes the user's behavior and may ask questions about their actions to get more information about the choices.
Example Usability Test with a Paper Prototype
Card sorting is helpful in optimizing the information architecture on your website, product, or software. This usability testing method lets you test how users view the information and its hierarchy on your website or product.
In moderated card sorting, the users are asked to organize different topics (labels) into categories. Once they are done, the moderator tries to find out the logic behind their grouping. Successful card sorting requires around 15 participants.
When to use card sorting for usability testing?
You can use card sorting to:
Types of card sorting tests:
Whether it’s an moderated or unmoderated card sorting test, there are three types of card sorting tests:
|Open Card Sorting||Closed Card Sorting||Free Card Sorting|
|The labels (topics) are predefined.||The topics and categories are both predefined.||The participants create both the categories and the topics.|
|The participants are asked to create their categories and group the given topics into them.||The participants are asked to group the topic into the given categories.||Useful during the very early stage of development when you want to come up with a website design.|
|Useful when you have the topics in mind and want to see how users will process and group the information||Useful when you want to gauge how users will sort the given content into categories.|
Advantages of card sorting:
In unmoderated card sorting, the users sort the cards alone without a moderator. You can set up the test remotely or in lab conditions. It is much quicker and inexpensive than a moderated sorting method.
Unmoderated card sorting is usually done using an online sorting tool like Trello or Mural. The tool records the user behavior and actions for analysis later.
If card sorting helps you design the website hierarchy, tree testing lets you test the efficiency of a given website architecture design.
You can evaluate how easily users can find the information from the given categories, subcategories, and topics.
The participants are asked to use the categories and subcategories to locate the desired information in a given task. The moderator assesses the user behavior and the time taken to find the information.
You can use Tree testing to:
Example of Card Sorting & Tree Sorting
Guerilla testing requires you to approach random people outside, such as parks, coffee houses, or any other public area, and ask them to take the test. Since it eliminates the need to find qualified participants and a testing venue, it is one of the most time-efficient and cost-effective testing methods to collect rich insights about your design prototype or the concept itself. The acceptable sample size is between 6 to 12 participants.
Session recording is a very effective way to visualize the user interactions on your functional website or product. It is one of the best unmoderated remote usability testing methods to identify the visitors' pain points, bugs, and other issues that might prevent them from completing the actions.
This type of testing requires screen recording tools such as SessionCam. Once set up, the tool anonymously records the users' actions on your website or product. You can analyze the recording later to evaluate usability and user experience.
It can help you visualize the user's journey to examine the checkout process, perform form analysis, uncover bugs and broken pathways, or any other issues leading to a negative experience.
Remote Screen recording with qualified participants
You can also use screen recording using specific participants and the think-aloud method, where people say their thoughts aloud as they perform the given tasks during the test.
In this method, the participants are selected and briefed before the test. It requires more resources than anonymous session recording. It's a fantastic method to collect in-the-moment feedback and actual thoughts of the participants.
Eye-tracking testing utilizes a pupil tracking device to monitor participants' eye movements as they perform the tasks on your website or product. Like session recording, it is an advanced testing technique that can help you collect nuanced information often missed by inquiry or manual observation.
The eye-tracking device follows the users' eye movements to measure the location and duration of a user's gaze on your website elements.
The results are rendered in the form of:
This type of testing is useful when you want to understand how users perceive the design UI.
Expert reviews involve a UX expert to review the website or product for usability and compliance issues.
There are different ways to conduct an expert review:
A typical expert review comprises of following elements:
An expert review can be conducted at any stage of the product development. It is an excellent method to uncover the issues with product design and other elements quickly.
But since it requires industry experts and in-depth planning, this type of testing can add substantial cost and time to your design cycle.
This last method is more of a proof of concept than a working usability testing methodology. Various papers and studies call for an automated usability tool that can iron out the limitations of conventional testing methods.
Here are two interesting studies that outline the possibilities and applications of an automated usability testing framework.
Conventional testing methods, though effective, carry various shortcomings such as inefficiency of the moderator, high resource demands, time consumption, and observer bias.
With automated usability testing, the tool would be self-sufficient to conduct the following functions:
It will allow developers and researchers to reduce development time as the testing and optimization iterations could be carried simultaneously.
One of the common questions asked about usability testing is - 'when can I do it?' The answer is anytime during the product life cycle. It means during the planning stage, design stage, and even after release.
Whether you are creating a new product or redesigning it, conducting usability tests during the planning or initial design stage can reveal useful information that can prevent you from wasting time in the wrong place.
It's when you are coming up with the idea of the product or website design. So testing it out can help you dispel initial assumptions and refine the product flows while still on paper.
For example, you can test whether the information architecture you are planning will be easy to understand and navigate for users. Since nothing is committed, it will help restructure it if needed without much effort.
You can use usability testing methods like paper prototyping, lab testing, and card sorting to test your design concept.
Now that you have moved into the development stage and produced a working prototype, you can conduct tests to do behavioral research.
At this point, usability testing aims to find out how the functionality and design come together for the users.
While usability test during the planning stage gives you qualitative insights, with design prototype, you can measure quantitative metrics as well to measure the product's usability such as:
This data can help validate the design and make the necessary adjustments to the process flows before continuing to the next phase of development.
There is always room for improvement, so usability testing is as crucial after product launch as well.
You may want to optimize the current design or add new features to improve the product or website.
It is beneficial to test the redesign or update for usability issues before deploying it. It will help to evaluate if the new planned update works better or worse than the current design.
Running a successful usability test depends on multiple factors such as time constraints, budget, and tools available at your disposal.
Though each usability testing method has a slightly different approach due to the testing conditions and depth of research, they share some common attributes, as explained in this section.
Let’s explore the eight common steps to conduct a usability test.
Irrespective of the usability testing method, the first step is to draw the plan for the test. It includes finding out what to test on your website or product.
It can be the navigation menu, checkout flow, new landing page design, or any other crucial process.
If it is a new website design, you probably have the design flow in mind. You can create a prototype or wire diagram depicting the test elements.
But if you are trying to test the usability of an existing website or product flow, you can use the data from various tools to find friction points.
a. Google Analytics (GA)
Use the GA reports and charts as the starting point to narrow your scope. You can locate the pages with low conversions and high bounce rates, compare the difference between desktop vs. mobile website performance, and compare the traffic sources and other details.
b. Survey feedback
The next step is to deploy surveys at the desired points and use the survey feedback to uncover the issues and problems with these pages.
You can choose from the below list of different types of survey tools based on your requirements:
c. Tickets, emails, and other communication mediums
Complete the circle by collating the data from tickets, live chat, emails, and other interaction points.
These can be valuable, especially when you are hosting a SaaS product. Customers’ emails can reveal helpful information about bugs and glitches in the process flows and other elements.
Once you have the data, compile it under a single screen and start marking the issues based on the number of times mentioned, criticality, number of requests, and other factors. It will let you set the priority to choose the element for the test. Plus, it will help set clear measurement goals.
It is necessary to set the goals for the test to examine its success or failure. The test goal can be qualitative or quantitative in nature, depending on what you want to test.
Let's say you want to test the usability of your navigation menu. Start by asking questions to yourself to identify the purpose of your test. For example,
Once you have the specific goals in mind, assign suitable metrics to measure during the test.
These metrics will help to establish the outcome of the test and plan the iteration.
Here is a sample goal template you can use in the usability test
This is how it will look once filled:
The next step is to find the most suited method to run the test and plan the essential elements for the chosen usability testing method.
Once you have decided on the method, it is time to think about the overheads.
Along with the task for the actual test, prepare a pre-test and introductory script to get to know about the participant (user persona) and tell them the purpose of the usability test. You can create scenarios to help the participants relate the product or website to their real-world experience.
Suppose you are testing a SaaS-based project management system. You can use the following warm up questions to build user personas:
Use the information to introduce the participant to the test's purpose and tell them about the product if they have never heard of the concept.
Probably the most important part of usability testing; tasks are designed as scenarios that prompt the participant to find the required information, get to a specific product page, or any other action.
The task can be a realistic scenario, straightforward instructions to complete a goal or a use case.
Pro tips: Use the data from customer feedback and knowledge of customer behavior to come up with practical tasks.
Using the previous example, let's say you have to create a task for usability testing of your project management tool. You can use first scenarios like this:
'You are a manager of a dev-ops team with 20 people. You have to add each team member to your main project - 'Theme development.' How will you do it?'
This scenario will help you assess the following:
The second scenario can be;
'Once you have added the team members, you want to assign a task to two lead developers, Jon and Claire, under the theme development project. The deadline for the task needs to be next Friday. How will you do it?
Use this scenario to test the following:
If the test is moderated, ask follow-up questions to find the reason behind user actions. If the test is unmoderated, use a screen recording tool or eye-tracking mechanism to record users' actions.
Remember the sequence of the task and the associated scenario will depend on the elements you want to test for usability.
There are multiple ways to choose the participants for your usability test.
To increase the chances of participation, always add an incentive for your participants, such as gift cards or discounts codes.
With everything in place, it is time to run a pre-test simulation to see if everything is working as intended or not. A pilot test can help you find issues with the scenarios, equipment, or other test-related processes. It is a quality check of your usability test preparation.
With pilot testing, you can check :
If it is an in-person moderated test, start with the warmup questions and introductions from the pre-test script. Make sure the participants are relaxed.
Start with an easier task to help the participants feel comfortable. Then transition into more specific tasks. Make sure to ask for their feedback and explore the reasons behind their actions, such as;
For the remote unmoderated tests, make sure that the instructions are clear and concise for the participants.
You can also include post-test questions for the participants, such as;
Once the test is over, it is time to analyze the results and turn the raw data into actionable insights.
a. Start by going over the recordings, notes, transcripts, and organize the data points under a single spreadsheet. Note down each error encountered by the user and associated task.
b. One way to organize your data is to list the tasks in one column, the issues encountered in the tasks in the next column, and then add the participant's name next to each issue. It will help you point out how many users faced the same problem.
c. Also, calculate the quantitative metrics for each task, such as success rate, average completion time, error-free rate, satisfaction ratings, and others. It will help you track the goals of the test as defined in point 2.
d. Next, mark each issue based on its criticality. According to NNGroup, the issues can be graded on five severity ratings ranging from 0-4 based on their frequency, impact, and persistence:
0 = I don't agree that this is a usability problem at all
1 = Cosmetic problem only: Need not be fixed unless extra time is available on project
2 = Minor usability problem: Fixing this should be given low priority
3 = Major usability problem: Important to fix, so should be given high priority
4 = Usability catastrophe: Imperative to fix this before product can be released
e. Create a final report detailing the highest priority issues on the top and lowest priority problems at the bottom. Add a short, clear description of each issue, where and how it occurred. You can add evidence like recordings to help the team to reproduce it at their end.
f. Add the proposed solutions to your report. Take the help of other teams to discuss the issues, find out the possible solutions, and include them in the usability testing report.
g. Once done, share the report with different teams to optimize the product/website/software for improving usability.
Do you feel that you know everything there is to know about your product and its users?
If you answer with a yes, then what is the purpose of a usability test, you may ask.
No matter how much you know about your customers, it isn’t wise to ignore the possibility that there is more to be learned about them, or about any shortcomings in your product.
That is why what you ask, when you ask, and how you ask is of uttermost importance.
Here are a few examples of usability testing questions to help you form your own.
Check if your design communicates what the product/website is at first glance.
Pro-Tip: If you’re testing digitally with a feedback tool like Qualaroo, you can even time your questions to pop up after a pre-set time spent on-site for a more accurate first glance test.
Develop task-specific questions for common user actions (depending upon your industry).
Pro-tip: If you want to test the ease with which users perform specific tasks (like the ones listed above), consider structuring your tasks as scenarios instead of questions.
Ask these questions after users complete test tasks to understand usability better.
Pro-tip: If you are getting users to complete more than one task, limit yourself to no more than 3 questions after each task to help prevent survey fatigue.
Finalize testing with broad questions that collect new information you haven’t considered.
Pro-tip: No matter which way you phrase your final questions, we recommend using an open-ended answer format so that you can provide users with a space to share feedback more freely. Doing so allows them to flesh out their experience during testing and might even inadvertently entice them to bring up issues that you may never have considered.
If you’re wondering how to conduct usability testing for the first time or without having to jump through hoops and loops of code, you can simply stroll over to Qualaroo’s survey templates.
We have created customizable templates for usability testing, like SUPR-Q (Standardized User Experience Percentile Rank Questionnaire - with or without NPS), UMUX (Usability Metric for User Experience - 2 positive & 2 negative statements), and UMUX Lite (2 positive statements).
SUPR-Q is a validated way to measure the general user experience on a website or application. It includes 8 questions within 4 areas: usability, trust/credibility, loyalty (including NPS), and appearance. However, it doesn’t identify bottlenecks or problems with navigation or specific elements of the interface. 50 is the comparison benchmark for assessing your product’s UX.
UMUX allows you to measure the general usability of a product (software, website, or app). It has 4 statements for users to rate on a 5- or 7-point Likert scale. However, it isn’t generally used to measure specific characteristics like usefulness or accessibility, nor for identifying navigation issues, bottlenecks, or problems that are related to specific elements of your product’s interface.
On a related note, if you have launched a product aimed specifically at smartphone users and you wish to understand the contextual in-app user experience (UX), simply take these 3 steps.
Even though there are multiple usability test methods, they share some general guidelines to ensure the accuracy of the test and results. Let’s discuss some of the do’s and don’ts to keep in mind while planning and conducting a usability test.
The first thing to remember is to do a quality check of the usability test before going live. You wouldn't want anything to fall apart during actual testing. Be it a broken link, faulty equipment, or ineffective questions/tasks. It would mean a wastage of time and resources.
Use a testee who is not associated with the usability test. It can be a member of another team in your organization. Run the usability test simulation under the actual conditions to gauge the efficiency of your tasks and prototype. Pilot testing can help to uncover previously undetected bugs.
It can help to:
It is always a good practice to let your team attend the usability test as observers. It can produce a two-pronged effect:
Pro tip: Be cautious as not to disturb or talk to the participants. The observer's role is to be invisible.
According to NNGroup, five users in any usability test can reveal 75% of the usability problems.
However, considering other external factors, there are different acceptable sample sizes for different usability testing methods.
Pro tip: If you aim to measure usability for multiple audience segments, the sample size would increase accordingly to include representation for each segment.
So, use the correct sample size for your usability test to make the results statistically significant.
Not all participants may show up for the usability test, whether it is in-person or remote. That's why it is helpful to recruit more participants than your target sample size. It will ensure that testing reaches the required statistical significance and you obtain reliable results.
A post-test interview is a potential gold mine to collect deeper insights into user behavior. The users are more relaxed than during the test to provide meaningful feedback about their difficulties in performing the task, delights about the product, and overall experience. Plus, it also presents the opportunity to ask follow-up questions that you may have missed during the tasks.
Pro tip: The best way is to calculate the total session time by including the post-test interview period in it. For example, if you are planning each session to be 10-15 minutes long, keep 2-3 minutes for post-test questions.
If your participants are nervous or stressed out, they won't be able to perform the tasks in the best way, which means skewed test results. So, try to make the participants feel relaxed before they start the test.
One way is to have a small introduction round during the pre-test session. Instead of strictly adhering to the question sheet, ask a few general friendly questions as the moderator to establish a relationship with the user. From there, you can smoothly transition into the testing phase without putting too much pressure on them.
The “observer effect” or “Hawthorne Effect'' is when people in studies change their behavior because they are being watched. In moderated usability testing, the participants may get nervous or shy away from being critical about the product. They may not share their actual feedback or ask questions that may come to their mind. All these behavioral traits may lead to test failure or unreliable test results.
So, make sure that the moderator does not influence the participants. A simple trick is to pretend that you are writing something instead of constantly watching over the participants.
The observer effect is one more reason to have a friendly pre-test conversation and tell the participants to ask questions when they don't understand something and share their feedback openly. Discuss the test's purpose so they understand their feedback is valuable to make the product better.
The overarching purpose of this usability testing guide was to help answer this essential question: how do I create the best product that satisfies customers (and as a bonus, outshines the competition)? We hope it threw a bright light on the possible ways to answer the question. Plus, here are a few pitfalls that are best avoided as you search for the answers:
The success of the usability test depends on the tasks and scenarios you provide to the participants. If the scenarios are hard to understand, the participants may get confused. It will lead to a drop in task success rate but not due to problems with usability but the questions. The problem may get compounded in unmoderated usability testing where the participant cannot approach a moderator if stuck.
That's why it is essential to keep your sentences concise and clear so that the tester can follow the instructions easily. A pilot test is an excellent way to test the quality of the questions and make changes.
Leading questions are those that carry a response bias in them. These questions can unintentionally steer the participants in a specific direction. It can point towards a step that you may want the participants to take or an element you want them to select.
Leading questions nullify the usability test as they help the participants to reach the answer. So, test your scenarios and questions during the pilot run to weed out such questions (if any).
It is essential to set the goals clearly to deliver a successful usability test. Whether the goals are qualitative or quantitative, assign suitable metrics to measure them properly.
For example, if the task aims to test the usability of your navigation menu, you may want to see whether the users can find the information or not. But to quantify this assessment, you can also calculate the success rate and time taken by participants to complete their tasks.
While the qualitative analysis will reveal points about user experience, the quantitative data will help calculate the reliability of your findings. In this way, you can approach the test results objectively.
One of the biggest mistakes in usability testing is using an incorrect audience sample, which leads to inaccurate results.
For example, If you use your friends or coworkers who may know about the product/software, they may not face the same problems that actual first-time users may experience.
In the same way, if they are entirely unaware of the product fundamentals, they might get stuck at points which your actual audience may easily navigate.
To recruit the right audience, start by focusing on the question - who will be the actual users of the test elements? Are they new users, verified customers, or any other user segment?
Once you have your answers, you can set the proper goals for the test.
Another grave mistake is repeatedly interrupting the participants. The usability test is aimed at observing the users testing the product/software without any outer influence. The moderator can ask the questions and guide them if necessary.
Constantly bugging the participants may make them nervous or frustrated, disturbing the testing environment and providing false results.
As mentioned before, a pilot test is a must in usability testing. It helps weed out the issues with your test conditions, scenarios, equipment, test prototype, and other elements.
With pilot testing, you can sort out these problems before you start the test.
The purpose of usability testing is to simulate actual user behavior to measure how easy it would be for them to use the product. If you guide the users through the scenario, you are compromising the test results.
The moderator can help the participants understand the scenario, but they may not help them complete the task or point towards the solution.
Another mistake to avoid is to draw conclusions from the test result of the first two or three users. It is necessary to be vigilant towards all the participants before concluding any presumptions.
Also, do not rush the testing process. You may be tempted to feel that you have all the information after testing a few participants but do not rush the test. Scan all the participants to establish the reliability of your results. It may point you towards new issues and problems.
Bonus Read: 30 Best A/B Testing Tools Compared
Experimentation and testing is an iterative process. Plus. since the sample size in usability testing is usually small, it is a big mistake to treat the results from one test phase as definitive.
The same problem is with the implemented solution of the issues found in the test. There can be many solutions to a single problem, so how will you know which one will work the best?
The only way to find out is to run successive tests after implementing the solution to optimize product usability. Without iterations, you cannot tell if the new solution is better or worse than the previous one.
It is true what they say: experience is the best teacher. As you do more tests, you will gain a better understanding of what usability testing actually is about - creating a perfect product.
It stands to reason that the easier it will be for your prospective customers to use your product, the more sales you will see. Usability testing helps you eliminate any unforeseen glitches and improve user experience by collecting pertinent user feedback for actionable insights. To get the best insights, Qualaroo makes usability testing a delightful experience for your testers.
Qualaroo surveys gather insights from real users & improve product design