Usability Testing Of Mobile Applications: A Step-By-Step Guide

The mobile market is huge and growing at a very fast rate. With an estimated 4.5 billion subscribers worldwide, it is forecasted that the number of mobile phones will surpass the world population.

Before Begin … A Couple of Words About This Guide

As the title of this article says, this is a step-by-step guide. The reason why I went for such a structure is to provide you with completeness. You can often find articles that describe in detail a particular phase of mobile application usability testing. While these articles are great (so great that I have cited several of them in this guide), they’re often confusing for someone who is new in the area of mobile usability testing. This is where this article comes in.

One note of caution: Before you start doing usability testing on mobiles, you should know how to design mobile interfaces based on best practices of mobile usability. If you don’t, I recommend you take the course Mobile User Experience (UX) Design (which also comes with an industry-recognized Course Certificate). Or you can buy a book on the topic.

Alright. Let’s get started. First, let’s start with a question:

Question no. 1 – Do you need to read each step? No. If you feel confident about any section, feel free to skip it. Here is an index of the main sections of this guide:

Key sections of this guide (TL;DR)

1. What is mobile application usability testing?
2. Setting the objectives of the usability test
3. Creating the tasks that will be performed
4. Creating the test documents
5. Finding and recruiting test participants
6. Implementing the usability testing method
7. Reporting usability test findings

Question no. 2 – Can this guide be followed for testing the usability of mobile websites? Yes. The testing methodology is for mobile testing. It can be used for both mobile website testing and mobile application testing. I focused on mobile application usability testing because mobile applications are extremely popular (as will be explained in the next section) and also because it seems that the majority of articles, or at least those that I came across tend to focus more on mobile website testing.

Mobile Application Usability

So let me kick off this guide with a question. What do phone users spend most of their time on? Interacting with mobile applications (or apps as most of us refer to them). Yes, you read that correctly. A recent study shows that phone users in the US spend 86% of their mobile usage time solely on apps. Another study actually calculated this figure to be as high as 89%. And taking this further, it has also been found that mobile users spend 80% of their mobile app usage time using just five apps (out of the total of 24 apps they typically use in a month).

Thus it comes as no surprise that there has been an explosive increase in the number of phone applications especially those for games and social media. Forbes actually estimates that by next year, there will be almost 270 billion apps downloaded.

That being said, mobile phone applications are still restrained by the relatively small screen size and limited performance capabilities of the devices on which they run. It is true that phones have come a long way with larger screen sizes and increased processing capabilities. It is also true that mobile app design has evolved considerably. However, they lack the screen sizes and processing capabilities of larger devices such as laptops or desktop computers.

Research shows that usability is key for the success of mobile apps. In fact, a common trend among successful mobile phone applications is that they are perceive by users as being easy to learn, user-friendly and less time-consuming when completing tasks. Other researchers have actually identified a direct link between mobile application usability and user acceptance.

Despite the importance of mobile application usability, there is a lack of an agreed-upon list of guidelines. In this regard, the best way to evaluate the usability of mobile applications is through usability testing.

Mobile Application Usability Testing

Way before you start thinking about what you will test and how, you need to devise a usability test plan. This serves as a blueprint for the actual test and although there is no specifically-set structure, it typically contains the following sections (Rubin et al. 2008):

1. Purpose, goals and objectives of the test
2. Research questions
3. Participant characteristics
4. Method (test design)
5. Task list
6. Test environment, equipment and logistics
7. Test facilitator role
8. Data to be collected and evaluation measures
9. Report content and presentation

An interesting variant that can be very handy when testing mobile applications (due to the smaller teams involved) is the 1-page usability test plan proposed by David Travis from UserFocus.

Based on the usability test plan, it can be stated that in order to test the usability of a mobile application, (apart from the application itself) you need the:

1. Objectives of the test
2. Tasks that will be performed
3. Test Documents (content form, orientation script, pre & post-test questionnaires)
4. Test Participants
5. Test Method

1. The Objectives of the Usability Test

The first step of any usability testing session is to set the goals straight. What questions do you want to answer with the usability test? What hypothesis do you want to test with the usability test?

So how does one set the goals? There are various ways one can do this. One handy technique that I use is a variaton of the methodology outlined by Michael Margolis from Google Ventures. This basically involves asking a number of questions (interviewing) to the app’s stakeholders (and that includes the development team behind it) to explore important areas:

• The app’s roadmap
• Users and markets for whom the app is targeted
• The app’s competitors
• Research that has already been done and other that the team requires
• The potential impact of the above research
• Timing and scope

The answers obtained from these interviews will give you two very important things:

1. What the stakeholders already know
2. What they would like to know

Based on the answers given to the above questions, it is now easier to start identifying the goals and what usability metrics one should use to measure them.

It is also very important that the identified goals are:

• Specific
• Measurable
• Prioritized

2. The Tasks that will be Performed

Once the objectives have been set, it is time to move to the next step – setting tasks. Tasks are typically just one sentence long and should consist of the interactions that need to be performed by the test users, example:

• Register an account
• Sign into your account
• Upload a photo
• Accept a friend request

Instead of asking the test user to perform a task and hence make them feel that they are being tested, tasks should be converted to task scenarios. These provide more context for why the participant is doing the task and hence look more like natural interations that a typical user will perform with your application. In this regard, the task scenarios that are set should be:

• Realistic, actionable and without any clues on how to perform the steps (Source)
• Ordered in a sequence that ensures a smooth flow of the test session
• Tied to one or more objectives (Source)

As with any form of testing, it is very important to perform a dry run of the usability test to ensure that the performance of the tasks will ultimately achieve the objectives that have been set.

3. The Usability Test Documents

There are a number of documents that you would need when conducting usability testing. While the number of documents and their content may vary, you would typically need the following:

• Consent form: (for minors or adults)
• Post-test questionnaire: This content of this questionnaire can range from the two simple post-test questions recommended by Uxpert Jared Spool to standard questionnaires such as the Post-Study Usability Questionnaire (PSSUQ) or the System Usability Scale (SUS)

For additional usability testing documents that you might find useful, please head over to the Usability.gov site.

4. The Test Participants

The mobile application usability testing method that will be discussed in the next section is a user-oriented testing technique, meaning it involves real users undertaking realistic tasks that the app is intended to achieve. Although testing with real users is more resource-consuming, this realistic scenario tends to yield more accurate results.

Raluca Buidu from the Nielsen Norman Group recommends recruiting test participants who have been using their devices for at least 3 months. This would overcome any difficulties resulting from the usage of the device rather than the app itself. In addition to this, using the test participants’ own devices tends to reveal more issues since they will be using real devices rather than top-of-the-range devices and fast internet connections that are usually available in development environments.

There are several considerations that need to be taken when choosing the participants for a usability test. Participants must:

• Be representative of the users for whom the app is intended (i.e. the target users)
• Own a device whose Operating System is the one on which the app is intended to run (including the appropriate version/s)
• Be considered in terms of the stage at which the application is currently in. This may range from an app that is in its initial phases to one that has been already on the market for some time. In this regard factors such as confidentiality and the expertise that the test participants have with mobile applications could be crucial
• Be available at the time, place, frequency of the intended usability tests
• Agree to the compensation terms that you are offering (if any)
• Be ready to sign a usability test participation consent form

An effective technique to ensure that the right test participants are recruited is to build user personas and then use them to screen potential test participants in order to find the right candidates. This is applicable of course if you already have a pool of potential test participants at hand (e.g. your registered users) or if you want to perform the recruitment process yourself.

If you would like to outsource this process, there are several recruiting agencies that can do that for you. Which one to choose is entirely up to you as both methods are fine and both have their pros and cons. An interesting survey conducted by the Nielsen Norman Group) shows the following interesting findings:

• Most companies recruit their own test participants
• Recruiting agencies are quite expensive (approximately $84 per person if recruiting average consumers or students and approximately $161 per participant if recruiting high-end professionals)
• Companies recruiting their own participants spend an average of 1.15hrs of staff time per participant recruited

For additional detail on how to recruit participants for usability studies, please head over to these articles:

• How to Recruit Participants for Usability Studies by Jakob Nielsen & Hinderer Sova
• Recruiting Usability Test Participants by Usability.gov
• 6 Ways To Recruit Participants for Remote Usability Testing by Natalia Bruzi
• 7 Ways To Find Users for Usability Testing by Jeff Sauro

5. The Mobile Application Usability Testing Methodology

There are two main methods for conducting usability testing of mobile applications. Needless to say, each comes with its own set of advantages and disadvantages. These are:

1. Laboratory-based usability testing
2. Remote usability testing

In this article, we will be using laboratory-based (or lab-based) usability testing. This testing method enables the testing of mobile applications by involving real users using real devices. In it, the evaluator (whose role will be explained below) has full control over the test and can easily set the tasks – thus enabling him/her to test all usability aspects. It has also been found to yield results that are more accurate and easier to understand. In fact, several usability experts such as Raluca Budiu from the Nielsen Norman Group and Jeff Sauro from MeasuringU recommend it as the preferred usability testing method for mobile.

The ‘Components’ of a Lab-based usability test

Lab-based usability testing involves the observation of test participants performing assigned tasks using a mobile device in a setup similar to the one shown below:

The above setup can be split into the following ‘components’:

A) Camera

There are two distinct camera setups. Luckily, both will help capture the users’ gestures.

1. Document / fixed-position cameras: The camera lies on a fixed point. (e.g. IPEVO’s Ziggi HD Plus)
2. Cradle-based cameras / Mobile testing sleds: The camera is fixed to a cradle on which the mobile device is placed. The user then holds the cradle in his / her hand when performing the assigned tasks. There are a number of cameras that one can choose from such as Mr. Tappy and MOD 1000 to name a few. One can also find a number of DIY setups such as this one from Rik Williams who assembled a testing camera sled in just 15 minutes, using materials that are readily available in any office and Harry Brignull who built a testing sled for less than $8. Although DIY setups are relatively cheap and fast to build, they do have their own limitations that can affect the testing procedure such as unnecessary additional weight and less durability and portability than a professional testing sled. The advantage of using a cradle-based camera is that it allows users to operate their mobile device using natural movement – just like they would be using the app in real-life, unlike the fixed-position camera which confines the device to remain on a flat surface. Conversely, cradle-based testing sleds are sometimes criticised for the additional size and weight that they add to the device being used for testing, although the increased usage of lightweight material has minimized this.

Wireless testing?: In a recent article on Smashing Magazine, Colman Walsh describe an interesting and cost-effective setup that makes use of Apple’s AirPlay technology. The reasoning behind it is that unlike document and cradle-based system, the testing equipment is entirely invisible to the test participant. Although it is not an ‘official’ testing methodology, the article is definitely worth a read.

B) Webcam (Optional)

A webcam can be pointed towards the user’s face so as to provide an alternative view that can assist the interviewer in understanding the user’s interaction by capturing his/her facial expressions.

C) Recording software

The recording software is needed to project the session onto the facilitator’s screen and ideally record it too in the process. Some software also allow the session to be projected in real-time on multiple screens via a network. This can be particularly useful when involving a number of stakeholders. Very often, both document cameras and cradle-based cameras come with their own software. If that is not the case, then there are some very good tools that one can use so it is entirely up to you which one to use. For instance Jeff Sauro from MeasuringU uses GoTo Meeting to save the audio and video as a compressed, shareable .wmv file and Camtasia to save the webcam video. One can also use WebEx or Morae although the price for the license is a bit on the high end.

D) Test venue

Although it has traditionally been carried out in specialized usability testing labs, the availability of sophisticated technology at a reasonable price, means that one can now conduct this type of testing without the need for hiring specialized labs. If you are going to set up your own testing room, it is very important to ensure that there is adequate lighting that is ideally not right above the mobile device being used by the user since it can cause a glare on the screen.

E) Facilitator

As the title implies, the role of the facilitator is to facilitate the testing process, that is, to ensure that the test runs smoothly by addressing any issues that the test participant may have with the task being assigned or device they are using. Other than that, the role of the facilitator, who should ideally be a usability specialist, is to observe the test on their screen and ensure that the test participants perform the assigned tasks.

The Testing Procedure

The testing procedure can be structured into 6 steps as shown below. The time next to each step is based on the typical time that each step takes as recommended by David Travis from UserFocus. If one adheres to these suggested times, then each session would take approximately an hour to conduct.

Step	Time
1. Welcome / Signing of the consent form	5 min
2. Pre-test interview	5 min
3. Carrying out the test tasks	35 min
4. Post-test questionnaire	5 min
5. Post-test interview	5 min
6. Debriefing	5 min

All of the above steps have been described earlier on. However, little has been said about step 6 – debriefing. Debriefing is a process that is undertaken at the end of each test session and it involves going through and analyzing the actions performed by the participant. Since the debriefing session is conducted with the participant, this provides additional insight on why that participant performed such actions. Thus, while the test session indicates the problems, it is the debriefing session that provides the insight on why those problems occured.

Reporting the Results of the Usability Test

After all the usability test sessions have been completed, you need to go through all the data, compile it, analyze it and present it in a way that it contains actionable recommendations.

First of all, the data needs to be split between quantitative and quantitative data.

• Quantitative data from the testing session may be used to compute usability metrics such as completion rates, success rates, task times, satisfaction ratings and error rates.
• Qualitative data can be compiled to provide insights about the paths taken by participants, problems experienced and answers that they provided in the questionnaire, post-test interview and debriefing sessions.

While there is no fixed structure for reporting, it is generally recommended to include the following sections:

1. Background summary: The application that was tested, where it was tested, what equipment was used, what was tested and who was the testing team.
2. Methodology: How the test was conducted, task scenarios, metrics that were collected. Also include information about the test participants (brief summaries of demographic data – do not release any confidential data such as participant names).
3. Test Results: The compiled qualitative and quantitative data together with an analysis of this data.
4. Findings and Recommendations: Based on the observations from point 3 (i.e. substantiated by data), provide a set of recommendations that one needs to implement in order to fix any usability issues that have been identified. However, do not just report the problems here. Be sure to highlight any findings that showed good usability.

Wrapping it Up

As you can see, there is indeed a structured approach to mobile application usability testing. Better still, this approach can be tweaked to meet the needs of any application that is being tested and any stakeholder who is interested in the outcome of such tests. Moreover, the importance of usability testing, especially for mobile – with the large array of devices and their respective specs has long-been documented. This is why there is absolutely no excuse as to why, at this day and age, one should not conduct usability testing.

Want to learn more?

If you’re interested in mobile UX, then consider to take the course on Mobile User Experience, which includes templates you can use in your own projects. If, on the other hand, you’d like to…

1. learn all the details of Usability Testing
2. get easy-to-use templates 
3. learn how to properly quantify the usability of a system/service/product/app/etc
4. learn how to communicate the result to your management

… then consider to take the online course Conducting Usability Testing.

Lastly, if you want to brush up on the basics of UX and Usability, then consider to take the online course on User Experience. Good luck on your learning journey!

(Lead image: Depositphotos)