As Steve Jobs once said,
“Design is not just what it looks like and feels like. Design is how it works.”
Even though the legendary tech leader spoke out about how design needs to be functional and have its usability, it is still treated as only an aesthetic part of the development process.
Even when we put aside that entire debate, the usability of a design is a much talked about subject. No matter what is being developed, be it a software, website or an app, the design usability is one of the metrics taken into consideration when measuring the success of a platform. And when it comes to measuring usability, there is nothing better than usability testing.
Usability testing, on the other hand,has its raging battle going on between formative and summative testing. The armies are divided between these two types of usability testing and for good reasons. Both formative and summative testings are so widely different that an argument on which one is more necessary is bound to break out.
So which one is more important during the development process? The only way to know is to compare and understand the differences between the two kinds of testings. But before that, let’s take a look at what formative and summative testing means.
What Is Formative Usability Testing?
Used during the early stages of the design and development process, the formative method identifies the issues with user interface (UI) design and provides solutions to solve those issues during the primary stages of the development process.
Considered to be an excellent tool to figure out which design features are useful and which are not, formative user testing heavily influences the design decisions you will make about your product. Conducted with a team of 5-7 users, formative testing is done mainly to design a good user experience for your product.
The discovery-based process moves the iteration process forward as the testers discover more and more issues with the design of the product. The data collected during formative user testing sessions are observational in nature, and deal with the quality of the design, hence the name qualitative usability testing.
Now before we move onto discussing the difference between formative and summative user testing, let’s take a look at what summative user testing is.
What Is Summative Usability Testing?
Summative user testing provides an indirect assessment of the usability of the design. The group of testers in this situation are given a task and based on their performance, the usability quotient of the design elements are measured.
Summative user testings are done after the product has been launched in the market. Conducted by a group of more than ten testers, the data collected are more about the quantity than the quality of the design (as in, what is the percentage of the users who could use the product without running into a roadblock?), and that is why another name for summative user testing is quantitative user testing.
The Difference Between The Two: Formative And Summative
By now, it is obvious that both formative and summative user testings have significant differences between them. Whether you consider the study setups, analysis methods, or different kinds of data outcome from the two kinds of testings, there are numerous differences, and we have elaborated on a few of the differences below.
What Questions Do They Answer?
One of the main reasons why usability tests are performed is to find the answers regarding the effectiveness of the design. And the main difference between Formative and Summative user testing is the questions they answer.
Formative or qualitative user testing answer the why and how questions of the design usability. This testing is performed during the initial stages of the design process and the questions it answers is basically why something is not working. When the users performing the test get stuck when performing an action, they look for the reason why the design is not working, and upon finding the reason, find the solution to fix it. You can also check out the 9 step UI UX guideline that helps in mapping the user journey better.
Summative or quantitative testing, on the other hand, answer the question regarding how many or how much. Quantitative testings are performed among a large number of users and the data collected evaluates the design usability of the product. With Summative study, the development team can find out what percentage of users had trouble performing a certain action on the platform. This statistical data helps them to redesign the platform according to the user needs.
Even though both Qualitative and Quantitative data are collected to determine whether the design elements are working or not and how they can be improved, the different usability tests are conducted differently so that they answer to different questions.
When Is It Used?
Even though both tests are performed to understand and improve the usability of the design, they are performed on different stages of the development process.
Formative or Qualitative tests are performed during the initial stages of design and development. It is most effective when the developers run formative testing as they are designing the product to see whether the primary design elements they are using are working out or not. Formative tests are also performed when they have a Minimum Viable Product as well. With the initial Formative testing, developers can launch a product with a functional design in the market.
Summative or Quantitative studies are performed when the product is already completed and launched in the market. This process is more about the evaluation of the design than improving the design. With Quantitative design testing, the developers aim to know whether the design elements they approved of through formative user testing are working out for the larger audience out there. Summative testing can be a costly affair, and that’s why so many enterprises out there are hesitant about performing this test on their online platforms. But quantitative data are of serious statistical significance and provides some serious insight into the effectiveness of design on the targeted user base.
What Goals Do They Complete?
Since there are major differences regarding the questions they answer and when they should be used, it goes without saying that these two testings would have serious differences when it comes to the goals they complete.
We have already discussed when these tests are used and what questions they answer. Based on that, the difference between the goals of Formative and Summative testings are pretty easy to understand.
A redesign process starts with the evaluation of the existing design, which is why in the above image, the first step of the redesign process is Summative. Once the design elements are evaluated and the data is collected, the evaluation of that data will reveal what is going wrong and where the users are struggling the most. Based on that we come to the second step, where we will perform qualitative testing to figure out the new design elements to be added to the platform.
Once the platform is redesigned, the two testings are repeated to ensure the new design works well. This process works well to clarify the goals of Summative and Formative testings. While summative helps to collect an extensive amount of data and evaluate the design, formative testing helps to quickly collect data during the design process.
Difference Of Methodology
The methodology of the two kinds of usability testing is what makes them so different from one another. Even though from the outside, both study deal with user experience, it is how they are collected that makes all the difference.
The methodology of both testings is standing upon three pillars- the number of participants, the flexibility of study conditions, and the protocol used.
For Qualitative or Formative testings, the number of participants is limited, most not preferring to go over the “magic number” five. The study conditions are extremely flexible, there is no need to strictly maintain a singular kind of testing atmosphere during Formative testings. The data is collected through a think-aloud protocol, wherein the users when interacting with a product or service during the testing process give their feedback verbally..
Summative testings have more than 20 users performing a single task. The study conditions are strictly maintained and each session of summative testing happens in the same predetermined atmosphere deemed fitting by the development team. The data collected through Quantitative testing is statistical, and usually, there is no think aloud protocol involved . Data collected through summative testing are much more precise than formative user testing.
What Are The Outcomes?
Qualitative data helps the developer to find the strengths and weaknesses of the design. However useful this data might be, it is still not concrete. The outcomes of Formative testing is the result of an irregular test done among a small number of people who may not even be able to represent the whole user demographic out there. On top of that, the results of Qualitative testing are based on the expertise and knowledge of the developers and how they interpret the user’s struggle with the UI elements. Which is why the Formative testing data might be extremely useful during the initial stages of design, but they are still not very reliable.
The quantitative study, however, has some concrete outcomes. The process involves a large number of users, which does a better job of representing an entire demographic than that of Formative user testing. The data is not collected arbitrarily, rather through statistical methods, which helps the developer team to understand the error margin of their design and where most people are struggling. This way they can take better action against lack of design usability on their platform.
Conclusion
Even though they are widely different, Quantitative and Qualitative testing are complementary to each other. Formative or Qual studies offer a direct view into why the design is failing, and Summative or Quant studies offer rather solid, yet indirect insight into the reasons for design failure and when the collected data is analyzed, they offer information sans the noise that can render the entire testing to be useless. And that’s why both kinds of user testing are needed to make the design flawless.
Want to learn more?
If you’d like to…
· learn all the details of Usability Testing
· get easy-to-use templates
· learn how to properly quantify the usability of a system/service/product/app/etc
· learn how to communicate the result to your management
… then consider to take the online course Conducting Usability Testing.
If, on the other hand, you want to brush up on the basics of UX and Usability, then consider to take the online course on User Experience. Good luck on your learning journey!
