The effect on the User Experience (UX) brought about by a newly-implemented product change is something that is both necessary yet difficult to measure. This is because changes in the user experience tend to be generic in nature and may not directly relate to the quality of the UX or the overall goals of the project. At the same time the ability to measure this effect will help you take decisions related to future product modifications.
This article discusses how the HEART framework can be applied in conjunction with the Goals-Signal-Metrics process in order to establish which metrics can be used to measure the UX of a product. It will also discuss how we have applied this combination when choosing UX metrics for our product, Sales Manager – an application that supports sales on Allegro.
The HEART Framework
HEART is a framework that identifies the quality of the user experience. It stands for Happiness, Engagement, Adoption, Retention and Task Success. All of these are actually user metrics that cover key product aspects. HEART was conceived by Kerry Rodden, Hillary Hutchinson and Xin Fu to address the need for defining large-scale user-centred attitudinal and behavioural metrics and was initially applied to analyzing user experience related to websites and web applications. Owing to the framework, the interpretation of introduced modifications to a product is clear.
Let us now take a look at each metric of the HEART Framwework:
- Happiness: provides information about attitude and user feelings. It tells you more about user satisfaction level, visual appeal, likelihood to recommend or perceived ease of use. A well-designed survey launched after each modification will help you evaluate the impact on users.
- Engagement: describes user involvement by determining interaction frequency and intensity within a given period of time. The rate of involvement may include, for example, the amount of content shared on a social networking site during a day per user or the average number of site visits within last 7 days.
- Adoption: indicates the number of new users who decided to use the product within a given period of time, e.g. they created an account or purchased the product.
- Retention: tells us how many users that registered during a certain period still use the product (it tells you how many users have renewed their subscription).
- Task Success: is related to a traditional measurement scale of user activity, such as effectiveness (e.g. time required to fill in a form), efficiency (e.g. percentage of completed tasks) or error rate (e.g. percentage of failed attempts to log on).
You can apply the HEART framework to a specific feature or to your whole product / project. You can choose to ignore some of the HEART categories if you feel that they do not make sense for your product. In fact, you will most likely end up choosing 1 or 2 categories using the key aspects of your product as the basis for the selection criteria. By focusing on these key aspects, you can take further decisions on the basis of adequate and precise information.
The Goals-Signals-Metrics Process
Each user metric from the HEART framework needs to be related to a goal. In the words of Rodden et al., the Goals-Signals-Metrics Process is “a simple process that steps teams through articulating the goals of a product or feature, then identifying signals that indicate success, and finally building specific metrics to track on a dashboard”.
Let us take a look at each of the words in bold in this statement:
- Goals: Before you set your framework, define product or feature goals concerning user experience. Goals may be related to tasks performed by users or problems they may encounter. If the scrum team members understand a given goal differently, it is worth taking into account these various perspectives in order to find a common solution. Feature goals do not have to match the product ones. However, it is definitely too early to worry about finding right metrics and goal signals.
- Signals: Define proper goal signals that will tell you, whether the goal is accomplished. It is a good moment to think about a method of data collection (e.g. surveys or event logs). Signals should closely correspond with respective goals, whereas their modification should reflect changes in user experience.
- Metrics: The last stage involves transforming signals into measurement scales, which you can observe for some time. Instead of raw data, use mean, fractions or percentage rates. Moreover, exclude traffic from automated sources and ensure that all user actions are logged. Make sure that your product can be compared with others, as it may be necessary to add metrics used by those products.
The next step is to plot the HEART categories against their Goals-Signal-Metrics. Google Ventures advise the use of a simple grid, such as the one below:
| Goals | Signals | Metrics | |
| Happiness | – | – | – |
| Engagement | – | – | – |
| Adoption | – | – | – |
| Retention | – | – | – |
| Task Success | – | – | – |
How we used HEART and the Goals-Signals-Metrics Process
We decided to try out the HEART framework for our product, Sales Manager – an application that supports sales on Allegro. The primary reason why we chose the HEART framwork is that we established that it will be able to provide us with constant access to data on how users use our application. Therefore, we would be able to monitor their behaviour. We would thus no longer be “blind” but we would be in a position where we know the impact of a newly-added feature has on users. We would know when to react and which aspect should be analysed.
Indeed, the framework provided us with large-scale metrics to measure user experience. Owing to HEART, we know how new features or interface modification have influenced users or their segment. We were also able to use metrics to compare and combine results collected with different research methods, e.g. usability tests. Combining data from these two sources will improve our decision-taking process.
When defining the HEART framework for our product, we focused on setting metrics for key areas related to drafting and listing offers. In terms of the HEART categories, these mapped as follows:
Happiness
Users must be happy with the process of drafting and listing offers. The most suitable goal signal is the level of user satisfaction. Metrics we apply to check the goal progress are mean and mode of ratings submitted by users.
Engagement
When defining engagement, we assume that new features facilitating drafting and listing of offers would increase the number of offers listed via Sales Manager. We want users to increase their sales conducted via our tool. Therefore, the number of offers listed via the application is our goal signal. To verify the progress of goal realisation, we compare the number of offers listed via the application within 30 days with the number of offers listed via the Allegro sale form.
Adoption and Retention
These metrics provide us with data on user behaviour. As a result, we can differentiate between new and experienced users. Both metrics are recommended when creating new products or redesigning the existing ones and both should become rather stable throughout the product’s life cycle, excluding seasonal trends and special cases. As some of our users list their offers using different channels, we want to improve their experience related to drafting and listing offers. As in case of Engagement, we assume that applied modifications should encourage users to list offers via our tool. The most relevant data provided by these metrics is the number of users who use the application to list offers.
With Adoption, we want new and current users to list their offers via Sales Manager. The expected goal signal is: increasing number of users that list their offers using the application. We check how many users decided to list an offer via the tool within 30 days compared to the number of all users.
With Retention, we want users to keep on listing offers via Sales Manager. The number of users who list their offers using the application tells us whether we go in the right direction. We check how many users from a group that listed an offer within last 2-3 months decided to list it again during the last 30 days.
Task Success
We want the listing process to be as smooth as possible. The signal we focus on is the number of errors while drafting and listing offers. The metric includes the number of displayed error messages compared to the number of offers to be listed. Therefore, we know how often users make mistakes, before they start selling and whether modifications decrease the number of errors.
Using the above reasoning, the UX metric selection table for our Sales Manager product would look as follows:
| Goals | Signals | Metrics | |
| Happiness | Users must be happy with the process of drafting and listing offers | The level of user satisfaction | Mean and mode of ratings submitted by users |
| Engagement | The new features must facilite drafting and listing of offers | The number of offers listed via Sales Manager | Compare the number of offers listed via Sales Manager within 30 days with the number of offers listed via the Allegro sale form |
| Adoption | New and current users must list their offers via Sales Manager | Increasing number of users who list their offers using Sales Manager | How many users decided to list an offer via Sales Manager within 30 days compared to the number of all users |
| Retention | Users must keep on listing offers via Sales Manager | The number of users who list their offers using Sales Manager | How many users from a group that listed an offer within last 2-3 months decided to list it again during the last 30 days |
| Task Success | The listing process must be as smooth as possible | The number of errors while drafting and listing offers | The number of displayed error messages compared to the number of offers to be listed |
Conclusions
The combination of the HEART framework with the Goals-Signals-Metrics process provides you with analytical solutions for monitoring and improving the user experience. Therefore, the framework complements traditional methods and research tools and offers you ways to measure large-scale user attitude and behaviour.
However, the framework cannot be the only source of information about a user. Nevertheless, metrics data and results obtained during traditional research give you the full image of user feelings and interaction.
Owing to the HEART framework you can notice results of implemented changes. Besides, it also improves the decision-making process. That is why identification of key product aspects and providing accurate analysis criteria is crucial.
In other words, setting the framework requires several iterations in order to find problematic areas and modify methods and tools used to test user activity. Nevertheless, it is worth indicating that the combination of goals related to user experience and proper metrics allows you to test hypotheses concerning further product development.
Want to learn more?
If you’d like to…
- get an industry-recognized Course Certificate in Usability Testing
- advance your career
- learn all the details of Usability Testing
- get easy-to-use templates
- learn how to properly quantify the usability of a system/service/product/app/etc
- learn how to communicate the result to your management
… then consider taking the online course Conducting Usability Testing.
If, on the other hand, you want to brush up on the basics of UX and Usability, then consider to take the online course on User Experience. Good luck on your learning journey!
(Lead image: Depositphotos)
