Photo by Ahmed Zayan on Unsplash
Moderated usability testing is probably the most commonly used design research technique, but I have ambivalent feelings about it. It’s definitely a powerful technique when used appropriately, but it is also the most overused, inconsistently applied and poorly interpreted approach to testing products and prototypes.
In simple terms, usability testing means getting people to complete a set of predefined tasks with a live or prototype system. ‘Moderated’ means that a researcher is guiding them through the process either by physically sitting with them or communicating via video-conferencing or a phone call (known as remote testing). Task completion metrics and observations of behaviour are usually recorded and often complemented with a post-completion interview.
Doing it:
In addition to recruiting participants, you’ll also need to identify a set of tasks that they will attempt and criteria for intervention when participants get stuck or wander off-task. The tasks themselves should be both representative of the service/product you're testing and designed to ensure that your findings are actionable; there is little point testing aspects of the system that you don’t have control over unless they are necessary to provide context for the parts of the system that you are able to influence.
What information you will capture during the session is also important. This can include measures of efficiency (time on task, error rate etc) and effectiveness (completion rate, level of success etc) as well as their subjective experience of the system. A common way of making sure you understand the participants’ experiences during the tasks is to use a ‘think out loud' protocol, which simply means asking them to talk about what they are doing and why, while they complete the tasks.
Key Advantages:
-
Quantifying usability - as a relatively tightly controlled method, moderated usability testing is useful for creating metrics to compare options and versions of a service in terms of usability.
-
Collect qualitative and quantitative data at the same time.
-
Clips of usability are a powerful tool for communicating with stakeholders - a few seconds of video footage will often be more effective than dozens of graphs and tables of stats.
-
By pre-defining the tasks and being on-hand to guide participants you can keep the session focused on exactly the parts of a service/product you are interested in.
Key Constraints:
-
Sessions are held in an artificial context, which can mean you overlook important contextual influences such as technology familiarity, environment effects and social factors.
-
The cost of running these sessions often means that you're usually not going to be able to run enough sessions to run meaningful inferential statistics.
-
Recording, running and analysing sessions is extremely time consuming, so can be overkill for early design ideas or low value services.
-
Researching with participants who are busy, hard to access or live in different time zones can be difficult.
-
Complex and easy to get it wrong. Inappropriate tasks, incorrect participants, poor instructions, bad choice of tasks, collecting the wrong data or not selecting the right methods for analysis can all have a big impact on the accuracy and usefulness of your findings.
Find out more:
Running remote moderated usability testing
How many participants is enough for usability testing
Finally, if there are any Research Methods in particular you'd be interested in, please just let us know and we may feature them in the up-coming series.
