Several third-party remote usability testing suites specialize in hosting unmoderated studies, and all promise powerful insights delivered at a low-cost. The companies can quickly recruit users, then run them through usability scenarios, establish benchmarks, conduct large-scale user research and more. But just because you have such a tool at your disposal, does that mean it’s the right tool for your project? When would you want to opt for an unmoderated online test instead of moderated sessions?
To answer that question of whether to choose unmoderated online vs. moderated onsite studies, define your goals and decide what you hope to learn. The more specific and better defined your goals are, the easier it is to quantify potential answers, and the more likely a remote unmoderated tool may be useful. Examples of research questions where unmoderated studies work well:
The key words here are specific and defined. Online unmoderated studies excel at evaluating clearly-defined computer-related tasks that participants can perform without guidance. The tools can recruit and execute study sessions much faster since they can run in parallel instead of serial scheduling at a research facility. The flip side of the faster execution time is that it’s much harder to make changes mid-stream to adapt to unforeseen circumstances or stakeholder requests. Online unmoderated studies also require reasonably tech-savvy participants who are comfortable and able to use their computer or mobile phone to perform the activities. They may not be as well-suited for specialized user groups, such as certain patient populations, lower-literary adults, or children, where they might be harder to recruit, or more likely to need assistance, or both.
The lack of researcher-participant interaction means that an individual session is a closed system with a rigid structure. The only instructions the participant sees, the only questions they can answer, are what has already been scripted. Studies where the participant might need additional guidance during a session (i.e. testing a partially functional prototype with an incomplete, buggy interface) aren’t good candidates for unmoderated studies because there is a higher risk the participants might quit in frustration, or be unable to continue without a moderator to steer the session back on task. Participants struggling to complete tasks provide critical insights into the flaws of a product, and the greatest weakness of an unmoderated study is the inability to ask additional targeted questions during the session. Probing for more information about why something happened is crucial for assessing the root cause of an issue.
That is where moderated studies excel; their primary virtue is their dynamic nature. Studies requiring flexibility, close observation, the ability to provide guidance or probe for additional information, are best executed in a research facility with a moderator.
Examples of research questions where moderated studies are the best fit:
As powerful as the dashboards and exported spreadsheets of remote study tools are, a trained user researcher still needs to review the sessions and analyze the findings. This requires a fair amount of secondary work, coding and organizing the data to tell a meaningful story. Also, while many remote study tools enable the quick and easy sharing of video clips from individual sessions, currently none provide ways to anonymize video footage – a valid concern in a field where protection of the participant is paramount.
The ultimate value of the data gathered by these studies, remote and unmoderated or onsite and moderated, is determined by the skill and training of the individuals designing them and analyzing the results. The first challenge is to determine the best method to answer the research questions at hand. That is where the UX consultants at Key Lime Interactive come in – to help your team choose the right tool for the job.
READ MORE: Task Analysis: What It Is And Why It Matters, 5 Tips to Recruit and Onboard Participants for Remote Usability Studies , Outstanding UX Research Naturally Leads Us to UX Design, ROI of User Research, Strengths and Weaknesses of a True Intent Study