Moderated vs. Unmoderated Online Studies

In the face of a remote working world, how can we decide which tool in our belt best fits the needs of our clients and the restrictions of our users? There are many third-party remote usability testing suites that specialize in moderated and unmoderated online studies with all promising the ability to deliver quick insights at a lower cost than using traditional methods. Many promise to quickly recruit users, run them through a variety of usability scenarios, establish benchmarks, conduct large-scale user research and many more options. While many of these tools seem to reach a high level of convenience, how do you know which one is the correct one for your project? When would you be interested in an unmoderated online tool instead of moderated online sessions?

When determining the direction and scope of your study to decide upon unmoderated online vs. moderated online, you have to be confident with your goals and what you are hoping to discover through the sessions. By creating specific goal outputs and clarity in quantifying potential answers, you are able to decide and make an educated decision on using the best tool.

Questions to answer prior to deciding upon a tool may include:
- What problem questions are we looking to answer during this study?
- What are the common problems users face when they visit our website?
- Is it easier for users to find this control if we put it at the top left of the screen or the top right?

It is important to answer questions at a variety of levels including higher-level questions about overall goals to tactical questions regarding precise goals to define a better usability understanding. Regardless of the moderated and unmoderated study, if the higher-level questions that define what success looks like are not aligned with overall goals, then the study will not be successful regardless of results. Moving into precise questions that will help establish the guidance or open-ended questions that will be included in the study will separate moderated questions from unmoderated questions. Online unmoderated studies are excellent at evaluating clearly-defined computer-related tasks that participants can perform without deeper inquiry or guidance. Tools associated with unmoderated studies can recruit and complete study sessions much faster than even a team of researchers. They are very efficient as they can run in parallel and outside of standard hours rather than a series of scheduling actions during standard hours. A downside of unmoderated sessions in a faster timeline is that if you were looking to adopt changes mid-session due to unforeseen circumstances, stakeholder requests, or evolving interests, it may be too late and all the sessions may be completed. Unmoderated online studies also require a certain level of technical savviness from participants that are more comfortable and able to use their computer/mobile devices to perform the activities. They may not be well-suited for hyper-specialized user groups that are harder to recruit, require assistance, or coaxing.

The primary difference between unmoderated and moderated online studies is researcher-participant interaction. This is the same as the primary difference between unmoderated online studies and moderated on-site studies. This means that unmoderated interactions are more of a rigid structure. Limited instructions are seen by the participants and the questions they answer are scripted. Any type of study where a participant might require additional guidance either through a not fully working prototype to an open-ended discussion can be recommended to be done through a moderated study. And in accordance with the current state of affairs, recommended being done remotely online. Importantly is that in a moderated online study, you can identify those moments of struggle, encourage participants, and learn from those moments to make truly meaningful improvements in a process or product.

As mentioned earlier, moderated studies excel in environments in which the researcher - participants relationship can answer dynamic questions and together discover additional information.

Questions that may be answered when aligning on a moderated online study:
- In a perfect interaction, what do people want to get out of an interaction with a product we could make?
- Is our understanding and interpretation of the user needs the same as what participants are saying would be most impactful to them?
- Any question that begins with ‘Why’

With a growing number of remote tools with a growing number of dashboards and exportable spreadsheets, user researchers are still critical to the process of review sessions, analyzing the information, and synthesizing it into actionable and impactful insights. The work following any type of moderated or unmoderated online study is what leads to its ultimate success for the client. While the challenge of determining the best method and tool is at first the point of discussion, Key Lime Interactive has a strong knowledge and historical understanding of using the best tools to find the best results.