Unmoderated Studies - When to be hands-off in research?

The topic of unmoderated studies stands out to me because it was how I was introduced to the UX research world, and what I did almost exclusively working at my first agency as a UX Researcher. It is a topic that’s near and dear to my heart and taught me many things throughout the years I have been running studies. It is not always the most in-depth study to facilitate, but if a client has a specific problem or question to address, the researcher can get user feedback and real-life perspectives within days. I noticed that this method can be very effective for certain projects and research objectives and goals, and wasn’t necessarily very well known among my peers. I thought I’d write about it to raise awareness of the pros and cons of unmoderated studies.

In this post, I’ll define what an unmoderated study is when it should be used and the benefits and disadvantages of this approach. I will also give you the 6 crucial steps to set up an unmoderated study.

What is an unmoderated study (and Why Does it Matter)?

An unmoderated study is a form of research that uses specialized research tools to capture participants' behaviors and attitudes, by giving them goals or scenarios to accomplish with a site or prototype. The participant completes the tasks on their own time without any interaction from the researcher. Usually done with software (like UserZoom, Usertesting, etc) In my experience, they have been typically shorter in length (30 mins or less) to minimize participant fatigue and have a higher sample size (usually 10+ participants).

This approach works best for live websites and apps or highly functional prototypes. They are appropriate for studying activities that don’t require a lot of imagination or emotion from participants (i.e., giving them a very specific task instead of a vague hypothetical situation). Unmoderated studies have focused research objectives and can answer specific questions that guide you and/or your client towards a solution or support a specific decision or change.

The Pros and Cons of Running an Unmoderated Study

I have encountered many benefits while conducting unmoderated studies like being able to gain insights much quicker than a moderated study and having real feedback from participants around the world within hours. Because these are done through a software platform and recorded, they do not require a researcher to attend and participants can complete them on their own time (alleviating time zone constraints). Many software platforms offer an end-to-end solution for the researcher - they can recruit through the platform, perform note-taking, tag themes, create highlight reels and share videos with stakeholders all in one place.

There are also some shortfalls when it comes to an unmoderated study. Software tools can become very costly and sometimes require an annual subscription. Unlike moderated studies, much of the work is done upfront (screener development, test plan, implementation into software tool, and analysis planning). Tasks and questions need to be very specific because once the study is launched it cannot be changed and follow-up questions cannot be asked (like in a live session with a moderator). Using an early prototype in an unmoderated session could be troublesome to the participant because they’d have to use a lot of imagination and some features/functions may not work. Because the researcher is giving the participant-specific tasks (that may not be a situation that they’d face), it’s common that the participant can be less engaged or behave less realistically.

6 Steps to Set Up an Unmoderated Study

Define the goals of your study:
The first step toward gathering useful feedback is setting a clear and focused objective. If you don’t know exactly what sort of information you’re looking to obtain before you conduct your research, you risk launching a study that fails to yield actionable insights.
Select software that fits you/your clients’ needs
Identify what you or your client wants from their study. Do they need videos, highlight reels, transcriptions, specific demographics, time on task metrics, anything fancy, etc.?
Identify needs and choose the best tool that fits within the budget.
Write tasks and questions
Your study guide is the list of instructions that your participants will follow, the tasks they’ll complete, and the questions they’ll answer during the study.
1. What is a task?
  An action or activity that you want a participant to complete.
  Example: Go through the checkout process as far as you can without actually making a purchase.
2. What is a question?
  Something that elicits verbal feedback from the study participant.
  Example: Was anything difficult or frustrating about this process?
Run a pilot test
For unmoderated sessions, this should be done before recruiting starts (if using a tool similar to Usertesting/UserZoom) because once you begin recruiting, you will launch the study at the same time. You want to make sure you test the flow of the study and the questions/tasks to ensure that the participant understands what is being asked and can provide valuable feedback to your questions. This can be done with usually 1 external participant that can be used towards the final count of your study.
Recruit Participants
Recruit participants through software tools or an outside source with specific demographics and screener questions for your project. Once you find the right participants, you can launch the study at the same time.
Analyze data
Use the software tool to your advantage by utilizing features provided and not manually pulling data. Watch participant videos, take notes, pull quotes, insights and themes, and make any clips to analyze the data. Many tools have built in functionality to pull insights for you on the demographic information, time on task information, etc.

Source: Unmoderated User Tests from Nielsen Norman Group

Closing

Over the years I realized unmoderated studies might be a quick and dirty approach to getting insights. The method is probably not always the best fit for every project. There are definitely other methods that can get better insights. But, in addition to providing quick, actionable insights for my clients, running many of these studies and having to think through these studies from start to finish, they have taught me how to be a better researcher. They helped me understand and think through a screener, test plan/guide, and analysis. They taught me the fundamentals of user experience research and ultimately helped me ask better questions to get the richest insights from my audience.

Reach out to us at keylimeinteractive.com/contact to learn more about how we can help you with unmoderated studies!