The topic of unmoderated studies stands out to me because it was how I was introduced to the UX research world, and what I did almost exclusively working at my first agency as a UX Researcher. It is a topic that’s near and dear to my heart and taught me many things throughout the years I have been running studies. It is not always the most in-depth study to facilitate, but if a client has a specific problem or question to address, the researcher can get user feedback and real-life perspectives within days. I noticed that this method can be very effective for certain projects and research objectives and goals, and wasn’t necessarily very well known among my peers. I thought I’d write about it to raise awareness of the pros and cons of unmoderated studies.
In this post, I’ll define what an unmoderated study is when it should be used and the benefits and disadvantages of this approach. I will also give you the 6 crucial steps to set up an unmoderated study.
An unmoderated study is a form of research that uses specialized research tools to capture participants' behaviors and attitudes, by giving them goals or scenarios to accomplish with a site or prototype. The participant completes the tasks on their own time without any interaction from the researcher. Usually done with software (like UserZoom, Usertesting, etc) In my experience, they have been typically shorter in length (30 mins or less) to minimize participant fatigue and have a higher sample size (usually 10+ participants).
This approach works best for live websites and apps or highly functional prototypes. They are appropriate for studying activities that don’t require a lot of imagination or emotion from participants (i.e., giving them a very specific task instead of a vague hypothetical situation). Unmoderated studies have focused research objectives and can answer specific questions that guide you and/or your client towards a solution or support a specific decision or change.
I have encountered many benefits while conducting unmoderated studies like being able to gain insights much quicker than a moderated study and having real feedback from participants around the world within hours. Because these are done through a software platform and recorded, they do not require a researcher to attend and participants can complete them on their own time (alleviating time zone constraints). Many software platforms offer an end-to-end solution for the researcher - they can recruit through the platform, perform note-taking, tag themes, create highlight reels and share videos with stakeholders all in one place.
There are also some shortfalls when it comes to an unmoderated study. Software tools can become very costly and sometimes require an annual subscription. Unlike moderated studies, much of the work is done upfront (screener development, test plan, implementation into software tool, and analysis planning). Tasks and questions need to be very specific because once the study is launched it cannot be changed and follow-up questions cannot be asked (like in a live session with a moderator). Using an early prototype in an unmoderated session could be troublesome to the participant because they’d have to use a lot of imagination and some features/functions may not work. Because the researcher is giving the participant-specific tasks (that may not be a situation that they’d face), it’s common that the participant can be less engaged or behave less realistically.
Source: Unmoderated User Tests from Nielsen Norman Group
Over the years I realized unmoderated studies might be a quick and dirty approach to getting insights. The method is probably not always the best fit for every project. There are definitely other methods that can get better insights. But, in addition to providing quick, actionable insights for my clients, running many of these studies and having to think through these studies from start to finish, they have taught me how to be a better researcher. They helped me understand and think through a screener, test plan/guide, and analysis. They taught me the fundamentals of user experience research and ultimately helped me ask better questions to get the richest insights from my audience.
Reach out to us at keylimeinteractive.com/contact to learn more about how we can help you with unmoderated studies!