How COVID-19 Affects User Research

COVID-19 has already drastically changed the way Americans go about our daily lives. Some people are out of a job, others are living paycheck to paycheck, and others are more fortunate to be able to work from home without as much disruption to their socioeconomic wellbeing. In what is essentially the world’s largest work-from-home experiment, COVID-19 will undoubtedly affect the way we conduct user research. As companies around the country try to adapt to rapidly changing circumstances, we at Key Lime Interactive are preparing our researchers to continue providing quality research, while taking into account how the stress and anxiety around COVID-19 can cause bias in the process.

Last week, we discussed different considerations for switching to remote research during this time. But how will this switch change user behavior? How will this affect response rates? How will this affect the way we design our studies? What type of studies should we run? It’s impossible to predict how the pandemic will truly affect user research, but we try to tackle some of these questions below.

What types of studies should we avoid?

The obvious answer here is to refrain from any in-person studies for the time being and to stick to remote research. The anxiety and fear of contracting the virus may negatively affect the participant’s state of mind, distracting them from the study. It also puts both the researcher and the participant at risk. Global guidelines right now are to practice social distancing to prevent the spread of the disease. For products that can’t be tested remotely, it may be better to wait until the fear dissipates to resume testing.

Unmoderated studies and surveys may also yield mixed results. It’s already hard to control for environmental or situational factors when conducting unmoderated studies, but global panic has complicated things further by forcing people into new situations and behavioral patterns. Their feedback may not approximate their “typical” behavior and can skew results. A large portion of the country now finds themselves working from home. That means a different daily routine for many Americans. This makes controlling for external factors even harder since we won’t know how people will alter their behaviors in order to adjust. There’s also no way to predict how this will affect the workforce even after the virus has been contained.

In the meantime, we stress the importance of having a skilled moderator that can effectively steer the conversation away from COVID-19, keep participants on task, and help pull the right insights. The advantage of moderated studies is the ability to untangle the 'why' of particular answers.

Will participants be more or less likely to want to participate in user research?

We have to take into consideration how the stress and anxiety from the virus will affect study invite response rates. Will people even check their personal emails right now, let alone fill out a screener form? Will people be too worried about their own well-being to care about participating in studies? This is something worth monitoring as you send out your initial rounds of invites.

As more companies implement work-from-home policies, that can open up a potential pool of participants that might otherwise find it hard to take time off from work to participate in user studies. Some work in an open-office setting where they can’t get easy access to an empty, quiet space. Others fear repercussions from their employer if they step away for a study. Work-from-home makes user studies more accessible to some people by allowing them to participate in the comfort of their own homes. On the other hand, as we may have already noticed in our video conference calls, a lot of folks now have to take care of their kids at home, which could also present a challenge in getting the participants to find a quiet place to do the study.

The reduction in human interaction because of social distancing can contribute to more people being open to using technology to connect and talk about something other than COVID-19. Maybe participants will appreciate a change in the topic to disrupt their current routines. People are already reaching out for human connections via various video tools. Maybe this will encourage the less tech-savvy to try something new, like giving feedback on products.

Certain industries like healthcare, food retail, and delivery services are impacted heavily by the virus. Be considerate of industries where employees have to work extra hours during this time. Some employees in these industries may not have the time or may be too stressed to participate in research studies.

It’s currently unclear how to study participation will be affected by COVID-19, but as you continue to run studies, take note of what works and what doesn’t work.

How will this affect how we recruit participants?

Inversely, some people who are out of a job or are living paycheck to paycheck during this period may turn to participate in user studies online as an extra source of money. That means we have to be even more diligent as issues of screening and data validation will come into play. While there may be plenty of new quality participants, there will also be those who are in it just for the compensation. It may be worth spending a little more effort than usual screening participants to make sure they are a good fit for the study. If you have the luxury of time, get on a brief phone call with participants to gauge their interest, and fit for the study. If they spend a disproportionate amount of time asking about the incentives or talking about what they would buy with the money, chances are they might just be in it for the incentives and say just enough to get the reward but are not actually interested in providing you with relevant feedback.

How will this affect bias in our research?

The fundamental attribution error is the tendency of people to overemphasize personal characteristics and ignore situational factors when judging others’ behavior. This is especially important since COVID-19 will surely have some sort of effect on user behavior. If a participant is frustrated when working through prototypes, how much of the frustration can we attribute to the actual prototype, and how much is carried over from how COVID-19 has affected their lives.

Over the next few months, we need to gain an understanding of how COVID-19 affects participants’ day-to-day life. We should try to understand the broader context and how participants' responses may be affected by this. “When we conduct research, we often ask users to dig into their past to find examples and anecdotal pieces of evidence. And often when we dig deep into the why’s, we hear several reasons about how they faced certain difficulties/took certain actions.” The changes in a participant’s behavior and lifestyle caused by COVID-19 can drastically skew their perception of past anecdotal evidence.

Be careful not to let our own experiences with COVID-19 affect our research. Some of us may be fortunate enough to work in companies that allow us to work from home or can offer more resources during this time. The participants we talk to may be going through different experiences. It’s important that we put our assumptions and personal experiences from the fallout of COVID-19 to the side when conducting interviews with participants. We also have to remain impartial and not let any frustration or anxiety from our regular lives seep into our conversations with participants. They can read our tone and body language just as well as we read theirs.

How can we better prepare ourselves for studies?

If response and participation rates are low during this time, there are a few actions you can take. Consider tailoring your study invite messages to sound more considerate when asking for participants’ time and energy. Avoid using template messages because those can get easily lost in the sea of emails that participants are currently receiving regarding COVID-19. Another option is to bump up the incentive amount that participants will receive. It may encourage participation, as long as you do a careful job of screening out those who are just in it for the money.

The flip side of conducting remote sessions means studies are more prone to technical issues. Every researcher knows the pain of conducting a session with shaky wifi. To combat this, provide more lead time for participants to install and get acquainted with any software or apps before the study even begins. Don’t waste precious minutes during the study doing technical troubleshooting. It can frustrate participants from the start and bias their feedback. Be proactive and get on a quick call with participants to resolve all tech questions a day or two before the study begins. On the researcher's side, make sure that all the tools and equipment needed for the study are set up beforehand. If you are exploring new tools for remote testing, make sure to thoroughly test them internally before you start testing.

Consider providing a little more time at the start of sessions to get the participant acclimated to the study environment and to allow you to mitigate any potential environmental or situational distractions. Time is a precious resource when it comes to research, but taking a bit of extra time to reassure both sides can help the participant feel more comfortable during the study.

The current political and socio-economic climate is sure to affect everyone in different ways, but as researchers, we should be extra conscientious of how we present ourselves in the sessions. We don’t want to carry any stress we may be feeling into our personal lives over to the participants. They will feed off of your energy. It may be a good idea to have discussions within your team to prepare for any possible questions or concerns that participants may have about COVID-19.