Methodology Spotlight: Online Diaries

Screen Shot 2018-11-13 at 2.11.04 PM Online diaries are a unique way to learn about respondent experiences in natural settings and allow researchers to gather and collect situational and qualitative feedback. In an online diary, users are given missions or tasks and just like a diary asked to log in entries. If you are interested to learn how customers go about considering and researching a product for purchase, or the usability of a piece of hardware over a longer period of time, then an online diary allows respondents to keep track of what they are doing, sites they are visiting, how they are using the products, as well as their overall impressions and satisfaction. Also, it incorporates time into research and helps get a more comprehensive perspective (e.g., what resources or sites are considered as part of the decision making journey or how is the usability of a product integrate within a user's’ lifestyle).

While usability labs, in-depth interviews, and surveys often form the core of a UX researchers’ repertoire, these have their limits in how they can be used to answer some research questions. In-person labs happen typically within a controlled setting and do not capture what happens outside of a testing room. On the other side, surveys that can be conducted while users are sitting naturally at their computer or phone are less qualitative and, in a way less personal. Online diaries can both gather qualitative feedback but still have users be in natural environments. A common setup for a usability would include an on-boarding in-person or remote sessions, followed by the journals / tracking activity, and then a debrief in-person or remote session. This flow lets respondents to document their experience over the course of the study and do so in whichever natural environment fits the mission or task.

Online diaries have distinct benefits:

Data can be gathered in various ways – Online diary tools offer a lot of different question formats with distinct advantages for users and researchers. For researchers, incorporating photos, videos clips, scaled questions, open text responses allow for more diverse data sources and types of information that can be observed and shared. On the flipside, for participants, it allows them to have more fun and be more creative than the standard written journal description.
Entries can be done at the time when respondents are completing a task - With several platforms offering mobile options, it is more available for respondents to capture events closer to when they occur or in situ, yielding reactions that are more immediate and fresh in their minds, rather than having users to recall details from a past event that can get manipulated with time.
It takes longitudinal factors into account with the experience – One of the key benefits of allowing respondents to use a product out “in the wild” is that external factors that may impact the overall experience can be factored in with an online diary. This gives a more comprehensive and accurate understanding of what happens. Also, more interestingly in addition to any external factors, learnability and familiarity with a product, website, or app can impact users’ impressions. While in usability sessions allow for these two factors to be absorbed, it is limited to what can be learned in a short interview, and both factors extend beyond the in-person session. For instance, when researching a product, terms and concepts may be more recognizable with the 3rd or 4th site that is visited vs. the first one.

Yet, there are things to be mindful of:

As with all methods, each version has its advantages and drawbacks. Specifically, with online diary studies, respondents and the researcher are not in the same room but are interacting over a longer period. Here are a few that we have come across during our more recent online diary studies.

It takes more planning – Setting up an online diary requires managing more logistics compared to the equivalent usability lab, as well as more guidance for how the tool and the study works. If there are multiple in-person sessions with the respondents, this may require more coordination and planning. If respondents are completing different missions (e.g., completing a couple of days of natural use tasks, followed by prompted tasks), ensuring that each respondent goes through a similar experience, even if they start at different times is helpful for synthesizing and comparing across participants.

Online diaries may require more troubleshooting – Since the respondents and the researcher are not together, then depending on what the research is (e.g., testing out hardware, or a prototype app), more interactive communication is needed to make sure that respondents are able to use the prototype. This helps to minimize artificially low satisfaction with the user experience because the prototype is not working properly.
Keeping participants motivated – Invariably, there will be respondents who will be very proactive, and others may not have the time or interest to log entries. In these cases, setting up expectations in the beginning, following up and asking respondents to log in, providing specific questions for them to answer, or establishing a “minimum” number of entries to complete, may help keep users going and log the data that you need.

Account for possible drop off – And, still there will be drop off and respondents may stop participating. Often the most direct way to account for this is to over recruit, and bear in mind that up to 20% may drop off during the study.

In summary:

An online diary is a creative and versatile method that allows researchers to gain a more complete understanding of a respondent’s experience, while still be able to delve into some nuances of the experience. With variety in the types of data that is gathered, the opportunity to have users experience a process or product over time, and in environments where they would look into a product, it is a tool that offers researcher to look into how different aspects impact the experience (specifically, time and environment) that are not always available in lab methods. Furthermore, being aware of the need for project management and interactive troubleshooting, as well as the possible decrease or lack in participation, will help run a successful study.