IA Issues? What do you do When Online Card Sorting is Not Enough?

Understanding how your users think about the organization of content.

If your users can’t find the information that they are seeking, it might as well not be there at all. An intuitive information architecture (IA) is a core part of a user’s experience, but how do you know what would make sense to them?

Card sorting is a technique that can help gain insights into how your users think about the organization of your content. This user research method can be performed using an online tool or in-person using physical cards.

The benefits of online card sorting
Online card sorting has become a very convenient and common way to collect this information from users. The benefits of online card sorting include:

Fast and easy data collection – participants can log into a website and perform the activity at their convenience.
Large sample size – it is easy to obtain a large sample size for statistical analysis.

Online card sorting has become the predominant way for user researchers to collect this type of data, but it can’t tell you everything you need to know about how to design an intuitive information architecture.

Online card sorting tells you what, but not why
What do you do when online card sorting is not enough? Collecting data on how your user's sort items into categories tell you what they did, but not why they did it. The reality is that designing information architecture is a messy job, and card sorting can sometimes lead to inconclusive results, especially if the items are particularly challenging for participants to sort.

Qualitative survey questions at the end of an online card are useful, but can’t really help you to get a full understanding of the user’s mental model. There is no way to probe deeper, and participants often give minimal responses to these questions.

Gaining deeper insights with moderated in-person card sorting
Moderated in-person card sort sessions are typically scheduled for an hour and are conducted one-on-one with a facilitator. Participants are provided with physical materials that include cards labeled with the name of each item to be sorted, pens or markers for making notes, and sometimes color-coded stickers for further annotations.

"Image from chapter 6, Information Architecture & Web Navigation, Eye Tracking in User Experience Design."

The ability to manipulate physical cards provides many benefits to the participants including:

Easier to complete – physically sorting cards into groups require less instruction compared with learning how to use a new online tool.
More flexibility – participants can put aside cards on a table and spread out as much as they need. They can write on the cards, cross things out, make suggestions, and draw out connections between cards, including cross-linking.
Higher engagement – participants tend to be more engaged when they are physically moving cards around. They are also less likely to be distracted compared with performing the activity remotely on their own.

The ability to sit with a participant as they sort cards provides many benefits to the researcher including:

Understand participant’s thought process – think-aloud protocol can be used to have the participant explain why they are sorting cards in a certain way.
Observe nonverbal responses – do they appear confused or frustrated? Do they visibly hesitate to place a card or create a label for a category?
Provide motivation – card sorting is a laborious and mentally demanding task. Participants often lose motivation to complete the activity. The facilitator can provide encouragement to the participant to keep going.

Moderated in-person card sorting is more time-consuming both in data collection and analysis than in online studies. However, tools are available to reduce the time needed to analyze data.

Bar code scanner – bar codes can be used to tag each card and can then be scanned into a computer.
Utilize software tools – the same analysis tools can often be used to analyze the results from an in-person card sort.

Combining the best of both methods

A hybrid approach that includes online and offline card sort activities will provide a more holistic understanding of how your users envision an information architecture. In this approach, it is recommended to start with an online open card sort study to see general trends in how users to sort items. Next, follow up with an in-person study with a smaller sample size to understand a participants’ thought process including insights into how they would use the content.