How to Plan for a Quantitative UX Competitive Benchmark Study

It’s important to start by building a strong foundation for your benchmarking study. This is not a one-off project. You’re designing it once, to run multiple times. So, before you jump in, first make sure you take the time for proper planning by asking and answering some essential questions, like how to plan for a quantitative UX competitive benchmark study.

A good place to start: Why? What? Who? How?

WHY benchmark yourself?

Benchmarking creates a baseline for understanding the current user experience (UX) on your website or app.

√ Provides a starting point to measure against in the future.
√ Helps deliver insights and establish goals to guide you where to focus efforts to improve and optimize the user experience.

However, once is not enough! Keep testing to track how the experience changes over time.

√ Compare how design iterations impact the user experience.
√ Track and measure changes in key performance indicators over time.
√ Keep abreast of changes in the needs and expectations of your users.
√ Conduct benchmark studies at regular intervals (e.g. every year, bi-annually, or quarterly).
√ Retest after rolling out major updates or redesigns.

**WHY benchmark your competitors, too?**

Don’t miss out on the many benefits of expanding your testing universe beyond your own site or app.

√ Competitive benchmarking provides valuable measurement of how you perform relative to your competitors.
√ Competitive analysis offers you a way to interpret your usability standing, feature set, and more in relation to the competition.
√ By benchmarking against your competition, you have the added benefit of learning from their successes and failures; gaining insight into what works and doesn’t work and why.
√ Continually track competitors’ changes—see whether they are making the user experience better or worse, and why.
√ Identify best-in-class examples to emulate.
√ Which competitors should you include? The ones you look at and discuss the most, the ones you feel you’re falling behind, and the ones that you think you can learn the most from.

WHAT are your goals and objectives?

Take the time up front to carefully identify the questions, concerns, areas of interest, and purpose of the research. Understand business concerns; review previous knowledge; develop testable hypotheses.

√ Be clear and specific when you define your benchmarking goals.
√ Include specific research questions and hypothesis you have.
√ Address problems you or your customers have identified.
√ Collect the data you need to assist in making sound design decisions.
√ Design your tasks and questions to get the answer you need and will confirm or refute your hypothesis.
√ Lastly, stay focused: don’t set too many goals or ask questions that are not directly tied back to helping answer your top questions.

WHO should the participants be?

The greatest insights are derived from collecting feedback from real users. But, you need more than just real users; you need your users and/or intended target audience.

√ Recruit participants who match your demographics (or better yet personas, if you have them).
√ Create a screener survey that will qualify or disqualify participants based on whether or not they meet your pre-defined criteria (e.g., age, previous experience, or other demographics, traits, or attitudes).
√ There are a number of different options for recruiting. Using a professional panel partner is a great way to find people who meet your specific criteria. Or if it’s your existing customers or site visitors you want, recruiting directly from a customer list or online intercept surveys may be good options to consider.
√ Keep the proportion even across all competitors. Otherwise, your competitive data will be skewed.
√ Use the same screener each time you run your benchmarking study. Consistency is imperative to yield reliable longitudinal data.

HOW many participants do I need?

Benchmarking studies require a larger sample size to get statistically significant and sound results. Consequentially, these studies require more time and recruiting resources than traditional usability studies. Allow for extra recruiting and fielding time in your schedule.

√ Need samples large enough to make projections to the population.
√ A “significant” difference is one that we are confident we didn’t find by chance alone.
√ Confidence increases with sample size and the accuracy of our measures.
√ For between-subjects benchmarking studies as a rule of thumb, sample sizes between 150n-300n per site are recommended. With these sample sizes differences of approximately 7%-10% are significant at a 90% confidence level.
√ Before launching your study, run the analysis using a statistical significance calculator to see if your sample size is sufficient to detect a meaningful difference.
√ An additional consideration is the inclusion of multiple segments or groups. If you are going to run any segmentation analysis (comparing two or more different groups such as customers and prospects) you will need to increase your sample size accordingly.

HOW do I collect the benchmarking data?

Once you’ve identified what you want to test and learn and whom you want to recruit to participate, select the best strategy and tools for conducting your benchmark study. Most benchmarking studies involve using an online application to collect the data. The tool and method you select will depend on a few factors including: what type of data and metrics you want to track, who your target market is, as well as how much time and budget you have.

√ There are many tools out there. Choose the right tool for conducting and analyzing your study. For instance, do you want users to complete tasks on the sites? Do you want to collect clickstream data to understand where users went to complete tasks? Would a virtual ‘card sort’ be helpful in answering your research questions?
√ Using online applications also save time by making it easy to repeat/ re-run the study in the future. Design once, run multiple times.

HOW do I analyze the benchmarking data?

Work smarter, not harder. If you took the time up front to carefully identify the questions, concerns, areas of interest, and purpose of the research and designed your study with data analysis in mind, then it will be a breeze.

√ Pre-planned analyses (know what you will do with each piece of data you collect).
√ No “ad-hoc” analyses.
√ See if your hypotheses were confirmed or debunked.
√ Synthesize findings into insights.
√ Present comparative analysis of key performance indicators and metrics.
√ Identify best-in-class examples.
√ Include qualitative feedback to support the “why” behind the results.
√ Think through implications/recommendations/next steps.

We conduct UX Competitive Intelligence studies for clients in many industries including financial services, retail, travel, healthcare, and more. The best way to see how our competitive reports can benefit you is to go straight to the source. Contact our report authors now.