Before we dive into a discussion regarding the population size needed for usability research, we need to make one thing clear - lot’s of people have written about this exact question. Nielsen, Lewis, Sauro, Krug, the list goes on. As I was doing research for my first mobile design project out of college, I was struck by the variation amongst these usability thought leaders. However, the more I read, the more I sided with Jacob Nielsen and the concept of iterative testing.
Some design departments have small budgets to conduct usability testing. At Key Lime Interactive, we can work with just about any budget to do usability testing!
In the works of Jacob Nielsen, “Elaborate usability tests are a waste of resources. The best results come from testing no more than five users and running as many small tests as you can afford.” 1 Nielsen states, it takes approximately 15 users to track 100% of the issues with a system. 2 So why, then, does he recommend a smaller n at five, which will only expose 85% of user issues? Early and often, iterative design is the key to this usability conundrum.
With early and often testing on a smaller n, for example five, system designers can expose 100% of the system issues that users will struggle with. Let’s say, for example, you start testing with low-fidelity wireframes on n=5 users, you will discover approximately 85% of the issues. Continue this testing method at major milestones, and you can potentially expose 100% of the design issues before going live. This iterative cycle is the key to allowing lower n’s and exposing higher percentages of problems. Iterative and often testing is the ultimate tool in the usability toolkit.
Another way to look at the iterative design cycle is through a Lean UX lense. Lean UX is a design process that rapidly designs, tests, adopts, and deploys new designs of a system. Key Lime Interactive has found that early and often are the key indicators of good system design, and with that an n=5 is a statistically sound number.
A best practice for iterative testing is not to overburden your testers - don’t try to test everything right off the bat. Break your tests into 4-5 task segments, which can be completed in about 30-minutes. You don’t want your users to get fatigued or become disinterested in testing halfway through your test. By segmenting your tests, you are ensuring you get unbiased accurate test results that can lead to actionable items for your design team. Additionally, be sure you are testing the right personas for each test. You want to make sure the persona matches the audience so your results are accurate and actionable. The wrong persona being tested on the wrong test could result in less than desirable tests.
Remember, too, that usability testing doesn’t need to be elaborate and expensive, there are plenty of resources on the internet to help you conduct a small, productive usability study. Just remember iterative and often testing is what the usability doctor ordered!
Happy testing!
1,2 Nielsen, Jakob, and Landauer, Thomas K.: "A mathematical model of the finding of usability problems," Proceedings of ACM INTERCHI'93 Conference (Amsterdam, The Netherlands, 24-29 April 1993), pp. 206-213.