As a behavior driven user experience consultant, I do a lot of user research and user testing.
Across a wide variety of customer segments and industry categories.
So over the last 19 years, I’ve learned a lot.
My goals in user testing are two-fold:
- Find and correct basic usability issues that prevent the user experience from failing
- Get insights that help my client(s) get a competitive edge
Choosing the right consumer research method
I’ve found one-on-one user research to be the best approach to hit both of these goals.
Focus groups are better at the why and the what – but the why is surface level. The very nature of social settings lends participants to present an ideal version of themselves. In doing so, they hide the personality quirks that define their real-world behavior.
So it’s no surprise, that when I talk to my product or marketing friends about one-on-research, I get the same question.
Q: How many customer research interviews do I need
For qualitative research, my answer used to be – well, it depends. Or sometimes 5-7 users – a Nielsen recommendation – if I sensed the desire to do just a few.
And fit is extremely important – fewer, representative users are far better than a larger pool of users who don’t fit your persona(s).
So back to the issue of how many participants.
Well, I went back and did some analysis. And reviewed it in subsequent research projects over the next year.
I was stunned to find that the answer could be illustrated in a simple sketch. Or, to borrow a photography term, near the top of an S curve.
Here’s the sketch that shows how many users to include in your customer research project.
5-7 Research Interviews:Basic UsabilityBasic usability issues are solved near the low end of the curve. 5-7 representative participants are enough to identify issues with usability.
- Basic comprehension (message clarity)
- Navigation – organization and labels
- Message clarity
- Content priority
- Calls to Action visibility
Secondary Insights:You might also get some insights into user journeys and feature prioritization, with this sample size, but not in much detail
7-11 Research Interviews: Good User Experience InsightsOnce you’ve identified basic usability issues, you’re at the next level. Identifying your user’s behavior and usage patterns. Once you go past the first 5 or so users, you now start to get deeper user experience insights. Besides what you learn about basic usability you also get:
- Actions taken/not taken & why
- Content formats and placement
- Deeper feature prioritization
- Navigation – starting points, alternates
- On-site user journeys (preliminary)
- Basic competitive insights (preliminary)
11+ Research Participants: Customer Journey InsightsNow you get a chance to dig deeper and get a fuller picture of your customer journey. Not simply on your site or app – but also the before and after. Digital and offline, if it is relevant. With your participant set reaching critical mass, you also get validations and patterns:
- Patterns in navigation, content usage, and flows
- Validation of critical features (consistent why)
- Customer journey insights (off-site + on-site)
- Insights into long cycles and repeat visits
- Platform comparisons (Desktop vs Mobile vs App)
- Richer competitive Insights
- What knowledge do they have about your product?
- Are they using the navigation to browse or get to the product they saw elsewhere
- If they started from a reseller like Amazon, what brought them over – and how could you turn that into an advantage
- Why would they buy from you, even if you were more expensive?
Secondary Insights:At this stage, you start to hear the true fears and motivators behind your users’ actions or lack thereof. You also get some insights into the decision-making process that includes other people. And the reasons to buy beyond features, ease of use and fit.
15-18 Research Participants: Competitive EdgeAfter you’ve identified the majority of issues, your insights should turn a corner. At this point you’re focusing on a few things:
- Validating your prior findings
- Digging deeper into the remaining areas of uncertainty
- Expanding the scope of your questions as you try to understand the broader customer behavior
- Basic usability
- Common patterns
- navigation and user flows
- content prioritization
- UI and aesthetics feedback (if using visual designs)
- content appropriateness
- Prioritization of features – from critical to nice-to-have, differentiating features vs expected
- Most used customer journeys (off-site + on-site)
- Long sales cycles insights
- Decision-making factors
- Understanding of the true fears and motivators
Secondary Insights:You also get a chance to explore the journeys that come up often or are most relevant or perhaps our unexpected
18+ Interview Participants: Incremental InsightsIn my experience, any user interviews past 18 bring only incremental results. Most of these are a rehash of what you already learned. You might pick up one insight or two per interview at this stage. At this point, you’re spending most of the time in the validation of your prior findings. Notable insights diminish fast beyond this point. You’re hitting the plateau at this stage and covering the same ground as before. And once you go beyond 25, the sheer number of conversations and your notes starts to overwhelm. You start to hit analysis paralysis and spend a lot of time checking and rechecking your conclusions. I wish I could say I haven’t done more customer interviews than I needed, ever. Sadly I have and suffered for it. Hopefully, this will help you avoid that pain.
- Define what your goals are for your consumer research or user testing first
- Identify the number of research interviews you need based on the guidelines above
- Get the participant fit* right using a screener
- Schedule more interviews than the smallest number to account for no-shows