Sketch: How many customer research interviews do you need?

Share on twitter
Share on facebook
Share on linkedin

As a behavior driven user experience consultant, I do a lot of user research and user testing.

Across a wide variety of customer segments and industry categories.

So over the last 19 years, I’ve learned a lot.

By listening.

My goals in user testing are two-fold:

  1. Find and correct basic usability issues that prevent the user experience from failing
  2. Get insights that help my client(s) get a competitive edge

Choosing the right consumer research method

I’ve found one-on-one user research to be the best approach to hit both of these goals.

Why one-on-one research over focus groups or surveys?

Focus groups are better at the why and the what – but the why is surface level. The very nature of social settings lends participants to present an ideal version of themselves. In doing so, they hide the personality quirks that define their real-world behavior.

So it’s no surprise, that when I talk to my product or marketing friends about one-on-research, I get the same question.

Q: How many customer research interviews do I need

For qualitative research, my answer used to be – well, it depends. Or sometimes 5-7 users – a Nielsen recommendation –  if I sensed the desire to do just a few.

And fit is extremely important – fewer, representative users are far better than a larger pool of users who don’t fit your persona(s). 

 - Dilbert by Scott Adams

So back to the issue of how many participants. 

Well, I went back and did some analysis. And reviewed it in subsequent research projects over the next year.

I was stunned to find that the answer could be illustrated in a simple sketch. Or, to borrow a photography term, near the top of an S curve.

Here’s the sketch that shows how many users to include in your customer research project.

5-7 Research Interviews:Basic Usability

Basic usability issues are solved near the low end of the curve. 5-7 representative participants are enough to identify issues with usability.
  • Basic comprehension (message clarity)
  • Navigation – organization and labels
  • Message clarity
  • Content priority
  • Calls to Action visibility
At this point, you’re simply trying to figure out what works or doesn’t work and why. And get consistent WHY so you can move on to the next step. At this point, you also start to identify any outliers. It’s also important that you focus on understanding and listening more than solving problems or pitching your product. Here’s a useful guide on how to be a good moderator.

Secondary Insights:

You might also get some insights into user journeys and feature prioritization, with this sample size, but not in much detail

7-11 Research Interviews: Good User Experience Insights

Once you’ve identified basic usability issues, you’re at the next level.  Identifying your user’s behavior and usage patterns. Once you go past the first 5 or so users, you now start to get deeper user experience insights. Besides what you learn about basic usability you also get:
  • Actions taken/not taken & why
  • Content formats and placement
  • Deeper feature prioritization
  • Navigation – starting points, alternates
  • On-site user journeys (preliminary)
  • Basic competitive insights (preliminary)
At this stage, you start to get insights specific to your product or service experience. For an e-commerce site, you might learn how people browse and search to refine a big list into a manageable set. For a B2B marketing site, you might learn the right experience at different stages of the journey. NOTE: 7-11 participants are a good ballpark, but yours might vary a bit.  This depends on how many primary segments you have, and how different they are from each other. Think an enterprise company that is doing research across product lines or geographies.

11+ Research Participants: Customer Journey Insights

Now you get a chance to dig deeper and get a fuller picture of your customer journey. Not simply on your site or app – but also the before and after. Digital and offline, if it is relevant. With your participant set reaching critical mass, you also get validations and patterns:
  • Patterns in navigation, content usage, and flows
  • Validation of critical features (consistent why)
  • Customer journey insights (off-site + on-site)
  • Insights into long cycles and repeat visits
  • Platform comparisons (Desktop vs Mobile vs App)
  • Richer competitive Insights
Taking our e-commerce example further. When a prospect arrives at your site/app:
  • What knowledge do they have about your product?
  • Are they using the navigation to browse or get to the product they saw elsewhere
  • If they started from a reseller like Amazon, what brought them over – and how could you turn that into an advantage
  • Why would they buy from you, even if you were more expensive?

Secondary Insights:

At this stage, you start to hear the true fears and motivators behind your users’ actions or lack thereof. You also get some insights into the decision-making process that includes other people. And the reasons to buy beyond features, ease of use and fit.

15-18 Research Participants: Competitive Edge

After you’ve identified the majority of issues, your insights should turn a corner. At this point you’re focusing on a few things:
  1. Validating your prior findings
  2. Digging deeper into the remaining areas of uncertainty
  3. Expanding the scope of your questions as you try to understand the broader customer behavior
It’s usually at this stage you’ll hear the participant say  “I shouldn’t be telling you this…” And that’s when some of the richest insights come out. These might shed competitive light or the true reason behind a fundamental feature. You need 15-18 research interviews for insights that deliver a competitive edge. With this set you get a full picture:
  • Basic usability
  • Common patterns
    • navigation and user flows
    • content prioritization
    • UI and aesthetics feedback (if using visual designs)
    • content appropriateness
  • Prioritization of features – from critical to nice-to-have, differentiating features vs expected
  • Most used customer journeys (off-site + on-site)
  • Long sales cycles insights
  • Decision-making factors
  • Understanding of the true fears and motivators
These insights deliver one or more aha moments. Ones that help you define the competitive edge for yourself or your client.

Secondary Insights:

You also get a chance to explore the journeys that come up often or are most relevant or perhaps our unexpected

18+ Interview Participants: Incremental Insights

In my experience, any user interviews past 18 bring only incremental results. Most of these are a rehash of what you already learned. You might pick up one insight or two per interview at this stage. At this point, you’re spending most of the time in the validation of your prior findings. Notable insights diminish fast beyond this point. You’re hitting the plateau at this stage and covering the same ground as before. And once you go beyond 25, the sheer number of conversations and your notes starts to overwhelm. You start to hit analysis paralysis and spend a lot of time checking and rechecking your conclusions. I wish I could say I haven’t done more customer interviews than I needed, ever. Sadly I have and suffered for it. Hopefully, this will help you avoid that pain.

Research Takeaways

  1. Define what your goals are for your consumer research or user testing first
  2. Identify the number of research interviews you need based on the guidelines above
  3. Get the participant fit* right using a screener
  4. Schedule more interviews than the smallest number to account for no-shows
* Fit – Participants that resemble your segment or persona. It’s better to test with a small representative set than a large number of participants, who may not match your profile.
June 12, 2019

Add Your Heading Text Here

About Me

About Me

I am Subir, a practicing UX strategist and designer. My goal with UX for All is to help you avoid mistakes and deliver exceptional customer experiences. Every time.

Keep Reading

Comments

Comment policy: We love comments and appreciate the time that readers spend to share ideas and give feedback. However, all comments are manually moderated and those deemed to be spam or solely promotional will be deleted.

Leave a Comment

Your email address will not be published. Required fields are marked *