How Many Interviews Is Enough for a Spec?

"How many interviews do we need?" tends to get answered with a number, and the numbers people reach for come from a real body of research on saturation, the point at which more interviews stop telling you anything new.

The research is careful. The trouble starts when the number travels without its conditions. Those studies measured when new themes stopped appearing in one fairly uniform group answering one narrow question. A spec asks something different: whether to build this, for whom, and what it has to do. That is usually several groups and several questions, and the answer to "how many?" changes with them.

Here is what the studies measured, where they stop applying, and how to work out a number for your own decision.

What the studies actually measured

Guest, Bunce and Johnson (2006). One of the first studies to test saturation with real data, and the origin of "about twelve". It was published in Field Methods as How Many Interviews Are Enough? An Experiment with Data Saturation and Variability. Its abstract (the full text is paywalled, so we stay within it) defines saturation as the point at which no new information or themes are observed in the data. Using sixty in-depth interviews with women in two West African countries, the authors tracked saturation over the course of thematic analysis, and found that it occurred within the first twelve interviews, with basic elements of metathemes present as early as six.

Hennink and Kaiser (2022). A systematic review in Social Science & Medicine, Sample sizes for saturation in qualitative research, which we read in full. It found 23 articles that tested saturation, 17 with real data and 6 with statistical models. The empirical interview studies reached saturation between 9 and 17 interviews once outliers were set aside, with a mean of 12 to 13.

The same review is clear about the conditions. Most studies had a relatively homogeneous population and narrowly defined objectives, and most measured code saturation. Fourteen of the sixteen articles were about health topics. The outliers needed more: meta-themes across four countries took 20 to 40 interviews. And the authors say plainly that the results should not be used as generic sample sizes for any qualitative study.

Their table describes the 2006 study as 60 interviews in a homogeneous population, about perceptions of social desirability bias in self-reported reproductive health behaviour, with 12 interviews yielding 88% of codes. Not all of them: the other 12% only turned up in later interviews.

What "saturation" counts

Code saturation is reached when new interviews stop adding new topics. That is a useful thing to know. It is not the same as knowing the topic well enough to design for it.

The interview study in the review that compared the two, Code Saturation Versus Meaning Saturation (Hennink, Kaiser and Marconi, 2017), shows the gap. Across 25 in-depth interviews, code saturation, where the range of issues had been identified, came at nine. Meaning saturation, a richly textured understanding of those issues, took 16 to 24. The authors' shorthand: code saturation is when you have "heard it all", meaning saturation is when you "understand it all".

It is also not prevalence. The review notes that how common an item is in a population is not an important focus of qualitative research. Twelve interviews can tell you that admins struggle with permission inheritance. They cannot tell you that 30% do.

Why a spec is asking something different

Your users are not one homogeneous group. Admins and members, small and large accounts, new and long-standing customers. The review could not say how much a mixed population changes the number, because few studies tested it, and the authors say their results cannot be confidently extrapolated to different kinds of sample. The safe reading is to treat each segment that might answer differently as its own group.

Your objective is rarely narrow. "What problems do admins have with permissions?" is narrow. "Should we build this, for whom, and what must it do?" is three questions, and they saturate at different points.

You need meaning, not a topic list. A spec needs to know why the problem happens and what the workaround costs. That is closer to meaning saturation, which took more interviews in the study that measured it.

The count depends on how the interviews were run. Our briefing on AI-moderated interviews covers a study where needs kept accumulating well past two dozen: in the AI-moderated arm, 24 interviews produced 111.5 unique needs on average and 139 produced 238. At the same count, interviews with follow-up questions surfaced 111.5 needs on average and interviews without them 75.9. Those counts come from the study's own extraction pipeline, so the absolute numbers will not carry over to your research; the direction is the point. "We ran twelve interviews" leaves out whether anyone asked a second question.

Work back from the decision

So the honest answer is a procedure rather than a number.

1. Name the decision. What will the interviews change? Which problem to solve first, who owns setup, whether the workaround is painful enough to replace.
2. Name the segments that could answer differently. Plan interviews per segment, not in total.
3. Decide what you are saturating on. A list of problems saturates sooner than an understanding of one problem.
4. Set a stopping rule in advance. Several studies in the review used one shape: an initial batch, then stop when two or three consecutive interviews add nothing new. For a spec, make it nothing that would change the decision.
5. Send the other questions elsewhere. How often, how many, how much: logs, tickets, a survey. That is the "how often" test in a problem statement that survives review.

What to write in the spec

An illustration of an interview line in the evidence section:

Interviews: 11 admins at multi-team accounts (recruited from people who filed permission tickets) and 7 at single-team accounts (random sample of active accounts), August. Stopped each segment after two consecutive interviews added no new cause of misconfiguration. Not designed to measure frequency; see the permission logs.

That line tells a reviewer who, how many, how they were chosen, when, and why the count stopped where it did. Without it, twelve interviews with the wrong segment produce a confident, coherent evidence section for the wrong premise, which is the well-formed wrong thing with a sample size attached.

Notice what it does not say: "we interviewed until saturation." Hennink and Kaiser spend part of their review on that sentence. Reviews they cite found saturation often claimed with little or no account of how it was assessed, which, borrowing another author's phrase, they describe as using saturation as a "mantle of rigor". Their suggested questions for reviewers carry straight over to a spec review: how did you assess it, and how do you know you reached it?

The count you choose is an assumption like any other, and it belongs in the spec with its reason.

FAQ

How many interviews are enough? No general number. Plan per segment, and stop when new interviews stop changing the decision.

Where does "twelve" come from? Guest, Bunce and Johnson (2006): saturation within the first twelve of sixty interviews in one fairly homogeneous group.

What is saturation? The point new interviews stop adding new themes. Understanding each theme in depth took more interviews.

Can interviews measure how many users have a problem? No. Use logs, tickets or a survey for that.