AI-moderated interviews found as many customer needs as human ones

The argument about how many interviews are enough is usually settled by a number someone half-remembers. A study posted to arXiv on 24 September 2026 is more useful than another number, because it measures what the question is really about, how many distinct customer needs you recover as interviews accumulate, and shows how much that depends on how the interviews are run.

The study

Researchers at Columbia Business School and the Wharton School ran a pre-registered, between-subjects study with three industry partners: a North American building-products company that supplied the research context and real marketing material, an AI-interview startup that supplied the interviewing platform, and GBK Collective, a research consultancy whose trained moderators ran the human interviews.

317 people took part, recruited from Prolific and screened as home-improvement decision-makers buying windows and doors. Each did a spoken interview of about 30 minutes on the same topic guide, in one of three conditions:

• AI-moderated (139 people). An LLM interviewer was given the objective for each block and probed adaptively.
• Static (154 people). The same questions on the same platform, one recorded answer each, no follow-up.
• Human-moderated (24 people). Trained moderators read the same fixed questions and probed to depth against the same objectives.

The arms were sized by budget, not headcount: roughly $5K of market cost per condition. That is why the human arm is small. The authors planned about 30 human interviews for that budget, and no-shows brought it to 24.

Customer needs were extracted from every transcript by an LLM pipeline and merged into one codebook of 252 unique needs across all interviews.

What they found

Per interview, AI and human moderation recovered about the same. At a matched 24 interviews, AI moderation recovered 111.5 unique needs on average (95% interval 98.0 to 125.0), human moderation 109.0, and static interviews 75.9. The authors put each AI interview at roughly 20% of the cost of a human one.

Per dollar, the cheaper adaptive method won. Under the fixed budget, AI moderation reached 238 unique needs, static 188 and human moderation 109. The implied cost per need was $19, $27 and $36.

The follow-up questions did a lot of the work. The static arm used the same platform and the same topics and differed mainly in having no probing. Its participants spoke for 4.3 minutes on average against 8.1 in the AI arm, and it accumulated needs more slowly.

The methods found different things. Of the 252 needs, 93 turned up in all three conditions. 52 appeared only in AI-moderated interviews, 7 only in static ones and 3 only in human ones.

People still talked differently to a person. Participants with a human moderator sounded more positive, more emotionally activated and more assertive. AI and static interviews were indistinguishable on those measures. The authors' own summary:

AI moderation matches human moderation on interview content, but not on the emotional experience of talking to a person.

Richer transcripts did not make better simulated customers. The second half of the paper built LLM "digital twins" of each participant and tested them against that person's held-out reactions to real mailers, commercials and product claims. Twins given the interview transcript and a few individual-difference scales generally beat twins given demographics alone. But twins built from AI-moderated interviews did no better than twins built from static ones, and the twins reasoned more analytically than the people they stood in for, most of all on images and video.

The caveats, from the paper

• One category. Replacing windows and doors is a high-involvement purchase with a lot of deliberation. The authors say AI moderation may perform differently for low-involvement, symbolic, emotional or sensitive topics.
• One platform, one moment. They evaluated a particular AI interviewing platform and set of models, and say the comparison should be repeated as the technology changes.
• A small human arm, added later. The first pre-registration covered only the two AI-run conditions. The human arm was added after peer feedback, pre-registered separately and recruited through a slightly different process. 24 interviews is the entire human sample.
• Not everything was pre-registered. Coverage at matched budget was. The comparison of transcript quality and the overlap of needs across conditions are labelled exploratory, and the robustness checks were not pre-registered.
• Costs are calculated from market prices supplied by the partners, and the human arm's no-shows raised that condition's marginal cost.

One caveat is ours rather than the authors'. "Unique needs" is a count produced by method choices: an LLM extracted needs from each transcript, and mentions were merged when their embeddings fell within a cosine distance of 0.60. A different threshold would give a different count. Every condition went through the same pipeline, so the comparisons between them are fair. The absolute numbers are not a property of customers you can carry over to your own research.

What this changes for how you plan research behind a spec

"How many interviews?" hides two other questions, and this study separates them.

The count kept climbing. In the AI arm, 24 interviews produced 111.5 needs on average and 139 produced 238. Two dozen interviews did not exhaust what this population had to say at this level of detail. Whether that matters depends on the decision. A spec that needs the handful of problems most people share can stop far earlier than one that needs the long tail, which is the argument in how many interviews is enough for a spec: the number is set by what the spec has to decide.

How the interviews were run changes what the number means. At the same count, the interviews with follow-up questions surfaced 111.5 needs on average and the ones without surfaced 75.9. "We ran 24 interviews" is missing its most important qualifier. An evidence section should say who you talked to, how they were recruited, whether anyone asked a second question, and what counted as a finding. That is exactly the part a reader cannot see from the citation alone.

Interviews find needs. They don't predict reactions. The digital-twin result is the one to keep if someone proposes replacing a test with simulated customers built from transcripts. More interview detail improved the twins against a demographics baseline and then stopped helping. If a spec's success depends on how people react to a specific screen, message or price, an interview transcript, or a model built from one, is weaker evidence than putting the thing in front of them. That is a distinction worth writing into the research that feeds the spec rather than leaving implicit.

Know what the machine interviewer misses. The authors name emotional engagement as the main limitation of AI moderation. If the research is about something people feel strongly about, or would find hard to say, that is the gap that matters.

FAQ

What did the study compare? AI-moderated, static and human-moderated versions of the same interview, 317 homeowners, equal budgets.

Did AI find as many needs? At 24 interviews each, yes: 111.5 against 109.0, with static at 75.9.

What did humans do better? Emotional engagement. People sounded more positive and more animated talking to a person.

Can transcripts stand in for customers? Not well. Richer transcripts did not make the simulated customers predict reactions any better.