Synthetic survey respondents got the direction right and the size wrong

The tempting shortcut in a sizing section is a population you can summon. If the adoption factor is unknown and the research budget is spent, a simulated panel will answer any question you ask, and the answer arrives with decimal places.

A study posted to arXiv on 26 September 2026 measures what that number is worth, and its result splits cleanly in two: the direction of a relationship survives the simulation, and the size of it does not.

The study

Two researchers at the Singapore University of Social Sciences evaluated three open-weight models — gemma4:31b, llama3.1:70b and qwen3.8:27b — against a national human panel of 2,058 respondents with complete answers to a battery of psychometric instruments.

The design rules out the obvious cheat. A persona is built from one real respondent's verbatim, item-level answers to a single instrument, quoted question by question and never summarised into a trait score. The model is then asked, in a fresh context, to complete a different instrument. Because it never sees the target instrument's items while being conditioned, agreement cannot be explained by echoing the scoring key back.

Fifteen instruments had coverage good enough to serve as either seed or target, giving 139 ordered pairs per model with 40 real respondents simulated per pair. Every pair was scored by fixed procedure on two axes: lexical overlap between the two instruments, and a four-tier judgment of how theoretically distant the two constructs are.

Two numbers that sound alike

The headline number is agreement about the pattern. For each pair the authors compute the real human correlation between the two instruments, and the simulated one, then correlate those two sets across all 139 pairs. That comes out at 0.703, 0.731 and 0.701 — strikingly consistent for three independently trained families, and the result the abstract calls substantively encouraging.

The second number is agreement about each size. The mean absolute error between a simulated correlation and the real human one is 0.276, 0.316 and 0.194. On a scale whose maximum is 1.0, the best model is off by about 0.19 on a typical pair, the worst by about 0.32.

Both are correct and they describe different things. The first says the simulated populations rank relationships roughly as humans do: strong ones come out strong, weak ones weak. The second says any individual relationship's size is off by an amount comparable to the size of many real effects. Sign agreement fills in the rest: 87.1%, 86.3% and 82.7% of pairs got the direction right.

The authors state the division plainly:

the models predict the sign of a relationship better than its magnitude

They add that magnitude error does not shrink as constructs get more distant the way the correlation does, so there is no region of the grid where the size becomes reliable.

Asking about one person is harder still

A sizing factor is usually a share of a population, so the population-level result is the relevant one. The study also measured whether a persona reproduces the specific person it was built from, and that is much weaker: averaged across pairs, the correlation between a persona's simulated score and its own respondent's real score was 0.210, 0.203 and 0.152, and it came out negative on 23, 22 and 28 of the 139 pairs.

Richer conditioning helps. On the one model tested that way, personas built from fourteen instruments at once recovered a held-out trait at a mean of 0.424, with sixteen of nineteen targets above 0.3 — though the best single instrument, when you already know which one it is, still won on ten of thirteen targets.

One finding bounds how much weight a single simulated result deserves. The human correlation between two instruments is symmetric; the simulated one need not be, because conditioning on A to produce B is a different computation from the reverse. Forward and reverse agreed at 0.666 and 0.718 for two models and at 0.139 for the third, whose largest swing — conscientiousness conditioned on empathy, at +0.629, against −0.413 reversed — changes sign depending only on which instrument was the seed.

Newer was not better

The authors ran three successive Llama releases on a fixed 14-pair panel. None of the three metrics improved monotonically, and the newest release was worst on two of them, with sign agreement of 71.4% against 92.9% for each older release. They are careful that a 14-pair panel carries real sampling variability and call the exact ranking indicative only. The finding they stand behind is the absence of a pattern:

it should be verified release by release, not validated once and assumed

The caveats, from the paper

• The statistics are descriptive. The 139 pairs share the same 40 respondents and reuse instruments, so the authors decline to attach significance tests that assume independence.
• There is no non-LLM baseline. Nothing says whether a regression fit to the human panel would reach 0.70 too, so the headline cannot be judged against a simpler method.
• The extremes of the grid are thin. The nearest-paraphrase tier is six pairs from only two seed instruments, about 4% of the grid; the unrelated tier's human correlations sit in a narrow band near zero where low error is easy. The authors flag both and say not to extrapolate from them.
• Side results are provisional. The behavioural extension covers one model and one respondent and may have been affected by seed text approaching the context limit. The authors report it for completeness and decline to treat it as confirmed.
• Model identity is a release tag. No weight checksums were collected, so exact reproducibility is not claimed.

One caveat is ours rather than the authors'. They read the result as a constructive case for open-weight simulation, and on their own question — does any real human structure survive? — that is fair. A sizing section asks a narrower question, and the same numbers answer it less kindly.

What this changes for sizing

A simulated panel cannot fill the factor you most want filled. In a bottom-up size, the factor usually labelled assumed is adoption: what share of the people with the problem would use the fix. That is a magnitude, and magnitude is what did not transfer here — which makes a synthetic panel the wrong instrument for the one gap it looks able to close. Writing assumed with a range, as sizing an opportunity without inventing numbers argues, is more defensible than importing a decimal from a simulation.

Direction is still worth something. If the question is which of three segments is most likely to care, the sign agreement here held up across three model families. That is a tool for sequencing research, not a business-case input.

An unconditioned panel looks like consensus. Without a real person's answers behind it, the null persona put 60% of its responses on the scale midpoint and skipped two of the five categories entirely on every item, at a mean entropy of 1.24 bits against 2.20 for the real population. A panel like that produces agreeable, tightly clustered answers to "would you use this?" — the shape most likely to be read as a clear signal.

Validation does not stay valid. A number from a simulated panel inherits the release it was produced on. That points the same way as our briefing on AI-moderated interviews, where richer transcripts did not make twins predict real reactions any better: the value of simulated people is not rising smoothly with model quality.

The honest sizing section still names the factor it cannot measure and says what it would take to find out. Borrowing a number from a population that does not exist is the same mistake as borrowing one from the wrong real population, with better production values.

FAQ

Can simulated respondents size an opportunity? Not the size. Direction held up at 83% to 87% sign agreement; magnitude was off by 0.19 to 0.32.

What was measured? Whether a persona built from one real person's answers to one instrument reproduces their relationships on a different one, over 139 pairs against a 2,058-person panel.

Are newer models better at it? Not monotonically. The newest of three Llama releases was worst on two of three metrics.

So what is it good for? Ordering hypotheses and guessing direction, which the authors call encouraging, rather than producing a number for a business case.