Ambiguous requirements make the strongest models fail hardest

A benchmark published in April takes the assumption underneath every code-generation evaluation — that the requirement is clear — and removes it.

The result is worth the attention of anyone who writes specs rather than code, because the failure mode it documents is not the one people worry about. The models do not get confused. They get confident.

What the benchmark does

Researchers at East China Normal University and the Shanghai Innovation Institute built Orchid, which they describe as the first function-level code-generation benchmark designed around ambiguous requirements: 1,304 tasks and 5,216 ambiguous requirement variants, spanning four types of ambiguity — lexical, syntactic, semantic, and vagueness.

The gap they are filling is real. As their own comparison table shows, the standard benchmarks — HumanEval, MBPP, BigCodeBench, SWE-Bench — are built from requirements characterised as "concise" or "code-aligned." All of them assume the specification is clear. None of them measured what happens when it is not, which is the condition most real specifications are actually in.

Ambiguity was injected via a multi-agent pipeline and then checked by human experts, rather than harvested from real-world specs. That is a limitation worth holding onto, and I will come back to it.

The example that explains the whole paper

The authors open with a requirement fragment that could appear in any spec:

"Return items filtered by the threshold."

One model keeps the items above the threshold. Another keeps the items below it. Both implementations are clean, both run, both are defensible readings of the English, and they are functionally opposite.

Nobody wrote a bug. The requirement had two meanings and the sentence did not choose between them.

The three findings

Ambiguity degrades every model tested. The five evaluated were GPT-4, Claude-3.5, DeepSeek-V3, DeepSeek-R1 and Qwen-2.5-Coder. The paper reports GPT-4 exhibiting "a performance drop exceeding 30%" on ambiguous specifications, and states that the most pronounced negative effects appeared in the most advanced models. Their conclusion is that current benchmarks "significantly overestimate the effectiveness of LLMs in real-world, 'noisy' software engineering scenarios."

The same requirement produces divergent implementations. Beyond correctness, ambiguity undermines consistency — the same ambiguous input yields functionally different code across runs. This is the property that matters most for anyone shipping, because it is invisible in a single sample. You get one implementation, it looks fine, and nothing signals that a different run would have built the opposite thing.

Models cannot locate the ambiguity. Asked directly, they show some capability to recognise that a requirement is ambiguous — but the authors find they "suffer from overprediction" and "consistently fail to precisely localize the source of ambiguity or provide valid resolutions." The paper names this gap precisely: between detecting a problem and understanding its logic.

The sentence worth stealing

The framing that does the most work is in the introduction:

While human developers can mitigate these uncertainties through iterative clarification, LLM-based solutions are typically forced into determinism.

That is the whole thing. A human engineer handed "filtered by the threshold" asks which way. It costs a Slack message. The model cannot ask — it is required to emit something — so it collapses the uncertainty into one implementation and moves on.

The ambiguity does not surface. It gets resolved, silently, by whichever reading the sampler preferred.

What this changes about how you spec

For years the practical cost of an ambiguous requirement was a question. Someone would read the line, not know what it meant, and ask — which was annoying and slow and also a working error- detection system. The ambiguity announced itself.

That system is being removed from an increasing share of the pipeline. When the thing reading your requirement is a model, an ambiguous line does not produce a question. It produces working, tested, plausible code that implements one of the two things you might have meant, with no signal about which.

Three consequences worth acting on:

Precision moved upstream and got more expensive. "Filtered by," "handle appropriately," "recent items," "the relevant user" — every one of these used to trigger a clarifying conversation. Now they resolve themselves.

A clean generation is not evidence the spec was clear. It is evidence the model found a reading. Reviewing generated output for correctness against your intent is now a distinct step from reviewing it for quality, and the first one is the one people skip.

Reviewing the requirement beats reviewing the output. The paper shows models cannot reliably locate ambiguity in a requirement even when asked directly. That is not a job to delegate to the same class of system yet — which makes the human pass over the spec more valuable than it was eighteen months ago, not less.

The limits of this result

The authors are working at function-level Python, which is a long way from a multi-service feature spec. Ambiguity was injected into existing benchmark tasks rather than collected from real requirements documents, so the distribution is constructed rather than observed. And the evaluated models are a 2023–2025 cohort — GPT-4, Claude-3.5, DeepSeek-V3 and R1, Qwen-2.5-Coder — so absolute numbers will move as models do.

None of that undermines the structural finding, which is about the shape of the interaction rather than any model's score: a system required to produce output cannot ask you what you meant. That property does not improve with scale. If anything, the paper's observation that the strongest models showed the most pronounced degradation suggests fluency makes the failure harder to see, not easier.

The requirement is still the thing. It is just being read by something that will never tell you it was unclear.

FAQ

What is Orchid? A benchmark of 1,304 function-level Python tasks and 5,216 ambiguous variants across four ambiguity types — the first designed around unclear requirements rather than clear ones.

Do models ask for clarification? No. The paper describes them as "forced into determinism," collapsing ambiguity into a single implementation.

Can they detect ambiguity? Partially, with overprediction — and they consistently fail to localise it or resolve it.

How large is the effect? Every model degraded; GPT-4 exceeded a 30% drop. Function-level Python only, with injected rather than observed ambiguity.