LLM peer reviews argued more and pointed to fixes less than human ones

Most comparisons of AI and human reviewers ask one question: is the AI's review as good? A study posted on arXiv on 1 September 2026 asks a more useful one. It treats a review as a bundle of different jobs — finding the weakness that matters, arguing it, asking questions, pointing to a fix — and measures which jobs show up in each kind of review.

The answer is that people and models did different jobs, and telling the model to act like an expert mostly made the difference larger, not smaller.

The study

Yunhan Yang and Mike Thelwall of the University of Sheffield and Guoxiu He of East China Normal University used the open peer reviews of ICLR 2025, the machine-learning conference: 11,560 submissions and 46,748 human reviews. For each paper, an open-weight model, Gemma3-27B, read the full text and wrote a review in the same format. A second model, Qwen3-32B, was used to check the results did not depend on one model family. Everything ran locally on a university cluster.

The authors looked only at the two sections where criticism lives — weaknesses and questions — and split them into individual points: 321,245 human weakness points and 155,738 from the model. Each point was then classified using five established theory-guided frameworks, covering:

• What a weakness targets — a factual or procedural detail, or a higher-order matter such as contribution, significance or limitations.
• How it is argued — a bare claim, or a claim backed by evidence and an explicit reason.
• How deep a question goes and how many elements of the study it connects.
• What a question is for — checking the task, probing the process, or feed-forward: pointing the author toward a concrete revision.

A further 600 papers — 200 accepted, 200 rejected, 200 withdrawn — were reviewed again with an "expert" prompt emphasising problem importance, claim–method–evidence alignment, robustness and contribution, to see whether better instructions made the model more human-like.

What they found

Humans aimed higher. 45.1% of human weakness points were higher-order — about contribution, positioning or significance — against 37.9% for the model.

The model argued harder. Its weakness points more often came with an explicit reason linking the observation to the judgment. Its argument-depth score was 1.752 against the humans' 1.581. A combined "expert-like weakness" measure came out slightly in the model's favour, but the authors note this was driven by argument structure, not by choosing more important problems.

The authors make the distinction in one sentence:

A report can be fluent, detailed, and well-structured without identifying the most scientifically consequential weakness

Humans told authors what to do. In questions, the model asked more deep, mechanism-level questions and more that connected several parts of a paper. Humans more often asked questions that pointed toward a specific revision — the feed-forward function.

These patterns held across accepted, rejected and withdrawn papers, and most of them reproduced when the generating and annotating models were swapped.

The expert prompt amplified the model, not the human. On the 600-paper sample, the expert prompt moved higher-order critique only modestly toward the human level (human 0.446, baseline model 0.377, expert-prompted model 0.406). It sent argument depth far past the humans (1.583 human, 2.961 with the expert prompt) and raised the expert-like weakness rate from 0.116 to 0.377, against 0.088 for humans. And it reduced revision guidance further: feed-forward questions fell from 0.364 to 0.307, against 0.437 for humans.

The authors' reading of that result is the one to keep:

expert-style language may make LLM critique appear more systematic and authoritative without necessarily improving the underlying evaluative judgment

The caveats, from the paper itself

• One venue. ICLR is a machine-learning conference with a structured review form. The authors say other disciplines, journals and review cultures need testing.
• One main model, and not a frontier one. The main analysis used Gemma3-27B, checked against Qwen3-32B. The authors say domain-specialised and frontier closed models would strengthen the findings. Readers should not assume the profile of the largest commercial models matches.
• Rates, not abilities. A lower rate means a function appeared less often in the review text, the authors write, not that the model cannot produce it.
• No outcomes. The study does not link critique profiles to what authors did with the reviews, revision quality or decisions. The authors list this as future work.
• Labels came from a model too. The point-level classification was done by an LLM. A human audit of 300 points found moderate agreement with the main annotator and higher agreement with the second one.

Taken together: a large, careful description of how two kinds of review text differ, not a verdict on which reviewer is better, and not evidence about outcomes.

Why this matters for product critique

A spec critique is not an ICLR review, but it has the same two halves: deciding which problems matter, and making them actionable. The study puts numbers on a split that anyone who has run a design crit or a spec review will recognise.

Prioritising is the scarce function. The human advantage was in higher-order critique: is this the right problem, is the contribution real, what is the limitation that sinks it. In product terms, that is "the problem statement and the solution describe different problems" — the finding we have argued beats twenty considerations of equal weight. It is also the category other work found models struggle to detect: judgments that need knowledge of the world rather than inspection of the text.

Feed-forward is the other one. A critique that explains the flaw in depth but does not say what to change leaves the author with the hard part. The model's questions leaned toward explanation; the humans' toward "do this next". Product teams run into the same thing with AI review output that reads like an essay about the spec rather than a list of changes to it.

Sounding expert is not the same as judging well. The expert-prompt result is a warning for anyone tuning an AI reviewer, ourselves included. Richer argument structure is easy to mistake for better critique. It made the model's output look more authoritative while moving it further from the human profile on revision guidance.

That suggests a division of labour for a product critique session rather than a contest:

1. Before the room: use a model to widen the question set, surface weak links between claims and evidence, and make reasoning explicit. The authors themselves suggest LLMs as pre-review tools for exactly this.
2. In the room: spend the time on the two functions the model under-supplied — ranking what matters, and agreeing what changes.
3. On the page: require every finding to end in a proposed change. If a point cannot say what to do, it is commentary, and it should be weighed as commentary.

The third point is a design target for tools in this category, ours included, not a claim that any of them meets it. This paper is a reminder of how easily the target slips.

FAQ

What was compared? Human ICLR 2025 reviews and Gemma3-27B reviews of the same papers, point by point, across five critique frameworks.

The main difference? Humans more often flagged higher-order weaknesses and guided revisions; the model more often structured arguments and asked deeper explanatory questions.

Did an expert prompt help? It mostly amplified the model's own style and reduced revision guidance further.

Biggest caveat? One venue, one main open-weight model, LLM-assisted labelling and no outcome data.