Reviewers approved more AI code over time, and their comments barely showed it

The worry about AI-generated work is usually framed around the author: the model writes something plausible and wrong. A preprint posted on arXiv on 5 September 2026 looks at the other side of the gate. As reviewers saw more AI-written pull requests, did they start approving them more easily — and could you tell from what they wrote?

The answers were yes, slightly, and barely.

The study

The authors — two independent researchers and a researcher at Nanchang Hangkong University — used AIDev, a public dataset of pull requests submitted to GitHub repositories with at least 100 stars by five AI coding agents: GitHub Copilot Autofix, Devin, OpenAI Codex CLI, Cursor and Claude Code. The pull requests date from January to July 2025.

They focused on 400 repeat reviewers who had each reviewed at least ten agent pull requests: 11,429 reviews across 207 days, with 10,104 inline comments written by those humans.

For each reviewer they sorted reviews by time, split the history at the midpoint, and asked two questions. Did the approval rate change? And did the language of the comments change with it?

What they found

Approvals went up. Across the 400 reviewers, the approval rate rose from 30.5% in the early half to 36.6% in the late half, a gain of 6.1 percentage points. It was statistically significant, with a Cohen's d of 0.25 — a small effect. The direction was not universal: 52% of reviewers became more approving, 28% less, and 20% did not change.

Among the 173 reviewers with at least 20 reviews, the approval rate climbed from 27.9% in the earliest tenth of their reviews to 42.8% in the latest. The authors stress the rise was gradual rather than a sudden switch into rubber-stamping.

The comments did not show it. The authors tested four standard measures of review-comment effort: lexical diversity, Shannon entropy, technical specificity (how often a comment references code, files or lines) and constructive actionability (imperative verbs, questions, code blocks). None declined as reviewers gained exposure. After correcting for multiple tests, every comparison was non-significant. A classifier built on those features did worse at telling early comments from late ones than simply guessing the majority class.

A subtler signal existed, faintly. Using sentence embeddings, they found each reviewer's late comments sat measurably further from their own early comments than chance would predict — but the drift went in different directions for different people. A small model on those features reached an F1 of 0.74 against a majority-class baseline of 0.71, a gap the authors themselves call modest.

Behaviour moved first. A time-series test found that changes in approval rate predicted later shifts in the technical specificity of comments, not the other way round. In the authors' words, the linguistic shift "trails the approval behavior shift rather than leading it."

They also checked the obvious confounders. Approval of human-written pull requests in the same months and overlapping repositories showed no directional change, which argues against reviewers simply becoming more lenient in general. Calendar date and pull-request size did not eliminate the effect.

The caveats, from the paper itself

The limitations section is short and pointed, and two of its items change how the headline should be read.

• The agents may have improved. The authors say they cannot rule out that AI-authored code got better over the period, which would raise approvals without any change in reviewer rigour. The human-PR control addresses general leniency, not this.
• No outcome data. The dataset has no post-merge defect rates, so the study cannot say whether the extra approvals let worse code through. The effect is small, and its practical importance depends on exactly that missing measure.
• Public repositories, seven months. Enterprise review, and longer reviewer careers, may look different.
• Short comments. The median inline comment was 11 words, and 24% were five words or fewer. The authors suggest this is part of why standard lexical measures found nothing.
• Off-the-shelf tools. The embedding model was general-purpose, not code-specific, and the "late phase" label was defined by position in a reviewer's sequence rather than by any observed habituation event.

This is also a preprint by a small team, with replication scripts published and no independent replication we know of. The fair summary: a real but small upward drift in approvals, of uncertain cause, invisible to the measures most people would reach for.

Why this matters for sign-off

Approvals of AI-written code are not the same thing as approvals of a spec. But the mechanism the paper is probing — a reviewer who has said yes many times finding it slightly easier to say yes again — is not specific to code, and the measurement problem it exposes applies anywhere approval is a single bit.

Three things follow for anyone who asks for, or gives, sign-off on a spec.

You cannot audit an approval by reading the comments around it. The study's central negative result is that comment text carried almost no trace of the change in approval behaviour. A thumbs-up with a few short comments attached looks the same whether the reviewer read every acceptance criterion or the first page. If the review record is going to tell you anything later, the reviewer has to write down what they checked.

A bare approval is ambiguous even to researchers. The authors could not separate "reviewers got softer" from "the work got better", because an approval records the outcome and nothing about the basis. The same ambiguity sits in every approved spec: was it approved because it was ready, or because the reviewer had approved the last six from the same author? That is the case for making sign-off a named commitment with a stated scope — "I approve the data model and the migration plan; I did not review the pricing section" — rather than a click.

Volume is the condition to watch. The drift appeared as reviewers' exposure to a single kind of author grew. As AI drafting makes specs faster to produce, more of them reach the same reviewers, looking equally complete. We have argued that confident drafts made review easier to skip; this paper is a small, careful data point on the reviewer's side of that equation, not proof of it.

What to take from it

Not that reviewers rubber-stamp AI work. The effect is small, the cause is unresolved, and nobody measured whether anything bad got through.

The narrower lesson is about records. If approval rates can drift without standard measures of the review text changing, then the text is not a record of rigour, and neither is the approval. What a reviewer says they checked is the only part of sign-off that carries information after the fact. The same logic sits behind ranking findings by severity rather than leaving a list: it forces the reviewer to show the judgment, not just the verdict.

FAQ

What changed? Approval of AI-agent pull requests rose from 30.5% to 36.6% across each reviewer's history, a small but significant effect.

Did the comments show it? No. Four standard comment-effort measures stayed flat.

Biggest caveat? The agents' code may have improved over the period, and there is no defect data to say whether the extra approvals mattered.

Lesson for sign-off? Make approval state what was checked, because the approval itself, and the comments around it, may not tell you.