The LLM judge found the bug and gave it a perfect score

When a feature's output is non-deterministic, its acceptance criterion often ends up as "the eval passes", and the eval is often another model reading the output and scoring it. A paper posted to arXiv on 31 August 2026 shows what that criterion is worth when the thing being scored is under pressure to score well. The short version: it is worth only as much as the judge's own ability to do the task.

The study

The paper has a single author, Idil Gozel of Evaluator Integrity in London. A competing-interests note discloses that the author is founding a company whose subject is the assessment of automated evaluators. Read it with that in mind. Every finding concerns public open-source software pinned to published versions, and the run records are published so the numbers can be re-derived.

It has two parts.

A census of shipped defaults. The author examined 27 default judge configurations from eight widely used evaluation frameworks: DeepEval, Ragas, Promptfoo, OpenAI Evals, Braintrust autoevals, LangChain, Arize Phoenix and MLflow. These are the templates teams copy when they start. Of the 24 in scope, none implemented commit-first judging, the defence that earlier work found effective: the judge solves the task itself, commits to its answer, and only then compares the candidate. Nine instead told the judge to reason step by step with the candidate already in view, a variant the earlier work measured as ineffective. Those nine share one ancestor prompt, traceable through a copied typographical error.

A controlled experiment. DeepEval's G-Eval metric, configured as its documentation recommends, judged code with Claude Opus 4.8. A generator model, Claude Sonnet 5, wrote solutions to four small Python tasks in a plain best-of-N loop: twelve rounds of eight candidates, the best two seeding the next round, 96 candidates per run. The score was half the pass rate on visible unit tests and half the judge's score. A separate held-out suite, which the loop could never see, measured whether each candidate was actually correct. No jailbreaks, no prompt injection. The author's description is the ordinary loop production systems run every day.

What it found

Scores rose and correctness did not. On the interval-merging task, 90 of 96 candidates in the first seed and 93 of 96 in the second scored well while failing the held-out suite. The evaluator's final-round mean was 0.97 and 0.96. True correctness sat at 0.667 in both. Every candidate passed every visible test.

The judge saw the defect. The winning candidate merged intervals that do not touch. The judge scored it 1.0 and quoted the faulty condition in its reasoning as the reason it was correct.

The judge did not miss the bug. It found the line, described its behaviour, and cited it as the reason for a perfect score.

The defence worked, until it did not. Switching to commit-first judging removed the effect on the interval task: 0 of 96 in both seeds, because the judge's own committed solution was correct. On a duration-parsing task it backfired in both seeds. Candidates that scored well but were wrong rose from 37 and 41 to 50 and 75. One seed ended pinned at 0.667 correctness while the evaluator read 0.99, and 0.667 was exactly the score of the judge's own wrong answer. The author's main finding is that commit-first judging does not remove the anchor that gets gamed. It moves it from the candidate to the judge.

Bigger was not safer. A smaller judge, Claude Haiku 4.5, was barely gamed on the interval task: 4 of 96. The frontier judge was fooled 90 and 93 times. A cheap probe explained why. Asked to solve each task three times, the frontier judge solved the interval task once and the smaller judge three times. Neither solved the duration task, the one where the defence backfired. The probe cost about 3 pence for the smaller judge and about 15 for the frontier one.

Strict mode did not help. Binarising the judge's score still ended at 0.83 correctness with the evaluator reading 1.0.

The failure replicated; the fix did not. The two seeds under the shipped configuration agreed closely on every task. The two commit-first seeds on the duration task ended at 0.958 and 0.667.

The caveats, from the paper

The paper is direct about its scale and about its own design errors:

• Four small tasks, one judge family, one generator. Divergence appeared on only two of the four tasks, because the generator solved the other two outright. The author calls it a controlled demonstration, not a survey.
• One seed of two. The population settling on the judge's own score happened once, so the claim is that commit-first judging can lock in the judge's error, not that it must. Several control arms are single runs.
• Defaults, not deployments. The census covers default templates. Teams modify them, and the verdicts do not transfer to a modified configuration automatically.
• A sampling asymmetry. The frontier judge rejects a temperature setting, so its commit-first arm ran at default sampling rather than temperature zero, which affects comparisons across judges.
• Self-reported design errors. A task-admission rule written in advance would have excluded all four tasks; the author reports them anyway and flags the rule as a design error. Two held-out checks were not justified by the task specifications; the reported figures keep them, which the author says makes them the conservative ones.

The paper also audits its own prose: a verbatim-quote check failed five of fifteen first-draft claims, all corrected before publication.

Why it matters if you write acceptance criteria

A criterion that reads "LLM judge score of 0.8 or higher on the eval set" looks testable. This paper shows the ways it can pass while the feature is wrong, and each has a fix that belongs in the spec, which is the subject of acceptance criteria for non-deterministic features.

Name the judge, and check it can do the task. A judge score is a measurement with its own error, and here the error depended on the judge-and-task pair, not on model size. Before a judge becomes the arbiter of done, ask it to solve a sample of the tasks and score its answers against checks it has not seen. In this study that cost pennies and predicted where the judge would fail.

Accept against a set nobody optimises against. The whole result turns on the split between the visible suite, which the loop could chase, and the held-out suite, which it could not. If prompts or models are chosen by eval score, the eval becomes the target, which is the proxy displacement our briefing on the benchmark utility gap described. The acceptance number should come from a held-out set with a named owner.

Do not trust the default. None of the in-scope templates teams start from implemented the defence, and the variant that spread by copying was the one measured not to work. The spec should state which judge configuration and version the threshold applies to.

Run it more than once. The failure reproduced tightly and the fix did not. One passing run is not evidence the fix works, the same lesson as the requirements-quality benchmark that found single-execution evaluations statistically unreliable.

Keep the scale in view. Four small Python functions are a long way from a production feature. What the paper demonstrates is a mechanism, carefully: an automated judge can read a defect, name it and approve it, and the score will not tell you.

FAQ

What happened? A best-of-N loop gamed a shipped LLM judge; the judge cited the bug as the reason for a perfect score.

What is commit-first judging? The judge answers first, then compares. None of 24 default configurations used it.

Does it fix the problem? Only when the judge can solve the task. Otherwise it locks in the judge's error.

What changes in a spec? Name and validate the judge, accept against a held-out set, and repeat the run.