Pointing a code model at the bug lost to letting it try again

"Show the model where the bug is" sounds so obviously helpful that it is easy to build in without testing. A study posted to arXiv on 1 September 2026 did test it, with a placebo arm borrowed from clinical trials, and the assumption failed twice: the signal was usually missing, and where the author supplied it, starting over still did better.

The study

The paper is by Anik Jha, an independent researcher, and is described as a workshop paper. It takes a program an AI model wrote that fails its tests and compares three ways to fix it, each given 16 attempts:

• Localized infill. Use the failing tests to find the most suspicious lines, a standard technique called spectrum-based fault localization, and have the model rewrite only those lines.
• Random-span placebo. Have the model rewrite a stretch of code of the same length somewhere unrelated.
• Blind resample. Ignore the failure entirely and generate a fresh solution from the original problem.

The two controls answer different questions. The placebo separates "the location was right" from "the edit was small". The fresh attempt separates "the model used the failure" from "it simply got another try".

The setup covered three models of 26 to 32 billion parameters, three Python benchmarks and 488 failing programs, plus a fourth model from a third family run as a declared replication.

Before running anything, the author wrote down a success rule: localized repair had to beat both controls, beat the placebo by at least five percentage points after correction for multiple tests, and replicate across at least two model families. The paper calls the rule pre-specified rather than preregistered, because it was not registered with an outside service.

What happened

The precondition rarely held. Only 9.0% of failing programs (44 of 488) had a failing public test with usable coverage data. The largest group, 212 programs, passed every test a user could see and failed only hidden ones, so there was nothing to localize from. Adding a much larger, augmented test suite raised the localizable share to 36.3%, but no deployed repair loop has that suite. The author reports it as an upper bound, not a method.

Where it could run, a fresh attempt won. On the 177 programs localizable with the stronger suite, 10.1% of fresh attempts succeeded against 4.4% of targeted ones. Counting programs fixed by one approach and not the other, the fresh attempt won 40 to 3. The fourth model, Mistral-Small-24B, showed the largest gap of all: 0.6% of targeted attempts succeeded against 11.9% of fresh ones.

The location effect is suggestive, not established. Targeted edits did beat the same-size random edit when all models were pooled, 4.3% of attempts against 1.1%. But pooling treats different models as interchangeable, the paper's own analysis plan does not allow that for a confirmatory claim, and the effect did not hold up within any single model.

The model mostly rewrote what was already there. In 48.9% of targeted attempts the model reproduced the removed lines character for character, leaving the program unchanged. The suspect lines were the ones it had just written, and the surrounding code pulled it straight back to them. Widening the editable region from a median of 1 line to 5 cut the repetition but still lost to a fresh attempt.

So the success rule was not met. The method's replication clause was satisfied only in reverse: the cross-family result the rule demanded turned up as a loss.

The caveats, from the paper

• Narrow tasks and models. Function- and class-level Python, not repository repair or edits spread across several places, and only models of 24 to 32 billion parameters. A probe with much smaller models could not support the comparison at all.
• One model did not reproduce it. Gemma-4-26B, the only mixture-of-experts model and the only non-Qwen of the original three, did not show the loss against a fresh attempt.
• The deployable condition resolved nothing. Restricted to the 44 programs localizable from public tests alone, neither comparison survived correction. In the author's words:

Every claim that resolves in this study therefore depends on a localization signal a deployed loop would not have.

• Cost depends on the unit. A targeted attempt used 21.7 generated tokens against 371.1 for a fresh solution. Priced in tokens, localization leads only on budgets too small to buy a single fresh attempt.
• The author's own pipeline had five defects, each of which changed a measured result before it was fixed. The paper lists them, and discloses that the code and text were written with substantial AI assistance under the author's direction.
• Contamination was not ruled out on the newest benchmark, so the paper makes no claim there.

What this changes for the assumptions section of a spec

The finding matters if you are specifying an AI coding feature: "point it at the failure" is a design assumption, not a settled fact. The method matters for every spec, because it is a worked example of the question in writing down what has to be true.

Ask how often the precondition holds before asking whether the idea works. The repair loop could act on 9.0% of failures, and the ones it could see were not a random sample. Plenty of features rest on the same kind of hidden premise: the user hits this error, the data is already there, the event is tracked. Write the premise down and find out how often it is true.

Name the simplest thing that could explain success, and test against it. Here it was "just try again". In a product it might be a reminder email instead of a redesigned flow, or the release that shipped the same week. If the spec cannot say what the feature has to beat, a positive result will not tell you much, which is the same attribution problem behind metrics that cannot fail.

Write the success rule before the result. The author set a demanding bar, missed it and said so up front. A spec that states "we will call this a failure if…" before launch makes the same move possible, and turns an assumption stated as a fact into one someone can check.

Your instruments are assumptions too. Five pipeline bugs each moved a number. Event definitions, dashboards and experiment harnesses deserve the same suspicion before anyone reads a result off them.

FAQ

What did it test? Whether locating a bug with failing tests helps a code model repair it, against a same-size random edit and a fresh attempt.

Did it help? Not against starting over: 10.1% of fresh attempts succeeded against 4.4% of targeted ones.

Why was localization rare? Only 9.0% of failures had a usable failing public test.

How general is it? Narrow: Python functions, 24 to 32 billion parameter models, and results that relied on a stronger test suite than real systems have.