An AI code reviewer at Ericsson was mostly right, which was only half the test

Most evaluations of AI review ask one question: were the findings right? A paper from Blekinge Institute of Technology and Ericsson, posted to arXiv on 14 September 2026, asks a second one, and its answer is the more useful of the two for anyone who runs a review. The authors put it plainly:

A correct finding is not necessarily a valuable one.

The study is small, its builders evaluated it, and the raters wrote the code under review. With those limits in view, it is still a clear account of how to organise a review whose output can be acted on, for a spec or a prototype as much as for a commit.

The study

Four of the five authors, Muhammad Laiq, Ricardo Britto, Muhammad Usman and Deepika Badampudi, are at Blekinge Institute of Technology in Sweden. Britto also lists Ericsson, and the fifth author, Nishrith Saini, is at Ericsson. The paper is accepted at PROFES 2026, the 27th International Conference on Product-Focused Software Process Improvement. They followed the Design Science Research Process: build an artefact, then evaluate it in a real setting.

The system splits code review into four agents, one per quality dimension: readability, maintainability, reliability and performance. Each has its own prompt, review instructions and catalogue of antipatterns taken from earlier literature. Magic numbers and deep nesting for readability; duplicated code and feature envy for maintainability; empty or over-broad exception handlers for reliability; repeated database queries and excessive I/O for performance.

The context is the part the authors stress. Instead of handing each agent the diff alone, a context builder pulls the changed files and metadata from GitLab, adds dependency and structural information from a code knowledge graph, and gives each agent a view tailored to its dimension. An orchestrator built on Kiro CLI, with tools connected over the Model Context Protocol, chooses the agents, runs them in parallel and merges their output into one report.

The output is fixed. Every finding has four fields: the antipattern's name, its location, a description of the problem and a suggested fix.

The evaluation ran the system on seven commits from four Ericsson projects, two small and two large, all in Python. It produced 206 findings. The five senior developers who wrote those commits rated each finding as correct or incorrect, and each correct one as minor, medium (should be fixed) or high (must be fixed). Four of them rated one commit each. One rated three commits, which accounted for 96 of the 206 findings.

What they found

Most findings were real. The developers judged 197 of 206 correct, approximately 96%, and 9 incorrect.

Fewer were important. Of the 197 correct findings, 65 (33%) were rated high, 70 (36%) medium and 62 (31%) minor. The authors count the first two together, about 69%, as worth fixing, and say the minor ones, often readability and style, still have some long-term value.

The lenses found different things. Findings were spread across all four dimensions: reliability 64 (31.1%), readability 58 (28.2%), maintainability 50 (24.3%) and performance 34 (16.5%). Each agent contributed findings the others did not, and the authors estimate that a review aimed at a single dimension would have covered at most about a third of what was reported.

The plumbing mattered as much as the agents. The orchestrator is a single point of failure: a mistake in choosing agents, building context or merging output reaches the final review however well each agent reasons. Constraining the output, with positive and negative examples, is what made it feasible to assess 206 findings one by one. In the authors' words, "Output format is part of the design, not a presentation detail."

The caveats

Several limits, most of them the authors' own, change how far the headline numbers travel.

• No ablation. The system was not tested without its context, or with one general agent in place of four. The authors describe their design observations as feasibility evidence rather than controlled comparative proof. Crediting the context for the low rate of wrong findings is their interpretation, not a measured effect.
• One company, seven commits, one language. Four projects, Python only. The authors say the results may not generalise to other organisations or languages.
• One model. Everything ran on Kiro CLI's default model. Other models or frameworks may score differently.
• Raters judged their own code. The authors note this can bias ratings either way, towards leniency or towards extra scrutiny, while arguing that a change's author is best placed to judge it. Our addition: with one developer rating 96 of the findings, one person's judgement carries a large share of the result.
• Accuracy covers the findings, not the misses. This one is ours, not the authors'. None of the three research questions asks what the agents failed to report. The 96% says the findings were mostly real. It says nothing about issues no agent raised.
• Hard to replicate. LLM runs are non-deterministic, and the commits and generated reviews are proprietary, which the authors acknowledge limits independent replication.

The fair reading: a field report on the authors' own tool in one setting, with design advice that is plausible but untested.

Why this matters if you review specs or prototypes

The paper's design choices are about how a review is organised, and they carry over to specs.

Give each reviewer a lens. "Have a look at the PRD" asks for one undifferentiated pass. The paper's agents each had a named dimension and a checklist for it, and each found things the others did not. The equivalent for a spec is asking one reviewer to walk the unhappy paths and another to check permissions and limits, with the checks written down rather than remembered.

Review the change with its context. The agents were given the surrounding system, not just the diff. A prototype screen is a diff: it shows the change and hides what it depends on. That is why the prototype review post pins every prototype to a spec version and tags each screen with the section it renders before anyone comments. A reviewer clicking through without the spec is reviewing the change alone. The authors credit context with keeping their agents' comments grounded, though they did not test that. It is the same gap that makes reviewing someone else's PRD hard: the reviewer's missing context.

Make every finding name a location and a fix. The four-field format is what let the authors evaluate 206 findings individually. A spec comment that names its section, states the problem and proposes a rewrite can be accepted, rejected or argued. "This feels off" can only be acknowledged. That is most of the argument in what makes a PRD critique useful.

Rank, because right is not enough. 31% of the correct findings were rated minor. A review that lists every correct observation at equal weight spends the author's time evenly on things that matter unequally. Severity is what turns the list into an order of work. Paired with saying whether each comment blocks, as the study of stalled pull requests recommends, it tells the author what to do first.

If you are trialling an AI reviewer

The evaluation template is worth borrowing. Run the tool on a few of your own documents and have each author mark every finding correct or incorrect, then minor, should-fix or must-fix. It is cheap, and it answers both of the questions this paper asks.

It also inherits the paper's blind spot: it measures what the tool said, not what it missed. For misses you need a document whose gaps you already know, such as an old spec and the problems found after it shipped. What AI still can't catch covers the gaps a reviewer reading only the document cannot see.

FAQ

What was tested? Four specialised AI review agents with repository context, on seven Ericsson commits that produced 206 findings.

How accurate was it? Developers confirmed 197 of 206 findings, about 96%. Misses were not measured.

Were the findings worth fixing? About 69% were rated should-fix or must-fix; 31% were minor.

What carries over to spec review? Named lenses, context alongside the change, findings with a location and a fix, and a severity on each.