AI
Better documentation did not help coding agents that could already read the code
A study optimised file descriptions for coding agents. With the source in view, none beat the issue alone. What that implies for what goes in a ticket.
Most advice about handing work to a coding agent says the same thing: give it more context. A paper posted to arXiv on 25 September 2026 built the best documentation its authors could, showed that it was good, and then found it made no difference to an agent fixing real issues, as long as the agent could already read the code.
The negative result is the headline. The more useful part is where the authors draw the line between documentation that helps and documentation that doesn't.
The study
The paper is by Md Shohel Arman of Daffodil International University in Dhaka and Igor Molybog of the University of Hawai'i at Mānoa. It is a preprint and has not been through peer review.
It has three parts:
- A roundtrip benchmark. One model writes a natural-language description of a source file. A separate model regenerates the file from the description alone. The file's original tests score the result. The fixtures are eleven files from SWE-bench Verified, all from the sympy project, plus three small hand-built codebases.
- An optimiser. A loop rewrites the description-writing prompt and keeps any change that raises the benchmark score, with a small penalty for length.
- A transfer test. Do those descriptions help an agent resolve real repository issues? The authors tested this across two model families and ten repositories, on their own fixtures and on an external benchmark, SWE-ContextBench.
What they found
Documentation helped only when it said something the code didn't. In one hand-built codebase, an access-control engine applies rules in whatever order they arrive. The intended rule, that higher priority wins and a deny beats an allow at equal priority, appears only in the description. With the description, a deliberately weak model went from zero of three attempts to three of three, on each of the three codebases built this way. When a description merely restated code the agent could read, it added nothing.
Completeness mattered; length didn't. When a description carried one, three or five of five required facts, the agent recovered exactly one, three or five. A 38-word description and a 664-word one carrying the same five facts gave identical results. An apparent length penalty in a harder version turned out to be missing information: the long description had never written out the exact key names the task needed.
Describing what code does is not describing what it should do. The authors compared their descriptions with OpenWiki, LangChain's open-source tool that generates repository documentation for coding agents. On the access-control task, OpenWiki correctly reported that the engine ignores priority. That is the observed behaviour, and the opposite of the rule the task required. With OpenWiki's documentation the agent failed all four attempts, exactly as it did with none; with the authors' description it passed all four.
A tool that documents what the code does cannot supply a contract the code does not exhibit
With the source hidden, good descriptions transformed the results. When the agent had to produce a file it could not see, the mean fraction of tests passed rose from 0.08 with the issue alone to 0.21 with baseline descriptions and 0.71 with optimised ones.
With the source available, nothing beat the issue alone. On the full SWE-ContextBench Lite run with Gemini 3.8 Flash, across the 58 tasks scored under every condition, the issue alone resolved 33, the benchmark's own retrieved past-task context 30, and a roughly eighty-word description of the target file 29. The differences were not significant (McNemar's exact test: p = 0.22 for the description, p = 0.25 for retrieved context), and they all pointed the same way. On a django subset with Gemini 3.1 Pro, the issue alone resolved 9 of 15, the compact description 8 and the full-length description 4. The authors' explanation: a complete description of a file the agent can already read invites it to rewrite rather than patch.
The caveats, from the paper and from us
The authors set out their limits plainly:
- A small benchmark. Eleven fixtures plus three hand-built codebases, one language, one project family: numbers that characterise the method, in their words, rather than powered measurements.
- A narrow headline run. One economical model and one description length of about eighty words; the stronger-model replications covered django subsets only.
- Attrition that isn't random. Tasks where any condition produced no edit were dropped, and the documentation conditions produced no edit slightly more often.
- A proof-of-concept optimiser, checked on two held-out files.
- Possible training exposure, which their contamination controls reduce but cannot rule out.
Two points of our own. The positive control the abstract leans on is thin. Retrieved context beat the issue alone by one task, 15 to 14 out of 29: the same one-task margin the authors call indistinguishable from zero when their compact description managed it, 16 to 15 out of 32. The null result doesn't depend on the control, but one task is not much of a demonstration that the evaluation can detect an improvement.
And the null is conditional. The authors say their issues are well specified, with everything needed already in the issue and the source, so a file description is redundant by construction. That is as much a finding about the issues as about the documentation.
Why it matters if you turn specs into tickets
The paper studies code documentation for agents, not product specs or tickets. But its central distinction maps onto a decision product teams make every sprint: what goes in the ticket, and what stays in the spec.
A restatement is not context. In the authors' own summary, "a restatement adds nothing when the original is already in context." A ticket that pastes the PRD's requirements into its body is a restatement. In this study the fullest restatement was the worst condition wherever it ran. Whether human engineers react the same way is our inference, not the paper's finding; what is certain is that a copy drifts the first time someone edits one version and not the other. That is the argument for tickets that link to requirements instead of restating them.
The ticket's job is the part nobody can see. The access-control rule is the template. Nothing in the existing system showed it; only the words did. In a ticket for a change, that is the rule the current code doesn't enforce yet, the exact threshold, the error message, the precedence when two conditions conflict. A ticket that accurately describes current behaviour can still point the builder the wrong way, as OpenWiki's documentation did.
Complete beats long. The optimiser converged, without being told, on instructions to state exact imports, literal constant values, exact signatures and defaults, and the conditions for every exception and return path. Translated for a ticket: the specifics a test would check, not more paragraphs around them.
The issue did the work because it was good. The result is not that context is useless; it is that a well-specified issue plus readable code was enough. That fits the finding that agents mostly read documentation written for agents rather than the classical kind. Extra material helps when it supplies what the reader can't see, and gets in the way when it repeats what they can.
What to take from it
Not "documentation doesn't matter". With the code out of view, a complete description lifted the pass fraction from 0.08 to 0.71.
The narrower point holds up. Before adding anything to a ticket, ask whether whoever builds it can already see it. If they can, link to it. If it is a rule or a threshold that lives only in the spec, that is what the ticket is for.
FAQ
What was tested? Whether optimised descriptions of source files help coding agents resolve real issues, across two model families and ten repositories.
Did they help? Only with the source withheld. With it present, the issue alone did as well or better.
What documentation did help? Documentation that stated a rule the code did not show.
The lesson for tickets? Carry what the builder can't see; link to the rest.