Coding agents mostly read documentation written for coding agents

A paper posted to arXiv does something unusual for the "how to write docs for AI" genre: instead of asserting what agents need, it watched what agents did.

The result complicates a premise a lot of tooling is currently built on.

What was measured

The authors analysed two datasets of real agent activity:

• 557 agentic coding sessions from SWE-chat, yielding 94,813 development events, of which 3,033 were documentation interactions
• 33,097 agentic pull requests from AIDev, yielding 690,260 file-level change records

They classified which documents agents opened, at what point in a session, and in what order relative to code changes. This is behavioural: it records what happened, not whether the documentation was any good.

The four findings

Agents mainly read documentation written for agents. Agent instruction files and agent working notes together account for 60.5% of all documentation interactions. Classical technical documentation — the README, the guides, the architecture notes — accounts for 10.6%. API references account for 1.3%.

The link between reading docs and writing code is weaker than assumed. The paper reports an adjacent transition probability of 0.002 between consultation and code editing, an unadjusted three-event lift of 1.05, and a stage-adjusted odds ratio of 1.33. The authors describe the association as unresolved rather than established — a conclusion a promotional write-up would not have reached.

No documentation-driven verification was observed. The authors found no explicit documentation-based validation sequence, and consultation was associated with less immediate testing, not more (adjusted OR 0.39, interval 0.25–0.60). Agents did not read the spec and then check their work against it.

Documentation trails code. Consultation is self-initiated in 70.2% of cases and failure-driven in 7.5% — agents consult by habit at the start rather than when stuck. And among multi-commit pull requests that changed both code and documentation, code was touched first 4.7 times more often.

From this the authors derive a two-lobed cycle — a read-ish phase and a write-ish phase that recur — rather than the linear read-then-build-then-verify journey the tooling narrative assumes.

What this is, and what the authors refuse to claim

An observational trace study. Associations, not causes, and the paper is disciplined about the distinction. Three specific reasons for caution:

The document-type classifier was not human-validated — the authors state this. Since the headline 60.5% / 10.6% split is a classification result, that caveat sits directly under the most quotable number in the paper.

It is an arXiv preprint, not yet through peer review.

Most notably, the paper contains a section titled "implications our data do not support". In it, the authors report that two properties almost universally recommended for "agent-friendly" documentation — actionability and verifiability — lack consistent behavioural support in this corpus. Not disproven; unsupported by the observed behaviour.

Publishing your own negative results is rare enough to be worth naming, and it is the main reason this paper is more trustworthy than the genre around it.

Why this is uncomfortable for the spec-driven story

Worth being straight about: this cuts against a framing covered here previously.

GitHub's spec-driven development material argues for treating the spec as the durable artefact and the code as its output, with an explicit clarify step to resolve ambiguity before generation. The reasoning is good and the clarify step is, I still think, the most useful part of it.

But this study suggests something awkward about the reading end. In practice agents spent most of their documentation attention on their own instruction files and working notes, not on human-authored documentation — and when they did consult, no verification sequence followed. The premise that a well-written spec functions as a reference the agent returns to and checks against is not what the traces show.

That is not a refutation. The corpus is general agentic coding, not teams deliberately practising spec-driven development with a maintained spec in the repo — a population this dataset barely contains. But it does mean the mechanism is currently assumed rather than observed, and anyone selling it should say so.

What actually follows

If agents read agent-facing files far more than prose documentation, the practical response is not "write better prose."

Put the constraints where they get read. If instruction files and working notes are 60.5% of attention, that is where the non-negotiables belong — the limits, the permission rules, the things that must not be invented. Not only in the spec.

Stop expecting the document to be the verification. Nothing in this data shows an agent checking its output against documentation. If a requirement has to hold, it needs a test, not a sentence — which is the same conclusion arrived at from a different direction entirely.

Expect documentation to lag. Code first, 4.7× more often. If your process assumes the spec is updated alongside the change, the data says it will not be, and the correction has to be a step somebody owns rather than an expectation.

Front-load, because consultation happens early. Consultation is self-initiated at the start rather than triggered by failure. A constraint that appears late in a document arrives after the approach is chosen — which is the same shape as the early-commitment problem in agent planning.

The part that should land

There are now two reader classes for a spec, and they behave differently in ways that are measurable rather than imagined.

The human reader skims, asks a question in a thread, and remembers the conversation. The agent reads its instruction file at the start, works, and does not come back to check. Writing one document and assuming it serves both is the assumption this paper makes visible — and the honest position is that nobody yet knows what the agent-facing half should look like, including the authors, who say as much.

FAQ

What was measured? Behaviour across 557 sessions, 94,813 events, 3,033 documentation interactions, plus 690,260 file-level change records.

Headline finding? Agent-facing files are 60.5% of documentation interactions; classical documentation 10.6%; API references 1.3%.

Do agents read docs when stuck? Rarely — 70.2% self-initiated versus 7.5% failure-driven, and code precedes documentation 4.7× in mixed pull requests.

Caveats? Observational, classifier not human-validated, preprint, and the authors flag actionability and verifiability as unsupported by their data.