Decision records mostly skip the options they rejected

Most writing about design docs, RFCs and decision records is about what they should contain. A study published on arXiv on 7 September 2026 looks at what more than four thousand of them actually contain. The gap sits in one predictable place: teams write down what they decided and leave out what they decided against.

The study

Nicolas Miccio, Antonela Tommasel and J. Andres Diaz-Pace, of the ISISTAN Research Institute (CONICET-UNCPBA) in Tandil, Argentina, analysed architecture decision records. An ADR is a short text note that engineering teams keep alongside the source code to record one decision and its reasoning. Tommasel is also affiliated with Johannes Kepler University Linz.

The records came from an existing dataset of open-source GitHub projects compiled by Buchgeher and colleagues, which covers 2013 to 2020. From 867 projects the team retrieved 6,061 ADRs, then removed duplicates, non-English records, unfilled templates, and projects with fewer than five ADRs or very short ones. That left 4,316 records from 547 projects.

They asked three questions:

• What topics do the records cover?
• How do they map onto established taxonomies of architectural decisions?
• How well do they follow the MADR template, a widely used format whose sections include Context and Problem Statement, Decision Drivers, Considered Options, Decision Outcome and Consequences?

The method is automated. Topics came from BERTopic. Classification and the template checks used GPT-4.1-mini at temperature 0, calibrated against 200 records that two of the authors labelled by hand. Two authors also reviewed a sample of 100 of the template assessments.

What they found

Records mostly say what will exist. Using Kruchten's ontology of decisions, 64% of records were existence decisions: a component, a layer or a technology will be part of the system. Executive decisions made up 18%, property decisions 14%, and decisions that something will not exist were rare.

About a fifth were about process, not architecture. On Zimmermann's classification, nearly 40% were design decisions and about a quarter were technology decisions. Approximately 20% were organisational or process decisions, covering CI/CD, change logs, approval workflows, testing and feature flags. Teams use decision records for how they work as well as for what they build.

The rationale sections were the weak ones. Context and Problem Statement and Decision Outcome were generally present, correctly placed and relevant. Consequences appeared relatively often, but with lower content quality. Decision Drivers were frequently misplaced, tucked into the context or the decision, and often lacked substantive content. Considered Options, which MADR treats as mandatory, was the most problematic. It was absent or misplaced in most records, and when it was present it often failed to do its job.

The authors' conclusion is that rationale and alternatives are under-documented, which weakens the traceability of architectural reasoning. Their advice to practitioners is blunt:

ADRs from open-source repositories should not necessarily be treated as exemplary documentation artifacts.

Teams that copy those examples, they warn, may inherit habits that emphasise the chosen solution and neglect the trade-offs.

The caveats, from the paper itself

• The classifier is only moderately reliable. On the hand-labelled sample, the LLM agreed with human labels between 46% and 66% of the time across the three taxonomies. The authors put part of that down to categories that overlap (a record can be about technology and implementation at once), but it means the percentages above are approximate. The direction of the findings is more trustworthy than any single figure.
• Structure was detected from headings. Records in custom or informal formats may have been misread, and prompt design or project-specific terminology may have affected the outputs.
• One label per record. Each record was assigned its dominant category, which hides secondary concerns, often quality attributes.
• Open source, English, markdown, and old. The data covers public GitHub projects up to 2020, and 40% of the projects date from 2020 alone. Decisions recorded in wikis or issue trackers were not included. The authors call for studies covering 2021 onwards and industrial projects, where practice may differ.
• The reason for the gap is a guess. They suggest time pressure, lightweight habits, or a sense that the rationale already lives in code reviews or issue trackers. They say plainly that these explanations remain hypothetical.

One more detail is worth keeping. The LLM sometimes confused executive and existence decisions because the context needed to tell them apart was not in the record. The authors say a better classifier would need project context that is often outside the ADR text.

Why this matters if you write specs

PRDs, RFCs, design docs and decision records tend to get compared as competing formats. They are better understood as answers to different questions. The PRD says what problem, for whom, and what done looks like. The RFC or design doc proposes how, and argues for it against the other options. The decision record keeps what was chosen and why the alternatives lost. Our comparison of the three covers who writes each and in what order.

Read against that split, this study finds that the question most often left unanswered is the second half of the decision record's job. The chosen option survives. The rejected ones, and the drivers that ruled them out, mostly do not.

That matches what we argued in what belongs in a decision record: the line that does the work is the rejected option with a reason, because "50 — rejected" reopens the question that "50 — billing can't invoice it" closes. It is more useful to see the gap measured across thousands of records than asserted in a blog post, ours included.

Three narrower points for anyone writing or reviewing these documents:

1. Review for the missing alternatives, not the present decision. A decision record or design doc usually reads well because its outcome is clear. The question to ask in review is the one the template already contains and authors skip: what else was on the table, and what ruled it out?
2. A template section is not a filled section. MADR makes Considered Options mandatory, and it was still absent or misplaced in most records. Adding a heading to your PRD or RFC template changes less than reviewing against the heading you already have.
3. Point the record at the context it depends on. This one is our reading, not the authors'. The classifier's confusion suggests that a decision record often does not carry enough about the product or business situation to be read on its own. In product work, that context is what the PRD is for. A record that links to it, kept in a decision log the next person can find, is easier to revisit than one that stands alone.

The authors also suggest reviewing decision records more systematically, for example during pull requests. Product teams make the same move when a decision is written into the spec instead of being left in a thread: someone other than the author reads it while the alternatives are still remembered.

FAQ

What was studied? 4,316 architecture decision records from 547 open-source GitHub projects, classified with an LLM pipeline and checked against the MADR template.

What was missing? Rationale. Decision drivers were often misplaced or thin, and Considered Options was absent or misplaced in most records.

How reliable is it? The LLM classifier agreed with human labels 46–66% of the time, so treat the percentages as approximate.

What should spec reviewers do? Ask what was rejected and why, every time. It is the section most likely to be missing.