A strong passive-voice finding shrank once the evidence was re-examined

When a spec cites a study, the reader sees a link and a sentence. They do not see whether the finding has been repeated, challenged or re-analysed since, or whether anyone tried. A paper posted to arXiv on 5 September 2026 offers a vocabulary for exactly that, and its best worked example is a requirements-writing claim plenty of product teams would recognise.

The paper

Seven empirical software-engineering researchers, led by Julian Frattini and funded through research projects at Blekinge Institute of Technology, start from a familiar complaint: single studies rarely settle anything for practitioners, and research synthesis in software engineering is rare and mostly narrative. Meta-analysis, the usual tool for combining studies, works well only when the studies are near-exact replications of each other.

Their proposal is easy to state. A piece of quantitative evidence has three parts:

• A hypothesis — which variables are involved and which is assumed to cause which.
• Data — the observations.
• A method — the analysis that turns the data into a conclusion.

Every follow-up study changes one of the three, and which one it changes decides what it can tell you:

• Replication keeps the hypothesis and method and collects new data. If it agrees, the finding holds in more contexts. If it disagrees, something is missing from the hypothesis.
• Revision proposes a new hypothesis, for example adding a variable suspected of confounding the effect. It improves confidence that the effect is causal.
• Reanalysis keeps the data and hypothesis and applies a better-suited statistical method. It improves the soundness of the statistical conclusion.

The framework encourages changing one part at a time, so that a different result can be traced to a single change.

The claim it was tested on

The authors apply the framework retrospectively to two research threads. The second is one several of them worked on: whether writing a requirement in passive voice makes the domain model built from it less complete. Their example requirement is this one, which leaves unclear who does the obtaining:

All transaction details shall be obtained from the Statement Database.

The thread as published:

• The original study (2014). 15 university students built domain models from 7 requirements each. A significance test gave p = 0.001 and a Cliff's delta of 0.75, and the original authors concluded that passive voice has a strong effect on missing associations.
• A revision and reanalysis (2024). Some of the current authors added variables such as missing entities and participant skill, and analysed the same data with a Bayesian model. Passive voice now caused more missing associations in only 34% of cases, the same number in 49% and fewer in 17%.
• A replication (2025). 25 mostly industrial participants. A comparable significance test under the original hypothesis supported the original finding, with p = 0.025.
• A further revision on that new data. The harmful outcome came out more likely than the 2024 analysis suggested: 45% rather than 34%.

Then the paper turns the framework on its own authors. The 2024 follow-up changed the hypothesis and the analysis method at the same time, so nobody could say which change produced the different answer. Pulling the two apart, the pure reanalysis largely agreed with the original test. That contradicts the 2024 paper's own explanation, which credited the difference to Bayesian statistics.

Taken one step at a time, the best-supported estimate is that passive voice causes more missed associations in 39% of cases, the same number in 47% and fewer in 14%.

The other thread, on whether test coverage predicts how many faults a test suite catches, follows a similar arc. In the original study only 4 of 9 programs showed a correlation. Later studies argued over whether test-suite size confounds the effect: one on 5 Java programs said it did, one on 250 Java projects said it did not. The most recent revision proposes an unobserved factor, the quality the test writers were aiming for, and the paper notes that no empirical evidence was offered for it.

The caveats, from the paper

• Quantitative evidence only. Qualitative findings can prompt a revision but are not part of the synthesis.
• No help choosing studies. Which studies to include in the first place is, the authors say, an ongoing challenge the framework does not address.
• Construct validity is set aside. Conclusions are about the specific measurement, such as missing associations or one coverage metric, not the broader concept of requirements quality or test adequacy.
• It rests on its parts. The framework is assembled from existing methods and is only as valid as they are — a threat the authors judge constrained, since replication, meta-analysis, causal inference and reanalysis are each well supported in their own right.

Two more observations are ours. This is a framework demonstrated on two research threads, not an evaluation showing that people who use it reach better conclusions. And in the passive-voice thread the authors are re-examining their own earlier work. They say so plainly, and the data sits in a public replication package, which is what makes the re-examination checkable.

What this changes for the evidence section of a spec

The paper is written for researchers, but its central move works for anyone citing research in a document someone else has to trust.

Cite the parts, not just the link. "Passive voice makes requirements worse" with a reference is a citation. "15 students, 7 requirements each, missing associations in domain models, a significance test" is evidence a reader can weigh. The same goes for your own research: who, how many, measured how, analysed how. That is the part a citation hides, and it is the argument of the evidence section post.

Say where the finding sits. Has anyone replicated it? Has a later study added a variable that changed the answer? Has anyone re-run the numbers? One study and five agreeing studies are different strengths of evidence, and a spec can say which in a clause.

Change one thing at a time in your own follow-ups. The decomposition applies directly to product experiments. If a second test changes the audience and the success metric at once and the result moves, you will not know why. That is the mistake the authors found in their own 2024 paper.

Expect strong single-study effects to shrink. Not because the first study was careless, but because its hypothesis was simpler than the world. The requirement-smells experiment we covered found declines it could not confirm statistically. Here, a "strong effect" became a lean.

None of this argues for writing requirements in passive voice. Harm stays likelier than help in the best-supported estimate, the interval on that hypothesis still excludes zero, and naming the actor costs one word. It argues for knowing what a cited finding actually rests on before it carries weight in a decision.

FAQ

What does the paper propose? Recording evidence as hypothesis, data and method, and classifying follow-ups as replications, revisions or reanalyses.

What happened to the passive-voice finding? A "strong effect" became more missed associations in 39% of cases, the same in 47% and fewer in 14%.

Is passive voice harmless? No. The effect is smaller and less certain, not gone.

The limits? Quantitative evidence only, no help selecting studies, and a demonstration rather than an evaluation.