Briefing
What moved in AI this week, and what it changes for you.
Model launches, tooling shifts, and the industry arguments worth having — read the way a product team reads them. Not what shipped, but what it changes about how you spec, scope, and review.
Decision records mostly skip the options they rejected
A study of 4,316 open-source decision records found the decision usually written down, and the considered options usually missing or misplaced.
Synthetic survey respondents got the direction right and the size wrong
AIThree open-weight models tracked human cross-instrument correlations at r = 0.70 to 0.73, while missing each correlation by 0.19 to 0.32 on average.
Galang Aulia · 6 min readAI-moderated interviews found as many customer needs as human ones
AIA pre-registered study of 317 interviews: AI moderation matched humans on needs found per interview, but richer transcripts did not predict reactions better.
Galang Aulia · 6 min readPointing a code model at the bug lost to letting it try again
AIA placebo-controlled test of fault localization for AI code repair: usable on 9.0% of failures, and beaten by simply generating a fresh attempt.
Galang Aulia · 5 min readHow often the problem occurred depended on how it was defined
TrendsOne phone-use dataset, three definitions of the problem: 51.4%, 28.3% or 14.9% of sessions. What a measurement study means for problem statements.
Galang Aulia · 5 min readA strong passive-voice finding shrank once the evidence was re-examined
TrendsA framework for tracing how evidence evolves, applied to a familiar requirements claim. The strong passive-voice effect became a modest lean once re-examined.
Galang Aulia · 6 min readFigma's own trial found prompt-to-design saved PMs the most time
ToolsA randomised trial of Figma Make with 50 designers and 50 PMs found about 20% faster completion, mostly for PMs, on tasks where the target design was given.
Galang Aulia · 6 min readAn AI code reviewer at Ericsson was mostly right, which was only half the test
AIDevelopers confirmed 197 of 206 issues raised by review agents at Ericsson and rated 31% of them minor. What the design and its limits mean for spec review.
Galang Aulia · 6 min readReviewers approved more AI code over time, and their comments barely showed it
AIAcross 11,429 reviews of AI-agent pull requests, approval rates rose with reviewer exposure while standard measures of comment effort stayed flat.
Galang Aulia · 6 min readPull requests mostly stall on who acts next, not on the code
TrendsA study of 14,234 stalled pull requests found coordination breakdowns, not technical faults, behind most stalls. The fix it proposes is a named next action.
Galang Aulia · 6 min readA wrong AI summary changed what people remembered seeing
AIIn a controlled experiment, one misleading detail in an AI summary cut correct recall from 83.6% to 44.8%. Knowing the summary came from AI did not help.
Galang Aulia · 6 min readLLM peer reviews argued more and pointed to fixes less than human ones
AIIn ICLR 2025 reviews, humans more often flagged higher-order weaknesses and pointed to fixes. An expert prompt made the model sound expert, not human.
Galang Aulia · 6 min readHallucination checkers scored near random on invented claims in peer reviews
AIA benchmark seeded false claims into 38K peer reviews. Off-the-shelf verifiers were near random, and prompted LLMs did better per sentence than per review.
Galang Aulia · 5 min readThe LLM judge found the bug and gave it a perfect score
AINone of 24 default judge setups in eight eval frameworks used the one defence shown to work. Applied, it made the eval only as good as the judge.
Galang Aulia · 6 min readAsked for the impossible, code models refused 27% of the time
AIA benchmark of 270 unsatisfiable coding tasks: models wrote ungrounded code on about 60% and refused 27%. Phrasing alone moved the rate 17 points.
Galang Aulia · 6 min readSpecs written for agents state the task and rarely the safeguards
AIA study of 1,248 GitHub Agentic Workflow files: in a labelled sample, task, output and constraints appeared in over 93%, prompt-injection defences in 9.4%.
Galang Aulia · 6 min readTwo independent checks agreed on a number more than four times too high
AIA redundancy gate passed seven counts, three wrong; the worst because both paths shared one definition. Asked for an independent check, models reproduced it.
Galang Aulia · 6 min readFor requirements work, the task decided more than the model did
AIA two-part study tested LLMs on five requirements tasks. No model or prompt won across all of them, and generating requirements was where fabrication showed up.
Galang Aulia · 5 min readThe best model found 47% of what the experts found
AIA benchmark of ten models against expert-derived ground truth. Necessity and correctness — the two criteria needing judgment — were almost never detected.
Galang Aulia · 5 min readVaguer requirements tended to produce worse AI code, but not conclusively
AIA controlled study injected defects into requirements before code generation. Pass rates often fell, yet no effect survived correction, and the authors say so.
Galang Aulia · 6 min readThe requirement was met and the outcome was still wrong
AIA requirements-engineering paper argues the moment an agent should stop and ask a human is a design-time requirement, not a runtime heuristic.
Galang Aulia · 5 min readCoding agents mostly read documentation written for coding agents
AIA study of 557 agent sessions finds agent-facing files are 60.5% of documentation interactions and classical docs 10.6%. Docs also trail code.
Galang Aulia · 5 min readMost teams measure AI by time saved, which is the half that looks good
TrendsA survey of 250+ product professionals finds time saved is the standard AI metric. Other research says the saved time reappears downstream as verification.
Galang Aulia · 4 min readAgents commit early, and the commitment compounds
ToolsA paper separates reasoning from planning: step-wise scoring makes early choices that amplify and cannot be undone. Specs have the same failure.
Galang Aulia · 4 min readEurope published a new accessibility standard that is not yet the legal one
IndustryEN 301 549 v4.1.1 moves to WCAG 2.2, but the binding reference is still v3.2.1. Two standards, one of them enforceable — your spec has to say which.
Galang Aulia · 4 min readThe benchmark utility gap has three named failure modes
TrendsA paper on AI evaluation names three ways a rising score hides a flat outcome. All three have exact equivalents in the metrics section of a PRD.
Galang Aulia · 4 min readSpec-driven development has a clarify step and no way to run it
IndustryMicrosoft made the spec the source of truth for AI agents. Step three of its own lifecycle is the one nothing in the toolchain can do for you.
Galang Aulia · 5 min readAmbiguous requirements make the strongest models fail hardest
AIA new benchmark injects four kinds of ambiguity into 1,304 coding tasks. Every model degrades, none reliably locates the ambiguity, and none asks.
Galang Aulia · 5 min readDevelopers Felt 20% Faster. They Were 19% Slower.
TrendsA controlled trial found developers slowed down with AI while believing they had sped up. The cost did not vanish — it moved somewhere nobody measures.
Galang Aulia · 4 min read