Briefing
What moved in AI this week, and what it changes for you.
Model launches, tooling shifts, and the industry arguments worth having — read the way a product team reads them. Not what shipped, but what it changes about how you spec, scope, and review.
Asked for the impossible, code models refused 27% of the time
A benchmark of 270 unsatisfiable coding tasks: models wrote ungrounded code on about 60% and refused 27%. Phrasing alone moved the rate 17 points.
Specs written for agents state the task and rarely the safeguards
AIA study of 1,248 GitHub Agentic Workflow files: in a labelled sample, task, output and constraints appeared in over 93%, prompt-injection defences in 9.4%.
Galang Aulia · 6 min readTwo independent checks agreed on a number more than four times too high
AIA redundancy gate passed seven counts, three wrong; the worst because both paths shared one definition. Asked for an independent check, models reproduced it.
Galang Aulia · 6 min readFor requirements work, the task decided more than the model did
AIA two-part study tested LLMs on five requirements tasks. No model or prompt won across all of them, and generating requirements was where fabrication showed up.
Galang Aulia · 5 min readThe best model found 47% of what the experts found
AIA benchmark of ten models against expert-derived ground truth. Necessity and correctness — the two criteria needing judgment — were almost never detected.
Galang Aulia · 5 min readVaguer requirements tended to produce worse AI code, but not conclusively
AIA controlled study injected defects into requirements before code generation. Pass rates often fell, yet no effect survived correction, and the authors say so.
Galang Aulia · 6 min readThe requirement was met and the outcome was still wrong
AIA requirements-engineering paper argues the moment an agent should stop and ask a human is a design-time requirement, not a runtime heuristic.
Galang Aulia · 5 min readCoding agents mostly read documentation written for coding agents
AIA study of 557 agent sessions finds agent-facing files are 60.5% of documentation interactions and classical docs 10.6%. Docs also trail code.
Galang Aulia · 5 min readMost teams measure AI by time saved, which is the half that looks good
TrendsA survey of 250+ product professionals finds time saved is the standard AI metric. Other research says the saved time reappears downstream as verification.
Galang Aulia · 4 min readAgents commit early, and the commitment compounds
ToolsA paper separates reasoning from planning: step-wise scoring makes early choices that amplify and cannot be undone. Specs have the same failure.
Galang Aulia · 4 min readEurope published a new accessibility standard that is not yet the legal one
IndustryEN 301 549 v4.1.1 moves to WCAG 2.2, but the binding reference is still v3.2.1. Two standards, one of them enforceable — your spec has to say which.
Galang Aulia · 4 min readThe benchmark utility gap has three named failure modes
TrendsA paper on AI evaluation names three ways a rising score hides a flat outcome. All three have exact equivalents in the metrics section of a PRD.
Galang Aulia · 4 min readSpec-driven development has a clarify step and no way to run it
IndustryMicrosoft made the spec the source of truth for AI agents. Step three of its own lifecycle is the one nothing in the toolchain can do for you.
Galang Aulia · 5 min readAmbiguous requirements make the strongest models fail hardest
AIA new benchmark injects four kinds of ambiguity into 1,304 coding tasks. Every model degrades, none reliably locates the ambiguity, and none asks.
Galang Aulia · 5 min readDevelopers Felt 20% Faster. They Were 19% Slower.
TrendsA controlled trial found developers slowed down with AI while believing they had sped up. The cost did not vanish — it moved somewhere nobody measures.
Galang Aulia · 4 min read