Acceptance Criteria for Non-Deterministic Features

Given-When-Then rests on an assumption so basic nobody states it: the same input produces the same output. Run the test twice, get the same result, and the criterion means something.

A model does not promise that. Run the same input twice and the wording changes; sometimes the answer does. A criterion like Then the summary mentions the refund amount can pass on Monday and fail on Tuesday with nothing changed.

So for the part of a feature a model decides, the acceptance criterion stops being a statement about one output and becomes a statement about many. Over a fixed set of cases, at least this share must pass this check, no segment may fall below this floor, and these outcomes must never occur. That is an eval. Written into the spec with a threshold and an owner, the eval is the acceptance criterion.

Split the feature first

Most of an AI feature is deterministic. The button, the loading state, the abstention message, the escalation routing, the fallback when the model API times out. Those take ordinary testable acceptance criteria, and they should, because they are where most user-visible bugs live.

Given the model returns no suggestion within 3 seconds, When the ticket is created, Then it enters the normal queue with no suggested category.

That criterion belongs to an AI feature and is fully deterministic. The states that surround a model, covered in the states nobody names, are mostly specified this way. Only the model's own output needs the new form. Splitting first keeps the eval small and keeps the rest of the feature testable in the usual way.

Six parts of a criterion for model behaviour

1 · A fixed evaluation set. A named, versioned collection of inputs, each with the expected result. Representative cases plus the hard ones: the inputs the feature should abstain on, the ones it must refuse, the segments you worry about. A release is judged against a named version. You can add cases later; you cannot swap the set after seeing the scores.

2 · A grading rule per case. What counts as a pass for one case. A match against a label, a short checklist a person applies, or an automated check. If people grade, have two of them grade the same sample first. Where they disagree, the rule is the problem, not the model.

3 · A threshold, set before you look. The minimum pass rate over the set, stated next to a baseline: the current system, or people doing the same task. A threshold chosen after the results arrive is a description of the results. A target with no baseline has the problem covered in the metric missing its baseline.

4 · Hard lines. Outcomes that fail the release on a single occurrence, whatever the pass rate. A commitment only a person can make, a category the model must never assign, personal data in an output. These are not percentages. Ninety-nine out of a hundred is a failure.

5 · Floors per segment. An average can pass while one language, customer tier or input type fails badly. A paper on AI evaluation calls this distributional concealment. The fix is a floor for each segment you care about, stated alongside the overall threshold.

6 · Runs. How many times the set is run, and how results are combined. A benchmark on requirements review found single-run evaluations statistically unreliable. The same logic applies here: one run is a sample. Say whether the criterion must hold on every run or on the average of several.

One criterion, written out

An illustration, continuing the support-ticket categoriser from the states post. The numbers show the shape and are not recommendations.

Evaluation set: tickets-eval v3. 400 tickets sampled from last quarter, labelled independently by two support leads, disagreements resolved by a third. Includes 40 non-English tickets and 30 with no body text.

Grading: a case passes if the suggested category matches the agreed label. Abstaining on a ticket labelled "unclassifiable" counts as a pass.

Threshold: at least 85% of cases pass. Baseline: [TBD: agreement between the two labellers before resolution, measured on the same 400 tickets, owner: support ops, due 20 Oct].

Floors: no language below 75%.

Hard lines: zero tickets auto-assigned to "legal" or "security incident".

Runs: three runs. The criterion is met only if all three meet the threshold and floors.

Owners: support operations lead owns the set, threshold and hard lines. The ML engineer owns the harness.

The line worth noticing is the baseline. If the two support leads only agree with each other on, say, 80% of tickets before resolving disagreements, then an 85% target for the model is already above the consistency of the labels. That is not a reason to lower the bar. It is a reason to fix the category definitions before anyone tunes a prompt.

Who owns it

An eval without an owner decays the same way an unowned document does, only faster.

The PM owns the threshold, the hard lines and the segments. They are decisions about acceptable error, which are product decisions. Left to whoever builds the harness, they become whatever the first run achieved.

Domain experts own the labels. The grading is only as good as the agreement on what "right" means.

Engineering owns the harness. Running the set, recording results per version, making a run cheap enough that people actually do it.

One named person owns additions. Every failure that reaches a user becomes a case in the next version of the set. Without that, the set describes the feature as it was imagined at launch.

Where these criteria go wrong

Tuning against the set. If the prompt is iterated until the set passes, the set now measures the tuning. Hold back part of it, and only look at the held-back part at release.

Easy cases only. A set built from typical inputs never exercises the ones that break the feature. The states you specified are the cases most worth including.

Only the average. Covered above, and the most common way a passing eval ships a failing feature.

Graders who drift. If people grade, the rule they apply on the four-hundredth case is not always the rule they applied on the first. Re-grade a handful of early cases at the end of the run. If the answers differ, tighten the grading rule before trusting the score.

No link to the success metric. The eval says the feature is ready. The success metric says it worked. Passing one says nothing about the other, and a spec with only the first has no way to learn it was wrong.

A set that never changes. Six months later it tests a product that no longer exists.

A template for it

We have published an Acceptance Criteria template that carries both halves: Given-When-Then scenarios and non-functional thresholds for the deterministic parts, and a model-behaviour section with fields for the set, grading rule, threshold, floors, hard lines, runs and owner. The AI Feature PRD template is where the eval plan sits in the wider spec.

Our generator asks about the brief before it drafts, and one of the critique's eleven passes, AI-Specific Readiness, applies to documents that describe model-driven behaviour. Neither can choose your threshold. That decision stays with the person accountable for the error rate.

FAQ

How do you write acceptance criteria for an AI feature? Split it. Given-When-Then for the deterministic parts, an eval with a threshold, floors and hard lines for the model's output.

What pass rate? No universal number. Set it against a baseline, before you look.

Who owns the set? PM owns thresholds, experts own labels, engineering owns the harness, one person owns additions.

Eval or success metric? Both. The eval decides readiness; the metric decides whether it worked.