Success criteria must be defined before testing
Aliases: completion criteria · operationalized success · a priori success rule
What it is
Task success rate scores each attempt against an observable endpoint. The scoring rule has to be locked before that attempt is seen: what counts as the end state, how much assistance is allowed, and when the attempt is abandoned or timed out. Recoding after the fact because someone “pretty much finished” turns success into advocacy for a preferred design. The rate answers how many people reached a stated standard, not whether the team feels they mostly got it.
Why it happens
Success is not a natural property of an interface; it is a cut the researcher places on a behavioral trace. If the cut can move after the recording is watched, scorers push ambiguous cases toward the design they already believe in: the favored variant gets a lenient “submitted,” the control gets failure for the same state. Writing an observable state in advance—account created and email verified, not “the user understood the flow”—is what lets independent scorers land on the same side of the same records. Once the rule is rewritten case by case, rates from earlier sessions are no longer the same metric.
Studying it
Before the pilot, write success, failure, stopping rules, and whether outside help is allowed for each task. Have a second scorer who did not design the interface apply the same rule blindly to several sessions and locate where disagreements cluster. Hold the rule fixed when comparing versions; do not rewrite it because a new UI created a new “almost done” path. Publish the rule with the rate so readers know which kind of arrival the number records.
Where it stops holding
Exploratory tests may uncover a legitimate completion path that the original definition missed. That is a revision of the operationalization; later confirmatory sessions must use the new rule and must not be pooled with the old rate. Safety-critical success sometimes includes “did not take a dangerous action,” which is not the same as reaching a target control. Collaborative endpoints may span people and days; a single session can operationalize only a segment. Unobservable “understanding” is not a success criterion.
Applying it
- Write each task as “reaching which interface state or external result counts as success”; ban words such as “smoothly” or “basically done.”
- Put the rule in the moderator script; if a dispute arises in session, score by the original rule and debate revisions only for the next round.
- Keep the same end state across redesign comparisons; if the product endpoint changed, report old and new rates separately rather than as one trend.
- Spot-check recordings: if a scorer cannot decide from the rule alone without seeing the variant label, the criterion is still not observable.