Meeting a standard is not evidence that people can complete the task

What it is

The checker is all green, Tab still works, the contrast numbers clear the line. A screen-reader user still gets lost at the third step of payment: every step sounds like “button,” errors are read as a string of digits, the success state never enters the speech stream. Conformance answers “were the sampled criteria met.” Usability answers “can this person finish this job in this situation.” The two scripts are not the same exam.

An ergonomics process can also finish and still be unusable — the “users” were two interns, the task was opening the home page rather than placing an order. Checklists and archives are not a road test.

Why it happens

Outcome criteria are necessary conditions plus a sample. Contrast, a name, keyboard reach: miss any one and the task can die, which is why they are worth writing as repeatable checks. They do not model the whole task: they do not measure too many steps, wording that misses this person’s model, live updates that never enter speech, a third-party widget the sample happened not to draw. A tool score displays “no known failure pattern found” as a pass. That is not the same as “success was found.”

Process standards hollow out the same way: activity names are present, participants are not the target users, the evaluation task is not the critical path, findings never flow back. The archive can prove “something correctly shaped was done.” It cannot prove “it holds for the people who will use it.” ISO 9241-11 locates usability in effectiveness, efficiency, and satisfaction in a context. Neither WCAG conformance nor process conformance emits those three by default.

The second layer is a mismatch of objects. Conformance’s object is a product (or sampled pages) against a table of criteria. Usability’s object is “this class of people × this class of tasks × this class of environments.” Different objects do not transfer a pass.

Studying it

On a version already claimed as conformant, run assistive-technology task tests on the critical path: the same order / booking / submit, on the person’s own device and reader (or switch, or voice). Record completion, where they stop, and whether the stop is a criterion the standard never covered or a criterion that was covered and still unusable in combination.

Independent variables: presence of a conformance report, whether the task is a real critical path, whether participants are target users of that task on their own AT. Dependent variables: task completion rate, time, count and coded cause of failures that still occur on steps the conformance report had ticked.

Expert review and automation are useful inputs to conformance. They do not substitute for task success. Samples can be small; write the claim as “these users failed these tasks,” not “therefore conformance is invalid.” Conformance can still be necessary. It is not sufficient.

Where it stops holding

In high-risk, narrow-task professional systems with trained users, passing an industry list sometimes sits very close to “usable,” because the standard’s assumed context already froze the task and the user. Consumer products, with diverging users and situations, open the widest gap. If a conformance sample happens to cover every critical path and includes AT-user tasks, the gap shrinks — that is a change in evaluation design, not conformance magically becoming usability. What the law often wants is conformance evidence. That does not stop a product from setting a task-success gate on top.

Applying it

• After the conformance report passes, list a separate AT-user pass mark on critical tasks (completed, no rescue). Missing that mark is not a ship.
• Code failures: a criterion that was not actually met, versus criteria all met and the flow, wording, or live update still breaking the task. Do not send the second kind back to “run the checker again.”
• Participants and tasks in process evaluation must overlap the delivery audience. Interns opening the home page must not be written up as a usability result.
• How to check: ask at least two people on their own assistive technology to finish checkout or an equivalent critical path. If the checker is green and the task is not, write “conformant” and “usable” as two lines in the release record. One score must not stand for both.

Related

• Same group: J1.03.1 Ergonomics standards often bind the process, not only the finished interface · J1.03.2 A legal citation of a standard is a citation of a dated version
• Nearby: J1.04 Legal requirements · J5.08 Limits of automated checking · J5.14 Testing with disabled users
• Search terms: conformance vs usability · ISO 9241-11 · task success