Passing an automated check does not mean the interface is usable
Aliases: auto-pass is not usable · Lighthouse score · false-confidence compliance
What it is
Lighthouse at one hundred, axe at zero violations, and checkout can still be impassable in a screen reader. An automated pass proves that the decidable subset did not catch a failure, not that a person can finish the task with their assistive technology. A green score is not a usability result.
Why it happens
Automation answers “did these computable predicates break.” Usability answers “can the intended user finish the primary task on their AT.” The two predicates have different extensions. A pass on the first only excludes a small class of defects; a keyboard trap, a wrong reading order, a button whose name lies, a mobile swipe that cannot finish, can all live under a perfect score.
Scores also stop testing. Teams treat the green gate as done, and reader walkthroughs and disabled-user sessions are squeezed out of the iteration. False confidence is more expensive than a few missed contrast failures — contrast can be patched next round; human tests blocked by a score let structural impassability live through launch. An automated pass is a necessary filter, not an end-to-end proof.
Studying it
Take real pages (or a release candidate) with high automated scores. Walk the primary task by keyboard and on at least one reader pair. Record task success and the step that stuck. When possible, have a disabled user run the same task on their own devices.
Independent variables: automated score / violation count, whether a human AT walkthrough was also done. Dependent variables: primary-task completion, count of blocking defects found only by humans.
Treat “zero violations” and “checkout succeeded” as two independent outcomes. Do not use the score to predict completion.
Where it stops holding
An automated fail is usually still a real fail and should be fixed; this claim is not cover for ignoring the tool. On tiny, almost non-interactive static pages, the seam between an automated pass and being usable is narrower. When a score is used to answer a procurement questionnaire, its social job is a compliance signal, not usability evidence — mixing the two in one report is how green gets read as usable. Disabled-user testing does not replace automated regression either: automation catches decidable regressions, humans catch whether it can be used. Both are required.
Applying it
- A green CI gate only blocks decidable regressions. The release list still includes “keyboard finishes the primary task” and “a reader finishes the primary task.”
- Do not put an automated score in a public statement as proof of usability; say which pairs were tested and whether the primary task completed.
- How to check: take a build with a perfect automated score, turn the display off, finish checkout by keyboard, then hear the same flow with a reader. If any step cannot be finished, a perfect score is not usable.