Automated tools and expert review cannot replace testing with AT users

What it is

A scanner catches missing contrast, missing alt, missing labels. An expert can walk checkout with a reader and file a list. After both, it is still unknown whether someone who uses JAWS every day will abandon at step three. AT user testing asks people who actually use assistive technology for their own goals to expose stuck points, workarounds, and strategies in real tasks. Automation and expert review are filters, not proxy users.

An expert launching NVDA tests “I can complete this against a checklist.” A user tests “at my speech rate, with my shortcuts, with my patience, can this order still be placed.” Those are different questions.

Why it happens

Automated rules can only judge machine-decidable presence of attributes. Whether semantics are right, whether a flow is completable, whether announcements are too noisy to think through — the rules have no truth value. Expert review adds human judgement, but the expert’s fluency, listening habits, and willingness to probe are not the target users’ distribution. Experts know to hunt a lost focus; users may think the app died. Experts finish every path to write the report; users have a time budget and leave after two detours.

The second layer is AT configuration space. Punctuation, hint level, braille table, browse mode: the lab expert almost always sits on defaults. “Works on default” does not imply works on the user’s profile. Without real users, that configuration space is never sampled.

Studying it

Run three layers on the same product: a scanner, an expert review (keyboard plus at least one reader), and paid AT users on critical tasks. Compare the three issue sets — overlap, and what each layer uniquely found. Tasks should be the user’s own goals (tickets, renewals, submitting work), not only “find every heading.” Record abandon points, not only severity scores.

Independent variables: evaluation layer (auto / expert / user), whether the task is user-owned. Dependent variables: unique issue count, task completion, step of abandon, strategies the expert did not foresee.

Where it stops holding

Paper prototypes have no runnable AT; user sessions collapse into interviews and cannot be claimed as “already tested with AT users.” When the target disability group cannot be recruited, expert review remains a required filter, but the conclusion must read “not yet hit by real AT users,” not pass. Some issues (flash frequency, contrast numbers) users may not report verbally; measurement stays necessary. A colleague who “can turn on VoiceOver” is not a user: the session measures politeness and insider knowledge.

Applying it

• After automation and expert review pass, still have real AT users complete end-to-end tasks; the pass line is abandon points, not the length of the issue list.
• Keep a heading in the expert report for “assumptions not validated with real users,” so a walkthrough is not treated as acceptance.
• Do not substitute engineers who can operate a reader for recruited users.
• How to check: pull the last “accessibility tested” record and see whether there is video or notes from external AT users. Scanner scores and an expert list only means this layer was skipped. File the step where the user abandoned as a defect, not as a “usability suggestion.”

Related

• Same group: J5.14.2 Variation within a disability type is large; a few participants do not cover the range · J5.14.3 Test on the user's own device and AT, not a lab-standard setup · J5.14.4 Paid disabled participants belong in early design, not only acceptance
• Nearby: J5.08 Limits of automated checking · J5.07 Compatibility testing
• Search terms: AT user testing · expert review · automated accessibility testing