Test on the user's own device and AT, not a lab-standard setup

What it is

A lab PC with factory-default NVDA, one headset, one keyboard, one keymap — that measures the default install, not the user’s assistive technology. Testing on users' own AT means the reader version and add-ons, the braille display, custom keys, speech rate and punctuation, magnification, external switches, come with them. A uniform environment erases the configuration layer that actually crashes in real use.

This is not the same cut as “strategies differ within a disability type.” That asks whom to recruit. This asks whose machine is in front of them when they sit down.

Why it happens

AT is a personal tool tuned over years. JAWS scripts, dictionaries, virtual-buffer settings, VoiceOver rotor items, invert-and-follow-focus in a magnifier, dwell time on a switch, are not what a lab image clones. After a uniform environment flattens those differences, the product faces a “standard user” who does not exist. Harder still: muscle memory is bound to the user’s keys. On a lab keyboard, failure may be “H is not where they thought,” misfiled as a product defect; the inverse also holds: a path that works on lab defaults never speaks the critical sentence at their 700 words/minute with hints off.

Remote sessions sit closer to this mechanism than inviting people into a lab with AT preinstalled, provided the remote tool does not block AT (some meeting apps swallow shortcuts or steal focus).

Studying it

Run the same tasks in two cells: lab-default AT versus the user’s own device (or remote with screen share). Record failures unique to one cell. On the bring-your-own cell, start with an unrelated page so the user can walk it their way and confirm the profile is loaded (rate, braille, magnification) before the product. Do not pause the session to ask “please switch to default punctuation so we can hear.”

Independent variables: environment (lab default / own), remote versus in-person, whether the meeting app intercepts shortcuts. Dependent variables: non-overlapping issues, key-binding errors misfiled as product bugs, states missed on the own-AT cell because hints were off.

Where it stops holding

First install on a new device (a just-bought phone, a company laptop) is a default profile; lab default then has ecological validity. Secure or classified settings cannot admit personal machines; import the user’s config onto a managed box rather than pretending default equals own. A compatibility matrix across reader/browser combinations needs a uniform environment to attribute cause; that is a different test, not to be merged into a user session. A broken personal device or expired AT licence will surface as equipment failure; budget time to separate that from a product accessibility fail, and do not log “would not open” as the latter by default.

Applying it

• Say in recruitment and scheduling “please use the computer or phone and AT you use daily”; prefer remote, lab as backup.
• At session start, confirm speech rate, braille, and magnification are the daily profile on a neutral page, then open the product under test.
• If observers need to hear, use a sidecar recording or a second output; do not ask the user to drop to a “demo” speech rate.
• How to check: compare the issue lists from a lab-default session and a bring-your-own session. Missed announcements, shortcut clashes, and magnification misalignment that appear only on own-AT are the layer the uniform environment erased. If own-AT was never run, current findings hold only for the default install.

Related

• Same group: J5.14.1 Automated tools and expert review cannot replace testing with AT users · J5.14.2 Variation within a disability type is large; a few participants do not cover the range · J5.14.4 Paid disabled participants belong in early design, not only acceptance
• Nearby: J5.07 Compatibility testing · J5.06 Refreshable braille displays
• Search terms: testing on users' own AT · ecological validity · JAWS profile