Cover mainstream combinations, not a single environment
Aliases: compatibility matrix · mainstream AT pairs · test coverage
What it is
Turning on VoiceOver on the designer’s Mac tests one environment, not a user population. Compatibility testing has to cover mainstream combinations: NVDA and JAWS on Windows, VoiceOver on Safari, TalkBack on Android — not treat a single pass as a full pass. Which pairs to include follows what people actually use, not which machine is easy to open in the lab.
Why it happens
A single environment systematically misses mapping differences. Teams skew Mac or Chrome; users’ readers skew Windows. Taking one’s own machine as the whole set is using one pair to stand in for other people’s runtimes. The unit of coverage is the pair, not the act of “having tested a reader” — ten runs of the same pair still cover one pair.
Mainstream is not everything. Windows+NVDA, Windows+JAWS, macOS+VoiceOver, iOS+VoiceOver, Android+TalkBack already catch most users of a public product; adding obscure engines past that has falling marginal return. If the missing pair is the highest-share one, extra obscure engines do not fill the hole. Rank the matrix by user data; when data is missing, at least do not keep only the machine the team already knows.
Studying it
List reader and browser share in the target market (WebAIM surveys, the product’s own AT feedback), build a minimal matrix, and run the same task script on every cell. Record which pair has never been tested, which pair appeared only in a demo.
Independent variables: pairs in the matrix, whether Windows is included, whether mobile is included. Dependent variables: defects first found on a second pair, fraction of tasks that pass in only one environment.
WCAG-EM sampling answers “which pages”; the matrix answers “on which runtimes those pages are tested.” Keep the two ledgers separate.
Where it stops holding
An intranet tool whose contract pins browser and reader may narrow the matrix and say so in the statement. Embedded or set-top devices have one system reader, so pair coverage does not apply. Share moves: a JAWS-heavy market a few years ago may be NVDA-heavier now; refresh the matrix from surveys, do not carve the lab’s five machines in stone. A VM that “had” a reader but never finished the primary task does not count as coverage.
Applying it
- Write a minimal matrix: at least one Windows reader+browser pair and one Apple reader+browser pair; add TalkBack or iOS VoiceOver when the product is mobile.
- Run the primary-task script in every cell; do not regress only on the developer’s laptop.
- How to check: tick the matrix. A record in only one cell is single-environment coverage. Fill a missing cell with the same script — failures that appear there are what the single environment dropped.