Automation can cover only some machine-decidable rules
Aliases: auto-check · axe · machine-testable rules
What it is
Contrast ratios, whether an image has an alt attribute, duplicate ids — a machine can give a definite answer. Whether a heading behaves as a heading, whether that alt carries the trend in the figure — it cannot. Automated checking covers only the machine-decidable slice of accessibility rules, not the whole table.
Why it happens
A rule is automatically decidable if and only if the predicate can be computed from structure, attributes, and rendered color, without understanding what the sentence means to a person. Presence of an attribute, whether a ratio clears a line, whether a node is focusable, belong here. The moment the predicate contains “correct / meaningful / equivalent,” computation stops — not a missed test, a problem outside the decidable set.
Tools therefore split results into fail / needs review / pass. A common industry figure is that roughly two to three tenths of success criteria are fully automatable; the rest are either only partly catchable or entirely human. Treating a tool report as rule coverage is mistaking the decidable subset for the whole set. What CI can block is what breaks inside that subset.
Studying it
Take pages with known defects (insufficient contrast, missing alt, alt of “image,” broken heading levels, a keyboard trap). Run automated engines (axe, Lighthouse, WAVE) and a WCAG-EM human sample. Align by success criterion: what automation caught, what it marked for review, what it never mentioned.
Independent variables: defect class (decidable / needs semantic judgment), tool. Dependent variables: detection rate, false positives, count of success criteria with zero coverage.
Do not use “number of issues” as the only metric. Decidable defects can be exhausted while semantic defects are all still missing.
Where it stops holding
Tools move. Fragments that used to need a human (some contrast, some focus visibility) enter the decidable set; coverage is not a constant. Heuristics will flag some semantic problems as “likely fail” — a hint, not a decision. For PDF, canvas, and native apps, the web engine’s decidable set does not transfer. If the rule itself is an open predicate such as “provide an equivalent alternative,” automation’s ceiling is set by the predicate, not by incomplete engineering.
Applying it
- Put automated checks in CI to catch decidable regressions: empty alt, contrast, duplicate ids, missing language.
- Items marked “needs review” go through a human path; they are neither a pass nor already a fail.
- How to check: against the tool’s rule list, mark the success criteria it claims to decide automatically. Every other criterion on the primary task needs a keyboard or reader check. Criteria the tool never lists are outside automated coverage.