Passing almost every check is not overall conformance

What it is

A scanner that reports “92” or “46 of 48 checks passed” sounds like conformance. At a claimed level, conformance is a conjunction: every applicable check must hold; fail one, and the claim at that level does not hold. Passing almost every check is not overall conformance is aimed at treating conformance as a score. Severity can tell you what to fix first. It cannot average away an open failure. What the levels themselves mean, and which rung law usually takes, is not this question.

Why it happens

A claim is bound to “all applicable success criteria at this level are met,” not to a pass rate. Tools like scores because scores compare and fit a dashboard; conjunction looks ugly, so organisations start to optimise the score: contrast the scanner can see gets fixed, name errors and keyboard traps remain — and those are what users hit. The second layer is “not applicable.” If there is no video, captions must not be padded into the denominator, and untested items must not be marked passed. Once the denominator is gamed, a high score looks even less like conformance.

Studying it

Issue two conclusions on the same page in parallel: tool score / pass ratio, versus “did every applicable item pass.” List the items that falsify the conjunction, and see how far they were diluted in the score.

Independent variables: scoring rule (equal-weight pass rate / severity-weighted / conjunction). Dependent variables: whether the level is still claimed; whether diluted failures include keyboard or name; whether not-applicable items entered the denominator.

Automated scores especially cannot stand in for conjunction — they do not cover everything, and a high score only means “most of what the machine can see is green.”

Where it stops holding

Progress tracking may use a score if it is not written as a conformance claim. Partial conformance has to list the failures explicitly; that is another document's job, and it cannot be rewritten as “mostly conforms.” Failures do have different weights for users; conjunction governs the truth value of the claim, not the repair order. Internal experiment pages may carry no level claim; once a public URL is in scope, conjunction applies.

Applying it

• Public material may not replace a conformance conclusion with a score or “almost all passed.” The conclusion is only: within the claimed scope, do all applicable items at that level hold.
• Dashboards keep “remaining failures” and “are we still claiming conformance” as two columns, so a greening score cannot silently change the conclusion.
• Record reasons for not-applicable items separately; they do not enter a pass rate.
• How to check: take the latest scan, ignore the score, and list every applicable failure. If any failure remains while public language still claims conformance, change the claim to non-conformant or narrow the scope and name the failures. Retest with keyboard and a screen reader the failures the score did not penalise, to confirm the tool simply did not see them.

Related

• Same group: J1.10.1 A page-level claim and a site-level claim can reach different conclusions · J1.10.3 Dynamically generated content must be retested after each update · J1.10.4 Third-party embeds usually sit outside the claimant's control
• Nearby: J1.02 Conformance Levels · J5.08 Limits of Automated Checking
• Search terms: full conformance requirement · conjunction not score · WCAG-EM