Readability formulas are only rough proxies

What it is

Readability formulas as surface-feature proxies map countable features—sentence length, word length, syllables, characters, or familiar-word lists—to a score or grade label. Flesch Reading Ease and Flesch–Kincaid Grade Level each have a particular equation, calibration corpus, and scope. They reveal something about the measured surface features and can screen for anomalies or compare controlled revisions. A score alone does not establish that target readers understood the content, could perform the task, or found the copy appropriate for its language and situation.

Why it happens

To remain computable, a formula discards whether terminology is known to this reader, conceptual relations are explicit, reference and logic are coherent, or conditions and negation are interpreted correctly. A short UI message also depends on controls, icons, prior state, and consequences. A score calculated without that context can be reproducible yet irrelevant to action comprehension. Tokenization, syllables, scripts, and sentence boundaries differ across languages, so transplanting a formula calibrated for one language can change what is being measured. Decimal output and a grade label create an appearance of precision; they do not imply low prediction error or guarantee comprehension by a particular age or education level.

Studying it

State the formula version, language, text genre, target readers, and criterion before evaluating it. In representative material, compare scores with independent comprehension and task measures, examining ranking, association, calibration error, and outlier yield. Break errors down by text length, task, reader knowledge, and language proficiency rather than allowing one aggregate relationship to conceal systematic misses. For interface copy, paraphrase accuracy, condition judgments, outcome prediction, first correct action, and recovery are closer criteria than a rating of “easy to read.” Cloze tasks can add evidence about local predictability but do not stand in for task understanding.

Where it stops holding

A locally validated formula can monitor a corpus within one language, genre, and comparable audience, or flag a sudden increase in long sentences and rare words. It should not directly rank scores from different languages, and very short controls or error labels may be too small for stable estimates. A score threshold may be required by law, organizational policy, or procurement; meeting it should be recorded without treating compliance as comprehension evidence. Human editorial judgment is also fallible. The limitation calls for formula-plus-task validation, not abandonment of automated measurement.

Applying it

• Expose the formula name, version, language, and validation scope in the tool, along with measured surface features and uncertainty. Do not merge algorithms behind an unexplained “readability score.”
• Use scores to trigger review of version jumps, corpus outliers, and clusters of long sentences—not as one release verdict across every interface and audience.
• For a score-driven rewrite, verify preservation of terminology, logic, references, conditions, negation, and decisive information, then test the copy in its actual control and consequence context.
• Re-estimate error periodically against target-reader paraphrase, decision, and task data. If the score improves while comprehension stays flat or falls, stop using that metric to drive rewriting and document its boundary.

Related

• Same group: T1.07.2 The target reader decides acceptable complexity · T1.07.3 Simplification in professional contexts loses precision
• Adjacent: T1.03.1 Deleting words does not lower comprehension cost · T3.01.2 Headings, lists, and code blocks carry the scan anchors
• Search terms: readability formula · Flesch-Kincaid · criterion validity