Evaluate by cohort, not by the overall number

What it is

The dashboard says overall word error rate is 8%. In open-plan logs, nine in ten speakers sit between twenty-five and forty-five. Children’s and older staff’s sessions are few, and the failed ones never leave a success mark. Eight percent is an average weighted by the majority. Cohort evaluation wants WER and completion for children, older adults, and younger adults each with enough N. Shipping reads the worst cell, not the total cell diluted by traffic. The overall number can be true and still false for the tails.

Why it happens

A weighted mean eats sparse, high-error cells. Younger adults at 90% of traffic and 6% WER, children at 10% and 24% WER, merge near 8% — looks shippable; the child cell is already unusable. Session count is further filtered by success: people who fail stop speaking, N for children and older adults shrinks, next week’s overall looks better. So the overall metric cleans twice: arithmetic weighting, and behavioral exit. Cohort evaluation splits the cells and sets a minimum N per cell, blocking “no one uses it so we need not measure it.” Without the split, the quality system rewards serving the group already succeeding.

Studying it

Turn acceptance into a cohort table: age band × scene (classroom, desk, car) of WER, sentence error, completion, N. Write the minimum N and maximum error per cell in advance. Cells under N are “unmeasured,” not filled with the overall number. Compare which skills change their ship decision under “overall may pass” versus “every cell must pass.” Open-plan voice logs need age or a proxy (account type), or field data will replay the weighting. Lab sampling must oversample the tails; drawing from natural traffic draws from a distribution already shaped by exclusion.

Where it stops holding

An internal tool whose only users are younger adults, and which will not be sold into classrooms or hotel rooms, can cover actual users only — and still has to name who was not covered. Tiny language or accent cells will be noisy; cohort evaluation wants honest intervals, not a fake fine table. Treating cohorting as “smoother prompts will flatten the cells” points at the wrong lever: evaluation answers which number to look at, not which fix closes the gap. Overall metrics remain useful as a coarse scan for capacity and regressions. They are not a release gate.

Applying it

• Replace “overall WER” on the release checklist with a table of children / older adults / younger adults × target scene. A missing cell or short N is a fail, not an equivalent of the overall.
• Keep fields in the log pipe that can cohort (an age-band consent item, account type), and count “spoke but did not finish” so failed sessions do not vanish from the denominator.
• If a demo or a sales sheet quotes recognition rate, it must quote the tails as well. Overall-only is an unqualified citation.
• How to check: recompute the live weighted mean. If error doubles once the 90% younger-adult cell is removed, the overall number is not qualified as a gate, and the cohort table becomes the only release evidence.

Related

• Same group: M4.04.1 Word error rate is substantially worse for children and older adults · M4.04.2 Recognition gaps become de facto exclusion
• Nearby: M4.09 Recognition differences for children and older adults · C7.04 Accent, dialect, and code-switching · M4.03 Multi-user recognition
• Search terms: cohort evaluation · age-stratified word error rate · disaggregated metrics