Substitutions, insertions, and deletions have different consequences
Aliases: WER · substitution · insertion · deletion
What it is
After automatic speech recognition aligns its output with a reference transcript, errors fall into three classes: a substitution writes the wrong word, an insertion adds a word the reference lacks, and a deletion drops a word that was said. Word error rate (WER) sums all three over the reference length, so the same WER can come from very different kinds of damage. In interaction, the three classes are not equivalent when they hit slots, negations, and proper names.
Why it happens
A decoder trades acoustic scores against a language model. Substitutions cluster on acoustic neighbours or on words the language model finds more fluent: hearing “cancel” as “query” can reverse intent. Insertions come from decoding noise, breaths, or filled pauses as words, or from a language model injecting high-frequency function words. Deletions hit quiet tails, particles masked by noise, and repetitions the language model treats as redundant. In a command interface, deleting a negation or inserting a phantom slot value is worse than mistaking a modifier. Intent error therefore cannot be recovered from WER: one substitution can change the intent, while three synonymous substitutions may leave it intact. Alignment itself depends on tokenization; different Chinese word breaks move the three counts.
Studying it
Align on a closed command set and on spontaneous dictation separately, split the three error types, and stratify by word class (negation, digits, names, function words). Also label whether the intent is still executable. Independent variables include noise, speaking rate, and domain vocabulary. Do not report a single WER: show the substitution/insertion/deletion mix and a separate figure for critical slots. Laboratory read speech suppresses insertions (fewer fillers) and deletions (clearer articulation), and overstates how ready recognition is.
Where it stops holding
Open-domain chat is more tolerant of synonymous substitutions; navigation and money movement almost never tolerate substitutions of digits or names. Insertions in streaming partials that are later withdrawn should not be charged like insertions in the final hypothesis. A speaker’s own slip is not a recognition error; protocols must say whether it is stripped from the reference. Calling every failure “inaccurate recognition” dumps substitutions that need language-model bias and insertions that need VAD into one optimisation queue.
Applying it
- Give error budgets to intent-changing classes (negation, amounts, names, addresses) rather than using overall WER as the ship gate.
- For low-confidence substitutions, offer tappable neighbours rather than only a full respeak.
- Log the three classes separately and mark whether they caused a wrong action; when insertions spike, inspect noise and endpointing first, when substitutions spike, inspect the lexicon and language-model bias.