Mid-utterance language switches require live language identification

What it is

Intra-sentential code-switching is a change of language inside one breath—Mandarin in the first half, English in the second. The recognizer cannot wait for the sentence to end before choosing a language. It must identify the current language live (language identification, LID) so later frames go to the matching phonology, lexicon, and language model. That is a routing problem for the decoding graph, not “a loanword appeared in this sentence.”

Why it happens

Monolingual recognition assumes one phoneme inventory and one lexicon for the whole clip. After a switch, acoustic evidence that still follows the previous language’s emission probabilities is broken into that language’s syllables. Streaming stacks therefore run LID on a short window, or a multilingual shared encoder with a language posterior. The decision has to be fast: waiting for the full sentence lets partials appear in the wrong script; deciding too fast treats accent, names, and noise as a switch. A real switch changes phonology and lexicon together. Inserting an English brand already in the lexicon into a Chinese sentence is different: a monolingual decoder with a mixed lexicon can still absorb it.

Studying it

On bilingual speech with marked switch points, measure LID frame delay, WER around the switch, and language-label accuracy. Scripts should include early, late, and back-and-forth switches, plus a no-switch control. Report errors before and after the switch separately so whole-sentence WER does not dilute a routing failure. Actors switching at a prompted index have lower ecological validity than unscripted bilingual conversation; keep at least one unscripted set.

Where it stops holding

If the user has locked a language in settings, live LID should not run; it will misread an accent as a foreign language. When two languages are acoustically close (some dialects and the standard), the LID window chatters, which is closer to variety identification than language ID. Read “half Chinese, half English” in translationese has clean switch points and overstates field performance. A product with no multilingual decoding graph has no engineering object for live identification.

Applying it

• For bilingual commands you actually claim to support, run a language posterior in the streaming decoder and allow a mid-capture change of track.
• Make the currently decided language visible (for example in the script of partial text) so a wrong switch can be corrected without repeating the whole sentence.
• Put mid-sentence switches in the regression set; do not test only “all English” and “all Chinese.”

Related

• Same group: C7.04.1 Training-data distribution determines whose speech is recognized well · C7.04.2 Recognition degrades on mixed Chinese–English and dialect words · C7.04.3 Accuracy gaps exclude particular populations · C7.04.5 A wrong language decision forces the rest of the sentence through the wrong phonology · C7.04.6 Dialects share vocabulary with the standard but differ in pronunciation, so they are misread as near-homophones · C7.04.7 Accent and dialect gains depend on training data from those speakers, not on algorithms alone
• Adjacent: C7.10 Live Partial Results · C7.14 Command Grammar versus Free Expression
• Search: language identification · code-switching · intra-utterance LID