Candidate ranking fuses word frequency, a contextual language model, and personal history

What it is

Pinyin, zhuyin, and romaji expand one key string into many homophones. The candidate bar is not shuffled; it fuses three signals into a score: word frequency (how common the word is in a corpus), a contextual language model (which successor the already-committed text makes likely), and personal history (what this user recently chose). The ranker decides which word sits in the first slot that Space commits. The topic is how scores fuse, not what the window covers, and not whether the IME was switched.

Why it happens

Frequency supplies an unconditioned prior: 的 always outranks a rare homophone. Context turns that into a conditional: after 北京, shi is more likely 市 than 是. Personal history then pulls the conditional toward the user's habit: a clinician typing tang more often wants 糖 than 唐. When the three conflict, implementations arbitrate with weights, interpolation, or a learned fusion. Cloud models may also inject trending terms. Top-1 hit rate depends on whether fusion fits the current domain; weights frozen on news will bury clinical terms behind a page turn. Space is bound to “first candidate,” so a fusion error becomes a wrong character, not merely an ugly list.

Studying it

On annotated pinyin–hanzi parallels, ablate the fusion: frequency only, frequency plus context, then add personal history, comparing top-1 and mean pick rank. Keep domains separate (chat, news, clinical notes, code comments). Personal history needs a cold-start segment and a steady-state segment. Corpus perplexity is not commit accuracy—users see a ranking, not a probability. Do not overlap train and test users, which reports memorization as general context skill.

Where it stops holding

Rare characters, classical verse, and chemical formulae have almost no frequency or user history; fusion degrades to pinyin equality and ranking stops meaning much. English autocomplete has a similar three-signal shape, but the grain is words, not homophones. Offline IMEs that drop cloud trends leave breaking news terms ranked low for a long time. Privacy modes that clear personal history leave only frequency and generic context.

Applying it

• Vertical products should ship a domain lexicon; do not assume a generic chat-and-placename model will put technical terms first.
• At cold start, keep personal-history weight low so the first few mis-picks cannot permanently outrank frequency.
• Let users see the first candidate before Space commits, rather than committing blind.
• Verify on target-domain text with history off (frequency and context only), record top-1, then turn domain history on and check that terms move forward without dragging chat slang with them.

Related

• Same group: C6.26.2 Learning from choices raises a word later, but can also cement typo candidates · C6.26.3 For polyphones and homophones, candidate order dominates first-hit rate · C6.26.4 Live re-ranking can move the target as the user is about to click, causing a mis-pick
• Adjacent: C6.12 IME candidates · C6.10 Autocomplete
• Search: IME ranking · language model · personalized candidates