Candidate ranking depends on context and personal history
Aliases: IME ranking · contextual candidates · personal frequency
What it is
Encodings such as pinyin or shape codes map one key string onto many possible words. The candidate window has to line that set up as a list a person can pick from. Order is not lexicographic and not by code length: it follows characters already committed above, and what this person usually picks. The same string shiji should not put the same word first in a meeting as in an experiment.
Why it happens
Encoding-to-character is one-to-many. An IME first expands a segmentation lattice over the code: different cuts yield different word sequences, each node holding homophone or same-code rivals. With no context, only general frequency can push high-frequency characters forward, so a handful of function words occupy the front of short codes for a long time. Prior context is a left constraint on the lattice: the last committed word weights some outgoing edges. Personal history is this user’s edge weights: jargon, names, and pet phrases rise to ranks general frequency never grants. Ranking therefore scores the lattice, not isolated characters. Users see only a truncated list after scoring; they cannot read the lattice off the list, and experience it as “it changed this time.”
Studying it
With controlled priors (empty, biased to word A, biased to word B) and personal lexicon on or off, measure top-1 for the same code string. Independent variables include prior length, personal-lexicon size, and whether the session may learn; dependent measures include top-1 hits, rank of the target, and whether users notice the order changed. Ranking tests on isolated characters write general unigram frequency down as the whole of an IME.
Where it stops holding
At sentence start with no prior, or on a new device with an empty lexicon, ranking falls back to general frequency. When the encoding already uniquely determines a glyph (some full shape codes), the list can be short and context matters less. “Order will change” is not permission for the list to rewrite positions while a finger is already down—that is a different interaction issue.
Applying it
- Make the first slot of the candidate window follow the current sentence’s prior, rather than always showing the globally most frequent character.
- Let personal-lexicon learning be turned off or cleared, so one person’s history is not the next person’s default ranking on a shared device.
- Accept top-1 on sentences with priors, not on isolated code strings; report sentence-initial and sentence-medial separately.