Matching inflected forms depends on stemming, not spelling correction

What it is

running hitting run, organizes hitting organization, depends on stemming or lemmatization: morphological rules fold inflections and some derivations back to one index term. That is not rewriting a misspelling onto a dictionary form. The user did not mistype; they used another legal word form. Porter, Snowball, and dictionary lemmas handle grammar. Levenshtein handles character accidents. Sending inflection through spelling correction labels a legal form as an error, or uses edit distance to guess a correspondence that already has a rule.

Chinese does not inflect like English: 计算机 / 电脑 is synonymy, not a stem; 走着 / 走了, if merged, is the analyzer’s job, not a fuzzy query.

Why it happens

Morphological change is systematic: plurals, tense, nominalization follow enumerable suffixes and lexicons. A stemmer uses that system and indexes a cluster of legal forms under one key, recalling different syntactic realizations of one concept. Spelling error has no such system; it follows the accidents of keyboard and attention, and its distribution is noise. Fitting a noise model to regular change is unstable (university / universe will be merged by an aggressive stemmer and by a wide k) and hides real spelling problems from the UI—the system thinks it is doing morphology when it is doing fuzz.

Conversely, exact terms with no stemming split one concept into recall holes in languages like English: the query is plural, the document is singular, the object exists and cannot be found. That is an analyzer gap, not a user typo, and it should not pop “did you mean.”

Studying it

Feed morphological minimal pairs and spelling minimal pairs to a stemmer and to fuzzy matching, and watch for boundary crossing.

• Paradigms: three query groups—legal inflection (mice/mouse), true spelling error (recieve/receive), known overstemming traps (university/universe). Compare stem-only, edit-distance-only, and both. Classic Porter-style evaluations (overstemming versus understemming) are the morphological reference.
• Independent variables: analyzer (none / light stem / dictionary lemma), whether fuzz is on at the same time.
• Dependent variables: recall on inflection pairs, false merges on trap pairs, fraction of spelling errors rescued by the stemmer alone (should be low).
• Methodological note: “overall recall went up” is not evidence that stemming worked; fuzz may have helped. Report by group. Porter conclusions from English do not transfer to Chinese, German compounds, or Arabic.

Where it stops holding

On morphologically poor, very short tokens (SKUs, command names, code identifiers) stemming is harm: cat should not eat anything beyond cats, and must not eat category. In medical and legal language, a small suffix changes reference (dosage form, subsection of a statute); lemma merges cause reference errors and want a controlled vocabulary, not a general stemmer. When the user is asking how a plural is spelled, merging forms hides the grammatical fact they came for.

Applying it

• Configure stemming or lemmatization in the analyzer pipeline. Do not reuse a fuzzy query to cover plurals and tense.
• Turn stemming off for identifiers, codes, and all-caps tokens; leave it on for ordinary prose, and watch known overstemming pairs.
• Do not write a stem hit as “spelling corrected.” If an explanation is needed, write “includes other word forms.”
• Verify: organize should reach a document containing organizing without the box turning into a correction. recieve should not be rescued by stemming alone—that is fuzz or spelling correction. university should not stably reach universe by stemming.

Related

• Within the group: G3.14.1 Approximate matching absorbs character differences in the index without rewriting the query the user sees · G3.14.2 An edit-distance threshold that is too large admits irrelevant hits; too small, and tolerance disappears · G3.14.3 Phonetic and lookalike matching are not the same as character edit distance · G3.14.5 Fuzzy hits should be downweighted so exact matches surface first
• Adjacent: G3.04 Spelling correction · G1.10 Controlled vocabularies and synonyms · G1.08 Term consistency
• Search terms: stemming · lemmatization · morphological matching