Dedup rules must be intelligible so known items do not silently vanish

What it is

Someone remembers that a result should be here—the URL clicked last week, the doc a colleague just sent, the version they uploaded—and the list does not show it. If it was merged as a duplicate and the UI never says “merged with X,” the record has vanished. Intelligible dedup criteria means the merge rule can be restated in the user’s own words, and that a trace remains on the merged-away side, not only inside the index.

This is not a demand to publish a SimHash threshold. It is a demand that people can answer “where did it go”: into which row, on what grounds they count as the same thing. Dedup that cannot be explained is attributed as lost data.

Why it happens

Re-finding leans on external memory: bookmarks, chat, spoken recommendations name a concrete identity, not “the cluster that is semantically closest.” Dedup works by content overlap or canonicalized URLs. When that fails to line up with the external identity, a known object is pulled from the list and external memory has nowhere to land. People who do not have a category called “dedup” can only read absence as not indexed, not permitted, or misremembered. All three stop the search; the object is still behind the representative.

A trace rewrites absence as relocation. “Merged with Title” or a source list under the representative gives the external identity a landing. Without a landing, dedup is random deletion for anyone who does not hold a mental model of the global index.

Studying it

Use re-finding tasks of the form “this known object should be in the list,” and manipulate whether dedup is traceable.

• Paradigms: people first open object A successfully; a later query merges A into representative B; compare no explanation, a merged-source list under B, and an explanation aimed at “why isn’t A here.” Query-log complaints of “can’t find” around a dedup launch are field evidence.
• Independent variables: form of the merge trace (none / source list / targeted note for a known URL), how far B’s title diverges from A’s.
• Dependent variables: success at recovering A, rate of attributing absence to “deleted / not indexed,” whether restatements of the rule match the real rule.
• Methodological note: participants who have never seen A will not experience vanishing, only one fewer row. Use their own known items. A heavily rewritten representative title adds a separate recognition failure and must be scored apart from “no trace at all.”

Where it stops holding

Public web search cannot write a targeted note for every omitted URL; the fallback is a clickable “omitted similar results” catch-all. Intranets, mail, and drive collections have stable identities and a much higher cost when a known object vanishes; the criterion must be more specific (same file hash, same message id). Security-sensitive settings cannot list every folded source domain (phishing mirrors); traces should be proportionate, not an attack-surface inventory in the name of intelligibility.

Applying it

• Give merged sources at least a discoverable list, hung on the representative. On an intranet, allow reverse lookup by title or URL to “which row did this merge into.”
• Summarize the rule in user language (“syndicated copies of the same body,” “different links to the same file”), not internal function names.
• If someone arrives by an external identity and lands on the representative, say “the item you wanted was merged here,” and offer the original address when it is still reachable.
• Verify by having someone note a result’s title or URL, then run a query that will merge it away. They should be able to point to the row it now lives in and say why. If they can only say “it’s gone,” the criterion is not intelligible.

Related

• Within the group: G3.15.1 Multiple sources of the same entity should collapse to one representative · G3.15.2 Near-identical but distinct results should be grouped, not listed side by side · G3.15.3 Grouping hides within-group differences and needs an expand path · G3.15.5 Group by the dimension the user is deciding on, not by a database field
• Adjacent: G3.17 Search result explainability · G3.07 Zero-result handling · G4.03 State persistence
• Search terms: omitted results · deduplication transparency · re-finding