Multiple sources of the same entity should collapse to one representative

What it is

One article syndicated on twenty sites, one product mirrored in two shops, one document as both HTML and PDF: several records in the index, one object for the user. Entity-level deduplication folds those records into a single canonical result in the list; the other sources no longer occupy their own rows. Broder’s shingling and SimHash-style near-duplicate detection exist to decide “is this still the same content.”

It handles identity, not similarity. Similar but distinguishable versions (different years, different sellers with truly different stock) must not be fused into one row—that is a later grouping layer. Dedup asks: if this takes another row, will people think a new object arrived.

Why it happens

Every row in a list is read as an independent candidate by default. One entity on many rows spends first-screen scent on copies: people compare site names among reprints instead of comparing objects. Clicks dilute; the judgment “is there anything else” is postponed. Merging changes the comparison unit from record to entity, so working memory counts “how many things” rather than “how many postings.”

Choice of representative rewrites the path that follows. Picking the busiest mirror sends people into the highest-traffic copy, which need not be the most complete or the most authoritative. Dedup is therefore not only fewer rows; it is the designation of one entrance that will be treated as “that’s it.” A wrong entrance, and the remaining sources—even if still in the back—are almost never walked.

Studying it

Use standard near-duplicate methods to test whether one entity becomes one row, then use tasks to test whether the representative was the right entrance.

• Paradigms: run shingling / SimHash / URL canonicalization on known syndication clusters and report whether only one row is shown; separately, tasks that ask for “the authoritative source” or “the downloadable PDF,” to see whether the representative blocked that path. IR near-duplicate evaluations (news-syndication corpora) supply gold clusters.
• Independent variables: decision threshold, representative rule (site authority / completeness / recency), whether an “N other sources” count is shown.
• Dependent variables: share of duplicate rows on the first screen, success at reaching the authoritative source, times a reprint is treated as a new object.
• Methodological note: reprints that are almost identical but differently bylined, and copies that are identical except for missing attachments, are not one kind of cluster. Gold labels should follow whether the user still needs to choose, not character overlap alone.

Where it stops holding

In archives, legal evidence, and version control, “the same text from different provenances” is itself the object of comparison; fusing them into one row destroys the chain of evidence. Goods whose price or stock varies by source look like one entity and are not, for a purchase decision; a hard merge hides the cheaper or in-stock row. Translations are not copies of one entity. For time-critical sources (official notice versus aggregator), the representative should prefer official, not clicks.

Applying it

• Collapse near-duplicate clusters to one representative at index or query time. Write the rule: authoritative source, complete text, or the user’s home site—not whichever URL was crawled first.
• Keep an “N other sources” count on the representative; do not let those sources occupy their own result rows.
• Treat reprints and “almost identical but different attachments” separately; do not merge the latter as identity.
• Verify with a query for a piece known to be syndicated. The list should show one row for it, pointing at an acceptable representative. Three copies in the top ten means dedup did not run; a single row pointing at a mutilated mirror, with the complete source unfindable, means the representative was wrong.

Related

• Within the group: G3.15.2 Near-identical but distinct results should be grouped, not listed side by side · G3.15.3 Grouping hides within-group differences and needs an expand path · G3.15.4 Dedup rules must be intelligible so known items do not silently vanish · G3.15.5 Group by the dimension the user is deciding on, not by a database field
• Adjacent: G1.07 Content inventory and audit · G3.05 Result ranking · G3.06 Result snippets
• Search terms: near-duplicate detection · canonical result · entity resolution