Searching several scopes at once requires one shared relevance scale

What it is

One query into several scopes—mail and drive, this site and Help, several repositories—then interleaved into one list, has to share one relevance scale. Each collection’s own scores (its BM25, its click rate) cannot be compared as magnitudes: Help’s 12.4 and mail’s 0.8 are different units, and interleaving becomes “whichever engine inflates harder ranks first.” Collection fusion in federated search, CombSUM / CombMNZ, and resource selection exist to hang heterogeneous scores on one comparable axis.

Merging is not shuffling several already-ranked lists. Without a shared scale, the shuffle only forces incomparable number-ones to sit together.

Why it happens

A relevance score is relative to the candidate set it was computed on. IDF moves with the collection; field weights move with the schema; click priors move with how the collection is used. Laying raw scores side by side is comparing lengths taken with different rulers. One scope then systematically owns the first screen: not because it is more relevant, but because its scores are larger in magnitude, or its collection is smaller and IDF is larger. What people see as “most relevant” is “the collection whose scores bloated most.”

A shared scale is a transform: normalize by within-collection percentile or z-score, train a cross-collection click model, or select collections first and re-score the admitted ones with one ranker. Whichever path, the merge happens after the transform. Scores from before the transform are valid only inside their collection.

Studying it

On queries whose known relevant documents are spread across collections, compare “interleave by raw score” with “interleave after normalization.”

• Paradigms: federated-search evaluations (TREC Federated Search, several verticals) report cross-collection nDCG and, separately, each collection’s share of the top ten. Lab tasks can ask whether the top three all come from one type that merely looks more relevant.
• Independent variables: score transform (none / within-collection percentile / cross-collection learned ranking), whether collection sizes are badly unbalanced.
• Dependent variables: cross-collection nDCG, deviation of top-ten collection share from the true distribution of relevant documents, frequency of one collection occupying the top three.
• Methodological note: if relevant documents really sit in one collection, occupation is correct. Gold labels follow whether the object is relevant, not a quota that “every collection should be represented.” Quotas are a diversity policy, not a relevance scale.

Where it stops holding

When each scope already has its own column (mail | files), no shared scale is required; rank inside the column. The problem is triggered by forcing a single list. Live and archive collections mean different things by “relevant” (new mail versus old contracts); one scale will crush the wrong time preference. Prefer type columns, or a task-level switch of scale. In secure retrieval some collections cannot export scores, only a truncated top-k; fusion has to work on truncated lists (CombMNZ-style variants) and cannot assume full score vectors.

Applying it

• Before one list across scopes, specify a cross-collection score: re-rank with one ranker, or normalize within each collection and then fuse. Do not compare engines’ raw scores.
• Inspect collection share in the top ten. If one collection stably fills it without holding more relevant objects, the scale is not unified.
• If diversity (at least one per type) is required, write it as a separate policy. Do not rely on one collection’s inflated scores to “naturally” surface it.
• Verify with two collections, relevant documents spread evenly, but collection A’s raw scores an order of magnitude larger. Merge-by-raw-score should let A own the top ten; after the transform, the top ten should approach the true spread. If A still owns it after the transform, fusion was not calibrated.

Related

• Within the group: G3.16.1 Default scope should be the current context, not the whole collection · G3.16.2 Nested scopes inherit: a child collection sits inside its parent · G3.16.3 Zero hits in a narrow scope should offer one-click expansion to the parent · G3.16.5 Scope changes ranking; top hits from different scopes are not comparable
• Adjacent: G3.05 Result ranking · G3.12 Search scope · G3.15 Result grouping and deduplication
• Search terms: collection fusion · federated search · score normalization