Scope changes ranking; top hits from different scopes are not comparable

What it is

The number-one hit for a query in “this section” need not be number one on “the whole site,” and need not be the “better” item. Ranks are not comparable across scopes: a rank is relative to the current candidate set. Change the set, and IDF, competitors, and click priors all move; the rank moves with them. Laying two scopes’ “firsts” side by side, or treating a local first as a global quality signal, is comparing champions from two different matches.

This is a different step from “merging several scopes needs one scale.” A shared scale is how to rank one list when several scopes are searched together. Incomparability says that when two scopes are run separately, their leading results cannot be used as evidence about each other.

Why it happens

Ranking is a tournament whose roster is set by scope. It is easy to finish first in a small set: fewer competitors, rarer terms have larger IDF, a middling match can surface. The same document, entered in a large set, meets more exact, newer, more-clicked opponents, and drops. People still read “ranked first” as an absolute property of the object, like a price. When the first changes after a scope switch, it is explained as system instability, or as the object itself having changed.

Local firsts also get misused downstream: “related document” recommendations, a cache of “best hit,” promoting the local champion onto a global home. Those uses treat a collection-relative rank as cross-collection quality. Change the scope, and the promoted row may not even make the new top ten.

Studying it

Hold query and documents fixed, change only the candidate-set boundary, watch ranks move, and test whether people treat rank as absolute quality.

• Paradigms: run one query on three nested scopes and record the target’s rank curve; separately, an interface that pairs two scopes’ firsts and asks which is “more relevant.” The effect of collection size on IDF and rank is known in IR; what interface studies must add is the absolutizing of rank.
• Independent variables: scope size, whether the UI pairs top results from two scopes, whether ranks are labeled “in this scope only.”
• Dependent variables: rank delta for the same document across scopes, rate of choosing a local first as “better overall,” distrust ratings when the first changes after a scope switch.
• Methodological note: if the two scopes share a first, incomparability is never experienced. Pick a document that surfaces locally and is pushed down globally. Do not use click rate as a cross-scope gold standard; clicks are themselves relative to the list then on screen.

Where it stops holding

When scopes are almost the same size and candidates overlap heavily, ranks will be close and the practical harm shrinks—though the logic is still relative. When people work in one scope and never switch, ranks are consumed only there and incomparability does not erupt; it appears the moment the product offers “see on the whole site” or promotes a local champion elsewhere. Showing raw scores strengthens absolutizing and is worse than showing ranks.

Applying it

• Do not export “first in this scope” as global quality. Do not fill another scope’s recommendations or home with it.
• After a scope switch, if the head of the list changes, label the list “results in scope name.” Do not let it read as a rematch of the same tournament.
• When two scopes must be compared side by side, compare whether objects appear and whether attributes differ, not the two rank numbers.
• Verify with a document that ranks first in a subsection and falls outside the top ten globally. Switching scope should move it. If the UI still marks it “best match” without naming the scope, or pairs two firsts for “which is better” without saying the matches differ, rank has been treated as an absolute.

Related

• Within the group: G3.16.1 Default scope should be the current context, not the whole collection · G3.16.2 Nested scopes inherit: a child collection sits inside its parent · G3.16.3 Zero hits in a narrow scope should offer one-click expansion to the parent · G3.16.4 Searching several scopes at once requires one shared relevance scale
• Adjacent: G3.05 Result ranking · G3.12 Search scope · G3.17 Search result explainability
• Search terms: collection-relative ranking · IDF · scope switch