Explanations exist to help reformulate the query, not to prove the ranker is right

What it is

A retrieval explanation written as “this row scores high because authority 0.81, freshness 0.12” is a brief for the ranker. Lookup does not need a verdict; it needs where to cut next. Bates’s berrypicking, Marchionini’s exploratory search, and Koenemann and Belkin’s experiments on transparent relevance feedback all define success as people rewriting the query or the constraints: add a word, drop a word that was too broad, switch fields, turn a filter off. Proving “the algorithm did not mis-rank” is neither testable nor a next step.

Field attribution, customization notices, and implicit-filter sources on the same screen should be kept or cut by this purpose: keep what changes the next sentence; drop what only raises confidence.

Why it happens

Finding is multi-turn. Each turn’s results are evidence about “how this sentence was understood,” so the next turn can move closer to the target. If the explanation points at internal features (authority, a net’s attention, a click prediction), the evidence lands where the person cannot act: they cannot turn authority up, only change words or scope. Attention is pulled onto “is the system fair,” the task flips from lookup to trying the algorithm, and turns spin in place.

Explanations that point at rewritable objects put the evidence back in query language: hit tags not title → add a field restriction; the roster is for you → contrast with the general list; safe search is cutting → turn it off and look again. Each line maps to an interface act. Explanation quality can then be measured by “did the next turn make a targeted rewrite,” not by “did they find the system trustworthy.” Trust may rise while lookup stops.

Studying it

Split explanation copy into “defends the ranking” and “points at a rewrite,” and measure whether the next query moves closer to the target.

• Paradigms: a task that takes two or three rewrites to finish; one explanation lists internal feature weights, the other names the rewritable item among field, filter, and scope. When Koenemann and Belkin compared transparent and opaque relevance feedback, transparency improved users’ control of the feedback, not their faith in the model—the same direction.
• Independent variables: whether the explanation points at internal features or at a changeable constraint, whether an action sits next to it (“title only,” “turn this off”).
• Dependent variables: targeted rewrites (the right layer), wasted rewrites, turns to completion, share of time spent judging the algorithm.
• Methodological note: trust questionnaires make defensive explanations look like winners. The primary metrics have to be rewrite quality and task completion. Simple lookups that finish in one turn cannot measure the purpose of explanation; use tasks that need a correction.

Where it stops holding

In audits, disputes, and security incidents the audience becomes an investigator, and the purpose may be to show why a ranking happened; query rewriting is no longer first. That explanation belongs in logs and tickets, not on every result row. Feature weights on an expert dashboard are a tuning tool, not copy for occasional lookup. When the results are already right and the person is only confirming, a short “title hit” is enough; a long menu of rewritable suggestions is noise.

Applying it

• Before writing an explanation, write “what will the person do after hearing this.” Delete sentences that yield no act.
• Wire explanations to controls: field name next to “this field only,” filter source next to shut-off, customization next to a general-list contrast.
• Do not use authority, model contribution, or score decomposition as result-page body copy. Leave those for internal debugging.
• Verify with a query that ranks wrong on the first turn (too broad a word, the wrong field, an implicit filter dropping the target). After the explanation, the next act should change that layer. If people start discussing whether the ranking is reasonable, or change nothing, the explanation is briefing, not helping lookup.

Related

• Within the group: G3.17.1 Results should say whether the match was in the title, body, tags, or another field · G3.17.2 Personalized ranking must be labeled as customized, not as the general result · G3.17.3 Implicit filters need their source disclosed · G3.17.4 Raw relevance scores are not fit to show end users
• Adjacent: G3.02 Query formulation · G3.05 Result ranking · G3.08 Filters
• Search terms: query reformulation · berrypicking · transparent relevance feedback