Ranking criteria need to be explainable to the user

What it is

The order of a result list is a claim about what “better” means. Ranking criteria need to be explainable means the user can say, in their own words, why this row sits above that one—match to the query, popularity, distance, paid placement—rather than facing a black-box sequence that cannot be questioned. Hearst treats understandability of ranking as a search-UI problem: people use order to decide which rows to see and when to stop; if order cannot be explained, the stopping rule cannot form.

What must be explainable is the kind and direction of the criterion, not an internal relevance score to three decimal places. The score has no operational meaning for the end user.

Why it happens

While scanning, people build an expectation that “further down is less worth seeing.” That expectation has to bind to a stable dimension: relevance means further down is more off-topic, recency means older, distance means farther. If the dimension is undeclared, three expectations collide—row three is newer but ranked lower, and the user does not know whether to keep going or to suspect a broken system. Working memory treats order as a property of the content (“first means official”); a wrong criterion is read as a wrong fact.

There is a second injury: people cannot change the query or a constraint to change the order. Relevance is not working, but they cannot tell whether to swap words or to swap “sort by time.” Even if a sort control exists, unnamed criteria are anonymous schemes to poke at. Once a criterion is speakable, order becomes a rebuttable claim, and the rebuttal has somewhere to land.

Studying it

Ask “what do you think this is sorted by,” not “do you think ranking is accurate.”

• Paradigms: the same result set with different criterion labels, or none, measuring predictions of “what the next row will be like”; post-task paraphrase of the criterion; Hearst’s SUI evaluations reporting ranking understandability separately from precision. SERP studies recording whether stop position moves with the declared criterion.
• Independent variables: whether the criterion is named, whether the name matches the real algorithm, whether an undeclared second criterion (popularity, personalization) is mixed in.
• Dependent variables: accuracy of paraphrasing the criterion, noticing of order anomalies, early stopping from not understanding order.
• Methodological note: high precision masks inexplainability—the list happens to work, and participants still cannot name the criterion. Score task success and “can name the criterion” separately. A lab with only one criterion never sees expectation conflict from mixed ranking. Showing internal scores induces people to use the number instead of a criterion name, which is exactly what should not happen.

Where it stops holding

A single, shared, commonsensical criterion (mail newest-first) can be labeled lightly, but a change (to “important”) must be redeclared. Personalized ranking’s criterion is “for you,” not the same sentence as general relevance; “comprehensive sort” will not cover it. Inserted ads break “further down is less relevant” and must be carved out of the organic criterion. In a collection that fits on one screen, explaining order is less valuable, but order should still not fight visible fields (date, price).

Applying it

• Near the list, name the current criterion in words people can repeat (relevance, newest, distance, popular); the name must match the real primary dimension.
• When a second dimension is mixed in, say so (“relevance; newer first among equally relevant”), rather than stopping at “smart” or “comprehensive.”
• When order fights a visible field (recently updated, ranked last), have a speakable reason ready, or let the user change criterion; do not hope nobody notices.
• Verify after a search: “what is this sorted by, and what will I see further down.” No answer, or an answer opposite the real primary dimension, means the criterion was not explained. Change the label so it disagrees with the algorithm; if people still act on the label, they trust that sentence—the sentence has to be true.

Related

• Within the group: G3.05.2 Ads and organic results must be visually distinct · G3.05.3 Relevance ranking and recency ranking must be switchable
• Adjacent: G3.09 Sort controls · G3.17 Search result explainability · G3.06 Result snippets
• Search terms: ranking · ranking explainability · search user interface