Raw relevance scores are not fit to show end users

What it is

BM25, cosine, a learning-to-rank logit, an “elastic score”—these numbers are for debugging a ranker, not for finding. Do not show raw relevance scores to end users. The number has no stable meaning: the same document, a different query, a different scope, a different model version, and the figure can jump an order of magnitude, yet people will read it as “matched N percent.” Explainability needs actionable reasons (which field hit, which filter applied), not false-precision decimals.

Debug modes and internal tools can keep the scores. That is a different user and a different UI, not to be mixed into the public result page.

Why it happens

A score is a relative quantity. It has no unit, no maximum, and cannot be compared across queries, but it looks like a measurement. Once 87.3 sits next to 12.1, people invent a story of “much worse,” even if both numbers are only rank codes inside one list. Later behaviour follows the story: drop the 12.1 row (still relevant), chase 87.3 (field inflation), even rewrite the query to “farm” the number. Explanation was supposed to support rewriting the query; the score sucks attention onto optimizing a figure with no external referent.

Scores also go stale without saying so. Swap the model, the number for the same query–document pair changes, and people who bookmarked a “high-score result” think the content changed. Rank is already easy to absolutize; writing the internal float supplies a finer scale for the same mistake.

Studying it

Compare “list only,” “ranks shown,” and “raw scores shown” on explanation quality and query rewriting.

• Paradigms: one result set, three annotations; ask “how do these two rows differ,” “which part of the query would you change next,” and watch for score-farming. In IR user studies, showing internal scores usually does not improve relevance judgments and sometimes hurts. Recruit assessors and search engineers separately from end users; do not pool them in one analysis.
• Independent variables: display (none / rank / raw score / normalized percentile), whether the score is stable across queries.
• Dependent variables: whether relative-quality judgments align with gold labels, whether rewrites aim at query words rather than at the score, rate of reading a score gap as “N percent worse.”
• Methodological note: assessors have already learned to ignore absolute score meaning; using them as participants underestimates harm. Use occasional users. Percentiles are a little better than raw scores and are still read as “percent match”; they are not a solution.

Where it stops holding

Relevance-assessment tools, TREC-style labeling UIs, and search-ops consoles exist in order to see scores; hiding them blocks the work. That is a workbench, behind a permission wall, not a product default. When a regulator must be told “why this row is first,” the score can go in an audit log without going on the result row. Gamified leaderboards (public contribution ranks) are not retrieval relevance scores; do not conflate them. Mapping scores onto three or five bins (“highly related / somewhat related”) still shows an internal judgment, only with less false precision, and does not replace actionable explanations such as field and filter.

Applying it

• Do not render BM25, model scores, or percent-match on the public result page. Put debug scores in an internally visible panel.
• Explain with attributes that have an external referent—field, source, applied constraints, time—not “score 73.”
• If the business insists on a “match” display, use at most a coarse band relative to this list, and say it “only marks position in this result set.”
• Verify by showing someone uninvolved a result page with raw scores, and asking what a gap between two scores means. If the answer is “this matched eighty percent, that only twenty,” or they start rewriting to raise the number, the score is already misleading. After removing it, the same person should talk about fields and whether the content is on topic.

Related

• Within the group: G3.17.1 Results should say whether the match was in the title, body, tags, or another field · G3.17.2 Personalized ranking must be labeled as customized, not as the general result · G3.17.3 Implicit filters need their source disclosed · G3.17.5 Explanations exist to help reformulate the query, not to prove the ranker is right
• Adjacent: G3.05 Result ranking · G3.06 Result snippets · T2.04 Error copy
• Search terms: relevance score · BM25 · search explainability