An industry benchmark whose source you cannot inspect is not a comparison

What it is

Vendor white papers, consultant decks, and “industry-average recommend scores” often offer a single number with no item wording, sampling frame, collection year, exclusion rules, weights, or response rate. That number is an opaque benchmark: you cannot see who was measured, at which touchpoint, with which sentence. Subtracting it from a company score yields a gap with no interpretable unit—it may be the industry, or the stem, the channel, or who was dropped. The objection is not that benchmarking is illegitimate; it is that an unauditable norm cannot serve as a comparison group.

Why it happens

Recommendation metrics move with the measurement process: when the prompt fires, whether skip is allowed, whether “non-real users” are deleted, whether the scale is 0–10 or 1–5, whether a comment is required. External benchmarks almost never publish those choices, yet they arrive with the authority of “the industry.” Citing them adopts someone else’s undeclared operationalization as your ruler. The nicer the number looks, the easier it is to treat as a fact that needs no methods appendix. When the supplier also sells score-improvement services, the incentive is to make clients look below average, not to make the average reproducible. An unauditable norm is therefore not a neutral backdrop; it is a control arm you cannot audit.

Studying it

Review a candidate benchmark as a study with missing methods: list which of item, population, window, response rate, and cleaning are unknown. If the unknowns cannot be filled, tag the figure as hearsay. If the vendor provides microdata or at least distributions by sector and channel, check whether that operationalization matches yours; on a mismatch, stop the numeric comparison and at most discuss direction. A company’s own prior window, a randomized control, or a pre-registered internal baseline is cheaper and inspectable. For an external number already in a report, trace the first slide it appeared on and see whether the definition changed in transit.

Where it stops holding

When a regulator or buyer names a public norm, the organization may have to report beside it, but the methods gap still needs a label and an inspectable internal contrast should sit next to it. A meta-analysis or open dataset with a complete protocol can serve as a norm if item and population match. Last-year-today internally is not an industry benchmark; its transparency comes from your own logs, and it should not share a reference line with an external black box.

Applying it

• Citation rule: a benchmark that cannot name its item, sample, and year does not enter a conclusions page, internal or external.
• Add an “inspectable?” column to any benchmarking table; rows marked no belong only in an appendix with the unknowns listed.
• Default the comparison to the product’s previous window or an experimental control, not to “the industry average.”
• Check: delete the benchmark from the report and see whether the claim still stands. If what remains is “we sit below a number of unknown origin,” drop the claim.

Related

• Same group: Q3.17.1 Cultures occupy the same scale differently · Q3.17.3 These scores are lagged attitude snapshots, not causes · Q3.17.4 Intercepting at a high-score moment inflates the number
• Adjacent: Q3.03 Net Promoter Score · Q3.15 Questionnaires and rating scales
• Search terms: opaque benchmark · industry norm · comparability