Competitor metrics cannot be compared directly when collection methods are unpublished

What it is

A competitor’s published “87% completion,” “NPS 62,” or “200 million daily actives” cannot be treated as ticks on the same ruler as yours when the collection method is unpublished. That is incomparable competitor metrics. Unknowns include task definition, exclusion rules, sampling frame, weighting, survey timing, and whether bots are counted. Without the method, what is visible is a number the other party chose. A side-by-side table manufactures a common scale that the methods do not actually share.

Why it happens

Every experience number is a function of a definition. The other party has a motive to choose the function that looks good: drop difficult users from the denominator, pop the survey after success, count silent presence as active. Published numbers also pass through communications, usually without failure branches or intervals. If your number is computed on a research definition, the gap contains both product difference and method difference, and the method difference cannot be estimated. Side-by-side display reads method difference as product difference, and then pushes resources toward “chase that number” rather than toward your users’ real failures. Incomparability is not an accusation that the other party is lying. It is that subtraction has no definition without a shared experimental protocol.

Studying it

On the few public reports that include a method appendix, list definition differences and estimate how far the same raw behavior would move under each definition, as a lower bound on method gap. For numbers with no method, research output should stop at “they claim,” and record which method items are missing. Run both products in the lab or a diary under the same task script to get a method-controlled comparison, and state clearly that it is not their published number. Sensitivity also works: recompute your own data under several common competitor definitions and see how far rank can be moved by definition alone.

Where it stops holding

Regulatory disclosures, standardized tests, and shared industry studies can be compared laterally if they truly share a protocol; still check versions. Structural observations such as whether a feature exists or whether a step is present do not depend on the other’s metrics and can be compared directly. Buying a third-party panel does not automatically confer comparability; the panel’s own frame and task script may still differ from your users. Incomparability also does not forbid studying the other’s interface structure; it only forbids using their number as your target value.

Applying it

• Label competitor numbers “method unknown / method known”; unknown numbers must not enter a gap calculation.
• When a lateral conclusion is required, measure both sides yourself with the same script and success criterion, rather than citing their site.
• Ban method-free side-by-side tables; write “they claim X; we measure Y on our definition,” and do not assert a difference in between.
• Recompute your data under several definitions and show decision-makers how sensitive rank is to definition.

Related

• Same group: Q6.11.1 Internal benchmarks show own trends; external benchmarks show industry position; they serve different uses · Q6.11.3 Benchmark values drift as the industry moves and must be recalibrated · Q6.11.4 Lagging a benchmark does not necessarily mean a problem; interpret it against product positioning
• Adjacent: Q2.17 Competitive analysis and benchmarking · Q6.07 Experience measurement frameworks
• Search terms: incomparable metrics · unpublished methodology · competitor benchmark