Ratings support ordering rather than absolute judgment
Aliases: prioritization · severity scale · issue triage · ordinal scale
What it is
A severity score is a relative signal that helps a team decide what to fix first; it is not an objective physical measurement of the problem. The same "3" can correspond to a different population size and risk level across products, samples, versions, or business cycles; severity rating should always be read together with its evidence, assumptions, and fix cost. This card builds on the previous one, "severity is composed of frequency, impact, and persistence": that card explains how the score gets computed, this one governs what you are allowed to do with it once computed — and the answer is: rank with it, do not compare it as an absolute quantity.
Why it happens
A severity scale is, at bottom, an ordinal scale: it only guarantees that "a 3 needs attention before a 2," not that "a 3 is exactly 1.5 times as severe as a 2." Rating compresses observations from multiple, qualitatively different sources — usability testing, logs, tickets, field observation — into a handful of discrete levels, and that compression necessarily discards the detail and units of the original evidence, detail that cannot be recovered afterward just by comparing the resulting numbers. This is why averaging scores from different raters is a common misuse: an average assumes the intervals between scores are equal, which an ordinal scale never guarantees — two raters who both give a 3 may be relying on completely different strength of evidence. The real value of the rating lives in relative resource allocation: under limited time and staff, address whatever ranks higher first, and keep a reviewable chain of reasoning for why — not argue in a meeting room over whether something is a 2.6 or a 2.8.
Where it stops holding
Some categories cannot be governed by this ranking logic at all and need a hard threshold that bypasses the queue: a fully blocked task, data loss, a security or legal compliance issue, or a violation of a minimum accessibility requirement must not be indefinitely deferred just because its composite score ranks lower — the order these get handled in is set by whether they cross a line, not by the severity score, and the scale steps aside for a rule here. Conversely, a low-scoring issue with a very low fix cost (a copy tweak, a spacing adjustment) is worth handling in passing rather than shelved purely on rank — what actually decides priority here is "score divided by fix cost," not the score alone. A rating also cannot substitute for genuine quantitative impact measurement: if the product can directly measure the conversion drop or ticket increase a given issue causes, that real business data should outrank a manual rating; rating is better suited to an earlier stage where such data does not yet exist and expert judgment has to provide a first-pass ordering.
Applying it
- In every issue record, log the severity level, the specific evidence behind it, the assumptions made at rating time, the estimated affected population, the fix cost, and the expected benefit, so the ranking can be traced later instead of leaving only an isolated number.
- Use the level for everyday ranking, but set a separate release gate for the highest-risk categories (blocking, data loss, security, compliance, accessibility) that does not participate in the ordinary score queue.
- Attach one or two real examples to each level as a reference anchor, so different raters share a common calibration point instead of two people both writing "3" for completely different reasons.
- How to check: periodically audit the ranking by asking the team why the top five items are ranked where they are — if the answer cannot cite specific evidence and only says "it felt more severe," the rating is being used as an absolute judgment rather than a ranking tool. Re-rate whenever the business context shifts (a new release, a changed user base) instead of carrying old scores forward.