A qualified entry must let two evaluators independently reach the same conclusion

What it is

A qualified check entry must specify its object, condition, evidence, and conclusion rule clearly enough that two qualified evaluators, without seeing each other's judgment, reach the same "pass / fail / needs evidence" call. Inter-rater decidability is what separates a genuine heuristic from a personal stylistic preference. This card follows the previous two: they settled whether extra entries are needed and where they should be induced from; this one settles whether an induced entry is actually fit for use — an entry distilled from a real incident can still fail in practice if its wording leaves too much room for interpretation.

Why it happens

A vague entry is unreliable because it leaves the core of the judgment to be filled in by each evaluator's own experience — "the interface should look professional" specifies neither the object of inspection nor the basis for judgment, so two evaluators each fill that gap using their own accumulated taste, and naturally arrive at different answers. This is not a difference in evaluator competence; it is that the entry never wrote the judgment rule down in the first place. A decidable entry makes this hidden fill-in-the-gap process explicit: it states exactly which field, process step, or state is being checked; it states the trigger condition — "both shown at once" or "one is disabled without a corresponding notice"; it states what evidence to look for — a screenshot, a log, or a full task-path recording; and it states which branch follows the judgment — block outright, issue a warning, or merely log for later review. Missing any one of these layers means two evaluators facing the same interface are actually working from two different, each individually reasonable, hidden assumptions, and disagreement is no surprise at all.

Where it stops holding

Agreement cannot be bought by writing an entry to an extreme narrowness — an entry that matches only one specific pixel position or one exact piece of copy can indeed get two evaluators to agree, but the cost is that it fails to recognize even a slight variant of the same problem, and it breaks the moment the interface has a minor redesign; trading scope for agreement this way is not worth it. A separate category of finding is inherently exploratory — "something about this flow feels off, but I can't pin down exactly what" — and that signal has value, but it should not be forced into a decidable acceptance entry pretending to be precise; it belongs as a lead to probe further in an interview or a deep-dive session, not as part of the acceptance checklist. Also, two evaluators agreeing only proves the entry itself is written clearly — it says nothing about whether the problem it catches actually matters. Decidability and severity are independent: once a decidable entry fires, the finding still needs a severity rating to determine where it lands in the fix queue.

Applying it

• Fix every entry's template to seven fields: purpose, object of inspection, trigger condition, decision rule, required evidence, known exceptions, and applicable role — fill in whatever is missing, and never leave a blank for the evaluator to improvise.
• Train new evaluators on a small test set with clear positive, clear negative, and ambiguous boundary examples, confirming their judgment on the boundary examples matches an experienced evaluator's, not just checking they get the easy ones right.
• Before every formal evaluation, have two evaluators render their judgment independently without communicating, then compare disagreements and feed each disagreement directly back into revising the entry's wording or adding an evidence requirement.
• How to check: log each entry's agreement result across evaluation rounds and track it by version; an entry that keeps producing disagreement should not be kept alive with the same wording — split it into finer entries, merge it into an existing one, or demote it to a suggestive cue that no longer serves as an acceptance criterion.

Related

• Same group: B3.20.1 Generic heuristics miss domain-specific failure modes and require additional entries · B3.20.2 Domain heuristics should be induced from the domain’s real problem base, not generic entries reworded · B3.20.4 Domain heuristics need retrospective testing against known problems; untested entries should not ship
• Nearby: Q4 Research Methods and Evaluation · B3.12 Severity Rating
• Search terms: inter-rater agreement · checklist item · decision rule · operational definition