Domain heuristics need retrospective testing against known problems; untested entries should not ship

What it is

Before a new domain entry is published, it should be back-tested against a set of interfaces with known problems and known non-problems: can it find what it should find, does it false-positive often, does it duplicate an existing entry. Retrospective validation turns a heuristic from "sounds reasonable" into an evidence-backed checking tool. This card is the final gate after the previous three: an entry induced from a real problem base (card two) and written so two evaluators can judge it consistently (card three) can still perform poorly in practice — back-testing is the last real check applied before an entry formally enters the list.

Why it happens

An entry a domain expert writes from experience, even when clearly worded and internally consistent, commonly distorts in one of two directions: too broad, so that almost any interface triggers it, making the entry look like it "always finds a problem" when it has actually lost any power to discriminate; or too narrow, precisely matching only the one specific case the author had in mind while writing it, and failing to catch a slightly different variant of the same interface entirely. Neither distortion is easy to spot just by reading the entry's text, because the wording reads as reasonable either way — only running it against a known set of positive and negative reference interfaces reveals how many hits it gets on positive samples, how many false positives on negative samples, and how much it overlaps with an existing entry. Only these three numbers together answer whether the entry is worth publishing, rather than an expert's intuition that "this sounds like it should be useful."

Where it stops holding

The back-test set itself carries bias, and passing it does not guarantee validity against every future case — a known problem set reflects failure modes that have already been observed and recorded in the past, and once new technology or new interface forms appear, an entry needs continuous monitoring rather than a one-time back-test that settles the matter forever. A low-detection entry should not automatically be discarded either: if it targets a failure with extremely severe consequences but that is inherently hard to observe (a safety issue that only triggers under an extreme edge condition), it can still be worth keeping as a supporting review cue even with a low hit rate in back-testing — it is simply unsuited to standing alone as a hard acceptance criterion. Conversely, an entry with a high false-positive rate should not enter the formal acceptance checklist no matter how professional it sounds, because it will keep generating noise that requires manual triage, steadily draining evaluators' trust and effort over time. Back-test results from a small sample need their uncertainty reported honestly — an entry that happens to perform well on a small sample should not be treated as already validated.

Applying it

• Build a known problem set where each record includes the interface version, supporting evidence, severity, domain category, and eventual fix outcome, paired with a set of already-fixed or never-problematic interfaces as negative controls.
• Have evaluators who did not write the candidate entries perform a blind evaluation — they should not know which interfaces are positive samples and which are negative — and log each candidate entry's hits and false positives, so knowing the answer in advance does not contaminate the judgment.
• Compute detection rate, miss rate, overlap with existing entries, and false-positive rate for each candidate entry, set a clear publication threshold, and demote anything below it to a supporting cue or send it back for re-induction.
• How to check: after an entry goes live, keep tracking what it finds in real evaluations, review it quarterly, and feed newly discovered real cases back into the known problem set, so the back-test set itself keeps updating with the domain instead of staying frozen as a one-time historical snapshot.

Related

• Same group: B3.20.1 Generic heuristics miss domain-specific failure modes and require additional entries · B3.20.2 Domain heuristics should be induced from the domain’s real problem base, not generic entries reworded · B3.20.3 A qualified entry must let two evaluators independently reach the same conclusion
• Nearby: Q4 Research Methods and Evaluation · Y3 Error Prevention and Recovery
• Search terms: retrospective validation · detection rate · false positive · blind evaluation