Cost asymmetry should appear as a concrete classification threshold, not only as a stated principle

What it is

“We would rather miss than false-alarm” has not entered the system if the probability is still cut at 0.5. Asymmetry has to become a number: above this score fire, below that suppress. The most common failure is a principle that lives in a design doc while the threshold lives at a library default.

Why it happens

Cost-sensitive classification supplies a cut: if the two error costs are (c_{\mathrm{FP}}) and (c_{\mathrm{FN}}), the optimal threshold moves with their ratio and the base rate, and does not sit at probability 0.5. Uncalibrated scores still need probability calibration (Platt, isotonic) or “0.8” does not mean an 80% event. The usual engineering substitutes—maximize F1 or Youden—encode an implicit cost ratio that is usually close to symmetric. If the product says “fire less” and the number is still the F1 peak, the two logics fight. Writing the principle as a number also includes a reject band: a middle range that does not fire and asks for more evidence, instead of forcing every score into a two-class decision.

Studying it

In review, ask teams to show the cost ratio, the base-rate estimate, the calibration curve, and the threshold those imply. Compare field false-positive rates between products that have “only a principle” and products that wrote the principle into the cut. Ablation is simple: freeze the model, move only the threshold from the F1 optimum to the cost-sensitive point, and watch interruptions and misses. Studies that report a threshold with no calibration are not comparable.

Where it stops holding

When scores are not probabilities, or the calibration set is far from the field distribution, the computed number is false precision. Under extreme class imbalance, a threshold quoted to three decimal places is shoved around by sampling noise; it needs an interval or a stratified threshold, not a point estimate. Mixed rule-and-model systems may have no single threshold; asymmetry then has to be written into each rule’s conditions rather than pretending a global 0.5 exists.

Applying it

• Put fire threshold, suppress threshold, and reject band in the spec with the cost ratio used; forbid uncommented 0.5 left in code.
• Before launch, sweep the threshold at field base rate and mark the chosen point on a precision–recall curve.
• When the principle changes (“this week we fear interruption more”), change the number and version it; do not only change copy.
• Verify by reading production config and confirming the live cut matches the spec’s cost ratio; if it is still the training-script default, the principle did not land.

Related

• Same group: C9.06.1 False-positive and false-negative costs are asymmetric · C9.06.2 Threshold choice is a product decision, not an algorithm decision · C9.06.3 High-consequence actions must not be decided by a single sensor · C9.06.4 Medical alarms usually prefer false positives over false negatives; consumer settings often reverse that · C9.06.6 Reusing one sensor across functions may require different false-positive tolerances · C9.06.7 Evaluating false-positive cost must include long-term trust loss, not only the harm of a single error
• Adjacent: C4.24 Recognition Confidence and Bias Direction · C9.10 Individual Calibration and Baseline Drift
• Search: cost-sensitive threshold · probability calibration · reject option