Threshold choice is a product decision, not an algorithm decision

What it is

A model emits a score or a probability. The line that cuts it into “fire / don’t fire” is the operating point. Where that line sits decides who is interrupted and who is missed. It is not a byproduct of training loss converging. It is a decision the product must own: which users, which consequences, written into the default threshold. The algorithm can supply a curve; it cannot make the choice.

Why it happens

The optimum under cross-entropy or F1 is a statistical compromise on the labeled distribution with symmetric cost, usually near 0.5 or at maximum Youden’s index. The product faces field base rates and costs in support tickets, harm, and uninstalls—none of which typically enter the loss. Labor therefore splits: the learner is responsible for ranking quality of scores; the product is responsible for driving a nail into that ranking. Handing the nail to “the split with highest accuracy” lets class balance in the labeled set decide for the user. Thresholds also enter operations: raising one cuts complaints, lowering one lets a team claim sensitivity. Those are product trades, often wrapped as a model version number that hides who actually decided.

Studying it

Freeze scores from one model, sweep only the threshold, and plot interruptions versus misses at the field base rate. Factor: who chooses (algorithm default / clinician / product manager / end user). Outcomes: the chosen point, later regret, and whether teams can restate why the number is that number. A/B tests of thresholds are ignored more often than A/B tests of model structure, and expose product judgment more clearly. Whether users may override the default is itself a research question.

Where it stops holding

A certified medical device may have its operating point fixed by a standard; the product cannot slide it freely. Some real-time systems are latency-bound to a fixed gate and cannot run a cost-sensitive search. User-set thresholds work for experts and become an unreadable slider for most consumer use. When the model is badly miscalibrated (scores are not probabilities), talking about “0.5” is meaningless; calibrate before asking who picks the cut.

Applying it

• Write the default threshold into the product spec, reviewed with copy and permissions, not left unsigned beside the training script.
• In release notes, say when an operating point moved (“sensitivity from x to y”), not only “model upgrade.”
• Give users who need it a bounded override (fewer interruptions / fewer misses) and show which side the current bias favors.
• Verify by asking someone who did not train the model who set the trigger and which loss table they used; if they cannot answer, the algorithm is overreaching.

Related

• Same group: C9.06.1 False-positive and false-negative costs are asymmetric · C9.06.3 High-consequence actions must not be decided by a single sensor · C9.06.4 Medical alarms usually prefer false positives over false negatives; consumer settings often reverse that · C9.06.5 Cost asymmetry should appear as a concrete classification threshold, not only as a stated principle · C9.06.6 Reusing one sensor across functions may require different false-positive tolerances · C9.06.7 Evaluating false-positive cost must include long-term trust loss, not only the harm of a single error
• Adjacent: C9.12 Implicit Interaction and System Initiative · C4.24 Recognition Confidence and Bias Direction
• Search: operating point · decision threshold · cost-sensitive classification