A predicted difference smaller than individual variation is not design evidence

What it is

When a modelled time difference between designs is smaller than natural variation across people, trials, or device conditions, it is not reliable design evidence. Individual differences are not disposable noise; they set the scale for whether a model difference can be detected and reproduced in real use.

Why it happens

Models yield average predictions, while skill, strategy, motor control, attention, and environment form broad distributions. A difference smaller than that spread can reverse through sampling, parameter uncertainty, or daily state. Even a correct mean direction may not justify sacrificing other objectives for an imperceptible gain.

Studying it

Report distributions, effect sizes, confidence intervals, and paired individual differences, not only means or model points. Replicate key tasks with representative users and devices to test stability across conditions. Define what counts as practically meaningful before interpreting any observed difference as improvement.

Where it stops holding

Small mean differences can accumulate for frequent, large-scale, or safety-critical work, but frequency, consequence, and uncertainty must be explicit. Conversely, large variation does not mean abandoning optimisation; seek designs that benefit weaker users too or provide adaptation.

Related

• Same group: B1.17.1 Laws calibrated under separate conditions double-count preparation and response when added · B1.17.2 Parallel operations cannot be estimated by adding time · B1.17.3 Adding group-average predictions amplifies rather than cancels error · B1.17.5 A law's ordering of alternatives is more credible than its absolute value
• Nearby: B1.01 Fitts's law · B1.04 Hick-Hyman law
• Search terms: individual differences · effect size · practical significance