Users misread a lucky correct answer as stable competence

What it is

The model writes a hard regex correctly on the first try and the user says “it knows regex.” A later miss on a cousin request is felt as regression or sabotage, not as “last time we drew a good region of the distribution.” The lucky-hit competence illusion treats a single sample as a point estimate of ability.

This is not the fact that outputs vary. It is the leap from one success to the next trial, the next class, the next domain. The raw material is success, not failure.

Why it happens

People attribute ability from tiny samples. Availability, plus a bias toward one-shot induction, lets a vivid hit outweigh unseen misses. Generative quality is high-variance: a prompt may pass 30% of the time while the user saw the one pass they happened to draw.

The interface fans this. Success arrives whole, fluent, confident; failure is what gets a warning. What is encoded is “it can,” not “it sometimes can.” Task selection then follows that portrait: neighbouring untested work is handed over until a loud miss yanks the estimate back, usually too far.

Studying it

Show one success (or one failure), then ask for estimated success on a new item from the same distribution, a near neighbour, and a far transfer, plus whether the next item should be handed to the system. Independent variables: correctness of the first sample, whether a base rate is shown, surface similarity of the tasks. Dependent variables: calibration error, scope of delegation, size of withdrawal after a miss.

The base rate has to be measured on the item in front of the user, not borrowed from a model card. A lab that then runs five similar items will correct the illusion with data; the dangerous case in products is sparse use — once a day.

Where it stops holding

If people can see a small distribution — several options side by side, or a history of the same prompt — a single hit is diluted. Experts in their own domain are better at spotting a fluke; novices and out-of-domain requesters most readily read fluency as mastery. On low-variance tasks (pulling a determined field from a database), one correct answer really is close to evidence of ability, and this claim weakens. It also does not cover the other asymmetry: a system that is usually right and then fails loudly.

Applying it

• Next to a first success, give a base rate or a range, not a celebration: “this class of item is usable as-is about one time in three; check the rest.”
• Offer try-again while keeping the last result, so success is visibly not the only point in the set.
• Do not stock empty states and marketing with only the best draw. Put typical misses on the same layer as typical hits.
• Check: give a new user one polished success, then ask what the next similar item will be like. If the answer is close to “it will nail it” and your pass rate is far below that, the illusion is already in place. Watch whether they also hand over a neighbouring task you never demonstrated.

Related

• Same group: L1.01.1 The same input can yield different outputs · L1.01.2 Interface conventions assume actions are repeatable and results are stable · L1.01.4 Controls promise that the same action yields the same result; generation breaks that promise · L1.01.5 Retry cannot tell whether the phrasing was wrong or the system itself is fluctuating · L1.01.6 Undo and redo lose their meaning when output cannot be reproduced · L1.01.7 Presenting regeneration as “refresh” implies the previous result was a failed load · L1.01.8 Presenting variability as several parallel options is more honest than hiding it behind a single result
• Nearby: L5.03 Trust calibration · L5.09 Overtrust and trust collapse · L3.03 Hallucination and the fact-checking burden
• Search terms: lucky-hit competence illusion · overgeneralization from success · base-rate neglect