People who submit feedback are a self-selected few; extreme experiences are over-represented

What it is

A thousand generations, twelve thumbs. Most of those twelve are the people most angry or most delighted. The large middle — “fine, a bit wrong, not worth a click” — is silent. Tuning a model on the twelve lets extreme experience stand in for everyone. Self-selected feedback means submitted samples are not a random quality audit; extremes are over-represented and the middle barely enters the data.

A down with no location is grain. Here even “who clicks” is already skewed.

Why it happens

Submitting costs a little; only people whose utility exceeds that cost pay: fury, delight, a job that requires reporting. Middling quality, middling affect, never crosses the threshold. The distribution then thickens at the ends and hollows the middle. Retraining or roadmaps that count feedback items go to fix extreme cases and miss the moderate failures that are the bulk of volume.

Silence is not neutrality. People who settled and sent, who finished with neither up nor down, are often the product’s actual everyday quality. Reading them as “no comment, therefore pass” treats nonresponse as a pass.

Studying it

Audit quality on all sessions (human or gold), then compare with the voluntary feedback set. Dependent variables: share of the two ends, coverage of middle quality inside feedback, whether tuning on feedback drops everyday error. Independent variables: friction of the feedback entrance, whether an active sample prompt appears.

Active sampling (a random “please rate this one”) is the control, to estimate bias in the voluntary set.

Where it stops holding

Safety and abuse must travel an extreme channel; do not suppress them because they “over-represent.” On a tiny product, voluntary feedback may be the only signal — still interpret it as extreme samples. Edits cover more of the silent users and can correct this skew — another card. This entry does not treat whether feedback gets an echo.

Applying it

• Do not use voluntary thumbs as the only numerator for training or roadmaps. Pair with a random audit that covers the silent middle.
• Label voluntary feedback as “extreme sample” on reports, not “user quality score.”
• Lower the cost of expression for middle users (one random item, one question, no long rationale), but do not expect them to become as frequent as extreme users.
• Check: compare error types in voluntary feedback with a random audit. If voluntary is almost only abuse and rapturous praise, and middling factual errors never appear, self-selection is already in charge.

Related

• Same group: L3.13.1 Thumbs up and down collect satisfaction, not correctness · L3.13.2 Negative feedback that does not point at a location cannot localise the problem · L3.13.4 Edits are implicit feedback with more information than an explicit rating · L3.13.5 If feedback produces no visible change, submission decays toward zero
• Nearby: L3.12 Editing and Taking Over Generated Content · L6.13 Negative Feedback Channels for Recommendations
• Search terms: self-selection bias · extreme responding · nonresponse