“Do users like it?” needs decomposition before it is researchable
Aliases: preference question · liking measure · attitude measurement
What it is
Liking is researchable; the raw question “Do users like it?” is underspecified because it lacks an object, population, use context, and operational meaning of like. It collapses visual appeal, perceived ease, brand attitude, choice, and polite approval into one preference question. A yes/no judgment neither identifies its basis nor establishes usability, sustained adoption, or purchase. The study must first decide whether it needs immediate attitude, relative choice, intention to use, or an explanation of how the judgment formed.
Why it happens
Overall liking is a low-resolution summary constructed at the time of questioning. Wording, presentation order, researcher presence, and social desirability affect which salient attributes a respondent samples. Different evaluations can therefore produce the same answer. Preference also exhibits an attitude–behavior gap: a person may like a novel interface yet retain the old one because of habit, switching cost, availability, or organizational rules. Liking indicates affective direction but cannot by itself locate a cause or predict action.
Studying it
For immediate attitude, define the object and time frame, then use attribute ratings, paired comparisons, or rankings according to whether the outcome is a score, relative preference, or order. To understand reasons, elicit a recent concrete episode, expectations, and consequences rather than asking respondents to defend a global score. To study adoption, add behavioral outcomes such as voluntary continuation, actual switching, return use, or payment, while recording opportunity and cost. There is no universally best order for open and closed questions: a prior rating can anchor a later account, while prior recall can alter a subsequent global judgment. Let the primary inference determine the sequence; when both matter, randomize or counterbalance order and report it as a design condition. An open response is also distinct from selecting among researcher-supplied reasons, which directly constrains what can be mentioned.
Where it stops holding
Overall liking can be useful for concept screening, aesthetic exploration, and satisfaction tracking. The error is treating it as sufficient evidence, not studying it at all. Anonymous large samples reduce interviewer effects but do not repair construct ambiguity. Forced choice yields a ranking without showing that either option is acceptable. Observed behavior is not automatically a truer preference because access, migration cost, and existing accounts constrain choice; attitude and behavior remain distinguishable evidence.
Applying it
- State which decision a liking judgment could change; remove the item if every answer leads to the same action.
- Specify object, population, context, and time frame, then decompose liking into decision-relevant attributes such as clarity, trust, comfort, or willingness to return.
- Pair one behavioral signal with one explanatory signal rather than projecting a rating onto adoption.
- Test the interpretation with a counterexample: if someone rates the design highly but fails the task, the report should preserve and explain both observations.