Bias-Aware Design for Informed Decisions: Raising Awareness of Self-Selection Bias in User Ratings and Reviews

People often take user ratings/reviews into consideration when shopping for products or services online. However, such user-generated data contains self-selection bias that could affect people’s decisions and it is hard to resolve this issue completely by algorithms. In this work, we propose to raise people’s awareness of the self-selection bias by making three types of information concerning user ratings/reviews transparent. We distill these three pieces of information, i.e., reviewers’ experience, the extremity of emotion, and reported aspect(s), from the definition of self-selection bias and exploration of related literature. We further conduct an online survey to assess people’s perceptions of the usefulness of such information and identify the exact facets (e.g., negative emotion) people care about in their decision process. Then, we propose a visual design to make such details behind user reviews transparent and integrate the design into an experimental website for evaluation. The results of a between-subjects study demonstrate that our bias-aware design significantly increases people’s awareness of bias and their satisfaction with decision-making. We further offer a series of design implications for improving information transparency and awareness of bias in user-generated content.