Dissecting users' needs for search result explanations

Document Title

Dissecting Users’ Needs for Search Result Explanations

Document Information

• Subject Area: Human-Computer Interaction (HCI), Information Retrieval, Search Engine Design
• Keywords: Search result explanations, user experience, transparency, search engines, information retrieval, trust, user behavior, non-technical users, explainability, design recommendations

Research Background and Problem

• Problems and Challenges:

  • Search engines generally lack transparency, leaving users unaware of how search results are filtered or ranked. This often leads to misconceptions about search engine mechanisms (e.g., believing in paid rankings beyond the ad domain). Such opacity can reduce users' trust in search results and hinder their ability to efficiently find information.
  • While prior research has attempted to improve user experience by providing search explanations, these efforts often focus on "how to explain" rather than exploring "whether explanations are needed" and "under what circumstances explanations are beneficial."

• Significance:

  • Search engines are a primary tool for information access, profoundly shaping user perceptions. For instance, biases in search results can significantly influence user decisions, such as political preferences.
  • Offering more transparent search engine explanations can enhance user efficiency, trust, and awareness of potential biases in search results.

• Motivation and Related Work:

  • Existing research focuses on model explainability, recommendation explanations in recommender systems, or improving transparency in AI systems. However, there is insufficient exploration of how users perceive and need explanations in search contexts.
  • The research aims to investigate how non-technical users understand and require search explanations, addressing gaps in current studies.

Solution

• Research Methodology:

  • The study adopts a two-phase approach: an online survey followed by semi-structured interviews with non-technical users.
  • Research Questions (RQs):
    1. Under what circumstances do users question search results? What information helps them evaluate results better?
    2. For which search goals do users find search result explanations useful? What are the benefits of explanations, and what are the characteristics of user needs?
    3. How do users perceive existing search result explanations provided by Google and Bing?

• Innovations:

  • A systematic exploration of user needs, practicality, and the value of search result explanations in different contexts for the first time.
  • Comparison of user needs with existing search engine features (e.g., explanation tools provided by Bing and Google) to generate design improvement recommendations.

• Key Techniques and Implementation Steps:

  1. Online Survey (Preliminary Screening and Needs Assessment):
     • Target group: Non-technical users in the U.S., with screening criteria including age, frequency of search engine use, and low-to-moderate knowledge of AI/information retrieval.
     • Utilized the search goal framework proposed by Rose et al. to classify and analyze user search scenarios.
  2. Interviews (In-depth Understanding and Validation):
     • Conducted contextual inquiry tasks, observing user screen sharing and recording their behaviors and feedback during specific search tasks.
     • Presented users with existing search engine explanation features and collected their reactions and improvement suggestions.

Research Findings

• Key Findings:

  1. Contexts in Which Users Question Search Results:
     • Users often question results when encountering irrelevant results, unknown sources, advertisements, or inappropriate content.
     • Curiosity about result rankings (e.g., why a particular result is ranked higher) was also common.
  2. Practical Uses of Search Explanations:
     • Explanations are particularly needed for complex or high-stakes tasks (e.g., health-related queries, product purchases).
     • For simple, closed-ended questions (e.g., seeking a clear single answer), users found explanations unnecessary.
     • For complex or open-ended questions, explanations were seen as helpful in building trust, guiding search direction, or evaluating diversity and bias.
  3. User Perceptions of Existing Search Features:
     • Google’s explanations were perceived as too broad and obvious, while Bing’s webpage preview feature was more favored.
     • Users valued features that allowed for "feedback and disputability" of search results (e.g., removing results, providing feedback).
     • Privacy and personalization settings were appreciated for enabling users to adjust their search experience, though these features were underexplored.

• User-Identified Functional Requirements and Priorities:

  1. Convenience and Efficiency: Explanations should be concise and easy to read, aiming to assist in quick filtering and understanding.
  2. Action-Oriented: Users expect explanations to provide clear guidance for actions (e.g., reordering search results).
  3. Credibility and Source Indication: Explanations should indicate the authority and trustworthiness of the sources.
  4. Diversity Indication: Explanations should reveal diverse perspectives or potential biases in the information.

• Design Recommendations:

  • Introduce categorization and guidance tools for search results (e.g., topic filters).
  • Provide contextual information to help users refine search terms and optimize queries.
  • Enhance the visibility of "issues and feedback" features for search results.
  • Design personalized, action-supportive explanation interfaces based on user preferences.

• Experimental Results and Advantages:

  • Compared to existing features provided by Bing and Google, designs with enhanced explanations and functional transparency significantly improved user trust and selection confidence.
  • The study identified potential directions for improving current search explanation tools, such as webpage previews and diversity indicators.

• Limitations and Future Directions:

  • Limited sample coverage, focusing primarily on non-technical users in the U.S., which may not fully represent global user perspectives.
  • Insufficient consideration of the impact of AI language models, necessitating further research on how they change user needs for search explanations.
  • Future research should emphasize diverse user backgrounds, the influence of personalization settings on explanation needs, and validating the effectiveness of different explanation formats (e.g., graphical or interactive).