"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction

Document Title

"Help Me Help the AI": Understanding How Explainability Supports Human-AI Interaction

Document Information

• Subject Area: Design and application of Explainable Artificial Intelligence (XAI) in Human-Computer Interaction.
• Keywords: Explainable Artificial Intelligence (XAI), Human-Computer Interaction, Human-AI Collaboration, Computer Vision, Local Explanations, AI Explanation Needs, Case Study

Research Background and Issues

• Identified Problems and Challenges:

  • Despite the growing popularity of Explainable Artificial Intelligence (XAI) methods in recent years, research on the specific needs and usage behaviors of AI end-users remains relatively scarce.
  • Current XAI methods primarily focus on the needs of researchers and developers, failing to adequately support the requirements of ordinary users in real-world scenarios.
  • Existing XAI methods often struggle to meet end-users' practical needs when visualizing the vast number of model parameters and pixel outputs in computer vision systems, as they are typically constrained by researchers' capabilities.

• Significance of the Research:

  • As AI systems are increasingly applied in fields such as healthcare, transportation, and consumer services, improving their explainability is crucial for fostering trust and enhancing collaboration.
  • Explainability not only aids in understanding AI decisions but also helps users improve their own capabilities and decision-making quality, thereby enabling better collaboration with AI systems.

• Research Motivation and Related Work:

  • The authors identified three key research questions (RQs):
    1. What are the explanation needs of end-users in real-world AI applications?
    2. How do end-users intend to use XAI explanations?
    3. How do end-users evaluate existing XAI methods?

Solution

• Methods and Solutions:

  • A mixed-methods study was conducted, involving interviews, surveys, and explanation preference evaluations with 20 users of a real-world AI application (the Merlin bird identification app).
  • Four representative XAI explanation prototypes were designed based on user needs:
    • Heatmaps
    • Example-based explanations
    • Concept-based explanations
    • Prototype-based explanations

• Innovations:

  1. Focused on XAI needs in real-world scenarios, bridging the gap between XAI methods and end-user applications.
  2. Introduced diverse XAI design prototypes to understand how existing methods meet user needs and their specific applications.
  3. Emphasized how XAI can improve human-AI collaboration, rather than merely aiding in understanding AI outputs.

• Implementation Steps and Key Techniques:

  1. User Research and Background Segmentation: Selected a diverse user sample based on expertise in bird identification and AI cognition.
  2. Interview Design: Included user background information, open-ended questions, a survey based on an XAI question bank, and feedback collection on XAI prototypes.
  3. XAI Prototype Development: Developed visual explanations (heatmaps, prototype region matching) and textual data explanations (concepts, examples).
  4. Data Analysis: Extracted user XAI needs, applications, and perception evaluations through interview data and coding processes.

Research Findings

• Key Findings:

  1. User needs varied significantly based on background and interests, but all users consistently desired information that could enhance human-AI collaboration:
     • Regardless of technical background, users sought actionable insights from AI (e.g., functional scope, system limitations).
  2. Users expected XAI explanations to serve multiple purposes:
     • Calibrating trust in AI.
     • Improving their task-related skills.
     • Providing detailed feedback to developers for system improvement.
  3. Users generally preferred "part-based" explanation methods (concepts/prototypes) as they aligned more closely with human cognitive patterns, particularly for enhancing user experience and education.

• Comparison with Existing Solutions:

  • Existing heatmap and example-based methods, while easy to understand, were considered too vague and insufficiently actionable.
  • Concept-based and prototype-based methods were favored for their detailed, actionable, and modular explanations.

• Experimental and Evaluation Results:

  • Evaluation of Explanation Needs: Users broadly desired system transparency but showed varying levels of engagement depending on their background.
  • Preferences for Four XAI Methods: Prototype-based methods were the most popular, with users widely recommending combining multiple explanation formats to enhance information richness.
  • User Collaboration Perspective: Explanations were viewed as critical tools for improving input quality or optimizing collaboration with AI.

• Limitations and Future Directions:

  • Limitations:
    • The study focused on a single application (Merlin), which may require validation in other applications.
    • The needs of developers and other stakeholders were not addressed.
  • Future Directions:
    • Expand the user base and increase scenario diversity.
    • Develop multimodal and multi-format combined explanations.
    • Further investigate user-suggested feedback loop designs to create efficient mechanisms for human-AI collaboration.

Summary: This study highlights the diversity of end-user needs for explanations and demonstrates the potential of XAI methods to enhance trust, task collaboration, and educational purposes. The research underscores the importance of user involvement in XAI design and provides a roadmap and design recommendations for future XAI studies grounded in real-world scenarios.