Visual, textual or hybrid: the effect of user experience on different explanations

Title of the Paper

Visual, textual or hybrid: the effect of user expertise on different explanations

Bibliographic Information

• Research Area: Human-Computer Interaction and Explainable Artificial Intelligence (XAI)
• Keywords: XAI, Explainable Artificial Intelligence, Visual Explanation, Textual Explanation, Hybrid Explanation, User Expertise, Domain Expertise

Research Background and Issues

• Problems or Challenges:

  1. With the widespread adoption of AI technologies, the demand for transparency and explainability has increased. However, mainstream methods still rely on "one-size-fits-all" universal explanation strategies, lacking customization for different user types.
  2. There is limited research on tailoring explanations based on user expertise, especially empirical evaluations of how different explanation approaches impact human interpretability.
  3. Studies in decision support and recommendation systems have found that simple explanations are often preferred, but this knowledge has not been sufficiently disseminated to the broader XAI field.

• Significance:

  • AI systems are becoming increasingly important in everyday decision-making. Researching explanation types tailored to different user needs can enhance system transparency, user trust, and efficiency.
  • Customized explanations can not only improve comprehension across diverse user groups but also reduce biases and misuse.

• Motivation and Related Work:

  1. Ribera et al. noted that the goals of explanations vary by user type (developers often use them for system verification, while general users focus on understanding their own situations).
  2. Mohseni et al. and Gunning et al. emphasized the importance of designing explanations based on users' professional backgrounds, a challenge that remains insufficiently addressed in existing research.
  3. Designing effective explanation models requires integrating insights from psychology, cognitive science, and other fields, yet many current designs lack such theoretical support.

Solution

• Methods or Solutions: The study designed and evaluated three types of explanations:

  1. Visual Explanation (Partial Dependence Plot, PDP): Visualizes how parameters influence outputs.
  2. Textual Explanation: Uses templated sentences to succinctly describe the relationship between parameters and predictions.
  3. Hybrid Explanation: Combines textual descriptions with visual explanations to help users better understand graphical content.

• Innovations:

  • Proposed and evaluated a new hybrid explanation model (text + graphics) aimed at bridging the gap between expert and general users by designing explanations tailored to their needs.
  • Combined task completion rates, user preferences, and user cognitive process modeling (via the Think Aloud method) to evaluate explanation effectiveness.

• Implementation Steps and Key Techniques:

  1. User Grouping: Divided users based on AI expertise (expert users vs. general users).
  2. Explanation Task Design: Users completed tasks based on C-local, C-counter, and C-importance capabilities.
  3. Evaluation Combining Subjective and Objective Metrics:
     • Subjective: Recorded user ratings on usability, intuitiveness, etc., using Likert scales.
     • Objective: Measured task completion accuracy.
  4. Think Aloud (TA) Protocol captured users' thought processes, analyzing sources of misunderstanding.

Research Findings

• Specific Findings:

  1. Textual explanations significantly outperformed visual explanations in accuracy (especially for general users).
  2. Despite this, most users preferred visual explanations due to their comprehensiveness and more efficient interaction experience.
  3. Hybrid explanations showed significant improvements in comprehension accuracy compared to pure visual explanations (accuracy for general users increased from 5% to 50%; expert users from 75% to 80%) without significantly increasing usage complexity.

• Advantages Over Existing Solutions:

  • Balanced user comprehension and preference: Hybrid explanations enhanced general users' understanding while satisfying expert users' needs for intuitiveness and time efficiency.
  • Textual supplements to preferred visual explanations retained high user preference while improving interpretability.

• Experimental or Evaluation Results:

  • First Round of Experiments:
    • General users achieved only 5% task completion accuracy with visual explanations but showed high preference (>70% chose visual explanations).
    • Expert users performed better but required additional familiarity with visual explanations, leading to increased time costs.
  • Second Round of Experiments:
    • Hybrid explanations significantly improved general users' accuracy and exploration behaviors; expert users primarily benefited from reduced explanation time costs.
  • Anomalous Behavior: General users were prone to cognitive biases (confirmation bias) under visual explanations.

• Limitations and Future Directions:

  1. The study did not incorporate more complex measurement dimensions, such as cognitive load or learning outcomes, which future research could address.
  2. The two experiments used different designs (Round 1: within-subject design; Round 2: between-subject design). Future studies should integrate visual and hybrid explanations into a single experimental scenario for comparative evaluation.
  3. Hybrid explanations showed limited effectiveness in enhancing task exploration depth (reducing visual information avoidance behaviors). Dynamic interaction techniques or optimized textual content could be explored further.