The Who in XAI: How AI Background Shapes Perceptions of AI Explanations

Title of the Paper

The Who in XAI: How AI Background Shapes Perceptions of AI Explanations

Paper Information

• Domain: Human-Computer Interaction, Explainable Artificial Intelligence (XAI)
• Keywords: Explainable Artificial Intelligence, Human-Computer Interaction, User Background, Cognitive Bias, Design Optimization, Algorithmic Trust

Research Background and Problem

• Identified Problems or Challenges:

  • Many current designs of Explainable Artificial Intelligence (XAI) fail to adequately consider users' background characteristics, such as whether they possess AI-related knowledge.
  • Different groups may perceive the same AI-generated explanations in fundamentally different ways, potentially leading to inappropriate trust or misunderstandings.
  • Although AI transparency is a critical aspect of system design, its actual impact may not align with design objectives.

• Significance:

  • AI explainability is a crucial foundation for ensuring user trust and responsible decision-making, particularly in high-risk domains such as healthcare, law, and finance, where misperceptions can have severe consequences.
  • Understanding how user backgrounds influence the perception of explanations can help bridge the gap between developers and end-user experiences, thereby fostering more efficient and responsible human-machine collaboration.

• Research Motivation and Related Work:

  • Research on explainability originated in the 1980s with expert systems, but the rise of technologies like deep learning has intensified the conflict between model visualization and user understanding.
  • Existing studies have explored the cognitive processes and user acceptance behaviors related to algorithmic explanations, but few have focused on user perception across different backgrounds.
  • The authors propose that explanations are not merely a transmission of information but also a process of meaning-making by users, significantly influenced by their characteristics.

Solution

• Method or Solution:

  • The study proposes a mixed-method research framework:
    • Quantitative analysis of preferences among different groups for three types of AI explanations (natural language explanations, plain natural language descriptions, numerical explanations);
    • Qualitative analysis to explore the underlying cognitive processes, differences, and commonalities.
  • Through scenario-based experiments, AI explanations were designed in three styles:
    1. Natural Language Explanation (with reasoning): Explains "why" a certain action was taken;
    2. Plain Natural Language Description: Describes "what action" was executed;
    3. Numerical Representation: Transparently displays the AI's decision basis (e.g., Q-values).

• Innovative Aspects:

  • Incorporating user background into the XAI research framework, systematically exploring how background differences affect the perception of explanations through experimental design.
  • Introducing "cognitive heuristics" and "technological appropriation" as analytical perspectives to elucidate the mechanisms behind user behavior.

• Implementation Steps and Key Techniques:

  • Sampling two user groups: one group of AI students and another group of end-users without AI backgrounds.
  • Participants watched videos containing different explanation styles and were asked to rate their preferences across five dimensions (comprehensibility, intelligence, trustworthiness, friendliness, and post-failure trust recovery) while analyzing the sources of their preferences.

Research Findings

• Specific Findings:

  1. Significant differences were observed in how AI-background and non-AI-background groups perceived AI explanations:
     • The AI-background group preferred numerical explanations and exhibited overconfidence in their intelligence;
     • The non-AI-background group focused more on the intuitiveness and credibility of natural language explanations.
  2. Instances of "unintentional information appropriation" and "misplaced trust behaviors" were identified, with differing attributes influencing the two groups:
     • The presence of numerical data was mistakenly associated with algorithmic logic and intelligence by both groups;
     • Non-natural language numerical explanations appeared "intelligent" to non-AI users, despite their limited understanding of the content.
  3. The designed natural language explanations were universally preferred by both groups, deemed more interactive and engaging, and suitable for applications across various scenarios.

• Comparison with Existing Solutions:

  • Compared to explanation methods solely focused on utility maximization, this study introduces a fine-grained consideration of user backgrounds, providing theoretical support for designing diversified explainable systems.
  • This is the first systematic validation of how users' educational backgrounds alter habitual cognition and tool appropriation, revealing the trust risks behind standard numerical representations.

• Experimental Results and Evaluation:

  • Quantitative preference analysis across five dimensions highlighted the comprehensive advantages of "natural language explanations."
  • Open-ended response decoding delved into the similarities and differences in heuristic and appropriation behaviors.

• Limitations and Future Directions:

  • Limitations:
    • The sample primarily consisted of students (AI group) and MTurk users (end-user group), limiting the generalizability of the data.
    • The explanation styles covered were limited in scope, excluding more complex multimodal combinations.
  • Future Directions:
    • Expand the study to different types of AI models (e.g., classification tasks, medical predictions) and applications across various industries.
    • Further explore the structured stratification of user groups and its specific impact on the perception of explanations (e.g., work background, years of expertise).

Conclusion

This study highlights the critical issue of user background differences in the design of explainable AI (XAI) through research on the perceptions of AI-background and end-user groups. The findings not only contribute to design improvements but also promote reflective AI education reform and more responsible AI development practices.