People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AI

Research Background and Issues

• What problems or challenges did the authors identify?

  • Current Explainable Artificial Intelligence (XAI) designs often fail to align with human cognitive patterns. Particularly in applications involving complex decision-making (e.g., autonomous vehicles), people tend to interpret system behavior in a teleological manner, while XAI systems typically generate mechanistic explanations.
  • Research in XAI has predominantly focused on algorithmic aspects, lacking support from cognitive science studies on how explanations are generated and understood by humans.
  • Existing XAI evaluation metrics, such as "fidelity," overly emphasize the consistency between algorithms and explanations, neglecting how humans assess and prefer these explanations.

• Why is this issue important?

  • Autonomous driving is a safety-critical domain where decision-making needs to be fully understood to build trust and facilitate human-machine collaboration.
  • Exploring how people generate and accept explanations can help design more intuitive, human-centered XAI systems.
  • If explanations fail to meet human psychological needs, users may misunderstand or distrust the system.

• Research Motivation and Related Work

  • Inspired by cognitive science theories on causality and explanation mechanisms, this study proposes a framework for analyzing explanations, encompassing teleological, mechanistic, counterfactual, and descriptive modes.
  • The authors aim to explore through experimental studies how these different explanation modes affect human understanding and trust, addressing the current gap in XAI research on human cognitive mechanisms of explanation.

Solutions

• What methods or solutions did the authors propose?

  • Developed a framework for human explanation generation and reception, covering multiple explanation modes such as teleological, mechanistic, and counterfactual.
  • Designed two experiments: the first investigates how participants generate explanations when observing autonomous driving scenarios; the second evaluates the quality and characteristics of these explanations.
  • Created a dataset called "Human Explanations for Autonomous Driving Decisions (HEADD)" for experimental analysis and future research.

• What are the innovative aspects of this solution?

  • Proposed a cognitive science-based "explanation mode framework" for analyzing and generating explanations, moving beyond traditional algorithm-centric approaches.
  • Conducted two experiments to investigate "how humans generate explanations" and "how humans evaluate explanations," achieving a closed-loop study of explanation generation and reception.
  • Introduced a causal explanation design concept combining teleological reasoning and cognitive science to improve existing XAI systems, making them more intuitive for humans.

• What are the implementation steps? What key technologies were used?

  • Experiment 1: Generating Explanations
    1. Presented videos of 14 driving scenarios and asked participants to generate natural language explanations for autonomous vehicle behavior using four explanation modes (teleological, mechanistic, counterfactual, descriptive).
    2. Collected and preprocessed these explanations for content analysis.
  • Experiment 2: Evaluating Explanations
    1. Presented a subset of explanations from Experiment 1 to another group of participants, asking them to evaluate the explanations based on dimensions such as satisfaction, causality, and trustworthiness.
    2. Analyzed evaluation results using mixed-effects models.
  • Data Processing and Analysis Techniques: Applied natural language processing techniques to analyze the syntactic and semantic complexity of generated explanations; used R and statistical models to analyze experimental data.

Research Outcomes

• What specific outcomes were achieved?

  • People prefer and are more likely to accept teleological explanations, which typically include the goals or intentions behind behaviors. This preference is consistent when explaining the behavior of both autonomous vehicles and human drivers.
  • Counterfactual explanations are more complex but have the lowest satisfaction and trust ratings, suggesting they may not be the optimal explanation mode for complex systems.
  • The study found that "perceived teleology" (whether goals or intentions are involved) is the most significant predictor of "explanation satisfaction" and "trustworthiness."

• What advantages does this research have compared to existing solutions?

  • This study provides a new perspective based on cognitive science, surpassing existing XAI approaches that rely on algorithmic implementation by focusing more on human psychology and behavior patterns.
  • Offers an open dataset (HEADD) for future research to explore additional dimensions of human explanation generation and evaluation.
  • Proposes different explanation modes as critical dimensions for designing and evaluating XAI systems, helping improve explanation performance in complex environments.

• What were the experimental or evaluation results?

  • Satisfaction with teleological and mechanistic explanations was significantly higher than with counterfactual explanations (p < 0.01).
  • Sentence length and linguistic complexity positively influenced explanation satisfaction, with a stronger effect observed for teleological explanations.
  • Participants' preference for teleological explanations did not differ based on whether they knew the vehicle was autonomous or human-driven (p = 0.44).

• Limitations and Future Directions

  • Limitations:
    1. The experiments were limited to the autonomous driving domain, and future studies are needed to verify the generalizability of the results to other domains.
    2. The data primarily came from participants in the United States, which may introduce cultural bias.
    3. Teleological and mechanistic explanations were sometimes conflated in the experiments, making it difficult to strictly separate them.
  • Suggestions for Future Directions:
    1. Extend the research to other complex domains, such as medical diagnosis or financial decision-making.
    2. Explore how preferences for XAI explanation modes vary across different cultural contexts.
    3. Investigate how to balance different explanation modes to enhance users' understanding and sense of control over system behavior.
    4. Develop more intuitive and reliable autonomous driving assistance explanation modules by incorporating teleological explanation design.