Impact of Model Interpretability and Outcome Feedback on Trust in AI

Title of the Paper

Impact of Model Interpretability and Outcome Feedback on Trust in AI

Paper Information

• Subject Area: Human-Computer Interaction (HCI), AI Trust, and Collaborative Performance
• Keywords: Human-Machine Systems, Hybrid Intelligence, Machine Learning, AI Trust, Human Experiments, AI-Assisted Decision-Making, Explainable AI, Outcome Feedback

Research Background and Problem

• Identified Problem or Challenge: While AI demonstrates strong performance across various domains, users have not widely adopted it. This resistance to algorithms, termed "algorithm aversion," may stem from two primary factors: a lack of interpretability and a lack of trust in model performance.

• Significance: Developing effective AI systems requires not only high performance but also fostering user trust to encourage adoption and benefit from AI recommendations. Trust can drive the widespread adoption of AI technologies and enhance human-AI collaboration.

• Motivation and Related Work: Previous studies have explored the impact of interpretability on user trust, but the findings are inconsistent. Some studies suggest that interpretability increases trust, while others find its effect limited. Additionally, information related to model performance (e.g., accuracy or error rates) has also garnered attention. This study aims to compare these two factors—interpretability and outcome feedback—and their respective impacts on user trust and collaborative performance.

Proposed Solution

• Proposed Solution: Using a pre-registered experimental design, this study investigates how the interpretability of AI models and outcome feedback independently affect user task performance and trust in AI.

• Innovations:

  • For the first time, this study compares the combined effects of interpretability and outcome feedback on AI trust and collaborative performance, while also exploring their interaction.
  • It employs "Weight of Advice" (WoA) as a behavioral trust metric to disentangle the effects of user trust and model accuracy.

• Implementation Steps:

  1. The experimental design involves a two-phase task: in the first phase, participants complete a prediction task without AI assistance; in the second phase, participants receive AI predictions and can adjust their own predictions accordingly.
  2. Six experimental conditions are set, combining different types of interpretability (no interpretability, global interpretability, local interpretability) with outcome feedback (with feedback or without feedback).
  3. A real dataset (speed dating dataset) is used to train the AI model, and participants are randomly assigned to ensure the stability of experimental results.

Research Findings

• Specific Findings:

  1. Trust: Outcome feedback had a significantly greater impact on behavioral trust (WoA) than interpretability. Neither global interpretability nor local interpretability significantly increased trust.
  2. Collaborative Performance: Outcome feedback improved user task performance, but the improvement was much smaller compared to its impact on trust. Interpretability did not significantly enhance task performance.

• Comparison with Existing Solutions: This study challenges the traditional view that interpretability is key to increasing trust in AI. It finds that outcome feedback may be a more effective method for improving AI credibility and adoption.

• Experimental or Evaluation Results:

  • When outcome feedback was provided, users were more likely to trust AI recommendations but also exhibited tendencies toward overtrust (overshooting) or distrust (contradicting), leading to a "trust-performance paradox."
  • Time trend analysis revealed that when users discovered that AI recommendations reduced their performance, their subsequent trust in AI recommendations significantly declined, further undermining collaboration efficiency.

• Limitations and Future Directions:

  • The study focused on a specific context (speed dating prediction task), and further validation is needed in other scenarios to generalize the effects of outcome feedback and interpretability.
  • The study tested only a limited set of interpretability presentation formats. Future research could explore other forms, such as "contrastive explanations" or "counterfactual explanations."
  • Future studies could combine behavioral trust (WoA) with trust calibration metrics to systematically understand the dynamic relationship between AI trust and collaborative performance.

In summary, this study reveals that outcome feedback is more effective than interpretability in enhancing user trust in AI. It also highlights that increasing trust does not always lead to significant improvements in collaborative performance, providing new insights for designing trustworthy AI systems in the HCI field.