Watch Out For Updates: Understanding the Effects of Model Explanation Updates in AI-Assisted Decision Making

Document Title

Watch Out for Updates: Understanding the Effects of Model Explanation Updates in AI-Assisted Decision Making

Document Information

• Topic Area: Human-Computer Interaction (HCI), Explainable Artificial Intelligence (XAI), Impact of AI Model Updates on User Decision-Making
• Keywords: Explainable Artificial Intelligence (XAI), Model Updates, User Trust, Human-AI Decision Making, Experimental Study, Domain Knowledge, Visual Explanations, Subjective Trust, User Satisfaction, AI-Assisted Decision Making

Research Background and Issues

• Identified Problems or Challenges:

  • The explainability of AI recommendation systems is increasingly critical in helping users understand model decisions.
  • AI models often require updates, which may lead to significant changes in model explanations.
  • How users perceive changes in model explanations and how these changes impact their trust and satisfaction with the model remain insufficiently understood.

• Significance:

  • Changes in model explanations may influence users' trust and reliance on the model, potentially affecting the quality of human-AI collaborative decision-making.
  • Understanding this issue can help design more reliable AI explanation methods, ensuring smoother user experiences after model updates.

• Research Motivation and Related Work:

  • Numerous studies have explored user interactions with static AI models, but research on user responses to dynamic AI model updates is limited.
  • While some studies have examined the impact of changes in model predictions and performance on user trust, there is a lack of in-depth research on the effects of changes in model explanations themselves.
  • Consistency with users' prior domain knowledge may be a key factor influencing user reactions.

Solution

• Method or Solution:

  • Propose a framework for studying changes in AI model explanations using experimental methods.
  • Conduct randomized controlled trials (RCTs) to investigate the impact of varying levels of similarity in model explanation updates on users' subjective trust, satisfaction, and objective adoption tendencies.

• Innovative Contributions:

  • Explore the role of consistency between updated model explanations and users' domain knowledge, a relatively underexplored area in prior research.
  • Design two experimental scenarios to compare the effects of domain knowledge levels: one task with low domain knowledge (mushroom toxicity prediction) and another with high domain knowledge (loan default prediction).
  • Systematically link perceived "similarity" to the consistency and deviation of model explanations before and after updates.

• Implementation Steps and Techniques:

  1. Design experiments: Two-stage decision-making tasks simulating AI model explanation updates.
  2. Set experimental variables: Compare the similarity of model explanations before and after updates (high similarity, medium similarity, low similarity).
  3. Recruit participants and collect data on decision-making behaviors and subjective feedback.
  4. Use statistical regression and path analysis modeling to identify mechanisms by which changes in model explanations impact user trust and satisfaction.

Research Findings

• Specific Findings:

  1. Users can perceive changes in model explanations after updates.
  2. For tasks with low domain knowledge, changes in explanations do not significantly impact users' subjective or objective trust.
  3. For tasks with high domain knowledge, changes in explanations significantly affect users' subjective trust and satisfaction, with the direction of impact depending on the consistency of updated explanations with domain knowledge.

• Advantages:

  • Provides extensive experimental data and path analysis models, revealing the complex mechanisms behind user reactions to AI explanation updates.
  • Highlights the importance of domain knowledge as a moderating factor, offering insights for designing more user-friendly explanation methods.

• Experimental or Evaluation Results:

  • In the mushroom toxicity prediction experiment, users in the low explanation similarity group were more likely to perceive changes in explanations, but this did not significantly affect subjective trust or satisfaction.
  • In the loan default prediction experiment, when updated model explanations were more consistent (or inconsistent) with users' domain knowledge, users' subjective trust and satisfaction significantly increased (or decreased).
  • Path analysis revealed the following mechanism: changes in explanations influence user trust and satisfaction by altering perceptions of model accuracy and consistency.

• Limitations and Future Directions:

  • Limitations:
    • Simplified experimental scenarios and tasks may not fully reflect the complexity of real-world situations.
    • Model updates only reflect changes in explanations, without incorporating changes in prediction results.
    • Quantitative metrics used (e.g., trust measurement) may not fully capture users' true perceptions of AI.
  • Future Directions:
    • Further study user reactions to simultaneous changes in model predictions and explanations.
    • Explore more complex real-world applications (e.g., medical diagnosis or autonomous driving).
    • Incorporate expert feedback or data transparency into model update designs, combining these with explanation updates.