Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on their Long-Term Collaboration with an AI Planning System

Research Background and Issues

• What problems or challenges did the authors identify?
  The study found that despite the strong performance of artificial intelligence (AI) systems, users may not fully trust these systems. Particularly, current research predominantly focuses on controlled laboratory environments, lacking in-depth exploration of trust development in real-world, long-term human-AI collaboration. In practical scenarios, human-AI cooperation may face issues such as insufficient trust or miscalibrated trust (overtrust or distrust), which negatively impact task completion and the long-term success of human-AI collaboration.

• Why is this issue important?
  AI has significant potential in complex decision-making scenarios, such as medical diagnosis or logistics optimization. However, if users lack trust in AI systems, it may hinder the adoption and practical efficiency of the technology. Furthermore, in long-term collaborations, the development and maintenance of trust are critical for improving the performance and satisfaction of human-AI teams.

• Research Motivation and Related Work
  This paper aims to address two major gaps in existing research: (1) an over-reliance on laboratory-based studies, neglecting ecological validity in real-world scenarios; (2) a lack of in-depth understanding of the dynamic development of trust in long-term human-AI collaboration. By investigating an AI planning system in the logistics domain and its performance in a real-world, long-term application scenario, the study seeks to deepen the understanding of human-AI interaction in practical contexts.

Solutions

• What methods or solutions did the authors propose?
  The authors conducted in-depth interviews with logistics experts who have long-term experience using AI planning systems. They employed qualitative analysis methods to explore the dynamic process of trust development, aiming to understand trust formation, challenges, and influencing factors in long-term collaboration.

• What are the innovative aspects of this solution?

  1. Extending research from laboratory settings to real-world domains, enhancing the ecological validity of the findings.
  2. Focusing not only on technical performance but also on the social and interpersonal factors influencing trust.
  3. Investigating trust dynamics from a macro and long-term perspective, revealing the complexity of trust formation through the perspectives of multiple stakeholders (dispatchers, truck drivers, and managers).

• What are the implementation steps and key techniques used?

  1. Conducting semi-structured, in-depth interviews with logistics company experts, including dispatchers, truck drivers, and managers.
  2. Collecting data and coding it using Reflexive Thematic Analysis to extract themes from the interview data.
  3. Analyzing trust dynamics through cross-role validation across different categories.
  4. Comparing findings with existing research to explore the ecological applicability of the results.

Research Findings

• What specific findings were achieved?

  1. Trust can develop over time even if the AI system does not perform perfectly. Experts gradually accepted and trusted the system through repeated interactions and familiarity.
  2. Inconsistencies in the AI system (e.g., frequent updates) and lack of transparency significantly disrupted trust-building, forcing users to adopt avoidance strategies.
  3. Despite the system being generally trustworthy, dispatchers often bypassed AI recommendations to protect the needs and well-being of truck drivers, highlighting the importance of human factors in AI trust.

• What advantages does it have compared to existing solutions?
  This study complements the limitations of short-term laboratory research by providing a longitudinal perspective and real-world scenario analysis, emphasizing the complexity of trust dynamics between users and AI systems. Additionally, it is the first to integrate interpersonal relationships (e.g., interactions among colleagues) into trust analysis, offering a more comprehensive perspective.

• What were the experimental or evaluation results?
  Dispatchers built trust in the system through learning and adaptation, basing their trust on an understanding of the system. However, frequent AI updates and lack of transparency reversed some of the trust development progress. Furthermore, ensuring humanitarian considerations (e.g., driver well-being) often took precedence over economic efficiency, indicating that users' use of AI is not solely based on technical or economic metrics.

• Limitations and Future Directions

  1. Limitations: The sample size was limited, and the study focused on a specific scenario (logistics), which may not fully generalize to other domains. The reliance on retrospective interviews may have overlooked some dynamic details.
  2. Future Directions:
     • Expand field studies to different industries to validate the generalizability of the findings.
     • Develop more transparent and explainable AI systems to reduce the need for human intervention.
     • Explore how to integrate personalized factors (e.g., employees' individual needs) into AI systems to balance efficiency and humanitarian considerations.

Conclusion

This paper, through a case study in the logistics domain, presents a perspective on the dynamic evolution of trust in long-term AI collaboration, emphasizing the importance of human and social factors. The research provides insights for future AI design and ecological studies, promoting the development of AI systems that better align with practical needs and specific value systems.