Trust in AI-assisted Decision Making: Perspectives from Those Behind the System and Those for Whom the Decision is Made

Title of the Paper

Trust in AI-assisted Decision Making: Perspectives from Those Behind the System and Those for Whom the Decision is Made

Paper Information

• Subject Area: Human-machine trust in AI-assisted decision-making
• Keywords: Human-machine trust, artificial intelligence, decision support, qualitative research, AI developers, decision subjects

Research Background and Problem

• Issues and Challenges: This study focuses on the issue of trust between humans and artificial intelligence in decision-making scenarios. Current research predominantly centers on the perspective of direct users (interactors), with insufficient consideration of other stakeholders (e.g., AI developers, decision subjects).
• Significance: Human-machine trust is a critical factor for the adoption of AI technologies and the optimization of decision-making processes. In high-risk domains (e.g., healthcare, finance, recruitment) where AI decision systems are applied, understanding the mechanisms of trust formation is essential.
• Research Motivation: Exploring the understanding and factors of human-machine trust from non-direct user groups (developers and those affected by decisions) can improve AI design and align trust-related differences among multiple stakeholders.

Solution

• Research Methodology: Semi-structured interviews were conducted with AI developers (7 participants) and decision subjects (7 participants) across various high-risk decision-making contexts (e.g., healthcare, finance, management).
• Innovations:
  1. Investigated the multifaceted definitions of human-machine trust, analyzing the trust dynamics and diverse influencing factors among different stakeholders.
  2. Proposed "task complexity" as a necessary precondition for human-machine trust, extending existing theoretical models of trust.
  3. Examined perspectives beyond direct users, such as AI developers and decision subjects, highlighting stakeholder differences regarding transparency, interactivity, and AI literacy.
• Implementation Steps:
  1. Interview Design: Adjusted questions based on the interviewee's role, focusing on trust definitions, influencing factors, and evaluation methods.
  2. Data Analysis: Utilized inductive thematic analysis to identify core themes of trust: preconditions for trust, the impact of mutual trust among stakeholders on human-machine trust, and divergences in key trust factors.
  3. Key Techniques: Interviews, semantic data coding, and thematic induction.

Research Findings

• Specific Findings:

  1. Identified three necessary preconditions for human-machine trust: positive expectations, risk perception related to decisions, and task complexity.
  2. Emphasized the importance of interpersonal relationships (e.g., developer-user relationships, trust among users, decision-maker-user trust) in shaping human-machine trust.
  3. Revealed differentiated impacts of transparency, AI literacy, and interactivity: developers prioritize information openness and workflow transparency, while decision subjects focus more on actionable information and their role in the decision loop.
  4. Made an initial distinction between "trust" (human attitudes toward AI), "trustworthiness" (attributes of the AI system), and "trust-related behaviors" (e.g., compliance).

• Advantages Compared to Existing Solutions:

  1. Deepened the understanding of trust relationships among stakeholder groups within the socio-technical ecosystem.
  2. Introduced a new dimension, "task complexity," into existing trust theoretical models, addressing a research gap.
  3. Provided specific design recommendations and calibration points, including interventions emphasizing non-user perspectives and dynamic relationships.

• Experimental or Evaluation Results:

  1. Found that decision subjects (e.g., patients, loan applicants) primarily base their trust in AI on an indirect reflection of trust in users and development teams.
  2. Identified differences in the role of AI explainability (transparency) and interactivity in trust: decision subjects are more concerned with control over predictive model outcomes and their points of influence.
  3. System certification was considered a potential factor for building trust, but the credibility of certifying institutions was deemed critical.

• Limitations and Future Directions:

  1. Individual differences (e.g., user experience, gender, background) were not fully detailed, requiring further research into cross-domain and cross-cultural variations.
  2. This study primarily focused on organizational contexts of AI trust; future research should examine the distribution of trust among multiple independent entities within the "algorithm supply chain."
  3. Suggested conducting in-depth investigations and action research to explore trust breakdown points and calibration methods, particularly in complex socio-technical networks.