Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making

Document Title

Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making

Document Information

• Subject Area: AI Ethical Decision-Making, Human-AI Collaboration
• Keywords: Ethical AI, Trust, Responsibility, Human-AI Collaboration, Algorithmic Aversion, Algorithmic Appreciation

Research Background and Issues

• Issues and Challenges:
  The authors point out that with the rapid development of artificial intelligence (AI) technologies and their application in various decision-making domains, many of these decisions involve high ethical risks. Human perceptions of ethical decision-making are divided, and the performance of AI in such issues, along with the allocation of responsibility, has become a focal point of discussion. Specifically, this study explores trust, responsibility attribution, and reliance when AI collaborates with human experts in ethical decision-making.
• Significance:
  Ethical decision-making determines how advanced decisions, such as drone search and rescue or national defense, are implemented. It also affects societal acceptance and the widespread application of AI technologies. Understanding human perceptions of AI in ethical decision-making is crucial for designing systems for practical applications.
• Motivation and Related Work:
  Previous research has separately investigated trust, reliance, and responsibility attribution, but no studies have examined how they interact in ethical decision-making. Moreover, past studies have primarily focused on topics such as autonomous vehicles, while this research extends to real-world scenarios like drone search and rescue and national defense.

Research Methods and Solutions

• Study Design:
  A 2x2 experimental design (expert type: human vs. AI; expert autonomy: human-in-the-loop (HITL) vs. human-on-the-loop (HOTL)) was used to investigate user perceptions in two ethical scenarios (maximizing the number of rescues vs. minimizing casualties).
• Specific Methods:
  • A complex simulation environment (developed using Unity) emphasized the immersion and realism of the decision-making scenarios.
  • Collaborative design of decision dilemmas, such as drone rescue missions or shooting down drones involved in terrorist attacks, with probabilistic weighting influencing ethical choices.
  • Questions included trust questionnaires (multi-dimensional trust measurement tools), responsibility attribution, and reliance behaviors (proportion of accepting or rejecting recommendations).
• Key Technologies:
  • Virtual simulations and random task assignments were used to enhance the controllability of overall responsibility allocation in the experiment.
  • Open scientific data was provided to ensure transparency.

Research Results

• Specific Findings:
  1. Human experts were attributed higher "moral trust" and responsibility, but AI outperformed in "competence trust," overall trust, and reliance. AI recommendations were more likely to be accepted (especially in later tasks).
  2. When AI made decisions, responsibility was partially shifted to AI developers and vendors, but it remained lower than the responsibility attributed to human experts.
  3. The impact of expert autonomy (HITL vs. HOTL) was minimal; humans were more concerned about whether they ultimately had control over the expert.
• Advantages:
  • Offers new directions for designing ethical AI, emphasizing the complementarity of AI and human expert strengths rather than substitution.
  • Provides guidance on designing systems to establish reasonable levels of trust and reliance, avoiding responsibility gaps.
• Experimental and Evaluation Results:
  • Experiments showed that AI was more trusted for its technical capabilities compared to human experts but was not considered a moral agent. The results were consistent across various task scenarios.
  • In the tasks, the absence of significant errors by AI further reinforced the proposed "algorithmic appreciation" effect.
• Limitations and Future Directions:
  • The experiment relied solely on general participants (non-domain experts). Future studies should include data from domain experts to understand decision-making variations.
  • The scenarios were limited to drone defense or rescue missions; other ethical decision-making domains (e.g., healthcare) need further exploration.
  • The range of autonomy levels was limited; future research should examine the effects of transitioning from HITL to fully autonomous AI decision-making.

Conclusion and Design Implications

• When designing ethical AI, the differences in human emotional tendencies and types of trust (moral trust vs. competence trust) should be considered. In high-ethics-demand tasks, appropriately integrating technological advantages with human moral responsibility may achieve greater societal acceptance than relying solely on AI efficiency.
• Autonomous system design should consider clarity in responsibility attribution within ethical contexts, such as incorporating transparent value judgments (e.g., prioritizing utility maximization).
• Future research should expand to other domains and application scenarios, optimizing task design to enhance the perception and acceptance of AI in practical applications.