Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making

Research Background and Issues

• Identified Challenges

  1. Traditional AI-assisted decision-making systems provide fixed recommendations, limiting user interaction with AI to merely accepting or rejecting suggestions, especially when disagreements arise.
  2. Users' reliance on AI recommendations may lead to over-reliance or complete disregard (under-reliance), ultimately affecting decision quality.
  3. Current systems fail to adequately address and resolve partial consistency and conflicts in reasoning between humans and AI during decision-making.

• Significance

  • AI decision support has been widely applied in critical domains such as medical diagnosis, investment decision-making, and criminal justice, but deficiencies in accuracy and transparency can result in severe consequences.
  • Improved human-AI collaboration can enhance overall team decision-making performance and help address ethical and fairness issues in real-world applications.

• Research Motivation and Related Work

  • This study draws inspiration from human deliberation theories and the "Weight of Evidence (WoE)" framework, aiming to introduce deeper interactions between AI and users to improve critical thinking and trust.
  • Current explainable AI (XAI) primarily simplifies AI decision processes by presenting model explanations, but research on enhancing AI-assisted decision-making through dynamic discussion and conflict resolution remains limited.

Solution

• Proposed Approach This paper introduces the "Human-AI Deliberation" framework and develops its core component, "Deliberative AI," which leverages large language models (LLMs) for dynamic interaction to address inconsistencies between human and AI perspectives.

• Innovations

  1. Support for Dimension-Level Opinion Expression and Updates: Human and AI can exchange and adjust opinions on specific dimensions (e.g., preference weights) rather than merely accepting or rejecting AI's overall suggestions.
  2. Encouraging Dynamic and Structured Discussions: A dialogue interface is designed to facilitate human-AI collaboration, supporting mutual questioning, evidence verification, and opinion updates.
  3. Integration of LLMs and Domain-Specific Models (DS Models): LLMs enable natural language interaction, while domain-specific models ensure reliable and accurate information provision.

• Implementation Steps and Techniques

  1. Expression and Alignment of Ideas: Domain models use SHAP explanations to generate dimension-specific feature weights (Weight of Evidence, WoE). Users are also required to clearly express their opinions on each dimension.
  2. Human-AI Discussion: The system uses LLMs to identify human intentions and dimension-specific queries, guiding the dialogue and automatically invoking domain models to extract specific evidence (e.g., data distributions, global correlations).
  3. Opinion Updates: Formulas quantify the strength of users' arguments and AI uncertainty, dynamically adjusting AI perspectives for more proactive responses.
  4. Interface Design: An intuitive interactive interface is developed to support dimension-level detailed discussions, overall decision updates, and summary viewing.

Research Outcomes

• Specific Results

  1. Through application in the research task "graduate admission evaluation," Human-AI Deliberation significantly improved collaborative decision accuracy.
  2. Compared to traditional Explainable AI (XAI), this approach effectively reduced over-reliance on erroneous AI recommendations while minimizing ineffective behaviors caused by unreliable AI explanations.

• Experimental Performance and Advantages

  • Experiments show that compared to traditional XAI, "Deliberative AI" improved participants' decision accuracy (accuracy increased from 52.4% to 59.8%).
  • Reduced over-reliance: Over-reliance rates dropped from 65% in traditional systems to 47% with this method.
  • Task complexity did not significantly increase, and users positively acknowledged the deep interaction and collaboration process with AI.

• Limitations and Future Directions

  1. Context and Task Applicability: The research task (graduate admission case) is based on synthetic datasets, which may differ from real-world high-stakes decision scenarios.
  2. User Experience and Satisfaction: Some user feedback indicated that the discussion process was lengthy, increasing decision-making burdens, necessitating optimization of input modules and dialogue generation speed.
  3. Applicability to Multi-Modal Data Tasks: The current study primarily focuses on tabular data; future research should explore extensions to tasks involving image or text analysis, such as through vision-language models.
  4. Ethics and Transparency: Users expressed concerns about the transparency of AI's opinion update mechanisms, requiring clearer presentation of the basis for AI adjustments in future work.

Conclusion

The "Human-AI Deliberation" framework proposed in this paper offers a novel perspective for AI-assisted decision-making by introducing dynamic, detailed user participation and AI interaction into critical domains. Experiments demonstrate its potential to significantly improve decision quality and provide initial insights into user perceptions of Deliberative AI. Future research should further expand task diversity and optimize human-AI interaction efficiency to adapt to broader real-world application scenarios.