AI, Help Me Think—but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Support

Research Background and Issues

• Identified Problems and Challenges: The authors point out the following challenges in traditional recommendation-based AI decision support systems (RecommendAI):
  • Users may inappropriately rely on AI, leading to "over-dependence."
  • AI recommendations are often difficult to integrate into users' decision-making processes.
  • For complex and "open-ended" problems (e.g., financial investment, medical diagnosis), the single predictive output of traditional AI recommendation models often fails to meet task requirements.
  • Users dealing with complex problems need to process subjective and unstructured information, which existing AI systems generally cannot capture.
• Significance of the Problem: With the widespread application of AI in fields such as healthcare, finance, and logistics, improving the interaction between users and AI and enhancing the complementarity between AI and human decision-making has become crucial. Additionally, AI tools that support deep reflection and decision improvement for complex tasks have not been adequately studied.
• Research Motivation and Related Work:
  • Traditional research has primarily focused on enhancing the interpretability and trustworthiness of AI recommendations, but these methods fail to address the inherent limitations of recommendation models.
  • Recent theories suggest that AI should support complex decision-making by enhancing users' reasoning processes rather than simply providing recommendations.
  • The natural language capabilities of large language models (LLMs) offer new opportunities to explore how AI can effectively integrate into users' reasoning processes.

Solution

• Proposed Solution:
  • The authors designed two AI decision support modes to explore new interaction models:
    1. RecommendAI: A traditional recommendation model that directly suggests specific solutions/ETF investment portfolios to users.
    2. ExtendAI: Requires users to first articulate their decision logic, after which AI provides expansive feedback based on the user's reasoning (without offering specific recommendations).
• Innovative Features:
  • The ExtendAI mode breaks the limitations of traditional recommendation models by embedding users' reasoning processes and enabling natural language interactions.
  • The study uses financial investment (ETF portfolios) as a test case for complex, multi-factor decision tasks, simulating real-world scenarios.
  • AI is positioned as a "collaborator" rather than an "authoritative advisor."
• Implementation Steps:
  1. Design a simulated trading platform to test the two AI decision support modes.
  2. Collect data on 31 ETFs and generate synthetic data to simulate future ETF prices and market contexts.
  3. Implement ExtendAI and RecommendAI using a large language model (GPT-4), providing tailored feedback or recommendations based on users' investment logic.
  4. Conduct experiments with 21 participants experienced in ETF investment, randomly assigning them to use the two AI modes in different sequences for comparison.

Research Findings

• Specific Findings:
  1. Both modes improved the quality of investors' portfolios in different aspects, particularly in geographic and industry diversification:
     • ExtendAI showed a slight advantage in enhancing global portfolio balance.
     • RecommendAI provided "novel inspiration" while requiring less cognitive effort.
  2. ExtendAI better integrated users' reasoning logic, encouraged deep reflection, and improved user satisfaction and decision quality in certain scenarios.
  3. RecommendAI increased decision efficiency but might lead to reduced user engagement and, in some cases, over-dependence.
• Advantages Over Existing Solutions:
  • ExtendAI is more closely aligned with users' reasoning, enabling them to "think for themselves."
  • The study highlights the issue of recommendation-based AI potentially causing users to disengage from decision-making in complex tasks.
  • Both modes go beyond single predictions, extending support to users' overall decision-making processes.
• Experimental or Evaluation Results:
  • Analysis of user behavior revealed:
    • Participants using ExtendAI felt the mode reduced interference with their decisions but found it less operationally efficient compared to RecommendAI.
    • ExtendAI users modified only 23% of their initial plans on average per decision, while RecommendAI's direct suggestions had an adoption rate of approximately 45%.
    • In terms of satisfaction, ExtendAI scored higher than RecommendAI but required greater cognitive effort.
• Limitations and Future Directions:
  • ExtendAI's feedback was perceived as overly "subtle," making its contributions easy for users to underestimate.
  • The decision format was relatively open-ended, without a clear "correct decision" standard, limiting evaluation precision.
  • Users were required to manually articulate their reasoning, which, while encouraging thought, added extra cognitive workload.
  • Future research could explore visualization tools and more intuitive interaction methods to simplify users' reasoning input processes.
  • The study could be extended to other complex, open-ended tasks such as healthcare and social welfare.

The analysis reveals design opportunities to shift from recommendation-driven AI to decision support centered on enhancing user reasoning, while also addressing the challenge of balancing cognitive load with user engagement.