AI-Moderated Decision-Making: Capturing and Balancing Anchoring Bias in Sequential Decision Tasks

Document Title

AI-Moderated Decision-Making: Capturing and Balancing Anchoring Bias in Sequential Decision Tasks

Document Information

• Subject Area: Artificial Intelligence, addressing cognitive bias in multi-sequential decision tasks, Human-Computer Interaction research
• Keywords: Anchoring Bias, Artificial Intelligence, Neural Networks, Human-Computer Interaction, Decision-Making Process, Fairness, Support Vector Machine, Machine Learning, Deep Reinforcement Learning, Human Decision Bias

Research Background and Problem Statement

• Identified Problems and Challenges:

  • In multi-sequential decision tasks, the most recent decision made by a decision-maker influences subsequent decisions, a phenomenon known as anchoring bias.
  • Anchoring can lead to unfair decisions, such as individuals with similar characteristics receiving inconsistent treatment due to differences in decision order, ultimately affecting decision accuracy.
  • This bias is present in both high-risk tasks (e.g., university admissions) and low-risk tasks (e.g., product reviews), posing significant societal implications.

• Research Motivation:

  • To mitigate this potential bias, this study aims to develop technical methods to identify, adjust anchoring states, and improve decision fairness.
  • Anchoring phenomena pose significant challenges to academic fields, social policies, and the fairness capabilities of AI-assisted judgment systems.

Solution

Methodology

1. Data Collection and Analysis:

   • Collect university admissions datasets (5,814 decision sequences, 117 evaluators) and product review datasets, completed by experiment participants on Amazon Mechanical Turk.
   • Use Support Vector Machine (SVM) as a "bias-free" prediction tool to detect bias points and perform subsequent adjustments.

2. Two Strategies:

   • Retrospective Probabilistic Adaptation (PA):
     • Fit evaluators' anchoring states using an exponential decay function and adjust completed decisions accordingly.
     • Recalibrate evaluation accuracy based on SVM predictions, correcting potential erroneous judgments.
   • Prospective Sequential Sampling Order Learning:
     • Simulate evaluators' anchoring states using Long Short-Term Memory (LSTM) neural networks.
     • Employ deep reinforcement learning (RL, such as Deep-Q networks and Actor-Critic methods) to dynamically determine the presentation order of instances in the sequence, thereby mitigating anchoring effects.

Implementation Steps

• Data Preprocessing:
  • Use TF-IDF techniques to vectorize data features.
  • Train SVM to define decision boundaries, serving as a benchmark for prediction accuracy.
• Retrospective Algorithm:
  • Adjust current decisions based on the number of past decisions and state probabilities, obtaining "bias markers" from the probabilistic model.
• Real-Time Sequential Learning:
  • Capture dynamic anchoring states using LSTM and input them into a deep reinforcement learning framework to optimize presentation order.
  • Employ a reward mechanism to measure the effectiveness of actions in reducing anchoring effects.

Research Outcomes

Specific Results

• Experimental Performance and Accuracy Improvement:

  1. Retrospective Method (PA):
     • University admissions task: 2% increase in decision accuracy, 0.01 reduction in bias (Pearson correlation coefficient).
     • Product review task: 5% increase in decision accuracy, 0.08 reduction in bias.
  2. Real-Time Sequential Learning: 7% increase in accuracy, 0.07 reduction in bias.

• Advantages Over Existing Solutions:

  • PA and real-time sequential learning algorithms do not require additional prompts or modifications to task settings, reducing the risk of introducing new biases.
  • More effective than heuristic methods (e.g., alternating presentation of strong and weak instances), which only improve accuracy by 3%.

Limitations and Future Research Directions

• Limitations:

  • Ethical concerns in high-risk tasks (e.g., university admissions) should prevent real-world experimentation.
  • LSTM's generation of anchoring states lacks interpretability, presenting a "black-box problem" that makes the reasoning mechanism difficult to understand.

• Future Directions:

  • Improve model interpretability: Analyze LSTM and RL behaviors using frameworks like LIME.
  • Explore supplementary models that collaborate with evaluators: Display anchoring states to enhance evaluators' decision-making capabilities.
  • Recommend re-reviewing instances marked as "suspected bias" in high-risk tasks and establishing consensus-based decision-making.

Conclusion

This study proposes a strategy integrating AI and human efforts to effectively mitigate anchoring bias in multi-sequential decision tasks while achieving efficient and fair decision-making processes. The research holds significant implications for human-computer collaboration and enhancing fairness, applicable across diverse social and technological domains.