As Confidence Aligns: Understanding the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making

Research Background and Issues

• What problems or challenges did the authors identify?
  As artificial intelligence (AI) becomes increasingly integrated into human decision-making processes, complementary collaboration between AI and users has emerged as a critical goal in human-machine decision-making. However, AI's expression of confidence can directly influence users' confidence levels, leading to issues of confidence calibration. This phenomenon may disrupt the efficiency of optimized human-machine collaboration. Currently, there is a lack of systematic exploration of how AI confidence affects user confidence.

• Why is this issue important?
  Complementary collaboration between AI and users requires human-machine teams to clearly determine when each party is best suited to make the final decision. If AI confidence impacts human confidence, leading to a mismatch between user confidence and actual decision-making ability (i.e., miscalibration), it can gradually undermine the efficiency of human-machine collaboration. This impact can be particularly detrimental in high-risk domains such as healthcare and financial decision-making.

• Research Motivation and Related Work
  This study is inspired by the phenomenon of confidence alignment among humans. Research on confidence convergence among individuals shows that participants in group decision-making often influence each other's confidence levels, leading to uniformity. However, there is insufficient research on the potential and lasting effects of AI confidence on user confidence in human-machine decision-making environments, especially under different collaboration modes and scenarios with or without real-time feedback.

Solution

• What methods or solutions did the authors propose?
  The authors designed and conducted a three-phase online randomized behavioral experiment to quantify and explore the alignment effect between user confidence and AI confidence, as well as its subsequent impact, in human-machine collaborative decision-making tasks (income prediction). The experiment further examined the role of real-time feedback and three human-machine collaboration modes (AI as an advisor, AI as a peer collaborator, AI as a decision-maker supervised by humans).

• What are the innovative aspects of this solution?

  1. Proposed an experimental framework to quantify the phenomenon of confidence alignment between AI and users.
  2. Introduced the definition of participant confidence calibration, which evaluates the consistency between participant confidence and actual decision accuracy.
  3. Considered various experimental variables (human-machine collaboration modes and real-time feedback) to provide broader applicability of the results.

• What are the implementation steps and key technologies used?

  1. Three-phase experimental design:
     • Phase 1: Baseline confidence measurement, where participants independently complete decision-making tasks.
     • Phase 2: Human-machine collaboration phase, recording AI confidence, participants' initial decision confidence, and final joint decision confidence.
     • Phase 3: Observation phase, analyzing the lasting changes in participants' confidence during subsequent independent decision-making.
  2. Variable settings: Two key variables—real-time feedback provision and different human-machine collaboration modes.
  3. Data analysis using linear regression and repeated measures ANOVA to quantify the alignment between AI confidence and user confidence, and calculate participant confidence error (ECE).

Research Outcomes

• What specific outcomes were achieved?

  1. Existence of confidence alignment phenomenon:
     • Users' confidence levels tend to align with AI-expressed confidence during collaboration.
     • This alignment phenomenon persists to some extent after the collaboration ends.
  2. Regulatory role of real-time feedback:
     • Providing real-time feedback reduces the degree of alignment between user confidence and AI confidence. Feedback may help users better calibrate their confidence with actual accuracy.
  3. Negative impact on confidence calibration:
     • When user confidence aligns with AI confidence, some users' confidence deviates from their actual decision accuracy, leading to poorer calibration.
  4. Impact on human-machine collaboration efficiency:
     • Poor confidence calibration results in irrational reliance (e.g., over-reliance or under-reliance on AI), reducing the accuracy of joint decision-making.
  5. Confidence alignment in joint human-machine decisions:
     • When users' final decisions align with AI recommendations, the degree of confidence alignment in joint decisions is higher.

• What advantages does it have compared to existing solutions?
  Unlike approaches that focus solely on optimizing decision-making through AI confidence, this study provides an in-depth exploration of the dynamic changes in human confidence, specifically revealing the direct and long-term impacts of AI confidence on user behavior. These insights contribute to improving human-machine collaboration design, particularly in understanding the psychological mechanisms influencing changes in user confidence.

• What were the experimental or evaluation results?
  Experimental results showed:

  • In environments with real-time feedback, the absolute confidence gap (average difference between AI and user confidence) was significantly reduced.
  • When AI acted as an "advisor" or "peer collaborator," confidence calibration issues were particularly prominent, with a noticeable decline in accuracy.

• Limitations and Future Directions:

  1. Task specificity: The experimental task focused on income prediction, and its applicability in complex professional domains (e.g., medical diagnosis, investment analysis) remains to be verified.
  2. Fixed and inflexible AI performance: The experiment used AI with fixed accuracy and confidence calibration, without exploring the potential impact of overconfidence or underconfidence in AI on the alignment phenomenon.
  3. Homogeneity of participant backgrounds: The sample group was biased toward high-confidence, high-accuracy individuals. Future research should expand to include users with lower confidence or accuracy levels.

Conclusion

This study is the first to experimentally validate the phenomenon of confidence alignment between human confidence and AI confidence, revealing its potential negative consequences, such as poorer confidence calibration and reduced efficiency in human-machine collaboration. It also provides design strategies, such as guiding user confidence calibration through real-time feedback to mitigate the negative effects of the alignment phenomenon. This research lays an important theoretical foundation for the design and development of future human-machine collaboration systems and offers insights into the complex impacts of AI on human cognition and behavior.