Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks

Title of the Paper

Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks

Paper Information

• Domain: Human-computer interaction in AI-assisted decision-making
• Keywords: Machine learning, appropriate reliance, human-AI interaction, limited feedback, high-confidence agreement, human decision-making

Research Background and Problem Statement

• What problems or challenges did the authors identify?

  1. How people determine the degree of reliance on machine learning models in the absence of explicit performance feedback is an underexplored issue.
  2. The lack of performance information may lead to inappropriate reliance on models, such as over-reliance or under-reliance.

• Why is this problem important?

  1. Machine learning models are widely used to support decision-making tasks such as music recommendation, financial risk assessment, and medical diagnosis. Whether people can use these models appropriately in specific scenarios directly impacts decision quality.
  2. Inappropriate reliance on machine learning models can result in decision failures or limited performance.

• Research Motivation and Related Work

  • Social psychology research indicates that in the absence of objective feedback, people tend to prioritize their own judgment and undervalue differing opinions.
  • Machine learning research has largely focused on the impact of external feedback, such as interpretability and confidence, on human behavior, but lacks systematic studies on reliance behavior under limited performance feedback.

Proposed Solution

• What methods or solutions did the authors propose?

  • Designed three sets of randomized experiments to study the heuristic strategies people adopt when machine learning model performance information is limited.
  • Explored whether people's agreement levels with models on high-confidence tasks affect their reliance on the models.

• What is innovative about this solution?

  1. Investigated a new research perspective: how people rely on machine learning models under limited performance feedback.
  2. Clarified the influence of high-confidence agreement on reliance behavior in different contexts (no performance feedback, overall performance feedback available).
  3. Conducted an in-depth study on the impact of human confidence levels in agreement and disagreement tasks.

• What are the implementation steps and key techniques used?

  1. Experiment 1: Studied the impact of agreement levels with models on high-confidence tasks on reliance when no performance feedback is available.
  2. Experiment 2: Investigated whether agreement levels still influence reliance after partial performance feedback (overall model accuracy) is provided.
  3. Experiment 3: Explored how people adjust their reliance on models in agreement and disagreement scenarios under different levels of self-confidence.

Research Findings

• What specific findings were obtained?

  1. Experiment 1: In the absence of any performance feedback, people relied more on the level of agreement with models on high-confidence tasks to assess model reliability.
  2. Experiment 2: Once performance feedback became available, people primarily relied on observed model accuracy, and the influence of agreement levels diminished.
  3. Experiment 3: In the absence of performance feedback, people's reliance on models depended on the interaction effect between their confidence levels and agreement/disagreement tasks.

• What advantages does this solution have compared to existing ones?

  • Provides more comprehensive experimental evidence on how humans adjust their behavior in human-computer interaction under limited feedback conditions.
  • Highlights how subjective evaluations in high-confidence tasks can distort objective assessments of model performance.

• What are the experimental or evaluation results?

  1. Experiment 1: The higher the proportion of high-confidence agreement, the stronger the reliance on the model, regardless of whether the model's predictions were correct.
  2. Experiment 2: Performance feedback weakened the impact of high-confidence agreement on reliance, with actual model performance becoming the primary consideration.
  3. Experiment 3: When confidence levels were high, disagreement with the model significantly reduced reliance, whereas disagreement had a smaller impact when confidence levels were low.

• Limitations and Future Directions

  1. Limitations:
     • The study focused only on general populations and specific tasks (e.g., speed dating predictions), which may limit the generalizability of the conclusions.
     • The impact of high-confidence agreement versus prediction ranking consistency was not distinguished.
  2. Future Work Directions:
     • Validate the applicability of the findings in professional domains and high-risk tasks.
     • Explore optimization of human-machine collaboration in more complex interaction dimensions.