Learning Behaviors Mediate the Effect of AI-powered Support for Metacognitive Calibration on Learning Outcomes

Research Background and Issues

• Problem or Challenge: Students often face difficulties in accurately assessing their own performance, especially without prior training. This inaccuracy can lead to inefficient learning and negatively impact academic outcomes. In online learning environments, this challenge may be exacerbated due to the lack of real-time feedback.
• Importance of the Problem: Accurate metacognitive calibration can help students allocate their study time and resources more effectively, thereby improving academic performance. Addressing this issue will promote self-directed learning and the development of online education.
• Research Motivation: Although there have been some intervention studies related to metacognitive calibration, these interventions are typically based on post-hoc feedback, potentially missing opportunities to correct calibration errors during the learning process. Therefore, the authors propose investigating whether real-time, early interventions can improve learning outcomes while examining the potential impact of interventions on behavioral changes.

Solution

• Method or Solution: The authors developed a real-time AI-based training tool that provides metacognitive calibration support by predicting students' performance scores at the end of their learning sessions. The intervention aims to help students adjust their learning strategies during the study process, improving calibration ability and learning outcomes.
• Innovations:
  • The tool provides real-time feedback rather than traditional post-hoc feedback.
  • A random forest model and attention mechanism are used to predict students' subsequent performance, dynamically analyzing their interaction behaviors and knowledge levels.
  • Feedback is used to guide students in adjusting their learning behaviors, such as focusing on challenging materials.
• Implementation Steps:
  1. Develop an online learning platform offering self-study content in statistics.
  2. Use an AI model to predict students' performance scores in real time, based on interaction data and test results.
  3. Provide feedback to students in the experimental group during the learning process (at 15, 30, and 45 minutes), including predicted scores and reflective prompts.
  4. Collect data on learning behaviors and metacognitive calibration to evaluate the intervention's effectiveness.

Research Outcomes

• Specific Results:
  1. The experimental group’s learning outcomes improved by an average of 8.9% (t=-2.384, p=0.019), significantly outperforming the control group.
  2. Overconfident students in the experimental group showed a 4.1% improvement in metacognitive calibration (t=2.001, p=0.049).
• Advantages:
  • Early intervention strategies proved effective in adjusting students' learning behaviors and improving learning outcomes.
  • The experimental group demonstrated more "knowledge-seeking" behaviors, which were confirmed as critical for improving learning outcomes.
• Experimental or Evaluation Results:
  • Multilevel analysis revealed that "knowledge-seeking" significantly enhanced students' learning gains, while "assessment-seeking" behaviors had relatively weak or negative effects on learning outcomes.
  • The intervention showed no significant differences in effectiveness across gender and racial groups, suggesting broad applicability of the strategy.
• Limitations and Future Directions:
  • The small sample size may limit further analysis of differences across demographic groups.
  • The current intervention tool primarily focuses on metacognitive calibration; incorporating other calibration dimensions, such as confidence and peer comparison, may pose new design requirements.
  • Future research is recommended to explore further optimization of "knowledge-seeking" behaviors through enhanced tools and to study user experience feedback for improving tool design.

By supporting metacognitive calibration through early real-time AI interventions, this study not only improved students' academic performance but also uncovered significant behavioral patterns. The findings provide practical insights for designing educational tools in online learning environments and point to directions for optimizing personalized interventions.