Students' Verbalized Metacognition during Computerized Learning

Title of the Paper

Students’ Verbalized Metacognition During Computerized Learning

Paper Information

• Research Area: Self-regulated learning, metacognition, and human-computer interaction
• Keywords: Metacognition, self-regulated learning, confusion, emotions, natural language processing, learning assessment, educational technology, human-computer interaction

Research Background and Problem

• Problem or Challenge: Students often need to self-regulate their learning processes in computerized learning environments, which requires metacognitive abilities. However, metacognition is difficult to measure accurately using traditional methods such as questionnaires. Existing research has yet to fully explore the relationship between students' metacognition, learning outcomes, and behavioral patterns.
• Importance: Metacognitive ability is critical for effective learning, especially in self-regulated learning environments. Understanding how metacognition impacts learning outcomes is essential for designing more supportive educational technologies.
• Research Motivation and Related Work: The authors reviewed existing methods and proposed using natural language processing (NLP) to analyze student interview transcripts to investigate the relationships between metacognition, learning, confusion, and metacognitive problem-solving strategies. This approach aims to overcome the limitations of traditional measurement methods and provide data-driven support for educational technology design.

Solution

• Method or Solution:
  • Apply improved NLP tools to analyze interview transcripts and extract students' "verbalized metacognition," including metacognitive knowledge, experiences, and strategies.
  • Combine interview data with behavioral logs from software usage through coherence analysis to study metacognitive strategies.
• Innovations:
  • Introduced a fully automated NLP-based solution for measuring metacognition.
  • Explored the potential role of metacognition across multiple dimensions (learning, confusion, behavior) to validate existing metacognitive strategies.
• Implementation Steps and Key Techniques:
  1. Collect interview data in middle school classrooms using unstructured interviews to capture students' metacognitive statements.
  2. Use a modified NLP tool to extract verbalized metacognitive phrases from interview records, adapting the tool's dictionary to suit middle school students' language characteristics.
  3. Extract metacognitive strategies from students' software usage logs, such as concept reading or map editing based on quiz results.
  4. Analyze correlations between metacognitive phrases and students' behavioral strategies in the software.

Research Findings

• Findings:
  • Metacognition and Learning (RQ1): The interview process itself may promote learning, but no significant relationship was found between verbalized metacognition and learning outcomes.
  • Metacognition and Confusion (RQ2): Recently resolved confusion (e.g., "confusion → delight" states) tends to be associated with more metacognitive expressions, while unresolved confusion (e.g., "confusion → frustration → boredom" states) is associated with fewer metacognitive expressions.
  • Metacognition and Behavioral Strategies (RQ3): Certain behavioral strategies (e.g., marking/editing concept maps based on quiz results) are positively correlated with verbalized metacognition, though not all strategies are significantly covered.
• Advantages and Significance: Provides a comprehensive, automated method for measuring metacognition; reveals the potential of interviews as effective learning support tools, particularly for resolving confusion.
• Experimental and Evaluation Results: Analysis of 493 interview data points and software interaction logs found that the interview process may have a potential positive impact on students' learning outcomes. The NLP tool's measurement results showed high consistency with manual annotations (Cohen's kappa = 0.684).
• Limitations and Future Directions:
  • The study cannot establish causality; future experiments are needed to verify the actual impact of interviews on learning.
  • Explore integrating similar interview functionalities into educational technologies to promote real-time metacognitive activities during learning.
  • Extend research to different age groups and learning environments to validate the generalizability of the findings.

Conclusion

This study provides an in-depth exploration of the significance and potential methods for assessing metacognition in computerized learning environments. The experimental results support the potential of student interviews as learning support tools and offer important insights into confusion resolution and metacognitive behavioral patterns. These findings provide empirical evidence and tool support for the design and optimization of educational technologies, while also demonstrating the application potential of NLP in the education field.