Piecing Together Teamwork: A Responsible Approach to an LLM-based Educational Jigsaw Agent

Research Background and Issues

• Identified Challenges and Issues:

  • Global collaboration skills "deficit": PISA 2015 revealed that over 70% of 15-year-old students failed to meet expected levels in simple collaborative problem-solving tasks. Many students exhibit poor collaboration skills.
  • Teachers face multitasking challenges in classrooms, making it difficult to monitor students' collaboration status in real-time and provide guidance.
  • Current computer-supported collaborative learning (CSCL) research primarily focuses on cognitive outcomes, with less attention given to fostering collaboration and knowledge sharing.
  • The application of AI technologies in classrooms raises significant ethical and practical concerns, such as data privacy, dependency, and the lack of standardized usage guidelines.

• Significance of the Research:

  • Collaboration is critical in future educational environments and workplaces. Designing an AI system capable of effectively monitoring group collaboration and intervening at critical moments will equip students with essential skills.
  • AI systems supporting education must adhere to responsible design and human-centered principles to address ethical concerns when applying educational technologies to sensitive populations.

• Research Motivation and Related Work:

  • Developing a conversational educational agent based on large language models (LLMs) to support collaborative learning, while integrating a responsible innovation framework for human-AI collaboration.
  • Existing research on conversational agents in education has rarely focused on collaborative learning in secondary school classrooms.
  • Aims to bridge the gap between ethical AI, CSCL, and the practical application of generative AI in education.

Solution

• Proposed Methods and Solutions:

  • Design an LLM-based educational agent named Jigsaw Interactive Agent (JIA) to support students in knowledge sharing and interaction during "jigsaw method" collaborative tasks.
  • Employ a human-centered perspective and a responsible innovation framework (including reflexivity, anticipation, inclusivity, and responsiveness) to co-develop and optimize the system with experts in education and technology.
  • Integrate "human-in-the-loop" (HITL) capabilities, enabling human experts to review and regulate JIA's intervention messages.

• Innovative Features:

  • Introduce an LLM-based conversational agent into a classic educational practice (jigsaw method) with dialogue strategies built on transparent rules.
  • Utilize multiple natural language processing (NLP) tools and speech analysis technologies to assess team collaboration status in real-time and optimize intervention strategies.
  • Implement multi-level control in agent interventions: (1) transparent decision tree models; (2) real-time human expert review of LLM-generated messages.

• Implementation Steps and Key Technologies:

  1. Data Collection: Develop jigsaw activities based on existing STEM curricula, record videos and dialogues, and simulate agent interventions using Wizard-of-Oz (WoZ) experiments.
  2. Behavior Annotation and Modeling: Use an adapted MOSAIC-AI protocol to annotate students' collaboration status before interventions, combined with NLP models to analyze dialogue features.
  3. Decision Tree Training: Train decision trees using factor analysis and dialogue features to predict students' collaboration status and construct interpretable dialogue strategy rules.
  4. LLM Generator Management: Generate supportive dialogue intervention messages using a revised Mistral LLM and manually designed prompt templates.
  5. Human Oversight: HITL mediates AI-generated outputs by reviewing (accepting, modifying, ignoring, or rejecting) messages.

Research Outcomes

• Specific Outcomes:

  • JIA effectively reduced social loafing and promoted respectful and deeper thinking in students' language.
  • Experiments showed that, compared to control conditions without agent support, the JIA intervention group demonstrated higher levels of "respectful collaboration" language and language that encouraged deeper thinking.
  • Compared to the WoZ-SME condition, the JIA group exhibited more authentic and sustained communication styles.

• Advantages Over Existing Solutions:

  • Transparent dialogue strategies make agent behavior more interpretable and easier to optimize.
  • HITL functionality enhances the system's perception and response capabilities while allowing humans to adjust model behavior in real-time, reducing common generative model issues like "hallucinations" (incorrect responses).
  • The system is fast, lightweight, and deployable in real classroom settings.

• Experimental or Evaluation Results:

  • Analysis of 58 experimental groups revealed that the JIA group was more effective in motivating student collaboration and reducing individualistic tendencies compared to the control group.
  • Social loafing significantly decreased in the WoZ-SME group, but no significant difference was observed between the JIA and WoZ-SME groups.
  • LLM-generated intervention messages required minimal modification in most cases, further demonstrating their practicality in educational interventions.

• Limitations and Future Directions:

  • Experiments were limited to single-session laboratory scenarios; future work should test the system's feasibility in real classroom environments and over longer periods.
  • The decision tree model achieved a testing accuracy of 62%. While interpretable, its performance is limited. Future work could explore integrating complex models (e.g., random forests) to improve predictive performance.
  • The system did not incorporate non-verbal signals (e.g., body language), limiting its ability to comprehensively detect phenomena like social loafing.
  • HITL feedback was treated as independent evaluation; future iterations could integrate it into machine learning processes, enabling interactive reinforcement learning.

Through this research, JIA not only demonstrates the potential of LLMs in educational collaboration but also provides a model for AI technology development centered on responsible innovation, offering significant academic and practical insights.