Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education

Title of the Paper

Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education

Paper Information

• Subject Area: Artificial Intelligence and Education, Programming Learning
• Keywords: Human-AI interaction, LLM agents, AI and education, generative AI, programming instruction, learning-based educational systems, conversational agents

Research Background and Issues

• Problems or Challenges Identified by the Authors:

  • Teachable agents are difficult to scale due to the labor-intensive programming required for specific domain knowledge.
  • Although large language models (LLMs) reduce the cost of creating teachable agents, their extensive knowledge can overwhelm learners during the teaching process.
  • Current teachable agents fail to effectively stimulate learners' knowledge construction and deep reflection.
  • Learners interacting with traditional agents often remain at the level of knowledge-telling, struggling to transition to knowledge-building.

• Importance of the Research:

  • The application of teachable agents in "Learning By Teaching" (LBT) can enhance learners' deep understanding and knowledge restructuring abilities.
  • Exploring cost-effective and efficient teachable agent systems can provide widely accessible teaching solutions for educators and learners.

• Motivation and Related Work:

  • Previous research has demonstrated the potential of teachable agents to improve learning outcomes, but their application is limited due to the high cost of building knowledge models.
  • LLMs have shown strong capabilities in dialogue, role simulation, and learning from examples, offering opportunities to enrich the interaction methods of teachable agents.
  • Scholars have studied how guiding questions and metacognitive feedback improve teaching effectiveness, but few have integrated these techniques into LLM-based teachable agents.

Solution

• Proposed Solution:

  • This study designed a learning system called TeachYou, which includes an LLM-based teachable agent named "AlgoBo" and a real-time analysis module for supporting metacognitive teaching feedback.
  • Introduced the Reflect-Respond prompting pipeline, enabling AlgoBo to simulate knowledge states and demonstrate a progressive learning curve through reflection and response mechanisms.
  • Implemented "Mode-shifting," allowing AlgoBo to alternate between answering and questioning during conversations to encourage learners' knowledge construction.
  • Added a "Teaching Helper" module to provide real-time teaching feedback, helping learners optimize their teaching methods.

• Innovations:

  • AlgoBo can exhibit misconceptions and incremental learning abilities based on predefined knowledge states.
  • Designed a question-generation system to provoke deep thinking, tailored to learners' knowledge expansion needs during teaching.
  • The Teaching Helper provides real-time analysis of conversations and metacognitive feedback to enhance teaching effectiveness, representing a novel approach.

• Implementation Steps and Key Technologies:

  • Used prompt engineering to develop the Reflect-Respond pipeline, simulating the knowledge learning process of LLMs.
  • Implemented Mode-shifting technology, introducing thought-provoking questions in conversations through a question generator and dialogue classifier to stimulate knowledge construction.
  • Integrated the Teaching Helper to detect flawed teaching patterns and provide feedback during conversations.

Research Outcomes

• Specific Outcomes:

  • Technical evaluations show that the Reflect-Respond pipeline effectively configures, maintains, and updates the knowledge states of the teachable agent.
  • User studies reveal that Mode-shifting significantly increases the density of knowledge-building messages in conversations (effect size Cohen's d=0.71).
  • The Teaching Helper aids learners in reflecting on teaching methods, though its impact on overall metacognitive ability improvement is limited.

• Advantages:

  • Compared to traditional teachable agents, the LLM-based design is more cost-effective and versatile, applicable to teaching in various domains.
  • Mode-shifting makes learner-agent interactions more dynamic and enhances the interactivity and immersion of the learning experience.

• Experimental or Evaluation Results:

  • The reflection and response flow maintains AlgoBo's knowledge state consistency (persistence) under random information inputs.
  • User study results indicate that TeachYou is better suited to inspire learners to reconstruct knowledge and engage in deep thinking.
  • TeachYou achieves a student-teacher role-sharing mechanism aligned with the characteristics of LBT.

• Limitations and Future Directions:

  • Limitations:

    • Current testing is limited to algorithm learning and has not been validated in other disciplines (e.g., mathematics, physics).
    • Teaching effectiveness evaluation lacks pre- and post-test comparisons, relying only on indirect dialogue analysis.
    • The system does not yet consider personalized learner settings, potentially reducing engagement.

  • Future Directions:

    • Explore the applicability of LLM-driven teachable agents for declarative knowledge learning.
    • Develop customizable interfaces for teachable agents to better match individual teaching needs.
    • Deploy the system in large-scale classroom environments and conduct longitudinal studies to observe learners' long-term metacognitive improvement and system acceptance.

Conclusion

This study successfully designed the TeachYou system and validated its effectiveness in encouraging learners to construct knowledge and optimize teaching interactions. Leveraging the flexibility and cost-efficiency of LLMs, the research demonstrates the potential of LBT-based teachable agents while proposing future design improvements. These findings provide significant insights for both academia and practitioners.