Playing Dumb to Get Smart: Creating and Evaluating an LLM-based Teachable Agent within University Computer Science Classes

Research Background and Issues

• What problems or challenges did the authors identify?
  When students learn programming through the "Learning by Teaching" (LBT) method, it can effectively deepen their understanding. However, traditional human-based LBT is limited in scalability and risks reinforcing misconceptions. Therefore, there is a need for a teachable agent driven by large language models (LLMs) to simulate the role of non-expert learners and expand the scope of LBT practices.

• Why is this issue important?
  LBT not only helps students consolidate knowledge but also promotes active learning, identifies knowledge gaps, and improves attitudes toward programming. However, tools for implementing this method often rely on human or even expert resources, making it difficult to scale in larger educational environments. Additionally, while students traditionally use LLMs as knowledge-providing tools, there is a lack of research exploring their potential and long-term impact as "teachable learners."

• Research Motivation and Related Work
  Existing studies show that using AI or robots as teaching objects can improve student learning outcomes, and LLMs, with their conversational generation capabilities, clearly have the potential to simulate the role of learners. However, the long-term impact of LLM-based LBT in extended course environments requires further empirical investigation.

Solution

• What methods or solutions did the authors propose?
  This study designed and evaluated an LLM-based teachable agent tool—MatlabTutee—using Chain-of-Thought prompts to guide LLMs in simulating the role of non-expert learners, enhancing students' LBT experiences. It also compared this system with humans pretending to be non-experts and explored its long-term use in real university computer science courses.

• What are the innovative aspects of this solution?

  1. Using simple Chain-of-Thought prompts to control LLM-generated teaching dialogues without complex knowledge state simulations.
  2. Incorporating multiple intentional errors and typical non-expert behaviors to maintain consistency in the learner role.
  3. Directly comparing the system with traditional LLMs and humans pretending to be learners to quantify its performance as a teachable agent and its interaction with students.

• What are the implementation steps and key technologies used?

  1. Preliminary Design and Evaluation: Developing initial prompts and observing student interactions with LLMs or human assistants through single-day experiments.
  2. Chain-of-Thought Optimization: Refining prompts based on feedback to introduce Chain-of-Thought methods and test the system's ability to generate incorrect code and explanations.
  3. Comprehensive Comparative Experiments: Conducting structured week-long experiments comparing MatlabTutee with similar human and traditional LLM systems, analyzing interaction quality, student behavior, and cognitive load.
  4. Long-term Usage Studies: Performing two month-long longitudinal experiments (partially controlled and fully uncontrolled) to evaluate students' autonomous adoption of the system in real teaching scenarios and its long-term effects.

Research Outcomes

• What specific outcomes were achieved?

  1. MatlabTutee successfully simulated the role of a non-expert learner, with its intelligence level perceived by students as similar to humans pretending to be uninformed experts.
  2. Interacting with MatlabTutee enabled students to identify knowledge gaps, accurately assess their own abilities, and improve their positive attitude toward computer science.

• What advantages does it have compared to existing solutions?

  1. Compared to traditional GPT models, MatlabTutee better facilitates active student engagement in learning rather than simply providing answers from AI.
  2. Its interaction model resembles human learners, effectively stimulating active knowledge construction and deep reflection during the LBT process.

• What were the experimental or evaluation results?

  • Under controlled conditions, MatlabTutee's LBT experience was similar to that of human interactions, prompting students to engage in active teaching behaviors such as explanation, reasoning, and deep learning. In contrast, traditional GPT models were primarily used for knowledge retrieval, leading to more passive learning behaviors.
  • In long-term experiments, while most students recognized the system's potential benefits, they reduced usage frequency due to incorrect feedback and conflicts with course schedules, with average interaction times being relatively short.

• Limitations and Future Directions

  1. MatlabTutee exhibited "learning too quickly" during extended interactions, leading to distrust among some students, necessitating further optimization of simulated learning trajectory design.
  2. Students had polarized reactions to teaching errors, which may impact sustained usage. Future designs could explore more motivational approaches to enhance feedback acceptance.
  3. Longitudinal experiments revealed low student autonomy in adopting the system, suggesting the need for integration with course objectives or grading mechanisms to encourage broader adoption.

Conclusion and Discussion

• This study demonstrated the potential of Chain-of-Thought prompts in constructing simple yet effective LLM-based teachable agents, while also revealing challenges such as insufficient student intent for natural usage, sensitivity to incorrect feedback, and inconsistent simulated learning trajectories.
• Open questions for future research include balancing transparency with customized learning trajectories, providing constructive feedback that does not undermine motivation, and expanding students' long-term autonomous use of such systems.