CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

Title of the Paper

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

Paper Information

• Subject Area: Educational Technology, Applications of Artificial Intelligence in Computer Science Education
• Keywords: Programming Education, Intelligent Tutoring Systems, Large Language Models, Educational Technology, AI Assistant, AI Tutoring, Generative AI, Classroom Deployment, Design Guidelines

Research Background and Problem

• Problems or Challenges Identified by the Authors: Providing immediate and personalized learning feedback in traditional computer programming courses is challenging, especially with the increasing number of students learning programming. Existing resources (e.g., teaching assistants' office hours) are limited in their utilization and may lead to inequities. Additionally, tools like ChatGPT can offer quick code solutions but may result in academic integrity issues and excessive dependence, hindering deep learning.

• Why the Problem is Important: Learning programming requires iterative practice and deep understanding. By providing tools that support students' self-directed learning, it is possible to enhance learning outcomes, improve the fairness and scalability of educational resources, and alleviate the pressures of increasing class sizes.

• Research Motivation and Related Work: Large Language Models (LLMs) have shown potential in computer science education by generating code, debugging, and explaining concepts. However, tools with "guardrails" need to be developed for educational settings to prevent directly providing code solutions and instead focus on supporting creative learning.

Solution

• Proposed Method or Solution: The authors designed and deployed a programming assistant system called CodeAid, based on LLMs, aimed at providing technically accurate feedback without directly displaying code solutions. CodeAid's features include answering conceptual questions, generating annotated pseudocode, debugging incorrect code, and explaining lines of student code.

• Innovative Aspects of the Solution: CodeAid is education-focused, with "guardrails" in place to limit the generation of direct code solutions, encouraging cognitive engagement and self-regulation. Key innovations include:

  • Assisting students in asking questions through natural language interaction
  • Providing pseudocode and detailed line-by-line explanations
  • Visual annotations of errors in student code with explanations on how to fix them

• Implementation Steps and Key Technologies:

  • User Interface Design: Offers five main functionalities (general Q&A, code explanation, code debugging, etc.) along with tutorial videos to help users understand the features.
  • Backend Architecture: Utilizes OpenAI API to generate responses and applies few-shot learning techniques to constrain the LLM's generation capabilities, ensuring outputs are structured, comprehensible, and technically reliable.
  • Data Collection and Analysis: After deployment, logs were used to record student queries and evaluate feedback. A "stepwise streaming generation" method was employed to optimize response speed.

Research Outcomes

• Specific Results:

  • CodeAid was deployed for an entire semester in a programming course with 700 students, handling approximately 8,000 student interactions.
  • Insights were gathered from analyzing the types of student queries, response accuracy, and student feedback based on real-world usage.
  • Four design principles were identified: leveraging AI's unique strengths, simplifying queries while promoting cognitive engagement, avoiding direct answers to encourage motivated learning, and ensuring transparency and control.

• Comparison with Existing Solutions and Advantages:

  • Compared to ChatGPT, CodeAid focuses more on educational objectives, avoiding direct display of code solutions.
  • The feedback provided is better structured to meet students' learning needs, promoting deeper integration of educational tools while reducing reliance on general-purpose AI tools.

• Experimental or Evaluation Results:

  • The overall response accuracy was 79%, with a helpfulness rate of 86%. After updates, the system's accuracy improved to 87%, with helpfulness reaching 91%.
  • Students reported that CodeAid's query interface and feature design enhanced their learning experience, particularly the pseudocode generation and error annotation features.
  • Despite some errors or incomplete answers, CodeAid demonstrated high relevance to course materials, significantly improving students' technical learning.

• Limitations and Future Directions:

  • The study focused on a single course and programming language (C language), and the generalizability of CodeAid needs further validation.
  • The performance of the AI output model affects system accuracy, necessitating the use of more advanced models in the future.
  • The long-term effects of using such AI educational tools on students' learning outcomes, skill development, and self-regulation capabilities warrant further investigation.
  • Future work could explore broader course integration designs and investigate how diverse student groups utilize such tools for learning.

This study provides valuable insights into developing AI-assisted programming learning tools with greater educational significance. Future AI designs should strike an optimal balance between student learning needs and educational principles to support broader educational goals and contexts.