Assisting Teaching Assistants with Automatic Code Corrections

Title of the Paper

Assisting Teaching Assistants with Automatic Code Corrections

Paper Information

• Subject Area: Computer science education, specifically the role and tools for Teaching Assistants (TAs) in helping students debug programming issues.
• Keywords: Student code debugging, programming education, automatic code correction, cognitive load, teaching assistants, user behavior analysis, computer science education, debugging tools, student intent understanding, programming education technology.

Research Background and Problems

• What problems or challenges did the authors identify?

  • Undergraduate Teaching Assistants (TAs) are often the first and only resource for students struggling with programming issues.
  • However, TAs are often novices in both programming and teaching, making it challenging for them to simultaneously understand students' code intent, identify and fix code errors, and explain solutions to students.
  • Research indicates that the primary difficulties for TAs do not stem from conceptual understanding but from high cognitive load and limitations in working memory, such as missing repetitive errors and being prone to distractions.

• Why is this problem important?

  • The abilities of TAs significantly impact students' learning and programming skill development.
  • Helping TAs identify issues and articulate explanations more efficiently can improve students' learning experiences and enhance course effectiveness.

• Research motivation and related work

  • Existing research has primarily focused on evaluating the overall effectiveness of TA programs, with limited detailed studies on TA-student interactions.
  • While there has been work on debugging and automatic code correction, most of it focuses on programmers' cognitive processes or tools that help students independently fix their code, with little exploration of the impact of providing correction information to teaching personnel assisting students.
  • This study aims to explore how providing automatically generated "corrected code" can assist TAs in better debugging and guiding students.

Solution

• What methods or solutions did the authors propose?

  • The study designed an interface that displays a comparison between students' code and automatically generated corrected code.
  • It investigates whether providing corrected code can help TAs debug faster and more accurately while reducing cognitive load.
  • Experiments were conducted to evaluate the impact of high-quality and low-quality corrected code on TAs' behavior.

• What is innovative about this solution?

  • This is the first study focusing on the interaction between TAs and automatic code correction, emphasizing aiding TAs in improving student feedback rather than directly providing hints for students' use.
  • It provides a detailed analysis of the challenges TAs face in debugging and offering feedback and evaluates how tools can effectively support them.

• What are the implementation steps? What key technologies were used?

  • Experimental design:
    • Two experiments were conducted to test the impact of high-quality and low-quality automatic corrections on TAs.
    • A web-based interface was implemented, allowing TAs to view student code, test case results, and automatically generated corrected code under experimental conditions.
  • Data collection:
    • Logs of TAs' code testing behavior, final submitted code, and feedback to students were recorded.
    • Pre-experiment data on participants' experience (e.g., Python experience, prior TA roles) and post-experiment surveys on system usability were collected.
  • Data analysis:
    • Quantitative analysis was used to compare debugging time, code modification distance, and error correction rates under different experimental conditions.
    • Qualitative methods analyzed whether TAs' written explanations covered all code errors.

Research Outcomes

• What specific outcomes were achieved?

  • When TAs could view automatically generated corrected code, debugging time was reduced by an average of 29%.
  • TAs' solutions were closer to students' original solutions, requiring fewer modifications (shorter edit distance).
  • Automatic code correction significantly improved TAs' accuracy in fixing and explaining code errors. Viewing corrected code increased the likelihood of fixing code errors by 17 times and explaining all errors by approximately 30%.

• What advantages does it have compared to existing solutions?

  • While most existing research focuses on helping students self-correct, this study is the first to explore indirect improvement of student learning outcomes through TA-support tools.
  • The experimental design provides detailed insights into the specific errors that pose the greatest challenges for TAs and identifies which tool features are most effective in alleviating these difficulties.

• What were the experimental or evaluation results?

  • Even low-quality automatic code corrections, while potentially slowing down debugging, did not significantly impact the accuracy of final code fixes and error explanations.
  • TAs faced significantly more challenges with complex code segments that involved difficult-to-infer "student intent" (e.g., non-standard algorithms). These issues notably affected their debugging performance and the accuracy of their communication with students.

• Limitations and future directions

  • Limitations:
    • The number of participants was limited (22), and larger-scale experiments are needed to validate the findings.
    • The experimental environment differed from real classroom settings, as participants did not interact with students in real-time.
    • The interface's comparison feature may still be insufficient for handling multiple errors, especially those involving small-scale modifications that are easier to overlook.
  • Future research directions:
    • Extend experiments to more complex and larger teaching scenarios, such as testing TAs' real-time feedback capabilities in online student interactions.
    • Improve interface tools, such as integrating error categorization and explanation features, to further analyze the impact of complex errors on TA assistance.
    • Investigate how TAs use generated corrected code to infer student intent and, based on this, help students learn more effectively.