Studying the Effect of AI Code Generators on Supporting Learners in Introductory Programming

Document Title

Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming

Document Information

• Subject Areas: Human-Computer Interaction, Computer Science Education, Applications of Artificial Intelligence in Programming Learning
• Keywords: Large Language Models, AI Code Assistants, AI-Assisted Programming, OpenAI Codex, GPT-3, ChatGPT, Copilot, Introductory Programming, K-12 Computer Science Education

Research Background and Issues

• Problems or Challenges:

  • Novices often struggle with learning text-based programming languages due to the complexity of syntax and logic.
  • The use of AI code generators (e.g., OpenAI Codex) may yield two contrasting outcomes: aiding learning or fostering dependency.

• Importance:

  • Investigating whether AI code generators can lower the barriers to programming and support novices in learning both programming logic and theory.
  • Understanding the impact of these tools on academic integrity and learning outcomes in educational contexts, as well as their potential limitations.

• Research Motivation and Related Work:

  • Previous studies have primarily focused on the usability of AI code generators rather than their impact on learning outcomes.
  • The authors aim to explore, through quantitative experiments, how these tools assist novice learners in programming and whether they lead to excessive reliance during the learning process.

Solution

• Methodology:

  • A controlled experiment was designed using the Python-based learning platform Coding Steps, involving 69 novices aged 10-17 with no prior experience in text-based programming.
  • The experiment consisted of three phases: introduction (using Scratch programming), training (completing tasks with AI code generators or manual programming), and evaluation (testing retained programming skills post-learning).
  • During the training phase, participants were divided into two groups: one using OpenAI Codex for assistance and the other manually completing tasks.

• Innovations:

  • The study examined novice learners' learning outcomes and behaviors when using code generators through two types of tasks: code creation and code modification.
  • Detailed tracking and quantification of AI tool usage patterns, such as frequency of use, proportion of generated code modified, and frequency of documentation access.

• Implementation Steps and Techniques:

  • Developed a customized learning environment, "Coding Steps," featuring syntax highlighting, real-time error detection, embedded Python documentation, and AI code generation tools.
  • A logging system was employed to track each learner's task completion, documentation access, code execution errors, and code generator usage behaviors.

Research Findings

• Specific Results:

  • Using AI code generators significantly improved task completion rates, accuracy scores, reduced task completion time, and minimized errors in code creation tasks.
  • In code modification tasks, both experimental groups performed similarly, indicating that AI code generators did not impair learners' ability to modify code during the training phase.

• Comparison with Existing Findings:

  • Compared to the manual learning baseline group, learners in the Codex-assisted group performed better during the training phase, but their advantages in retention tests were not statistically significant.
  • Learners with higher programming proficiency showed more significant improvements after using AI tools, while those with lower proficiency benefited less.

• Experimental or Evaluation Results:

  • During the evaluation phase (including retention tests), the AI code generator group performed slightly better, particularly when tackling complex topics such as loops and arrays.
  • Novices employed various strategies when using AI code generators, such as breaking tasks into smaller steps and writing detailed prompts, although 32% of prompts were simple copies of problem descriptions.

• Limitations and Future Directions:

  • The sample size limited the statistical significance of certain metrics.
  • The study did not delve into novices' cognitive processes, task decomposition strategies, or learning behaviors when interacting with AI-generated code.
  • Future studies could explore AI code generators' support for more complex algorithm design and their impact on non-native English-speaking learners.
  • Tools could be designed to mitigate overuse or dependency, such as providing explanations alongside generated code or requiring learners to engage in code modification tasks.

Design Recommendations and Future Research

• Design Recommendations:

  • AI code assistants should reduce cognitive load for learners by incrementally presenting code blocks with explanations.
  • Introduce restrictions, such as requiring learners to complete related subtasks before using generated code, to deepen their understanding of code logic.

• Future Research Directions:

  • Optimize and clarify how AI code assistants support novices in language expression and logical modeling.
  • Investigate the impact of introducing AI generators in more formal educational contexts, such as high school or university computer science courses.

This study is the first to quantify the impact of AI code generators on novice learners' ability to learn text-based programming, providing valuable insights for educators and researchers exploring the application of AI in programming education.