How Beginning Programmers and Code LLMs (Mis)read Each Other

Title of the Paper

How Beginning Programmers and Code LLMs (Mis)read Each Other

Paper Information

• Field of Study: Human-AI interaction, focusing on how novice programmers interact with large language models (LLMs) for code generation.
• Keywords: novice programmers, code generation, LLM, human-AI interaction, educational technology, programming education, code generation strategies, task difficulty, natural language descriptions, automated evaluation

Research Background and Problem

• Identified Problems or Challenges: Novice programmers face several critical challenges when interacting with large language models for code generation (Code LLMs), including articulating their programming intentions clearly, evaluating the correctness of the generated code, and iteratively refining their descriptions to address issues in the code.

• Significance: Code generation models have become productivity tools in fields like software engineering. Studying their impact on non-professional users, especially students, could uncover technological gaps and benefit programming education. However, systematic research on how beginners use these tools remains insufficient.

• Motivation and Related Work: Existing research has primarily focused on the impact of Code LLMs on professional programmers, with limited detailed studies on how beginners interact with these tools and the specific challenges they face. This paper aims to fill this gap and explore pathways to further democratize programming tools.

Solution

• Method or Solution: The paper designs a large-scale, multi-institutional experiment involving 120 university students who have just completed an introductory programming course. It investigates how they write initial descriptions and modify them to generate correct code.

• Innovations:

  • Introduced a unique experimental design that uses automated testing of code correctness to isolate specific difficulties students face in creating and editing code generation descriptions.
  • Conducted a large-scale experiment covering students from three universities and eight categories of programming problems.
  • Performed in-depth analysis of issues related to model output non-determinism and low-level task selection.

• Implementation Steps and Techniques:

  1. Provide input/output test cases and ask students to write natural language descriptions for code generation.
  2. Use the Codex model to generate code and automatically evaluate the correctness of the generated code against the test cases.
  3. When the generated code fails, students revise their descriptions and attempt again.
  4. Collect interaction data, including reasons for failure, students’ perceptions, and their mental models of the LLM.

Research Findings

• Specific Findings:

  • Over 43% of students successfully generated partial code, but some required multiple attempts.
  • Many students struggled to interact effectively with Code LLMs, including issues with unclear descriptions, unfamiliarity with the structure of the output code, and challenges in dealing with the randomness of model outputs.
  • The experiment revealed misconceptions in students’ mental models, such as believing that the model retrieves code based on keywords.

• Advantages and Comparisons:

  • Compared to existing studies, this experiment addressed the confounding effects of problem selection and description methods on results, systematically analyzing specific cognitive challenges faced by students.
  • Provided empirical support for programming education and tool design, including guidance on how to train beginners to craft effective descriptions.

• Experimental or Evaluation Results:

  • Students’ task success rates were correlated with their programming experience and family educational background. For instance, students with additional programming experience achieved higher success rates.
  • First-generation college students performed significantly worse in description generation tasks, highlighting fairness issues with such algorithmic tools.

• Limitations and Future Directions:

  • Limitations:
    • Students’ foundational programming backgrounds were not entirely uniform, which may have influenced the experimental results.
    • The rapidly evolving technological landscape means that the Codex model used may represent a limited level of technical capability or experience conditions.
  • Future Directions:
    • Develop more beginner-friendly Code LLM interaction interfaces, such as incorporating more interpretable system designs to reduce confusion.
    • Design educational training content based on users’ mental models to help beginners build a correct understanding of AI interactions.
    • Optimize model selection, such as using open-source models, to reduce the risk of technological dependency in research.

Through this experiment, the study not only opens up possibilities for applying Code LLMs in education but also reveals key challenges in promoting the democratization of programming.