From Code Generation to Conceptual Learning: Student Use of LLMs in a Web Programming Course

Paper Title

From Code Generation to Conceptual Learning: Student Use of LLMs in a Web Programming Course

Publication Info

• Topic area: Use of Large Language Models (LLMs) in computer science education.
• Keywords: LLMs, web programming, computer science education, prompt engineering, debugging, conceptual learning, metacognition, AI-assisted learning, student performance, programming assignments.

Background and Problem

• Problem / challenge: Existing studies on LLM use in education often rely on self-reports or controlled experiments with limited scope, failing to capture organic, real-world usage in advanced programming courses.
• Significance: Understanding how students use LLMs in realistic settings is crucial for designing effective educational strategies and preparing students for industry practices.
• Motivation and related work: Prior research has explored LLMs in introductory courses or group-based projects but lacks insights into individual usage patterns in advanced courses. This study builds on work like Arora et al. by focusing on individual interactions and their correlation with academic performance.

Solution

• Proposed approach: Analysis of 448 LLM chat logs from 147 students in a senior-level web programming course over two semesters, using open coding to categorize interactions and correlate them with academic outcomes.
• Novelty:
  1. Identification of 14 distinct prompt–response pairs across three categories: code generation, debugging, and conceptual explanation.
  2. Correlation of specific LLM interaction patterns with academic performance.
  3. Observation of a temporal shift from code generation to explanation-oriented interactions.
  4. Mapping of student prompts to established prompt-engineering techniques.
• Procedure and key techniques:
  1. Collection of LLM chat logs, programming assignments, surveys, and grades from students.
  2. Open coding of chat logs to categorize interactions into implement, debug, and explain categories.
  3. Statistical analysis to correlate interaction patterns with academic performance and code borrowing rates.
  4. Comparison of interaction trends across two semesters.

Results

• Concrete findings:
  • High-effort, detailed prompts for code generation correlated positively with final grades (r = 0.25, p < 0.01).
  • Low-effort behaviors, such as pasting raw error messages, correlated negatively with performance (r = −0.34, p < 0.01).
  • Explanation-oriented interactions increased over time, with all students using "explainhow" prompts by 2024.
• Advantage over baselines:
  • Detailed prompting strategies led to better academic outcomes compared to low-effort or generic prompting.
  • Explanation-oriented prompts provided richer learning opportunities compared to simple code generation.
• Experiments / evaluation:
  • Data from 147 students across two semesters, with 448 chat logs analyzed.
  • Correlations between prompt types and grades, as well as code borrowing rates, were assessed.
  • Assignments included real-world programming tasks in the MERN stack.
• Limitations and future work:
  • Voluntary participation may bias results toward higher-performing students.
  • Chat logs do not capture students' underlying intentions or metacognitive processes.
  • Future work could include longitudinal studies, adaptive GenAI systems, and comparisons with professional developers.

Summary

This study analyzed how students used LLMs in a senior-level web programming course, identifying 14 distinct interaction types and their correlations with academic performance. High-effort, detailed prompts were linked to better grades, while low-effort behaviors like pasting error messages were associated with poorer outcomes. Over time, students shifted from using LLMs primarily for code generation to leveraging them as conceptual tutors. These findings highlight the importance of effective prompting strategies and suggest that LLMs can support both learning and skill development when used thoughtfully. The results have implications for educators designing AI-integrated curricula and for the broader field of human-computer interaction.