Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science

Paper Title

Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science

Publication Info

• Topic area: AI literacy and its impact on programmatic data science education.
• Keywords: LLMs, AI literacy, data science, programming education, behavioral analysis, technical experience, AI-assisted learning, pedagogical design, computational notebooks, human-AI collaboration.

Background and Problem

• Problem / challenge: Despite the promise of LLMs to democratize technical tasks, their benefits are unevenly distributed. Prior studies have not sufficiently clarified how different types of user experience influence AI use and outcomes, particularly in real-world, time-constrained workflows.
• Significance: Understanding these disparities is critical for designing effective AI literacy curricula and tools that ensure equitable benefits from LLMs in technical education.
• Motivation and related work: Previous research has shown mixed results regarding the impact of AI on novice and expert users, with limited focus on real-world workflows and behavioral patterns. This study addresses gaps by analyzing how technical, LLM, and communication experiences shape AI use in a graduate-level data science course.

Solution

• Proposed approach: A mixed-methods study analyzing student interactions with LLMs in a graduate data science course, focusing on behavioral patterns and AI literacy competencies.
• Novelty:
  1. Empirical evidence that LLMs do not fully close the technical experience gap in realistic settings.
  2. Development of a behavior-oriented log annotation schema for analyzing AI use in computational notebooks.
  3. Identification of AI literacy competencies across conceptual, procedural, metacognitive, and dispositional dimensions.
• Procedure and key techniques:
  • Data collection from 36 students via homework submissions, logs, surveys, and screen recordings.
  • Behavioral log analysis using a custom annotation schema to segment workflows into episodes and steps.
  • Statistical and thematic analyses to explore the relationship between experience levels, AI usage behaviors, and performance.

Results

• Concrete findings:
  • Technical experience significantly predicted higher grades (β = 6.09, p = 0.041), while LLM and communication experience did not.
  • Under time pressure, LLMs reduced performance gaps, but these reappeared in more realistic, extended workflows.
  • Experienced students used AI more strategically, with clearer prompts and proactive planning, while novices relied on AI reactively.
• Advantage over baselines:
  • Post-instruction, students improved in prompt quality behaviors (30% increase in appropriate AI use ratio) but struggled with evaluation skills, highlighting the need for targeted pedagogical interventions.
• Experiments / evaluation:
  • Tasks included data cleaning, exploratory data analysis (EDA), machine learning, and storytelling, graded using detailed rubrics.
  • Behavioral logs (7,315 events) were analyzed to identify AI use patterns and their impact on task success.
  • Pre- and post-instruction comparisons revealed gains in specific AI use skills but persistent gaps in evaluative behaviors.
• Limitations and future work:
  • Small sample size (36 students) limits subgroup analysis.
  • Correlated experience variables (e.g., Python and data science) complicate disentangling their effects.
  • Future work should include more diverse tasks, controlled experiments, and robust pretests to isolate factors influencing effective AI use.

Summary

This study highlights that LLMs do not uniformly democratize programming tasks, as technical experience remains a key predictor of success. Behavioral analysis revealed that experienced students use AI more strategically, while novices often struggle with vague prompts and reactive usage. Although lightweight instruction improved some AI use skills, evaluative behaviors require deeper pedagogical investment. The findings inform AI literacy curricula and tool design, emphasizing the need for explicit mental models, scaffolding for evaluation, and fostering durable problem-solving competencies.