When Scaffolding Breaks: Investigating Student Interaction with LLM-Based Writing Support in Real-Time K-12 EFL Classrooms

Paper Title

When Scaffolding Breaks: Investigating Student Interaction with LLM-Based Writing Support in Real-Time K-12 EFL Classrooms

Publication Info

• Topic area: Integration of large language models (LLMs) in real-time K-12 English as a Foreign Language (EFL) classrooms.
• Keywords: LLMs, scaffolding, K-12 education, EFL classrooms, real-time learning, AI writing support, classroom equity, student engagement, teacher roles, grammar learning.

Background and Problem

• Problem / challenge: While LLMs show promise in scaffolding English writing skills, their effectiveness in real-time K-12 classrooms remains underexplored, particularly under constraints like diverse proficiency levels and limited time.
• Significance: Understanding the impact of LLMs in real-time classrooms is crucial for designing inclusive educational tools that support diverse learners without exacerbating existing inequities.
• Motivation and related work: Prior research has focused on asynchronous environments or higher education contexts, leaving gaps in understanding how LLMs function in dynamic, teacher-led K-12 settings. Earlier AI tools provided generic feedback, but LLMs offer adaptive, context-aware scaffolding that could address these limitations.

Solution

• Proposed approach: WriteAid, an LLM-based writing support tool designed as a technology probe to explore real-time classroom interactions and scaffold English writing tasks.
• Novelty:
  1. Creation of a large-scale dataset (14,863 utterances) from real-time K-12 classrooms, revealing divergent usage patterns across proficiency levels.
  2. Identification of limitations in LLM scaffolding, such as fostering dependency and demotivation among lower-proficiency students.
  3. Insights into classroom dynamics, including equity challenges and the impact on teacher-student interactions.
• Procedure and key techniques:
  • WriteAid provided step-by-step guidance via a chat-based interface, offering features like sentence construction, grammar revision, and vocabulary explanations.
  • Scaffolding strategies were implemented based on Van de Pol et al.'s framework, with prompts designed to encourage active engagement.
  • Deployment included six weeks of classroom use, qualitative coding of student queries, and naturalistic observation of classroom behaviors.

Results

• Concrete findings:
  • Students sent 14,863 messages, with high-performing students engaging more efficiently and focusing on higher-order tasks.
  • 48.9% of LLM-assisted sentences were incorporated into final essays, with 82.3% grammatically correct.
  • Lower-proficiency students retained fewer LLM-generated sentences and exhibited frustration with step-by-step scaffolding.
• Advantage over baselines: WriteAid improved grammatical accuracy and task completion rates, particularly for lower-proficiency students, but introduced challenges like dependency and reduced peer learning.
• Experiments / evaluation:
  • Participants: 157 eighth-grade students, with data from 133 consented participants.
  • Methods: Interaction logs, qualitative coding of queries, and written assessments graded using an analytic rubric.
  • Metrics: Engagement levels, query types, grammatical accuracy, and retention of LLM-assisted sentences.
• Limitations and future work:
  • Findings may not generalize to other age groups, educational settings, or cultural contexts.
  • Analysis relied on a subset of interaction data (9.3%), potentially missing nuanced patterns.
  • Future research should integrate student perspectives and explore cross-lingual scaffolding strategies.

Summary

This study investigated the use of WriteAid, an LLM-based writing support tool, in real-time K-12 EFL classrooms. While WriteAid improved grammatical accuracy and task completion rates, its step-by-step scaffolding demotivated lower-proficiency students and fostered reliance on the system. Classroom observations revealed reduced peer learning and challenges in teacher-student interactions, with struggling students’ difficulties masked by AI-polished outputs. These findings highlight the need for proficiency-aware scaffolding, clearer teacher-AI role definitions, and systems that promote equitable participation and collaboration. Future implementations should address these challenges to enhance the effectiveness and inclusivity of LLMs in education.