Designing Scaffolding Cards to Facilitate LLM-Based Socratic Instruction: An Exploratory Study of Response Strategies to Support Learning

Paper Title

Designing Scaffolding Cards to Facilitate LLM-Based Socratic Instruction: An Exploratory Study of Response Strategies to Support Learning

Publication Info

• Topic area: Enhancing learner-LLM interaction in Socratic instruction using scaffolding strategies.
• Keywords: Scaffolding Cards, Socratic instruction, LLM-based tutoring, critical thinking, cognitive load, learning satisfaction, response strategies, personalized scaffolding, learner agency.

Background and Problem

• Problem / challenge: Learners often produce superficial responses to open-ended questions in LLM-based Socratic instruction, undermining critical thinking and learning outcomes. Prior work has focused on LLMs' question design but neglected learners' response strategies.
• Significance: Addressing this issue is crucial to fostering deeper engagement, critical thinking, and learner autonomy in LLM-driven educational environments.
• Motivation and related work: The Socratic method promotes reflective dialogue, but its success depends on learners' ability to construct meaningful responses. Existing research emphasizes tutor-driven scaffolding, leaving a gap in understanding how to support learners' response formulation.

Solution

• Proposed approach: Scaffolding Cards—18 sentence-framing response strategies organized into three clusters (Structuring Expression, Constructing Reasoning, Activating Thinking) to guide learners in constructing thoughtful answers during LLM-based Socratic instruction.
• Novelty:
  1. Introduced a response-centered scaffolding approach targeting learners' immediate response states.
  2. Developed agency-preserving scaffolds that provide syntactic support without dictating content.
  3. Designed function-transparent scaffolds that explicitly convey their purpose and application.
  4. Evaluated the cards' impact on critical thinking, cognitive load, and learning satisfaction through a mixed-methods study.
• Procedure and key techniques:
  1. Conducted a formative study (N=20) to derive response strategies from dialogue logs and interviews.
  2. Designed Scaffolding Cards based on Bloom's revised taxonomy and learner feedback.
  3. Evaluated the cards in a mixed-methods user study (N=34) using a within-subjects design, comparing experimental and control conditions across critical thinking, cognitive load, and learning satisfaction metrics.

Results

• Concrete findings:
  • Critical thinking scores were significantly higher in the experimental condition (t33 = 6.675, p < .05).
  • Extraneous cognitive load was significantly reduced (t33 = -2.753, p < .05), while Germane Load showed a marginal increase (t33 = 1.904, p = .066).
  • Learning satisfaction was significantly higher in the experimental condition (t33 = 3.067, p < .05).
• Advantage over baselines: Scaffolding Cards improved analytical breadth, reasoning depth, and significance focus in learner responses compared to the no-scaffold control condition.
• Experiments / evaluation:
  • Participants used the cards extensively, with strategies like Simplifying, Questioning, and Speculating being the most frequent.
  • Interviews revealed purposeful strategy use shaped by dialogue context, knowledge proficiency, and individual preferences.
  • Questionnaire results confirmed reduced cognitive load and increased satisfaction with the learning experience.
• Limitations and future work:
  • Lack of personalized recommendations for card selection.
  • Potential risk of overreliance on scaffolds in long-term use.
  • Need to explore scaffolding effects across diverse learning contexts and stages.

Summary

This study developed and evaluated Scaffolding Cards to address the challenge of superficial learner responses in LLM-based Socratic instruction. The cards, organized into three clusters, provided structured yet flexible support for constructing meaningful answers. A mixed-methods evaluation demonstrated significant improvements in critical thinking, reduced extraneous cognitive load, and higher learning satisfaction compared to a no-scaffold baseline. The findings highlight the potential of state-aware, agency-preserving, and function-transparent scaffolding to enhance learner-LLM interaction, with implications for designing responsive and personalized educational tools.