Exploring Learners' Expectations and Engagement When Collaborating with Constructively Controversial Peer Agents

Paper Title

Exploring Learners' Expectations and Engagement When Collaborating with Constructively Controversial Peer Agents

Publication Info

• Topic area: Human-AI collaboration in asynchronous online learning environments
• Keywords: Constructive Controversy, LLM-based peer agents, collaborative learning, learner engagement, human-AI interaction, argumentation tasks, design transparency, regulated behaviors, unregulated behaviors, educational AI

Background and Problem

• Problem / challenge: Asynchronous online courses lack the real-time collaborative interactions that enhance learning outcomes, and while LLM-based peer agents can emulate human peers, their tendency toward conformity limits their ability to foster diverse perspectives and constructive controversy.
• Significance: Addressing this gap is crucial for improving isolated learners’ engagement, creativity, and critical thinking in online educational settings.
• Motivation and related work: Prior research highlights the benefits of collaborative learning and Constructive Controversy (CC) theory in enhancing cognitive and emotional engagement. However, applying CC principles to AI agents requires balancing skilled disagreement and rationality while aligning with user expectations. Transparency in agent design further complicates user perceptions and interactions.

Solution

• Proposed approach: Development of LLM-powered peer agents with regulated and unregulated CC behaviors, and exploration of the impact of transparency in agent design on learner engagement and perception.
• Novelty:
  1. Categorization of learners into Efficiency-Driven Learners (EDL) and Curiosity-Driven Learners (CDL) based on values, expectations, and collaborative strategies.
  2. Empirical analysis of how regulated and unregulated CC behaviors affect engagement, sense of agency, and argument quality.
  3. Investigation of the effects of design transparency on learners’ perceptions of agent abilities.
• Procedure and key techniques:
  • Conducted a 2 × 2 mixed factorial experiment with 144 undergraduate participants.
  • Participants interacted with regulated and unregulated peer agents on argumentation tasks, with transparent or opaque design disclosure.
  • Measured engagement, argument quality, and perceptions using task metrics, Likert-scale questionnaires, and qualitative feedback.
  • Developed regulated agents using a moderator module and persona expansion for skilled disagreement and rationality.

Results

• Concrete findings:
  • Regulated agents increased turn counts and task time but did not improve argument quality.
  • Unregulated agents were perceived as "Too Cooperative," while regulated agents were seen as "Too Contradictory."
  • Design transparency reduced learners’ perception of agent abilities across all participants.
• Advantage over baselines:
  • Regulated agents better aligned with CC principles of skilled disagreement for CDL learners.
  • Unregulated agents supported EDL learners’ sense of agency and emotional engagement.
• Experiments / evaluation:
  • Mixed-method study with quantitative and qualitative analyses.
  • Tasks included collaborative argumentation on debate topics, with metrics for engagement, argument strength, and variance.
  • Statistical models and G-eval techniques validated differences in agent behaviors and learner perceptions.
• Limitations and future work:
  • Limited generalizability due to participant demographics (young, experienced with LLMs).
  • Debate topics were generic and not domain-specific.
  • Controlled study design lacked longitudinal insights into sustained learner-agent interactions.

Summary

This study explored the design of LLM-powered peer agents with Constructive Controversy behaviors to enhance collaborative learning in asynchronous online environments. By categorizing learners into Efficiency-Driven and Curiosity-Driven types, it demonstrated how regulated and unregulated agent behaviors interact with learner values and expectations to influence engagement, sense of agency, and argument quality. Transparency in agent design negatively impacted learners’ perception of agent abilities. These findings provide foundational insights for tailoring AI-driven collaborative tools to diverse learner needs, with implications for broader educational applications. Future research should focus on long-term studies and domain-specific tasks to refine these interventions.