"Shall We Dig Deeper?": Designing and Evaluating Strategies for LLM Agents to Advance Knowledge Co-Construction in Asynchronous Online Discussions

Paper Title

"'Shall We Dig Deeper?': Designing and Evaluating Strategies for LLM Agents to Advance Knowledge Co-Construction in Asynchronous Online Discussions"

Publication Info

• Topic area: AI-driven facilitation of knowledge co-construction in online discussions.
• Keywords: LLM agents, asynchronous discussions, knowledge co-construction, intervention styles, task-oriented, relationship-oriented, AI facilitation, online platforms, human-AI interaction, collaborative learning.

Background and Problem

• Problem / challenge: Asynchronous online discussions often stagnate in surface-level phases of knowledge co-construction, failing to progress toward deeper synthesis and integration. Existing AI interventions lack mechanisms to orchestrate progression across phases and their impacts on discussion dynamics remain underexplored.
• Significance: Addressing stagnation in online discussions is critical for enhancing collective knowledge-building processes, improving user engagement, and increasing the value of platforms as sustainable repositories of synthesized knowledge.
• Motivation and related work: Prior work has focused on isolated phases of knowledge co-construction or interaction management but lacks a process-oriented approach to advance discussions systematically. Human facilitation styles—task-oriented and relationship-oriented—have shown promise but their application to AI agents in multi-party settings is underexplored.

Solution

• Proposed approach: A process-orchestrated intervention paradigm for LLM-powered agents, employing phase-sensitive strategies to advance knowledge co-construction across four phases: initiation, exploration, negotiation, and co-construction.
• Novelty:
  1. Design of actionable, phase-tailored intervention strategies for AI agents in task-oriented and relationship-oriented styles.
  2. Implementation of an LLM agent capable of monitoring discussion dynamics and applying phase-specific interventions.
  3. Mixed-methods evaluation revealing distinct impacts of intervention styles on knowledge progression, user perceptions, and human-human interaction.
• Procedure and key techniques:
  • Conducted a design workshop with 12 participants to co-design intervention strategies.
  • Developed an LLM agent with components including a Phase Classifier, Frequency Controller, Phase-Sufficiency Evaluator, Style Manager, and Response Generator.
  • Implemented a within-subject study (N=60) with five conditions (four intervention styles and one baseline) to evaluate agent impacts on asynchronous discussions.

Results

• Concrete findings:
  • Agent interventions advanced discussions to deeper phases, increasing the maximum phase reached by 23.8–38.1% compared to baseline.
  • Telling, Selling, and Participating styles achieved Phase 2 sufficiency in 60–80% of threads, compared to 10% in baseline.
  • Participating style received the highest appreciation, while Delegating showed minimal impact.
• Advantage over baselines: Agent-supported threads consistently progressed beyond surface-level phases, strengthened foundational phases, and elicited deeper knowledge co-construction compared to human-only discussions.
• Experiments / evaluation:
  • Mixed-methods study with 60 participants across five experimental conditions.
  • Metrics included thread-level measures (e.g., maximum phase reached, sufficiency criteria) and individual-level perceptions (e.g., depth, effectiveness, social presence).
  • Qualitative thematic analysis of interviews provided insights into user experiences and agent impacts.
• Limitations and future work:
  • Laboratory setting does not fully capture real-world dynamics; future field studies are needed.
  • Limited participant diversity and accessible discussion topics; future work should explore specialized knowledge contexts and diverse demographics.
  • Interface design and potential demand effects warrant further investigation.

Summary

This paper introduces a process-orchestrated intervention paradigm for LLM agents to advance knowledge co-construction in asynchronous online discussions. By employing phase-sensitive strategies derived from a design workshop, the agents effectively facilitated progression across four phases of knowledge co-construction. The study demonstrated that task-oriented (Telling, Selling) and relationship-oriented (Participating) styles each have distinct strengths, with Participating being the most favorably perceived. Findings highlight the promise of adaptive AI agents in fostering deeper and more robust online discourse, while suggesting design considerations for balancing task and relationship orientations, minimizing intrusiveness, and enhancing human-human interaction.