Exploring LLM-Powered Role and Action-Switching Pedagogical Agents for History Education in Virtual Reality

Research Background and Issues

• Identified Problems or Challenges: In current VR education, teaching agents (PAs) that support multi-role interaction face the following issues:

  • The complexity of multi-role interaction operations may lead to user fatigue and interruptions in learning.
  • A lack of dynamic adaptability to user input limits personalization and interactivity.
  • Many existing teaching agents rely solely on pre-scripted scenarios, lacking real-time responsiveness.
  • Virtual teaching agents with multi-role functionality (e.g., applications in literature, history, etc.) often fall short in narrative richness and interaction diversity.

• Significance:

  • History education is a critical domain for fostering cultural understanding and interdisciplinary exploration. Recreating historical scenes in VR can enhance students' hands-on engagement and interest in learning.
  • Multi-role teaching agents can improve the effectiveness of learning historical events from different perspectives, enriching narratives to promote deeper memory retention and understanding.

• Research Motivation and Related Work:

  • Inspired by the capabilities of large language models (LLMs) such as GPT-4, the authors aim to integrate LLMs into teaching agents to enable dynamic, multi-role, and multi-action interactions, addressing current issues and enhancing the effectiveness of history education.
  • Previous research has demonstrated the significant potential of VR combined with multi-role teaching agents in education. However, the personalization and diversity of interactions in practical applications still require improvement.

Proposed Solution

• Proposed Method or Solution: A VR prototype was developed, featuring two adaptive modules designed specifically for history education:

  • Adaptive Role-Switching Module: Dynamically switches the teaching agent's identity (e.g., historical figures, literary scholars, or archaeologists) as well as their tone, voice, and appearance based on user input.
  • Adaptive Action-Switching Module: Dynamically adjusts the agent's actions based on model output, enabling behaviors that reflect character traits (e.g., pointing, writing, or performing).

• Innovative Aspects of the Solution:

  1. Utilizes LLMs for dynamic multi-role switching, enabling adaptive representation of different identities in historical memory and perspectives.
  2. Combines role-switching and action-switching modules for the first time, making the teaching agent's behavior more vivid and natural.
  3. The modules provide context-sensitive interactive feedback based on real-time user input, enhancing immersion and learning outcomes.

• Implementation Steps and Key Technologies:

  1. System Design: The foundational structure includes three modules (dialogue, role-switching, and action-switching). The role-switching module analyzes user input through the LLM to match the most suitable role with the usage scenario, while the action-switching module selects appropriate actions based on context analysis.
  2. Virtual Environment Construction: The surrounding environment was built using the Unity engine, teaching agent appearances were generated using ReadyPlayerMe, and clothing and rigging were further customized using Blender.
  3. Experimental Setup: Testing was conducted using Oculus Quest 3 headsets, and user studies were performed to evaluate four experimental groups with or without the role-switching and action-switching modules.

Research Outcomes

• Specific Achievements:

  • Role-Switching:
    • Enhanced users' perception of the agent's credibility and professionalism.
    • Encouraged users to learn history from multiple perspectives, though frequent switching could lead to inconsistent learning experiences.
  • Action-Switching:
    • Significantly improved the sense of social presence, interaction naturalness, and the human-like qualities of the agent.
  • Impact on Learning Motivation and Cognitive Load:
    • Role and action switching had no significant effect on learning motivation or cognitive load (intrinsic, extraneous, and germane cognitive load).

• Advantages:

  • The solution addresses the challenge of unnatural interactions in multi-role teaching through adaptive role-switching and action-switching modules, sparking users' interest in exploring historical contexts.
  • The module design aligns well with real-world user needs, such as more realistic role interactions and efficient navigation guidance.

• Experimental or Evaluation Results:

  • A total of 84 participants took part in the experiment. Quantitative and qualitative data revealed:
    • Role-switching significantly aided the recall of factual knowledge but had limited impact on conceptual knowledge acquisition.
    • Action-switching significantly enhanced the sense of social presence and humanization in learning, though high-frequency role and action changes could introduce cognitive interference.
  • User suggestions included achieving smoother role transitions and optimizing the agent's action logic (e.g., more precise pointing actions).

• Limitations and Future Directions:

  • The research focused solely on history education; future work could expand to more complex application scenarios (e.g., medical simulations or vocational training).
  • The current study involved only single-agent interactions with users; future research should explore the design of teaching experiences involving multi-agent collaboration, such as interactions among multiple historical figures.
  • The effects of action and role switching need further optimization, including consideration of user preferences and adjustments to interaction frequency.

This study provides valuable insights for the design of future LLM-based virtual reality educational tools, demonstrating the potential of incorporating multi-perspective interactions. However, further exploration is needed to balance user cognitive load with interactive experiences.