Scientific and Fantastical: Creating Immersive, Culturally Relevant Learning Experiences with Augmented Reality and Large Language Models

Title of the Paper

Scientific and Fantastical: Creating Immersive, Culturally-Relevant Learning Experiences with Augmented Reality and Large Language Models

Paper Information

• Subject Area: Technology-Assisted Education, Augmented Reality, Human-Computer Interaction
• Keywords: Education, Learning, Augmented Reality, Narrative Learning, Large Language Models, Children, Cultural Relevance, Environmental Science, Technology Interaction Systems

Research Background and Issues

• Problems or Challenges Identified by the Authors:

  • Stimulating intrinsic motivation for learning in children is a major challenge in the field of education.
  • Although existing research has shown that augmented reality (AR) and narratives can enhance learning, how to integrate AR, narratives, and large language models (LLMs) to design culturally relevant and effective learning environments remains an underexplored area.
  • The effects of learning transfer and cognitive recognition between reality and fantasy require further investigation.

• Significance of the Problem:

  • High levels of intrinsic motivation are essential for better learning outcomes.
  • Balancing cultural relevance with a global perspective to create educational resources suitable for children from diverse cultural backgrounds can promote cultural integration and diversity in education.

• Research Motivation and Related Work:

  • The use of augmented reality technology to create immersive learning environments has been proven to have positive effects on education.
  • Narrative learning enhances engagement and immersion through storytelling while fostering creativity and critical thinking.
  • Large language models can provide learners with instant, interactive, and personalized feedback, but their application in education is still in its early stages of exploration.

Solution

• Proposed Method or Solution:

  • The authors developed a mobile application called “Moon Story,” which combines augmented reality, narrative learning, and large language models to create an immersive, interactive, culturally relevant learning experience based on Chinese mythology.
  • In “Moon Story,” children embark on an adventure to learn about traditional culture as well as real-world environmental and astronomical knowledge.

• Innovative Aspects:

  • Combining augmented reality with real physical markers to enhance understanding of astronomical distances and scales.
  • Using large language models to bring virtual characters to life, enabling users to interact dynamically with narrative characters.
  • Incorporating elements of Chinese mythology (e.g., the stories of Chang’e and Hou Yi) to create a culturally immersive experience, further fostering cultural identity and interest in learning.

• Implementation Steps and Key Technologies:

  1. Content Design: Learning objectives were developed based on the curriculum and Bloom’s taxonomy, covering scientific knowledge such as ecosystems, climate change, and the solar system.
  2. Narrative Design: The myth of Chang’e was adapted to connect mythological elements with modern topics like climate change and astronomy.
  3. AR Scene Creation: Seven interactive activities were designed, such as exploring the scale of the solar system and visualizing dynamic climate change effects.
  4. LLM Interaction: GPT-4 was used to create character interactions, enabling personalized feedback, instant responses, and reflective support during letter-writing activities.

Research Outcomes

• Specific Results:

  • Children demonstrated significant knowledge improvement after completing the course and exhibited high motivation for learning science.
  • The LLM-enhanced version effectively addressed potential issues of low engagement during learning tasks.
  • Quantitative analysis revealed statistically significant improvements in learning outcomes and confirmed medium effect sizes for knowledge retention.
  • Qualitative analysis indicated that the Chinese cultural narrative elements had a positive impact on all participants, with children from diverse cultural backgrounds showing high levels of engagement.

• Advantages Compared to Existing Solutions:

  • Created a highly culturally relevant learning environment, expanding the application scenarios of AR and LLM in education.
  • Combined environmental science and cultural education within a cohesive narrative framework, enriching the immersive educational experience.

• Experimental or Evaluation Results:

  • Learning assessments included quantitative tests (knowledge quizzes) and qualitative feedback (user experience surveys).
  • The LLM version outperformed in enhancing the quality of student reflections and the relevance of content while showing potential to mitigate declines in engagement.
  • Participants responded positively to the system’s immersive experience and the natural environment activities that encouraged autonomous exploration.

• Limitations and Future Directions:

  • Limitations:
    • The study required supervised implementation, which does not fully reflect a self-directed learning model.
    • Despite meticulous engineering and educational design, LLMs still face challenges in adapting to children’s cognitive levels, such as using long sentences and complex grammar.
    • AR activities rely on specific physical markers, which may limit widespread adoption.
  • Future Directions:
    • Develop more flexible grammar models in LLMs tailored to children’s language abilities, integrating concise communication with effective educational functionality.
    • Explore ways to dynamically adjust narrative content for diverse cultural contexts to achieve broader audience coverage.
    • Expand AR and LLM application scenarios and validate their knowledge transfer effects through long-term testing.

Conclusion

This study demonstrates the potential and challenges of integrating augmented reality technology with large language models to create novel interactive learning experiences through culturally relevant narrative design. The “Moon Story” system effectively combines ancient mythology with modern technology, revitalizing traditional educational methods and providing a reference path for the development of future global and multicultural education.