When XR and AI Meet - A Scoping Review on Extended Reality and Artificial Intelligence

Literature Title

When XR and AI Meet - A Scoping Review on Extended Reality and Artificial Intelligence

Literature Information

• Research Area: The integration of Extended Reality (XR) and Artificial Intelligence (AI)
• Keywords: Extended Reality, Virtual Reality, Augmented Reality, Artificial Intelligence, Machine Learning, Interaction Technology, User Studies, Datasets, Virtual Agents, Generative Models

Research Background and Issues

• Issues and Challenges:

  • Research in the intersection of XR and AI is a rapidly growing hotspot, but the main research topics and advantages of their integration remain unclear.
  • Current studies on AI applications in XR (e.g., vestibular rendering, object tracking) and XR supporting AI (e.g., neural network visualization) are emerging, yet systematic reviews and organized research directions in the field are still lacking.

• Significance:

  • The advancement of XR technologies and AI methods can drive profound research in areas such as interaction understanding, environment generation, and virtual assistants.
  • Identifying current research topics and directions can lay the groundwork for more impactful future outcomes.

• Research Motivation and Related Work:

  • Existing reviews often focus on specific applications (e.g., surgical simulation, medical education) rather than comprehensively addressing fundamental research issues in XR and AI.
  • For example, while there are fragmented studies on intelligent virtual agents and deep learning-enhanced augmented reality (AR), the field lacks cohesive integration.

Solution

• Methods and Solutions:

  • A scoping review methodology was employed to analyze 2,619 papers, ultimately refining 311 articles published between 2017 and 2021.
  • A 26-code framework was developed to evaluate research directions, contribution types, algorithms, tools, datasets, and other aspects.

• Innovations:

  • Proposed a research classification framework with five main themes:
    1. Using AI to create XR worlds (e.g., environment generation, virtual avatars, virtual agents).
    2. Using AI to understand users in XR scenarios (e.g., user feature prediction, physiological performance estimation in XR).
    3. Using AI to enhance XR interactions (e.g., gesture recognition, redirected walking techniques).
    4. Interaction with intelligent virtual agents.
    5. Using XR to support the improvement of AI methods.
  • Provided a list of commonly used datasets, software, algorithm networks, and evaluation mechanisms in the research.

• Implementation Steps and Core Technologies:

  1. Systematic literature retrieval using search keywords constructed from AI and XR-related terms.
  2. Multi-layer filtering and annotation to summarize 311 research outcomes.
  3. Extracted data categorized into four analytical dimensions: technical applications, innovative interactions, user evaluations, and analytical frameworks.

Research Results

• Specific Outcomes:

  • Summarized the current state of the intersection between XR and AI, extracting five core research themes that encompass frontier issues and relevant implementation technologies.
  • Provided a comprehensive list of existing tools and methods across diverse fields (e.g., user prediction to reduce VR sickness, multimodal interaction devices) (see Appendix for details).
  • Introduced 13 research opportunities, including future potential in synthetic content generation, new algorithm optimization, and ethical considerations.

• Comparison with Existing Solutions:

  • For the first time, systematically organized the major trends, issues, and methods in XR and AI research.
  • Compared to existing reviews, offered a broader and deeper research perspective on XR and AI applications, user evaluations, and social impacts.

• Experimental or Evaluation Results:

  • Data showed that in 77% of the papers, AI in XR aimed at real-time applications; 52% addressed real-time deployment issues.
  • Training data for many models predominantly focused on user groups from "Western, highly educated, industrialized backgrounds," highlighting deficiencies in diversity and generalizability.

• Limitations and Future Directions:

  • Limited to papers from recent years (2017-2021), leaving some cutting-edge developments uncovered.
  • Proposed future directions for exploration include but are not limited to: applications of AI technologies in virtual world synthesis, multi-task general models, deeper expansion in AR, user privacy protection, and ethical scrutiny.