Empowered XR through Generative AI: Balancing Superpowers and Risks

Paper Title

Empowered XR through Generative AI: Balancing Superpowers and Risks

Publication Info

• Topic area: Integration of generative AI and Extended Reality (XR) to enhance human capabilities and address associated risks.
• Keywords: Generative AI, Extended Reality, Large Language Models, cognitive augmentation, sensory enhancement, environmental manipulation, behavioral risks, user autonomy, ethical transparency, immersive technologies.

Background and Problem

• Problem / challenge: Existing XR technologies face limitations in replicating realistic scenarios, supporting collaboration, and providing detailed contextual understanding. Integrating Large Language Models (LLMs) with XR introduces new capabilities but also raises significant risks related to user autonomy, data security, and ethical transparency.
• Significance: The integration of LLMs with XR has the potential to revolutionize fields like healthcare, education, and professional training by enhancing human cognitive, sensory, and environmental control. However, addressing the associated risks is critical for responsible adoption.
• Motivation and related work: Prior research has explored XR's potential to overcome human sensory and cognitive limitations, and LLMs' ability to enhance contextual understanding. However, the combined use of LLMs and XR remains underexplored, particularly in terms of balancing empowerment with socio-technical risks.

Solution

• Proposed approach: A taxonomy of LLM-enabled XR superpowers and their associated risks, along with design guidelines and a forward research agenda to mitigate these risks.
• Novelty:
  1. Development of a comprehensive taxonomy categorizing LLM-enabled XR superpowers into internal, external, and mind-reading capabilities.
  2. Identification and analysis of risks associated with these superpowers, including user autonomy, data security, and ethical concerns.
  3. Creation of actionable design guidelines to balance empowerment and risk mitigation.
  4. Proposal of a scalable framework for updating the taxonomy as the field evolves.
• Procedure and key techniques:
  • Conducted a systematic literature review of 135 papers using the Grounded Theory Literature Review Methodology.
  • Developed taxonomies for superpowers and risks through iterative coding and refinement.
  • Proposed design guidelines based on the taxonomy and mapped them to existing theories like Distributed Cognition and Cognitive Load Theory.

Results

• Concrete findings:
  • Internal superpowers include cognitive load reduction (23/56), knowledge acquisition (17/56), and memory augmentation (5/56).
  • External superpowers involve creating environments (11/22), modifying environments (10/22), and social interaction enhancement (33/135).
  • Mind-reading superpowers enable gaze-based text entry, body movement interpretation, and semantic prediction.
  • Identified risks include manipulation of thoughts and behavior (e.g., 51/58 for internal superpowers), cross-reality security vulnerabilities (31/75 for external superpowers), and user data transparency issues (17/21 for mind-reading superpowers).
• Advantage over baselines: The taxonomy provides a structured framework for understanding both the capabilities and risks of LLM-enabled XR, addressing gaps in prior research that focused on either XR or LLMs in isolation.
• Experiments / evaluation:
  • Literature review spanned 576 unique papers, narrowed to 135 for in-depth analysis.
  • Inter-rater reliability achieved 92% agreement across coding decisions.
  • Proposed guidelines validated against existing theories and emerging research.
• Limitations and future work:
  • Subjective nature of grounded coding limits generalizability across all XR applications.
  • Lack of longitudinal data on the cognitive impact of LLM-enabled XR.
  • Future work includes empirical validation of guidelines, development of scalable review pipelines, and exploration of long-term user impacts.

Summary

This paper presents a taxonomy of LLM-enabled XR superpowers, categorizing them into internal, external, and mind-reading capabilities, and identifies associated risks such as data security, user autonomy, and ethical concerns. The authors propose design guidelines to mitigate these risks, emphasizing transparency, user control, and recoverability. The study is based on a systematic review of 135 papers and provides a foundation for future research and responsible development of XR technologies. The taxonomy and guidelines aim to balance the empowerment of users with the mitigation of socio-technical risks, ensuring sustainable and ethical adoption of LLM-enabled XR systems.