Understanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People

Paper Title

Understanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People

Publication Info

• Topic area: Accessibility in social virtual reality for blind and low vision users.
• Keywords: Virtual reality, accessibility, blind and low vision, AI guide, large language models, social interaction, embodiment, navigation, visual interpretation, assistive technology.

Background and Problem

• Problem / challenge: Current social VR platforms lack accessibility features for blind and low vision (BLV) users, and existing solutions like spatial audio or haptic feedback are inadequate for dynamic social environments.
• Significance: Making social VR accessible to BLV users is critical as these platforms grow in popularity, enabling inclusion and autonomy in virtual spaces.
• Motivation and related work: Previous research has focused on single-user VR accessibility or human guides, which are effective but limited by availability and user independence. AI-driven tools offer potential for scalable, autonomous solutions, but their efficacy has not been empirically studied.

Solution

• Proposed approach: An AI guide powered by a large language model (LLM) designed to assist BLV users in navigating and interpreting social VR environments.
• Novelty:
  1. First empirical study of an AI guide’s use by BLV users in social VR.
  2. Integration of customizable personas (dog, robot, human) for varied guidance experiences.
  3. Insights into BLV users’ behaviors towards AI guides in solo and social contexts.
  4. Design recommendations for future AI-powered accessibility tools.
• Procedure and key techniques:
  • Developed an AI guide with capabilities for visual descriptions, navigation, and spatial audio beacons.
  • Conducted a study with 16 BLV participants performing individual and social tasks in VR parks.
  • Analyzed participants’ interactions, task completion, and feedback through mixed-methods coding and thematic analysis.

Results

• Concrete findings:
  • Guide accuracy: 63.2% across 476 queries; response times averaged 6.1–11.2 seconds depending on query type.
  • Participants successfully used the guide to explore parks and lead tours, though challenges arose with landmark recall and guide errors.
  • Participants rated the guide’s usability (mean 3.2/5), usefulness (3.5/5), joy of use (4.1/5), social comfort (3.7/5), scene understanding (3.6/5), object perception (3.6/5), and navigation (3.1/5).
• Advantage over baselines:
  • Enabled BLV users to navigate and interact in social VR without human assistance, addressing autonomy and scalability limitations of prior solutions.
  • Provided high-level contextual information beyond sensory feedback.
• Experiments / evaluation:
  • Study tasks included solo exploration and group tours in VR parks using two guide personas (dog, robot).
  • Participants’ interactions were categorized by tone (utilitarian, polite, friendly) and request types (navigation, visual description, etc.).
  • Feedback collected via Likert-scale ratings and semi-structured interviews.
• Limitations and future work:
  • Technical limitations: delayed response times, low accuracy due to misinterpreted accents and incomplete queries.
  • Behavioral limitations: participants used simplistic commands, reflecting limited familiarity with LLMs.
  • Future work: longitudinal studies to examine user adaptation, improved guide responsiveness, and exploration of diverse guide embodiments.

Summary

This study presents the first empirical evaluation of an AI guide for BLV users in social VR, demonstrating its effectiveness for navigation, interpretation, and social interaction. Participants treated the guide differently in solo versus social contexts, often role-playing with it and rationalizing its mistakes. Despite usability challenges, the guide enabled participants to explore complex virtual environments and lead tours, highlighting its potential as an assistive tool. Findings inform design recommendations for future AI guides, emphasizing emotional connection, advanced usage, and alignment with user expectations. This work lays a foundation for accessible, adaptive AI tools in VR.