Virtual Worlds Beyond Sight: Designing and Evaluating an Audio-Haptic System for Non-Visual VR Exploration

Research Background and Issues

• Identified Problems or Challenges: The authors highlight the insufficient support for interaction and navigation for blind users in existing VR research. This is due to VR's heavy reliance on visual feedback, the lack of interaction tools similar to commonly used white canes, and the physical space limitations in exploring virtual environments (VEs). Additionally, the challenges of non-visual exploration in large-scale and complex VEs, such as urban landscapes, remain inadequately addressed.
• Importance of the Issue: Approximately 250 million visually impaired individuals are unable to fully access VR systems, which restricts the democratization of technology. Research on non-visual VR exploration can significantly enhance digital inclusivity for these individuals and support the development of accessible technologies that reflect real-world scenarios.
• Research Motivation and Related Work: Existing studies have developed some audio games and tactile feedback devices, but these technologies still fall short in terms of scale, complexity, and realistic scenario simulation. Particularly in non-visual navigation, current solutions often fail to adequately support the exploration of complex spaces while neglecting the dynamic integration of auditory and tactile interactions.

Solution

• Proposed Solution: The authors designed a non-visual interactive VR system comprising three key components: white cane simulation (tactile feedback), an omnidirectional slip device (for in-place walking navigation), and a generative audio engine based on spatial geometry rendering.
• Innovations:
  1. Integration of physical and virtual white cane interaction design to provide users with precise tactile feedback, simulating various ground textures and common surface materials.
  2. Development of a virtual audio engine with realistic spatial audio properties to help users understand environmental spatial conditions, such as sound diffraction, reflection, and occlusion.
  3. Expansion of the scale and complexity of urban environment simulations, enabling users to explore intricate urban streetscapes and dynamic scenes within a safe VR environment.
• Implementation Steps and Key Technologies: The system was developed using the Unity engine and integrated with optical motion capture and audio feedback systems. Tactile feedback for the white cane was implemented using the vibration motor of the Oculus Quest Pro, while in-place walking navigation was supported by the slip device providing body displacement data. For audio, the Audiokinetic Wwise engine was employed to generate high-precision spatial-temporal audio. Additionally, the study validated the effectiveness of the system in non-visual experiences through training and testing.

Research Outcomes

• Specific Outcomes:
  1. Developed a detailed white cane tactile feedback device capable of simulating various ground textures commonly found in urban environments.
  2. Designed a complex urban virtual environment featuring dynamic traffic vehicles, interactive non-player characters (NPCs), and natural scenes.
  3. Successfully demonstrated that users could navigate large virtual spaces under non-visual conditions using tactile and audio feedback with minimal training.
• Advantages Compared to Existing Solutions:
  1. The system overcame physical space limitations by supporting large-scale dynamic virtual environment exploration through in-place walking technology and the omnidirectional slip device.
  2. The integration of tactile and audio feedback enabled users to interact more naturally with the environment, improving task completion rates and accuracy.
• Experimental or Evaluation Results:
  • In three tasks (surface recognition, crossing the street, and object search), participants successfully completed tasks under non-visual conditions with high precision.
  • Even compared to visual-assisted conditions, participants exhibited behavior patterns and path choices in navigating complex environments that closely resembled real-world scenarios.
• Limitations and Future Directions:
  • Limitations: The current design does not fully reflect the interaction needs of real visually impaired users (e.g., white cane usage techniques), and the experiments did not directly involve visually impaired participants. The slip device also presents challenges in terms of learning difficulty and fatigue during prolonged use.
  • Future Directions: The authors propose collaborating with professional organizations to involve visually impaired users in system design and testing. Additionally, the system could further enhance the precision of tactile feedback devices, improve foot feedback technology, and extend its application to real-world adaptive training scenarios for visually impaired individuals while evaluating long-term skill transfer effects.

Through this study, the authors have laid the foundation for designing inclusive VR environments, expanded the possibilities for visually impaired individuals to explore the digital domain, and introduced innovative audio-tactile integration technologies.