Autonomous is Not Enough: Designing Multisensory Mid-Air Gestures for Vehicle Interactions Among People with Visual Impairments

Document Title

Autonomous is Not Enough: Designing Multisensory Mid-Air Gestures for Vehicle Interactions Among People with Visual Impairments

Document Information

• Subject Area: Accessibility design for autonomous vehicles and human-computer interaction
• Keywords: Autonomous vehicles, accessibility design, interfaces for blind and visually impaired (BVI) individuals, spatial audio, gestures, situational awareness

Research Background and Issues

• Issues or Challenges:

  • Fully autonomous vehicles (FAVs) have the potential to significantly enhance the independence and mobility of blind and visually impaired (BVI) individuals, but current FAV designs lack comprehensive accessibility support for this group.
  • In many driving scenarios, users need to exert some level of control over the vehicle (e.g., changing routes or emergency stops), yet there are no interaction mechanisms tailored to this demographic.
  • Situational awareness (understanding information about the vehicle and its surrounding environment) is critical for users to perceive vehicle behavior and make decisions, but existing systems lack multisensory support.

• Research Importance:

  • Globally, approximately 300 million people live with visual impairments. Designing accessible interfaces for this population can significantly improve their mobility and quality of life.
  • Enhancing FAV compatibility and support for special populations is crucial for advancing technology equity and widespread adoption.

• Research Motivation and Related Work:

  • Existing studies show that BVI users have a strong need for control and situational awareness in autonomous vehicles, but current solutions often rely on single modalities (e.g., voice or haptics), lacking multimodal support.
  • Gestures, as a non-visual-dependent interaction method, have recently gained attention in the driving domain, but research on their application for BVI individuals remains limited.
  • Based on user needs assessments and experimental studies, the authors propose this project to design a multisensory interface combining audio, haptics, and gestures.

Solution

• Proposed Solution:

  • Designed and evaluated an innovative multisensory interface that integrates mid-air gestures, multi-point spatial audio, and ultrasonic haptic feedback to provide BVI users with control capabilities for FAVs.
  • Proposed and tested the first inclusive gesture set specifically designed for FAVs.

• Innovations:

  • Developed a multimodal interaction system that does not rely on visual input, addressing the limitations of current automotive touchscreens and voice commands.
  • Combined spatial audio with gestures to convey environmental and vehicle behavior information, enhancing situational awareness.
  • Utilized haptic technology to represent complex road geometry, improving intuitiveness and usability through user interaction.

• Implementation Steps and Key Technologies:

  • Study 1: Conducted a user needs assessment (23 participants) to identify critical tasks and situational information requirements for FAV control.
  • Study 2: Designed and tested an inclusive gesture set with 15 BVI participants to determine the most effective and intuitive gestures.
  • Study 3: Tested an experimental interface combining audio and haptic support with 8 participants to evaluate its ability to enhance understanding of intersection environments and route control.

Research Outcomes

• Specific Outcomes:

  • Developed the first gesture set tailored for BVI users in FAVs.
  • Experimental results demonstrated that the audio-gesture interaction system was easy to use, enabling BVI users to successfully complete route-changing tasks.
  • The study validated the potential of combining audio and haptics to enhance situational awareness, though it identified a learning curve for purely haptic modes.

• Advantages Over Existing Solutions:

  • Compared to existing voice-based single-modal solutions, the multisensory interface addresses limitations in noisy environments or specific scenarios (e.g., sleeping passengers).
  • The non-visual design accommodates users with visual impairments, hearing impairments, or dual sensory loss, broadening accessibility.

• Experimental or Evaluation Results:

  • In the experiment, all participants were able to complete tasks using the audio-gesture interface, with most finding the system easy to use.
  • Integrating haptic feedback into the audio system did not significantly improve navigation speed but was subjectively preferred by users for stronger feedback.

• Limitations and Future Directions:

  • Limitations:

    • The experimental scenarios in this study were relatively simple, simulating only intersection navigation.
    • The haptic module did not achieve optimal intensity, leading to a steep learning curve for some users.
    • Gesture recognition relied on a Wizard-of-Oz approach, without utilizing computer vision recognition technology.

  • Future Directions:

    • Develop systems supporting 360-degree spatial audio within the vehicle cabin to accommodate future FAVs with more flexible seating configurations.
    • Optimize haptic feedback intensity and user learning processes, and explore its application in real-time driving scenarios.
    • Expand the applicability of multisensory systems to all user groups, particularly in nighttime or unfamiliar environments.

Conclusion

This paper presents an innovative multisensory interaction system based on gestures, audio, and haptics, successfully enhancing BVI users' control capabilities and situational awareness in FAVs. This design not only promotes inclusivity in transportation technologies but also provides important insights for mainstream automotive interface design.