SoundWeAR: Co-Designing AR Sound Cues to Support Outdoor Awareness for DHH Individuals

Paper Title

SoundWeAR: Co-Designing AR Sound Cues to Support Outdoor Awareness for DHH Individuals

Publication Info

• Topic area: Augmented Reality (AR) solutions for enhancing outdoor sound awareness for Deaf and Hard of Hearing (DHH) individuals.
• Keywords: AR glasses, sound visualization, DHH individuals, situational awareness, participatory design, sound cues, outdoor environments, accessibility, sound awareness technologies, user-centered design.

Background and Problem

• Problem / challenge: DHH individuals face challenges in outdoor environments due to limited access to auditory cues, which impacts their situational awareness and safety. Existing AR solutions focus primarily on speech-related information and lack comprehensive strategies for visualizing complex environmental sounds.
• Significance: Enhancing outdoor sound awareness for DHH individuals can improve their safety, confidence, and social engagement in dynamic environments.
• Motivation and related work: Prior research has explored sound awareness technologies (e.g., mobile apps, wearables, AR glasses) and sound visualization in AR, but these efforts have mostly focused on speech transcription or localization. There is a gap in understanding how DHH individuals perceive and interpret environmental sound visualizations in outdoor contexts.

Solution

• Proposed approach: The study introduces an AR-based system for visualizing environmental sounds tailored to the needs of DHH individuals. The system was co-designed through participatory workshops, interviews, and prototype evaluations.
• Novelty:
  1. Identification of three design dimensions (Sounds of Interest, Sound Information, Visualization Preferences) and nine design directions for AR sound visualization.
  2. Development of a prototype AR system that translates environmental sounds into visual cues, validated through real-world evaluations.
  3. Design recommendations for AR-based accessibility systems that prioritize safety, usability, and context-dependent customization.
• Procedure and key techniques:
  1. Conducted four participatory design workshops with DHH individuals, social workers, and designers to identify sound awareness needs and generate visualization ideas.
  2. Synthesized findings into nine design directions and 27 design options.
  3. Validated design options through semi-structured interviews with 15 DHH participants to select preferred visualizations.
  4. Developed and evaluated a prototype with eight DHH participants in an outdoor walking scenario, using predefined sound events to simulate real-world conditions.

Results

• Concrete findings:
  • Participants reported increased situational awareness, safety, and confidence when using the AR system.
  • Preferred visualizations included color-coded icons for sound identity, red border alerts for urgency, hybrid views for location, and bar indicators for loudness.
  • Limiting notifications to three simultaneous sounds reduced cognitive load.
• Advantage over baselines:
  • The system extends AR applications beyond speech-focused tasks to include dynamic environmental sound awareness.
  • Combines multiple sound characteristics (e.g., urgency, location, identity) into intuitive visual representations.
• Experiments / evaluation:
  • Workshops: 12 participants (DHH individuals, social workers, designers).
  • Interviews: 15 DHH participants validated 27 design options.
  • Prototype evaluation: 8 DHH participants tested the system in a controlled outdoor walking scenario with 15 predefined sounds.
• Limitations and future work:
  • Hardware limitations: AR glasses were bulky and interfered with hearing aids.
  • Evaluation limitations: Simulated sound events and controlled campus setting may not fully represent real-world complexity.
  • Participant diversity: Limited sample size and hearing condition variability. Future studies should involve larger, more diverse samples and real-time sound detection.

Summary

This study presents SoundWeAR, an AR-based system designed to enhance outdoor sound awareness for DHH individuals. Through participatory design workshops, interviews, and prototype evaluations, the researchers identified key design dimensions and validated user-preferred visualizations for sound cues. The system improves situational awareness, safety, and confidence by visualizing critical sound characteristics like urgency, location, and identity. While hardware and evaluation constraints remain, the findings provide actionable design recommendations for AR accessibility systems, emphasizing context-dependent customization and reduced cognitive load.