Spatialized Audio and Hybrid Video Conferencing: Where Should Voices be Positioned for People in the Room and Remote Headset Users?

Document Title

Spatialized Audio and Hybrid Video Conferencing: Where Should Voices be Positioned for People in the Room and Remote Headset Users?

Document Information

• Subject Area: Application of spatial audio in hybrid video conferencing and user experience research
• Keywords: Spatial audio, video conferencing, hybrid meetings, sound localization, remote participants, in-room users, auditory perception

Research Background and Issues

• Problems and Challenges:

  • In hybrid video conferencing, how to optimize sound positioning between in-room participants using speakers and remote participants using headsets to enhance user experience.
  • Spatial alignment of video and audio streams poses unique technical challenges in remote and face-to-face scenarios.
  • Limitations of the "sweet spot" in spatial audio for speakers and potential mismatches in audio and video width for remote users.

• Significance:

  • The widespread use of video conferencing post-COVID-19 pandemic underscores the urgency of improving user experience, including enhancing naturalness and reducing "Zoom fatigue."
  • Spatial audio technology has significant potential to improve voice distinction, reduce cognitive load, and enhance conversational fluidity among meeting participants.

• Research Motivation and Related Work:

  • Previous studies have shown that spatial audio aids in sound differentiation and speech comprehension in audio conferencing, but its effects in video conferencing remain unclear.
  • Current video conferencing audio technology predominantly uses mono sound, awaiting user experience studies to validate the value of spatial audio.
  • Designing audio layouts that balance the experiences of in-room and remote users remains an unresolved issue.

Solutions

• Methods and Solutions:

  • Two independent studies were proposed: exploring spatial audio positioning for in-room participants using speakers and investigating audio width for remote participants using headsets.
  • In-room experiments tested four audio conditions: mono, stereo, binaural stereo, and enhanced multi-channel spatial layouts.
  • Remote experiments tested different levels of sound separation angles (±5°, ±15°, ±30°) in two-person and four-person video conferencing scenarios.

• Innovations:

  • Combined in-room multi-channel spatial audio and binaural rendering for headsets to study the effects of different audio layouts on direct and indirect communication among multiple participants.
  • Explored the relationship between audio width and visual alignment, gaining insights into user perceptions of mismatched audio-visual widths.
  • Integrated user feedback with statistical analysis to guide audio design for hybrid meetings.

• Implementation Steps and Key Technologies:

  • Simulated conversational videos and segmented audio streams were used for experiments in both room environments and remote participation settings.
  • In-room experiments utilized Dolby Atmos systems to design different layouts; remote user experiments employed standard headphones for binaural audio rendering.
  • Tested users' accuracy and confidence in identifying audio streams, as well as their perceptions of audio-video positional alignment.

Research Findings

• Specific Results:

  • In-room participants showed the highest preference for enhanced multi-channel layouts, significantly outperforming mono and stereo layouts.
  • Remote users appreciated spatial audio rendered through headphones, though their preferred audio width was generally narrower than that of in-room participants.
  • In two-person video scenarios, users preferred narrower audio widths, while in four-person scenarios, users accepted wider sound distributions.

• Advantages Over Existing Solutions:

  • Provided more efficient audio layout design recommendations, enhancing sound separation and intelligibility.
  • Optimized the experiences of both in-room and remote users, avoiding negative impacts from overly wide or narrow designs.
  • Offered guidance on dynamically adjusting audio layouts based on the number of participants to comprehensively improve user experience in hybrid meetings.

• Experimental or Evaluation Results:

  • In-room speaker users were significantly faster at identifying sound sources and experienced smoother speech comprehension.
  • Remote users, despite lacking head tracking functionality, expressed satisfaction with narrower sound separation, reducing auditory cognitive load.
  • Investigated user demands for audio-visual alignment, finding that perceptions of width depend on the number of participants in the audio scenario.

• Limitations and Future Directions:

  • The current study did not fully compare the two experimental settings, such as cross-environment evaluations of the two user groups.
  • Did not explore the effects of audio layouts in large-scale meeting scenarios or their impact on cognitive load.
  • Future research is recommended to explore automated technologies for dynamically adjusting layout width and to incorporate head tracking devices to optimize remote user experiences.

Summary and Design Recommendations

• Spatial audio provides significant benefits for both in-room speaker users and remote headset users, but audio width should be adjusted based on the specific scenario.
• In-room setups should use wider audio layouts to cover the overall sound field, while remote users should maintain narrower widths to avoid sensory fatigue.
• Hybrid meeting system designs should separate in-room and remote user experiences to achieve the optimal overall meeting effect.