Enhancing XR Audio Realism via Multimodal Scene-Aware Acoustic Rendering

DOI

In Extended Reality (XR), rendering sound that accurately simulates real-world acoustics is pivotal in creating lifelike and believable virtual experiences. However, existing XR spatial audio rendering methods often struggle with real-time adaptation to diverse physical scenes, causing mismatches between visual and auditory cues. This sensory conflict can induce cognitive dissonance, disrupting user immersion. Drawing from domain knowledge and prior works, we explore three key scene-based audio realism enhancement strategies in the design space: 1) room geometry approximation, 2) material segmentation, and 3) semantic-driven acoustic parameter estimation. We introduce Samosa, a novel on-device system designed to render spatially accurate sound for XR by dynamically adapting to its physical environment. Leveraging multimodal Room Impulse Response (RIR) synthesis, Samosa estimates scene acoustic properties—informed by room geometry, surface materials, and acoustic context—and subsequently renders highly realistic acoustics suitable for diverse settings. We validate our system through technical evaluation using acoustic metrics for RIR synthesis across various room configurations and sound types, alongside a human expert evaluation (N=12). Evaluation results demonstrate Samosa feasibility and efficacy in enhancing XR audio realism.