Auditorily Embodied Conversational Agents: Effects of Spatialization and Situated Audio Cues on Presence and Social Perception

Paper Title

Auditorily Embodied Conversational Agents: Effects of Spatialization and Situated Audio Cues on Presence and Social Perception

Publication Info

• Topic area: Auditory embodiment of conversational agents and its impact on user perception and interaction.
• Keywords: auditory embodiment, conversational agents, spatialization, Foley audio, social presence, co-presence, human-computer interaction, auditory interfaces, social perception, user experience.

Background and Problem

• Problem / challenge: Existing research on conversational agents primarily focuses on visual embodiment, which is not always feasible in audio-only interactions (e.g., headphones or display-less devices). The potential of auditory embodiment to convey presence and influence user perceptions remains underexplored.
• Significance: Understanding auditory embodiment could enable richer, more socially engaging interactions with conversational agents in audio-only contexts, such as wearables or smart glasses.
• Motivation and related work: Prior work has shown that embodiment enhances social presence and trust in agents, but it has focused on visual modalities. Auditory cues like spatialized voice and Foley sounds (incidental environmental sounds) are known to influence perception in other domains (e.g., film, human interaction), but their application to conversational agents is limited.

Solution

• Proposed approach: The study explores auditory embodiment by introducing spatialized voice and situated Foley audio to conversational agents, aiming to convey bodily presence and evaluate its effects on user perception.
• Novelty:
  1. Investigates auditory embodiment as an alternative to visual representation for conversational agents.
  2. Examines the combined and individual effects of spatialized audio and Foley on social presence and perception.
  3. Provides empirical insights into the trade-offs of auditory embodiment in terms of co-presence, attention, and social impressions.
• Procedure and key techniques:
  • Developed a Unity-based system with spatial audio rendering (using Meta XR Audio SDK) and Foley sound playback.
  • Conducted a 2×2 within-subjects study (n=24) with four conditions: monaural/no Foley, monaural/Foley, spatialized/no Foley, spatialized/Foley.
  • Measured social presence (e.g., co-presence, attention), social impressions (e.g., likeability, attraction), and behavioral engagement (e.g., words exchanged, head rotation).

Results

• Concrete findings:
  • Spatialization and Foley increased feelings of co-presence (e.g., spatialized audio improved co-presence ratings by 16%, Foley by 23%).
  • Foley reduced perceptions of attention, message understanding, likeability, and social attraction (e.g., likeability ratings dropped by 10% with Foley).
  • Foley also reduced conversational engagement, with 20 fewer words exchanged on average.
• Advantage over baselines:
  • Spatialization alone enhanced co-presence without negatively affecting attention or social impressions.
  • Foley combined with spatialization yielded the highest co-presence ratings but introduced trade-offs in perceived attentiveness and conversational engagement.
• Experiments / evaluation:
  • Participants engaged in casual 3-minute conversations with the agent under four conditions.
  • Social presence was measured using the Networked Minds Social Presence Inventory (NMSPI), while verbal and nonverbal behaviors were recorded.
  • Qualitative interviews provided additional insights into user perceptions and preferences.
• Limitations and future work:
  • Limited to casual conversations in a controlled lab setting; results may not generalize to task-oriented or real-world contexts.
  • Foley and movement behaviors were pre-scripted, leading to potential misalignment with conversational norms.
  • Future work should explore dynamic, context-aware Foley and spatialization in diverse environments and tasks.

Summary

This study investigates auditory embodiment in conversational agents through spatialized voice and Foley audio. Results show that while both techniques enhance co-presence, Foley negatively impacts attention, message understanding, and social impressions. Spatialization alone offers a balanced improvement in presence without adverse effects. These findings highlight the potential of auditory embodiment to enrich interactions with agents in audio-only contexts, while emphasizing the need to align auditory cues with social norms and user expectations. Future work should address dynamic, context-aware implementations and explore broader applications.