Understanding Conversational and Expressive Style in a Multimodal Embodied Conversational Agent

Title of the Paper

Understanding Conversational and Expressive Style in a Multimodal Embodied Conversational Agent

Bibliographic Information

• Authors: Deepali Aneja, Rens Hoegen, Daniel McDuf, Mary Czerwinski
• Research Area: Socially Intelligent Virtual Agents (SIVA), multimodal conversational interface design, and user experience
• Keywords: Embodied agents, social behavior, conversational style, facial expression style, emotional expression, social dialogue, multimodality

Research Background and Problem

• Problem or Challenge: Current embodied conversational agent systems often exhibit monotony and fail to adapt to users' conversational styles and behaviors, lacking immediate and natural interaction experiences.
• Significance: As human-computer interfaces diversify, designing a virtual agent capable of understanding and matching human conversational and expressive styles holds potential value for enhancing user experience and applications in open-domain dialogues.
• Motivation and Related Work: Inspired by previous systems that only supported audio-based conversational style matching, the authors attempt to build a multimodal virtual agent capable of engaging in free-form conversations while dynamically matching users' conversational and expressive styles. Previous studies have shown that conversational style matching improves trust and naturalness in human-computer interaction, but these systems often have limited application domains.

Proposed Solution

• Proposed Approach: The authors developed a multimodal Socially Intelligent Virtual Agent (SIVA) capable of real-time analysis of users' audio and video signals. The system generates style-matching responses based on conversational and emotional states, including language, tone, facial expressions, and head movements.
• Innovations:
  • Combining linguistic conversational styles with non-verbal behaviors such as facial expressions and head gestures for real-time style matching.
  • Transforming multimodal inputs, including expressions and language, into real-time facial expression responses with synchronized lip movements.
  • Introducing a style-matching framework that integrates users' conversational styles (High Consideration HC or High Involvement HI) and facial expression sets to generate natural and credible interactive responses.
• Implementation Steps and Key Technologies:
  • At the conversational level, SIVA uses Microsoft Speech API for audio input analysis and deep learning to generate context-sensitive dialogues.
  • The conversational style manager captures user styles by extracting tonal variations, speaking speed, and linguistic features (e.g., pronoun usage).
  • At the visual level, facial recognition algorithms analyze users' expressions and head movements, while emotion APIs generate emotional expressions, and action units synthesize facial expressions.
  • Real-time lip synchronization and multimodal output responses are implemented.

Research Outcomes

• Specific Results:
  • SIVA demonstrated higher emotional responsiveness and non-verbal behavior under multimodal style-matching conditions, and participants rated it as more credible and expressive compared to the control group (without style matching).
  • Conversational style matching significantly improved "High Involvement HI" users' ratings of the agent's vitality and human-like performance.
  • The system successfully handled open-ended conversational tasks, such as planning events and discussing vacations.
• Advantages and Comparisons:
  • Compared to existing monotonous or domain-specific agents, SIVA is more natural and supports open-domain conversations.
  • Effectively combines conversational style and emotional expression, enhancing adaptability for diverse user groups.
• Experimental or Evaluation Results:
  • Style-matched SIVA showed significantly positive interactions with "High Involvement HI" users, but for "High Consideration HC" users, style matching might have adverse effects, indicating the need for further optimization for personalized design.
  • User feedback revealed that the naturalness of current facial expression imitation is insufficient, with some participants feeling it was "uncomfortable" or "forced."
• Limitations and Future Directions:
  • A dialogue delay of approximately 2 seconds affects the natural flow of interaction.
  • Facial expression imitation may lead to perceptions of being "unnatural" or "unapproachable," requiring expansion to more complex response mechanisms.
  • The current framework combines conversational style and expressive style into a single condition; future research should separate these conditions to explore the driving effects of each feature.
  • Future work could enhance dialogue generation models trained on larger datasets and incorporate non-verbal interaction forms such as body language.