Expressive Auditory Gestures in a Voice-Based Pedagogical Agent

Title of the Paper

Expressive Auditory Gestures in a Voice-Based Pedagogical Agent

Bibliographic Information

• Subject Area: Human-Computer Interaction, Educational Technology, Conversational Agent Design
• Keywords: Education, Voice, Interaction, Emotion, Motivation, Agent, Exclamations, Music

Research Background and Problem

• Identified Challenges:
  1. Conversational agents in the education domain overly emphasize cognitive needs, with limited attention to the impact of emotional factors on learning outcomes.
  2. Existing research primarily focuses on visual expressions (e.g., facial expressions, body language) or vocal tone, while studies on auditory expressions (e.g., exclamations or music) are relatively scarce.
  3. Particularly in educational contexts, the effects of these auditory expressions on learning motivation and human-agent relationships remain unclear.
• Significance:
  1. Emotional factors (e.g., emotions, attitudes) are crucial in the learning process, potentially enhancing motivation, memory, and comprehension.
  2. Optimizing interactions between learners and conversational agents could bring new breakthroughs to educational technology.
• Research Motivation and Related Work: This study explores the role of auditory expressions in a "learning-by-teaching" scenario, examining the effects of two types of auditory expressions (exclamations and short musical gestures) on learners' motivation and human-agent interaction, while expanding existing design concepts for conversational agents.

Solution

• Research Methodology: An experiment was designed where participants interacted with a voice-based agent as a tutor. The agent employed one of the following three auditory expression modes:
  1. Exclamations (e.g., "yay," "hmm").
  2. Short musical gestures (based on validated emotional music datasets).
  3. No additional auditory expressions (control group).
• Innovative Contributions:
  1. Introducing exclamations and musical auditory signals into the voice-based dialogue of educational agents to explore their impact on perceived rapport and learning motivation.
  2. Linking word-level auditory expressions to learning and emotional outcomes, addressing a research gap.
  3. Comparing language-dependent (exclamations) and language-independent (music) auditory expression methods.
• Implementation Steps:
  1. Develop a conversational teaching environment where participants teach the agent about rock types.
  2. Use Wizard-of-Oz techniques to have a human operator control the agent's responses.
  3. Evaluate participants' learning motivation, memory retention, and human-agent rapport through pre-tests, interaction phases, and post-tests.

Research Findings

• Specific Results:
  1. Emotional Rapport: Exclamations significantly enhanced learners' perceived emotional rapport with the agent, whereas the effect of music was weaker.
  2. Learning Motivation:
     • Exclamations significantly increased intrinsic motivation (e.g., "based on interest or sense of achievement") and extrinsic motivation (e.g., "cognitive growth").
     • Musical expressions also significantly improved certain dimensions of motivation.
  3. Learning Outcomes (Recall): Knowledge retention improved across all groups, but no statistically significant differences were observed between conditions.
• Comparative Advantages:
  • Exclamations significantly increased the emotional connection to learning and improved interaction quality.
  • Music enriched the agent's expressiveness; while its effect on learning motivation was weaker, it contributed to more precise emotional expression.
• Experimental and Evaluation Results:
  • Motivation and emotional rapport were significantly enhanced, but no significant impact on learning outcomes (memory) was observed.
  • Participants did not perceive an increase in cognitive load, indicating that such auditory expressions are suitable for educational scenarios.
• Limitations and Future Directions:
  1. The small sample size limits generalizability to larger populations.
  2. The short duration of the experiment may not fully capture the formation of long-term human-agent relationships.
  3. Further optimization of the compatibility between auditory expressions and speech synthesis is needed to avoid "uncanny" or discordant effects.
  4. Future research could explore the impact of auditory expressions in multimodal agents (including visual or tactile feedback) or in other application scenarios such as healthcare or customer service.