Crossmodal Interactions in Human-Robot Communication: Exploring the Influences of Scent and Voice Congruence on User Perceptions of Social Robots

Research Background and Issues

• Problem or Challenge: Current research in the field of Human-Computer Interaction (HCI) primarily focuses on visual and auditory cues, with limited exploration of the role of olfactory stimuli in shaping user perceptions of social robots. Furthermore, there is a lack of in-depth studies on whether consistency among different sensory cues in multimodal interaction enhances user experience.
• Significance: As social robots are increasingly applied across various domains (e.g., healthcare, education, customer service), understanding how multisensory design can enhance users' social affinity and trust toward robots is of critical importance.
• Research Motivation: Cross-modal interaction between scent and sound may provide new approaches for designing robots with enhanced social interactivity. By exploring the potential of such multisensory interaction, this study aims to address gaps in the current literature.

Solution

• Proposed Method: The authors employed combinations of scent (pleasant/unpleasant) and sound (friendly/unfriendly) in a 2x2 experimental design to evaluate the impact of these stimuli on four dimensions of social robot perception: "trust, friendliness, competence, and engagement." Additionally, the study investigates whether consistency between scent and sound enhances user experience.
• Innovations:
  • Conducting the first systematic study on the role of scent and sound in cross-modal sensory interaction for social robots.
  • Exploring the moderating effect of sensory consistency (e.g., friendly sound and pleasant scent) on user perception.
  • Expanding theoretical understanding of multimodal design while providing practical insights for robot design.
• Key Technologies and Steps:
  1. Developed a robot model integrating scent diffusion and sound playback modules, enabling remote control for switching experimental conditions.
  2. Selected lavender as the pleasant scent and a chemical odor (similar to ammonia) as the unpleasant scent; professional male voice actors recorded friendly and unfriendly speech.
  3. Conducted robot interaction experiments with 160 participants distributed across four experimental conditions (pleasant scent + friendly sound, unpleasant scent + unfriendly sound, etc.).
  4. Collected user ratings and subjective feedback on trust, friendliness, competence, and engagement through questionnaires and interviews.

Research Findings

• Specific Results:
  • Pleasant scent significantly improved users' perceptions of the robot's friendliness and engagement but had limited impact on trust and competence.
  • Friendly sound significantly enhanced users' perceptions of the robot's trust, friendliness, and engagement, with minimal effect on competence.
  • Sensory consistency between pleasant scent and friendly sound strengthened evaluations of friendliness and engagement but had no significant impact on trust or competence.
• Advantages:
  • Provides a multimodal design model combining scent and sound, offering a pathway for designing more socially interactive robots.
  • Explores the role of sensory consistency in enhancing users' emotional and cognitive responses.
• Experimental or Evaluation Results:
  • Quantitative Analysis: Variance analysis revealed that pleasant scent and friendly sound exhibited strong effects on "friendliness" and "engagement" in both independent and interactive conditions (significance p < 0.001).
  • Qualitative Feedback: Participants generally responded positively to the combination of pleasant scent and friendly sound, with some noting that inconsistent sensory feedback could lead to confusion or cognitive dissonance.
• Limitations and Future Directions:
  • Limitations:
    • The interaction duration was relatively short, potentially limiting the ability to fully assess the impact of scent and sound cues on complex cognitive evaluations (e.g., competence, trust).
    • The experiment did not include a no-scent condition as a baseline control, which is necessary for further elucidating the independent effects of stimuli.
    • Both scent and sound presentations were static, lacking dynamic interaction capabilities.
  • Future Directions:
    • Investigate the effects of dynamic scent and sound interactions, such as adaptive behaviors following emotional changes or task failures.
    • Extend the study to more contextual variables (e.g., healthcare scenarios, educational settings) to validate the generalizability of the findings.
    • Incorporate neutral scent and sound conditions to more finely distinguish the independent and combined effects of sensory inputs.

Conclusion

This study demonstrates the potential value of scent and sound stimuli in improving social interaction with robots. While sensory consistency can enhance certain dimensions of user experience, complex cognitive evaluations such as trust and competence require more sophisticated interaction designs. These findings provide new insights into the theory and practice of multisensory interaction and point to promising directions for future research.