EmoWear: Exploring Emotional Teasers for Voice Message Interaction on Smartwatches

Document Title

EmoWear: Exploring Emotional Teasers for Voice Message Interaction on Smartwatches

Document Information

• Subject Area: Human-Computer Interaction (HCI) and Wearable Technology, exploring emotional voice message interaction design
• Keywords: Emotion, Smartwatch, Voice Message, Animation, Emotional Teaser

Research Background and Problem

• Problems and Challenges:

  • Due to the non-visual nature of voice messages, recipients cannot predict the emotional tone before fully playing the message, limiting the shared emotional experience during the preview stage.
  • Traditional methods (e.g., color coding) have limitations in representing emotions, such as subjective interpretation errors and accessibility issues related to color perception.
  • Current research rarely explores the "teaser" function for voice messages, which provides emotional information without disclosing content.

• Significance:

  • With the surge in voice message usage (e.g., over 7 billion voice messages sent daily on WhatsApp), optimizing the emotional conveyance of voice messages can enhance user communication experiences.
  • Smartwatches, due to screen limitations and the inconvenience of manual interaction, require emotional designs tailored to their environment.

• Related Work and Research Motivation:

  • Related works, such as the EmoBalloon system and color-coded message bubbles, enhance emotional expression but are limited to text messages rather than voice messages.
  • Animation is considered an intuitive and effective way to convey emotions in smartwatch interactions, as seen in studies like Animo and Significant Otter, which use biometric data to communicate emotions.
  • Inspired by these works, this study attempts to combine animation with message bubbles to explore the potential of emotional interaction in voice messages.

Solution

• Proposed Solution:

  • Introduce an emotional voice message system, "EmoWear," which adds 30 animated emotional teaser bubbles to voice messages on smartwatches.
  • Use semantic and acoustic feature analysis of voice messages to recommend the most relevant emotional category, optimizing the user selection process.

• Innovations:

  • Combines animation and message bubble expressions, overcoming the limitations of single-color coding.
  • Provides an emotion recognition assistant for senders, simplifying user operations through intelligent prioritization while retaining diverse options.
  • Animation design is based on Ekman's basic emotion theory and incorporates rich visual effects.

• Implementation Steps and Techniques:

  1. Frontend Interface Design: Develop an Android Wear OS application supporting voice message recording and emotional animation bubble preview/selection.
  2. Backend Algorithm Development: Build a fusion model combining voice (MFCC features) and text (BERT model) data for emotion classification.
  3. Message Interaction Process: After recording audio, the system generates recommended emotional categories. The sender selects an animation bubble, and the message is sent; the recipient sees the animation bubble and clicks to play the voice message.

Research Outcomes

• Specific Results:

  • Experimental comparisons show that "EmoWear" significantly outperforms control systems (using color-coded bubbles) in helping recipients predict sender emotions, conveying emotions, and enhancing overall communication experience.
  • Animated bubbles are considered intuitive, vivid, and enrich the diversity and subtlety of emotional expression. Participants provided positive feedback on the system's usability and enjoyment.

• Advantages:

  • Animation design better meets user expression needs, capturing more complex and nuanced emotions.
  • The new recommendation mechanism simplifies the selection process while maintaining freedom, significantly improving user experience.
  • Enhances the transmission of nonverbal information and emotional connection in communication (e.g., increasing intimacy).

• Experimental or Evaluation Results:

  • EmoWear outperformed the control group in 13 out of 15 experience quality metrics. Participants expressed willingness to use the system for daily communication with family, friends, or partners.
  • Qualitative interviews further revealed the value of emotional teasers in supporting emotion prediction, enhancing communication enjoyment, aiding emotional regulation, and conveying intimacy.

• Limitations and Future Directions:

  • Limitations:
    • The experimental environment was a laboratory setting, lacking real-world ecological usage data.
    • The backend cloud-based recommendation mechanism may raise privacy concerns, and training data may have biases due to label imbalance.
  • Future Directions:
    • Conduct longer-term field tests to verify ecological applicability.
    • Expand emotional options and explore dynamic expressions of multiple emotional changes.
    • Introduce user customization or co-designed animations to enhance personalization and intimacy in interaction.
    • Integrate other information channels (e.g., sound and haptics) and biometric signals to achieve more authentic and rich emotional experiences.

This document provides an innovative exploration and practice in emotional smartwatch interaction design, not only enhancing the depth of communication in voice messages but also paving a meaningful path for future HCI research.