VibEmoji: Exploring User-authoring Multi-modal Emoticons in Social Communication

Title of the Paper

VibEmoji: Exploring User-authoring Multi-modal Emoticons in Social Communication

Paper Information

• Research Area: Human-Computer Interaction (HCI) and Affective Communication Technology
• Keywords: Social communication, emotional expression, emoticons, haptic feedback, animation, multimodal interaction, mobile interface

Research Background and Problem

• Identified Issues or Challenges:

  • Although static emoticons (e.g., emojis) have become highly popular in online social communication, they often fail to meet users' expectations for emotional expression in many scenarios.
  • An increasing number of social applications are beginning to support multimodal emoticons with animations or haptic feedback, but there is a lack of research on how users create, share, and experience these emoticons.
  • Current designs restrict user freedom, as emoticons, animations, and haptic feedback are often fixed combinations, preventing users from freely adjusting or creating new meanings.

• Significance of the Problem:

  • Emoticons serve as a cross-cultural visual language, effectively compensating for linguistic limitations and expressing emotions and non-verbal reactions, making them crucial in modern communication.
  • Multimodal emoticons have the potential to create richer emotional interactions, enhancing empathy between users.

• Motivation and Related Work:

  • The motivation stems from a trend where users demand more personalized and emotionally expressive emoticons.
  • This paper analyzes the limitations of current multimodal emoticons in detail and proposes a novel design to address these issues.

Solution

• Proposed Solution:

  • Designed and implemented the VibEmoji system, a mobile messaging application that supports users in creating and sharing multimodal emoticons in real-time.
  • VibEmoji allows users to freely combine different static emoticons, animations, and haptic feedback, enabling personalized expression and on-the-spot creation.

• Innovative Aspects of the Solution:

  • Creative Freedom: Users can dynamically select and combine elements from three independent multimodal libraries (emoticons, animations, vibration patterns), breaking the limitations of fixed combinations.
  • Non-intrusive Recommendations: The system provides suggestions based on users' previous usage patterns and emotional relevance but avoids automated decisions, ensuring creative freedom.
  • On-the-fly Creation: Unlike other systems that require pre-defined content, VibEmoji supports users in creating emoticons that meet real-time emotional needs during conversations.

• Key Technologies and Implementation Steps:

  • Utilized a rich library of Apple emojis, Kineticons animation library, and VibViz haptic feedback library as building blocks.
  • Developed a recommendation algorithm based on emotional relevance (valence-arousal model) and historical usage frequency.
  • Built a mobile application with front-end and back-end communication systems, enabling real-time messaging and emoticon generation.

Research Outcomes

• Specific Achievements:

  • Developed a fully functional multimodal emoticon generation system, VibEmoji, enabling users to achieve diverse emotional expressions.
  • Conducted a four-week field study to gather extensive user experiences with VibEmoji.
  • Identified the unique value of multimodal emoticons in enhancing conversational enjoyment, engagement, emotional accuracy, and expressive range.

• Advantages Compared to Existing Solutions:

  • VibEmoji enables users to dynamically create emoticons rather than being limited to pre-prepared options.
  • The combination of animation and haptic components expands the dimensions of communication, allowing for more nuanced emotional expression.
  • The non-intrusive recommendation design ensures flexibility and enjoyment in the user experience.

• Experimental and Evaluation Results:

  • Surveys revealed that users generally found VibEmoji to enhance the fun, engagement, and richness of emotional expression in communication.
  • Data analysis showed that participants sent one multimodal emoticon for every three text messages on average.
  • User feedback highlighted the "sound-like" quality of the haptic patterns and the "body language-like" quality of the animations.

• Limitations and Future Directions:

  • Limitations: The system currently does not support custom emoticons or dynamic import of new assets; data is primarily limited to a younger user demographic.
  • Future Directions:
    • Expand customization options, such as allowing users to upload new emoticons, create new animations, and design haptic patterns.
    • Explore the potential of multimodal emoticons in broader application scenarios (e.g., smart homes, social robots).
    • Further optimize the recommendation algorithm by incorporating collaborative filtering techniques.

Conclusion

This study designed, implemented, and evaluated an innovative multimodal emoticon system that not only empowers users in their daily emotional communication but also broadens the research perspective on multimodal communication in the field of human-computer interaction. It provides clear directions and practical insights for future design endeavors.