Raise Your Eyebrows Higher: Facilitating Emotional Communication in Social Virtual Reality Through Region-Specific Facial Expression Exaggeration

Research Background and Issues

• Issues and Challenges:
  • In social virtual reality (VR), exaggerated facial expressions for cartoon-like avatars can enhance emotional communication and user experience but may trigger the "Uncanny Valley Effect," a phenomenon that causes discomfort among users.
  • Current research on expression exaggeration techniques primarily focuses on static images or 2D displays, failing to fully capture the dynamic and multi-emotional interaction characteristics of social VR. Additionally, the physical limitations of head-mounted displays (HMDs) may lead to insufficient expression of the upper facial region.
• Significance: Non-verbal interactions (e.g., facial expressions) in social VR are core mechanisms for building trust and enhancing emotional resonance. Optimizing expression exaggeration techniques can directly impact the realism of virtual experiences and users' emotional engagement.
• Research Motivation: Addressing the gaps in existing research, the authors investigated the effects of real-time facial expression exaggeration in fully dynamic social VR and proposed a facial region-based expression enhancement strategy to optimize expression realism and emotional communication.

Solution

• Methods and Strategies:
  • Proposed a facial region-specific expression exaggeration strategy, applying differentiated exaggeration to areas such as eyebrows, eyelids, nose, cheeks, and mouth based on emotion type and expression characteristics.
  • The upper face (e.g., eyebrows) is exaggerated to a higher degree, while the lower face (e.g., mouth) maintains a natural intensity to avoid unnaturalness caused by over-exaggeration.
• Innovations:
  • Localized the degree of exaggeration rather than applying it to the entire face, adopting differentiated approaches for various emotion categories.
  • Proposed design principles for exaggeration, including quantitative analysis (e.g., activation intensity of specific facial regions) and qualitative feedback (e.g., user acceptance of exaggerated expressions for different emotions).
• Implementation Steps:
  1. Data Collection: Directed actors via scripts to express six basic emotions (joy, sadness, fear, anger, surprise, disgust) in a virtual environment, recording their facial expressions, eye movements, and body posture information.
  2. Experimental Validation: Conducted in two phases:
     • Study 1: Analyzed the impact of overall expression exaggeration on user empathy and the Uncanny Valley Effect.
     • Study 2: Validated the advantages of the region-specific exaggeration strategy in enhancing emotional communication and reducing the Uncanny Valley Effect compared to unmodified original expressions.

Research Findings

• Specific Findings:
  • Study 1:
    • Expression exaggeration had no significant impact on emotion recognition accuracy but showed an interactive effect on emotional contagion and the Uncanny Valley Effect. For instance, exaggerating joy and surprise increased the sense of "unnaturalness," while exaggerating fear and anger enhanced the naturalness and emotional communicability of expressions.
  • Study 2:
    • The region-specific exaggeration strategy significantly improved the emotional communication of disgust, anger, and fear while mitigating the Uncanny Valley Effect. Notably, highly exaggerated eyebrows were crucial for conveying anger.
    • The strategy had limited effects on joy and surprise, as the lower face expressions for these emotions were already prominent, and further exaggeration tended to cause unnaturalness.
• Experimental and Evaluation Results:
  • Across two experiments, the proposed exaggeration strategy was validated for its effectiveness in enhancing the communication of negative emotions (e.g., anger and disgust) while maintaining neutrality in positive emotion expressions.
  • Supplementary qualitative feedback from questionnaires and thematic analysis indicated that users tend to rely more on voice than facial expressions as primary emotional cues, suggesting that exaggeration strategies should consider multimodal optimization.
• Limitations and Future Directions:
  • Limitations: The participant sample was concentrated on young Chinese university students, potentially introducing cultural and age-related biases. The study focused on cartoon-like avatars, and the applicability to high-fidelity avatars requires further examination.
  • Future Directions:
    • Explore the influence of culture and age on the perception of expression exaggeration to improve the strategy's generalizability.
    • Investigate dynamic and adaptive expression enhancement algorithms in real-time interactive scenarios.
    • Conduct further research on the interaction of multimodal cues in complex multi-user settings.

This study addresses the technical challenges of effectively conveying emotions through facial expressions in social VR by introducing an innovative region-specific expression exaggeration strategy. It lays a foundation for optimizing emotional communication in future virtual reality experiences.