EmoShortcuts: Emotionally Expressive Body Augmentation for Social Mixed Reality Avatars

DOI

We present EmoShortcuts, a novel social Mixed Reality (MR) framework that enhances emotional expression by dynamically augmenting avatar body gestures to reflect users’ emotional states. While social MR enables immersive remote interactions through avatars, conveying emotions remains challenging due to limitations in head-mounted display (HMD) tracking (e.g., missing lower-body movements like stomping or defensive postures) and users' tendency to deprioritize nonverbal expression during multitasking. EmoShortcuts addresses these challenges by introducing an augmentation framework that generates expressive body gestures even when users' physical movements are restricted. We conducted a formative study with 12 participants to identify key challenges in emotional expression and explore user preferences for AI-assisted gesture augmentation. Based on these insights, we designed an interface that enables adaptive gesture augmentation and allows for both pre-set and real-time user control. Through an extensive user study (n = 16), our findings show that EmoShortcuts significantly improves emotion expression accuracy and reduces cognitive workload, demonstrating its potential for more immersive and emotionally rich virtual communication.