Emotion Embodied: Unveiling the Expressive Potential of Single-Hand Gestures

Document Title

Emotion Embodied: Unveiling the Expressive Potential of Single-Hand Gestures

Document Information

• Subject Area: Human-Computer Interaction (HCI), Affective Computing, Gesture Recognition
• Keywords: Emotion capture, gesture interaction, gesture guidance, emotion tracking, single-hand gestures, emotional expression, embodied cognition, physiological signal detection, high-low arousal emotions, gesture design

Research Background and Problems

• Identified Issues or Challenges:

  • Current emotion tracking systems often rely on self-report methods such as diaries or questionnaires, which are susceptible to recall bias, user burden, and lack of long-term engagement.
  • Automated emotion tracking methods (e.g., facial expression recognition, voice analysis, and physiological signal detection) face practical challenges in deployment and data interpretation (e.g., comfort and accuracy).
  • Although gestures are closely related to emotional expression, existing research and technologies primarily focus on the symbolic meaning of gestures, lacking studies on the relationship between emotional states and gesture details (e.g., finger direction, palm orientation, and movement intensity).

• Research Significance:

  • Gestures are an important component of emotional expression in daily life. Capturing this non-verbal form of expression can facilitate more natural and convenient emotion tracking.
  • Single-hand gestures are easy to perform and capture, making them suitable for simple, deployable emotion tracking systems.

• Research Motivation and Related Work:

  • Current gesture research is mostly applied to user interface control and lacks systematic analysis of the relationship between single-hand gestures and refined emotional dimensions (e.g., levels of valence or arousal).
  • Studies on the relationship between gestures and emotions often remain at the level of symbolic gestures (e.g., "thumbs up" or "victory") and overlook the impact of gesture attributes on emotional expression.

Solution

Proposed Method

• A comprehensive experimental design was proposed, combining emotion elicitation surveys and follow-up interviews to study how single-hand gestures convey emotions.
  • Data Collection: Conducted a survey with 63 participants, capturing 756 gesture photos and videos expressing 12 emotions.
  • Interview Analysis: 11 participants further explained the motivations behind their gestures.
  • Analysis Methods: Encoded and statistically analyzed gesture characteristics such as gesture names, finger direction, palm orientation, intensity, and movement frequency, supplemented by qualitative feedback to understand the psychological process behind gesture formation.

Novelty of the Solution

• Systematically revealed how emotional dimensions (valence and arousal levels) are related to single-hand gestures (e.g., finger direction and gesture intensity) for the first time in existing literature.
• Explored potential channels for emotional expression, including "communicative habits," "creative representation," "physical expression," and "abstract expression."
• Provided insights for designing multimodal emotion tracking systems, exploring the potential of gestures for real-time, low-burden emotional input.

Implementation Steps and Techniques

1. Emotion Elicitation and Data Collection:

   • Used 12 images selected from the OASIS database to induce emotions covering different valence and arousal levels, capturing participants' gestures for each emotion.
   • Each image corresponded to one static gesture photo and one short video.

2. Gesture Feature Encoding and Statistical Analysis:

   • Developed a gesture coding framework to extract features such as gesture morphology, finger direction, intensity, and dynamic frequency.
   • Used Chi-square statistical tests and residual analysis to evaluate the correlation between gesture features and emotional dimensions.

3. Follow-Up Interviews to Analyze Gesture Formation Motivation:

   • Explored the emotional drivers and psychological models behind gesture formation based on emotion-inducing images and participants' uploaded gestures.

Research Findings

Key Discoveries

1. Correlation Between Gesture Features and Emotional Dimensions:

   • Finger Direction:
     • Upward finger direction primarily expressed high-valence, high-arousal emotions (e.g., excitement).
     • Downward finger direction appeared in low-valence, low-arousal emotions (e.g., fatigue).
   • Gesture Intensity:
     • Tight grip gestures (e.g., clenched fists) often expressed high-arousal negative emotions (e.g., anger).
     • Looser gestures often expressed low-arousal negative emotions (e.g., fatigue).

2. Four Channels of Emotional Externalization:

   • Communicative Habits: Using familiar gesture symbols (e.g., "thumbs up") or cultural symbols to express emotions.
   • Creative Representation: Creating gestures to enrich expression based on personality and context.
   • Physical Expression: Directly venting emotions through intuitive actions (e.g., punching).
   • Abstract Expression: In some cases, gestures intuitively express emotions without specific origins.

3. Broad Application Prospects:

   • Single-hand gestures are suitable for emotion tracking systems, especially for low-burden, real-time emotional input.
   • Younger participants tend to use more diverse gestures, revealing a potential correlation between age and emotional expression tendencies.

Comparison with Existing Solutions and Advantages

• Compared to complex gestures requiring multiple hands or full-body involvement, this study focuses on simple single-hand gestures, exploring their applicability in everyday scenarios.
• Emphasizes the informational value of gesture dynamics (e.g., intensity and direction) rather than merely static shape recognition.

Experimental and Evaluation Results

• The study sample included 39 females and 23 males, covering a diverse age range from 18 to 54 years.
• Data analysis confirmed significant correlations between gesture features and emotional dimensions (p < .001), providing a practical gesture library and feature combinations for emotion inference.

Limitations and Future Directions

1. Limitations:

   • Gesture feature encoding relies on manual judgment, and limited cultural and geographical diversity may affect generalizability.
   • Insufficient analysis of the relationship between gesture dynamic features (e.g., movement range, fluidity) and emotions.

2. Future Directions:

   • Expand the study sample to include participants from diverse cultural backgrounds.
   • Introduce more precise motion capture technologies (e.g., sensor gloves) to quantify gesture dynamics.
   • Develop prototype systems combining multimodal inputs (e.g., heart rate, voice) for emotion tracking, enhancing contextual description and user engagement.

Conclusion

This study is the first to comprehensively reveal the multifaceted relationship between single-hand gestures and emotional expression, providing key evidence to support their use in real-time emotion tracking systems. It not only advances academic understanding of the relationship between gestures and emotions but also offers practical guidance for designing user-friendly, flexible multimodal affective computing systems.