Multimodal Emotion Recognition of Hand-Object Interaction

Document Title

Multimodal Emotion Recognition of Hand-Object Interaction

Document Information

• Subject Area: Multimodal Emotion Recognition in Hand-Object Interaction
• Keywords: Affective Touch, Emotion Classification, Hand-Object Interaction, Multimodal Data, Human-Computer Interaction

Research Background and Problem

• Identified Problems or Challenges:
  • Traditional touch interaction research typically focuses on social touch technologies between humans or between humans and robots, with little attention given to the relationship between touch and emotion during human interaction with rigid objects.
  • Inferring human emotional states from natural hand-object interactions remains an unresolved issue.
• Significance:
  • Emotion recognition is crucial for developing intelligent user interfaces, entertainment, rehabilitation, and communication for individuals unable to express themselves verbally.
  • Current emotion interaction technologies need to enhance their ability to process natural emotional expressions in hand-object interactions.
• Research Motivation and Related Work:
  • The study proposes classifying human emotional states based on touch and motion data collected through a specially designed device in natural interaction scenarios.
  • Unlike previous studies relying on symbolic touch gestures or pressure data from soft objects, this research focuses on natural interactions with rigid objects.

Solution

• Proposed Method or Solution:
  • Developed a device called iCube, a 5 cm cube embedded with touch sensors and an accelerometer, to collect touch and motion data from natural hand movements.
  • Designed a set of high-level handcrafted features to describe tactile and motion data, covering metrics such as touch density, variation, and rotation.
  • Classified emotions based on these features, aiming to recognize four emotions: anger, sadness, excitement, and gratitude.
• Innovative Aspects of the Solution:
  • First to use tactile and motion data from rigid objects as the basis for emotion classification.
  • Utilized a semantically neutral and simple-shaped object (similar to common everyday objects) to minimize external environmental influences.
  • Combined tactile and motion data for multimodal emotion recognition.
• Implementation Steps and Key Techniques:
  1. Collected tactile and motion data using iCube.
  2. Extracted 17 handcrafted features, including touch density, rate of change, and primary touch panels.
  3. Performed emotion classification using Support Vector Machine (SVM) and Localized Multiple Kernel Learning (LMKL) algorithms.
  4. Evaluated model performance using leave-one-out cross-validation.

Research Outcomes

• Specific Results:
  • Identified significant emotional effects on 12 features extracted from tactile and motion data, demonstrating the features' ability to capture variations in emotional states.
  • Achieved an accuracy of 0.75 (four emotion classes) in classification experiments using LMKL-SVM, indicating performance above random guessing.
• Advantages Compared to Existing Solutions:
  • Does not rely on pressure data, reducing device complexity and the need for additional data transmission.
  • Combining multimodal (tactile and motion) data significantly improves classification performance.
  • iCube's simple and semantically neutral design facilitates natural interaction and enhances user adaptability.
• Experimental or Evaluation Results:
  • Larger touch areas, more rotational movements, and shorter interaction times were observed for anger and excitement emotions; slower movements, fewer touch variations, and longer interaction times were observed for sadness.
  • Emotion classification results showed higher differentiation between anger and sadness, while excitement and gratitude were more easily confused.
• Limitations and Future Directions:
  • Limited dataset size necessitates expanded sample collection to improve classifier generalization.
  • Future research will explore the impact of different object shapes, materials, and sizes on emotional expression, investigating the potential to distinguish closely related emotions (e.g., anger vs. anxiety).
  • Plans to collect ecological data in real-world scenarios, such as gaming, to study spontaneous emotional interaction behaviors.