STMG: A Machine Learning Microgesture Recognition System for Supporting Thumb-Based VR/AR Input

Title of the Paper

STMG: A Machine Learning Microgesture Recognition System for Supporting Thumb-Based VR/AR Input

Paper Information

• Domain: Microgesture recognition and machine learning in Virtual Reality/Augmented Reality (VR/AR)
• Keywords: microgesture, neural network, machine learning, augmented reality, virtual reality, human-computer interaction, gesture recognition, temporal convolutional network, skeletal tracking, data augmentation

Research Background and Problem Statement

• Challenges:

  • Current hand input methods in VR/AR devices primarily rely on the "pinch" gesture between the thumb and index finger, which is limited in expressive capability.
  • Existing methods for mapping hand movements to practical applications often involve complex or large arm motions, with gesture recognition accuracy significantly affected by tracking jitter and noise.
  • Recognizing microgestures of the thumb on the surface of the index finger is challenging due to their small scale (2-3 cm sliding or tapping) and significant variability among users.

• Significance:

  • Expanding the gesture library to support more gesture types can enhance user experience, naturalness, efficiency, and immersion in hand input.
  • Efficient and accurate gesture recognition models are crucial for reducing misrecognition and improving user interaction satisfaction.

• Motivation and Related Work:

  • Existing gesture recognition research focuses on wearable devices (e.g., ring sensors, electric field sensing) and skeletal data-based recognition systems implemented with depth or RGB cameras.
  • The goal of STMG is to achieve high-precision thumb microgesture classification using skeletal data captured by cameras without relying on additional sensors.

Proposed Solution

• Proposed Solution:

  • A system named STMG (Skeletal Tracking MicroGesture) was designed to support the recognition of seven types of thumb microgestures on the surface of the index finger, including four directional slides (left, right, forward, backward), thumb tapping, pinch start, and pinch end.
  • Temporal Convolutional Network (TCN) was utilized to classify skeletal motion data, combined with multi-task loss functions (CTC loss, cross-entropy loss, and temporal alignment loss) to enhance model performance.

• Innovations:

  • Generalizable skeletal tracking model: A large-scale dataset capturing diverse user variations was collected and annotated, improving the model's adaptability to different users.
  • Data augmentation strategies: Training sample diversity was enhanced by simulating gestures at different speeds and variations in finger bending.
  • Integration of tapping and sliding: Small thumb movements were mapped to VR/AR remote interactions and scrolling tasks, providing a convenient and intuitive input method.

• Implementation Steps and Technical Details:

  1. Data Collection:
     • 228 participants were recruited, and their hand movements were recorded using OptiTrack and head-mounted camera systems.
     • Training data included positive samples (microgestures) and hard negative samples (non-gestures).
  2. Model Design:
     • The model used eight joint angles of the thumb and index finger as input features.
     • A 10-layer convolutional module TCN was constructed, employing dilated convolutions to expand the receptive field and capture the temporal dynamics of gestures.
     • CTC loss and temporal alignment loss were combined to address gesture alignment issues and prediction collapse problems.
  3. Experimental Evaluation:
     • Performance was evaluated using a test dataset from 20 participants and compared with traditional heuristic methods and hybrid recognition approaches.
     • Data augmentation experiments demonstrated further performance improvements.
  4. Model Implementation:
     • The model was trained on GPUs using the PyTorch framework, with ablation experiments conducted on different augmentation strategies.

Research Results

• Specific Results:

  • The model achieved an average F1 score of 95.1% across seven microgesture types, with the most challenging thumb tapping gesture reaching an F1 score of 90.8%.
  • Compared to heuristic methods (61.0%) and hybrid methods (81.6%), the machine learning approach showed significant performance improvements.
  • Balanced test datasets further validated the model's robustness across attributes such as gender, age, hand size, and handedness.

• Advantages Over Existing Solutions:

  • The STMG model achieved recognition accuracy competitive with or exceeding devices like ring sensors without requiring additional sensors (OptiRing 93.1%, EFRing 85.2%).
  • It enabled interaction through small hand movements while preserving natural gestures, allowing users to interact without large-scale motions.

• Experimental or Evaluation Results:

  • User testing with 20 participants showed that STMG was simpler and easier to learn compared to traditional pinch gestures, with 17 participants preferring STMG for virtual navigation.
  • In long-distance scrolling tasks, 60% of users reported that STMG's sliding gestures were more efficient and comfortable than "pinch drag."

• Limitations and Future Directions:

  • Limitations:
    • The accuracy of hand tracking still limits the reliability of certain gestures (e.g., subtle thumb tapping).
    • Thumb-to-index contact gestures are not yet fully optimized for highly dynamic scenarios (e.g., gaming).
  • Future Directions:
    • Introduce continuous recognition of thumb sliding and tapping for more refined continuous input functionality.
    • Further optimize adaptability for different user styles, especially those favoring small-scale motions.
    • Expand the gesture set to include bimanual interactions, supporting more complex tasks.