Enabling Hand Gesture Customization on Wrist-Worn Devices

Title of the Paper

Enabling Hand Gesture Customization on Wrist-Worn Devices

Paper Information

• Subject Area: Human-Computer Interaction (HCI), specifically gesture recognition and customization on smart wristbands
• Keywords: gesture customization, transfer learning, few-shot learning, human-computer interaction, wrist-worn devices, incremental learning, dynamic gesture recognition, neural networks, data augmentation, user experience

Research Background and Problem

• Identified Problems:

  1. Current gesture recognition technologies for smart wristbands are primarily based on predefined gesture sets, making it difficult to flexibly support user-defined gestures.
  2. Existing "few-shot learning" methods are mostly designed for simple gesture recognition tasks and rarely support complex, natural gestures.
  3. Gesture customization systems often face the following challenges: enabling rapid and minimal data collection while ensuring that recognition performance for existing gestures does not degrade.

• Significance:

  • Supporting user-defined gestures not only enhances the personalization of user experience but also improves memorability, interaction efficiency, and accessibility for individuals with disabilities.
  • This approach could become a core interaction method for future smart wearable devices.

• Research Motivation and Related Work:

  • Based on existing gesture recognition research, the authors observed that while traditional techniques like "Dynamic Time Warping" can handle simple gestures, they fall short when dealing with complex, diverse gestures that need to accommodate different user habits.
  • Some earlier studies attempted to develop customizable gestures but overlooked the challenges of adapting to incremental categories in existing models.

Solution

• Proposed Method:

  1. Developed a gesture customization framework that supports user-defined gestures with only 3-5 samples, without affecting the recognition performance of existing gestures.
  2. Combined transfer learning, incremental learning, and few-shot learning techniques to design a neural network architecture based on accelerometer and gyroscope data.
  3. Introduced mechanisms such as data augmentation, data synthesis, and adversarial training to improve model robustness under few-shot conditions.

• Innovations:

  1. A novel approach that separates the feature extraction layer of the pre-trained model, allowing independent support for newly added user-defined gestures.
  2. Provided an interactive feedback mechanism to notify users in real-time about the validity of newly defined gestures (e.g., whether they are too similar to existing gestures or distinguishable).
  3. The system enables users to create and test gestures in real-time on local devices, offering performance evaluation and adjustment options.

• Key Steps and Techniques:

  1. Trained a convolutional neural network (CNN) on accelerometer-gyroscope data from over 500 users to classify five categories (four gestures + non-gesture).
  2. Improved overall recognition accuracy for multi-window gestures through data aggregation and sliding window methods.
  3. For new gestures, used data augmentation (scaling, temporal transformations, etc.) and ∆-encoder data synthesis methods to generate additional samples.
  4. Applied adversarial training to adjust classification performance in boundary regions, reducing false positive rates.

Research Outcomes

• Specific Results:

  1. The pre-trained model achieved a gesture recognition accuracy of 95.7% on the test set, with an F1 score of 95.8%. It demonstrated fast classification speed and strong robustness (false positive rate for non-gesture samples as low as 0.6 per hour).
  2. For new gestures, an accuracy of 83.1% and an F1 score of 88.9% were achieved with only three samples.
  3. Even with the addition of 2-4 new gestures, the recognition performance of existing gestures remained high.

• Comparative Advantages:

  • Compared to methods like Dynamic Time Warping, standard SVM, and random forests, this approach showed significant improvements in accuracy and robustness.
  • While transfer learning (fine-tuning) exhibits some flexibility in few-shot scenarios, it impacts the performance of existing gestures. The incremental learning architecture in this study avoids this issue.

• Experiments and Evaluation:

  1. User experience evaluation (N=20) showed high scores for the learnability and usability of customized gestures (SUS average score of 87.2/100).
  2. Task load (NASA-TLX) indicated that the psychological and physical burdens during gesture customization and testing were low.

• Limitations and Future Directions:

  • The current experimental gesture set is not comprehensive, and future work needs to expand to more complex gesture features.
  • The impact of temporal drift in user gestures on performance has not been thoroughly explored.
  • The current model training relies on external computing devices. Future plans include optimizing for fully local device training or cloud-edge collaborative computing.

Conclusion

This study proposes an innovative gesture customization framework that supports few-shot customization while maintaining stable recognition performance for existing gestures. Combining user experience research with technical validation, the framework demonstrates strong potential and robustness in real-world human-computer interaction scenarios, providing valuable insights for broader wearable device interaction research.