Robust Finger Interactions with COTS Smartwatches via Unsupervised Siamese Adaptation

Document Title

Robust Finger Interactions with COTS Smartwatches via Unsupervised Siamese Adaptation (ViWatch)

Document Information

• Topic Area: Wearable device interaction technology, sensor-based gesture recognition
• Keywords: Gesture recognition, finger interaction, vibration sensing, unsupervised adversarial training, smartwatch, IMU sensor, deep learning, deployment variability, unsupervised adaptation, system user experience

Research Background and Issues

• Identified Problems or Challenges:
  • Smartwatches and wristbands face difficulties in enabling convenient input operations due to screen size limitations.
  • Current gesture recognition technologies lack robustness against deployment variability (e.g., hand shape differences, finger activity intensity, smartwatch position changes).
  • Supervised learning-based methods require frequent user involvement for annotation and calibration, which is time-consuming and less practical for real-world applications.
• Importance of the Problem:
  • As portable computing platforms, smartwatches are poised to become essential tools in industrial, medical, and consumer electronics fields.
  • Enhancing smartwatch interaction robustness and user experience can unlock more potential application scenarios.
• Research Motivation and Related Work:
  • Existing methods like FingerIO and LLAP attempt millimeter-level finger positioning via smartphones but are slow and unsuitable for smartwatches.
  • Systems such as Taprint focus on secure authentication but still require user-annotated data to adapt to variability; current unsupervised adaptation methods struggle with finer fingertip activities.

Solution

• Proposed Method or Solution:
  • A system named ViWatch was designed to enable accurate finger interaction using a single IMU sensor embedded in smartwatches.
  • Introduced unsupervised Siamese adversarial learning to address performance degradation caused by deployment variability in finger interactions.
  • The system architecture includes vibration signal preprocessing, training a general convolutional neural network (CNN), and optimizing a DANN (Domain-Adversarial Neural Network) model using Siamese learning.
• Innovations:
  • Algorithm Innovation: For the first time, unsupervised Siamese adversarial learning was applied to fine-grained finger interaction, significantly improving model robustness against deployment variability.
  • Sensor Innovation: Utilized only the single IMU sensor in commercial smartwatches without requiring additional hardware support.
  • Application Innovation: Achieved real-time interaction, providing a natural and user-friendly control system for smartwatches and related IoT devices.
• Implementation Steps and Key Techniques:
  1. Signal Preprocessing: Captured vibration signals using a dual-threshold segmentation method, denoised through Fourier analysis and high-pass filtering.
  2. CNN Backbone Model Training: Designed a five-layer convolutional network for finger interaction classification, avoiding overfitting.
  3. Unsupervised Adaptation: Optimized the model using DANN to adapt to user habitual behaviors.
  4. Siamese Optimization: Applied contrastive loss to adjust inter-domain embeddings, enabling the model to effectively handle diverse user influences.

Research Outcomes

• Specific Results:
  • ViWatch achieved an average real-time classification accuracy of 97% over one week.
  • The system demonstrated robustness against deployment variability, including hand shape differences, finger movement intensity, and smartwatch position changes.
  • Showcased ViWatch's versatility through various application scenarios (e.g., smartwatch gaming, remote control of head-mounted devices).
• Advantages Compared to Existing Solutions:
  • Compared to other methods like ViType, iDial, and Taprint, ViWatch improved classification accuracy by over 20%.
  • Eliminated the need for user-annotated data, significantly reducing user participation burden.
  • Maintained high robustness under diverse variability conditions, making it suitable for a wide range of practical applications.
• Experimental or Evaluation Results:
  • Offline Evaluation: Achieved an average classification accuracy of approximately 94%, significantly outperforming baseline methods.
  • Real-Time Evaluation: Adapted to varying behavior intensities, smartwatch positions, and finger types, maintaining over 97% accuracy.
  • User Experience Evaluation: Studies based on SUS and NASA-TLX showed significantly higher user acceptance of ViWatch compared to competing systems.
• Limitations and Future Directions:
  • Limitations:
    • The dataset size remains limited, which may affect the generalization capability of the predictive model.
    • The system requires further optimization for scenarios involving carrying other objects and extreme noise environments.
  • Future Directions:
    • Extend the model to support more user samples and a broader range of demographic groups, analyzing cross-domain transfer effects.
    • Investigate models capable of efficient on-device operation to enhance scalability.
    • Explore the application of unsupervised Siamese adaptation in other gesture recognition and sensor scenarios.

Conclusion

The ViWatch system leverages innovative algorithms to overcome deployment variability issues in commercial smartwatch interactions, achieving high accuracy and robustness with broad application potential. Unlike traditional supervised learning approaches, it eliminates the need for user annotation, significantly improving user experience. This research lays the foundation for future interaction technology innovations and provides valuable practical insights.