AO-Finger: Hands-free Fine-grained Finger Gesture Recognition via Acoustic-Optic Sensor Fusing

Paper Title

AO-Finger: Hands-free Fine-grained Finger Gesture Recognition via Acoustic-Optic Sensor Fusing

Paper Information

• Research Area: Human-Computer Interaction, Augmented Reality/Virtual Reality, Fingertip Gesture Recognition
• Keywords: Acoustic-optic fusion, high-precision gesture recognition, wearable devices, micro-gestures, XR interaction

Research Background and Problem

• Identified Problems or Challenges:

  • Current XR device gesture interactions require significant user effort, such as using touch surfaces or handheld devices, which increases user fatigue.
  • Many existing solutions require tracking within the field of view or direct observation of the hands, while fingertip movements are often difficult to observe.
  • Acoustic sensors are limited in accurately tracking micro-gestures due to low signal-to-noise ratios and susceptibility to background noise.
  • Many solutions only support discrete gestures and cannot achieve continuous, high-precision gesture tracking.

• Significance:

  • Micro-gestures (e.g., flicking, pinching) not only provide a more natural interaction experience but also enable users to perform operations effortlessly in various scenarios.
  • High-precision gesture detection, as a fundamental feature of next-generation XR device interactions, helps improve productivity and user experience.

• Research Motivation and Related Work:

  • Addressing the limitations of existing gesture recognition solutions, such as limited device wearability and inability to track fingertip movements, by exploring the integration of multiple sensors to build a more stable and efficient gesture detection system.

Solution

• Proposed Method or Solution:

  • A system named AO-Finger is proposed, combining optical motion sensors and an improved stethoscope microphone for high-precision micro-gesture recognition.
  • A set of natural, low-profile, and easy-to-perform micro-gestures is defined, including Flick, Pinch, Tap, Swipe Left, and Swipe Right.

• Innovations:

  • Innovatively combines the advantages of acoustic and optical sensors to enhance detection accuracy and signal robustness.
  • Designed a dual-branch neural network architecture for fast gesture classification and continuous sliding gesture detection.
  • Proposed a physics simulation-enhanced data augmentation technique to reduce model overfitting.

• Implementation Steps and Key Technologies:

  • Hardware Design:
    • Built a wristband device incorporating dual optical sensors and an improved stethoscope microphone.
    • Enhanced the microphone to reduce skin interference noise while amplifying gesture signals.
    • Optical sensors detect wrist skin movements via IR signals and process them in real-time.
  • Signal Processing:
    • Acoustic signals are converted into spectrograms to capture both time-domain and frequency-domain features.
    • Median filtering is applied to eliminate spike noise in optical and acoustic signals.
  • Model Design:
    • A multimodal CNN-Transformer model is used for fast gesture detection (e.g., Flick, Pinch, Tap).
    • A neural network detects finger contact states to trigger continuous sliding gesture tracking mode.
  • System Logic:
    • A finite state machine (FSM) aggregates prediction results to reduce false positives.

Research Results

• Specific Results:

  • The AO-Finger prototype achieved an overall gesture detection accuracy of 94.83%.
  • Provided continuous high-precision sliding gesture tracking functionality, supporting more precise interactions compared to discrete gesture detection solutions.

• Advantages Over Existing Solutions:

  • Compared to solutions using cameras or other types of sensors, AO-Finger's hardware design is more compact, consumes less power, and is not affected by ambient light.
  • In micro-gesture detection, the combination of acoustic and optical signals improved robustness and accuracy.

• Experimental or Evaluation Results:

  • Demonstrated good robustness in typical user environments, effectively reducing false triggers.
  • Experimental validation showed high usability as an interaction device for AR glasses.
  • User experiments revealed smooth responsiveness in sliding gesture tracking tasks.

• Limitations and Future Directions:

  • Limitations:
    • The current system relies on external devices (e.g., laptops) for model inference, and it has not yet been integrated into the wristband.
    • Users must keep their hands relatively still to trigger detection, reducing operational fluidity.
    • Applicability to large-scale user groups has not yet been validated.
  • Future Directions:
    • Advance model inference integration into the wristband device to enhance portability.
    • Optimize hardware design to improve sensor field of view and functionality.
    • Expand data collection to support a broader user base.
    • Conduct subjective usability studies, such as NASA TLX evaluations.

References

This paper includes numerous references related to XR interaction and gesture recognition (e.g., solutions based on pressure sensors, ultrasound, and cameras) and cites research on various data augmentation and machine learning techniques. For details, please refer to the references numbered [1] to [43] listed at the end of the paper.