FingerBar: A Mid-Air Touch Bar Interface for Earphones Using Finger-Generated Acoustics

Paper Title

FingerBar: A Mid-Air Touch Bar Interface for Earphones Using Finger-Generated Acoustics

Publication Info

• Topic area: Mid-air gesture recognition for earphone interaction using passive acoustic sensing.
• Keywords: Mid-air gestures, passive acoustic sensing, earphone interaction, friction sounds, gesture recognition, adversarial training, user-independent, noise filtering, wearable technology, real-time interaction.

Background and Problem

• Problem / challenge: Existing earphone interaction methods face limitations such as discomfort, hygiene concerns, restricted gesture sets, and energy inefficiency. Touch-free systems using active acoustic sensing may introduce audible signals, hearing risks, and energy burdens.
• Significance: Addressing these issues could enhance user experience, expand interaction capabilities, and improve the practicality of earphone controls in diverse environments.
• Motivation and related work: Prior work explored touch-based, facial, oral, and mid-air gestures, but faced challenges like limited gestures, social acceptability, and noise interference. Active acoustic sensing methods require additional hardware and energy. This paper aims to achieve mid-air gesture recognition using passive acoustic sensing with existing earphone microphones.

Solution

• Proposed approach: FingerBar—a mid-air gesture recognition system for earphones that uses finger-generated friction sounds captured by microphones, without active signal transmission.
• Novelty:
  1. First mid-air gesture recognition system for earphones relying solely on passive acoustic sensing.
  2. Gesture filtering pipeline to resist environmental noise and exclude non-interactive operations.
  3. Adversarial training for user-independent gesture recognition without personalization.
  4. Real-time prototype evaluations demonstrating usability and robustness.
• Procedure and key techniques:
  • Audio preprocessing: Short-Time Fourier Transform (STFT) to generate spectrograms, cropping to retain 12–22 kHz frequency range, and energy-based filtering.
  • Gesture recognition: Deep learning model with ResNet-18 backbone and domain-adversarial neural network (DANN) for user-independent features.
  • Real-time implementation: Desktop and portable versions with latency optimization and low power consumption.

Results

• Concrete findings:
  • Achieved an average F1-score of 0.91 across quiet, noisy, and semi-wild conditions.
  • End-to-end latency: 16.9 ms (desktop) and 48.5 ms (portable).
  • Power consumption: 136 mW (prototype); estimated 25 mW for commercial deployment.
  • Robustness to noise, hand cleanliness, and minor occlusions; limited by gloves and full ear occlusion.
• Advantage over baselines:
  • Passive sensing eliminates the need for active signal transmission, reducing energy consumption and hardware requirements.
  • Superior user-independence and noise resistance compared to prior systems.
• Experiments / evaluation:
  • User studies: Gesture selection (N=32), modular evaluation (N=16), and usability testing (N=12).
  • Scenarios: Quiet labs, noisy offices, outdoor environments, and real-world activities.
  • Metrics: F1-scores, confusion matrices, latency, power consumption, and user feedback.
• Limitations and future work:
  • Performance degradation in high-frequency noise environments and with gloves.
  • Limited gesture set; potential for expansion with higher-SNR gestures.
  • Need for additional data collection for older adults and extreme scenarios.
  • Integration with commercial earphones and exploration of new deployment strategies.

Summary

FingerBar introduces a novel, passive acoustic-based mid-air gesture recognition system for earphones, leveraging friction sounds captured by existing microphones. It achieves high accuracy (F1-score: 0.91) and robustness across diverse conditions, with low latency and power consumption. User studies confirm its practicality and acceptability. Future work will address high-noise environments, expand gesture sets, and optimize deployment on commercial devices. This system offers a promising, user-friendly interaction method for wearable technology.