SoundBubble: Finger-Bound Virtual Microphone using Headset/Glasses Beamforming

Paper Title

SoundBubble: Finger-Bound Virtual Microphone using Headset/Glasses Beamforming

Publication Info

• Topic area: Human-Computer Interaction (HCI) and acoustic sensing for XR devices.
• Keywords: acoustic beamforming, XR headset, virtual microphone, finger tracking, vibro-acoustic sensing, signal-to-noise ratio, passive interaction, wearable technology, augmented reality, machine learning.

Background and Problem

• Problem / challenge: Existing vibro-acoustic sensing systems often require users to wear specialized accessories like rings, wristbands, or watches, which may not be practical or scalable. These devices also face challenges such as limited battery life and difficulty isolating hand-generated sounds from background noise.
• Significance: Enabling robust, bare-hand sensing without additional accessories could improve user experience, reduce hardware requirements, and expand interaction possibilities in XR environments.
• Motivation and related work: Prior work in HCI has explored vibro-acoustic sensing using wearable devices, but these approaches often suffer from proximity vs. accuracy trade-offs and require instrumentation of the hands or objects. Acoustic beamforming has been used for sound localization and separation, but vision-guided acoustic beamforming for hand-centric interactions has not been explored.

Solution

• Proposed approach: SoundBubble—a vision-guided acoustic beamforming technique that creates virtual microphones at the user’s fingertips using an XR headset or glasses equipped with microphone arrays.
• Novelty:
  1. Vision-guided acoustic beamforming to isolate finger-generated sounds in noisy environments.
  2. Bare-hand sensing without requiring additional accessories or instrumented objects.
  3. Integration of spatial sound pressure level (SPL) maps and machine learning for robust interaction detection.
  4. Demonstration of diverse use cases, including ad hoc touch input, micro-gestures, passive widget interactions, and held object activity recognition.
• Procedure and key techniques:
  • Use XR headset cameras for 3D hand tracking to guide beamforming.
  • Apply delay-sum beamforming to align multichannel microphone signals and suppress background noise.
  • Generate SPL maps and spectrograms as input features for machine learning models.
  • Evaluate system performance across multiple tasks and noise conditions.

Results

• Concrete findings:
  • Overall true positive rate: 94.7%; false positive rate: 6.8%.
  • Ad hoc touch input: 97.5% true positive rate; 5.5% false positive rate.
  • Passive widget interaction: 97.1% true positive rate; 5.1% false positive rate.
  • Held object activity recognition: 89.4% true positive rate; 9.9% false positive rate.
  • Background noise (up to 100× louder than target sounds) reduced accuracy by only ~2.9%.
• Advantage over baselines:
  • Single-microphone systems showed significantly lower accuracy and higher false positive rates, especially in noisy environments.
  • Beamforming improved signal-to-noise ratio (SNR), enabling detection of subtle hand-generated sounds.
• Experiments / evaluation:
  • Three tasks evaluated: ad hoc touch input (4 surfaces), passive widget interaction (6 widgets), and held object activity recognition (8 objects).
  • Conducted under three noise conditions: typical office noise, music + cafe noise, and user talking.
  • Ablation study simulated different microphone array geometries and channel counts to assess performance trade-offs.
• Limitations and future work:
  • Noisy environments remain challenging despite improved SNR.
  • Compact glasses-like form factors may limit microphone count and array geometry, affecting accuracy.
  • Power consumption and hardware integration need optimization for commercial deployment.
  • Future work could explore new use cases, advanced beamforming techniques, and audio augmented reality applications.

Summary

SoundBubble introduces a novel vision-guided acoustic beamforming approach that enables robust, bare-hand vibro-acoustic sensing using XR headsets or glasses. By creating virtual microphones at the user’s fingertips, the system achieves high accuracy across diverse interaction tasks, even in noisy environments. Experiments demonstrate its efficacy compared to single-microphone systems, and an ablation study highlights the impact of microphone array design. SoundBubble opens new possibilities for XR applications, including ad hoc touch input, micro-gestures, and passive object interactions, while addressing limitations of prior wearable sensing systems.