PalateTouch : Enabling Palate as a Touchpad to Interact with Earphones Using Acoustic Sensing

Research Background and Issues

• Identified Problems and Challenges
  The authors highlight the limitations of existing headphone interaction methods, such as gestures or touch controls. Gesture-based interactions can cause arm fatigue and are unsuitable for elderly or disabled users, while voice input raises privacy concerns and is susceptible to environmental noise interference. Additionally, current interaction methods based on oral activities or ear canal deformation require expensive hardware, are inconvenient to operate, or fail to meet the needs of daily use.

• Significance
  As headphones become ubiquitous in daily life, developing interaction technologies that are privacy-preserving, low-cost, non-invasive, and adaptable to different user habits is increasingly important.

• Research Motivation and Related Work
  Based on sensors embedded in headphones, the authors propose a novel method that uses acoustic sensing to identify tongue-palate touch positions for interaction. By studying oral interaction systems, ear canal deformation-based techniques, and acoustic sensing technologies, the authors identify significant room for improvement in usability, cost, and privacy protection.

Solution

• Method or Solution
  The authors propose the PalateTouch system, whose core idea is to treat the "palate" as a touchpad and the "tongue" as a clicking tool. Using the built-in microphone and speaker in headphones, the system emits ultrasonic signals and collects the reflected signals to detect subtle deformations in the ear canal caused by oral activities. These deformations are then used to identify different tongue-palate touch gestures.

• Innovations

  1. Introduced an acoustic sensing method based on headphone sensors, eliminating the need for additional hardware.
  2. Used a normalized transfer function to mitigate the effects of wearing styles and individual differences, enabling user-independent gesture recognition.
  3. Designed a real-time gesture recognition pipeline to accurately identify gestures by monitoring the temporal changes in tongue-palate touch positions.

• Implementation Steps and Key Technologies

  1. Signal Generation: Designed Zadoff-Chu (ZC) sequences as probing signals and optimized signal characteristics using frequency domain modulation techniques.
  2. Ear Canal Transfer Function Extraction: Extracted the frequency response of ear canal reflections by removing direct sound interference and computed a normalized transfer function.
  3. Gesture Recognition Pipeline:
     • Determined whether the tongue is touching the palate and identified seven touch positions.
     • Matched position sequences to templates to recognize sliding and tapping gestures.
  4. Automated Calibration: Designed a standardized domain automated calibration algorithm to address the impact of wearing position and re-wearing on system performance.

Research Outcomes

• Specific Results

  1. Successfully recognized 9 gestures (5 tapping gestures and 4 sliding gestures) with an average F1 score of 0.92 and a false positive rate of only 0.02.
  2. Demonstrated excellent user-independent recognition performance in experiments with 16 participants.
  3. Showed high usability in real-time applications (e.g., video playback software).
  4. Evaluated the system under various environmental noise conditions (e.g., restaurant and outdoor noise) and body posture changes, demonstrating robust performance.

• Advantages Compared to Existing Solutions

  1. Fully based on built-in headphone sensors, compatible with existing headphone hardware, and requires no additional devices.
  2. Offers higher interaction concealment, suitable for privacy-sensitive scenarios.
  3. Provides automated calibration functionality, supporting different users and repeated wearing.

• Experimental or Evaluation Results

  1. Maintained stable performance across different wearing styles (initial wearing, re-wearing, etc.) through automated calibration, achieving an F1 score of 0.916 after re-wearing.
  2. Performance was only slightly affected under noisy environments (e.g., 55 dB laboratory noise, 91 dB bar noise).
  3. Supported operation under various body postures (e.g., looking down, walking) and background music playback without significant accuracy degradation.
  4. In user studies, the system achieved a System Usability Scale (SUS) score of 85.7/100, indicating high user satisfaction.

• Limitations and Future Directions

  1. Wireless Headphone Implementation: The current system uses wired connections; future work should develop a wireless version to align with mainstream wireless headphones.
  2. High-Intensity Activities: The system's performance during high-intensity activities (e.g., running) remains insufficient and requires further robustness improvements.
  3. Power Consumption: The current implementation has high power consumption, necessitating hardware circuit optimization and power reduction.
  4. Interaction Expansion: The current set of defined gestures includes 9 types; future work could expand the interaction command set through more precise tongue-tip trajectory recognition.

Conclusion

PalateTouch overcomes the limitations of previous oral activity-based interaction technologies by proposing a low-cost, high-privacy, interference-resistant, and user-independent headphone interaction method. Future research and development efforts to optimize power consumption and robustness, expand gesture interaction capabilities, and achieve wireless device compatibility are expected to make this system more applicable to various real-life scenarios.