EchoBreath: Continuous Respiratory Behavior Recognition in the Wild via Acoustic Sensing on Smart Glasses

Research Background and Problem Statement

• Identified Problems or Challenges:

  1. There is currently a lack of unobtrusive and reliable methods suitable for real-world environments to monitor abnormal respiratory behaviors such as coughing, sneezing, and sniffing.
  2. Traditional passive acoustic sensing methods often fail to distinguish between users' abnormal respiratory behaviors and environmental noise (limited subject-sensing capability), which may lead to false reports.
  3. Many existing methods have limited applicability in noisy environments or in the presence of non-target behaviors such as eating or speaking.
  4. Prolonged use of hardware devices such as headphones may cause user discomfort or inconvenience and appear conspicuous in social settings.

• Significance: Monitoring the frequency and duration of respiratory symptoms such as coughing, sniffing, and sneezing is crucial for personal health. It can be used for the early detection of chronic diseases (e.g., chronic obstructive pulmonary disease and asthma) or for predicting health risks.

• Research Motivation and Related Work:

  • Existing acoustic-based monitoring technologies often focus on a single modality (e.g., passive acoustic sensing), making it difficult to distinguish between user behaviors and environmental noise sources.
  • While some studies have integrated multimodal sensors into headphone platforms, the issue of long-term wearability remains unresolved.
  • This study proposes the design of a respiratory monitoring solution based on smart glasses, which combines subject-sensing capability, unobtrusiveness, durability, and user-friendliness to address the above challenges.

Solution

• Proposed Method or Solution: This study introduces EchoBreath, a monitoring system that combines active and passive acoustic sensing, embedded in smart glasses. It utilizes the glasses' speakers and microphones to emit and capture ultrasonic echoes to analyze respiratory behaviors.

• Innovative Aspects of the Solution:

  1. Integration of Active and Passive Sensing: Active acoustic sensing uses ultrasound to capture subtle facial movements and airflow changes caused by breathing, enhancing subject-sensing capability and noise resistance in complex scenarios.
  2. Lightweight Neural Network: A multimodal convolutional neural network (CNN) optimized for mobile devices is employed, using a "Null" class and open-set decision mechanisms to filter out non-target activities.
  3. Smart Glasses Hardware Integration: The sensing components (speakers and microphones) are embedded into the glasses' frame, making them suitable for prolonged wear without causing social discomfort.

• Implementation Steps and Key Technologies:

  1. Data Acquisition:
     • The glasses' speakers emit ultrasonic signals and capture active echo signals while simultaneously recording passive acoustic data.
  2. Feature Extraction:
     • Passive acoustics: A band-pass filter of 10kHz-16kHz extracts target frequency information, which is converted into STFT spectrograms.
     • Active acoustics: The correlation spectrum of ultrasonic echo signals is calculated, retaining specific frequency bands that reflect users' facial airflow and movements.
  3. Neural Network Inference:
     • Active and passive acoustic features are processed independently through a lightweight CNN.
     • A fusion strategy dynamically weights the results of the two sensing modalities, combining the "Null" class and open-set filtering mechanisms to exclude non-respiratory interferences.
  4. Evaluation and Optimization:
     • User-dependent and user-independent models are designed, and experiments validate performance under noisy and non-target behavior conditions.

Research Outcomes

• Specific Results:

  1. In laboratory settings, EchoBreath achieves an accuracy of 93.1% in identifying six typical respiratory behaviors (nasal breathing, mouth breathing, coughing, sneezing, sniffing, and throat clearing).
  2. In semi-outdoor tests (e.g., at home, in offices, and in restaurants), the overall accuracy remains between 82.8% and 89.8%, demonstrating the system's robustness.
  3. The integration of active acoustic sensing significantly enhances noise resistance and subject-sensing capability (e.g., accuracy improves by approximately 10% in the presence of external coughing interference).

• Comparison with Existing Solutions and Advantages:

  1. Compared to existing systems that rely solely on passive acoustic sensing, EchoBreath improves subject-sensing capability in multi-participant scenarios (accuracy under interference increases from 72% to 85%).
  2. Compared to methods based on headphones or other wearable devices, the glasses form factor offers superior comfort and privacy protection (no direct skin contact and frequency privacy safety).

• Experimental or Evaluation Results:

  1. In complex scenarios with non-target interferences (e.g., external coughing, restaurant noise), the system combining active and passive sensing performs best.
  2. By fine-tuning user-dependent models with just 10 minutes of specific user data, accuracy can be improved from a baseline of 66.4% to 89.8%.

• Limitations and Future Directions:

  1. Uncovered Symptom Categories: Currently, only six types of respiratory behaviors are supported. Future work could expand to more fine-grained respiratory symptom classifications.
  2. High-Intensity Motion Interference: The system's performance degrades during intense user movements. Future plans include improving robustness through advanced signal processing algorithms.
  3. Experimental Scenario Limitations: Experiments were primarily conducted in simple environments such as homes and offices. Future validation in more complex real-world settings, including hospitals and multi-user high-frequency environments, is needed.
  4. Development of Multimodal Integration: Further exploration of integration with other health platforms (e.g., smartwatches) is planned to build a more comprehensive health monitoring ecosystem.

In summary, EchoBreath provides a low-power, multimodal sensing respiratory monitoring solution based on smart glasses, addressing technical challenges in subject-sensing and noise resistance. It holds significant potential for application in complex real-world environments.