Document Title

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users

Document Information

• Domain: Human-Computer Interaction and Machine Learning
• Keywords: Customizable Sound Recognition, Few-Shot Learning, Technology for the Deaf, Environmental Sound Recognition, Sound Perception, Mobile Applications

Research Background and Problem Statement

• What problems or challenges did the authors identify?

  1. Current sound recognition systems rely on pre-trained generic models, which fail to meet the diverse needs of hearing-impaired users.
  2. Existing systems lack support for user-customized sound categories, such as household-specific sounds (e.g., a child’s voice or a pet’s bark).
  3. Generic models cannot adapt to the diversity of audio environments in daily life, such as transitions between indoor and outdoor settings.

• Why is this problem important?

  1. Sound recognition systems can provide critical alerts about the environment, activities, and emergencies for hearing-impaired users, improving their quality of life and safety.
  2. Personalization and environmental adaptability can significantly enhance system accuracy and user satisfaction.

• Motivation and related work:

  1. Based on previous research and a survey of 472 hearing-impaired participants, the authors identified a strong demand for personal sound recognition technology.
  2. The authors conducted an in-depth analysis of the limitations of existing sound recognition tools and their shortcomings in real-world scenarios.
  3. Inspired by prior research on few-shot learning, the authors proposed a more user-friendly system design and implementation.

Solution

• What methods or solutions did the authors propose?

  1. ProtoSound System: A few-shot learning model that customizes sound categories based on a small number of user-provided samples.
  2. Real-Time Personalized Model: Enables real-time model customization on mobile devices using a small amount of audio recordings.
  3. Context Generalization Techniques: Introduces data augmentation methods to adapt the model to background noise in different environments.
  4. Open-Set Recognition: Allows the system to identify unknown sound categories.

• What are the innovative aspects of this solution?

  1. Proposed the first scalable sound recognition system with low user involvement costs.
  2. The system design integrates the technical features of few-shot learning while addressing personalization and real-time operation needs.
  3. Supports the recognition of hard-to-record target sound categories by enhancing recognition capabilities with a built-in online resource library.

• What are the implementation steps? What key technologies were used?

  1. Data Collection: Users record a small number of samples for model training, requiring approximately 5 samples per category.
  2. Model Training: Features are extracted using the MobileNetV2 architecture, and category prototypes are generated through a prototypical network.
  3. Prediction Phase: Compares test audio with category prototypes and uses a nearest-neighbor classifier to predict sound categories.
  4. Enhanced Features: Includes context generalization (via background noise data augmentation), open-set classification (recognizing unknown categories), and real-time end-to-end application deployment.

Research Outcomes

• What specific results were achieved?

  1. Developed the ProtoSound system, implemented through open-source Python code and an Android application.
  2. Conducted experiments on two real-world datasets, demonstrating that the system outperforms other methods in accuracy.
  3. Field studies showed that ProtoSound can handle diverse environments and personalized sound categories, with an average user recording time of 10 minutes and a prediction accuracy of 87.4%.

• What advantages does it have compared to existing solutions?

  1. Enables real-time deployment of few-shot learning, allowing users to train models on mobile devices without requiring extensive computational resources.
  2. Supports personalization, addressing the limitations of generic models in handling user-specific sounds.
  3. Achieves higher accuracy (+9.7%) compared to current few-shot learning and traditional supervised learning methods.

• What were the experimental or evaluation results?

  1. Experiment 1: Achieved 90.4% accuracy on sound data from hearing-impaired participants, significantly outperforming supervised learning baseline models.
  2. Experiment 2: Achieved accuracy close to human labeling (average 91.3%) on detected sounds in real-world scenarios.
  3. Experiment 3: Field studies demonstrated that ProtoSound operates effectively in various locations (e.g., homes, restaurants, streets), with positive user feedback.

• Limitations and future directions:

  1. Limited Categories: Current experiments focus on a 5-way setup (5 categories); future work could expand to more categories.
  2. User Interface Improvements: Further optimization is needed for the interface used for sound recording and labeling, especially for hearing-impaired users.
  3. Long-Term Deployment Evaluation: Current research focuses on short-term evaluations; future studies should assess performance over extended periods and across multiple scenarios.
  4. Socio-Cultural Impact: Consider the diverse needs and cultural preferences of hearing-impaired user groups, including their acceptance of sound perception technology.

Open-Source Resources and Contributions

1. Python and Android open-source implementation: https://github.com/makeabilitylab/ProtoSound
2. Integration of open sound libraries and user interaction design.
3. Provided a dataset and evaluation methods to support multi-scenario research needs.