MELDER: The Design and Evaluation of a Real-time Silent Speech Recognizer for Mobile Devices

Document Title

MELDER: The Design and Evaluation of a Real-time Silent Speech Recognizer for Mobile Devices

Document Information

• Subject Area: Human-Computer Interaction, Silent Speech Recognition Technology
• Keywords: Silent Speech, Digital Lip Reading, Image Processing, Deep Learning, Transfer Learning, Language Modeling, Visual Feedback, Text Input

Research Background and Problem Statement

• Problems and Challenges:

  • Current silent speech recognition systems primarily operate on desktop devices, are slow in processing, lack real-time capabilities, and are difficult to adapt for mobile devices.
  • Existing models support only a limited vocabulary, lacking general applicability for everyday communication.
  • Training deep learning models requires large datasets, and building large-scale, domain-specific vocabulary datasets is time-consuming and expensive.
  • There is a lack of research on interfaces and feedback mechanisms for silent speech interaction on mobile devices.
  • Privacy and security demands are heightened in noisy or sensitive environments.

• Significance:

  • Silent speech input has potential applications, such as enhancing privacy, usability in noisy environments, and supporting individuals with speech impairments.
  • Improving model real-time performance and adapting it for mobile devices can facilitate its widespread use in daily communication.

• Research Motivation and Related Work:

  • A comparison of various sensor-based silent speech recognition methods (e.g., electrode implants, brain interfaces) and video-based lip-reading techniques revealed that the former traditionally has invasive and non-portable drawbacks, while the latter holds significant potential for mobile applications.
  • Existing lip-reading models (e.g., LipNet, Transformer) still require optimization for real-time performance and mobile scenarios.

Solution

• Methods and Innovations:

  1. Proposed MELDER (Mobile Lip Reader), a real-time silent speech recognition system optimized for mobile devices.
  2. Utilized short time window segmentation of input videos, processing them incrementally to balance real-time performance and accuracy.
  3. Introduced transfer learning methods to extract knowledge from high-resource vocabulary models, enhancing applicability to everyday communication vocabulary.
  4. Integrated a real-time visual feedback mechanism into the model to display the progress of recognition to users.
  5. Open-sourced data and code for community validation and dissemination.

• Implementation Steps and Key Techniques:

  1. Developed a pipeline processing framework based on "video slicing + parallel recognition" architecture.
  2. Used the LipType model as a foundation, incorporating optimized 3D-CNN and SE-ResNet modules to enhance temporal and spatial feature extraction.
  3. Applied three transfer learning strategies (e.g., freeze and fine-tune) to optimize pre-trained models for low-resource target vocabulary datasets.
  4. Designed a hybrid server-client processing architecture to balance performance optimization and device operational capabilities, considering mobile device constraints.

Research Outcomes

• Experiments and Specific Results:

  1. Window Function Optimization Experiment: Evaluated different video segmentation functions, identifying the optimal linear function (𝑦 = 𝑥 + 5) for the best balance between speed and accuracy.
  2. Transfer Learning Strategy Experiment: Verified that the "transfer + fine-tune" method (Finetune_Sequence) significantly improved pre-trained model performance on low-resource vocabulary.
  3. Static Performance Evaluation: Compared to other models, MELDER reduced word error rate and processing time by 6% and 50%, respectively.
  4. Applicability in Mobile Scenarios: MELDER continued to outperform comparison models during walking scenarios, particularly in speed (-46%) and error rate (-6%).
  5. Feedback Method Evaluation:
     • In a user experience comparison with Google Assistant, MELDER demonstrated significant advantages in privacy and security.
     • Phrase-level feedback design improved users' subjective evaluation of model performance (despite no significant difference in actual performance).

• Advantages Comparison:

  • MELDER significantly outperformed existing silent speech recognition methods in real-time performance and exhibited superior accuracy and user interaction experience.
  • The dual-channel architecture (transcriber + reviewer) and visual feedback design enhanced the model's user adaptability and interpretability.

• Limitations and Future Directions:

  • The system still relies on server-side processing, which may face challenges under poor network conditions.
  • Optimization is needed for complex scenarios such as dynamic lighting and multi-user interactions.
  • Plans to expand support for non-English datasets and improve adaptability for disadvantaged users (e.g., individuals with speech impairments).
  • Future research will focus on manual error correction, algorithm acceleration, and model robustness.

For access to the complete experimental code and data, please visit the publicly available resources (links provided in the document).