Watch Your Mouth: Silent Speech Recognition with Depth Sensing

Title of the Paper

Watch Your Mouth: Silent Speech Recognition with Depth Sensing

Paper Information

• Domain: Human-Computer Interaction, Deep Learning, Depth Sensing Technology, Silent Speech Recognition
• Keywords: Silent Speech Recognition, Visual Speech Recognition, Lip Reading, Depth Sensing, Deep Learning, Input Technology

Research Background and Problem

• Identified Problems or Challenges:
  Traditional silent speech recognition technologies rely on RGB cameras or other sensing methods (such as ultrasound imaging and electromagnetic sensors), which are highly susceptible to environmental factors (e.g., lighting changes or device orientation) and pose privacy risks.

• Significance:
  Silent speech recognition technology enables private human-computer interaction, particularly in applications for smart devices (smartwatches, AR/VR glasses, or IoT devices). With the advancement of large language models (LLMs), this technology can significantly enhance device interaction capabilities, eliminating the limitations of traditional interfaces.

• Research Motivation and Related Work:
  Depth sensors, which are insensitive to lighting changes, adaptable to different skin tones, and maintain data consistency despite changes in device position, are considered a promising solution to address these challenges. However, traditional methods lack generalization across different device positions and require extensive data calibration. The authors propose a depth sensor-based silent speech recognition technology to overcome these issues.

Solution

• Proposed Method or Solution:
  A novel depth sensor-based silent speech recognition method, "Watch Your Mouth," is introduced. This method captures users' lip movements using depth data and converts point cloud data into commands and sentences through a deep learning pipeline (PointVSR).

• Innovations:

  1. For the first time, depth sensors are employed for silent speech recognition, eliminating sensitive RGB data.
  2. By converting depth maps into point cloud data, the model's adaptability to different users and device position changes is enhanced.
  3. A deep learning pipeline combining signal alignment, point cloud feature extraction, and temporal sequence decoding is proposed.

• Implementation Steps and Key Techniques:

  1. Data Collection: Depth data of users' lip movements is captured using the iPhone TrueDepth camera.
  2. Lip Segmentation: The YOLOv7 object detection model is used to extract the lip region, removing irrelevant information.
  3. Depth Map to Point Cloud Conversion: Depth maps are converted into 3D point clouds, and normal vectors are calculated to enhance local features.
  4. Deep Learning Pipeline Design:
     • Includes 4D point cloud convolution layers, Transformer layers, and Bi-GRU layers, combined with Connectionist Temporal Classification (CTC) loss for sentence and command recognition.
     • TNet module is used for point cloud alignment, improving robustness to device position changes.

Research Outcomes

• Specific Results:

  1. Achieved high recognition accuracy across three sensor positions (wrist-mounted devices, head-mounted devices, environmental devices), demonstrating the feasibility of the method.
  2. For command recognition, the average accuracy reached 91.33% (within users), and when generalized to unseen users, accuracy remained at 74.88%.
  3. For sentence recognition, the character error rate (CER) within users was 4.13%, and the word error rate (WER) was 8.06%; for inter-user scenarios, CER and WER were 18.28% and 29.14%, respectively.

• Comparison with Existing Solutions:

  • In within-user scenarios: CER and WER were reduced by 48.05% and 38.10%, respectively, compared to mainstream RGB video methods.
  • In inter-user scenarios: CER and WER were reduced by 5% and 4.57%, respectively.
  • The model has fewer parameters (20M vs. 250M for RGB models) and higher training efficiency.

• Experimental or Evaluation Results:
  Errors in command and sentence recognition were mainly concentrated in the following types:

  • Substitution errors: 92.2%.
  • Deletion errors: 6.0%.
  • Insertion errors: 1.7%.

• Limitations and Future Directions:

  1. The sample size of participants is small; future studies should expand the participant pool to investigate the impact of population variability on results.
  2. System deployment is currently limited to laboratory environments; future testing in natural language interaction scenarios is needed.
  3. Performance of mid-to-low-end depth sensors has not been tested; further research is required to explore cost-effectiveness.
  4. The dataset size is relatively small; developing large-scale depth datasets is necessary to enhance model performance.

Conclusion

This paper proposes a depth sensor-based silent speech recognition technology, providing an efficient and reliable input method for smart devices. Experimental results show that the proposed method outperforms traditional RGB video-based methods in both command and sentence recognition and demonstrates adaptability across various device positions. The study further emphasizes the privacy protection advantages and potential societal applicability of depth sensors, paving the way for future research in related fields.