FaraPy: An Augmented Reality Feedback System for Facial Paralysis using Action Unit Intensity Estimation

Title of the Paper

FaraPy: An Augmented Reality Feedback System for Facial Paralysis using Action Unit Intensity Estimation

Paper Information

• Domain: Facial Paralysis Rehabilitation, Augmented Reality Technology, and Deep Learning
• Keywords: Facial Paralysis, Augmented Reality, Lightweight Model, Facial Action Unit Detection, Action Unit Intensity, Mirror Therapy
• Conference: The 34th ACM Symposium on User Interface Software and Technology (UIST '21)
• Publication Date: October 2021

Research Background and Problem

• Identified Issues or Challenges:

  1. Facial Paralysis (FP) significantly impacts patients' functionality, aesthetics, and psychological well-being. Traditional treatments (e.g., mirror therapy) face limitations such as lack of feedback mechanisms, insufficient user motivation, and inability to track recovery progress.
  2. There is a lack of standard datasets and models in the field of deep learning specifically targeting facial paralysis rehabilitation, particularly models capable of distinguishing unilateral facial action unit intensity.
  3. Existing deep learning models are often too complex to run in real-time on resource-constrained mobile devices.

• Significance of the Research:

  1. The large number of facial paralysis patients highlights the need for improved treatment methods to enhance functional recovery and quality of life.
  2. Real-time feedback-based home rehabilitation solutions can bridge the gap caused by limited offline treatment resources.

• Motivation and Related Work:

  1. While traditional mirror therapy is effective, it lacks technological support to clearly identify affected muscles or quantify recovery progress.
  2. Augmented reality (AR) technology has shown broad potential in medical fields such as surgery, rehabilitation, and training, but no applications specifically targeting facial paralysis currently exist.
  3. Research in deep learning has explored facial action unit (AU) detection, but most focus on bilateral predictions without addressing the need for unilateral muscle analysis.

Proposed Solution

• Proposed Method or Solution: FaraPy is a mobile augmented reality (AR) system that integrates a lightweight deep learning model, LW-FAU, with a real-time user feedback interface to support facial paralysis rehabilitation.

• Innovations:

  1. The first real-time AR-based mirror therapy feedback system.
  2. A novel lightweight deep learning model, LW-FAU, designed to independently predict action unit intensity (AUI) for both sides of the face.
  3. Development of a new facial paralysis dataset (FIFA) with unilateral action intensity labels.
  4. Integration of mirror therapy, muscle education, and gamification elements to enhance user engagement and rehabilitation motivation.

• Implementation Steps:

  1. Employ a knowledge distillation framework, using the complex FAU-Net model (teacher) to train the LW-FAU model (student).
  2. The LW-FAU architecture incorporates depthwise separable convolutions and multi-task learning to achieve efficient computation on mobile devices.
  3. Develop a user interface featuring AR facial overlay filters and real-time data visualization.
  4. Design and integrate a feedback mechanism and progress tracking system based on muscle symmetry scoring.

Research Outcomes

• Specific Outcomes:

  1. The LW-FAU model outperforms the state-of-the-art FAU-Net model in detecting action unit intensity and can run in real-time on mobile devices.
  2. The first dataset designed for facial paralysis, FIFA, was created to support future algorithmic research in this domain.
  3. User studies revealed that 85% of participants preferred using FaraPy for rehabilitation training, and 95% expressed willingness to continue using the system.

• Advantages Over Existing Solutions:

  1. The system supports unilateral facial action unit intensity detection with real-time feedback, addressing the limitations of traditional mirror therapy.
  2. AR technology reduces user anxiety related to mirror exposure, enhancing training effectiveness.
  3. Gamification design and progress tracking features significantly improve user engagement and rehabilitation motivation.

• Experimental or Evaluation Results:

  1. Technical evaluations demonstrated strong performance on the DISFA (healthy faces) and FIFA (facial paralysis) datasets, with low mean absolute error (MAE) and high intraclass correlation coefficients (ICC), indicating model reliability.
  2. User studies showed high scores across six user experience dimensions (attractiveness, transparency, efficiency, dependability, stimulation, novelty), with particularly high ratings for "stimulation" and "novelty."

• Limitations and Future Directions:

  1. The dataset and model require further expansion, such as incorporating more data from facial paralysis patients to improve model generalizability.
  2. Adding eye-tracking functionality to support syndrome-related training.
  3. The small sample size and experimental equipment limitations necessitate larger-scale user studies to validate the generalizability of the findings.
  4. Investigating the relationship between model performance and long-term rehabilitation outcomes.