EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses

Title of the Paper

EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses

Paper Information

• Field of Study: Low-power devices and facial expression tracking technology
• Keywords: Wearable glasses, facial expression tracking, acoustic sensing, low power, continuous tracking, human-computer interaction, frequency-modulated continuous wave (FMCW) technology, deep learning, scalability, real-time processing

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Existing technologies primarily rely on high-power sensors like cameras, making it difficult to achieve continuous facial expression tracking on lightweight devices such as glasses.
  • There is an asymmetry in capturing information from the upper and lower parts of the face, with lower facial expressions being particularly challenging to detect.
  • Limited battery life of devices, with traditional solutions consuming excessive power.
  • Performance stability in multi-scenario environments (e.g., re-wearing the device or during movement) remains a challenge.

• Why the Problem is Important:

  • Facial expressions are central to interactive applications (e.g., video calls, non-verbal communication, sign language), and tracking them can enhance user immersive experiences and expand interaction methods.
  • Tracking facial expressions using devices like smart glasses or augmented reality glasses is a promising and significant direction, enabling portable daily applications.

• Research Motivation and Related Work:

  • Previous wearable facial expression tracking solutions, such as cameras, earphones, or neck sensors, have not achieved continuous expression tracking.
  • Acoustic sensing technology has shown potential in static, discrete expression recognition but has not been developed into a high-resolution continuous facial expression tracking system on glasses.

Solution

• Proposed Solution:

  • EyeEcho System: A glasses-based device utilizing acoustic sensing, equipped with two sets of speakers and microphones to emit and receive acoustic signals. It captures subtle deformations of facial skin and analyzes these deformations to achieve continuous facial expression tracking.

• Innovations:

  • Employing FMCW (Frequency-Modulated Continuous Wave) audio signals to detect skin deformations, avoiding the use of cameras, thereby reducing power consumption and enhancing privacy.
  • Using deep learning (ResNet-34 model) to process differential echo data, enabling continuous prediction of 52 blended expression parameters.
  • Achieving lightweight and low-power device design for extended usage.
  • Real-time processing implemented on commercial smartphones, enhancing portability and application scenarios.

• Implementation Steps and Key Technologies:

  1. Hardware Design: Sensor modules, including MEMS microphones and speakers, are installed on the glasses' arms, with an embedded Bluetooth module for data transmission.
  2. Signal Processing: FMCW sensors detect skin deformations, compute echo profiles, and use differential processing to cancel background noise while detecting expression changes.
  3. Deep Learning Model: A predictive model trained on 52 blended parameters, optimized for mobile devices.
  4. Evaluation and Optimization: User testing and data augmentation improve robustness across various scenarios.
  5. Real-time Integration: Deploying a low-latency model on smartphones for real-time expression rendering and applications.

Research Outcomes

• Specific Results:

  • Performance:
    • Achieved a static sitting MAE (Mean Absolute Error) of 22.9; MAE of 26.9 in walking scenarios, maintaining high accuracy.
    • Tested in semi-natural environments (e.g., bedroom, living room, kitchen), achieving an average MAE of 44.4 (Day 1) and 45.5 (Day 2).
    • Continuous expression tracking demonstrated high stability across dynamic scenarios and after re-wearing the device.
  • Portability and Low Power Consumption:
    • Device power consumption is only 167mW, allowing approximately 14 hours of continuous use with a Google Glass battery; optimized to 71mW with efficient speakers.
    • Real-time processing achieved on Android smartphones, with end-to-end latency from sensing to prediction of just 34ms.
  • Application Expansion:
    • Enabled real-time visualization of expressions mapped to personalized 3D avatars.
    • Implemented health monitoring features, such as blink detection through periocular deformation, achieving an F1 score of 82%.

• Advantages Compared to Existing Solutions:

  • Compared to similar products like EarIO, this solution:
    • Demonstrates better MAE performance in expression tracking with significantly less training data (only 4 minutes required).
    • Provides blink detection capabilities and achieves longer-term performance stability.
    • The smart glasses integration is more suitable for long-term wear.

• Experimental or Evaluation Results:

  • Provides a rigorously validated solution with comparative performance under different frequency ranges and environmental noise conditions.
  • Outdoor low-temperature tests revealed the impact of cooling on sensor frequency response, offering insights for future research.

• Limitations and Future Directions:

  • Limitations:
    • Facial coverings (e.g., scarves or heavy beards) may interfere with signal transmission.
    • Performance may degrade during intense dynamic activities (e.g., running or vigorous movements).
  • Future Directions:
    • Explore higher frequency ranges and sensor optimizations to further reduce power consumption.
    • Extend experimental validation to diverse real-world scenarios and environments, such as outdoor or workplace settings.
    • Improve algorithms to enhance adaptability to various user characteristics.
    • Further strengthen power consumption control and privacy protection technologies.

For further details, please provide more specific requirements or additional questions!