FaceSight: Enabling Hand-to-Face Gesture Interaction on AR Glasses with a Downward-Facing Camera Vision

Document Title

FaceSight: Enabling Hand-to-Face Gesture Interaction on AR Glasses with a Downward-Facing Camera Vision

Document Information

• Domain: Augmented Reality (AR), Human-Computer Interaction, Gesture Recognition
• Keywords: Hand-to-face gestures, AR glasses, computer vision, infrared camera technology, interaction design

Research Background and Problem

• Identified Problems or Challenges:

  • Existing hand-to-face gesture interaction technologies primarily rely on electrical or audio signals, supporting only simple or discrete gestures and failing to recognize rich, continuous gestures.
  • Current head-mounted display devices, though equipped with cameras, are rarely utilized for advanced hand-to-face gesture recognition, leaving the potential of these devices underexplored.

• Significance:

  • Hand-to-face interaction is intuitive and user-friendly in augmented reality scenarios, leveraging facial tactile feedback to enable efficient, eyes-free operations.
  • AR glasses integrate various sensors, offering significant potential for recognizing complex gestures and enhancing interaction experiences.

• Motivation and Related Work:

  • The authors reviewed existing hand-to-face interaction technologies and camera deployment methods in head-mounted devices, highlighting the need for a compact sensing solution capable of recognizing rich gestures.
  • They discussed the limitations of previous gesture recognition approaches using electrical signals, audio signals, or RGB cameras, leading to the demand for infrared camera technology.

Solution

• Method and Solution:

  • The "FaceSight" system is proposed, featuring a downward-facing infrared camera mounted on the nose bridge of AR glasses to capture facial and hand behaviors.
  • An algorithmic pipeline is designed, including facial region segmentation, hand-face contact detection, gesture classification, and continuous input parameter computation (e.g., nose deformation estimation).

• Innovations:

  • The unique camera placement and infrared light setup provide high-resolution images of the face and hands while minimizing privacy concerns and background interference.
  • Supports 21 complex hand-to-face gestures, including 10 novel, creative gestures previously undescribed.
  • Offers a compact and socially acceptable interaction design solution for AR glasses.

• Implementation Steps and Key Technologies:

  • Hardware Design: A wide-angle infrared camera is used, secured with a 3D-printed mount.
  • Algorithmic Pipeline:
    • Facial regions (nose, mouth, cheeks, etc.) and hands are segmented using image brightness features.
    • Hand-face contact is detected by identifying overlaps and motion changes between fingers and facial regions.
    • A convolutional neural network (CNN) model classifies gesture types and estimates nose deformation or finger movement trajectories.

Research Outcomes

• Specific Results:

  • FaceSight achieved an overall gesture classification accuracy of 83.06%, demonstrating its effectiveness.
  • Developed five augmented reality applications compatible with 21 hand-to-face gestures (including home screen, video player, photo gallery, contacts, and voice assistant).
  • Proposed various interaction techniques, such as fast-forwarding a video by nudging the nose wing or activating the voice assistant by covering the lips.

• Advantages:

  • Compared to existing solutions, FaceSight detects more complex and diverse gestures with a more compact interaction form.
  • Infrared light sources provide stable image quality and reduce privacy concerns.

• Experimental or Evaluation Results:

  • Hand-to-face touch detection achieved a recall rate of 97.90%, with an average position recognition accuracy of 94.69%.
  • Gesture classification achieved an average accuracy of 96.42%, and nose-pushing pressure classification reached 94.12%.

• Limitations and Future Directions:

  • Limitations:

    • Infrared sensing is susceptible to interference from environmental infrared light, especially in outdoor settings.
    • The current gesture set was designed by the authors; future studies should conduct cross-cultural user research to optimize the design.
    • Computer vision algorithms currently run on servers, requiring further research into lightweight algorithms for local computation on AR glasses.

  • Future Directions:

    • Explore more robust algorithms to adapt to outdoor and complex environments.
    • Integrate multimodal information, such as facial expression recognition or audio signals, to enhance contact detection performance.
    • Customize gesture-to-function mappings to meet diverse user needs.

Conclusion

FaceSight introduces a rich hand-to-face interaction method by deploying a downward-facing infrared camera on AR glasses. This study demonstrates the potential of FaceSight in augmented reality interactions and outlines directions for future improvements, laying the foundation for the development of hand-to-face gesture interaction technologies.