InterFACE: Establishing a Facial Action Unit Input Vocabulary for Hands-Free Extended Reality Interactions, From VR Gaming to AR Web Browsing

Research Background and Problem

• Problems and Challenges: The authors highlight that current Virtual Reality (VR) and Extended Reality (XR) interactions typically rely on hand gestures or controller inputs, which are inaccessible to users with motor impairments or those temporarily unable to use their hands in specific scenarios. Additionally, alternative interaction methods such as voice input or eye tracking face issues like low social acceptability and limited functionality.
• Significance: VR and XR are considered to exhibit a degree of "ableism" in their design, which prevents some users from experiencing the benefits of these technologies, such as immersion or interaction capabilities.
• Research Motivation: Facial Action Units (FAUs) offer a hands-free interaction method, providing opportunities for users who cannot use traditional interaction methods and opening new possibilities for those temporarily or situationally unable to use their hands.
• Related Work: The authors draw on previous research that used facial expressions for game control, but these studies faced issues such as limited scope, unstable performance, and lack of generalizability. This paper aims to systematically evaluate and identify usable FAUs and apply them to a wide range of XR scenarios.

Proposed Solution

• Proposed Solution: The authors' solution includes:

  1. Systematically evaluating the performance of 53 FAUs supported by current consumer-grade VR devices (e.g., Meta Quest Pro) in terms of comfort, effort, and response time.
  2. Proposing a vocabulary of seven optimal FAUs to enable hands-free interaction with XR devices.
  3. Designing and evaluating two prototype applications that are fully operable through facial interaction: a VR game and an AR web browser.

• Innovations: The most notable innovation is the authors' development of a universal interaction method based on individual FAUs, rather than relying on emotional expressions or user-specific customization. This approach makes the method broadly adaptable for different users and provides a practical hands-free interaction solution for VR and AR design.

• Implementation Steps:

  1. Conduct user experiments to evaluate the subjective and objective performance of 53 FAUs, identifying suitable interactive FAUs.
  2. Train a classification model based on the selected FAUs to achieve high recognition accuracy (97% classification accuracy).
  3. Integrate the classifier into virtual environments for practical interaction testing, including a VR game and an AR browser.
  4. Conduct user evaluations to assess the performance and user acceptance of facial interaction in real-world scenarios.

Research Outcomes

• Specific Outcomes:

  1. Identified FAUs suitable for XR interaction (e.g., JawDrop, LidTightener).
  2. Trained an efficient classification model capable of accurately distinguishing between resting and activation states (overall F1 score of 0.95).
  3. Successfully developed two XR prototype applications driven entirely by facial interaction.
  4. Provided preliminary validation of the feasibility of facial interaction in practical use, particularly its potential to enhance XR accessibility and hands-free interaction.

• Advantages Compared to Alternatives:

  • Compared to voice- or eye-tracking-based interactions, facial interaction offers greater controllability and input dimensions, making it better suited for complex interaction tasks.
  • For users with permanent or temporary motor impairments, facial interaction provides a non-invasive and easily accessible solution.

• Experimental or Evaluation Results:

  • User experiments showed that facial interaction performs well in certain scenarios (e.g., web browsing) but may cause facial fatigue during prolonged use.
  • Social acceptability remains a concern, as users are more inclined to use facial interaction in private or semi-private settings rather than public spaces.

• Limitations and Future Directions:

  1. Headset Sensor Limitations: The current devices' recognition range and accuracy may limit the use of certain FAUs. Future research should explore new facial tracking technologies (e.g., tongue tracking and more composite actions).
  2. Target User Groups: The current study primarily focuses on non-disabled users and has yet to explore how to optimize facial interaction for specific disability groups.
  3. Interaction Design Improvements: Enable more personalized adjustments, such as user-selectable FAU bindings and customizable operation schemes.
  4. Fatigue Issues: Future research should further quantify the fatigue or health impacts of prolonged use of certain FAUs.
  5. Concurrent Inputs: Currently, multiple FAUs cannot be activated simultaneously. Future work could explore multi-label classification or combine facial interaction with other input methods (e.g., voice, gaze interaction).

Through this research, the authors demonstrate that hands-free interaction based on Facial Action Units has broad application potential. It not only addresses accessibility challenges in existing XR technologies but also opens new possibilities for hands-free interaction in everyday life. This provides important insights for the design of accessible technologies in XR and the broader field of human-computer interaction.