Hand Interfaces: Using Hands to Imitate Objects in AR/VR for Expressive Interactions

Title of the Paper

Hand Interfaces: Using Hands to Imitate Objects in AR/VR for Expressive Interactions

Paper Information

• Subject Area: Human-Computer Interaction and Augmented/Virtual Reality (AR/VR)
• Keywords: AR/VR, Interaction Design, Imitation, Experiential Interaction, Freehand Interaction, Embodied Interaction

Research Background and Problem Statement

• Identified Problems or Challenges: Traditional AR/VR interactions often rely on handheld controllers, which limit dynamic and expressive interaction scenarios. While existing research has explored the use of hands as interaction mediums, it primarily focuses on hands as controllers or contact surfaces, neglecting the three-dimensional expressiveness of human hands.
• Research Significance: With the widespread application of AR/VR technologies in education, entertainment, healthcare, and other fields, there is an urgent need for natural and expressive user interaction methods. Simulating real objects for interaction can combine both practicality and enjoyment.
• Research Motivation and Related Work: Analysis of existing literature reveals that although some studies have achieved controller-free hand gesture interactions, there is a lack of exploration into fully utilizing the three-dimensional structure of human hands to lead AR/VR interactions. Inspired by real-world activities such as the "rock-paper-scissors" game, this research explores using hands to imitate virtual objects to complete interaction tasks.

Proposed Solution

• Proposed Solution:

  • A novel interaction technique called "Hand Interfaces" is proposed, allowing users to mimic real objects with their hands to complete interaction tasks. This method enables dynamic binding between hands and objects.
  • The study demonstrates the applicability of this approach in object retrieval and interaction control tasks through the design of 28 "hand interfaces."

• Innovations:

  • Creatively transforms users' hand movements to simulate various three-dimensional virtual objects, such as fingers mimicking joysticks or switches, enabling more intuitive, spontaneous, and efficient interactions.
  • Avoids ambiguity caused by single grasping actions by designing specific gestures for complex objects, enhancing precision and expressiveness.

• Implementation Steps and Key Technologies:

  1. Design Phase: Develop a rich interaction design space, iteratively optimizing based on the morphological similarity of hand gestures, movement similarity, comfort, and socio-cultural acceptability.
  2. Implementation Phase: Develop a gesture recognition algorithm based on the hand-tracking functionality of Oculus Quest, including gesture template matching and a weighted sensitivity matrix for key points.
  3. Evaluation Phase: Implement 28 hand interfaces and conduct user studies comparing them with traditional interface designs and existing freehand interaction technologies to evaluate their interaction effectiveness.

Research Outcomes

• Specific Outcomes:

  • Designed 28 Hand Interfaces covering various application scenarios in entertainment, education, and ubiquitous computing.
  • Developed an open-source implementation toolkit to support future design efforts.
  • Validated the feasibility and innovativeness of the approach in object retrieval and interaction control tasks through experiments.

• Advantages Compared to Existing Methods:

  • Hand Interfaces demonstrate significant advantages over traditional dropdown menus or virtual grasping technologies in terms of realism, freedom, and expressiveness of interaction.
  • Provides natural tactile feedback and proprioception, enhancing the precision of the user experience.

• Experimental or Evaluation Results:

  • Two user studies explored the object retrieval and interaction control capabilities of the technique, collecting ratings from 17 participants on five metrics (realism, freedom of movement, speed, comfort, and ease of memory).
  • The object retrieval experiment showed that users found the gesture-based object imitation technique more realistic and engaging. The interaction control study supported its superior performance in dynamic interactions.

• Limitations and Future Directions:

  • Limitations: The detection system is sensitive to hand occlusion and performs inadequately for extremely small or large virtual objects. Excessive or overly similar gesture designs may affect memory and usability.
  • Future Directions:
    1. Research automated sensitivity matrix generation algorithms to accommodate complex interactions.
    2. Explore the combination of imitation and experiential interaction, such as enhancing immersion by adjusting object shapes or implementing visual changes.
    3. Expand the expressive space of hand interaction design, such as incorporating animations or extending the gesture library to support user customization.

This study clearly proposes an innovative interaction model, enriching the possibilities for AR/VR application design and providing valuable experimental data and open resources for future technological development.