GesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications

Title of the Paper

GesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications

Paper Information

• Domain: Freehand gesture interaction and application authoring in Augmented Reality (AR)
• Keywords: Freehand interaction, immersive authoring, augmented reality, gesture demonstration, visual programming

Research Background and Problem Statement

• Problems and Challenges:
  1. Freehand gestures are considered an intuitive interaction method for AR, but developing AR applications that support custom gestures remains highly challenging.
  2. Many existing studies focus on predefined interaction methods, failing to capture the complexity and diversity of gesture interactions in daily life.
  3. Current AR development tools (e.g., Unity3D and Unreal) have steep learning curves and are cumbersome for non-professional users.
• Significance of the Problem:
  • Freehand interaction not only enhances the sense of immersion but also allows users to intuitively manipulate virtual objects through gestures, offering more possibilities for fields such as education, industrial design, and entertainment.
• Motivation and Related Work:
  • Using gesture demonstration to intuitively generate gesture data without requiring in-depth knowledge of algorithms, thereby lowering the development barrier.
  • Combining immersive visual programming to allow users to instantly create and test AR interactive applications.
  • Integrating gesture classifications summarized from existing literature and proposing an interaction model linking inputs (gestures) and outputs (virtual object behaviors).

Proposed Solution

• Proposed Method:
  • Designed an end-to-end AR application authoring system, GesturAR, enabling users to create interactive freehand gesture AR applications through gesture demonstration and visual programming.
  • Proposed an interaction model based on freehand gestures, defining mapping rules between gesture inputs and virtual content behaviors.
  • Developed a real-time gesture recognition algorithm based on one-shot learning and time-series analysis.
• Innovations:
  1. Introduced a gesture demonstration authoring method in AR to lower the development barrier.
  2. Proposed a gesture-based input-output interaction model in AR, covering various types of static and dynamic inputs and responses.
  3. Adopted a "trigger-action" model as the interaction design logic, allowing users to create complex interactions through simple component connections.
• Implementation Steps and Techniques:
  1. Users define gestures through demonstration in an AR environment, with real-time recording of hand joint data.
  2. Introduced a visual programming interface where users can generate interaction logic by dragging and connecting triggers and action icons.
  3. Utilized a real-time gesture detection algorithm to instantly test the AR interactions created by users in Play mode.

Research Outcomes

• Specific Outcomes:
  • Developed the GesturAR system, enabling users to create freehand gesture interactive AR applications without professional coding skills.
  • Built various application scenarios, such as interactive virtual objects, robots and avatars, room-scale interactive games, and immersive AR presentations.
  • User studies showed a gesture detection accuracy rate of over 95%, with high user satisfaction and system usability scores (SUS score of 86/100).
• Advantages:
  • Supports personalized development for non-professional users, significantly lowering the barrier to AR application creation.
  • Implements the trigger-action model and visual logic connectors, allowing users to quickly get started and build complex gesture interaction logic.
  • Real-time authoring and testing mechanisms significantly improve development efficiency and user experience.
• Experimental or Evaluation Results:
  1. Static gesture detection achieved an F1 score of 95.93%, and dynamic gesture detection accuracy reached 91.67%.
  2. In user testing, 94.44% of interaction authoring and application experience tasks were successfully completed on the first attempt.
  3. Users positively evaluated the intuitiveness of the authoring process and the comprehensiveness of system functionality.
• Limitations and Future Directions:
  1. The current lack of haptic feedback mechanisms affects the immersion of freehand interactions.
  2. Differences in user-defined gestures and other users' operational preferences increase the complexity of application sharing.
  3. More sophisticated trigger and action mapping logic, such as conditional triggers, action delays, and multi-step chained reactions, remains to be developed.
  4. Accuracy is still constrained by hardware performance improvements (e.g., recognition capabilities under gesture occlusion).