User-Driven Constraints for Layout Optimisation in Augmented Reality

Document Title

User-Driven Constraints for Layout Optimization in Augmented Reality

Document Information

• Subject Area: Layout optimization and user interaction in augmented reality
• Keywords: augmented reality, user-defined constraints, layout optimization, gesture interaction, human-computer interface, interaction design, automation, spatial awareness, constraint optimization system

Research Background and Problem

• Problem or Challenge:

  • Manual arrangement of virtual content in augmented reality (AR) environments is complex and time-consuming.
  • Existing automated layout optimization methods often lack direct user control over the optimization results.
  • Current AR system user interfaces (e.g., menus and controls) disrupt the natural flow of user interaction, negatively impacting the immersive experience.

• Significance:

  • AR technology has reached a mature stage, with advancements in environmental tracking, display quality, and gesture recognition, but user interfaces remain a bottleneck for efficient virtual content arrangement.
  • Integrating human-computer interaction into optimization systems can enhance user satisfaction and trust in the optimization results.

• Research Motivation:

  • To provide an integrated, user-driven constraint framework that incorporates user participation in AR layout optimization, simplifying the arrangement of virtual content while maintaining user-defined flexibility.
  • To explore a fluid and intuitive interaction method that does not rely on traditional graphical user interfaces (GUIs).

Solution

• Method or Solution:

  • A user-driven constraint layout optimization method is proposed, allowing users to set constraints intuitively through gestures in real-world environments.
  • A design space is constructed, encompassing constraint types (e.g., attraction edges, repulsion edges, containment surfaces), focus areas (points, lines, surfaces, volumes), and constraint parameters (weights, minimum distances, priorities).

• Innovations:

  • Designed and validated a user-driven constraint framework for the first time, seamlessly integrating user interaction with the layout optimization process.
  • Introduced gesture-based constraint definition and optimization, offering an AR interaction design method free from traditional GUIs.
  • Incorporated voice input to enable more natural semantic constraint settings.

• Implementation Steps and Techniques:

  1. Design Space Definition:
     • User-driven constraints: attraction boundaries, repulsion boundaries, containment surfaces, exclusion zones, field-of-view areas, priority regions, user perspectives, semantic constraints.
     • Focus areas: points, lines, surfaces, volumes, etc.
     • Parameterization: including weights, distances, perspectives, and other dynamic variables.
  2. User Gesture Design and Validation:
     • Conducted a semantic gesture study to collect user-defined gestures, avoiding traditional GUIs.
     • Validated intuitiveness, learnability, and social acceptability.
  3. Implementation and Optimization:
     • Implemented gesture recognition and layout optimization functions based on Hololens 2 and the MRTK framework.
     • Used SolverHandler and SurfaceMagnetism to handle content parameter constraints and dynamic optimization.
  4. User Testing and Evaluation:
     • Provided two comparison methods: manual placement and user-driven constraints, and progressively tested functional performance.

Research Outcomes

• Specific Outcomes:

  • Developed an operational gesture interaction system capable of creating and optimizing various constraint types.
  • Offered a fluid and user-controllable AR content layout optimization method.

• Advantages:

  • Compared to traditional manual arrangement or automated optimization systems, user-driven constraints significantly reduced physical and time demands.
  • Interaction was more natural, allowing users to easily define personalized layouts.

• Experimental or Evaluation Results:

  • Average gesture intuitiveness and acceptability were high, particularly for "attraction edge" and "containment surface" gestures.
  • User-driven constraints significantly reduced time and physical effort compared to manual methods.
  • User feedback indicated that the system was overall efficient and easy to use.

• Limitations and Future Directions:

  1. Limitations:

     • The current implementation supports only a limited number of virtual content items.
     • Gesture recognition technology is constrained by hardware (e.g., Hololens 2).
     • Users still require an adaptation period to become familiar with complex constraint interactions.

  2. Future Directions:

     • Enhance the accuracy of gesture recognition and area definition, such as supporting palm or larger-scale operations.
     • Expand user-driven constraints to support 3D virtual content and more complex layouts, such as large-scale windows or intricate interactive interfaces.
     • Develop more robust 3D optimization algorithms to address multi-constraint conflict issues.
     • Explore new interaction methods (e.g., touch and pen interactions) to meet more flexible user needs.