ProObjAR: Prototyping Spatially-aware Interactions of Smart Objects with AR-HMD

Document Title

ProObjAR: Prototyping Spatially-aware Interactions of Smart Objects with AR-HMD

Document Information

• Domain: Human-Computer Interaction, Augmented Reality Technology, Spatial Interaction Design
• Keywords: Spatial interaction, smart objects, AR prototyping, augmented reality, interaction design tools

Research Background and Problem Statement

• What issues or challenges did the authors identify?

  • With technological advancements, interaction modes for smart objects are increasingly transcending traditional digital screens and incorporating spatial interaction elements. However, early-stage prototyping for spatial interactions of smart objects faces challenges such as poor adaptability to complex scenarios, lack of integrated tools, and high technical barriers.
  • Traditional prototyping methods, such as video demonstrations or physical models, struggle to fully present dynamic and interactive characteristics, while code-based design approaches hinder non-technical designers from quickly getting started.

• Why is this problem important?

  • As smart ecosystems become more prevalent, spatial interactions are increasingly applied in multi-scenario designs, such as information exchange and control functions between devices. This interaction mode is a critical step in enhancing the efficiency of digital content and physical facility interactions.
  • Improving prototyping efficiency can help designers quickly validate concepts and facilitate the implementation of innovative designs.

• Research Motivation and Related Work

  • Motivation: Address the inability of existing design tools to meet spatial interaction needs and provide a simplified, integrated solution for multi-scenario design through AR devices.
  • Related Work: Research spans interaction tool development in AR/VR, IoT interaction design, and applications of visual programming interfaces. However, most focus on virtual or static object interactions, with limited exploration of dynamic spatial behaviors.

Solution

• What methods or solutions did the authors propose?

  • The authors proposed a novel prototyping system called ProObjAR, which supports spatial interaction design for smart objects using AR head-mounted display devices (AR-HMD).
  • The system is based on an input-output event-trigger workflow framework, enabling the definition and testing of spatial events (input), virtual effects (output), and their relationships for smart objects. It also utilizes a visual programming interface to lower the learning and operational barriers.

• What are the innovative aspects of the solution?

  • Introduces a spatial interaction design model centered on real object manipulation, leveraging AR technology to achieve real-time interaction and visualization in prototyping.
  • Provides multiple classifications of spatial events based on logical operators (e.g., "AND," "OR," "NOT") and diverse virtual feedback effects, covering complex spatial interaction scenarios involving single and multiple objects.
  • Eliminates reliance on recoding and fixed hardware by enabling real-time acquisition of objects' 6 degrees of freedom poses (position and orientation), allowing designers to directly design and test interaction modes in real-world scenarios.

• What are the implementation steps and key technologies used?

  1. Spatial Information Acquisition: Utilize AR marker-based tracking technology to determine objects' poses in real time.
  2. Input Event Definition: Create discrete, synchronous, and gradient events for single or multiple objects, such as position changes and orientation changes.
  3. Output Effect Definition: Define virtual feedback effects using the system's asset library or free sketching functionality, including appearance, disappearance, synchronized motion, etc.
  4. Event-Effect Mapping: Specify the trigger relationships between input events and output effects by dragging and connecting lines in the visual programming interface.
  5. Prototype Testing: Trigger interactions through real object manipulation and observe the effects in real time to verify the design's rationality and naturalness.

Research Outcomes

• What specific outcomes were achieved?

  • The ProObjAR system significantly reduced the design barriers for spatial interaction prototyping, enabling users to quickly learn and efficiently complete interaction designs for complex scenarios.
  • Users could freely define and test spatial interaction modes, and the system supported real-time prototyping in various scenarios, including indoor and outdoor environments.

• How does it compare to existing solutions?

  • Supports real object manipulation instead of entirely virtual design, addressing spatial relationships and interaction experiences in real-world scenarios.
  • Outperforms existing AR/VR design tools in terms of coding barriers, adaptability to complex scenarios, and support for large-scale event types.

• What were the experimental or evaluation results?

  • In an initial usability study, 8 participants used ProObjAR to create 21 prototype designs in indoor and outdoor scenarios, covering applications such as device control, information exchange, and health management.
  • Participants were able to complete the creation and testing of a single design within approximately 2-8 minutes on average, with high satisfaction levels, finding the system intuitive and easy to use.

• Limitations and Future Directions

  • Limitations:
    • The types of supported spatial events remain limited, such as the ability to define complex spatial regions or non-uniform motion events.
    • Tracking performance is low in environments with reflective surfaces or occlusions.
    • Currently supports single-user design only, lacking multi-user collaborative prototyping functionality.
  • Future Work:
    • Expand spatial event classifications to provide designers with customizable region and angle interaction definitions.
    • Enhance marker recognition algorithms to improve tracking accuracy in complex environments.
    • Implement hybrid design for virtual objects and introduce real-time collaborative design features to support teamwork.
    • Recruit a more diverse sample for in-depth studies, including non-professional users and industry designers.