MechARspace: An Authoring System Enabling Bidirectional Binding of AR with Toys in Real-time

Title of the Paper

MechARspace: An Authoring System Enabling Bidirectional Binding of Augmented Reality with Toys in Real-time

Paper Information

• Subject Area: Interaction design and implementation of augmented reality (AR) with toys
• Keywords: Augmented reality, immersive authoring, demonstration-based programming, modular IoT toolkit, bidirectional interaction, toy augmentation, visual programming, human-computer interaction

Research Background and Problem

• Identified Problems or Challenges:

  • The creation of AR content is primarily carried out by professionals, with high technical barriers limiting participation by non-experts.
  • Existing tools mostly focus on unidirectional interactions (e.g., physical triggers for virtual content) and lack a unified framework for bidirectional interaction between toys and AR.
  • Traditional development tools (e.g., Unity3D) separate the programming and testing environments, making the development process time-consuming and complex.

• Significance:

  • Applying AR technology to traditional toys has the potential to significantly enhance their interactivity and entertainment value.
  • Empowering ordinary users to rapidly prototype personalized AR toys can stimulate creativity and promote the democratization of toy design.

• Research Motivation and Related Work:

  • Current AR-toy interaction systems are often tailored for specific applications, making it difficult to support personalized toy designs by general users.
  • While some immersive authoring systems support dynamic interactions, they are mostly limited to unidirectional content augmentation.
  • Modular IoT toolkits can simplify the process of creating electronic prototypes for physical toys, but their integration with AR requires further exploration.

Solution

• Proposed Solution:

  • MechARspace is a system that enables users to create toy-AR interactions through direct manipulation and visual programming in an immersive environment, including:
    • A bidirectional physical-virtual interaction model.
    • A plug-and-play IoT toolkit for adding AR compatibility to traditional toys.
    • A real-time scene authoring interface based on holographic headsets (Hololens 2).

• Innovations:

  • A unified bidirectional physical-virtual interaction design framework is proposed, covering interaction pathways from physical to virtual and vice versa.
  • A modular IoT toolkit is developed to enable rapid integration and responsive bidirectional interaction.
  • A user interface design based on demonstration-based programming and trigger-action visual logic is implemented, lowering the entry barrier for non-expert users.

• Implementation Steps and Techniques:

  1. Interaction Framework: An input-output classification model is proposed, including four input types (e.g., spatial placement, manipulation) and four output types (e.g., local motion, sensory effects), connected through bidirectional trigger-action logic.
  2. Toolkit Design: Eight IoT modules (e.g., rotation module, button module) with built-in sensors and networking capabilities communicate with Hololens via the MQTT protocol.
  3. Authoring Interface: Provides import mode (importing physical and virtual content), edit mode (defining input/output behaviors), and play mode (testing interactions). All data is recorded and quickly linked through user demonstration operations.
  4. Implementation Technology: The system is developed using Unity3D, leveraging the Microsoft Mixed Reality Toolkit (MRTK) and Vuforia for synchronizing physical and virtual scenes.

Research Outcomes

• Specific Achievements:

  • A comprehensive authoring system for interactions between virtual and physical objects was implemented.
  • Multiple application cases were demonstrated, including story toy design, toyification of everyday objects, micro-toy augmentation, and multiplayer games, validating the system's versatility and scalability.

• Advantages Compared to Existing Solutions:

  • Provides a unified design framework supporting bidirectional interaction, addressing the limitations of existing unidirectional interaction systems.
  • Significantly lowers the technical barriers to AR content creation through the IoT toolkit and visual interface.

• Experiment and Evaluation Results:

  • Two rounds of user studies evaluated the system's usability and effectiveness:
    1. In the first round of task testing, users completed six types of interactions, with high overall satisfaction (SUS score of 83/100).
    2. In the second round of open-ended creation, participants quickly designed and implemented personalized AR toys, reporting that the system simplified the creation process and enhanced the fun of creation.

• Limitations and Future Directions:

  • The interaction logic currently supports only simple trigger-action mappings; future work could expand to more complex logic (e.g., conditional triggers, parallel tasks).
  • Virtual content is preloaded; future iterations should support real-time import of external resources.
  • Toy tracking relies on physical markers, which are prone to occlusion; future research should explore marker-tracking fusion or markerless technologies.
  • IoT modules are relatively large, making them unsuitable for small toys; future efforts could focus on optimizing hardware integration via CAD plugins.

Conclusion

MechARspace provides a novel paradigm for AR toy interaction authoring, combining a bidirectional interaction model, modular hardware, and immersive visual programming. It expands the possibilities for ordinary users in AR application design and offers significant insights and references for the HCI field in exploring next-generation AR toy creation tools.